Given n feature vectors x 1=(x 11,…,x 1p ),..., x n =(x n1,…,x np ) of size p and a vector of class labels y=(y 1,…,y n ), where y i ∈ K = {0, ..., J-1} describes the class to which the feature vector x i belongs and J is the number of classes, the problem is to build a multi-class LogitBoost classifier.
Training Stage
The LogitBoost model is trained using the Friedman method [Friedman00].
Let y i,j = I {x i ∈ j} is the indicator that the i-th feature vector belongs to class j. The scheme below, which uses the stump weak learner, shows the major steps of the algorithm:
-
Start with weights w ij = 1/n, F j (x) = 0, p j (x) = 1/J, i = 1,...,n, j =0,...,J-1
-
For m=1,...,M
Do
- For j = 1,...,J
- Do
-
(i) Compute working responses and weights in the j-th class:
w ij = p i (x i )(1-p i (x i )), w ij = max(z ij ,Thr1)
z ij = (y ij - p i (x i )) /w ij , z ij = min(max(z ij ,-Thr2),Thr2)
- (ii) Fit the function f mj (x) by a weighted least-squares regression of z ij to x i with weights w ij using the stump-based approach.
-
-
End do
End do
The result of the model training is a set of M stumps.
Prediction Stage
Given the LogitBoost classifier and r feature vectors x 1,…,x r , the problem is to calculate the labels argmax j F j (x) of the classes to which the feature vectors belong.