Perceptron Algorithm for Classification
In classification tasks, we are given a training set T={X,t}, where X={xi} is the set of training data vectors and t={ti} is the set of vector labels.
Linear classifiers use linear functions to model the mapping from input vector to output labels, that is, the classifier is defined as a function of linear combination of the input vector
t=f(\sum_i x_i \theta_i) = f(x*\theta)
In binary classification problems, $f$ is usually a sign function where
f(y) = { +1, y >= 0;
{ -1, y < 0.
Therefore
x * \theta = 0
defined a decision plane, theta is the normal vector of this plane. So the problem turns into finding the optimal decision plane, i.e. optimal theta to separate the training points according to their labels.
We can use the number of classification mistakes as the evaluation metric for all the planes. Other metrics can also be used here. It is easy to observe that when we make a mistake t * f(x * \theta) < 0 which is equivalent to t x * \theta. So we hope to increase t x * \theta for all points that are mistakenly classified. Therefore, we maximize the loss function
L = \sum_i t_i x_i * \theta
The algorithm is to adapt theta for each mistakenly classified test point to increase the above target function. That is
1. Repeat until convergence
2. For each training point (x, t)
3. If tx * theta < 0 (i.e. there is a mistake)
4. \theta <-- \theta + tx
After each adaptation tx * theta will increase because
tx*(\theta + tx) = tx * \theta + ||x||^2
> tx * \theta
This is the perceptron algorithm.
After each adaptation, the \theta may be adjusted to perform better on the mistakenly classified training point, but other new mistakes may also appear. It is proved that this perceptron algorithm will converge when the points are linearly separable. Otherwise this algorithm may never converge.
