• Space of outputs \(\mathcal{Y}\) is finite. Often classes are given numbers starting from \(0\) or \(1\).

• Usually no notion of “similarity” between class labels in terms of loss. Remember our loss function \(\ell(h(\mathbf{x}),y)\):
  • Regression: \(\ell(9,10)\) is better than \(\ell(1,10)\)
  • Classification: \(\ell(9,10)\) and \(\ell(1,10)\) are equally bad.
    • Or, have explicit losses for every combination of predicted class and actual class.

• By linear models, we mean that the hypothesis function \(h_{\bf w}({\bf x})\) is a (transformed) linear function of the parameters \({\bf w}\).

• Predictions are a (transformed) linear combination of feature values

\[h_{\bf w}(\mathbf{x}) = g\left(\sum_{k=0}^{p} w_k \phi_k(\mathbf{x})\right) = g(\boldsymbol{\phi}(\mathbf{x})^\mathsf{T}{{\mathbf{w}}})\]

• again, \(\phi_k\) are called basis functions or feature functions As usual, we let \(\phi_0(\mathbf{x})=1, \forall \mathbf{x}\), so that we don’t force \(h_{\bf w}(0) = 0\)

• Classification tasks

• Loss functions for classification

• Logistic Regression

• Support Vector Machines

• Univariate real input: nucleus size
• Output coding: non-recurrence = 0, recurrence = 1
• Sum squared error minimized by the blue line

• The predictor shows an increasing trend towards recurrence with larger nucleus size, as expected.

• Output cannot be directly interpreted as a class prediction.

• Thresholding output (e.g., at 0.5) could be used to predict 0 or 1.
  (In this case, prediction would be 0 except for extremely large nucleus size.)

• Suppose we have two possible classes: \(y\in \{0,1\}\).

• The symbols “\(0\)” and “\(1\)” are unimportant. Could have been \(\{a,b\}\), \(\{\mathit{up},\mathit{down}\}\), whatever.

• Rather than try to predict the class label directly, ask:
  What is the probability that a given input \(\mathbf{x}\) has class \(y=1\)?

## Mean: 70.90

• If I know eruption time, can I do better?

## Mean: 55.60

• If I know eruption time, can I do better?

## Mean: 81.33

\[\varsigma(x) = \frac{1}{1+e^{-x}}\]

• How do we choose \({\bf w}\)?

• In the probabilistic framework, observing \(\langle \mathbf{x}_i , 1 \rangle\) does not mean \(h_\mathbf{w}(\mathbf{x}_i)\) should be as close to \(1\) as possible.

• Maximize probability the model assigns to the \(y_i\) in the training set given the \(\mathbf{x}_i\) by adjusting \(\mathbf{w}\).

• Two random variables \(X\) and \(Y\) that are part of a random vector are independent iff:

\[ F_{X,Y}(x,y) = F_X(x)F_Y(y) \]

If they have a joint density or joint PMF, then

\[ f_{X,Y}(x,y) = f_X(x)f_Y(y) \]

• Why are these so close?
• Recall the expected value of a discrete random variable \(Y\) is denoted

\[\mathrm{E}[Y] = \sum_{y \in \mathcal{Y}} y \cdot p_Y(Y = y)\]

• Consider a random variable \(Y \in \{0,1\}\)

\(\begin{aligned} \mathrm{E}[Y] & = \sum_{y \in \{0,1\}} y \cdot p_Y(Y = y)\\ & = 0 \cdot p_Y(Y = 0) + 1 \cdot p_Y(Y = 1)\\ & = p_Y(Y = 1) \end{aligned}\)

• Though we did not discuss in this way, linear regression tries to estimate the function \(\mathrm{E}[Y | X = x]\). So, makes sense that the OLS and logistic regression answers can be close.

• Both model the conditional mean of \(y\) using a (transformed) linear function
• Both use maximum conditional likelihood to estimate

• How complicated is a classifier?

• One way to think about it is in terms of its decision boundary, i.e. the line it defines for separating examples

• Linear classifiers draw a hyperplane between examples of the different classes. Non-linear classifiers draw more complicated surfaces between the different classes.

• For a probabilistic classifier with a cutoff of 0.5,
  the decision boundary is the curve on which: \[\frac{P(y=1|\mathbf{X}= \mathbf{x})}{P(y=0|\mathbf{X}= \mathbf{x})} = 1, \mbox{i.e., where } \log\frac{P(y=1|\mathbf{X}= \mathbf{x})}{P(y=0|\mathbf{X}= \mathbf{x})} = 0\]

• For logistic regression, this this is where \(\mathbf{x}^\mathsf{T}\mathbf{w}= 0\).

• Recall: predictions are a (transformed) linear combination of feature values

\[h_{\bf w}(\mathbf{x}) = g(\mathbf{x}^\mathsf{T}{{\mathbf{w}}})\]

• Suppose our decision boundary is \[h_{\bf w}(\mathbf{x}) = c\]

• This is equivalent to \[\mathbf{x}^\mathsf{T}{{\mathbf{w}}} = c'\]

where \(c' = g^{-1}(c)\).

Class = R if \(\mathrm{Pr}(Y=1|X=x) > 0.5\)

Class = R if \(\mathrm{Pr}(Y=1|X=x) > 0.25\)

Class = R if \(\mathrm{Pr}(Y=1|X=x) > 0.5\)

Class = R if \(\mathrm{Pr}(Y=1|X=x) > 0.25\)

Linear Support Vector Machines

• Linear classifiers that focus on learning the decision boundary rather than the conditional distribution \(P(Y=y|\mathbf{X}=\mathbf{x})\)

  • Perceptrons

    • Definition

    • Perceptron learning rule

    • Convergence

  • “Margin” idea and max margin classifiers

  • (Linear) support vector machines

    • Formulation as optimization problem

• Consider a binary classification problem with data \(\{{{\mathbf{x}_i}},y_i\}_{i=1}^n\), \(y_i\in\{-1,+1\}\). Note coding of \(y_i\).

• A perceptron (Rosenblatt, 1957) is a classifier of the form: \[h_{{{\mathbf{w}}},w_0}({{\mathbf{x}}}) = \mbox{sign}(\mathbf{x}^\mathsf{T}\mathbf{w}+ w_0) = \left\{ \begin{array}{ll} +1 & \mathrm{if}~ \mathbf{x}^\mathsf{T}\mathbf{w}+ w_0\geq 0 \\ -1 & \mathrm{otherwise} \end{array} \right.\] Here, \({{\mathbf{w}}}\) is a vector of weights, and \(w_0\) is a constant offset. (Note \(x_0 = 1\) is omitted.)

• The decision boundary is \(\mathbf{x}^\mathsf{T}\mathbf{w}+ w_0= 0\).

• Perceptrons output a class, not a probability

• An example \(( {{\mathbf{x}}}, y )\) is classified correctly if: \[y \cdot (\mathbf{x}^\mathsf{T}\mathbf{w}+ w_0) > 0\]

• The data set is linearly separable if and only if there exists \({{\mathbf{w}}}\), \(w_0\) such that:

  • For all \(i\), \(y_i(\mathbf{x}_i^\mathsf{T}\mathbf{w}+w_0)>0\).

  • Or equivalently, the 0-1 loss \(\sum_i \mathbf{1}_{y_i(\mathbf{x}_i^\mathsf{T}\mathbf{w}+w_0) < 0}\) is zero for some set of parameters \(({\bf w}, w_0)\).

• Consider the following procedure:

  1. Initialize \({{\mathbf{w}}}\) and \(w_0\) randomly

  2. While any training examples remain incorrecty classified

     1. Loop through all misclassified examples

     2. For misclassified example \(i\), perform the updates: \[{{\mathbf{w}}}\gets {{\mathbf{w}}}+ \delta y_i{{\mathbf{x}}}_i,~~~~~w_0\gets w_0 + \delta y_i\] where \(\delta\) is a step-size parameter.

• The update equation, or sometimes the whole procedure, is called the perceptron learning rule.

• Intuition: For positive examples misclassified as negative, change \({{\mathbf{w}}}\) to increase \(\mathbf{x}_i^\mathsf{T}\mathbf{w}+w_0\), and vice versa

• PLR can be interpreted as a gradient descent on the following function: \[{{J}}({{\mathbf{w}}},w_0) = \sum_{i=1}^n \left\{ \begin{array}{ll} 0 & \mathrm{if}~ y_i(\mathbf{x}_i^\mathsf{T}\mathbf{w}+ w_0)\geq 0 \\ -y_i(\mathbf{x}_i^\mathsf{T}\mathbf{w}+ w_0) & \mathrm{if}~ y_i(\mathbf{x}_i^\mathsf{T}\mathbf{w}+ w_0)<0 \end{array}\right.\]

• For correctly classified examples, the error is zero.

• For incorrectly classified examples, the error is by how much \(\mathbf{x}_i^\mathsf{T}\mathbf{w}+w_0\) is on the wrong side of the decision boundary.

• \(J\) is piecewise linear, so it has a gradient almost everywhere; stochastic gradient descent gives the perceptron learning rule.

• \(J\) is zero if and only if all examples are classified correctly – just like the 0-1 loss function.

• If classes are linearly separable then the perceptron learning rule will find a separater after some finite number of updates.

• The number of updates depends on the data set, and also on the step size parameter.

• If the classes are not linearly separable, there will be oscillation (which can be detected automatically).

• Recall percepton learning rule: \[{{\mathbf{w}}}\gets {{\mathbf{w}}}+ \delta y_i{{\mathbf{x}}}_i,~~~~~w_0\gets w_0 + \delta y_i\]

• If initial weights are zero, then at any step, the weights are a linear combination of feature vectors of the examples: \[{{\mathbf{w}}}= \sum_{i=1}^n \alpha_i y_i {{\mathbf{x}_i}},~~~~~w_0 =\sum_{i=1}^n \alpha_i y_i\] where \(\alpha_i\) is the sum of step sizes used for all updates based on example \(i\).

• This is called the dual representation of the classifier.

• Even by the end of training, some examples may have never participated in an update, just by chance. So their corresponding \(\alpha_i=0\).

• Perceptrons can be learned to fit linearly separable data, using a gradient descent rule.

• Blindingly fast

• Solutions are non-unique

• Support vector machines (SVMs) for binary classification can be viewed as a way of training perceptrons

• Three main new ideas:

  • A optimization criterion (the “margin”) guarantees uniqueness and has theoretical advantages

  • Natural handling nonseparable data by allowing mistakes

  • An efficient way of operating in expanded feature spaces: “kernel trick”

• SVMs can also be used for multiclass classification and regression.

• Consider a linearly separable binary classification data set

• There is an infinite number of hyperplanes that separate the classes:

• Which plane is best?

• For a given plane, for which points should we be most confident in the classification?

• For a given separating hyperplane, the margin is two times the (Euclidean) distance from the hyperplane to the nearest training example.

  

• Width of the “strip” around the decision boundary containing no training examples.

• A linear SVM is a perceptron for which we choose \({{\mathbf{w}}},w_0\) so that margin is maximized

• Suppose we have a decision boundary that separates the data.

  

• Let \(\gamma_i\) be the distance from instance \({{\mathbf{x}_i}}\) to the decision boundary.

• How can we write \(\gamma_i\) in terms of \({{\mathbf{x}_i}}, y_i, {{\mathbf{w}}}, w_0\)?

• The margin of the hyperplane is \(2M\), where \(M=\min_i y_i \gamma_i\)

• The most direct statement of the problem of finding a maximum margin separating hyperplane is thus

  \[\max_{\mathbf{w},w_0} \min_i y_i \gamma_i\]

  \[\equiv \max_{\mathbf{w},w_0} \min_i y_i\frac{\mathbf{x}_i^\mathsf{T}\mathbf{w}+ w_0}{||\mathbf{w}||}\]

• This turns out to be inconvenient for optimization, however

• From the definition of margin, we have: \[M \leq y_i \gamma_i = y_i \frac{\mathbf{x}_i^\mathsf{T}\mathbf{w}+ w_0}{||\mathbf{w}||} ~~~~\forall i\]

• This suggests:
  

• Problems:

  • \({{\mathbf{w}}}\) appears nonlinearly in the constraints.

  • This problem is underconstrained. If \(({{\mathbf{w}}},w_0,M)\) is an optimal solution, then so is \((\beta{{\mathbf{w}}},\beta w_0,M)\) for any \(\beta>0\).

Let’s add the constraint that \(M = 1 / \|{{\mathbf{w}}}\|\):

• This allows us to rewrite the objective function:

• This is really nice because the constraints are linear.

which is the same as

• Let’s minimize \(\frac{1}{2} \|{{\mathbf{w}}}\|^2\) instead of maximizing \(\frac{1}{||\mathbf{w}||}\). (Taking the square is a monotone transformation, as \(\|{{\mathbf{w}}}\|\) is postive, so this doesn’t change the optimal solution.)

• This gets us to:

  minimize	\(\frac{1}{2} \|{{\mathbf{w}}}\|^2\) w.r.t. \({{\mathbf{w}}}, w_0\)
  subject to	\(y_i(\mathbf{x}_i^\mathsf{T}\mathbf{w}+w_0)\geq1\)

• This we can solve! How?

  • It is a convex quadratic programming (QP) problem—a standard type of optimization problem for which many efficient packages are available.

We have a solution, but no “support vectors” yet…

• Turns out (HTF Ch. 4.5.2) we can write: \[{\bf w}=\sum_i \alpha_i y_i \mathbf{x}_i,~~\mbox{where $\alpha_i \ge 0$}\]

• As for the perceptron with zero initial weights, the optimal solution for \({{\mathbf{w}}}\) and \(w_0\) is a linear combination of the \({{\mathbf{x}_i}}\).

• The output is therefore:

  \[h_{\mathbf{w},w_0}(\mathbf{x}) = \mbox{sign} \left(\sum_{i=1}^n \alpha_i y_i ({{\mathbf{x}_i}}\cdot {{\mathbf{x}}}) +w_0\right)\]

• Output depends on weighted dot product of input vector with training examples

• We can actually solve directly for the \(\alpha_i\) (again see HTF Ch. 4.5.2): \[\max_{{{\boldsymbol{\alpha}}}} \sum_{i=1}^n \alpha_i -\frac{1}{2} \sum_{i=1}^n \sum_{j=1}^n y_i y_j \alpha_i \alpha_j (\mathbf{x}_i \cdot \mathbf{x}_j)\] with constraints: \(\alpha_i \geq 0 \mbox{ and} \sum_i \alpha_i y_i =0\)

• This is also a QP

• Suppose we find optimal \({{\boldsymbol{\alpha}}}\)s (e.g., using a standard QP package)

• The \(\alpha_i\) will be \(>0\) only for the points for which \(y_i(\mathbf{x}_i^\mathsf{T}\mathbf{w}+ w_0)=1\)

• These are the points lying on the edge of the margin, and they are called support vectors, because they define the decision boundary

• The output of the classifier for query point \(\mathbf{x}\) is computed as: \[\mbox{sgn}\left[\left(\sum_{i=1}^n \alpha_i y_i (\mathbf{x}_i \cdot \mathbf{x})\right) + w_0 \right]\] Hence, the output is determined by computing the dot product of the point with the support vectors

Support vectors are in bold

• SVMs are a state-of-the-art for classification when you don’t need probability estimates

• Inuitively, the large-margin property makes sense. Theory backs this up.

• SVMs offer “off-the-shelf” non-linear classification without having to do explicit feature construction, as we will see.

• Recall that in the linearly separable case, we compute the solution to the following optimization problem:

  min	\(\frac{1}{2}\|{{\mathbf{w}}}\|^2\)	w.r.t. \({{\mathbf{w}}}, w_0\)
  s.t.	\(y_i(\mathbf{x}_i^\mathsf{T}\mathbf{w}+ w_0)\geq1\)

• What if we can’t satisfy the constraints?

• To allow misclassifications, we relax the constraints to: \[y_i(\mathbf{x}_i^\mathsf{T}\mathbf{w}+ w_0) \geq 1-\xi_i\]

• If \(\xi_i \in (0,1)\), the data point is within the margin

• If \(\xi_i \geq 1\), then the data point is misclassified

• We define the soft error as \(\sum_i \xi_i\); each \(\xi_i\) is a slack variable

• Instead of:

  min	\(\frac{1}{2}\|{{\mathbf{w}}}\|^2\) w.r.t. \({{\mathbf{w}}}, w_0\)
  s.t.	\(y_i(\mathbf{x}_i^\mathsf{T}\mathbf{w}+ w_0)\geq1\)

  we want to solve:

  min	\(\frac{1}{2}\|{{\mathbf{w}}}\|^2+ C \sum_i \xi_i\) w.r.t. \({{\mathbf{w}}}, w_0, \xi_i\)
  s.t.	\(y_i(\mathbf{x}_i^\mathsf{T}\mathbf{w}+ w_0)\geq1-\xi_i\), \(\xi_i \geq 0\)

• Note that soft errors include points that are misclassified,
  as well as points within the margin

• There is a linear penalty for both categories

• The choice of the constant \(C\) controls boundary-fitting

• If \(C\) is very small, there is almost no penalty for soft errors, so the focus is on maximizing the margin, even if this means more mistakes

• If \(C\) is very large, the emphasis on the soft errors will decrease the margin, if this helps to classify more examples correctly.

• How could we choose \(C\)?

• Like before, we can formulate a “dual” problem that identifies the support vectors:

• All the previously described machinery can be used to solve this problem