Speeding Up Kernel Methods, and Intro To Unsupervised Learning

Speeding Up Kernel Methods, and
Intro to Unsupervised Learning
Piyush Rai
Introduction to Machine Learning (CS771A)
September 11, 2018
Intro to Machine Learning (CS771A) Speeding Up Kernel Methods, and Intro to Unsupervised Learning 1
Recap: Nonlinear Mappings
Idea: Use a nonlinear mapping φ : RD → RM to map original data to a high-dim space, e.g.,
Learn a linear model in the new space using the mapped inputs φ(x 1 ), . . . , φ(x N )
Equivalent to learning a nonlinear model on the original data x 1 , . . . , x N
The mappings can be explicitly defined, or implicitly defined via a kernel function k, s.t.
k(x n , x m ) = φ(x n )> φ(x m )
k(x n , x m ) = φ(x n )> φ(x m )
Benefit of using kernels: Don’t need to explicitly compute the mappings (M can be very large)
k(x n , x m ) = φ(x n )> φ(x m )
Benefit of using kernels: Don’t need to explicitly compute the mappings (M can be very large)
Many ML algos only have data appearing as inner products. Can kernelize such algos
Recap: Nonlinear Mappings and Kernels
A kernel function k(x n , x m ) = φ(x n )> φ(x m ) defines inner-product similarity between two inputs
This is a Euclidean similarity in φ space but a “nonlinear” similarity in original space
Some popular examples of kernel functions
k(x n , x m ) = x>
n xm (Linear kernel)
k(x n , x m ) = (1 + x>n x m)
2
(Quadratic kernel)
> d
k(x n , x m ) = (1 + x n x m) (Polynomial kernel of degree d)
k(x n , x m ) = exp[−γ||x n − x m ||2 ] (RBF/Gaussian kernel)
k(x n , x m ) = x>
k(x n , x m ) = (1 + x>n x m)
2
(Quadratic kernel)
> d
Each of these kernels have an associated feature mapping φ : RD → RM
k(x n , x m ) = x>
k(x n , x m ) = (1 + x>n x m)
2
(Quadratic kernel)
> d
Each of these kernels have an associated feature mapping φ : RD → RM , e.g.,

√ √
Quadratic kernel: φ(x) = [1, 2x1 , . . . , 2x1 x2 , . . . , x12 , . . .]
k(x n , x m ) = x>
k(x n , x m ) = (1 + x>n x m)
2
(Quadratic kernel)
> d

√ √
Quadratic kernel: φ(x) = [1, 2x1 , . . . , 2x1 x2 , . . . , x12 , . . .]. Here M = O(D + D 2 ) dimensional
k(x n , x m ) = x>
k(x n , x m ) = (1 + x>n x m)
2
(Quadratic kernel)
> d

√ √
RBF kernel: φ(x) is infinite dimensional (saw in the last class). Here M = ∞
k(x n , x m ) = x>
k(x n , x m ) = (1 + x>n x m)
2
(Quadratic kernel)
> d

√ √
RBF kernel: φ(x) is infinite dimensional (saw in the last class). Here M = ∞
Again, remember that when using kernels, we don’t have to compute φ explicitly
Not every high-dim mapping is helpful. The mapping φ must be nonlinear
Nonlinear Map
Nonlinear Map (Helps)
(Helps)
Linear Map
(not helpful)
Kernel Methods can be Slow
Training phase can be slow (if N is very large)
N x N size
Storing the learned model may be expensive
Possibly very
high-dimensional
Testing (prediction) phase can be slow (scales in N or at least the number of support vectors)
N x N size
Possibly very
high-dimensional
N x N size
Possibly very
high-dimensional
Dual form of Ridge Regression:
Kernelized Ridge Regression:
N x N size (also need to invert it)
Possibly very high-dimensional
Testing (prediction) phase can be slow (scales in N)
Speeding Up Kernel Methods
Kernel methods are slow at training and test time

Would have been nice if we could easily compute the mapping φ(x) associated with kernel k

Then we could apply linear models directly on φ(x) without having to kernelize

But this is in general not possible since φ(x) is very high dimensional

Instead of a high-dim φ(x), can we get a good set of low-dim features ψ(x) ∈ RL using the kernel?

If ψ(x) is a good approximation of φ(x), then we can just use ψ(x) in a linear model

“Goodness” Criterion: ψ(x n )> ψ(x m ) ≈ φ(x n )> φ(x m )

i.e., we want ψ(x n )> ψ(x m ) ≈ k(x n , x m )

i.e., we want ψ(x n )> ψ(x m ) ≈ k(x n , x m )

We will see two popular approaches: Landmarks and Random Features
Using Kernels to “Extract” Good Features: Landmarks
Suppose we choose a small set of L “landmark” inputs z 1 , . . . , z L in the training data
For each input x n , using a kernel k, define an L-dimensional feature vector as follows
ψ(x n ) = [k(z 1 , x n ), k(z 2 , x n ), . . . , k(z L , x n )]
ψ(x n ) ∈ RL is such that k(x n , x m ) = φ(x n )> φ(x m ) ≈ ψ(x n )> ψ(x m )
Can now apply a linear model on the ψ representation (L-dimensional now) of the inputs
This will be fast both at training as well as test time if L is small
No need to kernelize the linear model and work with kernels (but still reap their benefits :-))
No need to kernelize the linear model and work with kernels (but still reap their benefits :-))
Note: The landmarks need not be actual inputs. Can even be learned from data.
Using Kernels to “Extract” Good Features: Random Features
Many kernel functions can be written as†

k(x n , x m ) = φ(x n )> φ(x m ) = Ew ∼p(w ) [tw (x n )tw (x m )]
† Random Features for Large-Scale Kernel Machines (Ben and Retch, NIPS 2007. Note: This paper actually won the test-of-time award at NIPS 2017)

where tw (.) is a scalar-valued function with parameters w ∈ RD from some distribution p(w )

Example: For the RBF kernel, tw (.) is cosine function and p(w ) is zero mean Gausssian
k(x n , x m ) = Ew ∼p(w ) [cos(w > x n ) cos(w > x m )]

Given w 1 . . . , w L drawn from p(w ), using Monte-Carlo approximation of expectation above

L
1X
k(x n , x m ) ≈ cos(w > >
` x n ) cos(w ` x m )
L
`=1


L
1X
k(x n , x m ) ≈ cos(w > > >
` x n ) cos(w ` x m ) = ψ(x n ) ψ(x m )
L
`=1


L
1X
k(x n , x m ) ≈ cos(w > > >
` x n ) cos(w ` x m ) = ψ(x n ) ψ(x m )
L
`=1
where ψ(x n ) = √1 [cos(w > x n ), . . . , cos(w > x n )] is an L-dim. feature vector (L needs to be set)
L 1 L


L
1X
k(x n , x m ) ≈ cos(w > > >
` x n ) cos(w ` x m ) = ψ(x n ) ψ(x m )
L
`=1
L 1 L
Can apply a linear model on this L-dim representation of the data (no need to kernelize)


L
1X
k(x n , x m ) ≈ cos(w > > >
` x n ) cos(w ` x m ) = ψ(x n ) ψ(x m )
L
`=1
L 1 L
Can apply a linear model on this L-dim representation of the data (no need to kernelize)
Such techniques exist for several kernels (RBF, polynomial, etc)
Other Techniques for Speeding Up Kernel Methods
Reducing the number of support vectors (for SVM based models), For example,
Learn the kernelized SV. Identify the support vectors.
Cluster the support vectors
Pick one SV from each cluster, retrain SVM using the chosen SVs
Other Techniques for Speeding Up Kernel Methods
Reducing the number of support vectors (for SVM based models), For example,
Learn the kernelized SV. Identify the support vectors.
Cluster the support vectors
Pick one SV from each cluster, retrain SVM using the chosen SVs
Low-rank approximations of kernel matrix (Nyström approximation)
Kernel Methods: Some Final Comments
Sometimes, even linear models can be trained via kernelization (but with linear kernel)
Benefit? Well, this may be beneficial sometimes due to computational reasons
For example, ridge regression requires solving
w = (X> X + λI)−1 X> y
.. where we learn w by inverting a D × D matrix
w = (X> X + λI)−1 X> y
Instead, the dual version of Ridge Regression, as we saw earlier, requires solving
w = X> (XX> + λIN )−1 y
w = (X> X + λI)−1 X> y
X N
w = X> (XX> + λIN )−1 y = X> α = αn x n
n=1
.. where we learn w in terms of α by inverting N × N matrix
w = (X> X + λI)−1 X> y
X N
w = X> (XX> + λIN )−1 y = X> α = αn x n
n=1
Even when working with linear model, if D > N, the latter way may be preferable
w = (X> X + λI)−1 X> y
X N
w = X> (XX> + λIN )−1 y = X> α = αn x n
n=1
Similar considerations apply to other kernelizable models too (e.g., SVM)
w = (X> X + λI)−1 X> y
X N
w = X> (XX> + λIN )−1 y = X> α = αn x n
n=1
Similar considerations apply to other kernelizable models too (e.g., SVM)
If linear model is what you want, still makes sense to look at the relative values of N and D to
decide whether to go for the dual (kernelized) formulation of the problem with a linear kernel
Kernel methods give us good features to make learning easier
However, these features are pre-defined (due to the choice of kernel)

Example: Consider the quadratic kernel applied to input x = [x1 , x2 ]
√ √ √
φ(x) = [1, 2x1 , x12 , 2x1 x2 , x2 , x22 , 2x2 ] (fixed definition for φ)

√ √ √
Another alternative is to learn good features from data

√ √ √
Another alternative is to learn good features from data
We will revisit this when we talk about deep neural networks
Unsupervised Learning
Roughly speaking, it is about learning interesting structures in the data (unsupervisedly!)

There is no supervision (no labels/responses), only inputs x 1 , . . . , x N

Some examples of unsupervised learning

Clustering: Grouping similar inputs together (and dissimilar ones far apart)

Dimensionality Reduction: Reducing the data dimensionality

Estimating the probability density of data (which distribution “generated” the data)

Most unsupervised learning algos can also be seen as learning a new representation of data

Most unsupervised learning algos can also be seen as learning a new representation of data
Typically a compressed representation, e.g., clustering can be used to get a one-hot representation
K = 6 clusters
0 0 1 0 0 0
A one-hot (quantized) rep.
Clustering
Given: N unlabeled examples {x 1 , . . . , x N }; no. of desired partitions K
Goal: Group the examples into K “homogeneous” partitions
Clustering
Picture courtesy: “Data Clustering: 50 Years Beyond K-Means”, A.K. Jain (2008)
Clustering
Loosely speaking, it is classification without ground truth labels
Clustering
Loosely speaking, it is classification without ground truth labels

A good clustering is one that achieves:
High within-cluster similarity
Low inter-cluster similarity
Similarity can be Subjective
Clustering only looks at similarities, no labels are given
Without labels, similarity can be hard to define
Thus using the right distance/similarity is very important in clustering

Also important to define/ask: “Clustering based on what”?

Also important to define/ask: “Clustering based on what”?
Picture courtesy: http://www.guy-sports.com/humor/videos/powerpoint presentation dogs.htm
Clustering: Some Examples
Document/Image/Webpage Clustering
Image Segmentation (clustering pixels)
Clustering web-search results
Clustering (people) nodes in (social) networks/graphs

.. and many more..
Picture courtesy: http://people.cs.uchicago.edu/∼pff/segment/
Types of Clustering
1 Flat or Partitional clustering
Partitions are independent of each other
Types of Clustering
2 Hierarchical clustering
Partitions can be visualized using a tree structure (a dendrogram)
Types of Clustering
2 Hierarchical clustering
Partitions can be visualized using a tree structure (a dendrogram)
Possible to view partitions at different levels of granularities by “cutting” the tree at some level
Flat Clustering: K -means algorithm (Lloyd, 1957)
Input: N examples {x 1 , . . . , x N }; x n ∈ RD ; the number of partitions K
Desired Output: Cluster assignments of these N examples and K cluster means µ1 , . . . , µK
Initialize: K cluster means µ1 , . . . , µK , each µk ∈ RD
Usually initialized randomly, but good initialization is crucial; many smarter initialization heuristics
exist (e.g., K -means++, Arthur & Vassilvitskii, 2007)
Iterate:
Iterate:
(Re)-Assign each example x n to its closest cluster center (based on the smallest Euclidean distance)
Ck = {n : k = arg min ||x n − µk ||2 }

k
(Ck is the set of examples assigned to cluster k with center µk )
Iterate:
Ck = {n : k = arg min ||x n − µk ||2 }

k

Update the cluster means
1 X
µk = mean(Ck ) = xn
|Ck | n∈C
k
Iterate:
Ck = {n : k = arg min ||x n − µk ||2 }

k

1 X
|Ck | n∈C
k
Repeat while not converged
Iterate:
Ck = {n : k = arg min ||x n − µk ||2 }

k

1 X
|Ck | n∈C
k
Repeat while not converged

Stop when cluster means or the “loss” does not change by much
K -means = Prototype Classification
(with unknown labels)
● Guess the means

● Predict the labels
● Recompute the means
● Repeat
K -means: Initialization (assume K = 2)
K -means iteration 1: Assigning points
K -means iteration 1: Recomputing the centers
What Loss Function is K -means Optimizing?
Let µ1 , . . . , µK be the K cluster centroids (means)
Let znk ∈ {0, 1} be s.t. znk = 1 if x n belongs to cluster k, and 0 otherwise
Note: z n = [zn1 zn2 . . . znK ] represents a length K one-hot encoding of x n
Define the distortion or “loss” for the cluster assignment of x n

K
X
`(µ, x n , z n ) = znk ||x n − µk ||2
k=1

K
X
`(µ, x n , z n ) = znk ||x n − µk ||2
k=1
Total distortion over all points defines the K -means “loss function”
N X
X K
L(µ, X, Z) = znk ||x n − µk ||2
n=1 k=1

K
X
`(µ, x n , z n ) = znk ||x n − µk ||2
k=1
N X
X K
L(µ, X, Z) = znk ||x n − µk ||2 = ||X − Zµ||2F
| {z }
n=1 k=1
matrix factorization view
where Z is N × K (row n is z n ) and µ is K × D (row k is µk )

K
X
`(µ, x n , z n ) = znk ||x n − µk ||2
k=1
N X
X K
| {z }
n=1 k=1

The K -means problem is to minimize this objective w.r.t. µ and Z

K
X
`(µ, x n , z n ) = znk ||x n − µk ||2
k=1
N X
X K
| {z }
n=1 k=1

The K -means problem is to minimize this objective w.r.t. µ and Z
Alternating optimization would give the K -means (Lloyd’s) algorithm we saw earlier!
Next Class: Clustering (Contd.)

Speeding Up Kernel Methods, and Intro To Unsupervised Learning

Uploaded by

Copyright:

Available Formats

Speeding Up Kernel Methods, and Intro To Unsupervised Learning

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Speeding Up Kernel Methods, and Intro To Unsupervised Learning

Uploaded by

Copyright:

Available Formats

Speeding Up Kernel Methods, and

Intro to Unsupervised Learning

Introduction to Machine Learning (CS771A)

September 11, 2018

Some popular examples of kernel functions

Some popular examples of kernel functions

Each of these kernels have an associated feature mapping φ : RD → RM

Some popular examples of kernel functions

Each of these kernels have an associated feature mapping φ : RD → RM , e.g.,

Some popular examples of kernel functions

Each of these kernels have an associated feature mapping φ : RD → RM , e.g.,

Some popular examples of kernel functions

Each of these kernels have an associated feature mapping φ : RD → RM , e.g.,

Some popular examples of kernel functions

Each of these kernels have an associated feature mapping φ : RD → RM , e.g.,

Not every high-dim mapping is helpful. The mapping φ must be nonlinear

Storing the learned model may be expensive

Storing the learned model may be expensive

Storing the learned model may be expensive

Dual form of Ridge Regression:

Kernelized Ridge Regression:

N x N size (also need to invert it)

Storing the learned model may be expensive

Possibly very high-dimensional

Testing (prediction) phase can be slow (scales in N)

Kernel methods are slow at training and test time

Kernel methods are slow at training and test time

Kernel methods are slow at training and test time

Kernel methods are slow at training and test time

Kernel methods are slow at training and test time

Kernel methods are slow at training and test time

Kernel methods are slow at training and test time

“Goodness” Criterion: ψ(x n )> ψ(x m ) ≈ φ(x n )> φ(x m )

Kernel methods are slow at training and test time

“Goodness” Criterion: ψ(x n )> ψ(x m ) ≈ φ(x n )> φ(x m )

i.e., we want ψ(x n )> ψ(x m ) ≈ k(x n , x m )

Kernel methods are slow at training and test time

“Goodness” Criterion: ψ(x n )> ψ(x m ) ≈ φ(x n )> φ(x m )

i.e., we want ψ(x n )> ψ(x m ) ≈ k(x n , x m )

Many kernel functions can be written as†

Many kernel functions can be written as†

Many kernel functions can be written as†

Many kernel functions can be written as†

Given w 1 . . . , w L drawn from p(w ), using Monte-Carlo approximation of expectation above

Many kernel functions can be written as†

Given w 1 . . . , w L drawn from p(w ), using Monte-Carlo approximation of expectation above

Many kernel functions can be written as†

Given w 1 . . . , w L drawn from p(w ), using Monte-Carlo approximation of expectation above

Many kernel functions can be written as†

Given w 1 . . . , w L drawn from p(w ), using Monte-Carlo approximation of expectation above

Many kernel functions can be written as†

Given w 1 . . . , w L drawn from p(w ), using Monte-Carlo approximation of expectation above

Low-rank approximations of kernel matrix (Nyström approximation)

w = X> (XX> + λIN )−1 y

Kernel methods give us good features to make learning easier

However, these features are pre-defined (due to the choice of kernel)

Kernel methods give us good features to make learning easier

However, these features are pre-defined (due to the choice of kernel)

Another alternative is to learn good features from data