Principal Component Analysis- The MATH

3 min readMay 15, 2021

Hi, Isn’t PCA a hyped method for Dimensionality Reduction. All for good reasons. Let’s look out what maths has to say about this Non Parametric Method.

Principal component analysis (PCA) Orthogonally transforms the original n numeric dimensions of a dataset into a new set of m dimensions called principal components. As a result of the transformation, the first principal component has the largest possible variance; each succeeding principal component has the highest possible variance under the constraint that it is orthogonal to (i.e., uncorrelated with) the preceding principal components. Keeping only the first m < n principal components reduces the data dimensionality while retaining most of the data information, i.e., variation in the data.

Below is formulation of PCA

GOAL

Project entire D dimensional data onto K<D space, where K are orthogonal components of Covariance matrix(derived below).

Objective

Minimize projection error

2. Maximize the Variance of projected Data

Note Theoretically both are equivalent.

Note, only direction of u_1 matters to us, so we can restrict its magnitude/norm to be 1. By using fig1, On projecting vector x_{i} onto u_{1}, and calculating variance of projected data.

This essentially means that u_{1} is the eigen vector of Covariance Matrix(Sigma).Lambda_1 implying variance is maximized by choosing the eigenvector associated with the largest eigenvalue. This u_1 corresponds to the first Principal component.

Now we find second component by again maximizing variance subject to magnitude of it being 1 and it being perpendicular to already found out PC1(u_1).

This again leads to to eigen vector of Covariance matrix, corresponding to lambda_2, which should be second highest eigen value of Sigma matrix(Covariance matrix). And so on…we can find all principal components.