Principal Component Analysis- The MATH

Hi, Isn’t PCA a hyped method for Dimensionality Reduction. All for good reasons. Let’s look out what maths has to say about this Non Parametric Method.

  • Principal component analysis (PCA) Orthogonally transforms the original n numeric dimensions of a dataset into a new set of m dimensions called principal components. As a result of the transformation, the first principal component has the largest possible variance; each succeeding principal component has the highest possible variance under the constraint that it is orthogonal to (i.e., uncorrelated with) the preceding principal components. Keeping only the first m < n principal components reduces the data dimensionality while retaining most of the data information, i.e., variation in the data.

Below is formulation of PCA

Project entire D dimensional data onto K<D space, where K are orthogonal components of Covariance matrix(derived below).

Fig 1
  1. Minimize projection error

2. Maximize the Variance of projected Data

Note Theoretically both are equivalent.

Note, only direction of u_1 matters to us, so we can restrict its magnitude/norm to be 1. By using fig1, On projecting vector x_{i} onto u_{1}, and calculating variance of projected data.

This essentially means that u_{1} is the eigen vector of Covariance Matrix(Sigma).Lambda_1 implying variance is maximized by choosing the eigenvector associated with the largest eigenvalue. This u_1 corresponds to the first Principal component.

Now we find second component by again maximizing variance subject to magnitude of it being 1 and it being perpendicular to already found out PC1(u_1).

This again leads to to eigen vector of Covariance matrix, corresponding to lambda_2, which should be second highest eigen value of Sigma matrix(Covariance matrix). And so on…we can find all principal components.






Problem Solver, Data Science, Actuarial Science, Knowledge Sharer, Hardcore Googler

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store