Newton’s Method for Logistic Regression
More formally convergence achieved by Newton’s Raphson Method is called Fischer Scoring.
So Logistic Regression is a very useful algorithm to approach classification problems. If you want to check up basics of Logistic Regression visit my previous article here.
There are many ways to get to the convergence point of the cost function.
Here we discuss Newton’s Method as the solution which is also highlighted in Andrew Ng Machine Learning coursework.
Dataset used : artificially made from make_classification in sklearn.datasets
Process followed:
- Understanding of Newton’s method in brief is it allows us to find roots of any function iteratively and converge to a its exact roots. Here we want to maximize our log likelihood, thereby we would put derivative of loglikelihood to 0 to get values of feature set that maximises likelihood. Therefor if we take function to be l`(theta) = 0.
When we only have one parameter have one parameter expression would be
- When we have two parameters we have
The key difference in this iterative method is the update rule of parameters which essentially happens via Hessian and gradient matrix.
- Essential from above figure, we start at some point x1 and want to get the exact root point where l’ = 0 and we get parameter estimate which maximizes loglikelihood and hence likelihood
- The two variable update rule is
where first matrix is called Hessian Matrix substitute for second order differential of loglikelihood vector and Gradient or Jacobian Matrix which is multivariate substitute for first order differential of loglikelihood vector
and
- Now that we have the rules and theory let’s try it out on a dataset for which you can check my repository .
- Also Newton’s method converges much sooner than Batch Gradient thereby it is good to try it up.
I would really appreciate you to leave a note if you followed till the end. Any recommendation and thoughts are welcomed.
Cheers to learning!!! 💪
Shaily Jain
Be a part of my Instagram Community 👇
