Locally Weighted Linear Regression
Linear Regression is the way in which humans tend to think, but we can make machines think in semi linear way. One such technique is Locally weighted Linear Regression aka LOWESS aka LOESS. It is a Linear fit to parts of Non Linear Curve, so in end we are fitting a linear loss function but weighted one.
Weight here is nothing but importance that is given to immediate neighborhood of point considered. So this means that our initially Linear Equation fitted to static set of coefficient is now dynamic to the value of independent variable.
Dataset used: Constructed a hypothetical independent and deoendent variable with jitters
n = 1000#generate datasetX = np.linspace(-3, 3, num = n)
Y = np.log(np.abs(X ** 2 - 1) + 0.5)#jitter X
X +=np.random.normal(scale = 0.1, size = n)
Clearly the y vs x is a non linear relationship
Process Used:
- For providing weights, KERNELS is what is used here. Although there are pretty many kernels available but Radial Kernel is most frequent used ones.
- We next obtain a weight matrix w.r.t. to each observation in dataset such that observation nearest to say point x0 gets highest weight hence influence coefficient of linear equation there more and ones far away influence the least. This is controlled by radial kernel function as X closer to x0 implies kernel to be highest and x far away will make kernel tending to zero. This is denoted by xw in notebook (attached in end)
- Next we obtain solution to optimization of least square error through Normal equations similar to Ordinary Least squares as dealt with earlier, but with weights matrix now.
- Parameters here are ‘tau’(also referred to as span) which can be set by hyper parameter tuning. High value corresponds to high bias, low variance ; an under fitting status and vice versa. We can add those complexity for simplifying our task later.
If this article isn’t enough to satisfy your intellect goals,
Don’t worry!! Just visit my repository, it mentions all the articles, links I referred to while learning this topic.
I would really appreciate you to leave a note if you followed till the end. Any recommendation and thoughts are welcomed.
Cheers to learning!!! 💪
Shaily Jain
Be a part of my Instagram Community 👇
