Perceptron Logistic Regression
Logistic Regression is an important one, that is why there are significant number of algorithms at work to optimize its solutions. Here we look at another algorithm called Perceptron.
I tried again finding code for writing it from scratch. hope you enjoy it!
I also incorporated two other alternatives of perceptron implementation which are average perceptron and pegasos algorithm, which are fairly intuitive.
Dataset : Self created with random numbers
Process followed:
- Here the classes are encoded as 1 and -1 as opposed to 0 and 1 in logistic regression. Because that’s how this algorithm works. That doesn’t make much of the difference because they are just labels that we give them.
- After creation of data, we move to write single step update functions for perceptron and pegasos, and for average it follows directly in the final call function. All the family of perceptron algorithm follow same pattern with update rule being slightly different.
Perceptron psuedocode
# Perceptron Algorithm
# initialize θ and θ₀ with 0
θ = 0 (vector)
θ₀ = 0 (scalar)
# totally T epoches to iterate
for t = 1 .. T do
# totally m data points
for i = 1 .. m do
# misclassify data points
if y⁽ⁱ⁾(θ ⋅ x⁽ⁱ⁾ + θ₀) ≦ 0
then
θ = θ + y⁽ⁱ⁾ ⋅ x⁽ⁱ⁾
θ₀ = θ₀ + y⁽ⁱ⁾
return θ, θ₀Average Psuedocode
# Average Perceptron Algorithm# initialize θ, θ₀, sum_θ, sum_θ₀, and counter with 0
θ = 0 (vector)
θ₀ = 0 (scalar)
sum_θ = 0 (vector)
sum_θ₀ = 0 (scalar)
counter = 0# totally T epoches to iterate
for t = 1 .. T do
# totally m data points
for i = 1 .. m do
# misclassify data points
if y⁽ⁱ⁾(θ ⋅ x⁽ⁱ⁾ + θ₀) ≦ 0
then
θ = θ + y⁽ⁱ⁾ ⋅ x⁽ⁱ⁾
θ₀ = θ₀ + y⁽ⁱ⁾ sum_θ = sum_θ + θ
sum_θ₀ = sum_θ₀ + θ₀
counter = counter + 1return (sum_θ/counter), (sum_θ₀/counter)Pegasos Psuedocode
# Pegasos Algorithm# initialize θ, θ₀, and counter with 0
θ = 0 (vector)
θ₀ = 0 (scalar)
counter = 0# totally T epoches to iterate
for t = 1 .. T do
# totally m data points
for i = 1 .. m do
counter = counter + 1
η = 1/√counter
if y⁽ⁱ⁾(θ⋅x⁽ⁱ⁾ + θ₀) ≦ 1
then
θ = (1 - ηλ)θ + ηy⁽ⁱ⁾⋅x⁽ⁱ⁾
θ₀ = θ₀ + ηy⁽ⁱ⁾
else
then
θ = (1 - ηλ)θ
θ₀ = θ₀return θ, θ₀- Stochastic gradient descent has implemented here which updates the parameters after iterating through each observation. Also we try to obtain a plot which shows the comparative figures where all three types of perceptron plot a decision boundary for each iteration.
As simple execution repository looks, understanding them is far more important and difficult. So finding my way to learning them literally and not just mugging. Check out my other articles as well.
I would really appreciate you to leave a note if you followed till the end. Any recommendation and thoughts are welcomed.
Cheers to learning!!! 💪
Shaily Jain
Be a part of my Instagram Community 👇
