XGBoost- The MATH

Extreme Gradient Boosting- Nasty Math

XGBoost optimizes a loss function which needs to be prespecified, as below

Now objective is to minimize this by Gradient Descent, means

Here f(i) is the ith tree which is not necessarily a linear function. We need to get everything in terms of w(j) which is score of leaf j.

Let us see how XGBoost is comparatively better than other techniques:

1. Regularization:

• Standard GBM implementation has no regularisation like XGBoost.
• Thus XGBoost also helps to reduce overfitting.

2. Parallel Processing:

• XGBoost implements parallel processing and is faster than GBM .
• XGBoost also supports implementation on Hadoop.

3. High Flexibility:

• XGBoost allows users to define custom optimization objectives and evaluation criteria adding a whole new dimension to the model.

4. Handling Missing Values:

• XGBoost has an in-built routine to handle missing values. Where it assigns all missing values to left or right leaf, based on maximum gain. So each node has missing values going to leaves providing ultimate maximum gain.

5. Tree Pruning:

• XGBoost makes splits up to the max_depth specified and then starts pruning the tree backwards and removes splits beyond which there is no positive gain.

6. Built-in Cross-Validation:

• XGBoost allows a user to run a cross-validation at each iteration of the boosting process and thus it is easy to get the exact optimum number of boosting iterations in a single run.

MY SOCIAL SPACE

Instagram https://www.instagram.com/codatalicious/

LinkedIn https://www.linkedin.com/in/shaily-jain-6a991a143/

Medium https://codatalicious.medium.com/

YouTube https://www.youtube.com/channel/UCKowKGUpXPxarbEHAA7G4MA

Resources:

1. https://youtu.be/ufHo8vbk6g4
2. https://www.analyticsvidhya.com/blog/2018/06/comprehensive-guide-for-ensemble-models/