ensemble_boosting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Boosting ensembles\n",
    "\n",
    "Boosting is a method of converting a set of weak learners into strong learners.\n",
    "\n",
    "- Adaboost\n",
    "- CatBoost\n",
    "- GradientBoostingMachine (GBM)\n",
    "- LightGBM\n",
    "- XGboost\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adaptive Boosting\n",
    "\n",
    "- Alters the distribution of the training dataset to increase weights on sample observations that are difficult to classify.\n",
    "\n",
    "- The final prediction is based on a majority vote of the weak learner's predictions weighted by their individual accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AdaBoost learns from the mistakes by increasing the weight of misclassified data points. Gradient Boosting learns from the mistake — residual error directly, rather than update the weights of data points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Boosted Trees\n",
    "- The approach trains learners based upon minimizing the loss function of a strong learner (i.e., training on the residuals of the strong model) as an alternative means to focus on training upon misclassified observations.\n",
    "\n",
    "- The contribution of each weak learner to the final prediction is based on a gradient optimization process to minimize the overall error of the strong learner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extreme Gradient Boosting (XGBoost)\n",
    "XGBoost uses pre-sorted algorithm & Histogram-based algorithm for computing the best split. The histogram-based algorithm splits all the data points for a feature into discrete bins and uses these bins to find the split value of histogram.\n",
    "\n",
    "XGBoost cannot handle categorical features by itself, it only accepts numerical values similar to Random Forest. Therefore one has to perform various encodings like label encoding, mean encoding or one-hot encoding before supplying categorical data to XGBoost.\n",
    "\n",
    "Gradient Boosting algorithm also called gradient boosting machine including the learning rate. Stochastic Gradient Boosting with sub-sampling at the row, column and column per split levels. Regularized Gradient Boosting with both L1 and L2 regularization.\n",
    "\n",
    "Clever Penalisation of Trees A Proportional shrinking of leaf nodes Newton Boosting Extra Randomisation Parameter\n",
    "\n",
    "In XGBoost the trees can have a varying number of terminal nodes and left weights of the trees that are calculated with less evidence is shrunk more heavily.\n",
    "\n",
    "Newton Boosting uses Newton-Raphson method of approximations which provides a direct route to the minima than gradient descent.\n",
    "\n",
    "The extra randomisation parameter can be used to reduce the correlation between the trees, as seen in the previous article, the lesser the correlation among classifiers, the better our ensemble of classifiers will turn out."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LightGBM\n",
    "\n",
    "LightGBM uses a novel technique of Gradient-based One-Side Sampling (GOSS) to filter out the data instances for finding a split value\n",
    "\n",
    "LightGBM can also handle categorical features by taking the input of feature names. It does not convert to one-hot coding, and is much faster than one-hot coding. LGBM uses a special algorithm to find the split value of categorical features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CatBoost\n",
    "CatBoost has the flexibility of giving indices of categorical columns so that it can be encoded as one-hot encoding using one_hot_max_size (Use one-hot encoding for all features with number of different values less than or equal to the given parameter value).\n",
    "\n",
    "If you don’t pass any anything in cat_features argument, CatBoost will treat all the columns as numerical variables.\n",
    "\n",
    "Catboost improves over LightGBM by handling categorical features better. Traditionally categorical features are one-hot-encoded, this incurs the curse of dimensionality if the feature has many distinct values. Previous algorithms tackled this issue of sparse features as we saw with EFB above. Catboost deals with categorical features by, “generating random permutations of the dataset and for each sample computing the average label value for the sample with the same category value placed before the given one in the permutation”. They also process the data with GPU acceleration, and do feature discretization into a fixed number of bins (128 and 32)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bagging vs Boosting\n",
    "\n",
    "Bagging (stands for Bootstrap Aggregating) is a way to decrease the variance of your prediction by generating additional data for training from your original dataset using combinations with repetitions to produce multisets of the same cardinality/size as your original data. By increasing the size of your training set you can't improve the model predictive force, but just decrease the variance, narrowly tuning the prediction to expected outcome.\n",
    "\n",
    "Boosting is a two-step approach, where one first uses subsets of the original data to produce a series of averagely performing models and then \"boosts\" their performance by combining them together using a particular cost function (=majority vote). Unlike bagging, in the classical boosting the subset creation is not random and depends upon the performance of the previous models: every new subsets contains the elements that were (likely to be) misclassified by previous models.\n",
    "\n",
    "both bagging and boosting use a single learning algorithm for all steps; but they use different methods on handling training samples. both are ensemble learning method that combines decisions from multiple models\n",
    "\n",
    "### Bagging:\n",
    "1. resamples training data to get M subsets (bootstrapping);\n",
    "2. trains M classifiers(same algorithm) based on M datasets(different samples);\n",
    "3. final classifier combines M outputs by voting;\n",
    "samples weight equally;\n",
    "classifiers weight equally;\n",
    "decreases error by decreasing the variance\n",
    "\n",
    "### Boosting: here focus on adaboost algorithm\n",
    "1. start with equal weight for all samples in the first round;\n",
    "2. in the following M-1 rounds, increase weights of samples which are misclassified in last round, decrease weights of samples correctly classified in last round\n",
    "3. using a weighted voting, final classifier combines multiple classifiers from previous rounds, and give larger weights to classifiers with less misclassifications.\n",
    "step-wise reweights samples; weights for each round based on results from last round\n",
    "re-weight samples(boosting) instead of resampling(bagging)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradient boosting is based on Decision Tree and DTs are based on feature split not distances. Unlike Boosting (which is sequential), RF grows trees in parallel. Notice that for RF, each iteration, the classifier is trained independently from the rest."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both of these methods are built based on the idea of converting weak learners to a strong learner by updating based on the residuals (XGBoost) or misclassifications (AdaBoost). \n",
    "\n",
    "Both are the same XGboost and GBM, both works on the same principle. In Xg boost parallel computation is possible, means in XG boost parallelly many GBM's are working. In Xgboost tunning parameters are more."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}