correlation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Association"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Correlation__\n",
    "\n",
    "Correlation is the strength of association or measure of relationship between two variables. Correlation is dimensionless and is unit free measure. Correlation matrix/analysis is a tool for feature selection. PCA is one of the application of feature selection.There are several type of correlation \n",
    "- Pearson (measures the linear association between continuous variables)\n",
    "- Spearman (an special case of pearson applied to ranked(sorted) variables and is not restricted to linear relationship.\n",
    "- kendall\n",
    "\n",
    "- Corr(A and B) == Corr(B and A)\n",
    "- Coefficient of correlation is a numerical measure whose valuues lies between -1 to +1\n",
    "- Covariance ; 0 to 1\n",
    "- Spearman rank correlation is non parametric i.e distribution free works on order/rank\n",
    "- When the correlation (r) is negative, the regression slope (b) will be negative.\n",
    "- Correlation is a single statistic, whereas regression produces an entire equation\n",
    "\n",
    "__Covariance__\n",
    "\n",
    "Covariance is a measure of how changes in one variable are associated with changes in a second variable. Specifically, covariance measures the degree to which two variables are linearly associated. If two variables are independent, their covariance is 0. But, having a covariance of 0 does not imply the variables are independent.\n",
    "\n",
    "__Correlation Vs Covariance__\n",
    "- Correlation is a function of the covariance. Correlation values are standardized whereas, covariance values are not.\n",
    "- use the covariance matrix when the variable are on similar scales and the correlation matrix when the scales of the variables differ.\n",
    "- Correlation matrix are used if variables are measured on different scale whereas covariance matrix  measure variables that have different units of measurement.\n",
    "- Covariance doesn't really tell you about the strength of the relationship between the two variables, while correlation does\n",
    "\n",
    "\n",
    "__Auto Correlation__\n",
    "\n",
    "- Auto Correlation Function (ACF) is correlation of a variable with its lagged value.\n",
    "- Correlogram is the plot of ACF vst time_lag\n",
    "\n",
    "\n",
    "__Example__\n",
    "- Categorical - categorical ==> chi square\n",
    "- quantitative - quantitative ==> ANOVA\n",
    "- Quantitative - Quantitative ==> Pearson's correlaation\n",
    "- continuous - descrete ==> one way Anova ( Kruskal Wallis H Test non pararmetric form of one-way anova)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q. What is the difference between covariance and correlation?\n",
    "\n",
    "The covariance is a measure for how two variables are related to each other, i.e., how two variables vary with each other. A covariance of 0 indicates that two variables are totally unrelated. If the covariance is positive, the variables increase in the same direction, and if the covariance is negative, the variables change in opposite directions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q What are Rank Correlation techniques?\n",
    "\n",
    "Rank correlation is a technique that compares which features correlate with the output. One popular rank correlation method PCA. It’s a technique to find patterns in high dimensional data.Naturally, this comes at the expense of accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q What Is The Difference Between PCA and PLS (Principal Component Analysis VS. Partial Least Squares)?\n",
    "\n",
    "__PCA__ tries to explain the variance-covariance structure of a data set. Aim is to increase the variance of the features itself, like the loss of information is greatly reduced. PCA is a Dimensionality Reduction algorithm. Both PLS and PCA are used for dimension reduction.\n",
    "\n",
    "__PLS__ Partial Least Squares, use the annotated label to maximize inter-class variance. Principal components are pairwise orthogonal. Principal components are focus on maximize correlation. The main difference is that the PCA is unsupervised method and PLS is supervised method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multicollinearity"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Multicollinearity : arises when predictors are correlated with each other.\n",
    "Detection : Correlation Matrix\n",
    "Measure : Variance Inflation Factor (VIF), VIF>5 is a concern\n",
    "Diagnosis : PcA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Q What is collinearity and what to do with it? How to remove multicollinearity?__\n",
    "\n",
    "Collinearity/Multicollinearity:\n",
    "- In multiple regression: when two or more variables are highly correlated\n",
    "- They provide redundant information\n",
    "- In case of perfect multicollinearity: β=(XTX)−1XTy doesn’t exist, the design matrix isn’t invertible\n",
    "- It doesn’t affect the model as a whole, doesn’t bias results\n",
    "- The standard errors of the regression coefficients of the affected variables tend to be large\n",
    "- The test of hypothesis that the coefficient is equal to zero may lead to a failure to reject a false null hypothesis of no effect of the explanatory (Type II error)\n",
    "- Leads to overfitting\n",
    "\n",
    "Remove multicollinearity:\n",
    "- Drop some of affected variables\n",
    "- Principal component regression: gives uncorrelated predictors\n",
    "- Combine the affected variables\n",
    "- Ridge regression\n",
    "- Partial least square regression\n",
    "\n",
    "Detection of multicollinearity:\n",
    "- Large changes in the individual coefficients when a predictor variable is added or deleted\n",
    "- Insignificant regression coefficients for the affected predictors but a rejection of the joint\n",
    "hypothesis that those coefficients are all zero (F-test)\n",
    "- VIF: the ratio of variances of the coefficient when fitting the full model divided by the variance of the coefficient when fitted on its own\n",
    "- rule of thumb: VIF>5 indicates multicollinearity\n",
    "- Correlation matrix, but correlation is a bivariate relationship whereas multicollinearity is multivariate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q What is Variance Inflation Facotor(VIF)?\n",
    "\n",
    "- a way to measure the effect of multicollinearity among your predictors .More variation is bad. If the variance of the coefficients increases, our model isn't going to be as reliable. \n",
    "\n",
    "- Colinearity is the state where two variables are highly correlated and contain similiar information about the variance\n",
    "- multicollinearity is when three or more predictors in your regression are highly correlated\n",
    "- Multicollinearity is dangerous because it can increase the variance of the regression coefficients.\n",
    "- Multicollinearity can rise if the data is collected without and experimental design\n",
    "- The Variance Inflation Factor (VIF) is a measure of colinearity among predictor variables within a multiple regression\n",
    "-  a VIF is > 10, you have high multicollinearity.\n",
    "- Tolerance is the reciprocal of VIF.\n",
    "- VIF = 1 (Not correlated)\n",
    "- 1 < VIF < 5 (Moderately correlated)\n",
    "- VIF >=5 (Highly correlated)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}