CS613_Project_SourceCode_Preprocessing_FINAL.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CS613 Final Project\n",
    "### Notebook 1: Pre-processing\n",
    "### Submitted by: John Obuch, Fatih Catpinar, Meghan Colosimo, Colin Murphy\n",
    "### 3/19/20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Requirements\n",
    "#NOTE: Must pip install imblearn in terminal first!\n",
    "\n",
    "import time, pandas as pd, numpy as np\n",
    "start_time = time.time()\n",
    "from collections import Counter, defaultdict\n",
    "from pprint import pprint\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn import preprocessing\n",
    "import re\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set(style=\"white\")\n",
    "sns.set(style=\"whitegrid\", color_codes=True)\n",
    "from sklearn.model_selection import train_test_split\n",
    "from imblearn.over_sampling import SMOTE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Acquisition\n",
    "\n",
    "1) Behavioral Risk Factor Surveillance System (BRFSS) survey data is published annually, the data is avaliable via the following page:\n",
    "\n",
    "https://www.cdc.gov/brfss/annual_data/annual_data.htm\n",
    "\n",
    "2) To download the data, select a specific year. It will land to the the year page.\n",
    "\n",
    "3) Scroll down to \"Data Files\". The data files are provided in ASCII and SAS Transport formats.\n",
    "\n",
    "4) Dowload the appropriate file type. For this study we will use \"SAS Transport Format.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Documentation\n",
    "\n",
    "- To learn more information about the survey for specific year, read the files under \"Survey Data Information\". <br/>\n",
    "Check the codebook, to see variable names, location, and frequency of values for all reporting areas <br/>\n",
    "combined for the combined landline and cell phone data set.\n",
    "\n",
    "2017 Survey Codebook\n",
    "https://www.cdc.gov/brfss/annual_data/2017/pdf/codebook17_llcp-v2-508.pdf\n",
    "\n",
    "2018 Survey Codebook\n",
    "https://www.cdc.gov/brfss/annual_data/2018/pdf/codebook18_llcp-v2-508.pdf\n",
    "\n",
    "- BRFSS Survey Questions\n",
    "https://www.cdc.gov/brfss/questionnaires/index.htm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Our plan\n",
    "\n",
    "Dowloaded 2017 and 2018 survey data in SAS Transport format. \n",
    "\n",
    "https://www.cdc.gov/brfss/annual_data/annual_2017.html\n",
    "\n",
    "https://www.cdc.gov/brfss/annual_data/annual_2018.html\n",
    "\n",
    "Unzipped the data and placed the data files are in the \"data\" folder.\n",
    "\n",
    "The name of the survey data are: \n",
    "- LLCP2017.XPT\n",
    "- LLCP2018.XPT\n",
    "\n",
    "We will use the above data to create several models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "run_time:\n",
      " 110.07133102416992\n"
     ]
    }
   ],
   "source": [
    "# read data as a pandas dataframe. Note: This process will take time\n",
    "\n",
    "df_2017 = pd.read_sas('data/LLCP2017.XPT')\n",
    "df_2018 = pd.read_sas('data/LLCP2018.XPT')\n",
    "\n",
    "run_time = time.time() - start_time\n",
    "print('run_time:\\n', run_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_STATE</th>\n",
       "      <th>FMONTH</th>\n",
       "      <th>IDATE</th>\n",
       "      <th>IMONTH</th>\n",
       "      <th>IDAY</th>\n",
       "      <th>IYEAR</th>\n",
       "      <th>DISPCODE</th>\n",
       "      <th>SEQNO</th>\n",
       "      <th>_PSU</th>\n",
       "      <th>CTELENM1</th>\n",
       "      <th>...</th>\n",
       "      <th>_PA300R2</th>\n",
       "      <th>_PA30021</th>\n",
       "      <th>_PASTRNG</th>\n",
       "      <th>_PAREC1</th>\n",
       "      <th>_PASTAE1</th>\n",
       "      <th>_RFSEAT2</th>\n",
       "      <th>_RFSEAT3</th>\n",
       "      <th>_FLSHOT6</th>\n",
       "      <th>_PNEUMO2</th>\n",
       "      <th>_AIDTST3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>b'01302017'</td>\n",
       "      <td>b'01'</td>\n",
       "      <td>b'30'</td>\n",
       "      <td>b'2017'</td>\n",
       "      <td>1100.0</td>\n",
       "      <td>b'2017000001'</td>\n",
       "      <td>2.017000e+09</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>b'01122017'</td>\n",
       "      <td>b'01'</td>\n",
       "      <td>b'12'</td>\n",
       "      <td>b'2017'</td>\n",
       "      <td>1100.0</td>\n",
       "      <td>b'2017000002'</td>\n",
       "      <td>2.017000e+09</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>b'01102017'</td>\n",
       "      <td>b'01'</td>\n",
       "      <td>b'10'</td>\n",
       "      <td>b'2017'</td>\n",
       "      <td>1100.0</td>\n",
       "      <td>b'2017000003'</td>\n",
       "      <td>2.017000e+09</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>b'02082017'</td>\n",
       "      <td>b'02'</td>\n",
       "      <td>b'08'</td>\n",
       "      <td>b'2017'</td>\n",
       "      <td>1200.0</td>\n",
       "      <td>b'2017000004'</td>\n",
       "      <td>2.017000e+09</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>b'01302017'</td>\n",
       "      <td>b'01'</td>\n",
       "      <td>b'30'</td>\n",
       "      <td>b'2017'</td>\n",
       "      <td>1100.0</td>\n",
       "      <td>b'2017000005'</td>\n",
       "      <td>2.017000e+09</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 358 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   _STATE  FMONTH        IDATE IMONTH   IDAY    IYEAR  DISPCODE  \\\n",
       "0     1.0     1.0  b'01302017'  b'01'  b'30'  b'2017'    1100.0   \n",
       "1     1.0     1.0  b'01122017'  b'01'  b'12'  b'2017'    1100.0   \n",
       "2     1.0     1.0  b'01102017'  b'01'  b'10'  b'2017'    1100.0   \n",
       "3     1.0     1.0  b'02082017'  b'02'  b'08'  b'2017'    1200.0   \n",
       "4     1.0     1.0  b'01302017'  b'01'  b'30'  b'2017'    1100.0   \n",
       "\n",
       "           SEQNO          _PSU  CTELENM1    ...     _PA300R2  _PA30021  \\\n",
       "0  b'2017000001'  2.017000e+09       1.0    ...          1.0       1.0   \n",
       "1  b'2017000002'  2.017000e+09       1.0    ...          1.0       1.0   \n",
       "2  b'2017000003'  2.017000e+09       1.0    ...          3.0       2.0   \n",
       "3  b'2017000004'  2.017000e+09       1.0    ...          9.0       9.0   \n",
       "4  b'2017000005'  2.017000e+09       1.0    ...          3.0       2.0   \n",
       "\n",
       "   _PASTRNG  _PAREC1  _PASTAE1  _RFSEAT2  _RFSEAT3  _FLSHOT6  _PNEUMO2  \\\n",
       "0       1.0      1.0       1.0       1.0       1.0       1.0       1.0   \n",
       "1       1.0      1.0       1.0       1.0       1.0       1.0       1.0   \n",
       "2       2.0      4.0       2.0       1.0       1.0       1.0       1.0   \n",
       "3       9.0      9.0       9.0       9.0       9.0       9.0       9.0   \n",
       "4       2.0      4.0       2.0       1.0       2.0       2.0       2.0   \n",
       "\n",
       "   _AIDTST3  \n",
       "0       1.0  \n",
       "1       1.0  \n",
       "2       2.0  \n",
       "3       NaN  \n",
       "4       2.0  \n",
       "\n",
       "[5 rows x 358 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# take a look at data\n",
    "\n",
    "df_2017.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#df_2018.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The shape of the 2017 data is: (450016, 358)\n",
      "In 2017 450016 took the survey and there were 358 features in the original 2017 dataset.\n",
      "\n",
      "The shape of the 2018 data is: (437436, 275)\n",
      "In 2018 437436 took the survey and there were 275 features in the original 2018 dataset.\n"
     ]
    }
   ],
   "source": [
    "# the shape of the data\n",
    "\n",
    "print('The shape of the 2017 data is:', df_2017.shape)\n",
    "print('In 2017', df_2017.shape[0], 'took the survey and there were', df_2017.shape[1], \n",
    "      'features in the original 2017 dataset.\\n')\n",
    "\n",
    "print('The shape of the 2018 data is:', df_2018.shape)\n",
    "print('In 2018', df_2018.shape[0], 'took the survey and there were', df_2018.shape[1], \n",
    "      'features in the original 2018 dataset.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counter({dtype('float64'): 352, dtype('O'): 6})\n",
      "Counter({dtype('float64'): 269, dtype('O'): 6})\n"
     ]
    }
   ],
   "source": [
    "#check data types - categorical data is being interpreted as numeric data, so dummy coding is required\n",
    "\n",
    "print(Counter(df_2017.dtypes))\n",
    "print(Counter(df_2018.dtypes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can print the columns of the data\n",
    "# check the codebook for detailed information about the column\n",
    "\n",
    "#df_2017.columns\n",
    "#df_2018.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# answer counts to features\n",
    "# it counts non-empty cells\n",
    "\n",
    "#df_2017.count()\n",
    "#df_2018.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# descriptive statistics\n",
    "\n",
    "#df_2017.describe()\n",
    "#df_2018.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Data Preprocessing\n",
    "\n",
    "- Scales are diffent, we need to standardize the (continuous) features.\n",
    "- Convert categorical data to dummy variables.\n",
    "- Handle missing information.\n",
    "For example; in a binary Yes/No question. There may be 5 possible answers in this survey:\n",
    "    - 1 Yes\n",
    "    - 2 No\n",
    "    - 7 Don’t know/Not Sure\n",
    "    - 9 Refused\n",
    "    - BLANK Not asked or Missing\n",
    "\n",
    "- Reenginner the answers so that direction is consistent (e.g., reverse-coded variables). (Ex: Poor:5, Excellent:1)\n",
    "- Address imbalanced cases of outcome variable with SMOTE.\n",
    "- NOTE: There are some questions asked in specific states."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing Plan:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Part 1A: Drop columns that are:\n",
    "\n",
    "- Features related to metadata (e.g., interview details, filters, etc.)\n",
    "- Open-ended response data (strings)\n",
    "- Unscaled continuous features (e.g., in days, hours, or weeks)\n",
    "- Redundant features (e.g., height in meters)\n",
    "- Redundant outcomes (e.g., multiple versions of the question 'Have you ever been diagnoised with X disease?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of pre-selected columns to drop is: 46\n"
     ]
    }
   ],
   "source": [
    "#Names of columns to drop for the 2017 dataset\n",
    "\n",
    "metadata_features = ['IDATE', 'IMONTH', 'IDAY', 'IYEAR', 'SEQNO', 'FMONTH', 'DISPCODE', '_PSU', 'SAFETIME', 'QSTVER', \n",
    "                     '_STSTR', '_STRWT', '_RAWRAKE', '_WT2RAKE', '_CLLCPWT', '_DUALUSE', '_DUALCOR', '_LLCPWT2', '_LLCPWT',\n",
    "                    '_FRUITE1', '_VEGETE1', 'PAMISS1_']\n",
    "\n",
    "openended_features = ['DLYOTHER']\n",
    "\n",
    "#Scaled versions of these varaibles already exist in the dataset - drop the unscaled, redundant data\n",
    "unscaled_features = ['WEIGHT2', 'HEIGHT3', 'ALCDAY5', 'AVEDRNK2', 'DROCDY3_', 'FRUIT2', 'FRUITJU2', 'FVGREEN1', 'POTATOE1',\n",
    "                    'VEGEDA2_', 'EXEROFT1', 'EXEROFT2', 'STRENGTH', 'FRENCHF1', 'VEGETAB2']\n",
    "\n",
    "# NOTE: There are multiple columns representing the same data for a subset of features (e.g., race, age)\n",
    "# The majority of these redundicies will be addressed with PCA.\n",
    "\n",
    "redundant_features = ['HTM4']\n",
    "\n",
    "#NOTE: These features need to be dropped from the X data before training the model\n",
    "\n",
    "redundant_outcomes = ['BPHIGH4', 'TOLDHI2', 'CVDINFR4', 'CVDCRHD4', 'ASTHMA3',  'ASTHNOW', 'HAVARTH3']\n",
    "\n",
    "print('The number of pre-selected columns to drop is:', len(metadata_features) + len(openended_features) + len(unscaled_features) +\n",
    "     len(redundant_features) + len(redundant_outcomes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dataframe is the expected shape after dropping pre-selected columns: True\n"
     ]
    }
   ],
   "source": [
    "#Dropping pre-selected columns\n",
    "\n",
    "df_2017.drop(columns = metadata_features, inplace = True)\n",
    "df_2017.drop(columns = openended_features, inplace = True)\n",
    "df_2017.drop(columns = unscaled_features, inplace = True)\n",
    "df_2017.drop(columns = redundant_features, inplace = True)\n",
    "df_2017.drop(columns = redundant_outcomes, inplace = True)\n",
    "\n",
    "print('The dataframe is the expected shape after dropping pre-selected columns:', df_2017.shape[1] == 358 - 46)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Part 1B: Determine which variables are continuous or categorical for proper pre-processing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Continuous Features (based on documentation)\n",
    "\n",
    "countinuous_features_17 = ['NUMMEN', 'NUMADULT', 'NUMWOMEN', 'HHADULT', 'PHYSHLTH', 'MENTHLTH', 'POORHLTH', 'DIABAGE2',\n",
    "                            'JOINPAI1', 'NUMPHON2', 'CHILDREN', 'WEIGHT2', 'HEIGHT3', 'ALCDAY5', 'AVEDRNK2', 'DRNK3GE5',\n",
    "                          'MAXDRNKS', 'FRUIT2', 'FRUITJU2', 'FVGREEN1', 'POTATOE1', 'EXEROFT1',\n",
    "                          'EXERHMM1', 'EXEROFT2', 'EXERHMM2', 'STRENGTH', 'BLDSUGAR', 'FEETCHK2', 'DOCTDIAB', 'CHKHEMO3',\n",
    "                           'FEETCHK', 'COPDSMOK', 'ASTHMAGE', 'ASDRVIST', 'ASRCHKUP', 'ASACTLIM', 'PAINACT2', 'QLMENTL2',\n",
    "                          'QLHLTH2', 'SLEPTIM1', 'ADSLEEP', 'SLEPDAY1', 'DRVISITS', 'CNCRAGE', 'SSBSUGR2','SSBFRUT3', \n",
    "                          'MARIJANA', 'HPVADSHT', 'LCSFIRST', 'LCSLAST', 'LCSNUMCG', 'SDHMOVE', 'HTIN4', 'WTKG3', '_DRNKWEK',\n",
    "                          'DROCDY3_', 'FTJUDA2_', 'FRUTDA2_', 'GRENDA1_', 'FRNCHDA_', 'POTADA1_', 'VEGEDA2_', 'METVL11_',\n",
    "                          'METVL21_', 'MAXVO2_', 'FC60_', 'PADUR1_', 'PADUR2_', 'PAFREQ1_', 'PAFREQ2_', 'STRFREQ_', '_MINAC11',\n",
    "                          '_MINAC21', 'PAMIN11_', 'PAMIN21_', 'PA1MIN_', 'PAVIG11_', 'PAVIG21_', 'PA1VIGM_', '_FRUTSU1',\n",
    "                          '_VEGESU1', 'FLSHTMY2', '_BMI5', '_AGE80']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['_STATE',\n",
      " 'CTELENM1',\n",
      " 'PVTRESD1',\n",
      " 'COLGHOUS',\n",
      " 'STATERE1',\n",
      " 'CELLFON4',\n",
      " 'LADULT',\n",
      " 'CTELNUM1',\n",
      " 'CELLFON5',\n",
      " 'CADULT',\n",
      " 'PVTRESD3',\n",
      " 'CCLGHOUS',\n",
      " 'CSTATE1',\n",
      " 'LANDLINE',\n",
      " 'GENHLTH',\n",
      " 'HLTHPLN1',\n",
      " 'PERSDOC2',\n",
      " 'MEDCOST',\n",
      " 'CHECKUP1',\n",
      " 'BPMEDS',\n",
      " 'CHOLCHK1',\n",
      " 'CHOLMED1',\n",
      " 'CVDSTRK3',\n",
      " 'CHCSCNCR',\n",
      " 'CHCOCNCR',\n",
      " 'CHCCOPD1',\n",
      " 'ADDEPEV2',\n",
      " 'CHCKIDNY',\n",
      " 'DIABETE3',\n",
      " 'LMTJOIN3',\n",
      " 'ARTHDIS2',\n",
      " 'ARTHSOCL',\n",
      " 'SEX',\n",
      " 'MARITAL',\n",
      " 'EDUCA',\n",
      " 'RENTHOM1',\n",
      " 'NUMHHOL2',\n",
      " 'CPDEMO1A',\n",
      " 'VETERAN3',\n",
      " 'EMPLOY1',\n",
      " 'INCOME2',\n",
      " 'INTERNET',\n",
      " 'PREGNANT',\n",
      " 'DEAF',\n",
      " 'BLIND',\n",
      " 'DECIDE',\n",
      " 'DIFFWALK',\n",
      " 'DIFFDRES',\n",
      " 'DIFFALON',\n",
      " 'SMOKE100',\n",
      " 'SMOKDAY2',\n",
      " 'STOPSMK2',\n",
      " 'LASTSMK2',\n",
      " 'USENOW3',\n",
      " 'ECIGARET',\n",
      " 'ECIGNOW',\n",
      " 'EXERANY2',\n",
      " 'EXRACT11',\n",
      " 'EXRACT21',\n",
      " 'SEATBELT',\n",
      " 'FLUSHOT6',\n",
      " 'PNEUVAC3',\n",
      " 'SHINGLE2',\n",
      " 'HIVTST6',\n",
      " 'HIVTSTD3',\n",
      " 'HIVRISK5',\n",
      " 'PDIABTST',\n",
      " 'PREDIAB1',\n",
      " 'INSULIN',\n",
      " 'EYEEXAM',\n",
      " 'DIABEYE',\n",
      " 'DIABEDU',\n",
      " 'COPDCOGH',\n",
      " 'COPDFLEM',\n",
      " 'COPDBRTH',\n",
      " 'COPDBTST',\n",
      " 'HAREHAB1',\n",
      " 'STREHAB1',\n",
      " 'CVDASPRN',\n",
      " 'ASPUNSAF',\n",
      " 'RLIVPAIN',\n",
      " 'RDUCHART',\n",
      " 'RDUCSTRK',\n",
      " 'BPEATHBT',\n",
      " 'BPSALT',\n",
      " 'BPALCHOL',\n",
      " 'BPEXER',\n",
      " 'BPEATADV',\n",
      " 'BPSLTADV',\n",
      " 'BPALCADV',\n",
      " 'BPEXRADV',\n",
      " 'BPMEDADV',\n",
      " 'BPHI2MR',\n",
      " 'ARTTODAY',\n",
      " 'ARTHWGT',\n",
      " 'ARTHEXER',\n",
      " 'ARTHEDU',\n",
      " 'ASATTACK',\n",
      " 'ASERVIST',\n",
      " 'ASYMPTOM',\n",
      " 'ASNOSLEP',\n",
      " 'ASTHMED3',\n",
      " 'ASINHALR',\n",
      " 'QLSTRES2',\n",
      " 'SLEPSNO2',\n",
      " 'SLEPBRTH',\n",
      " 'MEDICARE',\n",
      " 'HLTHCVR1',\n",
      " 'DELAYMED',\n",
      " 'NOCOV121',\n",
      " 'LSTCOVRG',\n",
      " 'MEDSCOS1',\n",
      " 'CARERCVD',\n",
      " 'MEDBILL1',\n",
      " 'ASBIALCH',\n",
      " 'ASBIDRNK',\n",
      " 'ASBIBING',\n",
      " 'ASBIADVC',\n",
      " 'ASBIRDUC',\n",
      " 'CNCRDIFF',\n",
      " 'CNCRTYP1',\n",
      " 'CSRVTRT2',\n",
      " 'CSRVDOC1',\n",
      " 'CSRVSUM',\n",
      " 'CSRVRTRN',\n",
      " 'CSRVINST',\n",
      " 'CSRVINSR',\n",
      " 'CSRVDEIN',\n",
      " 'CSRVCLIN',\n",
      " 'CSRVPAIN',\n",
      " 'CSRVCTL1',\n",
      " 'WTCHSALT',\n",
      " 'DRADVISE',\n",
      " 'USEMRJN1',\n",
      " 'RSNMRJNA',\n",
      " 'PFPPRVN2',\n",
      " 'TYPCNTR7',\n",
      " 'NOBCUSE6',\n",
      " 'IMFVPLAC',\n",
      " 'HPVADVC2',\n",
      " 'TETANUS',\n",
      " 'LCSCTSCN',\n",
      " 'CAREGIV1',\n",
      " 'CRGVREL2',\n",
      " 'CRGVLNG1',\n",
      " 'CRGVHRS1',\n",
      " 'CRGVPRB2',\n",
      " 'CRGVPERS',\n",
      " 'CRGVHOUS',\n",
      " 'CRGVMST2',\n",
      " 'CRGVEXPT',\n",
      " 'CIMEMLOS',\n",
      " 'CDHOUSE',\n",
      " 'CDASSIST',\n",
      " 'CDHELP',\n",
      " 'CDSOCIAL',\n",
      " 'CDDISCUS',\n",
      " 'EMTSUPRT',\n",
      " 'LSATISFY',\n",
      " 'SDHBILLS',\n",
      " 'HOWSAFE1',\n",
      " 'SDHFOOD',\n",
      " 'SDHMEALS',\n",
      " 'SDHMONEY',\n",
      " 'SDHSTRES',\n",
      " 'SXORIENT',\n",
      " 'TRNSGNDR',\n",
      " 'FIREARM4',\n",
      " 'GUNLOAD',\n",
      " 'LOADULK2',\n",
      " 'RCSGENDR',\n",
      " 'RCSRLTN2',\n",
      " 'CASTHDX2',\n",
      " 'CASTHNO2',\n",
      " 'QSTLANG',\n",
      " 'MSCODE',\n",
      " '_IMPRACE',\n",
      " '_CHISPNC',\n",
      " '_CRACE1',\n",
      " '_CPRACE',\n",
      " '_RFHLTH',\n",
      " '_PHYS14D',\n",
      " '_MENT14D',\n",
      " '_HCVU651',\n",
      " '_RFHYPE5',\n",
      " '_CHOLCH1',\n",
      " '_RFCHOL1',\n",
      " '_MICHD',\n",
      " '_LTASTH1',\n",
      " '_CASTHM1',\n",
      " '_ASTHMS1',\n",
      " '_DRDXAR1',\n",
      " '_LMTACT1',\n",
      " '_LMTWRK1',\n",
      " '_LMTSCL1',\n",
      " '_PRACE1',\n",
      " '_MRACE1',\n",
      " '_HISPANC',\n",
      " '_RACE',\n",
      " '_RACEG21',\n",
      " '_RACEGR3',\n",
      " '_RACE_G1',\n",
      " '_AGEG5YR',\n",
      " '_AGE65YR',\n",
      " '_AGE_G',\n",
      " '_BMI5CAT',\n",
      " '_RFBMI5',\n",
      " '_CHLDCNT',\n",
      " '_EDUCAG',\n",
      " '_INCOMG',\n",
      " '_SMOKER3',\n",
      " '_RFSMOK3',\n",
      " '_ECIGSTS',\n",
      " '_CURECIG',\n",
      " 'DRNKANY5',\n",
      " '_RFBING5',\n",
      " '_RFDRHV5',\n",
      " '_MISFRT1',\n",
      " '_MISVEG1',\n",
      " '_FRTRES1',\n",
      " '_VEGRES1',\n",
      " '_FRTLT1A',\n",
      " '_VEGLT1A',\n",
      " '_FRT16A',\n",
      " '_VEG23A',\n",
      " '_TOTINDA',\n",
      " 'ACTIN11_',\n",
      " 'ACTIN21_',\n",
      " '_PACAT1',\n",
      " '_PAINDX1',\n",
      " '_PA150R2',\n",
      " '_PA300R2',\n",
      " '_PA30021',\n",
      " '_PASTRNG',\n",
      " '_PAREC1',\n",
      " '_PASTAE1',\n",
      " '_RFSEAT2',\n",
      " '_RFSEAT3',\n",
      " '_FLSHOT6',\n",
      " '_PNEUMO2',\n",
      " '_AIDTST3']\n"
     ]
    }
   ],
   "source": [
    "# Categorical Features\n",
    "\n",
    "categorical_features_17 = [col_name for col_name in df_2017.columns if col_name not in countinuous_features_17]\n",
    "\n",
    "pprint(categorical_features_17)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Part 1C: Rescale variables for consistency:\n",
    "\n",
    "Categorical Variables:\n",
    "- Possible values for all outcomes should be 0 = No dx, 1 = Yes dx\n",
    "\n",
    "Continuous Variables:\n",
    "- Rescale variables w/ multiple units of measurement (e.g., in days, hours, weeks) that were not previously dropped (e.g., SSBSUGR2, SSBFRUT3)\n",
    "- <i>Note: Rescaling continuous variables was ultimately not required because all remaining unscaled columns were later dropped due to high missigness"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Rescaling Categorical Variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The original value counts before recoding:\n",
      "\n",
      "[1.0    267519\n",
      "2.0    181098\n",
      "9.0      1399\n",
      "Name: _RFHYPE5, dtype: int64,\n",
      " 1.0    241794\n",
      "2.0    156651\n",
      "Name: _RFCHOL1, dtype: int64,\n",
      " 2.0    299934\n",
      "1.0    147288\n",
      "Name: _DRDXAR1, dtype: int64]\n",
      "The NaN counts before reocoding:\n",
      "\n",
      "[0, 51571, 2794]\n"
     ]
    }
   ],
   "source": [
    "# Partial Scaled Outcome Variables:\n",
    "\n",
    "# Possible Values vary by variable (see below)\n",
    "# Grouped according to coding schema for recoding\n",
    "\n",
    "# _RFHYPE5 = High blood presure (1 = No, 2 = Yes, 9 = Don't Know/Not Sure/Refused/Missing)\n",
    "# _RFCHOL1 = High Cholesterol (1 = No, 2 = Yes, Blank = Don't Know/Not Sure/Refused/Missing)\n",
    "# _MICHD = Coronary heart disease/myocardial infarction (1 = Yes, 2 = No, 9 = Don't Know/Not Sure/Refused/Mising)\n",
    "# _LTASTH1 = Lifetime asthma (1 = No, 2 = Yes, 9 = Don't Know/Not Sure/Resfued/Missing)\n",
    "# _DRDXAR1 = Arthritis d/x (1 = Yes, 2 = No, Blank = Don't Know/Not Sure/Refused/Missing)\n",
    "\n",
    "partial_scaled_outcome_vars1 = ['_RFHYPE5', '_RFCHOL1', '_DRDXAR1'] \n",
    "\n",
    "print('The original value counts before recoding:\\n')\n",
    "pprint([df_2017[col_name].value_counts() for col_name in partial_scaled_outcome_vars1])\n",
    "\n",
    "print('The NaN counts before reocoding:\\n')\n",
    "pprint([df_2017[col_name].isna().sum() for col_name in partial_scaled_outcome_vars1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The new value counts after recoding:\n",
      "\n",
      "[0.0    267519\n",
      "1.0    181098\n",
      "Name: _RFHYPE5, dtype: int64,\n",
      " 0.0    241794\n",
      "1.0    156651\n",
      "Name: _RFCHOL1, dtype: int64,\n",
      " 1.0    299934\n",
      "0.0    147288\n",
      "Name: _DRDXAR1, dtype: int64]\n",
      "The NaN counts after recoding:\n",
      "\n",
      "[1399, 51571, 2794]\n"
     ]
    }
   ],
   "source": [
    "partial_scaled_outcome_vars1_mapper = {1.0 : 0, 2.0 : 1, 9.0: np.NaN} #1 = No, 2 = Yes\n",
    "\n",
    "for column in partial_scaled_outcome_vars1:\n",
    "    df_2017[column] = df_2017[column].map(partial_scaled_outcome_vars1_mapper)\n",
    "    \n",
    "print('The new value counts after recoding:\\n')\n",
    "pprint([df_2017[col_name].value_counts() for col_name in partial_scaled_outcome_vars1])\n",
    "\n",
    "print('The NaN counts after recoding:\\n')\n",
    "pprint([df_2017[col_name].isna().sum() for col_name in partial_scaled_outcome_vars1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The original value counts before recoding:\n",
      "\n",
      "[2.0    406515\n",
      "1.0     39357\n",
      "Name: _MICHD, dtype: int64,\n",
      " 1.0    386142\n",
      "2.0     62411\n",
      "9.0      1463\n",
      "Name: _LTASTH1, dtype: int64]\n",
      "The NaN counts before reocoding:\n",
      "\n",
      "[4144, 0]\n"
     ]
    }
   ],
   "source": [
    "partial_scaled_outcome_vars2 = ['_MICHD', '_LTASTH1',]\n",
    "\n",
    "print('The original value counts before recoding:\\n')\n",
    "pprint([df_2017[col_name].value_counts() for col_name in partial_scaled_outcome_vars2])\n",
    "\n",
    "print('The NaN counts before reocoding:\\n')\n",
    "pprint([df_2017[col_name].isna().sum() for col_name in partial_scaled_outcome_vars2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The new value counts after recoding:\n",
      "\n",
      "[0.0    406515\n",
      "1.0     39357\n",
      "Name: _MICHD, dtype: int64,\n",
      " 1.0    386142\n",
      "0.0     62411\n",
      "Name: _LTASTH1, dtype: int64]\n",
      "The NaN counts after recoding:\n",
      "\n",
      "[4144, 1463]\n"
     ]
    }
   ],
   "source": [
    "partial_scaled_outcome_vars2_mapper = {2.0 : 0, 1.0 : 1, 9.0: np.NaN}  #1 = Yes, 2 = No\n",
    "\n",
    "for column in partial_scaled_outcome_vars2:\n",
    "    df_2017[column] = df_2017[column].map(partial_scaled_outcome_vars2_mapper)\n",
    "    \n",
    "print('The new value counts after recoding:\\n')\n",
    "pprint([df_2017[col_name].value_counts() for col_name in partial_scaled_outcome_vars2])\n",
    "\n",
    "print('The NaN counts after recoding:\\n')\n",
    "pprint([df_2017[col_name].isna().sum() for col_name in partial_scaled_outcome_vars2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The original value counts before recoding:\n",
      "\n",
      "[2.0    429710\n",
      "1.0     18956\n",
      "7.0      1200\n",
      "9.0       148\n",
      "Name: CVDSTRK3, dtype: int64,\n",
      " 2.0    406161\n",
      "1.0     42661\n",
      "7.0      1027\n",
      "9.0       165\n",
      "Name: CHCSCNCR, dtype: int64,\n",
      " 2.0    404637\n",
      "1.0     44289\n",
      "7.0       867\n",
      "9.0       220\n",
      "Name: CHCOCNCR, dtype: int64,\n",
      " 2.0    410141\n",
      "1.0     37577\n",
      "7.0      2099\n",
      "9.0       197\n",
      "Name: CHCCOPD1, dtype: int64,\n",
      " 2.0    358683\n",
      "1.0     89209\n",
      "7.0      1661\n",
      "9.0       460\n",
      "Name: ADDEPEV2, dtype: int64,\n",
      " 2.0    430890\n",
      "1.0     17547\n",
      "7.0      1368\n",
      "9.0       208\n",
      "Name: CHCKIDNY, dtype: int64]\n",
      "The NaN counts before reocoding:\n",
      "\n",
      "[2, 2, 3, 2, 3, 3]\n"
     ]
    }
   ],
   "source": [
    "# Unscaled Outcome Variables:\n",
    "\n",
    "# Possible Values: 1 = Yes, 2 = No, 7 = Don't Know/Not Sure, 9 = Refused, Blank = Not Asked/Missing\n",
    "\n",
    "# CVDSTRK3 = Stroke\n",
    "# CHCSCNCR = Skin Cancer\n",
    "# CHCOCNCR = Other Cancer\n",
    "# ADDEPEV2 = Depressive Disorder\n",
    "# CHCKIDNY = Chronic Kidney Disease\n",
    "\n",
    "unscaled_outcome_vars1 = ['CVDSTRK3', 'CHCSCNCR', 'CHCOCNCR', 'CHCCOPD1', 'ADDEPEV2', 'CHCKIDNY']\n",
    "\n",
    "print('The original value counts before recoding:\\n')\n",
    "pprint([df_2017[col_name].value_counts() for col_name in unscaled_outcome_vars1])\n",
    "\n",
    "print('The NaN counts before reocoding:\\n')\n",
    "pprint([df_2017[col_name].isna().sum() for col_name in unscaled_outcome_vars1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The new value counts after recoding:\n",
      "\n",
      "[0.0    429710\n",
      "1.0     18956\n",
      "Name: CVDSTRK3, dtype: int64,\n",
      " 0.0    406161\n",
      "1.0     42661\n",
      "Name: CHCSCNCR, dtype: int64,\n",
      " 0.0    404637\n",
      "1.0     44289\n",
      "Name: CHCOCNCR, dtype: int64,\n",
      " 0.0    410141\n",
      "1.0     37577\n",
      "Name: CHCCOPD1, dtype: int64,\n",
      " 0.0    358683\n",
      "1.0     89209\n",
      "Name: ADDEPEV2, dtype: int64,\n",
      " 0.0    430890\n",
      "1.0     17547\n",
      "Name: CHCKIDNY, dtype: int64]\n",
      "The NaN counts after recoding:\n",
      "\n",
      "[1350, 1194, 1090, 2298, 2124, 1579]\n"
     ]
    }
   ],
   "source": [
    "unscaled_outcome_vars1_mapper = {1.0 : 1, 2.0 : 0, 7.0: np.NaN, 9.0: np.NaN} #1 = Yes, 2 = No, 7 = Don't Know/Not Sure, 9 = Refused, Blank = Not Asked/Missing\n",
    "\n",
    "for column in unscaled_outcome_vars1:\n",
    "    df_2017[column] = df_2017[column].map(unscaled_outcome_vars1_mapper)\n",
    "    \n",
    "print('The new value counts after recoding:\\n')\n",
    "pprint([df_2017[col_name].value_counts() for col_name in unscaled_outcome_vars1])\n",
    "\n",
    "print('The NaN counts after recoding:\\n')\n",
    "pprint([df_2017[col_name].isna().sum() for col_name in unscaled_outcome_vars1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The original value counts before recoding:\n",
      "\n",
      "3.0    377264\n",
      "1.0     60440\n",
      "4.0      8232\n",
      "2.0      3283\n",
      "7.0       594\n",
      "9.0       199\n",
      "Name: DIABETE3, dtype: int64\n",
      "The NaN counts before reocoding: 4\n"
     ]
    }
   ],
   "source": [
    "unscaled_outcome_vars2 = ['DIABETE3']\n",
    "\n",
    "print('The original value counts before recoding:\\n')\n",
    "print(df_2017['DIABETE3'].value_counts())\n",
    "\n",
    "print('The NaN counts before reocoding:', df_2017['DIABETE3'].isna().sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The new value counts after recoding:\n",
      "\n",
      "0.0    385496\n",
      "1.0     63723\n",
      "Name: DIABETE3, dtype: int64\n",
      "The NaN counts after reocoding: 797\n"
     ]
    }
   ],
   "source": [
    "# Possible Values for DIABETE3 only: 1 = Yes, 2 = Yes but told during pregnancy, 3 = No, 4 = No (pre-diabetes or borderline),\n",
    "# 7 = Don't know/Not Sure, 9 = Refused, Blank = Missing\n",
    "\n",
    "unscaled_outcome_vars2_mapper = {1.0 : 1, 2.0 : 1, 3.0: 0, 4.0: 0, 7.0: np.NaN, 9.0: np.NaN}\n",
    "\n",
    "df_2017['DIABETE3'] = df_2017['DIABETE3'].map(unscaled_outcome_vars2_mapper)\n",
    "    \n",
    "print('The new value counts after recoding:\\n')\n",
    "print(df_2017['DIABETE3'].value_counts())\n",
    "\n",
    "print('The NaN counts after reocoding:', df_2017['DIABETE3'].isna().sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['_RFHYPE5',\n",
      " '_RFCHOL1',\n",
      " '_DRDXAR1',\n",
      " '_MICHD',\n",
      " '_LTASTH1',\n",
      " 'CVDSTRK3',\n",
      " 'CHCSCNCR',\n",
      " 'CHCOCNCR',\n",
      " 'CHCCOPD1',\n",
      " 'ADDEPEV2',\n",
      " 'CHCKIDNY',\n",
      " 'DIABETE3']\n"
     ]
    }
   ],
   "source": [
    "all_possible_outcomes = partial_scaled_outcome_vars1 + partial_scaled_outcome_vars2 + unscaled_outcome_vars1 + unscaled_outcome_vars2\n",
    "\n",
    "pprint(all_possible_outcomes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Part 1D: Deal with missingness:\n",
    "\n",
    "- Drop collapsed missing data (if > 60% missing columnwise) OR attempt Imputation (if < 60% missing)\n",
    "- Impute missing data using sklearn SimpleImputer (mode)\n",
    "- <i>NOTE: Multiple Imputation is a future direction of pre-processing and may provide more reliable imputation results given it's consideration of multiple vs. single features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 149 columns with >= 60% missing data to be dropped.\n",
      "['COLGHOUS',\n",
      " 'LADULT',\n",
      " 'CCLGHOUS',\n",
      " 'CHOLMED1',\n",
      " 'DIABAGE2',\n",
      " 'LMTJOIN3',\n",
      " 'ARTHDIS2',\n",
      " 'ARTHSOCL',\n",
      " 'JOINPAI1',\n",
      " 'NUMPHON2',\n",
      " 'PREGNANT',\n",
      " 'STOPSMK2',\n",
      " 'LASTSMK2',\n",
      " 'ECIGNOW',\n",
      " 'HIVTSTD3',\n",
      " 'INSULIN',\n",
      " 'BLDSUGAR',\n",
      " 'FEETCHK2',\n",
      " 'DOCTDIAB',\n",
      " 'CHKHEMO3',\n",
      " 'FEETCHK',\n",
      " 'EYEEXAM',\n",
      " 'DIABEYE',\n",
      " 'DIABEDU',\n",
      " 'COPDCOGH',\n",
      " 'COPDFLEM',\n",
      " 'COPDBRTH',\n",
      " 'COPDBTST',\n",
      " 'COPDSMOK',\n",
      " 'HAREHAB1',\n",
      " 'STREHAB1',\n",
      " 'CVDASPRN',\n",
      " 'ASPUNSAF',\n",
      " 'RLIVPAIN',\n",
      " 'RDUCHART',\n",
      " 'RDUCSTRK',\n",
      " 'BPEATHBT',\n",
      " 'BPSALT',\n",
      " 'BPALCHOL',\n",
      " 'BPEXER',\n",
      " 'BPEATADV',\n",
      " 'BPSLTADV',\n",
      " 'BPALCADV',\n",
      " 'BPEXRADV',\n",
      " 'BPMEDADV',\n",
      " 'BPHI2MR',\n",
      " 'ARTTODAY',\n",
      " 'ARTHWGT',\n",
      " 'ARTHEXER',\n",
      " 'ARTHEDU',\n",
      " 'ASTHMAGE',\n",
      " 'ASATTACK',\n",
      " 'ASERVIST',\n",
      " 'ASDRVIST',\n",
      " 'ASRCHKUP',\n",
      " 'ASACTLIM',\n",
      " 'ASYMPTOM',\n",
      " 'ASNOSLEP',\n",
      " 'ASTHMED3',\n",
      " 'ASINHALR',\n",
      " 'PAINACT2',\n",
      " 'QLMENTL2',\n",
      " 'QLSTRES2',\n",
      " 'QLHLTH2',\n",
      " 'SLEPTIM1',\n",
      " 'ADSLEEP',\n",
      " 'SLEPDAY1',\n",
      " 'SLEPSNO2',\n",
      " 'SLEPBRTH',\n",
      " 'MEDICARE',\n",
      " 'HLTHCVR1',\n",
      " 'DELAYMED',\n",
      " 'NOCOV121',\n",
      " 'LSTCOVRG',\n",
      " 'DRVISITS',\n",
      " 'MEDSCOS1',\n",
      " 'CARERCVD',\n",
      " 'MEDBILL1',\n",
      " 'ASBIALCH',\n",
      " 'ASBIDRNK',\n",
      " 'ASBIBING',\n",
      " 'ASBIADVC',\n",
      " 'ASBIRDUC',\n",
      " 'CNCRDIFF',\n",
      " 'CNCRAGE',\n",
      " 'CNCRTYP1',\n",
      " 'CSRVTRT2',\n",
      " 'CSRVDOC1',\n",
      " 'CSRVSUM',\n",
      " 'CSRVRTRN',\n",
      " 'CSRVINST',\n",
      " 'CSRVINSR',\n",
      " 'CSRVDEIN',\n",
      " 'CSRVCLIN',\n",
      " 'CSRVPAIN',\n",
      " 'CSRVCTL1',\n",
      " 'SSBSUGR2',\n",
      " 'SSBFRUT3',\n",
      " 'WTCHSALT',\n",
      " 'DRADVISE',\n",
      " 'MARIJANA',\n",
      " 'USEMRJN1',\n",
      " 'RSNMRJNA',\n",
      " 'PFPPRVN2',\n",
      " 'TYPCNTR7',\n",
      " 'NOBCUSE6',\n",
      " 'IMFVPLAC',\n",
      " 'HPVADVC2',\n",
      " 'HPVADSHT',\n",
      " 'TETANUS',\n",
      " 'LCSFIRST',\n",
      " 'LCSLAST',\n",
      " 'LCSNUMCG',\n",
      " 'LCSCTSCN',\n",
      " 'CAREGIV1',\n",
      " 'CRGVREL2',\n",
      " 'CRGVLNG1',\n",
      " 'CRGVHRS1',\n",
      " 'CRGVPRB2',\n",
      " 'CRGVPERS',\n",
      " 'CRGVHOUS',\n",
      " 'CRGVMST2',\n",
      " 'CRGVEXPT',\n",
      " 'CIMEMLOS',\n",
      " 'CDHOUSE',\n",
      " 'CDASSIST',\n",
      " 'CDHELP',\n",
      " 'CDSOCIAL',\n",
      " 'CDDISCUS',\n",
      " 'EMTSUPRT',\n",
      " 'LSATISFY',\n",
      " 'SDHBILLS',\n",
      " 'SDHMOVE',\n",
      " 'HOWSAFE1',\n",
      " 'SDHFOOD',\n",
      " 'SDHMEALS',\n",
      " 'SDHMONEY',\n",
      " 'SDHSTRES',\n",
      " 'FIREARM4',\n",
      " 'GUNLOAD',\n",
      " 'LOADULK2',\n",
      " 'RCSGENDR',\n",
      " 'RCSRLTN2',\n",
      " 'CASTHDX2',\n",
      " 'CASTHNO2',\n",
      " '_CRACE1',\n",
      " '_CPRACE',\n",
      " '_FLSHOT6',\n",
      " '_PNEUMO2']\n"
     ]
    }
   ],
   "source": [
    "#Identifying features w/ high missingness (>= 60% of total sample)\n",
    "\n",
    "missingness_threshold = round(df_2017.shape[0]*.60)\n",
    "\n",
    "high_missingness = [col_name for col_name in df_2017.columns if df_2017[col_name].isna().sum() >= missingness_threshold and col_name not in all_possible_outcomes]\n",
    "\n",
    "print('There are', len(high_missingness), 'columns with >= 60% missing data to be dropped.')\n",
    "pprint(high_missingness)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dataframe is the expected shape after dropping high missingness columns: True\n"
     ]
    }
   ],
   "source": [
    "# Dropping columns w/ high missingness (before collapsing missingness)\n",
    "\n",
    "df_2017.drop(columns = high_missingness, inplace = True)\n",
    "\n",
    "print('The dataframe is the expected shape after dropping high missingness columns:', df_2017.shape[1] == 312 - len(high_missingness))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Checking NaN counts before imputation\n",
    "\n",
    "#pprint((list(df_2017.isna().sum())))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Applying Simple Imputation, replacing missingness with mode values by columns\n",
    "\n",
    "imp = SimpleImputer(missing_values = np.NaN, strategy=\"most_frequent\")\n",
    "imputed_df_2017 = pd.DataFrame(imp.fit_transform(df_2017), columns = df_2017.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Checking NaN counts after imputation\n",
    "\n",
    "#pprint((list(imputed_df_2017.isna().sum())))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_STATE</th>\n",
       "      <th>CTELENM1</th>\n",
       "      <th>PVTRESD1</th>\n",
       "      <th>STATERE1</th>\n",
       "      <th>CELLFON4</th>\n",
       "      <th>NUMADULT</th>\n",
       "      <th>NUMMEN</th>\n",
       "      <th>NUMWOMEN</th>\n",
       "      <th>CTELNUM1</th>\n",
       "      <th>CELLFON5</th>\n",
       "      <th>...</th>\n",
       "      <th>_PAINDX1</th>\n",
       "      <th>_PA150R2</th>\n",
       "      <th>_PA300R2</th>\n",
       "      <th>_PA30021</th>\n",
       "      <th>_PASTRNG</th>\n",
       "      <th>_PAREC1</th>\n",
       "      <th>_PASTAE1</th>\n",
       "      <th>_RFSEAT2</th>\n",
       "      <th>_RFSEAT3</th>\n",
       "      <th>_AIDTST3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 163 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   _STATE  CTELENM1  PVTRESD1  STATERE1  CELLFON4  NUMADULT        NUMMEN  \\\n",
       "0     1.0       1.0       1.0       1.0       2.0       1.0  5.397605e-79   \n",
       "1     1.0       1.0       1.0       1.0       2.0       2.0  1.000000e+00   \n",
       "2     1.0       1.0       1.0       1.0       2.0       2.0  1.000000e+00   \n",
       "3     1.0       1.0       1.0       1.0       2.0       1.0  5.397605e-79   \n",
       "4     1.0       1.0       1.0       1.0       2.0       1.0  5.397605e-79   \n",
       "\n",
       "   NUMWOMEN  CTELNUM1  CELLFON5    ...     _PAINDX1  _PA150R2  _PA300R2  \\\n",
       "0       1.0       1.0       1.0    ...          1.0       1.0       1.0   \n",
       "1       1.0       1.0       1.0    ...          1.0       1.0       1.0   \n",
       "2       1.0       1.0       1.0    ...          2.0       3.0       3.0   \n",
       "3       1.0       1.0       1.0    ...          9.0       9.0       9.0   \n",
       "4       1.0       1.0       1.0    ...          2.0       3.0       3.0   \n",
       "\n",
       "   _PA30021  _PASTRNG  _PAREC1  _PASTAE1  _RFSEAT2  _RFSEAT3  _AIDTST3  \n",
       "0       1.0       1.0      1.0       1.0       1.0       1.0       1.0  \n",
       "1       1.0       1.0      1.0       1.0       1.0       1.0       1.0  \n",
       "2       2.0       2.0      4.0       2.0       1.0       1.0       2.0  \n",
       "3       9.0       9.0      9.0       9.0       9.0       9.0       2.0  \n",
       "4       2.0       2.0      4.0       2.0       1.0       2.0       2.0  \n",
       "\n",
       "[5 rows x 163 columns]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imputed_df_2017.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Part 1E: Collapse Missingness\n",
    "\n",
    "- Collapse values corresponding to missingness into a single category so Don't Know/Not Sure = Refused = Blank"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Checking which categorical values still exist in the dataset and require missigness recoding\n",
    "#Exceptions: EMPLOY1 - 7 = Retired, 8 = Unable to Work, 9 = Don't Know\n",
    "\n",
    "# Select variables that are categorical and have at least 1 np.NaN value\n",
    "to_collapse_cat = [col_name for col_name in imputed_df_2017.columns if col_name in categorical_features_17 and df_2017[col_name].isna().sum() != 0]\n",
    "\n",
    "# Select variables that have 7, 9, 77, 99, 88, 98, 777777, 999999 as possible values\n",
    "to_collapse_cat2 = []\n",
    "to_collapse_dict_cat = defaultdict(list)\n",
    "\n",
    "for col in to_collapse_cat:\n",
    "    if all(elem in np.unique(imputed_df_2017[col].values) for elem in [99., 77.]):\n",
    "        to_collapse_cat2.append(col)\n",
    "        to_collapse_dict_cat['map1'].append(col)\n",
    "    elif all(elem in np.unique(imputed_df_2017[col].values) for elem in [7.0, 9.0]):\n",
    "        to_collapse_cat2.append(col)\n",
    "        to_collapse_dict_cat['map2'].append(col)\n",
    "    elif all(elem in np.unique(imputed_df_2017[col].values) for elem in [777777.0, 999999.0]):\n",
    "        to_collapse_cat2.append(col)\n",
    "        to_collapse_dict_cat['map3'].append(col)\n",
    "    elif all(elem in np.unique(imputed_df_2017[col].values) for elem in [88.0, 98.0]):\n",
    "        to_collapse_cat2.append(col)\n",
    "        to_collapse_dict_cat['map4'].append(col)\n",
    "    elif 7.0 in imputed_df_2017[col].values:\n",
    "        to_collapse_cat2.append(col)\n",
    "        to_collapse_dict_cat['map5'].append(col)\n",
    "    elif 9.0 in imputed_df_2017[col].values:\n",
    "        to_collapse_cat2.append(col)\n",
    "        to_collapse_dict_cat['map6'].append(col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The original value counts before collapsing missingness:\n",
      "\n",
      "[2.0    365973\n",
      "1.0     83689\n",
      "7.0       200\n",
      "9.0       154\n",
      "Name: LANDLINE, dtype: int64,\n",
      " 2.0    145482\n",
      "3.0    142502\n",
      "1.0     74356\n",
      "4.0     62213\n",
      "5.0     24317\n",
      "7.0       707\n",
      "9.0       439\n",
      "Name: GENHLTH, dtype: int64,\n",
      " 1.0    412503\n",
      "2.0     35743\n",
      "7.0      1073\n",
      "9.0       697\n",
      "Name: HLTHPLN1, dtype: int64,\n",
      " 1.0    340810\n",
      "3.0     73313\n",
      "2.0     34083\n",
      "7.0      1212\n",
      "9.0       598\n",
      "Name: PERSDOC2, dtype: int64,\n",
      " 2.0    401191\n",
      "1.0     47528\n",
      "7.0      1001\n",
      "9.0       296\n",
      "Name: MEDCOST, dtype: int64,\n",
      " 1.0    331914\n",
      "2.0     50812\n",
      "3.0     29656\n",
      "4.0     27861\n",
      "7.0      5260\n",
      "8.0      3988\n",
      "9.0       525\n",
      "Name: CHECKUP1, dtype: int64,\n",
      " 1.0    417966\n",
      "2.0     31696\n",
      "7.0       232\n",
      "9.0       122\n",
      "Name: BPMEDS, dtype: int64,\n",
      " 2.0    316370\n",
      "3.0     43534\n",
      "1.0     26495\n",
      "4.0     25296\n",
      "7.0     21164\n",
      "5.0     16572\n",
      "9.0       585\n",
      "Name: CHOLCHK1, dtype: int64,\n",
      " 1.0    232898\n",
      "5.0     73939\n",
      "2.0     61437\n",
      "3.0     54633\n",
      "6.0     14544\n",
      "4.0      9426\n",
      "9.0      3139\n",
      "Name: MARITAL, dtype: int64,\n",
      " 6.0    168396\n",
      "5.0    124655\n",
      "4.0    122577\n",
      "3.0     21624\n",
      "2.0     10434\n",
      "9.0      1701\n",
      "1.0       629\n",
      "Name: EDUCA, dtype: int64,\n",
      " 1.0    317845\n",
      "2.0    107897\n",
      "3.0     21341\n",
      "9.0      2081\n",
      "7.0       852\n",
      "Name: RENTHOM1, dtype: int64,\n",
      " 2.0    437847\n",
      "1.0     11466\n",
      "9.0       527\n",
      "7.0       176\n",
      "Name: NUMHHOL2, dtype: int64,\n",
      " 1.0    412838\n",
      "2.0     36188\n",
      "9.0       876\n",
      "7.0       114\n",
      "Name: CPDEMO1A, dtype: int64,\n",
      " 2.0    391419\n",
      "1.0     57868\n",
      "9.0       609\n",
      "7.0       120\n",
      "Name: VETERAN3, dtype: int64,\n",
      " 1.0    184099\n",
      "7.0    133648\n",
      "2.0     39592\n",
      "8.0     32706\n",
      "5.0     24376\n",
      "6.0     12181\n",
      "3.0     10489\n",
      "4.0      9225\n",
      "9.0      3700\n",
      "Name: EMPLOY1, dtype: int64,\n",
      " 8.0     126094\n",
      "7.0      59632\n",
      "6.0      53148\n",
      "5.0      39751\n",
      "99.0     38426\n",
      "4.0      34222\n",
      "77.0     33328\n",
      "3.0      27735\n",
      "2.0      19334\n",
      "1.0      18346\n",
      "Name: INCOME2, dtype: int64,\n",
      " 1.0    367183\n",
      "2.0     81659\n",
      "9.0       735\n",
      "7.0       439\n",
      "Name: INTERNET, dtype: int64,\n",
      " 2.0    410716\n",
      "1.0     37768\n",
      "7.0      1007\n",
      "9.0       525\n",
      "Name: DEAF, dtype: int64,\n",
      " 2.0    426344\n",
      "1.0     22464\n",
      "7.0       815\n",
      "9.0       393\n",
      "Name: BLIND, dtype: int64,\n",
      " 2.0    401421\n",
      "1.0     46069\n",
      "7.0      1981\n",
      "9.0       545\n",
      "Name: DECIDE, dtype: int64,\n",
      " 2.0    372001\n",
      "1.0     76457\n",
      "7.0      1127\n",
      "9.0       431\n",
      "Name: DIFFWALK, dtype: int64,\n",
      " 2.0    429237\n",
      "1.0     20062\n",
      "7.0       403\n",
      "9.0       314\n",
      "Name: DIFFDRES, dtype: int64,\n",
      " 2.0    414737\n",
      "1.0     33949\n",
      "7.0       902\n",
      "9.0       428\n",
      "Name: DIFFALON, dtype: int64,\n",
      " 2.0    262038\n",
      "1.0    185596\n",
      "7.0      1937\n",
      "9.0       445\n",
      "Name: SMOKE100, dtype: int64,\n",
      " 3.0    386215\n",
      "1.0     44617\n",
      "2.0     18849\n",
      "9.0       237\n",
      "7.0        98\n",
      "Name: SMOKDAY2, dtype: int64,\n",
      " 3.0    434064\n",
      "1.0      8456\n",
      "2.0      6684\n",
      "9.0       640\n",
      "7.0       172\n",
      "Name: USENOW3, dtype: int64,\n",
      " 2.0    382318\n",
      "1.0     66793\n",
      "9.0       622\n",
      "7.0       283\n",
      "Name: ECIGARET, dtype: int64,\n",
      " 1.0    334183\n",
      "2.0    114783\n",
      "7.0       588\n",
      "9.0       462\n",
      "Name: EXERANY2, dtype: int64,\n",
      " 64.0    310430\n",
      "37.0     19644\n",
      "18.0     16127\n",
      "98.0     15979\n",
      "67.0     12927\n",
      "7.0       7785\n",
      "6.0       6579\n",
      "2.0       6237\n",
      "76.0      4597\n",
      "11.0      4458\n",
      "15.0      4376\n",
      "19.0      3796\n",
      "69.0      3607\n",
      "22.0      3305\n",
      "57.0      3164\n",
      "28.0      2659\n",
      "5.0       1971\n",
      "14.0      1817\n",
      "73.0      1762\n",
      "20.0      1570\n",
      "58.0      1468\n",
      "31.0      1240\n",
      "72.0      1208\n",
      "61.0      1160\n",
      "51.0       934\n",
      "54.0       807\n",
      "77.0       725\n",
      "34.0       659\n",
      "9.0        511\n",
      "74.0       508\n",
      "         ...  \n",
      "35.0       189\n",
      "56.0       165\n",
      "43.0       153\n",
      "36.0       151\n",
      "44.0       126\n",
      "12.0       120\n",
      "26.0       113\n",
      "68.0       110\n",
      "17.0       110\n",
      "1.0        107\n",
      "66.0        97\n",
      "50.0        85\n",
      "62.0        66\n",
      "33.0        61\n",
      "30.0        51\n",
      "59.0        49\n",
      "47.0        46\n",
      "4.0         43\n",
      "39.0        34\n",
      "21.0        34\n",
      "41.0        33\n",
      "46.0        29\n",
      "53.0        26\n",
      "3.0         23\n",
      "42.0        22\n",
      "29.0        22\n",
      "32.0        21\n",
      "27.0        13\n",
      "45.0        12\n",
      "55.0        11\n",
      "Name: EXRACT11, Length: 77, dtype: int64,\n",
      " 88.0    250527\n",
      "64.0     41207\n",
      "98.0     18746\n",
      "67.0     18699\n",
      "18.0     13248\n",
      "73.0     12283\n",
      "76.0      9738\n",
      "37.0      9484\n",
      "7.0       7928\n",
      "69.0      5620\n",
      "6.0       5583\n",
      "57.0      4962\n",
      "11.0      4890\n",
      "2.0       3945\n",
      "15.0      3139\n",
      "22.0      2924\n",
      "19.0      2745\n",
      "77.0      2583\n",
      "54.0      2509\n",
      "31.0      2493\n",
      "14.0      2217\n",
      "71.0      2024\n",
      "5.0       2008\n",
      "28.0      1816\n",
      "72.0      1793\n",
      "58.0      1521\n",
      "20.0      1410\n",
      "48.0      1052\n",
      "61.0       938\n",
      "99.0       901\n",
      "         ...  \n",
      "26.0       183\n",
      "33.0       179\n",
      "35.0       176\n",
      "38.0       162\n",
      "23.0       160\n",
      "1.0        160\n",
      "43.0       152\n",
      "44.0       143\n",
      "17.0       140\n",
      "50.0       133\n",
      "56.0       115\n",
      "68.0       111\n",
      "47.0       100\n",
      "39.0        73\n",
      "62.0        70\n",
      "66.0        63\n",
      "4.0         48\n",
      "59.0        48\n",
      "53.0        41\n",
      "42.0        31\n",
      "30.0        30\n",
      "32.0        29\n",
      "3.0         26\n",
      "29.0        24\n",
      "45.0        24\n",
      "41.0        22\n",
      "27.0        21\n",
      "21.0        21\n",
      "55.0        20\n",
      "46.0        18\n",
      "Name: EXRACT21, Length: 78, dtype: int64,\n",
      " 1.0    398563\n",
      "2.0     27960\n",
      "3.0     10843\n",
      "5.0      6453\n",
      "4.0      4608\n",
      "8.0       900\n",
      "9.0       406\n",
      "7.0       283\n",
      "Name: SEATBELT, dtype: int64,\n",
      " 2.0    258392\n",
      "1.0    189998\n",
      "7.0      1184\n",
      "9.0       442\n",
      "Name: FLUSHOT6, dtype: int64,\n",
      " 2.0    241399\n",
      "1.0    174476\n",
      "7.0     33683\n",
      "9.0       458\n",
      "Name: PNEUVAC3, dtype: int64,\n",
      " 2.0    356948\n",
      "1.0     87890\n",
      "7.0      4973\n",
      "9.0       205\n",
      "Name: SHINGLE2, dtype: int64,\n",
      " 2.0    309928\n",
      "1.0    126734\n",
      "7.0     12174\n",
      "9.0      1180\n",
      "Name: HIVTST6, dtype: int64,\n",
      " 2.0    432626\n",
      "1.0     16030\n",
      "9.0      1163\n",
      "7.0       197\n",
      "Name: HIVRISK5, dtype: int64,\n",
      " 1.0    349015\n",
      "2.0     89609\n",
      "7.0     11219\n",
      "9.0       173\n",
      "Name: PDIABTST, dtype: int64,\n",
      " 3.0    417292\n",
      "1.0     28972\n",
      "2.0      2609\n",
      "7.0       998\n",
      "9.0       145\n",
      "Name: PREDIAB1, dtype: int64,\n",
      " 1.0    436842\n",
      "9.0      3585\n",
      "3.0      3579\n",
      "2.0      3167\n",
      "7.0      1954\n",
      "4.0       889\n",
      "Name: SXORIENT, dtype: int64,\n",
      " 4.0    446869\n",
      "9.0      1491\n",
      "7.0       822\n",
      "1.0       412\n",
      "2.0       240\n",
      "3.0       182\n",
      "Name: TRNSGNDR, dtype: int64,\n",
      " 9.0    390804\n",
      "2.0     49617\n",
      "1.0      9595\n",
      "Name: _CHISPNC, dtype: int64,\n",
      " 3.0    302728\n",
      "1.0     74357\n",
      "2.0     71721\n",
      "9.0      1210\n",
      "Name: _LMTACT1, dtype: int64,\n",
      " 3.0    302728\n",
      "2.0     96393\n",
      "1.0     47809\n",
      "9.0      3086\n",
      "Name: _LMTWRK1, dtype: int64,\n",
      " 4.0    302728\n",
      "3.0     78726\n",
      "2.0     38329\n",
      "1.0     29313\n",
      "9.0       920\n",
      "Name: _LMTSCL1, dtype: int64,\n",
      " 1.0     363803\n",
      "2.0      38825\n",
      "4.0      10951\n",
      "3.0      10715\n",
      "6.0      10026\n",
      "99.0      7012\n",
      "77.0      4120\n",
      "5.0       3127\n",
      "7.0       1437\n",
      "Name: _PRACE1, dtype: int64,\n",
      " 1.0     359267\n",
      "2.0      37655\n",
      "7.0      10297\n",
      "4.0      10166\n",
      "3.0       9823\n",
      "6.0       9760\n",
      "99.0      7012\n",
      "77.0      4120\n",
      "5.0       1916\n",
      "Name: _MRACE1, dtype: int64,\n",
      " 2.0    309928\n",
      "1.0    126734\n",
      "9.0     13354\n",
      "Name: _AIDTST3, dtype: int64]\n",
      "The NaN counts before collapsing missingness:\n",
      "\n",
      "[0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0,\n",
      " 0]\n"
     ]
    }
   ],
   "source": [
    "print('The original value counts before collapsing missingness:\\n')\n",
    "pprint([imputed_df_2017[col_name].value_counts() for col_name in to_collapse_cat2])\n",
    "\n",
    "print('The NaN counts before collapsing missingness:\\n')\n",
    "pprint([imputed_df_2017[col_name].isna().sum() for col_name in to_collapse_cat2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Recategorizing misclassified column 'EMPLOY1'\n",
    "\n",
    "to_collapse_dict_cat['map2'].remove('EMPLOY1')\n",
    "to_collapse_dict_cat['map6'].append('EMPLOY1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The new value counts after recoding:\n",
      "\n",
      "[ 8.0       126094\n",
      "-9999.0     71754\n",
      " 7.0        59632\n",
      " 6.0        53148\n",
      " 5.0        39751\n",
      " 4.0        34222\n",
      " 3.0        27735\n",
      " 2.0        19334\n",
      " 1.0        18346\n",
      "Name: INCOME2, dtype: int64,\n",
      "  64.0      310430\n",
      " 37.0       19644\n",
      " 18.0       16127\n",
      " 98.0       15979\n",
      " 67.0       12927\n",
      " 7.0         7785\n",
      " 6.0         6579\n",
      " 2.0         6237\n",
      " 76.0        4597\n",
      " 11.0        4458\n",
      " 15.0        4376\n",
      " 19.0        3796\n",
      " 69.0        3607\n",
      " 22.0        3305\n",
      " 57.0        3164\n",
      " 28.0        2659\n",
      " 5.0         1971\n",
      " 14.0        1817\n",
      " 73.0        1762\n",
      " 20.0        1570\n",
      " 58.0        1468\n",
      " 31.0        1240\n",
      " 72.0        1208\n",
      "-9999.0      1177\n",
      " 61.0        1160\n",
      " 51.0         934\n",
      " 54.0         807\n",
      " 34.0         659\n",
      " 9.0          511\n",
      " 74.0         508\n",
      "            ...  \n",
      " 35.0         189\n",
      " 56.0         165\n",
      " 43.0         153\n",
      " 36.0         151\n",
      " 44.0         126\n",
      " 12.0         120\n",
      " 26.0         113\n",
      " 17.0         110\n",
      " 68.0         110\n",
      " 1.0          107\n",
      " 66.0          97\n",
      " 50.0          85\n",
      " 62.0          66\n",
      " 33.0          61\n",
      " 30.0          51\n",
      " 59.0          49\n",
      " 47.0          46\n",
      " 4.0           43\n",
      " 39.0          34\n",
      " 21.0          34\n",
      " 41.0          33\n",
      " 46.0          29\n",
      " 53.0          26\n",
      " 3.0           23\n",
      " 29.0          22\n",
      " 42.0          22\n",
      " 32.0          21\n",
      " 27.0          13\n",
      " 45.0          12\n",
      " 55.0          11\n",
      "Name: EXRACT11, Length: 76, dtype: int64,\n",
      "  88.0      250527\n",
      " 64.0       41207\n",
      " 98.0       18746\n",
      " 67.0       18699\n",
      " 18.0       13248\n",
      " 73.0       12283\n",
      " 76.0        9738\n",
      " 37.0        9484\n",
      " 7.0         7928\n",
      " 69.0        5620\n",
      " 6.0         5583\n",
      " 57.0        4962\n",
      " 11.0        4890\n",
      " 2.0         3945\n",
      "-9999.0      3484\n",
      " 15.0        3139\n",
      " 22.0        2924\n",
      " 19.0        2745\n",
      " 54.0        2509\n",
      " 31.0        2493\n",
      " 14.0        2217\n",
      " 71.0        2024\n",
      " 5.0         2008\n",
      " 28.0        1816\n",
      " 72.0        1793\n",
      " 58.0        1521\n",
      " 20.0        1410\n",
      " 48.0        1052\n",
      " 61.0         938\n",
      " 34.0         852\n",
      "            ...  \n",
      " 26.0         183\n",
      " 33.0         179\n",
      " 35.0         176\n",
      " 38.0         162\n",
      " 23.0         160\n",
      " 1.0          160\n",
      " 43.0         152\n",
      " 44.0         143\n",
      " 17.0         140\n",
      " 50.0         133\n",
      " 56.0         115\n",
      " 68.0         111\n",
      " 47.0         100\n",
      " 39.0          73\n",
      " 62.0          70\n",
      " 66.0          63\n",
      " 59.0          48\n",
      " 4.0           48\n",
      " 53.0          41\n",
      " 42.0          31\n",
      " 30.0          30\n",
      " 32.0          29\n",
      " 3.0           26\n",
      " 29.0          24\n",
      " 45.0          24\n",
      " 41.0          22\n",
      " 21.0          21\n",
      " 27.0          21\n",
      " 55.0          20\n",
      " 46.0          18\n",
      "Name: EXRACT21, Length: 77, dtype: int64,\n",
      "  1.0       363803\n",
      " 2.0        38825\n",
      "-9999.0     11132\n",
      " 4.0        10951\n",
      " 3.0        10715\n",
      " 6.0        10026\n",
      " 5.0         3127\n",
      " 7.0         1437\n",
      "Name: _PRACE1, dtype: int64,\n",
      "  1.0       359267\n",
      " 2.0        37655\n",
      "-9999.0     11132\n",
      " 7.0        10297\n",
      " 4.0        10166\n",
      " 3.0         9823\n",
      " 6.0         9760\n",
      " 5.0         1916\n",
      "Name: _MRACE1, dtype: int64]\n"
     ]
    }
   ],
   "source": [
    "#Designated missingness value = -9999\n",
    "missingness_mapper1 = {77.: -9999, 99.: -9999}\n",
    "\n",
    "for column in to_collapse_dict_cat['map1']:\n",
    "    orig_column = imputed_df_2017[column]\n",
    "    imputed_df_2017[column] = imputed_df_2017[column].map(missingness_mapper1).fillna(orig_column)\n",
    "    \n",
    "print('The new value counts after recoding:\\n')\n",
    "pprint([imputed_df_2017[col_name].value_counts() for col_name in to_collapse_dict_cat['map1']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The new value counts after recoding:\n",
      "\n",
      "[ 2.0       365973\n",
      " 1.0        83689\n",
      "-9999.0       354\n",
      "Name: LANDLINE, dtype: int64,\n",
      "  2.0       145482\n",
      " 3.0       142502\n",
      " 1.0        74356\n",
      " 4.0        62213\n",
      " 5.0        24317\n",
      "-9999.0      1146\n",
      "Name: GENHLTH, dtype: int64,\n",
      "  1.0       412503\n",
      " 2.0        35743\n",
      "-9999.0      1770\n",
      "Name: HLTHPLN1, dtype: int64,\n",
      "  1.0       340810\n",
      " 3.0        73313\n",
      " 2.0        34083\n",
      "-9999.0      1810\n",
      "Name: PERSDOC2, dtype: int64,\n",
      "  2.0       401191\n",
      " 1.0        47528\n",
      "-9999.0      1297\n",
      "Name: MEDCOST, dtype: int64,\n",
      "  1.0       331914\n",
      " 2.0        50812\n",
      " 3.0        29656\n",
      " 4.0        27861\n",
      "-9999.0      5785\n",
      " 8.0         3988\n",
      "Name: CHECKUP1, dtype: int64,\n",
      "  1.0       417966\n",
      " 2.0        31696\n",
      "-9999.0       354\n",
      "Name: BPMEDS, dtype: int64,\n",
      "  2.0       316370\n",
      " 3.0        43534\n",
      " 1.0        26495\n",
      " 4.0        25296\n",
      "-9999.0     21749\n",
      " 5.0        16572\n",
      "Name: CHOLCHK1, dtype: int64,\n",
      "  1.0       317845\n",
      " 2.0       107897\n",
      " 3.0        21341\n",
      "-9999.0      2933\n",
      "Name: RENTHOM1, dtype: int64,\n",
      "  2.0       437847\n",
      " 1.0        11466\n",
      "-9999.0       703\n",
      "Name: NUMHHOL2, dtype: int64,\n",
      "  1.0       412838\n",
      " 2.0        36188\n",
      "-9999.0       990\n",
      "Name: CPDEMO1A, dtype: int64,\n",
      "  2.0       391419\n",
      " 1.0        57868\n",
      "-9999.0       729\n",
      "Name: VETERAN3, dtype: int64,\n",
      "  1.0       367183\n",
      " 2.0        81659\n",
      "-9999.0      1174\n",
      "Name: INTERNET, dtype: int64,\n",
      "  2.0       410716\n",
      " 1.0        37768\n",
      "-9999.0      1532\n",
      "Name: DEAF, dtype: int64,\n",
      "  2.0       426344\n",
      " 1.0        22464\n",
      "-9999.0      1208\n",
      "Name: BLIND, dtype: int64,\n",
      "  2.0       401421\n",
      " 1.0        46069\n",
      "-9999.0      2526\n",
      "Name: DECIDE, dtype: int64,\n",
      "  2.0       372001\n",
      " 1.0        76457\n",
      "-9999.0      1558\n",
      "Name: DIFFWALK, dtype: int64,\n",
      "  2.0       429237\n",
      " 1.0        20062\n",
      "-9999.0       717\n",
      "Name: DIFFDRES, dtype: int64,\n",
      "  2.0       414737\n",
      " 1.0        33949\n",
      "-9999.0      1330\n",
      "Name: DIFFALON, dtype: int64,\n",
      "  2.0       262038\n",
      " 1.0       185596\n",
      "-9999.0      2382\n",
      "Name: SMOKE100, dtype: int64,\n",
      "  3.0       386215\n",
      " 1.0        44617\n",
      " 2.0        18849\n",
      "-9999.0       335\n",
      "Name: SMOKDAY2, dtype: int64,\n",
      "  3.0       434064\n",
      " 1.0         8456\n",
      " 2.0         6684\n",
      "-9999.0       812\n",
      "Name: USENOW3, dtype: int64,\n",
      "  2.0       382318\n",
      " 1.0        66793\n",
      "-9999.0       905\n",
      "Name: ECIGARET, dtype: int64,\n",
      "  1.0       334183\n",
      " 2.0       114783\n",
      "-9999.0      1050\n",
      "Name: EXERANY2, dtype: int64,\n",
      "  1.0       398563\n",
      " 2.0        27960\n",
      " 3.0        10843\n",
      " 5.0         6453\n",
      " 4.0         4608\n",
      " 8.0          900\n",
      "-9999.0       689\n",
      "Name: SEATBELT, dtype: int64,\n",
      "  2.0       258392\n",
      " 1.0       189998\n",
      "-9999.0      1626\n",
      "Name: FLUSHOT6, dtype: int64,\n",
      "  2.0       241399\n",
      " 1.0       174476\n",
      "-9999.0     34141\n",
      "Name: PNEUVAC3, dtype: int64,\n",
      "  2.0       356948\n",
      " 1.0        87890\n",
      "-9999.0      5178\n",
      "Name: SHINGLE2, dtype: int64,\n",
      "  2.0       309928\n",
      " 1.0       126734\n",
      "-9999.0     13354\n",
      "Name: HIVTST6, dtype: int64,\n",
      "  2.0       432626\n",
      " 1.0        16030\n",
      "-9999.0      1360\n",
      "Name: HIVRISK5, dtype: int64,\n",
      "  1.0       349015\n",
      " 2.0        89609\n",
      "-9999.0     11392\n",
      "Name: PDIABTST, dtype: int64,\n",
      "  3.0       417292\n",
      " 1.0        28972\n",
      " 2.0         2609\n",
      "-9999.0      1143\n",
      "Name: PREDIAB1, dtype: int64,\n",
      "  1.0       436842\n",
      "-9999.0      5539\n",
      " 3.0         3579\n",
      " 2.0         3167\n",
      " 4.0          889\n",
      "Name: SXORIENT, dtype: int64,\n",
      "  4.0       446869\n",
      "-9999.0      2313\n",
      " 1.0          412\n",
      " 2.0          240\n",
      " 3.0          182\n",
      "Name: TRNSGNDR, dtype: int64]\n"
     ]
    }
   ],
   "source": [
    "missingness_mapper2 = {7.0: -9999, 9.0: -9999}\n",
    "\n",
    "for column in to_collapse_dict_cat['map2']:\n",
    "    orig_column = imputed_df_2017[column]\n",
    "    imputed_df_2017[column] = imputed_df_2017[column].map(missingness_mapper2).fillna(orig_column)\n",
    "    \n",
    "print('The new value counts after recoding:\\n')\n",
    "pprint([imputed_df_2017[col_name].value_counts() for col_name in to_collapse_dict_cat['map2']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The new value counts after recoding:\n",
      "\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "missingness_mapper3 = {777777.0: -9999, 999999.0: -9999}\n",
    "\n",
    "for column in to_collapse_dict_cat['map3']:\n",
    "    orig_column = imputed_df_2017[column]\n",
    "    imputed_df_2017[column] = imputed_df_2017[column].map(missingness_mapper3).fillna(orig_column)\n",
    "    \n",
    "print('The new value counts after recoding:\\n')\n",
    "pprint([imputed_df_2017[col_name].value_counts() for col_name in to_collapse_dict_cat['map3']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The new value counts after recoding:\n",
      "\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "missingness_mapper4 = {88.0: -9999, 98.0: -9999}\n",
    "\n",
    "for column in to_collapse_dict_cat['map4']:\n",
    "    orig_column = imputed_df_2017[column]\n",
    "    imputed_df_2017[column] = imputed_df_2017[column].map(missingness_mapper4).fillna(orig_column)\n",
    "    \n",
    "print('The new value counts after recoding:\\n')\n",
    "pprint([imputed_df_2017[col_name].value_counts() for col_name in to_collapse_dict_cat['map4']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The new value counts after recoding:\n",
      "\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "missingness_mapper5 = {7.0: -9999}\n",
    "\n",
    "for column in to_collapse_dict_cat['map5']:\n",
    "    orig_column = imputed_df_2017[column]\n",
    "    imputed_df_2017[column] = imputed_df_2017[column].map(missingness_mapper5).fillna(orig_column)\n",
    "    \n",
    "print('The new value counts after recoding:\\n')\n",
    "pprint([imputed_df_2017[col_name].value_counts() for col_name in to_collapse_dict_cat['map5']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The new value counts after recoding:\n",
      "\n",
      "[ 1.0       232898\n",
      " 5.0        73939\n",
      " 2.0        61437\n",
      " 3.0        54633\n",
      " 6.0        14544\n",
      " 4.0         9426\n",
      "-9999.0      3139\n",
      "Name: MARITAL, dtype: int64,\n",
      "  6.0       168396\n",
      " 5.0       124655\n",
      " 4.0       122577\n",
      " 3.0        21624\n",
      " 2.0        10434\n",
      "-9999.0      1701\n",
      " 1.0          629\n",
      "Name: EDUCA, dtype: int64,\n",
      " -9999.0    390804\n",
      " 2.0        49617\n",
      " 1.0         9595\n",
      "Name: _CHISPNC, dtype: int64,\n",
      "  3.0       302728\n",
      " 1.0        74357\n",
      " 2.0        71721\n",
      "-9999.0      1210\n",
      "Name: _LMTACT1, dtype: int64,\n",
      "  3.0       302728\n",
      " 2.0        96393\n",
      " 1.0        47809\n",
      "-9999.0      3086\n",
      "Name: _LMTWRK1, dtype: int64,\n",
      "  4.0       302728\n",
      " 3.0        78726\n",
      " 2.0        38329\n",
      " 1.0        29313\n",
      "-9999.0       920\n",
      "Name: _LMTSCL1, dtype: int64,\n",
      "  2.0       309928\n",
      " 1.0       126734\n",
      "-9999.0     13354\n",
      "Name: _AIDTST3, dtype: int64,\n",
      "  1.0       184099\n",
      " 7.0       133648\n",
      " 2.0        39592\n",
      " 8.0        32706\n",
      " 5.0        24376\n",
      " 6.0        12181\n",
      " 3.0        10489\n",
      " 4.0         9225\n",
      "-9999.0      3700\n",
      "Name: EMPLOY1, dtype: int64]\n"
     ]
    }
   ],
   "source": [
    "missingness_mapper6 = {9.0: -9999}\n",
    "\n",
    "for column in to_collapse_dict_cat['map6']:\n",
    "    orig_column = imputed_df_2017[column]\n",
    "    imputed_df_2017[column] = imputed_df_2017[column].map(missingness_mapper6).fillna(orig_column)\n",
    "    \n",
    "print('The new value counts after recoding:\\n')\n",
    "pprint([imputed_df_2017[col_name].value_counts() for col_name in to_collapse_dict_cat['map6']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 1 column(s) with >= 60% missing data to be dropped after collapsing missingness.\n",
      "['_CHISPNC']\n"
     ]
    }
   ],
   "source": [
    "#Identifying categorical features w/ high missingness (>= 60% of total sample) after collapsing missingness\n",
    "\n",
    "missingness_threshold2 = round(imputed_df_2017.shape[0]*.60)\n",
    "\n",
    "high_missingness2 = []\n",
    "\n",
    "for col_name in imputed_df_2017.columns:\n",
    "    if -9999.0 in imputed_df_2017[col_name].value_counts().keys():     \n",
    "        if imputed_df_2017[col_name].value_counts()[-9999.0] >= missingness_threshold2 and col_name not in all_possible_outcomes:\n",
    "            high_missingness2.append(col_name)\n",
    "\n",
    "print('There are', len(high_missingness2), 'column(s) with >= 60% missing data to be dropped after collapsing missingness.')\n",
    "pprint(high_missingness2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dropping columns w/ high missingness (after collapsing missingness)\n",
    "\n",
    "imputed_df_2017.drop(columns = high_missingness2, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dataframe is the expected shape after dropping high missingness columns post-missingness collapsing: True\n"
     ]
    }
   ],
   "source": [
    "print('The dataframe is the expected shape after dropping high missingness columns post-missingness collapsing:', imputed_df_2017.shape[1] == 163 - len(high_missingness2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LANDLINE Missing Count: 354\n",
      "GENHLTH Missing Count: 1146\n",
      "HLTHPLN1 Missing Count: 1770\n",
      "PERSDOC2 Missing Count: 1810\n",
      "MEDCOST Missing Count: 1297\n",
      "CHECKUP1 Missing Count: 5785\n",
      "BPMEDS Missing Count: 354\n",
      "CHOLCHK1 Missing Count: 21749\n",
      "MARITAL Missing Count: 3139\n",
      "EDUCA Missing Count: 1701\n",
      "RENTHOM1 Missing Count: 2933\n",
      "NUMHHOL2 Missing Count: 703\n",
      "CPDEMO1A Missing Count: 990\n",
      "VETERAN3 Missing Count: 729\n",
      "EMPLOY1 Missing Count: 3700\n",
      "INCOME2 Missing Count: 71754\n",
      "INTERNET Missing Count: 1174\n",
      "DEAF Missing Count: 1532\n",
      "BLIND Missing Count: 1208\n",
      "DECIDE Missing Count: 2526\n",
      "DIFFWALK Missing Count: 1558\n",
      "DIFFDRES Missing Count: 717\n",
      "DIFFALON Missing Count: 1330\n",
      "SMOKE100 Missing Count: 2382\n",
      "SMOKDAY2 Missing Count: 335\n",
      "USENOW3 Missing Count: 812\n",
      "ECIGARET Missing Count: 905\n",
      "EXERANY2 Missing Count: 1050\n",
      "EXRACT11 Missing Count: 1177\n",
      "EXRACT21 Missing Count: 3484\n",
      "SEATBELT Missing Count: 689\n",
      "FLUSHOT6 Missing Count: 1626\n",
      "PNEUVAC3 Missing Count: 34141\n",
      "SHINGLE2 Missing Count: 5178\n",
      "HIVTST6 Missing Count: 13354\n",
      "HIVRISK5 Missing Count: 1360\n",
      "PDIABTST Missing Count: 11392\n",
      "PREDIAB1 Missing Count: 1143\n",
      "SXORIENT Missing Count: 5539\n",
      "TRNSGNDR Missing Count: 2313\n",
      "_LMTACT1 Missing Count: 1210\n",
      "_LMTWRK1 Missing Count: 3086\n",
      "_LMTSCL1 Missing Count: 920\n",
      "_PRACE1 Missing Count: 11132\n",
      "_MRACE1 Missing Count: 11132\n",
      "_AIDTST3 Missing Count: 13354\n"
     ]
    }
   ],
   "source": [
    "# Checking NaN counts before imputation (round 2)\n",
    "\n",
    "nan_after_collapse = []\n",
    "\n",
    "for col_name in imputed_df_2017.columns:\n",
    "    if -9999.0 in imputed_df_2017[col_name].value_counts().keys():\n",
    "        nan_after_collapse.append(col_name)\n",
    "        print(col_name, 'Missing Count:', imputed_df_2017[col_name].value_counts()[-9999.0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Applying Simple Imputation a second time, after collapsing categorical missingness\n",
    "\n",
    "imp2 = SimpleImputer(missing_values = -9999.0, strategy=\"most_frequent\")\n",
    "imputed_df_2017_v2 = pd.DataFrame(imp2.fit_transform(imputed_df_2017), columns = imputed_df_2017.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Checking NaN counts after imputation (round 2)\n",
    "\n",
    "nan_after_collapse2 = []\n",
    "\n",
    "for col_name in imputed_df_2017_v2.columns:\n",
    "    if -9999.0 in imputed_df_2017_v2[col_name].value_counts().keys():\n",
    "        nan_after_collapse2.append(col_name)\n",
    "        print(col_name, 'Missing Count:', imputed_df_2017_v2[col_name].value_counts()[-9999.0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The continuous value counts before recoding:\n",
      "\n",
      "[2.0     349069\n",
      "1.0      78139\n",
      "3.0      16179\n",
      "4.0       5093\n",
      "5.0       1115\n",
      "6.0        297\n",
      "7.0         63\n",
      "8.0         27\n",
      "9.0         11\n",
      "10.0         9\n",
      "12.0         6\n",
      "14.0         3\n",
      "17.0         1\n",
      "15.0         1\n",
      "25.0         1\n",
      "13.0         1\n",
      "23.0         1\n",
      "Name: NUMADULT, dtype: int64,\n",
      " 1.000000e+00    374761\n",
      "5.397605e-79     59970\n",
      "2.000000e+00     13020\n",
      "3.000000e+00      1942\n",
      "4.000000e+00       261\n",
      "5.000000e+00        39\n",
      "6.000000e+00        12\n",
      "7.000000e+00         4\n",
      "1.200000e+01         2\n",
      "9.000000e+00         2\n",
      "1.000000e+01         1\n",
      "8.000000e+00         1\n",
      "1.400000e+01         1\n",
      "Name: NUMMEN, dtype: int64,\n",
      " 1.000000e+00    406229\n",
      "5.397605e-79     25613\n",
      "2.000000e+00     15721\n",
      "3.000000e+00      2094\n",
      "4.000000e+00       275\n",
      "5.000000e+00        58\n",
      "6.000000e+00        12\n",
      "8.000000e+00         5\n",
      "7.000000e+00         3\n",
      "1.100000e+01         3\n",
      "9.000000e+00         2\n",
      "2.400000e+01         1\n",
      "Name: NUMWOMEN, dtype: int64,\n",
      " 2.0     352810\n",
      "1.0      48179\n",
      "3.0      28803\n",
      "4.0      13174\n",
      "5.0       4100\n",
      "6.0       1275\n",
      "99.0       742\n",
      "7.0        407\n",
      "8.0        169\n",
      "10.0        76\n",
      "9.0         61\n",
      "77.0        58\n",
      "12.0        48\n",
      "13.0        29\n",
      "14.0        22\n",
      "11.0        21\n",
      "15.0        10\n",
      "18.0        10\n",
      "16.0         7\n",
      "21.0         3\n",
      "17.0         3\n",
      "32.0         2\n",
      "22.0         1\n",
      "30.0         1\n",
      "52.0         1\n",
      "20.0         1\n",
      "54.0         1\n",
      "40.0         1\n",
      "33.0         1\n",
      "Name: HHADULT, dtype: int64,\n",
      " 88.0    276112\n",
      "30.0     35676\n",
      "2.0      25524\n",
      "1.0      18411\n",
      "3.0      14919\n",
      "5.0      13180\n",
      "10.0      9725\n",
      "15.0      8929\n",
      "7.0       8367\n",
      "4.0       7851\n",
      "77.0      7213\n",
      "20.0      5841\n",
      "14.0      4686\n",
      "25.0      2318\n",
      "6.0       2246\n",
      "99.0      2117\n",
      "8.0       1428\n",
      "21.0      1058\n",
      "12.0      1049\n",
      "28.0       818\n",
      "29.0       433\n",
      "9.0        338\n",
      "18.0       298\n",
      "16.0       243\n",
      "27.0       232\n",
      "17.0       171\n",
      "13.0       154\n",
      "22.0       139\n",
      "11.0       138\n",
      "26.0       132\n",
      "24.0       129\n",
      "23.0        90\n",
      "19.0        51\n",
      "Name: PHYSHLTH, dtype: int64,\n",
      " 88.0    300135\n",
      "30.0     24843\n",
      "2.0      21661\n",
      "5.0      15711\n",
      "1.0      13973\n",
      "3.0      12862\n",
      "10.0     11494\n",
      "15.0     10974\n",
      "20.0      6623\n",
      "4.0       6441\n",
      "7.0       6308\n",
      "77.0      5026\n",
      "14.0      2399\n",
      "25.0      2306\n",
      "99.0      2176\n",
      "6.0       1876\n",
      "8.0       1257\n",
      "12.0       880\n",
      "28.0       663\n",
      "21.0       391\n",
      "29.0       378\n",
      "18.0       232\n",
      "9.0        207\n",
      "16.0       205\n",
      "27.0       183\n",
      "17.0       161\n",
      "13.0       144\n",
      "22.0       129\n",
      "24.0        99\n",
      "26.0        88\n",
      "11.0        88\n",
      "23.0        73\n",
      "19.0        30\n",
      "Name: MENTHLTH, dtype: int64,\n",
      " 88.0    342297\n",
      "30.0     20574\n",
      "2.0      13267\n",
      "1.0      11373\n",
      "5.0       8846\n",
      "3.0       8331\n",
      "15.0      7925\n",
      "10.0      7358\n",
      "20.0      5335\n",
      "7.0       4684\n",
      "4.0       4664\n",
      "77.0      3601\n",
      "14.0      2449\n",
      "25.0      2038\n",
      "6.0       1493\n",
      "99.0      1198\n",
      "8.0       1104\n",
      "12.0       729\n",
      "21.0       605\n",
      "28.0       452\n",
      "9.0        221\n",
      "18.0       215\n",
      "29.0       190\n",
      "16.0       172\n",
      "27.0       143\n",
      "13.0       130\n",
      "17.0       115\n",
      "24.0       110\n",
      "22.0        98\n",
      "26.0        94\n",
      "11.0        91\n",
      "23.0        78\n",
      "19.0        36\n",
      "Name: POORHLTH, dtype: int64,\n",
      " 88.0    330175\n",
      "1.0      47839\n",
      "2.0      40555\n",
      "3.0      17808\n",
      "4.0       6611\n",
      "99.0      3547\n",
      "5.0       2149\n",
      "6.0        834\n",
      "7.0        279\n",
      "8.0        122\n",
      "9.0         40\n",
      "10.0        16\n",
      "12.0        14\n",
      "11.0        10\n",
      "17.0         3\n",
      "14.0         3\n",
      "15.0         3\n",
      "16.0         3\n",
      "13.0         3\n",
      "72.0         1\n",
      "23.0         1\n",
      "Name: CHILDREN, dtype: int64,\n",
      " 88.0    391136\n",
      "1.0      20174\n",
      "2.0      11747\n",
      "3.0       5416\n",
      "4.0       4992\n",
      "5.0       3735\n",
      "77.0      2316\n",
      "30.0      1870\n",
      "10.0      1860\n",
      "6.0       1437\n",
      "8.0       1126\n",
      "15.0       903\n",
      "7.0        809\n",
      "20.0       672\n",
      "99.0       569\n",
      "12.0       402\n",
      "25.0       243\n",
      "14.0        84\n",
      "28.0        78\n",
      "9.0         73\n",
      "16.0        46\n",
      "29.0        41\n",
      "18.0        37\n",
      "13.0        29\n",
      "24.0        26\n",
      "27.0        25\n",
      "26.0        24\n",
      "21.0        23\n",
      "17.0        23\n",
      "23.0        22\n",
      "22.0        21\n",
      "11.0        18\n",
      "76.0        14\n",
      "40.0         5\n",
      "19.0         5\n",
      "50.0         4\n",
      "31.0         3\n",
      "36.0         2\n",
      "38.0         2\n",
      "32.0         2\n",
      "48.0         1\n",
      "60.0         1\n",
      "Name: DRNK3GE5, dtype: int64,\n",
      " 1.0     289847\n",
      "2.0      57985\n",
      "3.0      33926\n",
      "4.0      17910\n",
      "5.0      12989\n",
      "6.0      10737\n",
      "77.0      5797\n",
      "8.0       4509\n",
      "10.0      4253\n",
      "7.0       3695\n",
      "12.0      2934\n",
      "15.0      1080\n",
      "99.0      1043\n",
      "9.0        778\n",
      "20.0       646\n",
      "18.0       333\n",
      "30.0       265\n",
      "24.0       195\n",
      "14.0       194\n",
      "11.0       151\n",
      "16.0       141\n",
      "13.0       136\n",
      "25.0       101\n",
      "40.0        56\n",
      "17.0        50\n",
      "22.0        34\n",
      "50.0        28\n",
      "35.0        24\n",
      "23.0        20\n",
      "76.0        16\n",
      "         ...  \n",
      "32.0        13\n",
      "19.0        12\n",
      "48.0        10\n",
      "27.0         9\n",
      "45.0         9\n",
      "60.0         8\n",
      "26.0         8\n",
      "28.0         5\n",
      "38.0         5\n",
      "31.0         3\n",
      "42.0         3\n",
      "33.0         3\n",
      "70.0         3\n",
      "69.0         2\n",
      "65.0         2\n",
      "52.0         2\n",
      "37.0         2\n",
      "72.0         2\n",
      "53.0         2\n",
      "47.0         2\n",
      "39.0         2\n",
      "73.0         1\n",
      "58.0         1\n",
      "64.0         1\n",
      "75.0         1\n",
      "29.0         1\n",
      "43.0         1\n",
      "67.0         1\n",
      "46.0         1\n",
      "66.0         1\n",
      "Name: MAXDRNKS, Length: 62, dtype: int64,\n",
      " 30.0     223570\n",
      "100.0     67800\n",
      "45.0      27628\n",
      "200.0     21386\n",
      "20.0      20757\n",
      "130.0     13802\n",
      "15.0      11407\n",
      "40.0       9181\n",
      "300.0      8568\n",
      "400.0      6493\n",
      "777.0      5313\n",
      "10.0       5075\n",
      "25.0       3982\n",
      "35.0       2985\n",
      "230.0      2492\n",
      "500.0      2409\n",
      "50.0       2065\n",
      "600.0      1676\n",
      "115.0      1396\n",
      "5.0        1182\n",
      "1.0        1023\n",
      "800.0       980\n",
      "7.0         638\n",
      "120.0       599\n",
      "2.0         551\n",
      "430.0       460\n",
      "330.0       449\n",
      "999.0       447\n",
      "110.0       442\n",
      "3.0         349\n",
      "          ...  \n",
      "888.0         1\n",
      "111.0         1\n",
      "128.0         1\n",
      "256.0         1\n",
      "902.0         1\n",
      "458.0         1\n",
      "206.0         1\n",
      "955.0         1\n",
      "420.0         1\n",
      "123.0         1\n",
      "325.0         1\n",
      "610.0         1\n",
      "615.0         1\n",
      "154.0         1\n",
      "158.0         1\n",
      "51.0          1\n",
      "212.0         1\n",
      "639.0         1\n",
      "649.0         1\n",
      "234.0         1\n",
      "740.0         1\n",
      "424.0         1\n",
      "820.0         1\n",
      "701.0         1\n",
      "353.0         1\n",
      "707.0         1\n",
      "214.0         1\n",
      "355.0         1\n",
      "925.0         1\n",
      "255.0         1\n",
      "Name: EXERHMM1, Length: 221, dtype: int64,\n",
      " 100.0    297191\n",
      "30.0      42107\n",
      "200.0     18380\n",
      "20.0      14663\n",
      "45.0      13230\n",
      "15.0       9974\n",
      "130.0      7787\n",
      "300.0      7721\n",
      "400.0      5710\n",
      "10.0       5498\n",
      "40.0       3885\n",
      "777.0      3492\n",
      "25.0       2085\n",
      "500.0      2080\n",
      "230.0      2072\n",
      "600.0      1662\n",
      "5.0        1492\n",
      "35.0       1166\n",
      "800.0      1011\n",
      "50.0        853\n",
      "1.0         816\n",
      "115.0       612\n",
      "2.0         599\n",
      "7.0         554\n",
      "999.0       467\n",
      "3.0         426\n",
      "900.0       334\n",
      "330.0       321\n",
      "120.0       314\n",
      "430.0       289\n",
      "          ...  \n",
      "218.0         1\n",
      "256.0         1\n",
      "123.0         1\n",
      "955.0         1\n",
      "44.0          1\n",
      "440.0         1\n",
      "127.0         1\n",
      "715.0         1\n",
      "356.0         1\n",
      "707.0         1\n",
      "353.0         1\n",
      "453.0         1\n",
      "346.0         1\n",
      "345.0         1\n",
      "43.0          1\n",
      "333.0         1\n",
      "57.0          1\n",
      "41.0          1\n",
      "655.0         1\n",
      "920.0         1\n",
      "650.0         1\n",
      "121.0         1\n",
      "915.0         1\n",
      "113.0         1\n",
      "154.0         1\n",
      "305.0         1\n",
      "255.0         1\n",
      "603.0         1\n",
      "820.0         1\n",
      "510.0         1\n",
      "Name: EXERHMM2, Length: 181, dtype: int64,\n",
      " 102016.0    320378\n",
      "112016.0     28018\n",
      "102017.0     21801\n",
      "92016.0      18788\n",
      "777777.0     10455\n",
      "92017.0      10276\n",
      "122016.0      8625\n",
      "12017.0       7362\n",
      "112017.0      6972\n",
      "22017.0       4358\n",
      "82016.0       2473\n",
      "32017.0       2135\n",
      "82017.0       2006\n",
      "42017.0       1231\n",
      "122017.0      1189\n",
      "52017.0        808\n",
      "62017.0        682\n",
      "72017.0        512\n",
      "72016.0        376\n",
      "62016.0        358\n",
      "999999.0       327\n",
      "52016.0        209\n",
      "32016.0        177\n",
      "22016.0        176\n",
      "42016.0        154\n",
      "12016.0        104\n",
      "12018.0         60\n",
      "22018.0          6\n",
      "Name: FLSHTMY2, dtype: int64,\n",
      " 80.0    35831\n",
      "65.0    11679\n",
      "70.0    11106\n",
      "60.0    10885\n",
      "62.0    10645\n",
      "64.0    10419\n",
      "67.0    10306\n",
      "63.0    10241\n",
      "66.0    10130\n",
      "68.0     9697\n",
      "69.0     9591\n",
      "58.0     9278\n",
      "59.0     9176\n",
      "57.0     9126\n",
      "55.0     9005\n",
      "61.0     8975\n",
      "56.0     8827\n",
      "53.0     8168\n",
      "50.0     8061\n",
      "54.0     8010\n",
      "52.0     7935\n",
      "72.0     7746\n",
      "71.0     7694\n",
      "73.0     7486\n",
      "74.0     7424\n",
      "75.0     7242\n",
      "47.0     6701\n",
      "51.0     6465\n",
      "49.0     6362\n",
      "48.0     6242\n",
      "        ...  \n",
      "40.0     5870\n",
      "77.0     5797\n",
      "35.0     5495\n",
      "37.0     5437\n",
      "78.0     5407\n",
      "42.0     5379\n",
      "36.0     5367\n",
      "38.0     5241\n",
      "32.0     5179\n",
      "30.0     5135\n",
      "43.0     5044\n",
      "34.0     5028\n",
      "33.0     4973\n",
      "44.0     4919\n",
      "39.0     4911\n",
      "79.0     4880\n",
      "27.0     4680\n",
      "28.0     4571\n",
      "31.0     4520\n",
      "41.0     4437\n",
      "25.0     4422\n",
      "29.0     4414\n",
      "26.0     4310\n",
      "24.0     4043\n",
      "23.0     3991\n",
      "21.0     3828\n",
      "22.0     3796\n",
      "20.0     3550\n",
      "19.0     3521\n",
      "18.0     3507\n",
      "Name: _AGE80, Length: 63, dtype: int64,\n",
      " 66.0    57373\n",
      "64.0    39201\n",
      "65.0    35064\n",
      "67.0    34579\n",
      "68.0    32044\n",
      "70.0    31435\n",
      "63.0    31399\n",
      "62.0    30036\n",
      "69.0    28963\n",
      "72.0    28833\n",
      "71.0    24337\n",
      "61.0    14969\n",
      "60.0    14895\n",
      "73.0    13803\n",
      "74.0    11450\n",
      "75.0     6431\n",
      "59.0     4694\n",
      "76.0     3921\n",
      "77.0     1721\n",
      "58.0     1559\n",
      "57.0      861\n",
      "78.0      823\n",
      "56.0      388\n",
      "79.0      317\n",
      "80.0      234\n",
      "53.0      117\n",
      "55.0      110\n",
      "54.0       86\n",
      "48.0       70\n",
      "81.0       66\n",
      "82.0       38\n",
      "51.0       35\n",
      "50.0       27\n",
      "49.0       24\n",
      "52.0       23\n",
      "83.0       22\n",
      "84.0       20\n",
      "36.0        7\n",
      "87.0        6\n",
      "86.0        6\n",
      "89.0        5\n",
      "85.0        4\n",
      "88.0        3\n",
      "38.0        2\n",
      "41.0        2\n",
      "40.0        2\n",
      "39.0        2\n",
      "44.0        2\n",
      "45.0        1\n",
      "91.0        1\n",
      "46.0        1\n",
      "93.0        1\n",
      "47.0        1\n",
      "43.0        1\n",
      "37.0        1\n",
      "Name: HTIN4, dtype: int64,\n",
      " 9072.0     52806\n",
      "8165.0     20891\n",
      "6804.0     20307\n",
      "7257.0     19172\n",
      "7711.0     17298\n",
      "8618.0     14758\n",
      "6350.0     14654\n",
      "7938.0     12245\n",
      "7484.0     11387\n",
      "5897.0     11338\n",
      "9979.0     10554\n",
      "8391.0     10534\n",
      "9525.0      9692\n",
      "6577.0      9416\n",
      "6123.0      9101\n",
      "7031.0      8576\n",
      "10433.0     8025\n",
      "11340.0     7758\n",
      "8845.0      7331\n",
      "5443.0      7263\n",
      "5670.0      6882\n",
      "10886.0     6799\n",
      "9752.0      5268\n",
      "9299.0      4816\n",
      "10206.0     4783\n",
      "11793.0     3809\n",
      "5216.0      3682\n",
      "4990.0      3322\n",
      "10659.0     2968\n",
      "13608.0     2514\n",
      "           ...  \n",
      "11300.0        1\n",
      "15400.0        1\n",
      "11800.0        1\n",
      "17735.0        1\n",
      "18300.0        1\n",
      "10900.0        1\n",
      "19913.0        1\n",
      "22770.0        1\n",
      "26082.0        1\n",
      "23042.0        1\n",
      "12900.0        1\n",
      "24000.0        1\n",
      "25401.0        1\n",
      "2676.0         1\n",
      "15600.0        1\n",
      "21863.0        1\n",
      "13600.0        1\n",
      "18552.0        1\n",
      "15200.0        1\n",
      "18000.0        1\n",
      "2631.0         1\n",
      "24630.0        1\n",
      "12300.0        1\n",
      "22045.0        1\n",
      "2313.0         1\n",
      "11100.0        1\n",
      "2585.0         1\n",
      "24313.0        1\n",
      "24267.0        1\n",
      "20729.0        1\n",
      "Name: WTKG3, Length: 561, dtype: int64,\n",
      " 2663.0    41379\n",
      "2744.0     3648\n",
      "2746.0     3643\n",
      "2441.0     3611\n",
      "2712.0     3216\n",
      "2510.0     2893\n",
      "2575.0     2577\n",
      "2929.0     2551\n",
      "2870.0     2548\n",
      "2421.0     2471\n",
      "3228.0     2455\n",
      "2953.0     2431\n",
      "2584.0     2384\n",
      "2834.0     2321\n",
      "2509.0     2275\n",
      "2496.0     2261\n",
      "2905.0     2215\n",
      "2506.0     2206\n",
      "2582.0     2156\n",
      "2661.0     2151\n",
      "2658.0     2005\n",
      "2737.0     1980\n",
      "2403.0     1976\n",
      "3041.0     1968\n",
      "2583.0     1954\n",
      "2819.0     1953\n",
      "2657.0     1943\n",
      "2439.0     1925\n",
      "2349.0     1906\n",
      "2433.0     1904\n",
      "          ...  \n",
      "6876.0        1\n",
      "7821.0        1\n",
      "4522.0        1\n",
      "7001.0        1\n",
      "6423.0        1\n",
      "1250.0        1\n",
      "4888.0        1\n",
      "4022.0        1\n",
      "5854.0        1\n",
      "5599.0        1\n",
      "4890.0        1\n",
      "4941.0        1\n",
      "2651.0        1\n",
      "4555.0        1\n",
      "4656.0        1\n",
      "4468.0        1\n",
      "1214.0        1\n",
      "1323.0        1\n",
      "5141.0        1\n",
      "4541.0        1\n",
      "1235.0        1\n",
      "1473.0        1\n",
      "5899.0        1\n",
      "6420.0        1\n",
      "7146.0        1\n",
      "1285.0        1\n",
      "9300.0        1\n",
      "5903.0        1\n",
      "7653.0        1\n",
      "5183.0        1\n",
      "Name: _BMI5, Length: 3798, dtype: int64,\n",
      " 5.397605e-79    203183\n",
      "9.990000e+04     30066\n",
      "2.300000e+01     25114\n",
      "4.700000e+01     19291\n",
      "9.300000e+01     14046\n",
      "7.000000e+02     12739\n",
      "1.400000e+03      9232\n",
      "7.000000e+01      8987\n",
      "2.000000e+02      8858\n",
      "1.000000e+02      8419\n",
      "1.400000e+02      8137\n",
      "2.330000e+02      7755\n",
      "4.000000e+02      7699\n",
      "4.670000e+02      7400\n",
      "6.000000e+02      5798\n",
      "1.870000e+02      5733\n",
      "1.170000e+02      5356\n",
      "3.000000e+02      4910\n",
      "3.500000e+02      4797\n",
      "2.800000e+02      4480\n",
      "9.330000e+02      3673\n",
      "2.100000e+03      3479\n",
      "3.730000e+02      2342\n",
      "8.000000e+02      2329\n",
      "5.000000e+02      2217\n",
      "2.100000e+02      2155\n",
      "1.000000e+03      2054\n",
      "1.200000e+03      2034\n",
      "5.600000e+02      1896\n",
      "5.830000e+02      1831\n",
      "                 ...  \n",
      "1.470000e+04         1\n",
      "3.080000e+03         1\n",
      "7.583000e+03         1\n",
      "1.516700e+04         1\n",
      "7.933000e+03         1\n",
      "5.973000e+03         1\n",
      "8.050000e+03         1\n",
      "2.368300e+04         1\n",
      "8.100000e+03         1\n",
      "4.830000e+03         1\n",
      "6.650000e+03         1\n",
      "8.820000e+03         1\n",
      "3.966700e+04         1\n",
      "9.917000e+03         1\n",
      "5.100000e+03         1\n",
      "1.026700e+04         1\n",
      "1.440000e+04         1\n",
      "2.660000e+03         1\n",
      "4.560000e+04         1\n",
      "2.170000e+04         1\n",
      "1.092000e+04         1\n",
      "4.375000e+04         1\n",
      "5.320000e+03         1\n",
      "8.960000e+03         1\n",
      "2.777000e+03         1\n",
      "2.250000e+04         1\n",
      "2.823000e+03         1\n",
      "2.280000e+04         1\n",
      "5.717000e+03         1\n",
      "8.167000e+03         1\n",
      "Name: _DRNKWEK, Length: 278, dtype: int64,\n",
      " 5.397605e-79    206963\n",
      "1.000000e+02     62535\n",
      "1.400000e+01     26683\n",
      "2.900000e+01     24440\n",
      "7.000000e+00     19043\n",
      "3.000000e+00     18680\n",
      "4.300000e+01     17517\n",
      "2.000000e+00     10040\n",
      "1.000000e+01      8468\n",
      "1.700000e+01      7334\n",
      "2.000000e+02      7292\n",
      "5.700000e+01      7043\n",
      "1.300000e+01      5563\n",
      "3.300000e+01      4909\n",
      "7.100000e+01      4572\n",
      "5.000000e+01      4245\n",
      "6.700000e+01      2354\n",
      "3.000000e+02      2345\n",
      "2.000000e+01      1926\n",
      "2.300000e+01      1413\n",
      "2.700000e+01       934\n",
      "8.600000e+01       818\n",
      "4.000000e+02       693\n",
      "8.300000e+01       683\n",
      "4.000000e+01       618\n",
      "5.000000e+02       492\n",
      "4.700000e+01       380\n",
      "7.000000e+02       378\n",
      "1.430000e+02       227\n",
      "3.000000e+03       113\n",
      "                 ...  \n",
      "1.830000e+02         3\n",
      "2.930000e+02         3\n",
      "1.370000e+02         2\n",
      "2.430000e+02         2\n",
      "1.070000e+02         2\n",
      "4.570000e+02         2\n",
      "2.500000e+02         2\n",
      "7.140000e+02         2\n",
      "2.290000e+02         1\n",
      "4.430000e+02         1\n",
      "5.140000e+02         1\n",
      "1.300000e+02         1\n",
      "1.770000e+02         1\n",
      "1.600000e+02         1\n",
      "1.414000e+03         1\n",
      "2.670000e+02         1\n",
      "1.071000e+03         1\n",
      "2.400000e+03         1\n",
      "4.710000e+02         1\n",
      "3.860000e+02         1\n",
      "9.000000e+02         1\n",
      "2.200000e+03         1\n",
      "1.630000e+02         1\n",
      "2.600000e+03         1\n",
      "2.800000e+03         1\n",
      "2.830000e+02         1\n",
      "1.400000e+02         1\n",
      "2.330000e+02         1\n",
      "1.600000e+03         1\n",
      "2.530000e+02         1\n",
      "Name: FTJUDA2_, Length: 103, dtype: int64,\n",
      " 1.000000e+02    159969\n",
      "2.000000e+02     71634\n",
      "4.300000e+01     33967\n",
      "3.000000e+02     27693\n",
      "2.900000e+01     25478\n",
      "5.700000e+01     19689\n",
      "7.100000e+01     15622\n",
      "1.400000e+01     12780\n",
      "5.397605e-79     12753\n",
      "7.000000e+00      7693\n",
      "1.700000e+01      6325\n",
      "1.000000e+01      6076\n",
      "3.300000e+01      5820\n",
      "4.000000e+02      5350\n",
      "5.000000e+01      5068\n",
      "3.000000e+00      4702\n",
      "1.300000e+01      4487\n",
      "6.700000e+01      4269\n",
      "8.600000e+01      2739\n",
      "5.000000e+02      2603\n",
      "2.000000e+01      2045\n",
      "2.000000e+00      1940\n",
      "8.300000e+01      1506\n",
      "1.430000e+02      1342\n",
      "2.300000e+01      1226\n",
      "2.700000e+01      1011\n",
      "4.000000e+01       913\n",
      "7.000000e+02       788\n",
      "6.000000e+02       567\n",
      "1.140000e+02       417\n",
      "                 ...  \n",
      "3.070000e+02         2\n",
      "4.000000e+03         2\n",
      "8.570000e+02         2\n",
      "6.430000e+02         2\n",
      "3.170000e+02         2\n",
      "1.230000e+02         2\n",
      "1.300000e+02         2\n",
      "3.500000e+03         1\n",
      "2.230000e+02         1\n",
      "1.600000e+02         1\n",
      "1.130000e+02         1\n",
      "3.600000e+03         1\n",
      "2.130000e+02         1\n",
      "1.071000e+03         1\n",
      "5.140000e+02         1\n",
      "1.343000e+03         1\n",
      "2.600000e+03         1\n",
      "8.800000e+03         1\n",
      "2.070000e+02         1\n",
      "3.300000e+03         1\n",
      "1.300000e+03         1\n",
      "2.030000e+02         1\n",
      "2.300000e+03         1\n",
      "7.570000e+02         1\n",
      "1.870000e+02         1\n",
      "2.900000e+03         1\n",
      "4.200000e+03         1\n",
      "2.800000e+02         1\n",
      "1.730000e+02         1\n",
      "8.000000e+03         1\n",
      "Name: FRUTDA2_, Length: 128, dtype: int64,\n",
      " 1.000000e+02    118025\n",
      "4.300000e+01     50840\n",
      "2.900000e+01     46561\n",
      "1.400000e+01     36775\n",
      "5.700000e+01     29981\n",
      "5.397605e-79     29683\n",
      "7.100000e+01     20371\n",
      "2.000000e+02     16107\n",
      "7.000000e+00     13412\n",
      "5.000000e+01     10197\n",
      "3.000000e+00     10128\n",
      "6.700000e+01      8436\n",
      "3.300000e+01      8175\n",
      "1.000000e+01      8146\n",
      "1.700000e+01      7860\n",
      "1.300000e+01      6753\n",
      "8.600000e+01      3463\n",
      "8.300000e+01      3419\n",
      "3.000000e+02      3328\n",
      "2.000000e+00      3231\n",
      "2.000000e+01      3033\n",
      "2.300000e+01      1865\n",
      "4.000000e+01      1858\n",
      "2.700000e+01      1821\n",
      "4.000000e+02       860\n",
      "5.000000e+02       759\n",
      "7.000000e+02       545\n",
      "4.700000e+01       490\n",
      "9.300000e+01       415\n",
      "1.430000e+02       363\n",
      "                 ...  \n",
      "1.270000e+02         2\n",
      "1.530000e+02         2\n",
      "2.700000e+03         2\n",
      "7.140000e+02         2\n",
      "3.100000e+03         2\n",
      "3.710000e+02         2\n",
      "1.414000e+03         2\n",
      "1.830000e+02         2\n",
      "6.000000e+03         1\n",
      "1.400000e+02         1\n",
      "4.000000e+03         1\n",
      "1.230000e+02         1\n",
      "4.860000e+02         1\n",
      "2.170000e+02         1\n",
      "3.430000e+02         1\n",
      "1.600000e+03         1\n",
      "5.290000e+02         1\n",
      "7.860000e+02         1\n",
      "2.300000e+03         1\n",
      "3.860000e+02         1\n",
      "1.870000e+02         1\n",
      "1.300000e+03         1\n",
      "5.300000e+03         1\n",
      "3.130000e+02         1\n",
      "5.500000e+03         1\n",
      "6.860000e+02         1\n",
      "9.000000e+03         1\n",
      "9.100000e+03         1\n",
      "2.830000e+02         1\n",
      "1.370000e+02         1\n",
      "Name: GRENDA1_, Length: 119, dtype: int64,\n",
      " 5.397605e-79    105454\n",
      "1.400000e+01     71439\n",
      "7.000000e+00     46921\n",
      "2.900000e+01     43573\n",
      "3.000000e+00     41371\n",
      "4.300000e+01     25128\n",
      "1.000000e+01     21020\n",
      "1.700000e+01     14220\n",
      "1.300000e+01     13329\n",
      "1.000000e+02     11896\n",
      "3.300000e+01      9709\n",
      "5.700000e+01      9216\n",
      "2.000000e+00      7903\n",
      "5.000000e+01      5803\n",
      "7.100000e+01      4377\n",
      "2.000000e+01      3963\n",
      "6.700000e+01      3474\n",
      "2.300000e+01      2314\n",
      "2.700000e+01      2188\n",
      "2.000000e+02      1454\n",
      "4.000000e+01      1262\n",
      "8.600000e+01       676\n",
      "8.300000e+01       676\n",
      "3.000000e+02       551\n",
      "4.700000e+01       275\n",
      "4.000000e+02       189\n",
      "5.000000e+02       183\n",
      "3.000000e+01       177\n",
      "5.300000e+01       167\n",
      "1.430000e+02       135\n",
      "                 ...  \n",
      "8.000000e+02         4\n",
      "1.100000e+02         4\n",
      "1.860000e+02         4\n",
      "7.500000e+03         4\n",
      "4.710000e+02         3\n",
      "9.900000e+03         3\n",
      "7.140000e+02         2\n",
      "1.400000e+03         2\n",
      "2.330000e+02         2\n",
      "2.930000e+02         2\n",
      "1.830000e+02         2\n",
      "1.200000e+02         2\n",
      "1.057000e+03         1\n",
      "1.370000e+02         1\n",
      "2.400000e+02         1\n",
      "5.000000e+03         1\n",
      "1.600000e+02         1\n",
      "1.200000e+03         1\n",
      "2.830000e+02         1\n",
      "8.000000e+03         1\n",
      "3.030000e+02         1\n",
      "3.070000e+02         1\n",
      "6.430000e+02         1\n",
      "6.000000e+03         1\n",
      "5.860000e+02         1\n",
      "1.700000e+02         1\n",
      "1.930000e+02         1\n",
      "1.730000e+02         1\n",
      "9.000000e+02         1\n",
      "1.800000e+02         1\n",
      "Name: FRNCHDA_, Length: 96, dtype: int64,\n",
      " 1.400000e+01    105729\n",
      "2.900000e+01     54383\n",
      "5.397605e-79     49768\n",
      "7.000000e+00     38457\n",
      "4.300000e+01     32880\n",
      "3.000000e+00     29792\n",
      "1.000000e+01     21347\n",
      "1.700000e+01     17622\n",
      "1.300000e+01     15909\n",
      "1.000000e+02     15648\n",
      "5.700000e+01     12923\n",
      "3.300000e+01     12308\n",
      "5.000000e+01      7046\n",
      "7.100000e+01      5987\n",
      "2.000000e+00      5717\n",
      "2.000000e+01      5554\n",
      "6.700000e+01      4643\n",
      "2.700000e+01      3073\n",
      "2.300000e+01      2882\n",
      "4.000000e+01      1676\n",
      "2.000000e+02      1405\n",
      "8.600000e+01      1104\n",
      "8.300000e+01      1036\n",
      "3.000000e+02       552\n",
      "4.700000e+01       383\n",
      "3.000000e+01       278\n",
      "4.000000e+02       229\n",
      "5.300000e+01       185\n",
      "6.000000e+01       183\n",
      "5.000000e+02       166\n",
      "                 ...  \n",
      "8.000000e+02         6\n",
      "7.500000e+03         5\n",
      "1.570000e+02         5\n",
      "1.100000e+02         5\n",
      "3.570000e+02         5\n",
      "1.070000e+02         5\n",
      "4.570000e+02         4\n",
      "3.140000e+02         3\n",
      "9.900000e+03         3\n",
      "3.290000e+02         3\n",
      "4.710000e+02         3\n",
      "2.930000e+02         2\n",
      "1.830000e+02         2\n",
      "1.200000e+03         2\n",
      "1.370000e+02         2\n",
      "2.670000e+02         1\n",
      "2.470000e+02         1\n",
      "2.570000e+02         1\n",
      "2.290000e+02         1\n",
      "1.800000e+03         1\n",
      "7.000000e+03         1\n",
      "7.430000e+02         1\n",
      "1.071000e+03         1\n",
      "2.100000e+02         1\n",
      "1.400000e+03         1\n",
      "1.100000e+03         1\n",
      "1.700000e+02         1\n",
      "5.300000e+03         1\n",
      "1.300000e+03         1\n",
      "1.270000e+02         1\n",
      "Name: POTADA1_, Length: 92, dtype: int64,\n",
      " 1.000000e+02    95231\n",
      "2.000000e+02    53759\n",
      "3.000000e+02    27584\n",
      "4.300000e+01    16641\n",
      "1.140000e+02    12730\n",
      "5.700000e+01    11509\n",
      "4.000000e+02    10362\n",
      "2.900000e+01    10347\n",
      "1.290000e+02    10331\n",
      "1.430000e+02     9695\n",
      "5.397605e-79     8372\n",
      "7.100000e+01     7716\n",
      "1.400000e+01     6531\n",
      "8.600000e+01     6100\n",
      "1.070000e+02     5562\n",
      "1.030000e+02     5190\n",
      "1.700000e+01     4702\n",
      "5.000000e+01     4430\n",
      "2.140000e+02     4377\n",
      "7.200000e+01     4357\n",
      "2.290000e+02     4213\n",
      "7.000000e+00     4122\n",
      "1.570000e+02     4120\n",
      "1.000000e+01     3958\n",
      "5.000000e+02     3714\n",
      "1.020000e+02     3329\n",
      "2.430000e+02     3247\n",
      "1.170000e+02     2886\n",
      "1.100000e+02     2878\n",
      "3.000000e+00     2871\n",
      "                ...  \n",
      "4.320000e+02        1\n",
      "5.850000e+02        1\n",
      "1.550000e+03        1\n",
      "5.830000e+02        1\n",
      "3.890000e+02        1\n",
      "2.300000e+03        1\n",
      "7.567000e+03        1\n",
      "8.400000e+02        1\n",
      "8.470000e+02        1\n",
      "1.229000e+03        1\n",
      "1.980000e+02        1\n",
      "2.529000e+03        1\n",
      "7.503000e+03        1\n",
      "4.690000e+02        1\n",
      "1.010000e+04        1\n",
      "2.540000e+02        1\n",
      "2.523000e+03        1\n",
      "5.033000e+03        1\n",
      "1.529000e+03        1\n",
      "7.507000e+03        1\n",
      "2.507000e+03        1\n",
      "7.513000e+03        1\n",
      "2.502000e+03        1\n",
      "4.700000e+02        1\n",
      "1.243000e+03        1\n",
      "5.800000e+02        1\n",
      "9.913000e+03        1\n",
      "9.903000e+03        1\n",
      "3.090000e+02        1\n",
      "1.940000e+02        1\n",
      "Name: _FRUTSU1, Length: 594, dtype: int64,\n",
      " 200.0      57325\n",
      "157.0       8782\n",
      "214.0       7161\n",
      "143.0       7074\n",
      "243.0       6822\n",
      "186.0       6397\n",
      "171.0       5742\n",
      "100.0       5611\n",
      "228.0       5471\n",
      "172.0       5469\n",
      "114.0       5029\n",
      "129.0       4698\n",
      "257.0       4661\n",
      "300.0       4616\n",
      "229.0       4382\n",
      "128.0       4165\n",
      "86.0        4040\n",
      "217.0       3771\n",
      "150.0       3702\n",
      "142.0       3473\n",
      "314.0       3402\n",
      "185.0       3063\n",
      "115.0       3016\n",
      "158.0       2849\n",
      "117.0       2842\n",
      "207.0       2790\n",
      "160.0       2770\n",
      "203.0       2739\n",
      "400.0       2739\n",
      "343.0       2726\n",
      "           ...  \n",
      "9917.0         1\n",
      "3164.0         1\n",
      "19843.0        1\n",
      "19850.0        1\n",
      "9928.0         1\n",
      "3271.0         1\n",
      "9933.0         1\n",
      "826.0          1\n",
      "9940.0         1\n",
      "19881.0        1\n",
      "9942.0         1\n",
      "1633.0         1\n",
      "6527.0         1\n",
      "9949.0         1\n",
      "9950.0         1\n",
      "1631.0         1\n",
      "1420.0         1\n",
      "19907.0        1\n",
      "9958.0         1\n",
      "9960.0         1\n",
      "9966.0         1\n",
      "9967.0         1\n",
      "1104.0         1\n",
      "9971.0         1\n",
      "2493.0         1\n",
      "9973.0         1\n",
      "6502.0         1\n",
      "1328.0         1\n",
      "3736.0         1\n",
      "1513.0         1\n",
      "Name: _VEGESU1, Length: 1578, dtype: int64,\n",
      " 3.500000e+01    316179\n",
      "6.000000e+01     26622\n",
      "5.000000e+01     21304\n",
      "5.397605e-79     17538\n",
      "4.500000e+01     15979\n",
      "6.800000e+01     14364\n",
      "7.300000e+01      7423\n",
      "3.800000e+01      5227\n",
      "7.000000e+01      4723\n",
      "4.000000e+01      4646\n",
      "6.500000e+01      1971\n",
      "3.300000e+01      1823\n",
      "7.800000e+01      1817\n",
      "4.300000e+01      1779\n",
      "5.500000e+01      1730\n",
      "5.800000e+01      1659\n",
      "4.800000e+01      1208\n",
      "3.000000e+01       917\n",
      "9.000000e+01       807\n",
      "8.000000e+01       588\n",
      "5.300000e+01       544\n",
      "1.030000e+02       508\n",
      "1.280000e+02       380\n",
      "1.250000e+02       120\n",
      "2.500000e+01        46\n",
      "1.100000e+02        34\n",
      "1.200000e+02        34\n",
      "6.300000e+01        33\n",
      "9.800000e+01        13\n",
      "Name: METVL11_, dtype: int64,\n",
      " 5.397605e-79    279588\n",
      "3.500000e+01     46807\n",
      "4.500000e+01     18746\n",
      "6.000000e+01     17910\n",
      "5.000000e+01     17294\n",
      "6.800000e+01     13511\n",
      "3.300000e+01     12462\n",
      "4.000000e+01      9786\n",
      "3.800000e+01      6024\n",
      "7.300000e+01      4924\n",
      "7.000000e+01      3916\n",
      "5.500000e+01      3130\n",
      "9.000000e+01      2509\n",
      "7.800000e+01      2217\n",
      "6.500000e+01      2008\n",
      "5.800000e+01      1867\n",
      "4.800000e+01      1793\n",
      "4.300000e+01      1738\n",
      "5.300000e+01      1185\n",
      "3.000000e+01      1143\n",
      "8.000000e+01       446\n",
      "1.280000e+02       313\n",
      "1.030000e+02       269\n",
      "1.250000e+02       193\n",
      "2.500000e+01       100\n",
      "1.100000e+02        73\n",
      "6.300000e+01        22\n",
      "9.800000e+01        21\n",
      "1.200000e+02        21\n",
      "Name: METVL21_, dtype: int64,\n",
      " 2210.0     6333\n",
      "99900.0    6298\n",
      "2395.0     6189\n",
      "2580.0     6033\n",
      "2506.0     5775\n",
      "2321.0     5609\n",
      "2469.0     5562\n",
      "2358.0     5526\n",
      "2432.0     5511\n",
      "2247.0     5500\n",
      "2284.0     5474\n",
      "2654.0     5209\n",
      "2617.0     5005\n",
      "2543.0     4962\n",
      "2425.0     4952\n",
      "2691.0     4942\n",
      "2765.0     4935\n",
      "3690.0     4912\n",
      "2728.0     4853\n",
      "2150.0     4740\n",
      "2700.0     4733\n",
      "2590.0     4645\n",
      "2136.0     4474\n",
      "2099.0     4467\n",
      "2173.0     4454\n",
      "2062.0     4450\n",
      "2480.0     4423\n",
      "2315.0     4419\n",
      "2839.0     4414\n",
      "2535.0     4396\n",
      "           ... \n",
      "1581.0     1387\n",
      "1544.0     1266\n",
      "1435.0     1225\n",
      "1380.0     1115\n",
      "1325.0     1046\n",
      "1507.0      966\n",
      "1470.0      909\n",
      "1270.0      879\n",
      "1215.0      709\n",
      "1160.0      640\n",
      "1433.0      605\n",
      "1105.0      528\n",
      "1396.0      517\n",
      "1050.0      414\n",
      "1359.0      359\n",
      "995.0       277\n",
      "1322.0      263\n",
      "940.0       257\n",
      "1137.0      205\n",
      "885.0       169\n",
      "1285.0      158\n",
      "1248.0      128\n",
      "555.0       128\n",
      "830.0       105\n",
      "1211.0       70\n",
      "775.0        66\n",
      "720.0        52\n",
      "1174.0       40\n",
      "665.0        26\n",
      "610.0        19\n",
      "Name: MAXVO2_, Length: 163, dtype: int64,\n",
      " 379.0      6333\n",
      "99900.0    6298\n",
      "411.0      6189\n",
      "442.0      6033\n",
      "430.0      5775\n",
      "398.0      5609\n",
      "423.0      5562\n",
      "404.0      5526\n",
      "417.0      5511\n",
      "385.0      5500\n",
      "392.0      5474\n",
      "595.0      5473\n",
      "455.0      5209\n",
      "449.0      5005\n",
      "436.0      4962\n",
      "416.0      4952\n",
      "461.0      4942\n",
      "474.0      4935\n",
      "633.0      4912\n",
      "303.0      4867\n",
      "468.0      4853\n",
      "614.0      4799\n",
      "369.0      4740\n",
      "463.0      4733\n",
      "444.0      4645\n",
      "366.0      4474\n",
      "360.0      4467\n",
      "373.0      4454\n",
      "353.0      4450\n",
      "425.0      4423\n",
      "           ... \n",
      "277.0      1661\n",
      "702.0      1614\n",
      "696.0      1583\n",
      "709.0      1512\n",
      "255.0      1416\n",
      "271.0      1387\n",
      "227.0      1309\n",
      "237.0      1115\n",
      "258.0       966\n",
      "252.0       909\n",
      "218.0       879\n",
      "208.0       779\n",
      "199.0       640\n",
      "189.0       528\n",
      "239.0       517\n",
      "180.0       414\n",
      "233.0       359\n",
      "171.0       277\n",
      "161.0       257\n",
      "195.0       205\n",
      "152.0       169\n",
      "220.0       158\n",
      "214.0       128\n",
      "95.0        128\n",
      "142.0       105\n",
      "133.0        66\n",
      "123.0        52\n",
      "201.0        40\n",
      "114.0        26\n",
      "105.0        19\n",
      "Name: FC60_, Length: 156, dtype: int64,\n",
      " 30.0     229330\n",
      "60.0      67800\n",
      "45.0      27628\n",
      "120.0     21386\n",
      "20.0      20757\n",
      "90.0      13802\n",
      "15.0      11407\n",
      "40.0       9181\n",
      "180.0      8568\n",
      "240.0      6493\n",
      "10.0       5075\n",
      "25.0       3982\n",
      "35.0       2985\n",
      "150.0      2492\n",
      "300.0      2409\n",
      "50.0       2065\n",
      "360.0      1676\n",
      "75.0       1396\n",
      "5.0        1182\n",
      "1.0        1023\n",
      "480.0       980\n",
      "7.0         638\n",
      "80.0        599\n",
      "2.0         551\n",
      "270.0       460\n",
      "210.0       449\n",
      "70.0        442\n",
      "3.0         349\n",
      "420.0       339\n",
      "55.0        306\n",
      "          ...  \n",
      "485.0         1\n",
      "51.0          1\n",
      "495.0         1\n",
      "145.0         1\n",
      "253.0         1\n",
      "598.0         1\n",
      "97.0          1\n",
      "308.0         1\n",
      "345.0         1\n",
      "155.0         1\n",
      "315.0         1\n",
      "280.0         1\n",
      "235.0         1\n",
      "325.0         1\n",
      "83.0          1\n",
      "136.0         1\n",
      "500.0         1\n",
      "118.0         1\n",
      "173.0         1\n",
      "84.0          1\n",
      "175.0         1\n",
      "312.0         1\n",
      "460.0         1\n",
      "361.0         1\n",
      "362.0         1\n",
      "363.0         1\n",
      "370.0         1\n",
      "375.0         1\n",
      "98.0          1\n",
      "126.0         1\n",
      "Name: PADUR1_, Length: 219, dtype: int64,\n",
      " 60.0     301150\n",
      "30.0      42107\n",
      "120.0     18380\n",
      "20.0      14663\n",
      "45.0      13230\n",
      "15.0       9974\n",
      "90.0       7787\n",
      "180.0      7721\n",
      "240.0      5710\n",
      "10.0       5498\n",
      "40.0       3885\n",
      "25.0       2085\n",
      "300.0      2080\n",
      "150.0      2072\n",
      "360.0      1662\n",
      "5.0        1492\n",
      "35.0       1166\n",
      "480.0      1011\n",
      "50.0        853\n",
      "1.0         816\n",
      "75.0        612\n",
      "2.0         599\n",
      "7.0         554\n",
      "3.0         426\n",
      "540.0       334\n",
      "210.0       321\n",
      "80.0        314\n",
      "270.0       289\n",
      "420.0       278\n",
      "4.0         272\n",
      "          ...  \n",
      "43.0          1\n",
      "173.0         1\n",
      "87.0          1\n",
      "410.0         1\n",
      "41.0          1\n",
      "175.0         1\n",
      "99.0          1\n",
      "225.0         1\n",
      "195.0         1\n",
      "98.0          1\n",
      "93.0          1\n",
      "83.0          1\n",
      "81.0          1\n",
      "285.0         1\n",
      "226.0         1\n",
      "595.0         1\n",
      "363.0         1\n",
      "176.0         1\n",
      "301.0         1\n",
      "302.0         1\n",
      "124.0         1\n",
      "435.0         1\n",
      "57.0          1\n",
      "190.0         1\n",
      "310.0         1\n",
      "427.0         1\n",
      "213.0         1\n",
      "280.0         1\n",
      "115.0         1\n",
      "138.0         1\n",
      "Name: PADUR2_, Length: 179, dtype: int64,\n",
      " 7000.0     208126\n",
      "3000.0      51529\n",
      "2000.0      34160\n",
      "5000.0      29269\n",
      "4000.0      27331\n",
      "1000.0      16839\n",
      "6000.0      10275\n",
      "4667.0       8781\n",
      "3500.0       6405\n",
      "933.0        5829\n",
      "700.0        5545\n",
      "2333.0       5445\n",
      "467.0        5404\n",
      "1167.0       4875\n",
      "5833.0       4185\n",
      "2800.0       3305\n",
      "1867.0       3282\n",
      "1400.0       2984\n",
      "233.0        2592\n",
      "14000.0      2386\n",
      "1633.0       1672\n",
      "6533.0        966\n",
      "3733.0        906\n",
      "3267.0        754\n",
      "10000.0       700\n",
      "21000.0       580\n",
      "5600.0        449\n",
      "2100.0        419\n",
      "11667.0       359\n",
      "4200.0        341\n",
      "            ...  \n",
      "26000.0         3\n",
      "19000.0         2\n",
      "84000.0         2\n",
      "73000.0         2\n",
      "34000.0         2\n",
      "29000.0         1\n",
      "63000.0         1\n",
      "15400.0         1\n",
      "74000.0         1\n",
      "23000.0         1\n",
      "52000.0         1\n",
      "20300.0         1\n",
      "32000.0         1\n",
      "13533.0         1\n",
      "80000.0         1\n",
      "38000.0         1\n",
      "48000.0         1\n",
      "9100.0          1\n",
      "9567.0          1\n",
      "88000.0         1\n",
      "43000.0         1\n",
      "36000.0         1\n",
      "72000.0         1\n",
      "10733.0         1\n",
      "20533.0         1\n",
      "14233.0         1\n",
      "11900.0         1\n",
      "9800.0          1\n",
      "56000.0         1\n",
      "47000.0         1\n",
      "Name: PAFREQ1_, Length: 120, dtype: int64,\n",
      " 2000.0     290240\n",
      "3000.0      28557\n",
      "1000.0      25335\n",
      "7000.0      23822\n",
      "4000.0      10691\n",
      "5000.0      10483\n",
      "467.0        8389\n",
      "933.0        7471\n",
      "700.0        6228\n",
      "1167.0       4763\n",
      "2333.0       4353\n",
      "3500.0       3867\n",
      "233.0        3719\n",
      "4667.0       3631\n",
      "6000.0       3111\n",
      "1867.0       2960\n",
      "1400.0       2910\n",
      "2800.0       1948\n",
      "5833.0       1319\n",
      "1633.0       1220\n",
      "14000.0       801\n",
      "3267.0        443\n",
      "3733.0        401\n",
      "6533.0        283\n",
      "2100.0        276\n",
      "10000.0       262\n",
      "21000.0       196\n",
      "11667.0       161\n",
      "5600.0        155\n",
      "4200.0        141\n",
      "            ...  \n",
      "7467.0          4\n",
      "90000.0         3\n",
      "9800.0          3\n",
      "23000.0         3\n",
      "22000.0         2\n",
      "8867.0          2\n",
      "54000.0         2\n",
      "13067.0         2\n",
      "8633.0          2\n",
      "10733.0         2\n",
      "84000.0         2\n",
      "31000.0         1\n",
      "36000.0         1\n",
      "48000.0         1\n",
      "32000.0         1\n",
      "7700.0          1\n",
      "11200.0         1\n",
      "74000.0         1\n",
      "72000.0         1\n",
      "14700.0         1\n",
      "58000.0         1\n",
      "14233.0         1\n",
      "55000.0         1\n",
      "88000.0         1\n",
      "76000.0         1\n",
      "12600.0         1\n",
      "19600.0         1\n",
      "80000.0         1\n",
      "10267.0         1\n",
      "14467.0         1\n",
      "Name: PAFREQ2_, Length: 104, dtype: int64,\n",
      " 5.397605e-79    181114\n",
      "1.800000e+02     18903\n",
      "1.200000e+02     17579\n",
      "9.000000e+01     16522\n",
      "2.100000e+02     16188\n",
      "6.000000e+01     15945\n",
      "4.200000e+02     12567\n",
      "2.400000e+02      9757\n",
      "1.400000e+02      8670\n",
      "1.500000e+02      8120\n",
      "3.000000e+02      7230\n",
      "3.600000e+02      7106\n",
      "1.050000e+02      5558\n",
      "3.000000e+01      5435\n",
      "1.350000e+02      5392\n",
      "7.000000e+01      4863\n",
      "8.400000e+02      4608\n",
      "3.150000e+02      4466\n",
      "2.800000e+02      4134\n",
      "2.250000e+02      3409\n",
      "2.700000e+02      3115\n",
      "4.000000e+01      2875\n",
      "2.800000e+01      2811\n",
      "5.600000e+01      2712\n",
      "8.400000e+01      2655\n",
      "4.800000e+02      2539\n",
      "1.000000e+02      2502\n",
      "6.300000e+02      2493\n",
      "4.200000e+01      2467\n",
      "4.500000e+01      2463\n",
      "                 ...  \n",
      "2.516000e+03         1\n",
      "1.010000e+02         1\n",
      "1.610000e+03         1\n",
      "4.020000e+02         1\n",
      "1.564000e+03         1\n",
      "6.120000e+02         1\n",
      "4.050000e+04         1\n",
      "3.170000e+02         1\n",
      "6.375000e+03         1\n",
      "2.541000e+03         1\n",
      "6.370000e+02         1\n",
      "1.830000e+03         1\n",
      "3.266000e+03         1\n",
      "8.170000e+02         1\n",
      "1.330000e+03         1\n",
      "2.576000e+03         1\n",
      "1.008000e+04         1\n",
      "8.210000e+02         1\n",
      "2.491000e+03         1\n",
      "7.180000e+02         1\n",
      "5.800000e+02         1\n",
      "1.108000e+03         1\n",
      "4.170000e+02         1\n",
      "6.960000e+03         1\n",
      "4.300000e+02         1\n",
      "6.150000e+02         1\n",
      "5.900000e+01         1\n",
      "1.225000e+03         1\n",
      "8.370000e+02         1\n",
      "2.989000e+03         1\n",
      "Name: _MINAC11, Length: 814, dtype: int64,\n",
      " 5.397605e-79    289533\n",
      "6.000000e+01     14813\n",
      "1.200000e+02     12242\n",
      "9.000000e+01      9332\n",
      "1.800000e+02      8327\n",
      "3.000000e+01      6250\n",
      "2.100000e+02      6003\n",
      "2.400000e+02      5248\n",
      "4.200000e+02      5156\n",
      "1.400000e+02      4241\n",
      "7.000000e+01      3829\n",
      "3.600000e+02      3742\n",
      "2.800000e+01      3640\n",
      "5.600000e+01      3584\n",
      "8.400000e+01      2958\n",
      "1.050000e+02      2897\n",
      "1.500000e+02      2823\n",
      "4.200000e+01      2753\n",
      "4.500000e+01      2684\n",
      "8.400000e+02      2651\n",
      "4.000000e+01      2602\n",
      "1.400000e+01      2555\n",
      "1.120000e+02      2234\n",
      "3.000000e+02      2154\n",
      "1.350000e+02      1803\n",
      "1.680000e+02      1773\n",
      "2.000000e+01      1746\n",
      "2.800000e+02      1689\n",
      "2.100000e+01      1683\n",
      "4.800000e+02      1618\n",
      "                 ...  \n",
      "1.056000e+03         1\n",
      "6.700000e+01         1\n",
      "6.500000e+02         1\n",
      "4.600000e+02         1\n",
      "1.428000e+03         1\n",
      "2.835000e+03         1\n",
      "2.257500e+04         1\n",
      "4.492500e+04         1\n",
      "8.200000e+02         1\n",
      "1.397000e+03         1\n",
      "4.455000e+04         1\n",
      "3.720000e+03         1\n",
      "1.867000e+03         1\n",
      "1.108800e+04         1\n",
      "6.440000e+02         1\n",
      "4.690000e+02         1\n",
      "3.773000e+03         1\n",
      "2.737000e+03         1\n",
      "3.420000e+02         1\n",
      "3.180000e+02         1\n",
      "6.830000e+02         1\n",
      "3.300000e+03         1\n",
      "1.360000e+03         1\n",
      "3.390000e+02         1\n",
      "3.380000e+02         1\n",
      "1.596000e+03         1\n",
      "2.695000e+03         1\n",
      "1.710000e+03         1\n",
      "6.700000e+02         1\n",
      "4.095000e+03         1\n",
      "Name: _MINAC21, Length: 663, dtype: int64,\n",
      " 5.397605e-79    291189\n",
      "3.000000e+03     28915\n",
      "2.000000e+03     25604\n",
      "7.000000e+03     16790\n",
      "1.000000e+03     15712\n",
      "5.000000e+03     11415\n",
      "4.000000e+03     11214\n",
      "4.670000e+02      7134\n",
      "9.330000e+02      4392\n",
      "2.330000e+02      4318\n",
      "7.000000e+02      4246\n",
      "1.167000e+03      4105\n",
      "2.333000e+03      3668\n",
      "6.000000e+03      3518\n",
      "3.500000e+03      3441\n",
      "4.667000e+03      3108\n",
      "1.867000e+03      2178\n",
      "2.800000e+03      1917\n",
      "1.400000e+03      1551\n",
      "5.833000e+03      1077\n",
      "1.633000e+03       770\n",
      "1.400000e+04       549\n",
      "3.733000e+03       467\n",
      "3.267000e+03       353\n",
      "6.533000e+03       226\n",
      "2.100000e+03       207\n",
      "5.600000e+03       199\n",
      "1.000000e+04       186\n",
      "4.200000e+03       150\n",
      "6.067000e+03       107\n",
      "                 ...  \n",
      "8.800000e+04         3\n",
      "3.700000e+04         2\n",
      "1.866700e+04         2\n",
      "1.120000e+04         2\n",
      "1.446700e+04         2\n",
      "7.933000e+03         2\n",
      "7.600000e+04         2\n",
      "2.700000e+04         2\n",
      "5.600000e+04         2\n",
      "2.200000e+04         2\n",
      "2.300000e+04         2\n",
      "2.053300e+04         2\n",
      "4.700000e+04         1\n",
      "1.960000e+04         1\n",
      "1.726700e+04         1\n",
      "9.000000e+04         1\n",
      "7.700000e+03         1\n",
      "2.216700e+04         1\n",
      "3.200000e+04         1\n",
      "4.300000e+04         1\n",
      "1.423300e+04         1\n",
      "1.283300e+04         1\n",
      "2.600000e+04         1\n",
      "1.820000e+04         1\n",
      "7.300000e+04         1\n",
      "5.300000e+04         1\n",
      "1.026700e+04         1\n",
      "5.400000e+04         1\n",
      "6.400000e+04         1\n",
      "4.900000e+04         1\n",
      "Name: STRFREQ_, Length: 104, dtype: int64,\n",
      " 5.397605e-79    184322\n",
      "1.800000e+02     16375\n",
      "1.200000e+02     15681\n",
      "4.200000e+02     13578\n",
      "2.100000e+02     13293\n",
      "6.000000e+01     12678\n",
      "2.400000e+02     12316\n",
      "9.000000e+01     12314\n",
      "3.600000e+02     11281\n",
      "1.400000e+02      7391\n",
      "3.000000e+02      7249\n",
      "8.400000e+02      6537\n",
      "1.500000e+02      6162\n",
      "2.800000e+02      5475\n",
      "4.800000e+02      5060\n",
      "7.200000e+02      4001\n",
      "3.000000e+01      3995\n",
      "7.000000e+01      3653\n",
      "2.700000e+02      3610\n",
      "6.000000e+02      3548\n",
      "1.350000e+02      3533\n",
      "1.050000e+02      3467\n",
      "3.150000e+02      3222\n",
      "6.300000e+02      2927\n",
      "5.600000e+01      2588\n",
      "4.000000e+01      2548\n",
      "8.400000e+01      2515\n",
      "2.800000e+01      2499\n",
      "8.000000e+01      2383\n",
      "2.250000e+02      2355\n",
      "                 ...  \n",
      "4.120000e+03         1\n",
      "4.536000e+03         1\n",
      "1.465000e+03         1\n",
      "3.584000e+03         1\n",
      "9.550000e+02         1\n",
      "2.079000e+03         1\n",
      "2.745000e+03         1\n",
      "2.640000e+03         1\n",
      "3.550000e+02         1\n",
      "6.870000e+02         1\n",
      "3.024000e+04         1\n",
      "3.450000e+02         1\n",
      "2.410000e+02         1\n",
      "3.854000e+03         1\n",
      "7.700000e+03         1\n",
      "7.686000e+03         1\n",
      "1.220000e+03         1\n",
      "1.130000e+02         1\n",
      "1.396000e+03         1\n",
      "1.397000e+03         1\n",
      "2.030000e+03         1\n",
      "4.492500e+04         1\n",
      "7.100000e+01         1\n",
      "2.390000e+02         1\n",
      "2.190000e+03         1\n",
      "7.050000e+02         1\n",
      "1.350000e+04         1\n",
      "7.100000e+02         1\n",
      "2.842000e+03         1\n",
      "8.190000e+03         1\n",
      "Name: PAMIN11_, Length: 955, dtype: int64,\n",
      " 5.397605e-79    291356\n",
      "1.200000e+02     13082\n",
      "6.000000e+01     10654\n",
      "2.400000e+02      8614\n",
      "1.800000e+02      8510\n",
      "9.000000e+01      6327\n",
      "3.600000e+02      5899\n",
      "4.200000e+02      5609\n",
      "2.100000e+02      5009\n",
      "1.400000e+02      4209\n",
      "3.000000e+01      3743\n",
      "5.600000e+01      3619\n",
      "8.400000e+02      3528\n",
      "4.800000e+02      3224\n",
      "2.800000e+01      3114\n",
      "1.120000e+02      2935\n",
      "8.400000e+01      2866\n",
      "2.800000e+02      2782\n",
      "7.000000e+01      2650\n",
      "3.000000e+02      2429\n",
      "1.680000e+02      2402\n",
      "7.200000e+02      2379\n",
      "4.200000e+01      2192\n",
      "1.500000e+02      2176\n",
      "4.000000e+01      2170\n",
      "8.000000e+01      1933\n",
      "1.400000e+01      1581\n",
      "2.240000e+02      1515\n",
      "1.050000e+02      1475\n",
      "6.000000e+02      1390\n",
      "                 ...  \n",
      "1.288000e+03         1\n",
      "6.062000e+03         1\n",
      "1.064000e+03         1\n",
      "3.038000e+03         1\n",
      "9.700000e+02         1\n",
      "4.440000e+03         1\n",
      "7.080000e+02         1\n",
      "7.600000e+02         1\n",
      "2.905000e+03         1\n",
      "3.620000e+02         1\n",
      "1.790000e+02         1\n",
      "1.956000e+03         1\n",
      "8.985000e+04         1\n",
      "2.570000e+02         1\n",
      "2.257500e+04         1\n",
      "3.200000e+03         1\n",
      "3.860000e+02         1\n",
      "2.835000e+03         1\n",
      "7.546000e+03         1\n",
      "2.640000e+03         1\n",
      "7.230000e+02         1\n",
      "3.913000e+03         1\n",
      "2.850000e+03         1\n",
      "1.082000e+03         1\n",
      "1.952000e+03         1\n",
      "2.196000e+03         1\n",
      "3.910000e+03         1\n",
      "1.260000e+04         1\n",
      "1.056000e+03         1\n",
      "4.095000e+03         1\n",
      "Name: PAMIN21_, Length: 772, dtype: int64,\n",
      " 5.397605e-79    167091\n",
      "1.800000e+02     10420\n",
      "1.200000e+02     10111\n",
      "4.200000e+02      9912\n",
      "2.100000e+02      9468\n",
      "6.000000e+01      8532\n",
      "2.400000e+02      8287\n",
      "9.000000e+01      8101\n",
      "3.600000e+02      7698\n",
      "3.000000e+02      6612\n",
      "1.400000e+02      4913\n",
      "8.400000e+02      4899\n",
      "1.500000e+02      4864\n",
      "4.800000e+02      4552\n",
      "6.000000e+02      3693\n",
      "2.800000e+02      3645\n",
      "2.700000e+02      3638\n",
      "5.400000e+02      3204\n",
      "7.200000e+02      3144\n",
      "3.000000e+01      2977\n",
      "6.300000e+02      2723\n",
      "7.000000e+01      2590\n",
      "4.500000e+02      2477\n",
      "1.050000e+02      2397\n",
      "3.150000e+02      2240\n",
      "1.350000e+02      2221\n",
      "3.300000e+02      2087\n",
      "1.260000e+03      1879\n",
      "4.000000e+01      1853\n",
      "2.800000e+01      1795\n",
      "                 ...  \n",
      "2.969000e+03         1\n",
      "2.993000e+03         1\n",
      "3.645000e+03         1\n",
      "2.973000e+03         1\n",
      "2.974000e+03         1\n",
      "4.933000e+03         1\n",
      "2.975000e+03         1\n",
      "2.466000e+03         1\n",
      "3.272000e+03         1\n",
      "2.465000e+03         1\n",
      "1.190800e+04         1\n",
      "5.955000e+03         1\n",
      "2.462000e+03         1\n",
      "2.978000e+03         1\n",
      "7.188000e+03         1\n",
      "1.229000e+03         1\n",
      "5.973000e+03         1\n",
      "3.892000e+03         1\n",
      "4.908000e+03         1\n",
      "5.974000e+03         1\n",
      "9.810000e+03         1\n",
      "2.451000e+03         1\n",
      "7.786000e+03         1\n",
      "5.982000e+03         1\n",
      "4.240000e+03         1\n",
      "2.148000e+03         1\n",
      "3.092000e+03         1\n",
      "1.197000e+04         1\n",
      "4.893000e+03         1\n",
      "1.723000e+03         1\n",
      "Name: PA1MIN_, Length: 2995, dtype: int64,\n",
      " 5.397605e-79    359216\n",
      "1.800000e+02      7197\n",
      "1.200000e+02      6330\n",
      "9.000000e+01      4903\n",
      "2.100000e+02      4692\n",
      "6.000000e+01      4628\n",
      "2.400000e+02      3673\n",
      "4.200000e+02      3510\n",
      "3.600000e+02      2949\n",
      "1.400000e+02      2790\n",
      "1.500000e+02      2494\n",
      "3.000000e+02      2381\n",
      "1.050000e+02      2026\n",
      "1.350000e+02      1815\n",
      "7.000000e+01      1617\n",
      "3.000000e+01      1563\n",
      "8.400000e+02      1516\n",
      "2.800000e+02      1403\n",
      "2.700000e+02      1286\n",
      "3.150000e+02      1191\n",
      "4.800000e+02      1117\n",
      "5.600000e+01      1015\n",
      "2.250000e+02       996\n",
      "8.400000e+01       984\n",
      "7.200000e+02       924\n",
      "2.800000e+01       920\n",
      "4.500000e+01       882\n",
      "4.200000e+01       877\n",
      "1.680000e+02       863\n",
      "1.000000e+02       816\n",
      "                 ...  \n",
      "2.930000e+02         1\n",
      "3.750000e+04         1\n",
      "5.900000e+01         1\n",
      "4.300000e+02         1\n",
      "3.420000e+03         1\n",
      "3.800000e+01         1\n",
      "1.281000e+03         1\n",
      "8.210000e+02         1\n",
      "4.110000e+02         1\n",
      "1.275000e+03         1\n",
      "6.370000e+02         1\n",
      "9.200000e+01         1\n",
      "3.170000e+02         1\n",
      "4.170000e+02         1\n",
      "2.250000e+03         1\n",
      "8.370000e+02         1\n",
      "1.675000e+03         1\n",
      "3.720000e+03         1\n",
      "2.514000e+03         1\n",
      "2.513000e+03         1\n",
      "1.350000e+04         1\n",
      "2.700000e+01         1\n",
      "7.800000e+02         1\n",
      "3.760000e+02         1\n",
      "8.680000e+02         1\n",
      "2.640000e+02         1\n",
      "2.460000e+02         1\n",
      "1.512000e+04         1\n",
      "2.491000e+03         1\n",
      "4.095000e+03         1\n",
      "Name: PAVIG11_, Length: 604, dtype: int64,\n",
      " 5.397605e-79    377934\n",
      "6.000000e+01      6753\n",
      "1.200000e+02      5770\n",
      "9.000000e+01      4223\n",
      "1.800000e+02      3939\n",
      "3.000000e+01      2773\n",
      "2.400000e+02      2349\n",
      "2.100000e+02      2311\n",
      "7.000000e+01      1851\n",
      "2.800000e+01      1844\n",
      "1.400000e+02      1829\n",
      "4.200000e+02      1796\n",
      "5.600000e+01      1765\n",
      "3.600000e+02      1737\n",
      "8.400000e+01      1480\n",
      "4.200000e+01      1415\n",
      "1.050000e+02      1382\n",
      "1.400000e+01      1356\n",
      "4.500000e+01      1334\n",
      "4.000000e+01      1193\n",
      "1.500000e+02      1123\n",
      "1.120000e+02      1046\n",
      "8.400000e+02       882\n",
      "2.100000e+01       878\n",
      "1.680000e+02       830\n",
      "3.000000e+02       825\n",
      "1.350000e+02       790\n",
      "2.000000e+01       790\n",
      "4.800000e+02       726\n",
      "2.800000e+02       724\n",
      "                 ...  \n",
      "4.780000e+02         1\n",
      "3.380000e+02         1\n",
      "6.830000e+02         1\n",
      "2.970000e+03         1\n",
      "1.770000e+02         1\n",
      "1.850000e+02         1\n",
      "4.730000e+02         1\n",
      "1.467000e+03         1\n",
      "4.960000e+02         1\n",
      "7.290000e+02         1\n",
      "1.056000e+03         1\n",
      "2.370000e+02         1\n",
      "3.794000e+03         1\n",
      "7.180000e+02         1\n",
      "4.600000e+02         1\n",
      "2.850000e+03         1\n",
      "2.184000e+03         1\n",
      "3.180000e+02         1\n",
      "4.492500e+04         1\n",
      "3.540000e+02         1\n",
      "1.397000e+03         1\n",
      "2.420000e+02         1\n",
      "3.470000e+02         1\n",
      "1.925000e+03         1\n",
      "9.640000e+02         1\n",
      "1.736000e+03         1\n",
      "1.960000e+03         1\n",
      "1.369000e+03         1\n",
      "3.420000e+02         1\n",
      "1.150000e+03         1\n",
      "Name: PAVIG21_, Length: 493, dtype: int64,\n",
      " 5.397605e-79    317751\n",
      "1.200000e+02      8133\n",
      "1.800000e+02      7706\n",
      "6.000000e+01      7594\n",
      "9.000000e+01      6067\n",
      "2.100000e+02      4519\n",
      "2.400000e+02      4458\n",
      "4.200000e+02      3649\n",
      "3.600000e+02      3505\n",
      "1.400000e+02      2925\n",
      "3.000000e+01      2920\n",
      "3.000000e+02      2688\n",
      "1.500000e+02      2632\n",
      "7.000000e+01      2246\n",
      "1.050000e+02      2128\n",
      "2.800000e+01      1940\n",
      "5.600000e+01      1885\n",
      "1.350000e+02      1809\n",
      "8.400000e+02      1680\n",
      "8.400000e+01      1671\n",
      "2.700000e+02      1662\n",
      "2.800000e+02      1569\n",
      "4.200000e+01      1567\n",
      "4.800000e+02      1553\n",
      "1.400000e+01      1484\n",
      "4.500000e+01      1397\n",
      "4.000000e+01      1289\n",
      "1.120000e+02      1245\n",
      "3.150000e+02      1137\n",
      "1.680000e+02      1127\n",
      "                 ...  \n",
      "1.947000e+03         1\n",
      "2.324000e+03         1\n",
      "1.425000e+03         1\n",
      "1.818000e+03         1\n",
      "2.044000e+03         1\n",
      "2.950000e+03         1\n",
      "1.223000e+03         1\n",
      "2.947000e+03         1\n",
      "1.821000e+03         1\n",
      "5.250000e+03         1\n",
      "4.905000e+03         1\n",
      "3.584000e+03         1\n",
      "5.810000e+02         1\n",
      "9.590000e+02         1\n",
      "3.752000e+03         1\n",
      "9.130000e+02         1\n",
      "3.893000e+03         1\n",
      "5.873000e+03         1\n",
      "6.680000e+02         1\n",
      "6.060000e+03         1\n",
      "2.490000e+03         1\n",
      "1.608000e+03         1\n",
      "2.150000e+03         1\n",
      "3.892000e+03         1\n",
      "2.424000e+03         1\n",
      "5.836000e+03         1\n",
      "7.290000e+02         1\n",
      "3.885000e+03         1\n",
      "3.402000e+03         1\n",
      "4.095000e+03         1\n",
      "Name: PA1VIGM_, Length: 1596, dtype: int64]\n"
     ]
    }
   ],
   "source": [
    "# Checking which categorical values still exist in the dataset and require missingness recoding\n",
    "\n",
    "updated_continuous_predictors_17 = [col_name for col_name in imputed_df_2017_v2.columns if col_name in countinuous_features_17]\n",
    "\n",
    "print('The continuous value counts before recoding:\\n')\n",
    "pprint([imputed_df_2017_v2[col_name].value_counts() for col_name in updated_continuous_predictors_17])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_collapse_cont = ['HHADULT', 'PHYSHLTH', 'MENTHLTH', 'POORHLTH', 'CHILDREN', 'DRNK3GE5', 'MAXDRNKS', 'EXERHMM1',\n",
    "                   'EXERHMM2', 'FLSHTMY2', '_DRNKWEK', 'MAXVO2_', 'FC60_', 'PAFREQ1_', 'PAFREQ2_', 'STRFREQ_']\n",
    "\n",
    "# Select continuous variables that have 77, 88, 99, 777, 999, 777777, 99900 as possible values\n",
    "to_collapse_cont2 = []\n",
    "to_collapse_dict_cont = defaultdict(list)\n",
    "\n",
    "for col in to_collapse_cont:\n",
    "    if all(elem in np.unique(imputed_df_2017_v2[col].values) for elem in [88.0, 77.0, 99.0]):\n",
    "        to_collapse_cont2.append(col)\n",
    "        to_collapse_dict_cont['map1'].append(col)\n",
    "    elif all(elem in np.unique(imputed_df_2017_v2[col].values) for elem in [88.0, 99.0]):\n",
    "        to_collapse_cont2.append(col)\n",
    "        to_collapse_dict_cont['map2'].append(col)        \n",
    "    elif all(elem in np.unique(imputed_df_2017_v2[col].values) for elem in [99., 77.]):\n",
    "        to_collapse_cont2.append(col)\n",
    "        to_collapse_dict_cont['map3'].append(col)\n",
    "    elif all(elem in np.unique(imputed_df_2017_v2[col].values) for elem in [777.0, 999.0]):\n",
    "        to_collapse_cont2.append(col)\n",
    "        to_collapse_dict_cont['map4'].append(col)\n",
    "    elif all(elem in np.unique(imputed_df_2017_v2[col].values) for elem in [777777.0, 999999.0]):\n",
    "        to_collapse_cont2.append(col)\n",
    "        to_collapse_dict_cont['map5'].append(col)\n",
    "    elif 99000.0 in imputed_df_2017_v2[col].values:\n",
    "        to_collapse_cont2.append(col)\n",
    "        to_collapse_dict_cont['map6'].append(col)\n",
    "    elif 99900.0 in imputed_df_2017_v2[col].values:\n",
    "        to_collapse_cont2.append(col)\n",
    "        to_collapse_dict_cont['map7'].append(col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The original value counts before collapsing missingness (continuous vars):\n",
      "\n",
      "[2.0     352810\n",
      "1.0      48179\n",
      "3.0      28803\n",
      "4.0      13174\n",
      "5.0       4100\n",
      "6.0       1275\n",
      "99.0       742\n",
      "7.0        407\n",
      "8.0        169\n",
      "10.0        76\n",
      "9.0         61\n",
      "77.0        58\n",
      "12.0        48\n",
      "13.0        29\n",
      "14.0        22\n",
      "11.0        21\n",
      "15.0        10\n",
      "18.0        10\n",
      "16.0         7\n",
      "21.0         3\n",
      "17.0         3\n",
      "32.0         2\n",
      "22.0         1\n",
      "30.0         1\n",
      "52.0         1\n",
      "20.0         1\n",
      "54.0         1\n",
      "40.0         1\n",
      "33.0         1\n",
      "Name: HHADULT, dtype: int64,\n",
      " 88.0    276112\n",
      "30.0     35676\n",
      "2.0      25524\n",
      "1.0      18411\n",
      "3.0      14919\n",
      "5.0      13180\n",
      "10.0      9725\n",
      "15.0      8929\n",
      "7.0       8367\n",
      "4.0       7851\n",
      "77.0      7213\n",
      "20.0      5841\n",
      "14.0      4686\n",
      "25.0      2318\n",
      "6.0       2246\n",
      "99.0      2117\n",
      "8.0       1428\n",
      "21.0      1058\n",
      "12.0      1049\n",
      "28.0       818\n",
      "29.0       433\n",
      "9.0        338\n",
      "18.0       298\n",
      "16.0       243\n",
      "27.0       232\n",
      "17.0       171\n",
      "13.0       154\n",
      "22.0       139\n",
      "11.0       138\n",
      "26.0       132\n",
      "24.0       129\n",
      "23.0        90\n",
      "19.0        51\n",
      "Name: PHYSHLTH, dtype: int64,\n",
      " 88.0    300135\n",
      "30.0     24843\n",
      "2.0      21661\n",
      "5.0      15711\n",
      "1.0      13973\n",
      "3.0      12862\n",
      "10.0     11494\n",
      "15.0     10974\n",
      "20.0      6623\n",
      "4.0       6441\n",
      "7.0       6308\n",
      "77.0      5026\n",
      "14.0      2399\n",
      "25.0      2306\n",
      "99.0      2176\n",
      "6.0       1876\n",
      "8.0       1257\n",
      "12.0       880\n",
      "28.0       663\n",
      "21.0       391\n",
      "29.0       378\n",
      "18.0       232\n",
      "9.0        207\n",
      "16.0       205\n",
      "27.0       183\n",
      "17.0       161\n",
      "13.0       144\n",
      "22.0       129\n",
      "24.0        99\n",
      "26.0        88\n",
      "11.0        88\n",
      "23.0        73\n",
      "19.0        30\n",
      "Name: MENTHLTH, dtype: int64,\n",
      " 88.0    342297\n",
      "30.0     20574\n",
      "2.0      13267\n",
      "1.0      11373\n",
      "5.0       8846\n",
      "3.0       8331\n",
      "15.0      7925\n",
      "10.0      7358\n",
      "20.0      5335\n",
      "7.0       4684\n",
      "4.0       4664\n",
      "77.0      3601\n",
      "14.0      2449\n",
      "25.0      2038\n",
      "6.0       1493\n",
      "99.0      1198\n",
      "8.0       1104\n",
      "12.0       729\n",
      "21.0       605\n",
      "28.0       452\n",
      "9.0        221\n",
      "18.0       215\n",
      "29.0       190\n",
      "16.0       172\n",
      "27.0       143\n",
      "13.0       130\n",
      "17.0       115\n",
      "24.0       110\n",
      "22.0        98\n",
      "26.0        94\n",
      "11.0        91\n",
      "23.0        78\n",
      "19.0        36\n",
      "Name: POORHLTH, dtype: int64,\n",
      " 88.0    330175\n",
      "1.0      47839\n",
      "2.0      40555\n",
      "3.0      17808\n",
      "4.0       6611\n",
      "99.0      3547\n",
      "5.0       2149\n",
      "6.0        834\n",
      "7.0        279\n",
      "8.0        122\n",
      "9.0         40\n",
      "10.0        16\n",
      "12.0        14\n",
      "11.0        10\n",
      "17.0         3\n",
      "14.0         3\n",
      "15.0         3\n",
      "16.0         3\n",
      "13.0         3\n",
      "72.0         1\n",
      "23.0         1\n",
      "Name: CHILDREN, dtype: int64,\n",
      " 88.0    391136\n",
      "1.0      20174\n",
      "2.0      11747\n",
      "3.0       5416\n",
      "4.0       4992\n",
      "5.0       3735\n",
      "77.0      2316\n",
      "30.0      1870\n",
      "10.0      1860\n",
      "6.0       1437\n",
      "8.0       1126\n",
      "15.0       903\n",
      "7.0        809\n",
      "20.0       672\n",
      "99.0       569\n",
      "12.0       402\n",
      "25.0       243\n",
      "14.0        84\n",
      "28.0        78\n",
      "9.0         73\n",
      "16.0        46\n",
      "29.0        41\n",
      "18.0        37\n",
      "13.0        29\n",
      "24.0        26\n",
      "27.0        25\n",
      "26.0        24\n",
      "21.0        23\n",
      "17.0        23\n",
      "23.0        22\n",
      "22.0        21\n",
      "11.0        18\n",
      "76.0        14\n",
      "40.0         5\n",
      "19.0         5\n",
      "50.0         4\n",
      "31.0         3\n",
      "36.0         2\n",
      "38.0         2\n",
      "32.0         2\n",
      "48.0         1\n",
      "60.0         1\n",
      "Name: DRNK3GE5, dtype: int64,\n",
      " 1.0     289847\n",
      "2.0      57985\n",
      "3.0      33926\n",
      "4.0      17910\n",
      "5.0      12989\n",
      "6.0      10737\n",
      "77.0      5797\n",
      "8.0       4509\n",
      "10.0      4253\n",
      "7.0       3695\n",
      "12.0      2934\n",
      "15.0      1080\n",
      "99.0      1043\n",
      "9.0        778\n",
      "20.0       646\n",
      "18.0       333\n",
      "30.0       265\n",
      "24.0       195\n",
      "14.0       194\n",
      "11.0       151\n",
      "16.0       141\n",
      "13.0       136\n",
      "25.0       101\n",
      "40.0        56\n",
      "17.0        50\n",
      "22.0        34\n",
      "50.0        28\n",
      "35.0        24\n",
      "23.0        20\n",
      "76.0        16\n",
      "         ...  \n",
      "32.0        13\n",
      "19.0        12\n",
      "48.0        10\n",
      "27.0         9\n",
      "45.0         9\n",
      "60.0         8\n",
      "26.0         8\n",
      "28.0         5\n",
      "38.0         5\n",
      "31.0         3\n",
      "42.0         3\n",
      "33.0         3\n",
      "70.0         3\n",
      "69.0         2\n",
      "65.0         2\n",
      "52.0         2\n",
      "37.0         2\n",
      "72.0         2\n",
      "53.0         2\n",
      "47.0         2\n",
      "39.0         2\n",
      "73.0         1\n",
      "58.0         1\n",
      "64.0         1\n",
      "75.0         1\n",
      "29.0         1\n",
      "43.0         1\n",
      "67.0         1\n",
      "46.0         1\n",
      "66.0         1\n",
      "Name: MAXDRNKS, Length: 62, dtype: int64,\n",
      " 30.0     223570\n",
      "100.0     67800\n",
      "45.0      27628\n",
      "200.0     21386\n",
      "20.0      20757\n",
      "130.0     13802\n",
      "15.0      11407\n",
      "40.0       9181\n",
      "300.0      8568\n",
      "400.0      6493\n",
      "777.0      5313\n",
      "10.0       5075\n",
      "25.0       3982\n",
      "35.0       2985\n",
      "230.0      2492\n",
      "500.0      2409\n",
      "50.0       2065\n",
      "600.0      1676\n",
      "115.0      1396\n",
      "5.0        1182\n",
      "1.0        1023\n",
      "800.0       980\n",
      "7.0         638\n",
      "120.0       599\n",
      "2.0         551\n",
      "430.0       460\n",
      "330.0       449\n",
      "999.0       447\n",
      "110.0       442\n",
      "3.0         349\n",
      "          ...  \n",
      "888.0         1\n",
      "111.0         1\n",
      "128.0         1\n",
      "256.0         1\n",
      "902.0         1\n",
      "458.0         1\n",
      "206.0         1\n",
      "955.0         1\n",
      "420.0         1\n",
      "123.0         1\n",
      "325.0         1\n",
      "610.0         1\n",
      "615.0         1\n",
      "154.0         1\n",
      "158.0         1\n",
      "51.0          1\n",
      "212.0         1\n",
      "639.0         1\n",
      "649.0         1\n",
      "234.0         1\n",
      "740.0         1\n",
      "424.0         1\n",
      "820.0         1\n",
      "701.0         1\n",
      "353.0         1\n",
      "707.0         1\n",
      "214.0         1\n",
      "355.0         1\n",
      "925.0         1\n",
      "255.0         1\n",
      "Name: EXERHMM1, Length: 221, dtype: int64,\n",
      " 100.0    297191\n",
      "30.0      42107\n",
      "200.0     18380\n",
      "20.0      14663\n",
      "45.0      13230\n",
      "15.0       9974\n",
      "130.0      7787\n",
      "300.0      7721\n",
      "400.0      5710\n",
      "10.0       5498\n",
      "40.0       3885\n",
      "777.0      3492\n",
      "25.0       2085\n",
      "500.0      2080\n",
      "230.0      2072\n",
      "600.0      1662\n",
      "5.0        1492\n",
      "35.0       1166\n",
      "800.0      1011\n",
      "50.0        853\n",
      "1.0         816\n",
      "115.0       612\n",
      "2.0         599\n",
      "7.0         554\n",
      "999.0       467\n",
      "3.0         426\n",
      "900.0       334\n",
      "330.0       321\n",
      "120.0       314\n",
      "430.0       289\n",
      "          ...  \n",
      "218.0         1\n",
      "256.0         1\n",
      "123.0         1\n",
      "955.0         1\n",
      "44.0          1\n",
      "440.0         1\n",
      "127.0         1\n",
      "715.0         1\n",
      "356.0         1\n",
      "707.0         1\n",
      "353.0         1\n",
      "453.0         1\n",
      "346.0         1\n",
      "345.0         1\n",
      "43.0          1\n",
      "333.0         1\n",
      "57.0          1\n",
      "41.0          1\n",
      "655.0         1\n",
      "920.0         1\n",
      "650.0         1\n",
      "121.0         1\n",
      "915.0         1\n",
      "113.0         1\n",
      "154.0         1\n",
      "305.0         1\n",
      "255.0         1\n",
      "603.0         1\n",
      "820.0         1\n",
      "510.0         1\n",
      "Name: EXERHMM2, Length: 181, dtype: int64,\n",
      " 102016.0    320378\n",
      "112016.0     28018\n",
      "102017.0     21801\n",
      "92016.0      18788\n",
      "777777.0     10455\n",
      "92017.0      10276\n",
      "122016.0      8625\n",
      "12017.0       7362\n",
      "112017.0      6972\n",
      "22017.0       4358\n",
      "82016.0       2473\n",
      "32017.0       2135\n",
      "82017.0       2006\n",
      "42017.0       1231\n",
      "122017.0      1189\n",
      "52017.0        808\n",
      "62017.0        682\n",
      "72017.0        512\n",
      "72016.0        376\n",
      "62016.0        358\n",
      "999999.0       327\n",
      "52016.0        209\n",
      "32016.0        177\n",
      "22016.0        176\n",
      "42016.0        154\n",
      "12016.0        104\n",
      "12018.0         60\n",
      "22018.0          6\n",
      "Name: FLSHTMY2, dtype: int64,\n",
      " 5.397605e-79    203183\n",
      "9.990000e+04     30066\n",
      "2.300000e+01     25114\n",
      "4.700000e+01     19291\n",
      "9.300000e+01     14046\n",
      "7.000000e+02     12739\n",
      "1.400000e+03      9232\n",
      "7.000000e+01      8987\n",
      "2.000000e+02      8858\n",
      "1.000000e+02      8419\n",
      "1.400000e+02      8137\n",
      "2.330000e+02      7755\n",
      "4.000000e+02      7699\n",
      "4.670000e+02      7400\n",
      "6.000000e+02      5798\n",
      "1.870000e+02      5733\n",
      "1.170000e+02      5356\n",
      "3.000000e+02      4910\n",
      "3.500000e+02      4797\n",
      "2.800000e+02      4480\n",
      "9.330000e+02      3673\n",
      "2.100000e+03      3479\n",
      "3.730000e+02      2342\n",
      "8.000000e+02      2329\n",
      "5.000000e+02      2217\n",
      "2.100000e+02      2155\n",
      "1.000000e+03      2054\n",
      "1.200000e+03      2034\n",
      "5.600000e+02      1896\n",
      "5.830000e+02      1831\n",
      "                 ...  \n",
      "1.470000e+04         1\n",
      "3.080000e+03         1\n",
      "7.583000e+03         1\n",
      "1.516700e+04         1\n",
      "7.933000e+03         1\n",
      "5.973000e+03         1\n",
      "8.050000e+03         1\n",
      "2.368300e+04         1\n",
      "8.100000e+03         1\n",
      "4.830000e+03         1\n",
      "6.650000e+03         1\n",
      "8.820000e+03         1\n",
      "3.966700e+04         1\n",
      "9.917000e+03         1\n",
      "5.100000e+03         1\n",
      "1.026700e+04         1\n",
      "1.440000e+04         1\n",
      "2.660000e+03         1\n",
      "4.560000e+04         1\n",
      "2.170000e+04         1\n",
      "1.092000e+04         1\n",
      "4.375000e+04         1\n",
      "5.320000e+03         1\n",
      "8.960000e+03         1\n",
      "2.777000e+03         1\n",
      "2.250000e+04         1\n",
      "2.823000e+03         1\n",
      "2.280000e+04         1\n",
      "5.717000e+03         1\n",
      "8.167000e+03         1\n",
      "Name: _DRNKWEK, Length: 278, dtype: int64,\n",
      " 2210.0     6333\n",
      "99900.0    6298\n",
      "2395.0     6189\n",
      "2580.0     6033\n",
      "2506.0     5775\n",
      "2321.0     5609\n",
      "2469.0     5562\n",
      "2358.0     5526\n",
      "2432.0     5511\n",
      "2247.0     5500\n",
      "2284.0     5474\n",
      "2654.0     5209\n",
      "2617.0     5005\n",
      "2543.0     4962\n",
      "2425.0     4952\n",
      "2691.0     4942\n",
      "2765.0     4935\n",
      "3690.0     4912\n",
      "2728.0     4853\n",
      "2150.0     4740\n",
      "2700.0     4733\n",
      "2590.0     4645\n",
      "2136.0     4474\n",
      "2099.0     4467\n",
      "2173.0     4454\n",
      "2062.0     4450\n",
      "2480.0     4423\n",
      "2315.0     4419\n",
      "2839.0     4414\n",
      "2535.0     4396\n",
      "           ... \n",
      "1581.0     1387\n",
      "1544.0     1266\n",
      "1435.0     1225\n",
      "1380.0     1115\n",
      "1325.0     1046\n",
      "1507.0      966\n",
      "1470.0      909\n",
      "1270.0      879\n",
      "1215.0      709\n",
      "1160.0      640\n",
      "1433.0      605\n",
      "1105.0      528\n",
      "1396.0      517\n",
      "1050.0      414\n",
      "1359.0      359\n",
      "995.0       277\n",
      "1322.0      263\n",
      "940.0       257\n",
      "1137.0      205\n",
      "885.0       169\n",
      "1285.0      158\n",
      "1248.0      128\n",
      "555.0       128\n",
      "830.0       105\n",
      "1211.0       70\n",
      "775.0        66\n",
      "720.0        52\n",
      "1174.0       40\n",
      "665.0        26\n",
      "610.0        19\n",
      "Name: MAXVO2_, Length: 163, dtype: int64,\n",
      " 379.0      6333\n",
      "99900.0    6298\n",
      "411.0      6189\n",
      "442.0      6033\n",
      "430.0      5775\n",
      "398.0      5609\n",
      "423.0      5562\n",
      "404.0      5526\n",
      "417.0      5511\n",
      "385.0      5500\n",
      "392.0      5474\n",
      "595.0      5473\n",
      "455.0      5209\n",
      "449.0      5005\n",
      "436.0      4962\n",
      "416.0      4952\n",
      "461.0      4942\n",
      "474.0      4935\n",
      "633.0      4912\n",
      "303.0      4867\n",
      "468.0      4853\n",
      "614.0      4799\n",
      "369.0      4740\n",
      "463.0      4733\n",
      "444.0      4645\n",
      "366.0      4474\n",
      "360.0      4467\n",
      "373.0      4454\n",
      "353.0      4450\n",
      "425.0      4423\n",
      "           ... \n",
      "277.0      1661\n",
      "702.0      1614\n",
      "696.0      1583\n",
      "709.0      1512\n",
      "255.0      1416\n",
      "271.0      1387\n",
      "227.0      1309\n",
      "237.0      1115\n",
      "258.0       966\n",
      "252.0       909\n",
      "218.0       879\n",
      "208.0       779\n",
      "199.0       640\n",
      "189.0       528\n",
      "239.0       517\n",
      "180.0       414\n",
      "233.0       359\n",
      "171.0       277\n",
      "161.0       257\n",
      "195.0       205\n",
      "152.0       169\n",
      "220.0       158\n",
      "214.0       128\n",
      "95.0        128\n",
      "142.0       105\n",
      "133.0        66\n",
      "123.0        52\n",
      "201.0        40\n",
      "114.0        26\n",
      "105.0        19\n",
      "Name: FC60_, Length: 156, dtype: int64,\n",
      " 7000.0     208126\n",
      "3000.0      51529\n",
      "2000.0      34160\n",
      "5000.0      29269\n",
      "4000.0      27331\n",
      "1000.0      16839\n",
      "6000.0      10275\n",
      "4667.0       8781\n",
      "3500.0       6405\n",
      "933.0        5829\n",
      "700.0        5545\n",
      "2333.0       5445\n",
      "467.0        5404\n",
      "1167.0       4875\n",
      "5833.0       4185\n",
      "2800.0       3305\n",
      "1867.0       3282\n",
      "1400.0       2984\n",
      "233.0        2592\n",
      "14000.0      2386\n",
      "1633.0       1672\n",
      "6533.0        966\n",
      "3733.0        906\n",
      "3267.0        754\n",
      "10000.0       700\n",
      "21000.0       580\n",
      "5600.0        449\n",
      "2100.0        419\n",
      "11667.0       359\n",
      "4200.0        341\n",
      "            ...  \n",
      "26000.0         3\n",
      "19000.0         2\n",
      "84000.0         2\n",
      "73000.0         2\n",
      "34000.0         2\n",
      "29000.0         1\n",
      "63000.0         1\n",
      "15400.0         1\n",
      "74000.0         1\n",
      "23000.0         1\n",
      "52000.0         1\n",
      "20300.0         1\n",
      "32000.0         1\n",
      "13533.0         1\n",
      "80000.0         1\n",
      "38000.0         1\n",
      "48000.0         1\n",
      "9100.0          1\n",
      "9567.0          1\n",
      "88000.0         1\n",
      "43000.0         1\n",
      "36000.0         1\n",
      "72000.0         1\n",
      "10733.0         1\n",
      "20533.0         1\n",
      "14233.0         1\n",
      "11900.0         1\n",
      "9800.0          1\n",
      "56000.0         1\n",
      "47000.0         1\n",
      "Name: PAFREQ1_, Length: 120, dtype: int64,\n",
      " 2000.0     290240\n",
      "3000.0      28557\n",
      "1000.0      25335\n",
      "7000.0      23822\n",
      "4000.0      10691\n",
      "5000.0      10483\n",
      "467.0        8389\n",
      "933.0        7471\n",
      "700.0        6228\n",
      "1167.0       4763\n",
      "2333.0       4353\n",
      "3500.0       3867\n",
      "233.0        3719\n",
      "4667.0       3631\n",
      "6000.0       3111\n",
      "1867.0       2960\n",
      "1400.0       2910\n",
      "2800.0       1948\n",
      "5833.0       1319\n",
      "1633.0       1220\n",
      "14000.0       801\n",
      "3267.0        443\n",
      "3733.0        401\n",
      "6533.0        283\n",
      "2100.0        276\n",
      "10000.0       262\n",
      "21000.0       196\n",
      "11667.0       161\n",
      "5600.0        155\n",
      "4200.0        141\n",
      "            ...  \n",
      "7467.0          4\n",
      "90000.0         3\n",
      "9800.0          3\n",
      "23000.0         3\n",
      "22000.0         2\n",
      "8867.0          2\n",
      "54000.0         2\n",
      "13067.0         2\n",
      "8633.0          2\n",
      "10733.0         2\n",
      "84000.0         2\n",
      "31000.0         1\n",
      "36000.0         1\n",
      "48000.0         1\n",
      "32000.0         1\n",
      "7700.0          1\n",
      "11200.0         1\n",
      "74000.0         1\n",
      "72000.0         1\n",
      "14700.0         1\n",
      "58000.0         1\n",
      "14233.0         1\n",
      "55000.0         1\n",
      "88000.0         1\n",
      "76000.0         1\n",
      "12600.0         1\n",
      "19600.0         1\n",
      "80000.0         1\n",
      "10267.0         1\n",
      "14467.0         1\n",
      "Name: PAFREQ2_, Length: 104, dtype: int64,\n",
      " 5.397605e-79    291189\n",
      "3.000000e+03     28915\n",
      "2.000000e+03     25604\n",
      "7.000000e+03     16790\n",
      "1.000000e+03     15712\n",
      "5.000000e+03     11415\n",
      "4.000000e+03     11214\n",
      "4.670000e+02      7134\n",
      "9.330000e+02      4392\n",
      "2.330000e+02      4318\n",
      "7.000000e+02      4246\n",
      "1.167000e+03      4105\n",
      "2.333000e+03      3668\n",
      "6.000000e+03      3518\n",
      "3.500000e+03      3441\n",
      "4.667000e+03      3108\n",
      "1.867000e+03      2178\n",
      "2.800000e+03      1917\n",
      "1.400000e+03      1551\n",
      "5.833000e+03      1077\n",
      "1.633000e+03       770\n",
      "1.400000e+04       549\n",
      "3.733000e+03       467\n",
      "3.267000e+03       353\n",
      "6.533000e+03       226\n",
      "2.100000e+03       207\n",
      "5.600000e+03       199\n",
      "1.000000e+04       186\n",
      "4.200000e+03       150\n",
      "6.067000e+03       107\n",
      "                 ...  \n",
      "8.800000e+04         3\n",
      "3.700000e+04         2\n",
      "1.866700e+04         2\n",
      "1.120000e+04         2\n",
      "1.446700e+04         2\n",
      "7.933000e+03         2\n",
      "7.600000e+04         2\n",
      "2.700000e+04         2\n",
      "5.600000e+04         2\n",
      "2.200000e+04         2\n",
      "2.300000e+04         2\n",
      "2.053300e+04         2\n",
      "4.700000e+04         1\n",
      "1.960000e+04         1\n",
      "1.726700e+04         1\n",
      "9.000000e+04         1\n",
      "7.700000e+03         1\n",
      "2.216700e+04         1\n",
      "3.200000e+04         1\n",
      "4.300000e+04         1\n",
      "1.423300e+04         1\n",
      "1.283300e+04         1\n",
      "2.600000e+04         1\n",
      "1.820000e+04         1\n",
      "7.300000e+04         1\n",
      "5.300000e+04         1\n",
      "1.026700e+04         1\n",
      "5.400000e+04         1\n",
      "6.400000e+04         1\n",
      "4.900000e+04         1\n",
      "Name: STRFREQ_, Length: 104, dtype: int64]\n"
     ]
    }
   ],
   "source": [
    "print('The original value counts before collapsing missingness (continuous vars):\\n')\n",
    "pprint([imputed_df_2017_v2[col_name].value_counts() for col_name in to_collapse_cont])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The new value counts after recoding:\n",
      "\n",
      "[ 0.0       276112\n",
      " 30.0       35676\n",
      " 2.0        25524\n",
      " 1.0        18411\n",
      " 3.0        14919\n",
      " 5.0        13180\n",
      " 10.0        9725\n",
      "-9999.0      9330\n",
      " 15.0        8929\n",
      " 7.0         8367\n",
      " 4.0         7851\n",
      " 20.0        5841\n",
      " 14.0        4686\n",
      " 25.0        2318\n",
      " 6.0         2246\n",
      " 8.0         1428\n",
      " 21.0        1058\n",
      " 12.0        1049\n",
      " 28.0         818\n",
      " 29.0         433\n",
      " 9.0          338\n",
      " 18.0         298\n",
      " 16.0         243\n",
      " 27.0         232\n",
      " 17.0         171\n",
      " 13.0         154\n",
      " 22.0         139\n",
      " 11.0         138\n",
      " 26.0         132\n",
      " 24.0         129\n",
      " 23.0          90\n",
      " 19.0          51\n",
      "Name: PHYSHLTH, dtype: int64,\n",
      "  0.0       300135\n",
      " 30.0       24843\n",
      " 2.0        21661\n",
      " 5.0        15711\n",
      " 1.0        13973\n",
      " 3.0        12862\n",
      " 10.0       11494\n",
      " 15.0       10974\n",
      "-9999.0      7202\n",
      " 20.0        6623\n",
      " 4.0         6441\n",
      " 7.0         6308\n",
      " 14.0        2399\n",
      " 25.0        2306\n",
      " 6.0         1876\n",
      " 8.0         1257\n",
      " 12.0         880\n",
      " 28.0         663\n",
      " 21.0         391\n",
      " 29.0         378\n",
      " 18.0         232\n",
      " 9.0          207\n",
      " 16.0         205\n",
      " 27.0         183\n",
      " 17.0         161\n",
      " 13.0         144\n",
      " 22.0         129\n",
      " 24.0          99\n",
      " 11.0          88\n",
      " 26.0          88\n",
      " 23.0          73\n",
      " 19.0          30\n",
      "Name: MENTHLTH, dtype: int64,\n",
      "  0.0       342297\n",
      " 30.0       20574\n",
      " 2.0        13267\n",
      " 1.0        11373\n",
      " 5.0         8846\n",
      " 3.0         8331\n",
      " 15.0        7925\n",
      " 10.0        7358\n",
      " 20.0        5335\n",
      "-9999.0      4799\n",
      " 7.0         4684\n",
      " 4.0         4664\n",
      " 14.0        2449\n",
      " 25.0        2038\n",
      " 6.0         1493\n",
      " 8.0         1104\n",
      " 12.0         729\n",
      " 21.0         605\n",
      " 28.0         452\n",
      " 9.0          221\n",
      " 18.0         215\n",
      " 29.0         190\n",
      " 16.0         172\n",
      " 27.0         143\n",
      " 13.0         130\n",
      " 17.0         115\n",
      " 24.0         110\n",
      " 22.0          98\n",
      " 26.0          94\n",
      " 11.0          91\n",
      " 23.0          78\n",
      " 19.0          36\n",
      "Name: POORHLTH, dtype: int64,\n",
      "  0.0       391136\n",
      " 1.0        20174\n",
      " 2.0        11747\n",
      " 3.0         5416\n",
      " 4.0         4992\n",
      " 5.0         3735\n",
      "-9999.0      2885\n",
      " 30.0        1870\n",
      " 10.0        1860\n",
      " 6.0         1437\n",
      " 8.0         1126\n",
      " 15.0         903\n",
      " 7.0          809\n",
      " 20.0         672\n",
      " 12.0         402\n",
      " 25.0         243\n",
      " 14.0          84\n",
      " 28.0          78\n",
      " 9.0           73\n",
      " 16.0          46\n",
      " 29.0          41\n",
      " 18.0          37\n",
      " 13.0          29\n",
      " 24.0          26\n",
      " 27.0          25\n",
      " 26.0          24\n",
      " 21.0          23\n",
      " 17.0          23\n",
      " 23.0          22\n",
      " 22.0          21\n",
      " 11.0          18\n",
      " 76.0          14\n",
      " 40.0           5\n",
      " 19.0           5\n",
      " 50.0           4\n",
      " 31.0           3\n",
      " 38.0           2\n",
      " 36.0           2\n",
      " 32.0           2\n",
      " 48.0           1\n",
      " 60.0           1\n",
      "Name: DRNK3GE5, dtype: int64]\n"
     ]
    }
   ],
   "source": [
    "#Designated missingness value = -9999\n",
    "missingness_mapper1_cont = {88.: 0, 77.: -9999, 99.: -9999}\n",
    "\n",
    "for column in to_collapse_dict_cont['map1']:\n",
    "    orig_column = imputed_df_2017_v2[column]\n",
    "    imputed_df_2017_v2[column] = imputed_df_2017_v2[column].map(missingness_mapper1_cont).fillna(orig_column)\n",
    "    \n",
    "print('The new value counts after recoding:\\n')\n",
    "pprint([imputed_df_2017_v2[col_name].value_counts() for col_name in to_collapse_dict_cont['map1']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The new value counts after recoding:\n",
      "\n",
      "[ 0.0       330175\n",
      " 1.0        47839\n",
      " 2.0        40555\n",
      " 3.0        17808\n",
      " 4.0         6611\n",
      "-9999.0      3547\n",
      " 5.0         2149\n",
      " 6.0          834\n",
      " 7.0          279\n",
      " 8.0          122\n",
      " 9.0           40\n",
      " 10.0          16\n",
      " 12.0          14\n",
      " 11.0          10\n",
      " 15.0           3\n",
      " 14.0           3\n",
      " 17.0           3\n",
      " 13.0           3\n",
      " 16.0           3\n",
      " 72.0           1\n",
      " 23.0           1\n",
      "Name: CHILDREN, dtype: int64]\n"
     ]
    }
   ],
   "source": [
    "missingness_mapper2_cont = {88.: 0, 99.: -9999}\n",
    "\n",
    "for column in to_collapse_dict_cont['map2']:\n",
    "    orig_column = imputed_df_2017_v2[column]\n",
    "    imputed_df_2017_v2[column] = imputed_df_2017_v2[column].map(missingness_mapper2_cont).fillna(orig_column)\n",
    "    \n",
    "print('The new value counts after recoding:\\n')\n",
    "pprint([imputed_df_2017_v2[col_name].value_counts() for col_name in to_collapse_dict_cont['map2']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The new value counts after recoding:\n",
      "\n",
      "[ 2.0       352810\n",
      " 1.0        48179\n",
      " 3.0        28803\n",
      " 4.0        13174\n",
      " 5.0         4100\n",
      " 6.0         1275\n",
      "-9999.0       800\n",
      " 7.0          407\n",
      " 8.0          169\n",
      " 10.0          76\n",
      " 9.0           61\n",
      " 12.0          48\n",
      " 13.0          29\n",
      " 14.0          22\n",
      " 11.0          21\n",
      " 15.0          10\n",
      " 18.0          10\n",
      " 16.0           7\n",
      " 21.0           3\n",
      " 17.0           3\n",
      " 32.0           2\n",
      " 22.0           1\n",
      " 30.0           1\n",
      " 52.0           1\n",
      " 20.0           1\n",
      " 54.0           1\n",
      " 40.0           1\n",
      " 33.0           1\n",
      "Name: HHADULT, dtype: int64,\n",
      "  1.0       289847\n",
      " 2.0        57985\n",
      " 3.0        33926\n",
      " 4.0        17910\n",
      " 5.0        12989\n",
      " 6.0        10737\n",
      "-9999.0      6840\n",
      " 8.0         4509\n",
      " 10.0        4253\n",
      " 7.0         3695\n",
      " 12.0        2934\n",
      " 15.0        1080\n",
      " 9.0          778\n",
      " 20.0         646\n",
      " 18.0         333\n",
      " 30.0         265\n",
      " 24.0         195\n",
      " 14.0         194\n",
      " 11.0         151\n",
      " 16.0         141\n",
      " 13.0         136\n",
      " 25.0         101\n",
      " 40.0          56\n",
      " 17.0          50\n",
      " 22.0          34\n",
      " 50.0          28\n",
      " 35.0          24\n",
      " 23.0          20\n",
      " 76.0          16\n",
      " 36.0          14\n",
      "            ...  \n",
      " 21.0          13\n",
      " 19.0          12\n",
      " 48.0          10\n",
      " 27.0           9\n",
      " 45.0           9\n",
      " 26.0           8\n",
      " 60.0           8\n",
      " 38.0           5\n",
      " 28.0           5\n",
      " 31.0           3\n",
      " 33.0           3\n",
      " 70.0           3\n",
      " 42.0           3\n",
      " 53.0           2\n",
      " 65.0           2\n",
      " 69.0           2\n",
      " 47.0           2\n",
      " 39.0           2\n",
      " 37.0           2\n",
      " 72.0           2\n",
      " 52.0           2\n",
      " 43.0           1\n",
      " 29.0           1\n",
      " 58.0           1\n",
      " 73.0           1\n",
      " 75.0           1\n",
      " 46.0           1\n",
      " 67.0           1\n",
      " 66.0           1\n",
      " 64.0           1\n",
      "Name: MAXDRNKS, Length: 61, dtype: int64]\n"
     ]
    }
   ],
   "source": [
    "missingness_mapper3_cont = {77.: -9999, 99.: -9999}\n",
    "\n",
    "for column in to_collapse_dict_cont['map3']:\n",
    "    orig_column = imputed_df_2017_v2[column]\n",
    "    imputed_df_2017_v2[column] = imputed_df_2017_v2[column].map(missingness_mapper3_cont).fillna(orig_column)\n",
    "    \n",
    "print('The new value counts after recoding:\\n')\n",
    "pprint([imputed_df_2017_v2[col_name].value_counts() for col_name in to_collapse_dict_cont['map3']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The new value counts after recoding:\n",
      "\n",
      "[ 30.0      223570\n",
      " 100.0      67800\n",
      " 45.0       27628\n",
      " 200.0      21386\n",
      " 20.0       20757\n",
      " 130.0      13802\n",
      " 15.0       11407\n",
      " 40.0        9181\n",
      " 300.0       8568\n",
      " 400.0       6493\n",
      "-9999.0      5760\n",
      " 10.0        5075\n",
      " 25.0        3982\n",
      " 35.0        2985\n",
      " 230.0       2492\n",
      " 500.0       2409\n",
      " 50.0        2065\n",
      " 600.0       1676\n",
      " 115.0       1396\n",
      " 5.0         1182\n",
      " 1.0         1023\n",
      " 800.0        980\n",
      " 7.0          638\n",
      " 120.0        599\n",
      " 2.0          551\n",
      " 430.0        460\n",
      " 330.0        449\n",
      " 110.0        442\n",
      " 3.0          349\n",
      " 700.0        339\n",
      "            ...  \n",
      " 902.0          1\n",
      " 256.0          1\n",
      " 422.0          1\n",
      " 845.0          1\n",
      " 128.0          1\n",
      " 855.0          1\n",
      " 435.0          1\n",
      " 888.0          1\n",
      " 358.0          1\n",
      " 355.0          1\n",
      " 124.0          1\n",
      " 214.0          1\n",
      " 958.0          1\n",
      " 610.0          1\n",
      " 615.0          1\n",
      " 154.0          1\n",
      " 955.0          1\n",
      " 158.0          1\n",
      " 212.0          1\n",
      " 840.0          1\n",
      " 639.0          1\n",
      " 649.0          1\n",
      " 325.0          1\n",
      " 235.0          1\n",
      " 234.0          1\n",
      " 820.0          1\n",
      " 701.0          1\n",
      " 353.0          1\n",
      " 707.0          1\n",
      " 255.0          1\n",
      "Name: EXERHMM1, Length: 220, dtype: int64,\n",
      "  100.0     297191\n",
      " 30.0       42107\n",
      " 200.0      18380\n",
      " 20.0       14663\n",
      " 45.0       13230\n",
      " 15.0        9974\n",
      " 130.0       7787\n",
      " 300.0       7721\n",
      " 400.0       5710\n",
      " 10.0        5498\n",
      "-9999.0      3959\n",
      " 40.0        3885\n",
      " 25.0        2085\n",
      " 500.0       2080\n",
      " 230.0       2072\n",
      " 600.0       1662\n",
      " 5.0         1492\n",
      " 35.0        1166\n",
      " 800.0       1011\n",
      " 50.0         853\n",
      " 1.0          816\n",
      " 115.0        612\n",
      " 2.0          599\n",
      " 7.0          554\n",
      " 3.0          426\n",
      " 900.0        334\n",
      " 330.0        321\n",
      " 120.0        314\n",
      " 430.0        289\n",
      " 700.0        278\n",
      "            ...  \n",
      " 106.0          1\n",
      " 121.0          1\n",
      " 204.0          1\n",
      " 155.0          1\n",
      " 445.0          1\n",
      " 154.0          1\n",
      " 610.0          1\n",
      " 113.0          1\n",
      " 453.0          1\n",
      " 57.0           1\n",
      " 256.0          1\n",
      " 915.0          1\n",
      " 315.0          1\n",
      " 123.0          1\n",
      " 715.0          1\n",
      " 356.0          1\n",
      " 707.0          1\n",
      " 353.0          1\n",
      " 346.0          1\n",
      " 345.0          1\n",
      " 43.0           1\n",
      " 333.0          1\n",
      " 41.0           1\n",
      " 655.0          1\n",
      " 920.0          1\n",
      " 650.0          1\n",
      " 44.0           1\n",
      " 440.0          1\n",
      " 310.0          1\n",
      " 510.0          1\n",
      "Name: EXERHMM2, Length: 180, dtype: int64]\n"
     ]
    }
   ],
   "source": [
    "missingness_mapper4_cont = {777.: -9999, 999.: -9999}\n",
    "\n",
    "for column in to_collapse_dict_cont['map4']:\n",
    "    orig_column = imputed_df_2017_v2[column]\n",
    "    imputed_df_2017_v2[column] = imputed_df_2017_v2[column].map(missingness_mapper4_cont).fillna(orig_column)\n",
    "    \n",
    "print('The new value counts after recoding:\\n')\n",
    "pprint([imputed_df_2017_v2[col_name].value_counts() for col_name in to_collapse_dict_cont['map4']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The new value counts after recoding:\n",
      "\n",
      "[ 102016.0    320378\n",
      " 112016.0     28018\n",
      " 102017.0     21801\n",
      " 92016.0      18788\n",
      "-9999.0       10782\n",
      " 92017.0      10276\n",
      " 122016.0      8625\n",
      " 12017.0       7362\n",
      " 112017.0      6972\n",
      " 22017.0       4358\n",
      " 82016.0       2473\n",
      " 32017.0       2135\n",
      " 82017.0       2006\n",
      " 42017.0       1231\n",
      " 122017.0      1189\n",
      " 52017.0        808\n",
      " 62017.0        682\n",
      " 72017.0        512\n",
      " 72016.0        376\n",
      " 62016.0        358\n",
      " 52016.0        209\n",
      " 32016.0        177\n",
      " 22016.0        176\n",
      " 42016.0        154\n",
      " 12016.0        104\n",
      " 12018.0         60\n",
      " 22018.0          6\n",
      "Name: FLSHTMY2, dtype: int64]\n"
     ]
    }
   ],
   "source": [
    "missingness_mapper5_cont = {777777.: -9999, 999999.: -9999}\n",
    "\n",
    "for column in to_collapse_dict_cont['map5']:\n",
    "    orig_column = imputed_df_2017_v2[column]\n",
    "    imputed_df_2017_v2[column] = imputed_df_2017_v2[column].map(missingness_mapper5_cont).fillna(orig_column)\n",
    "    \n",
    "print('The new value counts after recoding:\\n')\n",
    "pprint([imputed_df_2017_v2[col_name].value_counts() for col_name in to_collapse_dict_cont['map5']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The new value counts after recoding:\n",
      "\n",
      "[7000.0     208126\n",
      "3000.0      51529\n",
      "2000.0      34160\n",
      "5000.0      29269\n",
      "4000.0      27331\n",
      "1000.0      16839\n",
      "6000.0      10275\n",
      "4667.0       8781\n",
      "3500.0       6405\n",
      "933.0        5829\n",
      "700.0        5545\n",
      "2333.0       5445\n",
      "467.0        5404\n",
      "1167.0       4875\n",
      "5833.0       4185\n",
      "2800.0       3305\n",
      "1867.0       3282\n",
      "1400.0       2984\n",
      "233.0        2592\n",
      "14000.0      2386\n",
      "1633.0       1672\n",
      "6533.0        966\n",
      "3733.0        906\n",
      "3267.0        754\n",
      "10000.0       700\n",
      "21000.0       580\n",
      "5600.0        449\n",
      "2100.0        419\n",
      "11667.0       359\n",
      "4200.0        341\n",
      "            ...  \n",
      "49000.0         3\n",
      "73000.0         2\n",
      "84000.0         2\n",
      "19000.0         2\n",
      "34000.0         2\n",
      "48000.0         1\n",
      "63000.0         1\n",
      "9100.0          1\n",
      "32000.0         1\n",
      "36000.0         1\n",
      "52000.0         1\n",
      "74000.0         1\n",
      "47000.0         1\n",
      "9567.0          1\n",
      "20533.0         1\n",
      "23000.0         1\n",
      "80000.0         1\n",
      "43000.0         1\n",
      "10733.0         1\n",
      "9800.0          1\n",
      "72000.0         1\n",
      "20300.0         1\n",
      "29000.0         1\n",
      "13533.0         1\n",
      "38000.0         1\n",
      "88000.0         1\n",
      "14233.0         1\n",
      "11900.0         1\n",
      "56000.0         1\n",
      "15400.0         1\n",
      "Name: PAFREQ1_, Length: 120, dtype: int64,\n",
      " 2000.0     290240\n",
      "3000.0      28557\n",
      "1000.0      25335\n",
      "7000.0      23822\n",
      "4000.0      10691\n",
      "5000.0      10483\n",
      "467.0        8389\n",
      "933.0        7471\n",
      "700.0        6228\n",
      "1167.0       4763\n",
      "2333.0       4353\n",
      "3500.0       3867\n",
      "233.0        3719\n",
      "4667.0       3631\n",
      "6000.0       3111\n",
      "1867.0       2960\n",
      "1400.0       2910\n",
      "2800.0       1948\n",
      "5833.0       1319\n",
      "1633.0       1220\n",
      "14000.0       801\n",
      "3267.0        443\n",
      "3733.0        401\n",
      "6533.0        283\n",
      "2100.0        276\n",
      "10000.0       262\n",
      "21000.0       196\n",
      "11667.0       161\n",
      "5600.0        155\n",
      "4200.0        141\n",
      "            ...  \n",
      "7467.0          4\n",
      "90000.0         3\n",
      "9800.0          3\n",
      "23000.0         3\n",
      "13067.0         2\n",
      "8867.0          2\n",
      "54000.0         2\n",
      "22000.0         2\n",
      "84000.0         2\n",
      "8633.0          2\n",
      "10733.0         2\n",
      "32000.0         1\n",
      "88000.0         1\n",
      "14467.0         1\n",
      "55000.0         1\n",
      "74000.0         1\n",
      "36000.0         1\n",
      "14233.0         1\n",
      "58000.0         1\n",
      "14700.0         1\n",
      "48000.0         1\n",
      "80000.0         1\n",
      "31000.0         1\n",
      "7700.0          1\n",
      "12600.0         1\n",
      "19600.0         1\n",
      "10267.0         1\n",
      "72000.0         1\n",
      "11200.0         1\n",
      "76000.0         1\n",
      "Name: PAFREQ2_, Length: 104, dtype: int64,\n",
      " 5.397605e-79    291189\n",
      "3.000000e+03     28915\n",
      "2.000000e+03     25604\n",
      "7.000000e+03     16790\n",
      "1.000000e+03     15712\n",
      "5.000000e+03     11415\n",
      "4.000000e+03     11214\n",
      "4.670000e+02      7134\n",
      "9.330000e+02      4392\n",
      "2.330000e+02      4318\n",
      "7.000000e+02      4246\n",
      "1.167000e+03      4105\n",
      "2.333000e+03      3668\n",
      "6.000000e+03      3518\n",
      "3.500000e+03      3441\n",
      "4.667000e+03      3108\n",
      "1.867000e+03      2178\n",
      "2.800000e+03      1917\n",
      "1.400000e+03      1551\n",
      "5.833000e+03      1077\n",
      "1.633000e+03       770\n",
      "1.400000e+04       549\n",
      "3.733000e+03       467\n",
      "3.267000e+03       353\n",
      "6.533000e+03       226\n",
      "2.100000e+03       207\n",
      "5.600000e+03       199\n",
      "1.000000e+04       186\n",
      "4.200000e+03       150\n",
      "6.067000e+03       107\n",
      "                 ...  \n",
      "8.800000e+04         3\n",
      "2.700000e+04         2\n",
      "3.700000e+04         2\n",
      "1.446700e+04         2\n",
      "1.866700e+04         2\n",
      "7.933000e+03         2\n",
      "5.600000e+04         2\n",
      "2.200000e+04         2\n",
      "7.600000e+04         2\n",
      "1.120000e+04         2\n",
      "2.053300e+04         2\n",
      "2.300000e+04         2\n",
      "6.400000e+04         1\n",
      "1.960000e+04         1\n",
      "2.216700e+04         1\n",
      "1.726700e+04         1\n",
      "1.423300e+04         1\n",
      "4.300000e+04         1\n",
      "4.700000e+04         1\n",
      "3.200000e+04         1\n",
      "4.900000e+04         1\n",
      "7.700000e+03         1\n",
      "5.400000e+04         1\n",
      "5.300000e+04         1\n",
      "1.820000e+04         1\n",
      "7.300000e+04         1\n",
      "1.026700e+04         1\n",
      "2.600000e+04         1\n",
      "1.283300e+04         1\n",
      "9.000000e+04         1\n",
      "Name: STRFREQ_, Length: 104, dtype: int64]\n"
     ]
    }
   ],
   "source": [
    "missingness_mapper6_cont = {99000.: -9999}\n",
    "\n",
    "for column in to_collapse_dict_cont['map6']:\n",
    "    orig_column = imputed_df_2017_v2[column]\n",
    "    imputed_df_2017_v2[column] = imputed_df_2017_v2[column].map(missingness_mapper6_cont).fillna(orig_column)\n",
    "    \n",
    "print('The new value counts after recoding:\\n')\n",
    "pprint([imputed_df_2017_v2[col_name].value_counts() for col_name in to_collapse_dict_cont['map6']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The new value counts after recoding:\n",
      "\n",
      "[ 5.397605e-79    203183\n",
      "-9.999000e+03     30066\n",
      " 2.300000e+01     25114\n",
      " 4.700000e+01     19291\n",
      " 9.300000e+01     14046\n",
      " 7.000000e+02     12739\n",
      " 1.400000e+03      9232\n",
      " 7.000000e+01      8987\n",
      " 2.000000e+02      8858\n",
      " 1.000000e+02      8419\n",
      " 1.400000e+02      8137\n",
      " 2.330000e+02      7755\n",
      " 4.000000e+02      7699\n",
      " 4.670000e+02      7400\n",
      " 6.000000e+02      5798\n",
      " 1.870000e+02      5733\n",
      " 1.170000e+02      5356\n",
      " 3.000000e+02      4910\n",
      " 3.500000e+02      4797\n",
      " 2.800000e+02      4480\n",
      " 9.330000e+02      3673\n",
      " 2.100000e+03      3479\n",
      " 3.730000e+02      2342\n",
      " 8.000000e+02      2329\n",
      " 5.000000e+02      2217\n",
      " 2.100000e+02      2155\n",
      " 1.000000e+03      2054\n",
      " 1.200000e+03      2034\n",
      " 5.600000e+02      1896\n",
      " 5.830000e+02      1831\n",
      "                  ...  \n",
      " 3.966700e+04         1\n",
      " 8.820000e+03         1\n",
      " 5.100000e+03         1\n",
      " 1.026700e+04         1\n",
      " 7.583000e+03         1\n",
      " 3.300000e+03         1\n",
      " 1.516700e+04         1\n",
      " 6.650000e+03         1\n",
      " 4.830000e+03         1\n",
      " 1.470000e+04         1\n",
      " 8.120000e+03         1\n",
      " 5.040000e+04         1\n",
      " 1.440000e+04         1\n",
      " 9.473000e+03         1\n",
      " 3.173000e+03         1\n",
      " 5.973000e+03         1\n",
      " 3.547000e+03         1\n",
      " 1.241300e+04         1\n",
      " 4.340000e+04         1\n",
      " 1.866700e+04         1\n",
      " 5.200000e+03         1\n",
      " 4.560000e+04         1\n",
      " 2.170000e+04         1\n",
      " 1.092000e+04         1\n",
      " 3.920000e+04         1\n",
      " 4.620000e+03         1\n",
      " 9.200000e+03         1\n",
      " 4.375000e+04         1\n",
      " 1.829300e+04         1\n",
      " 8.167000e+03         1\n",
      "Name: _DRNKWEK, Length: 278, dtype: int64,\n",
      "  2210.0    6333\n",
      "-9999.0    6298\n",
      " 2395.0    6189\n",
      " 2580.0    6033\n",
      " 2506.0    5775\n",
      " 2321.0    5609\n",
      " 2469.0    5562\n",
      " 2358.0    5526\n",
      " 2432.0    5511\n",
      " 2247.0    5500\n",
      " 2284.0    5474\n",
      " 2654.0    5209\n",
      " 2617.0    5005\n",
      " 2543.0    4962\n",
      " 2425.0    4952\n",
      " 2691.0    4942\n",
      " 2765.0    4935\n",
      " 3690.0    4912\n",
      " 2728.0    4853\n",
      " 2150.0    4740\n",
      " 2700.0    4733\n",
      " 2590.0    4645\n",
      " 2136.0    4474\n",
      " 2099.0    4467\n",
      " 2173.0    4454\n",
      " 2062.0    4450\n",
      " 2480.0    4423\n",
      " 2315.0    4419\n",
      " 2839.0    4414\n",
      " 2535.0    4396\n",
      "           ... \n",
      " 1581.0    1387\n",
      " 1544.0    1266\n",
      " 1435.0    1225\n",
      " 1380.0    1115\n",
      " 1325.0    1046\n",
      " 1507.0     966\n",
      " 1470.0     909\n",
      " 1270.0     879\n",
      " 1215.0     709\n",
      " 1160.0     640\n",
      " 1433.0     605\n",
      " 1105.0     528\n",
      " 1396.0     517\n",
      " 1050.0     414\n",
      " 1359.0     359\n",
      " 995.0      277\n",
      " 1322.0     263\n",
      " 940.0      257\n",
      " 1137.0     205\n",
      " 885.0      169\n",
      " 1285.0     158\n",
      " 555.0      128\n",
      " 1248.0     128\n",
      " 830.0      105\n",
      " 1211.0      70\n",
      " 775.0       66\n",
      " 720.0       52\n",
      " 1174.0      40\n",
      " 665.0       26\n",
      " 610.0       19\n",
      "Name: MAXVO2_, Length: 163, dtype: int64,\n",
      "  379.0     6333\n",
      "-9999.0    6298\n",
      " 411.0     6189\n",
      " 442.0     6033\n",
      " 430.0     5775\n",
      " 398.0     5609\n",
      " 423.0     5562\n",
      " 404.0     5526\n",
      " 417.0     5511\n",
      " 385.0     5500\n",
      " 392.0     5474\n",
      " 595.0     5473\n",
      " 455.0     5209\n",
      " 449.0     5005\n",
      " 436.0     4962\n",
      " 416.0     4952\n",
      " 461.0     4942\n",
      " 474.0     4935\n",
      " 633.0     4912\n",
      " 303.0     4867\n",
      " 468.0     4853\n",
      " 614.0     4799\n",
      " 369.0     4740\n",
      " 463.0     4733\n",
      " 444.0     4645\n",
      " 366.0     4474\n",
      " 360.0     4467\n",
      " 373.0     4454\n",
      " 353.0     4450\n",
      " 425.0     4423\n",
      "           ... \n",
      " 277.0     1661\n",
      " 702.0     1614\n",
      " 696.0     1583\n",
      " 709.0     1512\n",
      " 255.0     1416\n",
      " 271.0     1387\n",
      " 227.0     1309\n",
      " 237.0     1115\n",
      " 258.0      966\n",
      " 252.0      909\n",
      " 218.0      879\n",
      " 208.0      779\n",
      " 199.0      640\n",
      " 189.0      528\n",
      " 239.0      517\n",
      " 180.0      414\n",
      " 233.0      359\n",
      " 171.0      277\n",
      " 161.0      257\n",
      " 195.0      205\n",
      " 152.0      169\n",
      " 220.0      158\n",
      " 95.0       128\n",
      " 214.0      128\n",
      " 142.0      105\n",
      " 133.0       66\n",
      " 123.0       52\n",
      " 201.0       40\n",
      " 114.0       26\n",
      " 105.0       19\n",
      "Name: FC60_, Length: 156, dtype: int64]\n"
     ]
    }
   ],
   "source": [
    "missingness_mapper7_cont = {99900.: -9999}\n",
    "\n",
    "for column in to_collapse_dict_cont['map7']:\n",
    "    orig_column = imputed_df_2017_v2[column]\n",
    "    imputed_df_2017_v2[column] = imputed_df_2017_v2[column].map(missingness_mapper7_cont).fillna(orig_column)\n",
    "    \n",
    "print('The new value counts after recoding:\\n')\n",
    "pprint([imputed_df_2017_v2[col_name].value_counts() for col_name in to_collapse_dict_cont['map7']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 0 column(s) with >= 60% missing data to be dropped after collapsing missingness.\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "#Identifying continuous features w/ high missingness (>= 60% of total sample) after collapsing missingness\n",
    "\n",
    "missingness_threshold3 = round(imputed_df_2017_v2.shape[0]*.60)\n",
    "\n",
    "high_missingness3 = []\n",
    "\n",
    "for col_name in imputed_df_2017_v2.columns:\n",
    "    if -9999.0 in imputed_df_2017_v2[col_name].value_counts().keys():     \n",
    "        if imputed_df_2017_v2[col_name].value_counts()[-9999.0] >= missingness_threshold3:\n",
    "            high_missingness3.append(col_name)\n",
    "\n",
    "print('There are', len(high_missingness3), 'column(s) with >= 60% missing data to be dropped after collapsing missingness.')\n",
    "pprint(high_missingness3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HHADULT Missing Count: 800\n",
      "PHYSHLTH Missing Count: 9330\n",
      "MENTHLTH Missing Count: 7202\n",
      "POORHLTH Missing Count: 4799\n",
      "CHILDREN Missing Count: 3547\n",
      "DRNK3GE5 Missing Count: 2885\n",
      "MAXDRNKS Missing Count: 6840\n",
      "EXERHMM1 Missing Count: 5760\n",
      "EXERHMM2 Missing Count: 3959\n",
      "FLSHTMY2 Missing Count: 10782\n",
      "_DRNKWEK Missing Count: 30066\n",
      "MAXVO2_ Missing Count: 6298\n",
      "FC60_ Missing Count: 6298\n",
      "PAFREQ1_ Missing Count: 26\n",
      "PAFREQ2_ Missing Count: 29\n",
      "STRFREQ_ Missing Count: 15\n"
     ]
    }
   ],
   "source": [
    "# Checking NaN counts before imputation (round 3)\n",
    "\n",
    "nan_after_collapse_cont = []\n",
    "\n",
    "for col_name in imputed_df_2017_v2.columns:\n",
    "    if -9999.0 in imputed_df_2017_v2[col_name].value_counts().keys():\n",
    "        nan_after_collapse_cont.append(col_name)\n",
    "        print(col_name, 'Missing Count:', imputed_df_2017_v2[col_name].value_counts()[-9999.0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Applying Simple Imputation a third time, after collapsing continuous missingness\n",
    "\n",
    "imp3 = SimpleImputer(missing_values = -9999.0, strategy=\"most_frequent\")\n",
    "imputed_df_2017_v3 = pd.DataFrame(imp3.fit_transform(imputed_df_2017_v2), columns = imputed_df_2017_v2.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_STATE</th>\n",
       "      <th>CTELENM1</th>\n",
       "      <th>PVTRESD1</th>\n",
       "      <th>STATERE1</th>\n",
       "      <th>CELLFON4</th>\n",
       "      <th>NUMADULT</th>\n",
       "      <th>NUMMEN</th>\n",
       "      <th>NUMWOMEN</th>\n",
       "      <th>CTELNUM1</th>\n",
       "      <th>CELLFON5</th>\n",
       "      <th>...</th>\n",
       "      <th>_PAINDX1</th>\n",
       "      <th>_PA150R2</th>\n",
       "      <th>_PA300R2</th>\n",
       "      <th>_PA30021</th>\n",
       "      <th>_PASTRNG</th>\n",
       "      <th>_PAREC1</th>\n",
       "      <th>_PASTAE1</th>\n",
       "      <th>_RFSEAT2</th>\n",
       "      <th>_RFSEAT3</th>\n",
       "      <th>_AIDTST3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 162 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   _STATE  CTELENM1  PVTRESD1  STATERE1  CELLFON4  NUMADULT        NUMMEN  \\\n",
       "0     1.0       1.0       1.0       1.0       2.0       1.0  5.397605e-79   \n",
       "1     1.0       1.0       1.0       1.0       2.0       2.0  1.000000e+00   \n",
       "2     1.0       1.0       1.0       1.0       2.0       2.0  1.000000e+00   \n",
       "3     1.0       1.0       1.0       1.0       2.0       1.0  5.397605e-79   \n",
       "4     1.0       1.0       1.0       1.0       2.0       1.0  5.397605e-79   \n",
       "\n",
       "   NUMWOMEN  CTELNUM1  CELLFON5    ...     _PAINDX1  _PA150R2  _PA300R2  \\\n",
       "0       1.0       1.0       1.0    ...          1.0       1.0       1.0   \n",
       "1       1.0       1.0       1.0    ...          1.0       1.0       1.0   \n",
       "2       1.0       1.0       1.0    ...          2.0       3.0       3.0   \n",
       "3       1.0       1.0       1.0    ...          9.0       9.0       9.0   \n",
       "4       1.0       1.0       1.0    ...          2.0       3.0       3.0   \n",
       "\n",
       "   _PA30021  _PASTRNG  _PAREC1  _PASTAE1  _RFSEAT2  _RFSEAT3  _AIDTST3  \n",
       "0       1.0       1.0      1.0       1.0       1.0       1.0       1.0  \n",
       "1       1.0       1.0      1.0       1.0       1.0       1.0       1.0  \n",
       "2       2.0       2.0      4.0       2.0       1.0       1.0       2.0  \n",
       "3       9.0       9.0      9.0       9.0       9.0       9.0       2.0  \n",
       "4       2.0       2.0      4.0       2.0       1.0       2.0       2.0  \n",
       "\n",
       "[5 rows x 162 columns]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imputed_df_2017_v3.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Checking NaN counts after imputation (round 3)\n",
    "\n",
    "nan_after_collapse_cont2 = []\n",
    "\n",
    "for col_name in imputed_df_2017_v3.columns:\n",
    "    if -9999.0 in imputed_df_2017_v3[col_name].value_counts().keys():\n",
    "        nan_after_collapse_cont2.append(col_name)\n",
    "        print(col_name, 'Missing Count:', imputed_df_2017_v3[col_name].value_counts()[-9999.0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Part 1E: Dummy code categorical predictors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "#******Dummy Coding Categorical Variables******\n",
    "\n",
    "# Updating list of categorical features for dummy coding\n",
    "updated_categorical_features_17 = [col_name for col_name in imputed_df_2017_v3.columns if col_name in categorical_features_17 and col_name not in all_possible_outcomes]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Making copy of imputed dataframe; renaming for dummy coding\n",
    "dummy_df_2017 = imputed_df_2017_v3.copy(deep=True)\n",
    "\n",
    "# Creating dummy coded variables for categorical predictors \n",
    "for predictor in updated_categorical_features_17:\n",
    "    cat_list = 'predictor' + '_' + predictor\n",
    "    cat_list = pd.get_dummies(imputed_df_2017_v3[predictor], prefix=predictor)\n",
    "\n",
    "    #Adding dummy coded variables to imputed df\n",
    "    dummy_df_2017 = dummy_df_2017.join(cat_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The shape of the dummy coded dataframe (before addressing multicolinearity) is: (450016, 731)\n"
     ]
    }
   ],
   "source": [
    "print('The shape of the dummy coded dataframe (before addressing multicolinearity) is:', dummy_df_2017.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 104 features that must have one column dropped (reference column) to avoid multicolinearity.\n",
      "There are 104 redundant columns identified to be dropped to avoid multicolinearity.\n"
     ]
    }
   ],
   "source": [
    "# Determining which dummy coded columns will required dropping of redundant columns to avoid multicolinearity\n",
    "\n",
    "multicolinearity_columns = []\n",
    "\n",
    "for predictor in updated_categorical_features_17:\n",
    "    if len(dummy_df_2017[predictor].value_counts().keys()) >= 2:\n",
    "        multicolinearity_columns.append(predictor)\n",
    "        \n",
    "print('There are', len(multicolinearity_columns), 'features that must have one column dropped (reference column) to avoid multicolinearity.')\n",
    "\n",
    "# Drop the first column of every multicolinearity column\n",
    "reference_cols_to_drop = [col_name + '_1.0' for col_name in multicolinearity_columns]\n",
    "print('There are', len(reference_cols_to_drop), 'redundant columns identified to be dropped to avoid multicolinearity.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dummy coded dataframe is the expected shape after addressing multicolinearity: True\n"
     ]
    }
   ],
   "source": [
    "dummy_df_2017.drop(columns = reference_cols_to_drop, inplace = True)\n",
    "print('The dummy coded dataframe is the expected shape after addressing multicolinearity:', dummy_df_2017.shape[1] == 627)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dropping original, non-dummy coded categorical features\n",
    "\n",
    "dummy_df_2017.drop(columns = multicolinearity_columns, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dataframe is the expected shape after dropping original, non-dummy coded categorical features: True\n"
     ]
    }
   ],
   "source": [
    "print('The dataframe is the expected shape after dropping original, non-dummy coded categorical features:', dummy_df_2017.shape[1] == 523)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The shape of the dummy coded dataframe is: (450016, 523)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CTELENM1</th>\n",
       "      <th>STATERE1</th>\n",
       "      <th>NUMADULT</th>\n",
       "      <th>NUMMEN</th>\n",
       "      <th>NUMWOMEN</th>\n",
       "      <th>CTELNUM1</th>\n",
       "      <th>CELLFON5</th>\n",
       "      <th>HHADULT</th>\n",
       "      <th>PHYSHLTH</th>\n",
       "      <th>MENTHLTH</th>\n",
       "      <th>...</th>\n",
       "      <th>_PAREC1_3.0</th>\n",
       "      <th>_PAREC1_4.0</th>\n",
       "      <th>_PAREC1_9.0</th>\n",
       "      <th>_PASTAE1_2.0</th>\n",
       "      <th>_PASTAE1_9.0</th>\n",
       "      <th>_RFSEAT2_2.0</th>\n",
       "      <th>_RFSEAT2_9.0</th>\n",
       "      <th>_RFSEAT3_2.0</th>\n",
       "      <th>_RFSEAT3_9.0</th>\n",
       "      <th>_AIDTST3_2.0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 523 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   CTELENM1  STATERE1  NUMADULT        NUMMEN  NUMWOMEN  CTELNUM1  CELLFON5  \\\n",
       "0       1.0       1.0       1.0  5.397605e-79       1.0       1.0       1.0   \n",
       "1       1.0       1.0       2.0  1.000000e+00       1.0       1.0       1.0   \n",
       "2       1.0       1.0       2.0  1.000000e+00       1.0       1.0       1.0   \n",
       "3       1.0       1.0       1.0  5.397605e-79       1.0       1.0       1.0   \n",
       "4       1.0       1.0       1.0  5.397605e-79       1.0       1.0       1.0   \n",
       "\n",
       "   HHADULT  PHYSHLTH  MENTHLTH      ...       _PAREC1_3.0  _PAREC1_4.0  \\\n",
       "0      2.0       0.0       0.0      ...                 0            0   \n",
       "1      2.0       0.0       0.0      ...                 0            0   \n",
       "2      2.0       0.0       0.0      ...                 0            1   \n",
       "3      2.0       0.0       0.0      ...                 0            0   \n",
       "4      2.0      14.0       0.0      ...                 0            1   \n",
       "\n",
       "   _PAREC1_9.0  _PASTAE1_2.0  _PASTAE1_9.0  _RFSEAT2_2.0  _RFSEAT2_9.0  \\\n",
       "0            0             0             0             0             0   \n",
       "1            0             0             0             0             0   \n",
       "2            0             1             0             0             0   \n",
       "3            1             0             1             0             1   \n",
       "4            0             1             0             0             0   \n",
       "\n",
       "   _RFSEAT3_2.0  _RFSEAT3_9.0  _AIDTST3_2.0  \n",
       "0             0             0             0  \n",
       "1             0             0             0  \n",
       "2             0             0             1  \n",
       "3             0             1             1  \n",
       "4             1             0             1  \n",
       "\n",
       "[5 rows x 523 columns]"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View dummy coded dataframe\n",
    "\n",
    "print('The shape of the dummy coded dataframe is:', dummy_df_2017.shape)\n",
    "\n",
    "dummy_df_2017.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Part 1F: Identify disease-specific features for removal from features based on outcome of interest prior to modeling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Identify features to drop that correspond to questions only asked to respondents w/ a particular diagnosis\n",
    "# Avoids artificial inflation of model performance\n",
    "\n",
    "diabetes_features = ['DIABAGE2', 'INSULIN', 'BLDSUGAR', 'FEETCHK2', 'DOCTDIAB', 'CHKHEMO3', 'FEETCHK', 'EYEEXAM',\n",
    "                    'DIABEYE', 'DIABEDU']\n",
    "\n",
    "heart_disease_attack_features = ['HAREHAB1']\n",
    "\n",
    "hypertension_features =  ['BPMEDS', 'BPEATHBT', 'BPSALT', 'BPALCHOL', 'BPEXER', 'BPEATADV', 'BPSLTADV', 'BPALCADV',\n",
    "                           'BPEXRADV',  'BPMEDADV', 'BPHI2MR']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The shape of the diabetes dataframe is: (450016, 523)\n"
     ]
    }
   ],
   "source": [
    "# Checking to see if any of the identified disease specific features ar still present in the dataset\n",
    "\n",
    "# Diabetes\n",
    "diabetes_df_2017 = dummy_df_2017.copy(deep=True)\n",
    "\n",
    "for feature in diabetes_features:\n",
    "    pattern = re.compile(feature + '*')\n",
    "    for col_name in diabetes_df_2017.columns:\n",
    "        if pattern.findall(col_name):\n",
    "            print(pattern.match(col_name))\n",
    "            \n",
    "print('The shape of the diabetes dataframe is:', diabetes_df_2017.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The shape of the heart disease/heart attack dataframe is: (450016, 523)\n"
     ]
    }
   ],
   "source": [
    "# Heart Disease/Heart Attack\n",
    "heart_disease_attack_df_2017 = dummy_df_2017.copy(deep=True)\n",
    "\n",
    "for feature in heart_disease_attack_features:\n",
    "    pattern = re.compile(feature + '*')\n",
    "    for col_name in heart_disease_attack_df_2017.columns:\n",
    "        if pattern.findall(col_name):\n",
    "            print(pattern.match(col_name))\n",
    "            \n",
    "print('The shape of the heart disease/heart attack dataframe is:', heart_disease_attack_df_2017.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<re.Match object; span=(0, 6), match='BPMEDS'>\n",
      "The shape of the hypertension dataframe is: (450016, 522)\n"
     ]
    }
   ],
   "source": [
    "# Hypertension\n",
    "hypertension_df_2017 = dummy_df_2017.copy(deep=True)\n",
    "\n",
    "for feature in hypertension_features:\n",
    "    pattern = re.compile(feature + '*')\n",
    "    for col_name in hypertension_df_2017.columns:\n",
    "        if pattern.findall(col_name):\n",
    "            print(pattern.match(col_name))\n",
    "            \n",
    "hypertension_df_2017 = hypertension_df_2017.drop(columns = ['BPMEDS_2.0'])\n",
    "\n",
    "print('The shape of the hypertension dataframe is:', hypertension_df_2017.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Part 1H: Apply SMOTe for imbalanced cases prior to modeling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ8AAAEPCAYAAACdhMnXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAHRJJREFUeJzt3X2QXXWd5/F356mNCwEtgiQiT2byJdpKZm9ASh6EEcfJ4i7rKjAmQ2SUYASk3Bl1dstkXCxcax4WJGgWiwfjVhDYBZkHSaypQde0MjCmFdkG84VlAxgSCsR1UyBpEtL7x/l1ctPm4fbDPd3Veb+qUt7zPb/f6d+xbvG5v3PO/d2O/v5+JEmq06SxHoAk6dBj+EiSamf4SJJqZ/hIkmpn+EiSamf4SJJqZ/hIkmpn+EiSamf4SJJqZ/hIkmpn+EiSajel3X8gIv4aOCozL42I+cAtwAxgPbAsM3dGxHHAGuBoIIHFmflSRBwJ3A6cBLwAXJSZz0XENOBWYAHwCrAoMzdGRAfwV8AHgF3A0sz8Uatj7enp6QROBbYCr43G+UvSBDcZmAX8uNFo9LXaqa3hExHvBT4K3FdKa4DLMvPBiLgVWAr8V2AVsCoz74yIFcAK4M+Aa4HuzDw/Ii4BbgAuBq4GXs7MeRFxNrAaOB34EDAPeBswB7gvIuZl5s4Wh3wq0D3S85akQ9BZwA9bbdy28ImINwJfAv4zcEpEHA9Mz8wHS5PVwDURcQtwNvBvm+o/oAqf88s+gDuAr0XE1FL/c4DMXB8RM8vs6XzgzszcBTweEc8A76aaZbViK8DcuXOZNm3asM5bkg4lr776Ko8//jiU/362qp0zn68DnwfeUrZns/fgtgLHAkcB25pmJwP1vfqUy3PbgJkHONb+6q16DWDatGl0dnYOoZskHfKGdKuiLeETEZcBv8jM+yPi0lKeBDT/eFAH1X2ZwXVKfaBNs/31OVh9SHp7e4faRZI0BO2a+VwMzIqIh4E3AodRhcKspjbHAFuA54EjImJyZr5W2mwpbZ4t7TZHxBTgcOBFYHNp9+SgYw3UB/+NIenq6nLmI0kt6OvrG9YH9rY8ap2Z78vMrsycT3Vv5u8y84+B7RFxRml2CbAuM3dQ3eS/uNSXAOvK67Vlm7K/u7TfXY+IM4HtmflMqS+OiMkRMQeYC/y4HecoSRq+tj9qPchi4OaImAH8BFhZ6lcA34yI5cAzwEdKfQWwOiIeBX5d+gPcCHy91PuoggzgbuBdwCNl++OZ+Uobz0eSNAwd/f2Db7ccunp6ek4ANnnZTZJa03TZ7cRGo/FUq/1c4UCSVDvDR5JUO8NnlO3asWOsh6BxyPeFtLe6HziY8CZNncqGq5eN9TA0zixYedNYD0EaV5z5SJJqZ/hIkmpn+EiSamf4SJJqZ/hIkmpn+EiSamf4SJJqZ/hIkmpn+EiSamf4SJJqZ/hIkmpn+EiSamf4SJJqZ/hIkmrX1p9UiIgvAh8G+oFbM/O6iPgGcCbwcml2TWbeGxHnAdcB04G7MnN5OcZ84BZgBrAeWJaZOyPiOGANcDSQwOLMfCkijgRuB04CXgAuyszn2nmekqShadvMJyLeA/we8E5gAfCpiIjy+uzMnF/+3RsR04HbgAuAecCpEbGwHGoNcFVmzgU6gKWlvgpYlZknAxuAFaV+LdCdmfOAm4Eb2nWOkqThaVv4ZOYPgHMzcyfV7GQK8ApwHHBbRDwSEddExCTgNOCJzNxU2q8BLoyI44HpmflgOezqUp8KnA3c3Vwvr8+nmvkA3AEsLO0lSeNEW+/5ZOaOiLgGeAy4H5gKfA/4GHA6cBbwcWA2sLWp61bg2APUjwK2laBqrtPcp+zfBswc7XOTJA1f239GOzO/EBF/Afw98N7M/ODAvoi4EVhCNYPpb+rWAeyiCsdW6pT6QJtmHU37WtLb2zuU5ntpNBrD7quJraenZ6yHII0bbQufiDgZeF1mPpyZv4mIbwMXR8SLmXlPadYB7AA2A7Oauh8DbDlA/XngiIiYnJmvlTZbSptnS7vNETEFOBx4cShj7+rqorOzcyhdpIPyg4kmor6+vmF9YG/nZbeTgJsjojMiplE9TPAD4CsR8YZyH+Zy4F7gISAiYk5ETAYWAesy82lge0ScUY55SanvALqBi0t9CbCuvF5btin7u0t7SdI40c4HDtYC9wE/BXqABzLzi8CXgR9R3Qd6ODPvyMztwKXAPaW+kT0PEywGro+IjcBhwMpSvwK4PCIeo7p3tLzUVwCnR8Sjpc2V7TpHSdLwdPT3D751cujq6ek5Adg00stuG65eNmpj0sSwYOVNYz0EqS2aLrud2Gg0nmq1nyscSJJqZ/hIkmpn+EiSamf4SJJqZ/hIkmpn+EiSamf4SJJqZ/hIkmpn+EiSamf4SJJqZ/hIkmpn+EiSamf4SJJqZ/hIkmpn+EiSamf4SJJqZ/hIkmpn+EiSajelnQePiC8CHwb6gVsz87qIOA+4DpgO3JWZy0vb+cAtwAxgPbAsM3dGxHHAGuBoIIHFmflSRBwJ3A6cBLwAXJSZz0XENOBWYAHwCrAoMze28zwlSUPTtplPRLwH+D3gnVRB8KmIOAW4DbgAmAecGhELS5c1wFWZORfoAJaW+ipgVWaeDGwAVpT6tUB3Zs4DbgZuKPWrgZdL/dPA6nadoyRpeNoWPpn5A+DczNxJNWuZAhwJPJGZm0p9DXBhRBwPTM/MB0v31aU+FTgbuLu5Xl6fTzXzAbgDWFja765n5npgZpk9SZLGibZedsvMHRFxDfAZ4H8As4GtTU22AsceoH4UsK0EVXOd5j7l8tw2YOYBjvVMq+Pu7e1ttelvaTQaw+6ria2np2eshyCNG20NH4DM/EJE/AXw98Bcqvs/AzqAXVQzsFbqlPpAm2YHO1bLurq66OzsHEoX6aD8YKKJqK+vb1gf2Nt5z+fk8hABmfkb4NvAOcCspmbHAFuAzfupPw8cERGTS31WqQM8W9oREVOAw4EXD3AsSdI40c5HrU8Cbo6IzvIE2gXA14GIiDklUBYB6zLzaWB7RJxR+l5S6juAbuDiUl8CrCuv15Ztyv7u0n53PSLOBLZnZsuX3CRJ7dfOBw7WAvcBPwV6gAcy807gUuAe4DFgI3seJlgMXB8RG4HDgJWlfgVweUQ8BpwFLC/1FcDpEfFoaXNlqd8IdJb6SqogkySNIx39/YNvqRy6enp6TgA2jfSez4arl43amDQxLFh501gPQWqLpns+JzYajada7ecKB5Kk2hk+kqTaGT6SpNoZPpKk2hk+kqTaGT6SpNoZPpKk2hk+kqTaGT6SpNoZPpKk2hk+kqTaGT6SpNoZPpKk2hk+kqTaGT6SpNoZPpKk2hk+kqTaGT6SpNpNaefBI+ILwEVl877M/FxEfAM4E3i51K/JzHsj4jzgOmA6cFdmLi/HmA/cAswA1gPLMnNnRBwHrAGOBhJYnJkvRcSRwO3AScALwEWZ+Vw7z1OSNDRtm/mUMPl94HeB+UAjIj4ILADOzsz55d+9ETEduA24AJgHnBoRC8uh1gBXZeZcoANYWuqrgFWZeTKwAVhR6tcC3Zk5D7gZuKFd5yhJGp52XnbbCvxpZr6amTuAnwPHlX+3RcQjEXFNREwCTgOeyMxNmbmTKnAujIjjgemZ+WA55upSnwqcDdzdXC+vz6ea+QDcASws7SVJ40TbLrtl5qMDryPid6guv50FnANcAfw/4DvAx4GXqMJqwFbgWGD2fupHAdtKUDXXae5TLs9tA2YCW1ode29vb6tNf0uj0Rh2X01sPT09Yz0Eadxo6z0fgIh4O3Af8NnMTOCDTftuBJZQzWD6m7p1ALuoZmat1Cn1gTbNOpr2taSrq4vOzs6hdJEOyg8mmoj6+vqG9YG9rU+7RcQZwP3Af8jMb0bEOyLiQ01NOoAdwGZgVlP9GKqZyv7qzwNHRMTkUp/FnpnNs6UdETEFOBx4cTTPS5I0Mu184OAtwN8AizLzzlLuAL4SEW8o92EuB+4FHqq6xJwSKIuAdZn5NLC9hBjAJaW+A+gGLi71JcC68npt2abs7y7tJUnjRDsvu30GeB1wXUQM1G4Cvgz8CJgK3JOZdwBExKXAPaXPWvY8TLAYuDkiZgA/AVaW+hXANyNiOfAM8JFSXwGsjohHgV+X/pKkcaSjv3/wrZNDV09PzwnAppHe89lw9bJRG5MmhgUrbxrrIUht0XTP58RGo/FUq/1c4UCSVDvDR5JUO8NHklQ7w0eSVDvDR5JUO8NHklQ7w0eSVLuWwici3ryP2ttGfziSpEPBAVc4iIg3lpdrI+Ic9izaORX4NnBy+4YmSZqoDra8zh3A+8rr5sU5d7Jn+RtJkobkgOGTme8HiIjbMvNj9QxJkjTRtbSwaGZ+rPyq6Btp+r2czPxJuwYmSZq4WgqfiLgG+CzV7+gMrETaD5zUpnFJkiawVn9SYQkwJzNb/ilqSZL2p9Xv+fzC4JEkjZZWZz73R8RfAn8LvDJQ9J6PJGk4Wg2fS8v/XthU856PJGlYWn3a7cThHDwivgBcVDbvy8zPRcR5wHXAdOCuzFxe2s4HbgFmAOuBZZm5MyKOA9YARwMJLM7MlyLiSOB2qgB8AbgoM5+LiGnArcACqlnaoszcOJzxS5Lao9Xldf5kX/8O0uc84PeB3wXmA42I+AhwG3ABMA84NSIWli5rgKsycy7V49xLS30VsCozTwY2ACtK/VqgOzPnATcDN5T61cDLpf5pYHUr5yhJqk+rDxy8o+lfA/gT4JSD9NkK/GlmvpqZO4CfA3OBJzJzU2bupAqcC8t3iKZn5oOl7+pSnwqczZ7VFFaz59Lf+VQzH6hWYlhY2u+uZ+Z6YGaZPUmSxolWL7v9cfN2RMymurR1oD6PNrX/HarLbzdShdKArcCxwOz91I8CtpWgaq7T3KdcntsGzDzAsZ452HlKkurR6gMHe8nMLRFxQittI+LtwH1UX1LdSTX7GdAB7KKagfW3UKfUB9o0O9ixWtbb2zuU5ntpNBrD7quJraenZ6yHII0bra5w0Hx/p4PqZv7zLfQ7A7gH+HRm3hkR7wFmNTU5BtgCbN5P/XngiIiYnJmvlTYD3zd6trTbHBFTgMOpFj8dONaTg47Vsq6uLjo7O4fSRTooP5hoIurr6xvWB/bh3PN5O9UlrMUH6hARbwH+huppsztL+aFqV8yJiMnAImBdZj4NbC9hBXBJqe8AuoGLS30JsK68Xlu2Kfu7S/vd9Yg4E9iemV5yk6RxZEj3fMqDAVMz83+30O0zwOuA6yJioHYT1XeG7in71rLnYYLFwM0RMQP4CbCy1K8AvhkRy6lC7yOlvgJYHRGPAr9mTxjeCHy91PuogkySNI509PcPvqXy2yJiDtXqBrOpZku/BD6QmT9v7/Dq1dPTcwKwaaSX3TZcvWzUxqSJYcHKm8Z6CFJbNF12O7HRaDzVar9WL7t9FfjLzHxDZh5B9R2brw15lJIk0Xr4vCkzvzmwkZnfoHqsWZKkIWs1fKZExBsHNiLiKH77EWhJklrS6vd8bgQejIi7qELnD4Hr2zYqSdKE1urMZy1V6EwD3ga8Gbi3XYOSJE1srYbPauBrmflnwB8Bn6daIFSSpCFrNXyOysyVAJm5PTO/wt4rEkiS1LKhPHAwe2AjIt7Eb6+tJklSS1p94OA64OGI+C7VvZ/zqBYKlSRpyFqa+WTmbVSB81OqH3R7f2Z+q50DkyRNXC3/pEJmPgI80saxSJIOEa3e85EkadQYPpKk2hk+kqTaGT6SpNoZPpKk2hk+kqTaGT6SpNq1/D2f4YqIGcADVD+7/VREfAM4E3i5NLkmM++NiPOoVlKYDtyVmctL//nALcAMYD2wLDN3RsRxwBrgaCCBxZn5UkQcCdwOnAS8AFyUmc+1+zwlSa1r68wnIt4F/BCY21ReAJydmfPLv3sjYjrVKtkXAPOAUyNiYWm/BrgqM+dSrSe3tNRXAasy82SqVRdWlPq1QHdmzgNuBm5o3xlKkoaj3ZfdlgJXAlsAIuL1wHHAbRHxSERcExGTgNOAJzJzU2bupAqcCyPieGB6Zj5Yjre61KcCZwN3N9fL6/OpZj4AdwALS3tJ0jjR1vDJzMsys7updAzwPeBjwOnAWcDHgdnA1qZ2W4FjD1A/CthWgqq5TnOfsn8bMHP0zkqSNFJtv+fTLDP/D/DBge2IuBFYQjWD6W9q2gHsogrHVuqU+kCbZh1N+1rS29s7lOZ7aTQaw+6ria2np2eshyCNG7WGT0S8A5ibmfeUUgewA9jM3j9OdwzVpbr91Z8HjoiIyZn5WmmzpbR5trTbHBFTgMOBF4cyzq6uLjo7O4fSRTooP5hoIurr6xvWB/a6H7XuAL4SEW8o92EuB+4FHgIiIuZExGRgEbAuM58GtkfEGaX/JaW+A+gGLi71JcC68npt2abs7y7tJUnjRK3hU36W4cvAj4DHgIcz847M3A5cCtxT6hvZ8zDBYuD6iNgIHAasLPUrgMsj4jGqe0fLS30FcHpEPFraXNnu85IkDU1Hf//gWyeHrp6enhOATSO97Lbh6mWjNiZNDAtW3jTWQ5Daoumy24mNRuOpVvu5woEkqXaGjySpdoaPJKl2ho8kqXaGjySpdoaPJKl2ho8kqXaGjySpdoaPJKl2ho8kqXaGjySpdoaPJKl2ho8kqXaGjySpdoaPJKl2ho8kqXaGjySpdoaPJKl2U9r9ByJiBvAA8IHMfCoizgOuA6YDd2Xm8tJuPnALMANYDyzLzJ0RcRywBjgaSGBxZr4UEUcCtwMnAS8AF2XmcxExDbgVWAC8AizKzI3tPk9JUuvaOvOJiHcBPwTmlu3pwG3ABcA84NSIWFiarwGuysy5QAewtNRXAasy82RgA7Ci1K8FujNzHnAzcEOpXw28XOqfBla37QQlScPS7stuS4ErgS1l+zTgiczclJk7qQLnwog4HpiemQ+WdqtLfSpwNnB3c728Pp9q5gNwB7CwtN9dz8z1wMwye5IkjRNtDZ/MvCwzu5tKs4GtTdtbgWMPUD8K2FaCqrm+17HK/m3AzAMcS5I0TrT9ns8gk4D+pu0OYNcQ6pT6QJtmBztWy3p7e4fSfC+NRmPYfTWx9fT0jPUQpHGj7vDZDMxq2j6G6pLc/urPA0dExOTMfK20GbiE92xptzkipgCHAy82HevJQcdqWVdXF52dnUPpIh2UH0w0EfX19Q3rA3vdj1o/BEREzImIycAiYF1mPg1sj4gzSrtLSn0H0A1cXOpLgHXl9dqyTdnfXdrvrkfEmcD2zHymzeclSRqCWsMnM7cDlwL3AI8BG9nzMMFi4PqI2AgcBqws9SuAyyPiMeAsYHmprwBOj4hHS5srS/1GoLPUV1IFmSRpHKnlsltmntD0+n7glH20+RnV03CD608D5+yj/ivg3+yjvh346IgGLElqK1c4kCTVzvCRJNXO8JEk1c7wkSTVzvCRJNXO8JEk1c7wkSTVzvCRJNXO8JEk1c7wkSTVzvCRJNXO8JEk1c7wkSTVzvCRJNXO8JEOITt2DekX5XWIGIv3Rd0/oy1pDE2dNIllD2wY62FonLnp3Qtq/5vOfCRJtTN8JEm1G5PLbhHxfeBoYEcpfQJ4K7AcmAp8JTO/VtqeB1wHTAfuyszlpT4fuAWYAawHlmXmzog4DlhTjp/A4sx8qa5zkyQdXO0zn4joAOYCp2Tm/MycD2wGvgScCcwHLo+It0XEdOA24AJgHnBqRCwsh1oDXJWZc4EOYGmprwJWZebJwAZgRU2nJklq0Vhcdovyv/8QET+LiKuA84DvZeavMvNl4G7gw8BpwBOZuSkzd1IFzoURcTwwPTMfLMdaXepTgbNL/931Ok5KktS6sbjs9gbgfuBTVJfY/idwF7C1qc1WquCZvY/6sQeoHwVsK0HVXB+S3t7eoXbZrdFoDLuvJraenp6xHoLvT+1X3e/P2sMnM/8J+KeB7Yi4leqezrVNzTqAXVQzs/4R1Cn1Ienq6qKzs3Oo3aQD8j/8Gs+G+/7s6+sb1gf2sbjnc2ZEvLep1AE8Bcxqqh0DbKG6FzSU+vPAERExudRnlbokaRwZi3s+RwJ/FRGvi4jDgY8CfwS8NyJmRsTrgQ8B3wUeAiIi5pRAWQSsy8ynge0RcUY55iWlvgPoBi4u9SXAutrOTJLUktrDJzO/A9wH/BToAW7LzB8Bnwe+DzwMfCsz/zkztwOXAvcAjwEb2fMwwWLg+ojYCBwGrCz1K6ielnsMOIvq8W1J0jgyJt/zycwVDHoEOjO/BXxrH23vB07ZR/1nVA8lDK4/DZwzWmOVJI0+VziQJNXO8JEk1c7wkSTVzvCRJNXO8JEk1c7wkSTVzvCRJNXO8JEk1c7wkSTVzvCRJNXO8JEk1c7wkSTVzvCRJNXO8JEk1c7wkSTVzvCRJNXO8JEk1c7wkSTVbkx+RrvdImIRsByYCnwlM782xkOSJDWZcDOfiHgz8CXgTGA+cHlEvG1sRyVJajYRZz7nAd/LzF8BRMTdwIeBL7bQdzLAq6++OrIR/IvDRtZfE05fX99YD2E3350abCTvz6b/Xk4eSr+JGD6zga1N21uB01rsOwvg8ccfH9EAOj78kRH118TT29s71kPY7SOdHWM9BI0zo/T+nAU82WrjiRg+k4D+pu0OYFeLfX8MnEUVWK+N8rgkaSKaTBU8Px5Kp4kYPpupAmTAMcCWVjo2Go0+4IftGJQkTWAtz3gGTMTw+UfgP0XETOBl4EPA5WM7JElSswn3tFtmPgt8Hvg+8DDwrcz857EdlSSpWUd/f//BW0mSNIom3MxHkjT+GT6SpNoZPpKk2hk+kqTaTcRHrVWzgy3kGhHzgVuAGcB6YFlm7qx9oDokRcQM4AHgA5n51KB9vjfHiDMfjUiLC7muAa7KzLlUK04srXeUOlRFxLuovjg+dz9NfG+OEcNHI7V7IdfMfBkYWMgVgIg4HpiemQ+W0mrgwtpHqUPVUuBK9rHKie/NseVlN43UwRZy3df+Y2sYl0RmXgYQEfva7XtzDDnz0UgdbCHXkSz0KrWT780xZPhopDZTfoqiGLyQ68H2S2PF9+YYMnw0Uv8IvDciZkbE66kWcv3uwM7MfBrYHhFnlNIlwLr6hyntzffm2DJ8NCL7W8g1ItZGxILSbDFwfURspPohzZVjM1oJfG+ODy4sKkmqnTMfSVLtDB9JUu0MH0lS7QwfSVLtDB9JUu1cXkcapog4AXgS+F+lNAl4iWpl7/8eEecAX83MrqY+RwG/AFZn5ieb6udQfcckS2kK8Evgk5n58338rWanAd8C5pTtU4Be4DXg/2bmuRHxFNAHvDKo7xWZ+UBEXEi1MnlH+bufyMwnhvB/hzQkho80Mq9k5vyBjbJY5f0R8Rrw4j7afxz4W2BRRHw+M3/VtO/JQcf6HNX3Tt63r781SPNirv3AuZn5y0FtFmfmhsEdI+IY4OvAOzNzc0RcBXwVeP9+/pY0Yl52k0ZR+db8nwOfHbwvIiYBn6BaPbkbuHx/x4mIDuCN7L3wZVtk5nPAm0rwTAGOZ9/BKY0aZz7S6PsZ8I591P8AeD3VkkSHU32z/q+bfrzsrRHxcHn9ptL23Kb+05v2D/hRZl7Z4rhuj4jmy259mfkugMzcUb71/x1gehmr1DaGjzT6+oHf7KP+SeD2zNwZEX9HdanrQuCOsn/wZbclwD9ExImldKDLbq3Y52W3AWXfMRHxB8B9EXFSZv56BH9P2i8vu0mj71QGPRhQ7gX9K+APy83/pPrw9+/3d5DM/G+lzeBfhh1VETE7Inbf38nM7wLbgLe28+/q0Gb4SKMoIuYCK4D/MmjXJ4AfZuabM/OEzDwBaAD/MiLevZ9jDay2nPvaP4peB9wZEXPK3z2XKvR+3ua/q0OYl92kkWm+D7ML2A78x8y8rzw+TURMo3rK7WPNHTPziYi4g2r28zX2vuczierR6H+Xmb+OiCPZ9z0fgEszc1/1wQbf84HqUfBbIuIy4J7ypNyvgX+dmfu6dCiNCle1liTVzstukqTaGT6SpNoZPpKk2hk+kqTaGT6SpNoZPpKk2hk+kqTaGT6SpNr9f9zgKsN8Pb+hAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Plot showing that Diabetes group frequencies are unbalanced before SMOTe\n",
    "\n",
    "sns.countplot(x='DIABETE3', data=diabetes_df_2017, palette='hls')\n",
    "plt.show\n",
    "plt.savefig('Diabetes_Count_Plot_Before_SMOTe')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ8AAAEPCAYAAACdhMnXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAHIVJREFUeJzt3X+MXeV95/H3YJvBKRgS2QQTan4s9Rcnk8bRNQlbfiRpaCMvaFnEDwt7cWgSsxZQlG2SqlLMprCkUdrKBNO4RAbHtCaEFYTsttjRqiQbOyHQ+CYEDdRf2KyBGpuFJZu1oHjwj9k/zjP2tYPtOzO+507G75dkce/3PM+Z50hXfO5znnPP6RkcHESSpDod1e0BSJKOPIaPJKl2ho8kqXaGjySpdoaPJKl2ho8kqXaGjySpdoaPJKl2ho8kqXaGjySpdoaPJKl2E7s9gLGk2Wz2AmcDW4FdXR6OJP06mABMB37caDQG2u1k+OzrbGB9twchSb+Gzgd+0G5jw2dfWwFmzpzJ0Ucf3e2xSNKY9+abb/LMM89A+f9nuwyffe0COProo+nt7e32WCTp18mwliq84ECSVDvDR5JUO8NHklQ7w0eSVDvDR5JUO8NHklQ7w0eSVDvD5zDbvWNHt4egMcjPhbQvf2R6mB01aRIbblzc7WFojJmz7M5uD0EaUzoePhHxl8DUzLwmImYDdwFTgHXA4szcGREzgNXAiUACCzLztYg4AbgXOAN4BbgyM1+KiKOBu4E5wBvA/MzcGBE9wF8AFwO7gUWZ+cNOH6MkaXg6etotIj4KfLyltBq4ITNnAj3AolJfDizPzLOADcBNpX4rsD4zZwErgNtL/Ubg9VL/NLCq1C8DZgHvBv4dsCoinN1J0hjTsfCJiHcAXwT+rLw/FZicmY+VJquAKyJiEnAB8EBrvby+iGrmA3AfMLe031PPzHXAtDJ7ugj4ZmbuzsxngBeA3+nUMUqSRqaTM5+vAZ8H/m95fzL73vV0K3AKMBXYlpk796vv06ds3wZMO8i+DlSXJI0hHTklFRGfAv45Mx+JiGtK+ShgsKVZD9W6zP51Sn2oTasD9TlUfVj6+/uH22WPRqMx4r4a35rNZreHII0ZnVoPmQdMj4gngHcAx1KFwvSWNicBW4CXgeMjYkJm7ipttpQ2L5Z2m8vazXHAq8Dm0u7n++1rqL7/3xiWvr4+H6mgw84vJhqPBgYGRvSFvSOn3TLz9zKzLzNnA/8J+G+Z+QfA9og4tzS7GlibmTuonh46r9QXAmvL6zXlPWX7+tJ+Tz0izgO2Z+YLpb4gIiZExJnATODHnThGSdLI1X0l2AJgRURMAX4CLCv164B7ImIJ1UUCV5X6TVRXrD0F/LL0B7gD+FqpD1AFGVQXLXwQeLK8/2RmvtHB45EkjUDP4OD+yy1HrmazeRqwabSn3fyRqfbnj0w1XrWcdju90Wg8124/b68jSaqd4SNJqp3hI0mqneEjSaqd4SNJqp3hI0mqneEjSaqd4SNJqp3hI0mqneEjSaqd4SNJqp3hI0mqneEjSaqd4SNJqp3hI0mqneEjSapdR59kGhG3AJcDg8Ddmbk0Ir4OnAe8XprdnJkPRcSFwFJgMnB/Zi4p+5gN3AVMAdYBizNzZ0TMAFYDJwIJLMjM1yLiBOBe4AzgFeDKzHypk8cpSRqejs18IuJDwO8Cvw3MAf4wIqK8viAzZ5d/D0XEZGAlcAkwCzg7IuaWXa0GbsjMmUAPsKjUlwPLM/MsYAPVI7cBbgXWZ+YsYAVwe6eOUZI0Mh0Ln8z8PvCRzNxJNTuZCLwBzABWRsSTEXFzRBwFfAB4NjM3lfargSsi4lRgcmY+Vna7qtQnARcAD7TWy+uLqGY+APcBc0t7SdIY0dE1n8zcERE3A08DjwCTgO8CnwDOAc4HPgmcDGxt6boVOOUg9anAthJUrXVa+5Tt24Bph/vYJEkj19E1H4DM/EJEfBn4O+CjmXnp0LaIuANYSDWDGWzp1gPspgrHduqU+lCbVj0t29rS398/nOb7aDQaI+6r8a3ZbHZ7CNKY0bHwiYizgGMy84nM/JeI+BYwLyJezcwHS7MeYAewGZje0v0kYMtB6i8Dx0fEhMzcVdpsKW1eLO02R8RE4Djg1eGMva+vj97e3uF0kQ7JLyYajwYGBkb0hb2Tp93OAFZERG9EHE11McH3ga9ExNvLOsy1wEPA40BExJkRMQGYD6zNzOeB7RFxbtnn1aW+A1gPzCv1hcDa8npNeU/Zvr60lySNEZ284GAN8DDwU6AJPJqZtwBfAn5ItQ70RGbel5nbgWuAB0t9I3svJlgA3BYRG4FjgWWlfh1wbUQ8TbV2tKTUbwLOiYinSpvrO3WMkqSR6Rkc3H/p5MjVbDZPAzaN9rTbhhsXH7YxaXyYs+zObg9B6oiW026nNxqN59rt5x0OJEm1M3wkSbUzfCRJtTN8JEm1M3wkSbUzfCRJtTN8JEm1M3wkSbUzfCRJtTN8JEm1M3wkSbUzfCRJtTN8JEm1M3wkSbUzfCRJtTN8JEm1m9jJnUfELcDlwCBwd2YujYgLgaXAZOD+zFxS2s4G7gKmAOuAxZm5MyJmAKuBE4EEFmTmaxFxAnAv1eO6XwGuzMyXyiO77wbmAG8A8zNzYyePU5I0PB2b+UTEh4DfBX6bKgj+MCLeB6wELgFmAWdHxNzSZTVwQ2bOBHqARaW+HFiemWcBG6gekw1wK7A+M2cBK4DbS/1G4PVS/zSwqlPHKEkamY6FT2Z+H/hIZu6kmrVMBE4Ans3MTaW+GrgiIk4FJmfmY6X7qlKfBFwAPNBaL68vopr5ANwHzC3t99Qzcx0wrcyeJEljREfXfDJzR0TcDDwNPAKcDGxtabIVOOUg9anAthJUrXVa+5Tt24BpB9mXJGmM6OiaD0BmfiEivgz8HTCTav1nSA+wmyoE26lT6kNtWh1qX23r7+8fTvN9NBqNEffV+NZsNrs9BGnM6Fj4RMRZwDGZ+URm/ktEfIvq4oNdLc1OArYAm4Hpb1F/GTg+IiZk5q7SZktp82JptzkiJgLHAa+27Ovn++2rbX19ffT29g6ni3RIfjHReDQwMDCiL+ydPO12BrAiInrLFWiXAF8DIiLOjIgJwHxgbWY+D2yPiHNL36tLfQewHphX6guBteX1mvKesn19ab+nHhHnAdsz84UOHqckaZg6ecHBGuBh4KdAE3g0M78JXAM8SLUOtJG9FxMsAG6LiI3AscCyUr8OuDYingbOB5aU+k3AORHxVGlzfanfAfSW+jKqIJMkjSE9g4P7L6kcuZrN5mnAptGedttw4+LDNiaND3OW3dntIUgd0XLa7fRGo/Fcu/28w4EkqXaGjySpdoaPJKl2ho8kqXaGjySpdoaPJKl2ho8kqXaGjySpdoaPJKl2ho8kqXaGjySpdoaPJKl2ho8kqXaGjySpdoaPJKl2ho8kqXaGjySpdhM7ufOI+AJwZXn7cGb+cUR8HTgPeL3Ub87MhyLiQmApMBm4PzOXlH3MBu4CpgDrgMWZuTMiZgCrgROBBBZk5msRcQJwL3AG8ApwZWa+1MnjlCQNT8dmPiVMfh94PzAbaETEpcAc4ILMnF3+PRQRk4GVwCXALODsiJhbdrUauCEzZwI9wKJSXw4sz8yzgA3ATaV+K7A+M2cBK4DbO3WMkqSR6eRpt63AZzLzzczcAfwTMKP8WxkRT0bEzRFxFPAB4NnM3JSZO6kC54qIOBWYnJmPlX2uKvVJwAXAA6318voiqpkPwH3A3NJekjRGdOy0W2Y+NfQ6In6L6vTb+cCHgeuA/wf8PfBJ4DWqsBqyFTgFOPkA9anAthJUrXVa+5TTc9uAacCWdsfe39/fbtNf0Wg0RtxX41uz2ez2EKQxo6NrPgAR8R7gYeBzmZnApS3b7gAWUs1gBlu69QC7qWZm7dQp9aE2rXpatrWlr6+P3t7e4XSRDskvJhqPBgYGRvSFvaNXu0XEucAjwJ9k5j0R8d6IuKylSQ+wA9gMTG+pn0Q1UzlQ/WXg+IiYUOrT2TuzebG0IyImAscBrx7O45IkjU4nLzj4TeDbwPzM/GYp9wBfiYi3l3WYa4GHgMerLnFmCZT5wNrMfB7YXkIM4OpS3wGsB+aV+kJgbXm9prynbF9f2kuSxohOnnb7LHAMsDQihmp3Al8CfghMAh7MzPsAIuIa4MHSZw17LyZYAKyIiCnAT4BlpX4dcE9ELAFeAK4q9ZuAVRHxFPDL0l+SNIb0DA7uv3TyqyLiXZn54n61d2fm0x0bWRc0m83TgE2jXfPZcOPiwzYmjQ9zlt3Z7SFIHdGy5nN6o9F4rt1+B535RMQ7yss1EfFh9i7mTwK+BZw17JFKko54hzrtdh/we+V166L9TvaeFpMkaVgOGj6Z+TGAiFiZmZ+oZ0iSpPGurQsOMvMT5W4D76DldzSZ+ZNODUySNH61FT4RcTPwOarf1wxdoTBIdfNOSZKGpd1LrRcCZ2Zm27eokSTpQNr9kek/GzySpMOl3ZnPIxHx58B/Bd4YKrrmI0kaiXbD55ry3ytaaq75SJJGpN2r3U7v9EAkSUeOdq92+6O3qmfm0sM7HEnSkaDd027vbXl9NPAhqkclSJI0bO2edvuD1vcRcTJwd0dGJEka90b0PJ9y2fVph3cokqQjxUjWfHqAOVR3O5AkadhGsuYzSPXwts8d/uFIko4Ew1rzKTcXnZSZ/7OdfhHxBeDK8vbhzPzjiLgQWApMBu7PzCWl7WzgLmAKsA5YnJk7I2IGsBo4EUhgQWa+FhEnAPdS/dboFeDKzHwpIo6mWo+aQ/WD2PmZubGd8UqS6tHWmk9EnFkeS/0E0IyIn0fErEP0uRD4feD9wGygERFXASuBS4BZwNkRMbd0WQ3ckJkzqU7tLSr15cDyzDwL2ED1mGyAW4H1mTkLWAHcXuo3Aq+X+qeBVe0coySpPu1ecPBXwJ9n5tsz83iq//F/9RB9tgKfycw3M3MH8E/ATODZzNyUmTupAueKMqOanJmPlb6rSn0ScAF7H1y3ir13WbiIauYD1UPv5pb2e+qZuQ6YVmZPkqQxot01n3dm5j1DbzLz6wf64WlLm6eGXkfEb1GdfruDKpSGbAVOAU4+QH0qsK0EVWud1j7l9Nw2YNpB9vVCW0cKQ88jH5FGozHivhrfms1mt4cgjRnths/EiHhHZv4CICKmsve5PgcVEe8BHqa6QGEn1exnSA+wm2oGNthGnVIfatPqUPtqW19fH729vcPpIh2SX0w0Hg0MDIzoC3u7p93uAB6LiP8cEbcAjwJ/fahOEXEu1Z0Q/qTMnDYD01uanARsOUj9ZeD4iJhQ6tNLHeDF0o6ImAgcB7x6kH1JksaIdsNnDdVs4mjg3cC7gIcO1iEifhP4NtXVZt8s5cerTXFmCZT5wNrMfB7YXsIK4OpS3wGsB+aV+kJgbcuYFpbX86guPtjRWo+I84Dtmdn2KTdJUue1e9ptFfDVzFwWEccAi6muWvs3B+nzWeAYYGlEDNXupHo8w4Nl2xr2XkywAFgREVOAnwDLSv064J6IWEK1bnNVqd8ErCpX4f2y9Idqlva1Uh+gCjJJ0hjSMzh46KWbiHgiM2fvV/tpZr6/YyPrgmazeRqwabRrPhtuXHzYxqTxYc6yO7s9BKkjWtZ8Tm80Gs+126/d024Ty81EAYiId/KrC/6SJLWl3dNuS4EnIuI7VGs/F+LtdSRJI9TWzCczV1IFzk+p7jLwscz8RicHJkkav9qd+ZCZTwJPdnAskqQjxIie5yNJ0mgYPpKk2hk+kqTaGT6SpNoZPpKk2hk+kqTaGT6SpNoZPpKk2hk+kqTaGT6SpNoZPpKk2hk+kqTatX1j0ZEqTyZ9FLg4M5+LiK8D5wGvlyY3Z+ZDEXEh1aMbJgP3Z+aS0n82cBcwBVgHLM7MnRExA1gNnAgksCAzX4uIE4B7gTOAV4ArM/OlTh+nJKl9HZ35RMQHgR8AM1vKc4ALMnN2+fdQREymeiz3JcAs4OyImFvarwZuyMyZVA+wW1Tqy4HlmXkW1WMebir1W4H1mTkLWAHc3rkjlCSNRKdPuy0Crge2AETE24AZwMqIeDIibo6Io4APAM9m5qbM3EkVOFdExKnA5Mx8rOxvValPAi4AHmitl9cXUc18AO4D5pb2kqQxoqPhk5mfysz1LaWTgO8CnwDOAc4HPgmcDGxtabcVOOUg9anAthJUrXVa+5Tt24Bph++oJEmj1fE1n1aZ+b+AS4feR8QdwEKqGcxgS9MeYDdVOLZTp9SH2rTqadnWlv7+/uE030ej0RhxX41vzWaz20OQxoxawyci3gvMzMwHS6kH2AFsBqa3ND2J6lTdgeovA8dHxITM3FXabCltXiztNkfEROA44NXhjLOvr4/e3t7hdJEOyS8mGo8GBgZG9IW97kute4CvRMTbyzrMtcBDwONARMSZETEBmA+szcznge0RcW7pf3Wp7wDWA/NKfSGwtrxeU95Ttq8v7SVJY0St4ZOZTwJfAn4IPA08kZn3ZeZ24BrgwVLfyN6LCRYAt0XERuBYYFmpXwdcGxFPU60dLSn1m4BzIuKp0ub6Th+XJGl4egYH9186OXI1m83TgE2jPe224cbFh21MGh/mLLuz20OQOqLltNvpjUbjuXb7eYcDSVLtDB9JUu0MH0lS7QwfSVLtDB9JUu0MH0lS7QwfSVLtDB9JUu0MH0lS7QwfSVLtDB9JUu0MH0lS7QwfSVLtDB9JUu0MH0lS7QwfSVLtJnb6D0TEFOBR4OLMfC4iLgSWApOB+zNzSWk3G7gLmAKsAxZn5s6ImAGsBk4EEliQma9FxAnAvcAZwCvAlZn5UkQcDdwNzAHeAOZn5sZOH6ckqX0dnflExAeBHwAzy/vJwErgEmAWcHZEzC3NVwM3ZOZMoAdYVOrLgeWZeRawgeox2QC3AuszcxawAri91G8EXi/1TwOrOnaAkqQR6fRpt0XA9cCW8v4DwLOZuSkzd1IFzhURcSowOTMfK+1Wlfok4ALggdZ6eX0R1cwH4D5gbmm/p56Z64BpZfYkSRojOho+mfmpzFzfUjoZ2NryfitwykHqU4FtJaha6/vsq2zfBkw7yL4kSWNEx9d89nMUMNjyvgfYPYw6pT7UptWh9tW2/v7+4TTfR6PRGHFfjW/NZrPbQ5DGjLrDZzMwveX9SVSn5A5Ufxk4PiImZOau0mboFN6Lpd3miJgIHAe82rKvn++3r7b19fXR29s7nC7SIfnFROPRwMDAiL6w132p9eNARMSZETEBmA+szcznge0RcW5pd3Wp7wDWA/NKfSGwtrxeU95Ttq8v7ffUI+I8YHtmvtDh45IkDUOt4ZOZ24FrgAeBp4GN7L2YYAFwW0RsBI4FlpX6dcC1EfE0cD6wpNRvAs6JiKdKm+tL/Q6gt9SXUQWZJGkMqeW0W2ae1vL6EeB9b9HmZ1RXw+1ffx748FvUfwH827eobwc+PqoBS5I6yjscSJJqZ/hIkmpn+EiSamf4SJJqZ/hIkmpn+EiSamf4SJJqZ/hIkmpn+EiSamf4SJJqZ/hIkmpn+EiSamf4SJJqZ/hIkmpn+EiSamf4SJJqV8vD5PYXEd8DTgR2lNJ/AP4V1VNKJwFfycyvlrYXAkuBycD9mbmk1GcDdwFTgHXA4szcGREzgNVl/wksyMzX6jo2SdKh1T7ziYgeYCbwvsycnZmzgc3AF4HzgNlUj81+d0RMBlYClwCzgLMjYm7Z1WrghsycCfQAi0p9ObA8M88CNlA9bluSNIZ047RblP/+94j4WUTcAFwIfDczf5GZrwMPAJdTPVb72czclJk7qQLniog4FZicmY+Vfa0q9UnABaX/nnodByVJal83wuftwCPApcBHgcXADGBrS5utwCnAycOsTwW2laBqrUuSxpDa13wy80fAj4beR8TdVGs6t7Y06wF2U4Xj4CjqlPqw9Pf3D7fLHo1GY8R9Nb41m81uD0EaM2oPn4g4D+jNzEdKqQd4Dpje0uwkYAvVWtBw6i8Dx0fEhMzcVdpsGe4Y+/r66O3tHW436aD8YqLxaGBgYERf2Ltx2u0E4C8i4piIOA74OPDvgY9GxLSIeBtwGfAd4HEgIuLMiJgAzAfWZubzwPaIOLfs8+pS3wGsB+aV+kJgbW1HJklqS+3hk5l/DzwM/BRoAisz84fA54HvAU8A38jMf8zM7cA1wIPA08BG9l5MsAC4LSI2AscCy0r9Oqqr5Z4Gzqe6fFuSNIb0DA7uv0Ry5Go2m6cBm0Z72m3DjYsP25g0PsxZdme3hyB1RMtpt9MbjcZz7fbzDgfSEWTH7mFff6MjQDc+F125w4Gk7ph01FEsfnRDt4ehMebO35lT+9905iNJqp3hI0mqneEjSaqd4SNJqp3hI0mqneEjSaqd4SNJqp3hI0mqneEjSaqd4SNJqp3hI0mqneEjSaqd4SNJqp3hI0mqneEjSarduHyeT0TMp3p89iTgK5n51S4PSZLUYtzNfCLiXcAXgfOA2cC1EfHu7o5KktRqPM58LgS+m5m/AIiIB4DLgVva6DsB4M033xzdCH7j2NH117gzMDDQ7SHs4adT+xvN57Pl/5cThtNvPIbPycDWlvdbgQ+02Xc6wDPPPDOqAfRcftWo+mv86e/v7/YQ9riqt6fbQ9AYc5g+n9OBn7fbeDyGz1HAYMv7HmB3m31/DJxPFVi7DvO4JGk8mkAVPD8eTqfxGD6bqQJkyEnAlnY6NhqNAeAHnRiUJI1jbc94hozH8PkH4E8jYhrwOnAZcG13hyRJajXurnbLzBeBzwPfA54AvpGZ/9jdUUmSWvUMDg4eupUkSYfRuJv5SJLGPsNHklQ7w0eSVDvDR5JUu/F4qbVqdqgbuUbEbOAuYAqwDlicmTtrH6iOSBExBXgUuDgzn9tvm5/NLnHmo1Fp80auq4EbMnMm1R0nFtU7Sh2pIuKDVD8cn3mAJn42u8Tw0WjtuZFrZr4ODN3IFYCIOBWYnJmPldIq4IraR6kj1SLget7iLid+NrvL024arUPdyPWttp9Sw7gkMvNTABHxVpv9bHaRMx+N1qFu5DqaG71KneRns4sMH43WZsqjKIr9b+R6qO1St/jZ7CLDR6P1D8BHI2JaRLyN6kau3xnamJnPA9sj4txSuhpYW/8wpX352ewuw0ejcqAbuUbEmoiYU5otAG6LiI1UD9Jc1p3RSuBnc2zwxqKSpNo585Ek1c7wkSTVzvCRJNXO8JEk1c7wkSTVzvCRJNXOe7tJNYmIPwW+AHwiM7/eUv8N4H8D/yMzL46IDwN/lZl9ZfsxVL+lupjqFjATqO7G/OeZOVj2OzUzb9jv7z0HXJ6ZGyJiEOgHdpV97ATuzszlHTtg6SCc+Uj1eoHql/StLgNee6vGEdEDfBs4HvjXmTkb+DBwKXDLMP/2RzJzdma+D/gYsDAiPjPMfUiHheEj1es7wHsiovXuyR+nmsm8lQuAWcB/zMztAJn5KlWArRvpIDLz/wB/BHy2BJxUK0+7SfXaAfwXqtu6fDkiZgDHUZ0SO+st2s8BHs/MXa3FzHwWeLalNC8iztuv78mHGMvPqG6mORV4pe0jkA4Dw0eq398AK4EvU81g/uYgbXfT3hmK+w+w5nMwQ/fWeqON/UuHlafdpJpl5o+BCRExG5gHfOMgzR8Dzo6ICa3FiDg7Iv52lEM5G9iUmW+53iR1kuEjdcffArcBz2TmLw7UKDN/BGwElpar3oiIdwJ3AJtG+scj4mSqmddfjnQf0mh42k3qjtXArcAlbbS9DPgzoBkRO6kutb6H4QfH9yJiF9Xl1oPASi+1Vrf4SAVJUu2c+UiHUUQEcP8BNmdmzqtzPNJY5cxHklQ7LziQJNXO8JEk1c7wkSTVzvCRJNXO8JEk1e7/AwLC4ht5hP5yAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Plot showing that Heart Disease/Heart Attack group frequencies are unbalanced before SMOTe\n",
    "\n",
    "sns.countplot(x='_MICHD', data=heart_disease_attack_df_2017, palette='hls')\n",
    "plt.show\n",
    "plt.savefig('HeartDiseaseAttack_Count_Plot_Before_SMOTe')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ8AAAEPCAYAAACdhMnXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAFldJREFUeJzt3X+MXtV95/H3+NfgNBiSxRS7FNKU+otTI9wOP1bhx6KGErlhYVEDLHYg7DamlmHTbNO0K8VWl4qqalrxwwQrEY5jtAZCi5euFHBalWRrJxQShgZkCF9oCs4ajGCDUi9WPLax9497xn6YjD3PeOY5zzB+v6SRn/u959w517qaz5x779zbs3//fiRJqmlKtwcgSTr6GD6SpOoMH0lSdYaPJKk6w0eSVJ3hI0mqzvCRJFVn+EiSqjN8JEnVGT6SpOoMH0lSddO6PYCJor+/vxc4G9gOvN3l4UjSu8VUYA7wvb6+voF2Oxk+B50NbO72ICTpXeoC4NvtNjZ8DtoOMG/ePGbMmNHtsUjSu8Lu3bt54YUXoPwMbZfhc9DbADNmzKC3t7fbY5Gkd5tRXa7whgNJUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+42zfnj3dHoImII8L6Z38I9NxNmX6dJ789LJuD0MTzFmrvtTtIUgTijMfSVJ1ho8kqTrDR5JUneEjSarO8JEkVWf4SJKqM3wkSdUZPpKk6gwfSVJ1HX3CQUT8MXBVWXw4M/8wIr4KnA/sLPWbM/OhiLgYuBWYCTyQmSvKNhYCa4BZwCZgWWbujYhTgPXAiUACSzLzrYg4HrgX+CDwBnBVZr7Wyf2UJI1Ox2Y+JUwuAX4NWAj0RcQVwFnAhZm5sHw9FBEzgbXA5cB84OyIWFQ2tR64KTPnAT3A0lJfDazOzNOBJ4GVpX4LsDkz5wN3A3d0ah8lSUemk6fdtgOfzczdmbkH+AFwSvlaGxHPRMTNETEFOAd4MTNfysy9NIFzZUScCszMzMfLNteV+nTgQuDB1nr5/DGamQ/A/cCi0l6SNEF07LRbZj47+DkifoXm9NsFwEXAcuBfga8DvwO8RRNWg7YDJwNzD1E/AdhRgqq1TmufcnpuBzAbeHX89k6SNBYdf6p1RPwq8DDwucxM4IqWdXcC19HMYPa3dOsB9tHMzNqpU+qDbVr1tKwb0ZYtW9ptOqy+vr4x9dfk1d/f3+0hSBNGp284OA/YAHwmM78WEWcA8zJzQ2nSA+wBtgFzWrqeRDNTOVT9deC4iJiamW+XNoMzm1dKu20RMQ04Fvhxu2NesGABvb29o9tRqQ3+YqLJaGBg4Ih+ae/kDQe/CPwNsDgzv1bKPcDtEfG+ch3mBuAh4ImmS5wWEVOBxcDGzNwK7CohBnBtqe8BNgNXl/p1wMby+ZGyTFm/ubSXJE0QnZz5/AFwDHBrRAzWvgT8GfAdYDqwITPvB4iI62lmScfQBMjgzQRLgLsjYhbwFLCq1JcD90TECuBHwDWlvhJYFxHPAj8p/SVJE0jP/v1DL50cnfr7+z8AvDQep918k6mG8k2mmqxaTrv9Ul9f38vt9vMJB5Kk6gwfSVJ1ho8kqTrDR5JUneEjSarO8JEkVWf4SJKqM3wkSdUZPpKk6gwfSVJ1ho8kqTrDR5JUneEjSarO8JEkVWf4SJKqM3wkSdUZPpKk6gwfSVJ1ho8kqTrDR5JUneEjSarO8JEkVWf4SJKqM3wkSdUZPpKk6gwfSVJ1ho8kqTrDR5JUneEjSarO8JEkVTetkxuPiD8GriqLD2fmH0bExcCtwEzggcxcUdouBNYAs4BNwLLM3BsRpwDrgROBBJZk5lsRcTxwL/BB4A3gqsx8LSJmAF8BzgJ+CizOzOc7uZ+SpNHp2MynhMwlwK8BC4G+iLgGWAtcDswHzo6IRaXLeuCmzJwH9ABLS301sDozTweeBFaW+i3A5sycD9wN3FHqnwZ2lvpngHWd2kdJ0pHp5Gm37cBnM3N3Zu4BfgDMA17MzJcycy9N4FwZEacCMzPz8dJ3XalPBy4EHmytl88fo5n5ANwPLCrtD9QzcxMwu8yeJEkTRMdOu2Xms4OfI+JXaE6/3UkTSoO2AycDcw9RPwHYUYKqtU5rn3J6bgcw+zDb+lE7496yZUs7zQ6pr69vTP01efX393d7CNKE0dFrPgAR8avAw8DngL00s59BPcA+mhnY/jbqlPpgm1YjbastCxYsoLe3t93mUtv8xUST0cDAwBH90t7Ru90i4jzgUeC/ZeY9wDZgTkuTk4BXD1N/HTguIqaW+pxSB3iltCMipgHHAj8+zLYkSRNEJ284+EXgb2juNvtaKT/RrIrTSqAsBjZm5lZgVwkrgGtLfQ+wGbi61K8DNpbPj5RlyvrNpf2BekScD+zKzLZOuUmS6ujkabc/AI4Bbo2IwdqXgOuBDWXdIxy8mWAJcHdEzAKeAlaV+nLgnohYQXPd5ppSXwmsi4hngZ+U/tBcV/pyqQ/QBJkkaQLp5A0Hvwf83iFWnzlM+6eBc4apbwUuGqb+JnDZMPVdwCdHOVxJUkU+4UCSVJ3hI0mqzvCRJFVn+EiSqjN8JEnVGT6SpOoMH0lSdYaPJKk6w0eSVJ3hIx1F9uxr+wHvOop047jo+CsVJE0c06dMYdljT3Z7GJpgvvThs6p/T2c+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1bUVPhHxC8PUPjT+w5EkHQ0O+ybTiHh/+fhIRFwE9JTl6cD/BE7v3NAkSZPVSK/Rvh/4zfL5xy31vcCD7XyDiJgFPAZcmpkvR8RXgfOBnaXJzZn5UERcDNwKzAQeyMwVpf9CYA0wC9gELMvMvRFxCrAeOBFIYElmvhURxwP3Ah8E3gCuyszX2hmrJKmOw552y8yPZuYUYF1mTmn5mpGZi0faeEScC3wbmNdSPgu4MDMXlq+HImImsBa4HJgPnB0Ri0r79cBNmTmPZua1tNRXA6sz83TgSWBlqd8CbM7M+cDdwB0j/i9IkqoaaeYDQGb+54g4FXg/B0+9kZlPjdB1KXAj8D8AIuI9wCnA2nId6SHgZuAc4MXMfKm0Ww9cGRHPATMz8/GyvXXAzRGxBrgQ+A8t9X8A/gj4WFkHzcztroiYnpl72tlXSVLntRU+EXEz8DngdWB/Ke+nObV1SJn5qdJ/sHQS8E1gOfCvwNeB3wHeAra3dN0OnAzMPUT9BGBHZu4dUqe1Tzk9twOYDbzazr5KkjqvrfABrgNOy8wx/QDPzH8Brhhcjog7y7Yf5GCoQTO72kdzWrCdOqU+2KZVT8u6EW3ZsqXdpsPq6+sbU39NXv39/d0egsenDqn28dlu+PyfsQYPQEScAczLzA2l1APsAbYBc1qankQzUzlU/XXguIiYmplvlzaD43ultNsWEdOAY3nnzRKHtWDBAnp7e0e7a9KI/MGviexIj8+BgYEj+qW93T8yfTQivhAR50XErw9+jfq7NWFze0S8LyKmAzfQXPd5AoiIOC0ipgKLgY2ZuRXYFRHnlf7XlvoeYDNwdalfB2wsnx8py5T1m73eI0kTS7szn+vLv1e21Ea85jNUZj4TEX8GfIfmb4U2ZOb9ABFxPbABOIYmQAZv5V4C3F1u2X4KWFXqy4F7ImIF8CPgmlJfCayLiGeBn5T+kqQJpN273X5pLN8kMz/Q8nk1zW3SQ9s8Cpw5TP1pmrvhhta3AhcNU38TuGws45UkdVa7d7v9/nD1zLx1fIcjSToatHva7YyWzzOAfwc8Ov7DkSQdDdo97fafWpcjYi7wlY6MSJI06R3RKxXKbdcfGN+hSJKOFkdyzaeH5vlsr3dkRJKkSe9Irvnsp7m1+XPjPxxJ0tFgVNd8ysNFp2fmP3d0VJKkSa3d026nAf+L5qGdUyLi/9K8n+cHnRycJGlyaveGgy8CX8jM92XmcTTvzLmrc8OSJE1m7YbPz2fmPYMLmflVmtcUSJI0au2Gz7SIeP/gQkScwM++0kCSpLa0e7fbncDjEfEATej8R+C2jo1KkjSptTvzeYQmdGYAHwIGX4EtSdKotRs+64C7MvOPgE8AnwfWdmpQkqTJrd3wOSEzVwFk5q7MvJ13vmFUkqS2jeaGg7mDCxHx8zSP2ZEkadTaveHgVuD7EfENmms/F+PjdSRJR6itmU9mrqUJnH8CngQ+mpn3dXJgkqTJq92ZD5n5DPBMB8ciSTpKHNH7fCRJGgvDR5JUneEjSarO8JEkVWf4SJKqM3wkSdUZPpKk6gwfSVJ1ho8kqTrDR5JUneEjSaqu7We7HYmImAU8BlyamS9HxMU0T8ieCTyQmStKu4XAGmAWsAlYlpl7I+IUYD1wIpDAksx8KyKOB+4FPgi8AVyVma9FxAzgK8BZwE+BxZn5fCf3UZI0eh2b+UTEucC3gXlleSbN208vB+YDZ0fEotJ8PXBTZs6jeU/Q0lJfDazOzNNpnqa9stRvATZn5nzgbuCOUv80sLPUP0PzBlZJ0gTTydNuS4EbgVfL8jnAi5n5UmbupQmcKyPiVGBmZj5e2q0r9enAhcCDrfXy+WM0Mx+A+4FFpf2BemZuAmaX2ZMkaQLp2Gm3zPwUQEQMluYC21uabAdOPkz9BGBHCarW+ju2VU7P7QBmH2ZbP2p33Fu2bGm36bD6+vrG1F+TV39/f7eH4PGpQ6p9fHb0ms8QU2jegjqoB9g3ijqlPtim1UjbatuCBQvo7e0dTRepLf7g10R2pMfnwMDAEf3SXvNut23AnJblk2hOyR2q/jpwXERMLfU5HDyF90ppR0RMA44FfnyYbUmSJpCa4fMEEBFxWgmUxcDGzNwK7IqI80q7a0t9D7AZuLrUrwM2ls+PlGXK+s2l/YF6RJwP7MrMtk+5SZLqqBY+mbkLuB7YADwHPM/BmwmWALdFxPPAe4FVpb4cuCEingMuAFaU+krg30bEs6XNjaV+J9Bb6qtogkySNMF0/JpPZn6g5fOjwJnDtHma5m64ofWtwEXD1N8ELhumvgv45JgGLEnqOJ9wIEmqzvCRJFVn+EiSqjN8JEnVGT6SpOoMH0lSdYaPJKk6w0eSVJ3hI0mqzvCRJFVn+EiSqjN8JEnVGT6SpOoMH0lSdYaPJKk6w0eSVJ3hI0mqzvCRJFVn+EiSqjN8JEnVGT6SpOoMH0lSdYaPJKk6w0eSVJ3hI0mqzvCRJFVn+EiSqjN8JEnVGT6SpOqmdeObRsS3gBOBPaX0u8AvAyuA6cDtmXlXaXsxcCswE3ggM1eU+kJgDTAL2AQsy8y9EXEKsL5sP4ElmflWrX2TJI2s+swnInqAecCZmbkwMxcC24A/Bc4HFgI3RMSHImImsBa4HJgPnB0Ri8qm1gM3ZeY8oAdYWuqrgdWZeTrwJLCy0q5JktrUjdNuUf79u4h4OiJuAi4GvpmZb2bmTuBB4OPAOcCLmflSZu6lCZwrI+JUYGZmPl62ta7UpwMXlv4H6jV2SpLUvm6Ez/uAR4ErgI8Ay4BTgO0tbbYDJwNzR1k/AdhRgqq1LkmaQKpf88nMfwT+cXA5Ir5Cc03nlpZmPcA+mnDcP4Y6pd62LVu2jKb5z+jr6xtTf01e/f393R6Cx6cOqfbxWT18IuJ8oDczHy2lHuBlYE5Ls5OAV2muBY2m/jpwXERMzcy3S5tXRzO+BQsW0NvbO5ouUlv8wa+J7EiPz4GBgSP6pb0bp92OB/4iIo6JiGOBTwKfAD4SEbMj4j3AbwPfAJ4AIiJOi4ipwGJgY2ZuBXZFxHllm9eW+h5gM3B1qV8HbKy2Z5KktlQPn8z8OvAw8E9AP7A2M78DfB74FvB94L7M/G5m7gKuBzYAzwHPc/BmgiXAbRHxPPBeYFWpL6e5W+454AKa27clSRNIV/7OJzNXMuQW6My8D7hvmLaPAmcOU3+a5m64ofWtwEXjNVZJ0vjzCQeSpOoMH0lSdYaPJKk6w0eSVJ3hI0mqzvCRJFVn+EiSqjN8JEnVGT6SpOoMH0lSdYaPJKk6w0eSVJ3hI0mqzvCRJFVn+EiSqjN8JEnVGT6SpOoMH0lSdYaPJKk6w0eSVJ3hI0mqzvCRJFVn+EiSqjN8JEnVGT6SpOoMH0lSdYaPJKk6w0eSVJ3hI0mqzvCRJFU3rdsD6ISIWAysAKYDt2fmXV0ekiSpxaSb+UTELwB/CpwPLARuiIgPdXdUkqRWk3HmczHwzcx8EyAiHgQ+DvzJCP2mAuzevXvsI/i59459G5pUBgYGuj2EAzw6NdRYjs+Wn5lTR9NvMobPXGB7y/J24Jw2+s0BeOGFF8Y8gJ6PXzPmbWhy2bJlS7eHcMA1vT3dHoImmHE6PucAP2y38WQMnynA/pblHmBfG/2+B1xAE1Zvd2BckjQZTaUJnu+NptNkDJ9tNCEy6CTg1ZE69fX1DQDf7tSgJGkSa3vGM2gyhs/fA/89ImYDO4HfBm7o7pAkSa0m3d1umfkK8HngW8D3gfsy87vdHZUkqVXP/v37R24lSdI4mnQzH0nSxGf4SJKqM3wkSdUZPpKk6ibjrdaqbKQHuUbEQmANMAvYBCzLzL3VB6qjUkTMAh4DLs3Ml4es89jsEmc+GpM2H+S6HrgpM+fRPHFiad1R6mgVEefS/PH4vEM08djsEsNHY3XgQa6ZuRMYfJArABFxKjAzMx8vpXXAldVHqaPVUuBGhnnKicdmd3naTWM10oNch1t/coVxSWTmpwAiYrjVHptd5MxHYzXSg1yP9EGvUqd5bHaR4aOx2kZ5HUUx9EGuI62XusVjs4sMH43V3wMfiYjZEfEemge5fmNwZWZuBXZFxHmldC2wsf4wpXfy2Owuw0djcqgHuUbEIxFxVmm2BLgtIp6neZHmqu6MVgKPzYnBB4tKkqpz5iNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnU+200aJxFxPXAH8FIp9dA8qn8zcENm7oqIdcBvAm8M6f5bwCXAxzPz0iHb/d/AF8vi3cAZmbmtZf0XgHOB3wB+CAwAP6V5dMwM4O+Az2bmvoh4uWV9q+WZ+VhE/Beav9t6rdT/X2ZeMJr/B6kdho80vja3hkdEHEPzSP9PAl8u5dsy8y+HdjzEwy8PyMwHI2IR8NWIuCQz90fEBcB1wK9n5ttlG0sy88myzRnAPwDLORhgB9YP48PA72fmfe3trnRkPO0mdda/AY4D3hyn7X0aOAVYHhHvpXkNwPWZOewzyTJzN83M6/Q2t/9hYElEPBMRfxsRZ4zDmKWf4cxHGl8XRMT3gWNogucF4C8z869b2vzXiPhEy/IXM3PNkP6tThv8kJk7I+IamlNpFwF/lZnf4BAiYi7w72neNDvo3ohoPe02kJnnRsTPAc8Df56ZmyLiKmBjRJyemW+1se9S2wwfaXxtzsxLI2IKzQ/8xcBfD2kz7Gm31v6thXLN54DMfCoiVgNXANcMs43BcJkC7AHWZOaGlvXDnnYrLwP8aMvyX0XESuBsmmf3SePG8JE6IDP3AX8SER+mOTV22Th/i38Btmbm3mHWHe6aziGVN3telpl3tpR7aAJMGlde85E660bgkoi4vNsDacNO4JaIOAcgIn4LeA/w3a6OSpOSMx+pgzLzhxHx5zSP7f/bbo+nGHrNB8p1p3Kd58vlLrkdwBXlpgVpXPlKBUlSdc58pFGI5g9pHjjE6szMq2uOR3q3cuYjSarOGw4kSdUZPpKk6gwfSVJ1ho8kqTrDR5JU3f8HTTGIhPsW42UAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Plot showing that Hypertension group frequencies are unbalanced before SMOTe\n",
    "\n",
    "sns.countplot(x='_RFHYPE5', data=hypertension_df_2017, palette='hls')\n",
    "plt.show\n",
    "plt.savefig('Hypertension_Count_Plot_Before_SMOTe')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 341,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Applying SMOTe\n",
    "\n",
    "# Diabetes\n",
    "\n",
    "diabetes_X = diabetes_df_2017[[col_name for col_name in diabetes_df_2017.columns if col_name != 'DIABETE3']]\n",
    "diabetes_y = diabetes_df_2017['DIABETE3']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 342,
   "metadata": {},
   "outputs": [],
   "source": [
    "os = SMOTE(random_state=0)\n",
    "\n",
    "X_train_diabetes, X_test_diabetes, y_train_diabetes, y_test_diabetes = train_test_split(diabetes_X, diabetes_y, test_size=0.3, random_state=0)\n",
    "\n",
    "diabetes_columns = X_train_diabetes.columns\n",
    "os_X_diabetes,os_y_diabetes = os.fit_sample(X_train_diabetes, y_train_diabetes)\n",
    "os_X_diabetes = pd.DataFrame(data = os_X_diabetes, columns = diabetes_columns )\n",
    "os_y_diabetes = pd.DataFrame(data = os_y_diabetes, columns=['DIABETE3'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length of oversampled diabetes data is 540712\n",
      "Number of columns in oversampled diabetes data is 522\n",
      "Number of no diabetes dx in oversampled data 270356\n",
      "Number of diabetes dx 270356\n",
      "Proportion of no diabetes dx data in oversampled data is  0.5\n",
      "Proportion of diabetes dx data in oversampled data is  0.5\n"
     ]
    }
   ],
   "source": [
    "# Prints results of SMOTe - Diabetes\n",
    "\n",
    "print(\"Length of oversampled diabetes data is\", len(os_X_diabetes))\n",
    "print(\"Number of columns in oversampled diabetes data is\", os_X_diabetes.shape[1])\n",
    "print(\"Number of no diabetes dx in oversampled data\", len(os_y_diabetes[os_y_diabetes['DIABETE3'] == 0]))\n",
    "print(\"Number of diabetes dx\", len(os_y_diabetes[os_y_diabetes['DIABETE3'] == 1]))\n",
    "print(\"Proportion of no diabetes dx data in oversampled data is \", len(os_y_diabetes[os_y_diabetes['DIABETE3'] == 0])/len(os_X_diabetes))\n",
    "print(\"Proportion of diabetes dx data in oversampled data is \", len(os_y_diabetes[os_y_diabetes['DIABETE3'] == 1])/len(os_X_diabetes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ8AAAEPCAYAAACdhMnXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAFn5JREFUeJzt3X+MXeV95/H3+NfgbDAkwsQ4FChx/cWpszgdIKvyo0ShTb1Jg9LGUOxCaBKIAyjJbpusVrG3y4qqUrtLCAQrFcRxVuZXhUu7Cjiq6kTChELDpIEOgS8sC6bGRiSw1ILFg3/tH+cZc5nM4Ds/7nMn4/dLsrjne57n3Oeio/nMOeeZ5/YcOHAASZJqmtHtAUiSDj+GjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklTdrG4PYKro7+/vBU4HdgL7ujwcSfpFMRM4DvhhX1/fYLudDJ83nA5s7fYgJOkX1NnAfe02NnzesBNg8eLFzJkzp9tjkaRfCK+//jpPPPEElJ+h7TJ83rAPYM6cOfT29nZ7LJL0i2ZMjyuccCBJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNnku3fs6fbQ9AUNFXOiz3793d7CJqCunFe+Eemk2zG7Nk89PnV3R6GppjTrv9Gt4cAwOwZM1h9/0PdHoammG/8+mnV39MrH0lSdYaPJKk6w0eSVJ3hI0mqzvCRJFVn+EiSqjN8JEnVGT6SpOoMH0lSdYaPJKm6ji6vExF/AlxQNu/OzC9HxLeAs4BXS/3qzLwrIs4DrgXmAndk5ppyjGXAzcA84F5gdWbujYgTgI3AsUACqzLzlYg4GrgFOBn4KXBBZj7fyc8pSRqbjl35lDD5LeD9wDKgLyI+DpwGnJOZy8q/uyJiLrAeOB9YApweEcvLoTYCV2XmYqAHuKzU1wHrMvMU4CFgbalfA2zNzCXATcDXOvUZJUnj08nbbjuBP8rM1zNzD/AYcEL5tz4iHomIqyNiBnAG8GRmPp2Ze2kCZ0VEnAjMzcwHyjE3lPps4BzgztZ6ef0RmisfgNuA5aW9JGmK6Nhtt8x8dOh1RPwKze23s4FzgSuAfwW+A3waeIUmrIbsBI4HFo5SPwbYVYKqtU5rn3J7bhcwH9gxeZ9OkjQRHf9KhYj4VeBu4EuZmcDHW/bdAFxCcwVzoKVbD7Cf5sqsnTqlPtSmVU/LvkMaGBhot+mI+vr6JtRf01d/f3+3h+D5qVHVPj87PeHgTGAT8MXMvD0i3gcszsxNpUkPsAfYDhzX0nUBzZXKaPUXgKMiYmZm7itthq5snivttkfELOBI4MV2x7x06VJ6e3vH9kGlNviDX1PZeM/PwcHBcf3S3skJB78E/A2wMjNvL+Ue4LqIeEd5DnM5cBfwYNMlFkXETGAlsDkztwG7S4gBXFzqe4CtwIWlfgmwuby+p2xT9m8t7SVJU0Qnr3z+GDgCuDYihmrfAP4M+AEwG9iUmbcBRMSlNFdJR9AEyNBkglXATRExD/gRcH2pXwF8OyLWAM8CF5X6WmBDRDwKvFz6S5KmkE5OOPgC8IVRdq8bof0W4NQR6g/TzIYbXt9GM3lheP0l4GNjHK4kqSJXOJAkVWf4SJKqM3wkSdUZPpKk6gwfSVJ1ho8kqTrDR5JUneEjSarO8JEkVWf4SJKqM3wkSdUZPpKk6gwfSVJ1ho8kqTrDR5JUneEjSarO8JEkVWf4SJKqM3wkSdUZPpKk6gwfSVJ1ho8kqTrDR5JUneEjSarO8JEkVWf4SJKqM3wkSdUZPpKk6gwfSVJ1szp58Ij4E+CCsnl3Zn45Is4DrgXmAndk5prSdhlwMzAPuBdYnZl7I+IEYCNwLJDAqsx8JSKOBm4BTgZ+ClyQmc9HxBzgm8BpwGvAysx8vJOfU5I0Nh278ikh81vA+4FlQF9EXASsB84HlgCnR8Ty0mUjcFVmLgZ6gMtKfR2wLjNPAR4C1pb6NcDWzFwC3AR8rdQ/D7xa6l8ENnTqM0qSxqeTt912An+Uma9n5h7gMWAx8GRmPp2Ze2kCZ0VEnAjMzcwHSt8NpT4bOAe4s7VeXn+E5soH4DZgeWl/sJ6Z9wLzy9WTJGmK6Fj4ZOajQ2ESEb9Cc/ttP00oDdkJHA8sHKV+DLCrBFVrndY+Zf8uYP5bHEuSNEV09JkPQET8KnA38CVgL83Vz5AemkCaARxoo06pD7VpdahjtWVgYKDdpiPq6+ubUH9NX/39/d0eguenRlX7/Oz0hIMzgU3AFzPz9oj4DeC4liYLgB3A9lHqLwBHRcTMzNxX2uwobZ4r7bZHxCzgSODFlmM9NexYbVm6dCm9vb1j+pxSO/zBr6lsvOfn4ODguH5p7+SEg18C/oZmttntpfxgsysWRcRMYCWwOTO3AbtLWAFcXOp7gK3AhaV+CbC5vL6nbFP2by3tD9Yj4ixgd2Y+26nPKUkau05e+fwxcARwbUQM1b4BXEpzNXQETVAMTSZYBdwUEfOAHwHXl/oVwLcjYg3wLHBRqa8FNkTEo8DLpT/ADcBflvogTZBJkqaQjoVPZn4B+MIou08dof3DwBkj1LcB545Qfwn42Aj13cAnxzhcSVJFrnAgSarO8JEkVWf4SJKqM3wkSdUZPpKk6gwfSVJ1ho8kqTrDR5JUneEjSarO8JEkVWf4SJKqM3wkSdUZPpKk6gwfSVJ1ho8kqTrDR5JUneEjSarO8JEkVWf4SJKqM3wkSdUZPpKk6gwfSVJ1ho8kqTrDR5JUneEjSarO8JEkVWf4SJKqayt8IuLdI9TeO/nDkSQdDma91c6IeGd5eU9EnAv0lO3ZwF8Dp3RuaJKk6eotwwe4DfjN8vrFlvpe4M6OjEiSNO29Zfhk5ocBImJ9Zn5qPG8QEfOA+4GPZuYzEfEt4Czg1dLk6sy8KyLOA64F5gJ3ZOaa0n8ZcDMwD7gXWJ2ZeyPiBGAjcCyQwKrMfCUijgZuAU4GfgpckJnPj2fskqTOaOuZT2Z+KiJOjIj3R8SvDf07VL+I+ABwH7C4pXwacE5mLiv/7oqIucB64HxgCXB6RCwv7TcCV2XmYprbfpeV+jpgXWaeAjwErC31a4CtmbkEuAn4WjufUZJUT7sTDq4GHgPuAjaVf+3cdrsMuBLYUY7zNuAEYH1EPBIRV0fEDOAM4MnMfDoz99IEzoqIOBGYm5kPlONtKPXZwDktY9gArCivP0Jz5QPNbcPlpb0kaYo41DOfIZcAizJzx1gOnpmfAYiIodIC4HvAFcC/At8BPg28Auxs6boTOB5YOEr9GGBXCarWOq19yu25XcB8SgAeysDAQNufbyR9fX0T6q/pq7+/v9tD8PzUqGqfn+2Gz7+MNXhGkpn/B/j40HZE3EATbHcCB1qa9gD7aa7M2qlT6kNtWvW07DukpUuX0tvb225zqW3+4NdUNt7zc3BwcFy/tLcbPlsi4s+BvwVeGypm5o/G8mYR8T5gcWZuKqUeYA+wHTiupekCmiuV0eovAEdFxMzM3FfaDIXjc6Xd9oiYBRzJm2fqSZK6rN0VDi6leaaykbE98xmuB7guIt5RnsNcTvMc6UEgImJRRMwEVgKbM3MbsDsiziz9Ly71PcBW4MJSvwTYXF7fU7Yp+7eW9pKkKaKtK5/M/OXJeLPMfCQi/gz4Ac0fqm7KzNsAIuJSmlA7giZAhsJtFXBTmbL9I+D6Ur8C+HZErAGeBS4q9bXAhoh4FHi59JckTSFthU9E/MeR6pl5bTv9M/OkltfraKZJD2+zBTh1hPrDNLPhhte3AeeOUH8J+Fg745IkdUe7z3ze1/J6DvAbwJbJH44k6XDQ7m23P2zdjoiFwDc7MiJJ0rQ3rq9UKNOuT5rcoUiSDhfjeebTQ7NEzgsdGZEkadobzzOfAzSzy740+cORJB0OxvTMp6y1Njsz/3dHRyVJmtbave22iGZ1g4XAjIj4Gc1XJDzWycFJkqandiccfB3488x8R2YeRfO1BTd2bliSpOms3fB5V2Z+e2gjM79Fs1K0JElj1m74zIqIdw5tRMQx/Pyq0pIktaXd2W43AA9ExB00ofP7wFc7NipJ0rTW7pXPPTShMwd4L/BumtWoJUkas3bDZwNwY2b+J+APgK8A6zs1KEnS9NZu+ByTmdcDZObuzLyON3/JmyRJbRvLhIOFQxsR8S5+/uuqJUlqS7sTDq4FfhwR36V59nMeLq8jSRqntq58MnM9TeD8E/AQ8OHMvLWTA5MkTV/tXvmQmY8Aj3RwLJKkw8S4vs9HkqSJMHwkSdUZPpKk6gwfSVJ1ho8kqTrDR5JUneEjSarO8JEkVWf4SJKqM3wkSdUZPpKk6tpe2208ImIecD/w0cx8JiLOo1khey5wR2auKe2WATcD84B7gdWZuTciTgA2AscCCazKzFci4mjgFuBk4KfABZn5fETMAb4JnAa8BqzMzMc7+RklSWPXsSufiPgAcB+wuGzPpfn20/OBJcDpEbG8NN8IXJWZi2m+J+iyUl8HrMvMU2hW015b6tcAWzNzCXAT8LVS/zzwaql/keYbWCVJU0wnb7tdBlwJ7CjbZwBPZubTmbmXJnBWRMSJwNzMfKC021Dqs4FzgDtb6+X1R2iufABuA5aX9gfrmXkvML9cPUmSppCOhU9mfiYzt7aUFgI7W7Z3Ase/Rf0YYFcJqtb6m45V9u8C5r/FsSRJU0hHn/kMM4PmW1CH9AD7x1Cn1IfatDrUsdo2MDAwluY/p6+vb0L9NX319/d3ewienxpV7fOzZvhsB45r2V5Ac0tutPoLwFERMTMz95U2Q7fwnivttkfELOBI4MWWYz017FhtW7p0Kb29vWPpIrXFH/yaysZ7fg4ODo7rl/aaU60fBCIiFkXETGAlsDkztwG7I+LM0u7iUt8DbAUuLPVLgM3l9T1lm7J/a2l/sB4RZwG7M/PZDn8uSdIYVQufzNwNXApsAn4CPM4bkwlWAV+NiMeBtwPXl/oVwOUR8RPgbGBNqa8F/l1EPFraXFnqNwC9pX49TZBJkqaYjt92y8yTWl5vAU4doc3DNLPhhte3AeeOUH8J+NgI9d3AJyc0YElSx7nCgSSpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqqb1Y03jYjvA8cCe0rps8B7gDXAbOC6zLyxtD0PuBaYC9yRmWtKfRlwMzAPuBdYnZl7I+IEYGM5fgKrMvOVWp9NknRo1a98IqIHWAycmpnLMnMZsB34U+AsYBlweUS8NyLmAuuB84ElwOkRsbwcaiNwVWYuBnqAy0p9HbAuM08BHgLWVvpokqQ2deO2W5T//l1EPBwRVwHnAd/LzJcy81XgTuATwBnAk5n5dGbupQmcFRFxIjA3Mx8ox9pQ6rOBc0r/g/UaH0qS1L5uhM87gC3Ax4EPAauBE4CdLW12AscDC8dYPwbYVYKqtS5JmkKqP/PJzH8A/mFoOyK+SfNM55qWZj3AfppwPDCBOqXetoGBgbE0/zl9fX0T6q/pq7+/v9tD8PzUqGqfn9XDJyLOAnozc0sp9QDPAMe1NFsA7KB5FjSW+gvAURExMzP3lTY7xjK+pUuX0tvbO5YuUlv8wa+pbLzn5+Dg4Lh+ae/Gbbejgb+IiCMi4kjgk8AfAB+KiPkR8Tbg94DvAg8CERGLImImsBLYnJnbgN0RcWY55sWlvgfYClxY6pcAm6t9MklSW6qHT2Z+B7gb+CegH1ifmT8AvgJ8H/gxcGtm/mNm7gYuBTYBPwEe543JBKuAr0bE48DbgetL/Qqa2XI/Ac6mmb4tSZpCuvJ3Ppm5lmFToDPzVuDWEdpuAU4dof4wzWy44fVtwLmTNVZJ0uRzhQNJUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqqb1e0BdEJErATWALOB6zLzxi4PSZLUYtpd+UTEu4E/Bc4ClgGXR8R7uzsqSVKr6Xjlcx7wvcx8CSAi7gQ+Afy3Q/SbCfD6669PfAT/5u0TP4amlcHBwW4P4SDPTg03kfOz5WfmzLH0m47hsxDY2bK9EzijjX7HATzxxBMTHkDPJy6a8DE0vQwMDHR7CAdd1NvT7SFoipmk8/M44Kl2G0/H8JkBHGjZ7gH2t9Hvh8DZNGG1rwPjkqTpaCZN8PxwLJ2mY/hspwmRIQuAHYfq1NfXNwjc16lBSdI01vYVz5DpGD5/D/zXiJgPvAr8HnB5d4ckSWo17Wa7ZeZzwFeA7wM/Bm7NzH/s7qgkSa16Dhw4cOhWkiRNoml35SNJmvoMH0lSdYaPJKk6w0eSVN10nGqtyg61kGtELANuBuYB9wKrM3Nv9YHqsBQR84D7gY9m5jPD9nludolXPpqQNhdy3QhclZmLaVacuKzuKHW4iogP0Pzx+OJRmnhudonho4k6uJBrZr4KDC3kCkBEnAjMzcwHSmkDsKL6KHW4ugy4khFWOfHc7C5vu2miDrWQ60j7j68wLonM/AxARIy023Ozi7zy0UQdaiHX8S70KnWa52YXGT6aqO2Ur6Mohi/keqj9Urd4bnaR4aOJ+nvgQxExPyLeRrOQ63eHdmbmNmB3RJxZShcDm+sPU3ozz83uMnw0IaMt5BoR90TEaaXZKuCrEfE4zRdpXt+d0UrguTk1uLCoJKk6r3wkSdUZPpKk6gwfSVJ1ho8kqTrDR5JUncvrSOMUEScBTwH/XEozgFdoVvb+q4g4F/h6Zi5t6XMM8C/Ahsz8XEv9XJq/MclSmgX8DPhcZj42wnu1OgO4FVhUtk8FBoB9wP/NzA9GxDPAIPDasL5XZOb9EbGCZmXynvK+n83MJ8fwv0MaE8NHmpjXMnPZ0EZZrHJLROwDXhyh/aeBvwVWRsRXMvOlln1PDTvWl2n+7uQ3R3qvYVoXcz0AfDAzfzaszarMfGh4x4hYAPwl8G8zc3tEXAV8HfjwKO8lTZi33aRJVP5q/r8AXxq+LyJmAJ+lWT15K3D5aMeJiB7gnbx54cuOyMzngXeV4JkFnMjIwSlNGq98pMn3MPC+Eeq/DbyNZkmiI2n+sv6/t3x52Xsi4sfl9btK2w+29J/bsn/IDzLzyjbHdUtEtN52G8zMDwBk5p7yV//fAeaWsUodY/hIk+8A8P9GqH8OuCUz90bE/6K51bUCuK3sH37b7RLg7yLil0vprW67tWPE225Dyr4FEfHbwN0RcXJmvjyB95NG5W03afKdzrCJAeVZ0L8Hfr88/E+aX/7+w2gHycz/WdoM/2bYSRURCyPi4POdzPwusAt4TyffV4c3w0eaRBGxGFgL/I9huz4L3JeZ787MkzLzJKAP+LWI+PVRjjW02nKOtH8SHQHcHhGLyvt+kCb0Huvw++ow5m03aWJan8PsB3YD/zkz7y7Tp4mIOTSz3D7V2jEzn4yI22iufm7kzc98ZtBMjf7dzHw5Io5m5Gc+AJdm5kj14YY/84FmKvjNEfEZYFOZKfcy8DuZOdKtQ2lSuKq1JKk6b7tJkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRV9/8BPkmWxN1uC94AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot showing that Diabetes group frequencies are balanced after SMOTe\n",
    "\n",
    "sns.countplot(x='DIABETE3', data=os_y_diabetes, palette='hls')\n",
    "plt.show\n",
    "plt.savefig('Diabetes_Count_Plot_After_SMOTe')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 343,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Heart Disease/Heart Attack\n",
    "\n",
    "heart_disease_attack_X = heart_disease_attack_df_2017[[col_name for col_name in heart_disease_attack_df_2017.columns if col_name != '_MICHD']]\n",
    "heart_disease_attack_y = heart_disease_attack_df_2017['_MICHD']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 344,
   "metadata": {},
   "outputs": [],
   "source": [
    "os2 = SMOTE(random_state=0)\n",
    "\n",
    "X_train_heart, X_test_heart, y_train_heart, y_test_heart = train_test_split(heart_disease_attack_X, heart_disease_attack_y, test_size=0.3, random_state=0)\n",
    "\n",
    "heart_disease_attack_columns = X_train_heart.columns\n",
    "os_X_heart,os_y_heart = os2.fit_sample(X_train_heart, y_train_heart)\n",
    "os_X_heart = pd.DataFrame(data = os_X_heart, columns = heart_disease_attack_columns)\n",
    "os_y_heart = pd.DataFrame(data = os_y_heart, columns = ['_MICHD'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length of oversampled heart disease/attack data is  574900\n",
      "Number of columns in oversampled diabetes data is 522\n",
      "Number of no heart diease/attack dx in oversampled data 287450\n",
      "Number of heart disease/attack dx 287450\n",
      "Proportion of no heart disease/attack dx data in oversampled data is  0.5\n",
      "Proportion of heart disease/attack dx data in oversampled data is  0.5\n"
     ]
    }
   ],
   "source": [
    "# Prints results of SMOTe - Heart Disease/Attack\n",
    "\n",
    "print(\"Length of oversampled heart disease/attack data is \", len(os_X_heart))\n",
    "print(\"Number of columns in oversampled diabetes data is\", os_X_heart.shape[1])\n",
    "print(\"Number of no heart diease/attack dx in oversampled data\", len(os_y_heart[os_y_heart['_MICHD'] == 0]))\n",
    "print(\"Number of heart disease/attack dx\", len(os_y_heart[os_y_heart['_MICHD'] == 1]))\n",
    "print(\"Proportion of no heart disease/attack dx data in oversampled data is \", len(os_y_heart[os_y_heart['_MICHD'] == 0])/len(os_X_heart))\n",
    "print(\"Proportion of heart disease/attack dx data in oversampled data is \", len(os_y_heart[os_y_heart['_MICHD'] == 1])/len(os_X_heart))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ8AAAEPCAYAAACdhMnXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAF8BJREFUeJzt3X+MXeV95/H3YDuD02BIZBPsUJOy1F9InOJqgGQXcOiGbWSBwkb88GIXQlNMEbBJNz+aSrE3JaJbJbsyYIqbCnBMa354BSWrDXZU1UlrJ8Q0TBLQQPwNuwsmgFkoKLVg8WBj7x/nGXxxZjx3ZnyfGY3fL8ninu95nnOfi47mM885zz3TtW/fPiRJqumI8R6AJOnwY/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqqb2smDR8RXgYuAfcAdmbkyIs4FVgLTgfWZuby0XQDcDswANgNXZ+aeiJgLrAOOBRJYmpmvRsQxwF3AicBLwCWZ+UJEvAO4AzgNeB1YkpnbOvk5JUkj07HwiYiPAv8W+C1gGvBERGwC1gAfBX4BPBgRizJzI03AXJmZWyPiDmAZ8JfAamB1Zt4bESuAFcCXgBuALZl5XkRcBtwMLAY+A7yWmadExEJgLfCR4cbb29vbDZwO7ADePFT/HyRpkpsCzAZ+1NPT099up46FT2b+Y0T8Tpm9vK+81zHAk5n5FEBErAMujogngOmZubV0XwtcHxG3AwuBf99S/0ea8Dmv7AO4B7g1IqaV+n8uY9gcEbMiYm5mPjPMkE8Htoz1c0vSYeps4PvtNu7oPZ/M3B0R1wNPAJuAOTQziwE7gOMPUp8J7MzMPQfUae1T9u8EZh3kWMPZMXwTSdIQRvQztKP3fAAy8ysR8TXgfwLzaO7/DOgC9tKEYDt1Sn2gTavhjjWcNwHmz59Pd3d3G80lSf39/fT19cEIb1d0bOYTESeXRQRk5v8D/hY4h+ba4IDjgOeBZ4eovwgcHRFTSn12qQM8V9oREVOBo4CXD3IsSdIE0cnLbicCt0VEd1mBdgHwV0BExEklUJYAGzNzO7ArIs4sfS8r9d0092EWl/rlwMbyekPZpuzfUtq/VY+Is4BdbdzvkSRV1LHwycwNwIPAT4Be4KHMvBe4Arif5j7QNuC+0mUpcGNEbAPeBawq9WuAq8qihLOB5aW+AvhIRDxe2lxb6rcA3aW+iibIJEkTSJd/RrvR29v7fuAp7/lIUvta7vn8Rk9Pz9Pt9vMJB5Kk6gwfSVJ1ho8kqTrD5xDbu3v3eA9BE9BEOS92723nK2863IzHedHxL5kebo6YNo1HPnP1eA9DE8xpq74x3kMAYNoRR3D1Q4+M9zA0wXzj35xW/T2d+UiSqjN8JEnVGT6SpOoMH0lSdYaPJKk6w0eSVJ3hI0mqzvCRJFVn+EiSqjN8JEnVGT6SpOoMH0lSdYaPJKk6w0eSVJ3hI0mqzvCRJFVn+EiSqjN8JEnVGT6SpOoMH0lSdYaPJKk6w0eSVN3UTh48Ir4CXFI2H8zMP46IbwJnAa+V+vWZ+UBEnAusBKYD6zNzeTnGAuB2YAawGbg6M/dExFxgHXAskMDSzHw1Io4B7gJOBF4CLsnMFzr5OSVJI9OxmU8Jk98FfhtYAPRExCeB04CFmbmg/HsgIqYDa4ALgFOA0yNiUTnUOuC6zJwHdAHLSn01sDozTwYeAVaU+g3Alsw8BbgNuLlTn1GSNDqdvOy2A/h8Zr6RmbuBnwFzy781EfFYRFwfEUcAZwBPZuZTmbmHJnAujogTgOmZubUcc22pTwMWAve11svr82hmPgD3AItKe0nSBNGxy26Z+fjA64j4TZrLb2cD5wDXAP8CfBv4A+BVmrAasAM4HpgzRH0msLMEVWud1j7l8txOYBbwfDvj7uvra/cjDqqnp2dM/TV59fb2jvcQPD81pNrnZ0fv+QBExAeBB4EvZmYCn2zZdwtwOc0MZl9Lty5gL83MrJ06pT7QplVXy75hzZ8/n+7u7nabS23zB78mstGen/39/aP6pb2jq90i4kxgE/AnmXlnRHwoIi5sadIF7AaeBWa31I+jmakMVX8RODoippT6bPbPbJ4r7YiIqcBRwMuH8nNJksamkwsOfh34FrAkM+8t5S7gpoh4d7kPcxXwAPBw0yVOKoGyBNiYmduBXSXEAC4r9d3AFmBxqV8ObCyvN5Rtyv4tpb0kaYLo5GW3LwBHAisjYqD2DeDPgR8A04D7M/MegIi4Ari/9NnA/sUES4HbImIG8GNgValfA9wZEcuBZ4BLS30FsDYiHgd+WfpLkiaQTi44+Czw2SF2rx6k/Sbg1EHqj9Kshjuwvp1m8cKB9VeAT4xwuJKkinzCgSSpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqm5qJw8eEV8BLimbD2bmH0fEucBKYDqwPjOXl7YLgNuBGcBm4OrM3BMRc4F1wLFAAksz89WIOAa4CzgReAm4JDNfiIh3AHcApwGvA0syc1snP6ckaWQ6NvMpIfO7wG8DC4CeiLgUWANcAJwCnB4Ri0qXdcB1mTkP6AKWlfpqYHVmngw8Aqwo9RuALZl5CnAbcHOpfwZ4rdT/CFjbqc8oSRqdTl522wF8PjPfyMzdwM+AecCTmflUZu6hCZyLI+IEYHpmbi1915b6NGAhcF9rvbw+j2bmA3APsKi0f6uemZuBWWX2JEmaIDp22S0zHx94HRG/SXP57RaaUBqwAzgemDNEfSawswRVa53WPuXy3E5g1kGO9Uw74+7r62un2ZB6enrG1F+TV29v73gPwfNTQ6p9fnb0ng9ARHwQeBD4IrCHZvYzoAvYSzMD29dGnVIfaNNquGO1Zf78+XR3d7fbXGqbP/g1kY32/Ozv7x/VL+0dXe0WEWcCm4A/ycw7gWeB2S1NjgOeP0j9ReDoiJhS6rNLHeC50o6ImAocBbx8kGNJkiaITi44+HXgWzSrze4t5YebXXFSCZQlwMbM3A7sKmEFcFmp7wa2AItL/XJgY3m9oWxT9m8p7d+qR8RZwK7MbOuSmySpjk5edvsCcCSwMiIGat8ArgDuL/s2sH8xwVLgtoiYAfwYWFXq1wB3RsRymvs2l5b6CmBtRDwO/LL0h+a+0l+Vej9NkEmSJpBOLjj4LPDZIXafOkj7R4EzBqlvB84ZpP4K8IlB6ruAT41wuJKkinzCgSSpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpurbCJyLeN0jtA4d+OJKkw8FB/4x2RLynvNwQEecAXWV7GvC3wMmdG5okabI6aPgA9wD/rrx+uaW+B7ivIyOSJE16Bw2fzPw4QESsycxP1xmSJGmyG27mA0BmfjoiTgDew/5Lb2Tmjzs1MEnS5NVW+ETE9cAXgReBfaW8DzixQ+OSJE1ibYUPcDlwUmY+38nBSJIOD+1+z+cXBo8k6VBpd+azKSK+DvwP4PWBovd8JEmj0W74XFH+e3FLzXs+kqRRaXe122+M9g0iYgbwEHB+Zj4dEd8EzgJeK02uz8wHIuJcYCUwHVifmctL/wXA7cAMYDNwdWbuiYi5wDrgWCCBpZn5akQcA9xFE4wvAZdk5gujHb8k6dBrd7Xb5warZ+bKYfp9GLgNmNdSPg1YmJk7WtpNB9YAHwV+ATwYEYsycyNNwFyZmVsj4g5gGfCXwGpgdWbeGxErgBXAl4AbgC2ZeV5EXAbcDCxu53NKkupod8HBh1r+9QCfA05to98y4FrgeYCIeCcwF1gTEY9FxPURcQRwBvBkZj6VmXtoAufi8t2i6Zm5tRxvbalPAxay/ykLa9l/SfA8mpkPNE9oWFTaS5ImiHYvu/1+63ZEzAHuaKPflaX9QOk44LvANcC/AN8G/gB4FdjR0nUHcDwwZ4j6TGBnCarWOq19yuW5ncAsSgAOp6+vr51mQ+rp6RlTf01evb294z0Ez08Nqfb52e6Cg7fJzOcj4v2j6Pd/gE8ObEfELTTfIbqP/V9eheYpCntpZmbt1Cn1gTatulr2DWv+/Pl0d3e321xqmz/4NZGN9vzs7+8f1S/to7nn00Vz3+bFkb5ZRHwImJeZ97ccazfwLDC7pelxNDOVoeovAkdHxJTMfLO0GZjZPFfaPRsRU4GjePtDUSVJ42w093w+CDwDLB3F+3UBN0XEu8t9mKuAB4CHgYiIkyJiCrAE2JiZ24FdEXFm6X9Zqe8GtrB/IcHlwMbyekPZpuzfUtpLkiaIEd3zKQsApmXm/xrNm2XmYxHx58APaP4m0P2ZeU859hXA/cCRNAEysJhgKXBbWbL9Y2BVqV8D3BkRy2nC8NJSXwGsjYjHgV8yupCUJHVQu5fdTqJ5usEc4IiI+Gea7+38rJ3+mfn+lteraZZJH9hmE4OsoMvMR2lWwx1Y3w6cM0j9FeAT7YxLkjQ+2r3s9hfA1zPz3Zl5NM13aW7t3LAkSZNZu+Hz3sy8c2AjM79Js3xZkqQRazd8pkbEewY2ImImv7rUWZKktrT7PZ9bgK0RsZ4mdP4DcGPHRiVJmtTanflsoAmddwAfAN5Hs0RakqQRazd81gK3ZuaXgN8DvkzzIFBJkkas3fCZmZmrADJzV2bexNufPCBJUttGsuBgzsBGRLyXX32GmiRJbWl3wcFK4KcR8R2aez/nAl/s2KgkSZNaWzOfzFxDEzg/AR4BPp6Zd3dyYJKkyavtP6mQmY8Bj3VwLJKkw0S793wkSTpkDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1bX9x+RGIyJmAA8B52fm0xFxLs2f5J4OrM/M5aXdAuB2YAawGbg6M/dExFxgHXAskMDSzHw1Io4B7gJOBF4CLsnMFyLiHcAdwGnA68CSzNzWyc8oSRq5js18IuLDwPeBeWV7OrAGuAA4BTg9IhaV5uuA6zJzHtAFLCv11cDqzDyZ5s93ryj1G4AtmXkKcBtwc6l/Bnit1P8IWNupzydJGr1OXnZbBlwLPF+2zwCezMynMnMPTeBcHBEnANMzc2tpt7bUpwELgfta6+X1eTQzH4B7gEWl/Vv1zNwMzCqzJ0nSBNKxy26ZeSVARAyU5gA7WprsAI4/SH0msLMEVWv9bccql+d2ArMOcqxn2h13X19fu00H1dPTM6b+mrx6e3vHewienxpS7fOzo/d8DnAEsK9luwvYO4I6pT7QptVwx2rb/Pnz6e7uHkkXqS3+4NdENtrzs7+/f1S/tNdc7fYsMLtl+ziaS3JD1V8Ejo6IKaU+m/2X8J4r7YiIqcBRwMsHOZYkaQKpGT4PAxERJ5VAWQJszMztwK6IOLO0u6zUdwNbgMWlfjmwsbzeULYp+7eU9m/VI+IsYFdmtn3JTZJUR7XwycxdwBXA/cATwDb2LyZYCtwYEduAdwGrSv0a4KqIeAI4G1he6iuAj0TE46XNtaV+C9Bd6qtogkySNMF0/J5PZr6/5fUm4NRB2jxKsxruwPp24JxB6q8Anxikvgv41JgGLEnqOJ9wIEmqzvCRJFVn+EiSqjN8JEnVGT6SpOoMH0lSdYaPJKk6w0eSVJ3hI0mqzvCRJFVn+EiSqjN8JEnVGT6SpOoMH0lSdYaPJKk6w0eSVJ3hI0mqzvCRJFVn+EiSqjN8JEnVGT6SpOoMH0lSdYaPJKk6w0eSVJ3hI0mqzvCRJFVn+EiSqps6Hm8aEd8DjgV2l9IfAv8KWA5MA27KzFtL23OBlcB0YH1mLi/1BcDtwAxgM3B1Zu6JiLnAunL8BJZm5qu1PpskaXjVZz4R0QXMA07NzAWZuQB4Fvgz4CxgAXBVRHwgIqYDa4ALgFOA0yNiUTnUOuC6zJwHdAHLSn01sDozTwYeAVZU+miSpDaNx2W3KP/9u4h4NCKuA84FvpuZr2Tma8B9wEXAGcCTmflUZu6hCZyLI+IEYHpmbi3HWlvq04CFpf9b9RofSpLUvvG47PZuYBPwH2kusf0DsB7Y0dJmB03wzBmkfvxB6jOBnSWoWutt6+vrG0nzX9HT0zOm/pq8ent7x3sInp8aUu3zs3r4ZOYPgR8ObEfEHTT3dG5oadYF7KWZme0bQ51Sb9v8+fPp7u4eSRepLf7g10Q22vOzv79/VL+0j8c9n7Mi4mMtpS7gaWB2S+044Hmae0Ejqb8IHB0RU0p9dqlLkiaQ8bjncwzwXyPiyIg4CvgU8HvAxyJiVkS8E7gQ+A7wMBARcVIJlCXAxszcDuyKiDPLMS8r9d3AFmBxqV8ObKz2ySRJbakePpn5beBB4CdAL7AmM38AfBn4HvBT4O7M/KfM3AVcAdwPPAFsY/9igqXAjRGxDXgXsKrUr6FZLfcEcDbN8m1J0gQyLt/zycwVHLAEOjPvBu4epO0m4NRB6o/SLEo4sL4dOOdQjVWSdOj5hANJUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklTd1PEeQCdExBJgOTANuCkzbx3nIUmSWky6mU9EvA/4M+AsYAFwVUR8YHxHJUlqNRlnPucC383MVwAi4j7gIuCrw/SbAvDGG2+MfQS/9q6xH0OTSn9//3gP4S2enTrQWM7Plp+ZU0bSbzKGzxxgR8v2DuCMNvrNBvj5z38+5gF0XXTpmI+hyaWvr2+8h/CWS7u7xnsImmAO0fk5G/jf7TaejOFzBLCvZbsL2NtGvx8BZ9OE1ZsdGJckTUZTaILnRyPpNBnD51maEBlwHPD8cJ16enr6ge93alCSNIm1PeMZMBnD5++BP42IWcBrwIXAVeM7JElSq0m32i0znwO+DHwP+Clwd2b+0/iOSpLUqmvfvn3Dt5Ik6RCadDMfSdLEZ/hIkqozfCRJ1Rk+kqTqJuNSa1U23INcI2IBcDswA9gMXJ2Ze6oPVIeliJgBPAScn5lPH7DPc3OcOPPRmLT5INd1wHWZOY/miRPL6o5Sh6uI+DDNl8fnDdHEc3OcGD4aq7ce5JqZrwEDD3IFICJOAKZn5tZSWgtcXH2UOlwtA65lkKeceG6OLy+7aayGe5DrYPuPrzAuicy8EiAiBtvtuTmOnPlorIZ7kOtoH/QqdZrn5jgyfDRWz1L+HEVx4INch9svjRfPzXFk+Gis/h74WETMioh30jzI9TsDOzNzO7ArIs4spcuAjfWHKb2d5+b4Mnw0JkM9yDUiNkTEaaXZUuDGiNhG84c0V43PaCXw3JwYfLCoJKk6Zz6SpOoMH0lSdYaPJKk6w0eSVJ3hI0mqzvCRJFXns92kSiLiT4GvAJ/OzG+21H8N+L/AP2Tm+RFxDvAXmTm/7D+S5rtU59M8AmYKzdOYv56Z+8pxZ2bmdQe839PARZn5SETsA/qAN8sx9gB3ZObqjn1g6SCc+Uh1PUPzTfpWFwKvDtY4IrqAbwFHA/86MxcA5wCfBL46wvf+ncxckJmnAh8HLo+Iz4/wGNIhYfhIdX0H+GBEtD49+VM0M5nBLAROAf5TZu4CyMyXaQJs82gHkZn/DHwO+EIJOKkqL7tJde0G/jvNY12+FhFzgaNoLomdPEj704CHM/PN1mJmPgk82VJaHBFnHdB3zjBjeZTmYZozgZfa/gTSIWD4SPX9NbAG+BrNDOavD9J2L+1doVg/xD2fgxl4ttbrbRxfOqS87CZVlpk/AqZExAJgMXD3QZpvBU6PiCmtxYg4PSL+ZoxDOR14KjMHvd8kdZLhI42PvwFuBH6ema8M1SgzfwhsA1aWVW9ExHuBW4CnRvvmETGHZub130Z7DGksvOwmjY91wA3ABW20vRD4L0BvROyhWWp9JyMPju9FxJs0y633AWtcaq3x4p9UkCRV58xHOoQiIoD1Q+zOzFxcczzSROXMR5JUnQsOJEnVGT6SpOoMH0lSdYaPJKk6w0eSVN3/BxNlCoahfb7qAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot showing that Heart Disease/Attack group frequencies are balanced after SMOTe\n",
    "\n",
    "sns.countplot(x='_MICHD', data=os_y_heart, palette='hls')\n",
    "plt.show\n",
    "plt.savefig('HeartDiseaseAttack_Count_Plot_After_SMOTe')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 345,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hypertension\n",
    "\n",
    "hypertension_X = hypertension_df_2017[[col_name for col_name in hypertension_df_2017.columns if col_name != '_RFHYPE5']]\n",
    "hypertension_y = hypertension_df_2017['_RFHYPE5']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 346,
   "metadata": {},
   "outputs": [],
   "source": [
    "os3 = SMOTE(random_state=0)\n",
    "\n",
    "X_train_hypertension, X_test_hypertension, y_train_hypertension, y_test_hypertension = train_test_split(hypertension_X, hypertension_y, test_size=0.3, random_state=0)\n",
    "\n",
    "hypertension_columns = X_train_hypertension.columns\n",
    "os_X_hypertension,os_y_hypertension = os.fit_sample(X_train_hypertension, y_train_hypertension)\n",
    "os_X_hypertension = pd.DataFrame(data = os_X_hypertension, columns = hypertension_columns )\n",
    "os_y_hypertension = pd.DataFrame(data = os_y_hypertension, columns=['_RFHYPE5'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length of oversampled hypertension data is  376392\n",
      "Number of columns in oversampled diabetes data is 521\n",
      "Number of no hypertension dx in oversampled data 188196\n",
      "Number of hypertension dx 188196\n",
      "Proportion of no hypertension dx data in oversampled data is  0.5\n",
      "Proportion of hypertension dx data in oversampled data is  0.5\n"
     ]
    }
   ],
   "source": [
    "# Prints results of SMOTe - Hypertension\n",
    "\n",
    "print(\"Length of oversampled hypertension data is \", len(os_X_hypertension))\n",
    "print(\"Number of columns in oversampled diabetes data is\", os_X_hypertension.shape[1])\n",
    "print(\"Number of no hypertension dx in oversampled data\", len(os_y_hypertension[os_y_hypertension['_RFHYPE5'] == 0]))\n",
    "print(\"Number of hypertension dx\", len(os_y_hypertension[os_y_hypertension['_RFHYPE5'] == 1]))\n",
    "print(\"Proportion of no hypertension dx data in oversampled data is \", len(os_y_hypertension[os_y_hypertension['_RFHYPE5'] == 0])/len(os_X_hypertension))\n",
    "print(\"Proportion of hypertension dx data in oversampled data is \", len(os_y_hypertension[os_y_hypertension['_RFHYPE5'] == 1])/len(os_X_hypertension))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ8AAAEPCAYAAACdhMnXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAGktJREFUeJzt3X+QXWWd5/F3JyRNFAK6gCSyQRkmX3DDkLX5YcmPoVZGF2VgKAWESGSVRCowzg/XnSpNxmGKWWucERAkq0WMsSbCoGTZ2VGi7qAOQQaUVmBbhi+sQtyYWLAyM9mwpklI9o/zNLk2neT2j/vctnm/qlLc8z3POf0c6lR/+jnnuef07N69G0mSaprW7Q5Ikl5+DB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpugO63YHJor+/vxc4GdgCvNDl7kjSr4rpwBzge319fYPtbmT47HEysKHbnZCkX1FnAPe229jw2WMLwPz585k5c2a3+yJJvxKef/55Hn/8cSi/Q9tl+OzxAsDMmTPp7e3tdl8k6VfNqG5XOOFAklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGzwTbtWNHt7ugSWiynBc7du3qdhc0CXXjvPBLphNs2owZPPjBK7vdDU0yJ934mW53AYAZ06Zx5X0PdrsbmmQ+8+aTqv9MRz6SpOoMH0lSdYaPJKk6w0eSVJ3hI0mqzvCRJFVn+EiSqjN8JEnVGT6SpOo6/oSDiJgN3AecC7wB+M8tq18LPJCZ50bEx4D3Af9U1t2SmTdHxEJgFTAbuAe4MjN3RsQ8YC1wBJDAoszcFhGHAl8EjgGeAS7KzJ91+jglSe3r6MgnIk4F7gXmA2TmXZm5MDMXAv8e2Ar8QWl+EvDuofWZeXOprwWuzsz5QA+wpNRXAisz8zjgQWBFqV8LbMjM44FbgE918hglSaPX6ctuS4CrgM0jrPsL4DOZ+URZPgn4SEQ8EhGfjogDI+JoYFZm3l/arAEujIgZwJnAHa318vkdNCMfgNuAc0p7SdIk0dHwycwrMnPD8HpE/DpwFnBjWT4I+AHwYeCNwKE0I5m5wJaWTbcARwGHAVszc+ewOq3blPVbgcMn8rgkSePTradaL6W5ZDYIkJnbgLcPrYyITwKrgbuA3S3b9QC7aEKztU6pD7Vp1dOybr8GBgbabTqivr6+cW2vqau/v7/bXfD81F7VPj+7FT6/A7x1aKFMHjg7M1eXUg+wA9gEzGnZ7kiaS3hPA4dExPTMfKG0Gbq099PSblNEHAAcDPy83Y4tWLCA3t7eMR2UtC/+4tdkNtbzc3BwcEx/tFefah0Rh9Hcx3mypfwL4BMR8fqI6KG5T3RnZm4EtkfEaaXdZcD6zNwBbAAuLvXFwPry+a6yTFm/obSXJE0S3fiezzE0I5oXZeYzwAeAv6WZNt0DfLKsXgRcHxGPAQdR7hMBy4ClEfEocAawvNRXAG+KiB+WNld17lAkSWNR5bJbZr6u5fN3gTeN0GYdsG6E+sPAKSPUN9JMWhhefxY4b1wdliR1lE84kCRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1R3Q6R8QEbOB+4BzM/OpiPg8cDrwXGlyTWbeGRFnA9cBs4DbM3N52X4hsAqYDdwDXJmZOyNiHrAWOAJIYFFmbouIQ4EvAscAzwAXZebPOn2ckqT2dXTkExGnAvcC81vKJwFnZubC8u/OiJgFrAbOB44HTo6Ic0r7tcDVmTkf6AGWlPpKYGVmHgc8CKwo9WuBDZl5PHAL8KnOHaEkaSw6fdltCXAVsBkgIl4BzANWR8QjEXFNREwDTgGeyMwnM3MnTeBcGBFHA7My8/6yvzWlPgM4E7ijtV4+v4Nm5ANwG3BOaS9JmiQ6etktM68AiIih0pHAN4FlwL8AXwHeD2wDtrRsugU4Cpi7l/phwNYSVK11Wrcpl+e2AodTAnB/BgYG2j6+kfT19Y1re01d/f393e6C56f2qvb52fF7Pq0y88fABUPLEXETsJhmBLO7pWkPsItmZNZOnVIfatOqp2Xdfi1YsIDe3t52m0tt8xe/JrOxnp+Dg4Nj+qO96my3iDghIt7ZUuoBdgCbgDkt9SNpRip7qz8NHBIR00t9DntGNj8t7YiIA4CDgZ9P7JFIksaj9lTrHuCGiHhVuQ+zFLgTeACIiDi2BMqlwPrM3Ahsj4jTyvaXlfoOYANwcakvBtaXz3eVZcr6DaW9JGmSqBo+mfkI8HHgO8CjwEOZeVtmbgcuB9aV+mPsmUywCLg+Ih4DDgJuLPVlwNKIeBQ4A1he6iuAN0XED0ubqzp9XJKk0alyzyczX9fyeSXNNOnhbe4GThyh/jDNbLjh9Y3AWSPUnwXOG1eHJUkd5RMOJEnVGT6SpOoMH0lSdYaPJKk6w0eSVJ3hI0mqzvCRJFVn+EiSqjN8JEnVGT6SpOoMH0lSdYaPJKk6w0eSVJ3hI0mqzvCRJFVn+EiSqjN8JEnVGT6SpOo6/hrtiJgN3Aecm5lPRcRS4IPAbuBB4AOZ+XxEfAx4H/BPZdNbMvPmiFgIrAJmA/cAV2bmzoiYB6wFjgASWJSZ2yLiUOCLwDHAM8BFmfmzTh+nJKl9HR35RMSpwL3A/LI8H/gw8GbgN8rPv6o0Pwl4d2YuLP9uLvW1wNWZOR/oAZaU+kpgZWYeRxNiK0r9WmBDZh4P3AJ8qoOHKEkag05fdltCEy6by/IgsCwzt2bmbuB/AvPKupOAj0TEIxHx6Yg4MCKOBmZl5v2lzRrgwoiYAZwJ3NFaL5/fQTPyAbgNOKe0lyRNEh0Nn8y8IjM3tCxvzMz/ARARhwNXA38TEQcBP6AZFb0ROJRmJDMX2NKyyy3AUcBhwNbM3DmsTus2Zf1W4PCOHKAkaUw6fs9nJBHxWmA98LnM/HYpv71l/SeB1cBdNPeGhvQAu2hCs7VOqQ+1adXTsm6/BgYG2m06or6+vnFtr6mrv7+/213w/NRe1T4/q4dPRBwHfB24MTM/WWrzgLMzc3Vp1gPsADYBc1o2P5LmEt7TwCERMT0zXyhthi7t/bS02xQRBwAHAz9vt38LFiygt7d3rIcn7ZW/+DWZjfX8HBwcHNMf7VWnWkfEwcA3gOVDwVP8AvhERLw+Inpo7hPdmZkbge0RcVppdxmwPjN3ABuAi0t9Mc1ICprR0uLy+WKayQc7OnZQkqRRqz3yuQJ4DfChiPhQqf33zPzjiPgA8LfATJoZckPhtAi4pUzZ/j5wY6kvA74QEcuBnwCXlPoKYE1E/BD457K9JGkSqRI+mfm68vH68m+kNuuAdSPUHwZOGaG+EThrhPqzwHlj760kqdN8woEkqTrDR5JUneEjSarO8JEkVWf4SJKqM3wkSdUZPpKk6gwfSVJ1ho8kqTrDR5JUneEjSaqurfAp798ZXnvDxHdHkvRysM8Hi0bEq8vHuyLiLPa8qG0G8F+B4zrXNUnSVLW/p1rfBvxW+dz6QradwB0d6ZEkacrbZ/hk5tsAImJ1Zr6vTpckSVNdW+/zycz3RcTRwKvZc+mNzPx+pzomSZq62gqfiLgG+DDwNLC7lHcDx3SoX5KkKazdN5kuBo7NzM2d7Iwk6eWh3e/5/G+DR5I0Udod+dwdEZ8A/gb4xVBxf/d8ImI2cB9wbmY+FRFnA9cBs4DbM3N5abcQWAXMBu4BrszMnRExD1gLHAEksCgzt0XEocAXaS77PQNclJk/i4iZwOeAk0o/L83Mx9o8RklSJe2OfC4HLqQJgnXl3z6nWkfEqcC9wPyyPAtYDZwPHA+cHBHnlOZrgaszcz7NhIYlpb4SWJmZxwEPAitK/VpgQ2YeD9wCfKrUPwg8V+q/D6xp8/gkSRW1O9vt9WPY9xLgKuCvyvIpwBOZ+SRARKwFLoyIR4FZmXl/abcGuCYiVgFnAr/TUv974I+Ad5R10HwX6eaImFHqf1z6fE9EHB4R8zLzJ2PovySpQ9qd7faHI9Uz87q9bZOZV5Rth0pzgS0tTbYAR+2jfhiwNTN3Dqv/0r7K5bmtwOH72Ffb4TMwMNBu0xH19fWNa3tNXf39/d3uguen9qr2+dnuPZ8TWj7PBH4TuHuUP2sae6ZpQ3N5bdco6pT6UJtW+9tX2xYsWEBvb+9oNpHa4i9+TWZjPT8HBwfH9Ed7u5fd/kPrckTMpbmxPxqbgDkty0cCm/dRfxo4JCKmZ+YLpc3QjLuflnabIuIA4GCax/8M7etHw/YlSZpExvRKhTLt+nWj3OwBICLi2IiYDlwKrM/MjcD2iDittLus1HcAG4CLS30xsL58vqssU9ZvKO1frEfE6cB27/dI0uQzlns+PTRTmZ8ezQ/KzO0RcTnNTLkDaYJiaMbcIuCWMjX7+8CNpb4M+EJELKe5b3NJqa8A1kTED4F/LtsD3AR8ttQHaYJMkjTJjOWez26aIPhwOxtm5utaPt8NnDhCm4dpZsMNr28Ezhqh/ixw3gj17cB72+mXJKl7RnXPpzxcdEZm/q+O9kqSNKW1e9ntWJqnG8wFpkXE/6F5asE/drJzkqSpqd0JB58GPpGZr8rMQ2ieMHBz57olSZrK2g2f12TmF4YWMvPzNF/qlCRp1NoNnwMi4tVDCxFxGC/9AqgkSW1pd7bbTcD9EXE7Tei8G7i+Y72SJE1p7Y587qIJnZnAG4DXAnd2qlOSpKmt3fBZA9ycmX8EvAf4KM3rESRJGrV2w+ewzLwRmi9yZuYN/PLz2CRJattoJhzMHVqIiNfw0idLS5LUlnYnHFwHPBQRX6O593M2bT5eR5Kk4doa+WTmaprA+QHN66zflpm3drJjkqSpq92RD5n5CPBIB/siSXqZGNP7fCRJGg/DR5JUneEjSarO8JEkVWf4SJKqa3u220SJiCuAq1tKrwf+CnglcDrwXKlfk5l3RsTZNN8zmgXcnpnLy34WAquA2cA9wJWZuTMi5gFrgSOABBZl5rbOH5kkqV3VRz6ZuSozF2bmQmAR8DTwJ8BJwJlD60rwzKJ5htz5wPHAyRFxTtnVWuDqzJxP87SFJaW+EliZmcfRfCdpRa1jkyS1p9uX3f4L8BHg/wHzgNUR8UhEXBMR04BTgCcy88nM3EkTOBdGxNHArMy8v+xnTanPAM4E7mitVzsaSVJbuhY+5XLarMz8MnAk8E3gfcCbgDOA9wNzgS0tm20BjtpH/TBgawmq1rokaRKpfs+nxQdo7uWQmT8GLhhaERE3AYtpRjCtb0ztAXbRhGY7dUq9bQMDA6Np/hJ9fX3j2l5TV39/f7e74Pmpvap9fnYlfCJiJvCbwOVl+QRgfmauK016gB3AJn751Q1HApv3UX8aOCQipmfmC6XN5tH0bcGCBfT29o72kKT98he/JrOxnp+Dg4Nj+qO9W5fdfgN4PDOHZrb1ADdExKvKfZulNG9KfQCIiDg2IqYDlwLrM3MjsD0iTivbX1bqO4ANwMWlvhhYX+eQJEnt6lb4HEMzegFefGjpx4HvAI8CD2XmbZm5nWZ0tK7UH2PPZIJFwPUR8RhwEHBjqS8DlkbEozT3jpZ3/GgkSaPSlctumfkl4EvDaitppkkPb3s3cOII9YdpZsMNr28EzpqovkqSJl63p1pLkl6GDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpugO68UMj4lvAEcCOUvoA8GvAcmAGcENm3lzang1cB8wCbs/M5aW+EFgFzAbuAa7MzJ0RMQ9YW/afwKLM3Fbr2CRJ+1d95BMRPcB84MTMXJiZC4FNwJ8BpwMLgaUR8YaImAWsBs4HjgdOjohzyq7WAldn5nygB1hS6iuBlZl5HPAgsKLSoUmS2tSNy25R/vuNiHg4Iq4Gzga+mZnPZuZzwB3Au4BTgCcy88nM3EkTOBdGxNHArMy8v+xrTanPAM4s279Yr3FQkqT2deOy26uAu4HfpbnE9m3gdmBLS5stNMEzd4T6UfuoHwZsLUHVWm/bwMDAaJq/RF9f37i219TV39/f7S54fmqvap+f1cMnM/8B+Ieh5Yj4HM09nWtbmvUAu2hGZrvHUafU27ZgwQJ6e3tHs4nUFn/xazIb6/k5ODg4pj/au3HP5/SIeEtLqQd4CpjTUjsS2ExzL2g09aeBQyJieqnPKXVJ0iTSjXs+hwJ/EREHRsTBwHuB9wBviYjDI+IVwDuBrwEPABERx5ZAuRRYn5kbge0RcVrZ52WlvgPYAFxc6ouB9dWOTJLUlurhk5lfAb4K/ADoB1Zn5neAjwLfAh4Cbs3M72bmduByYB3wKPAYeyYTLAKuj4jHgIOAG0t9Gc1suUeBM2imb0uSJpGufM8nM1cwbAp0Zt4K3DpC27uBE0eoP0wzKWF4fSNw1kT1VZI08XzCgSSpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklSd4SNJqs7wkSRVZ/hIkqozfCRJ1Rk+kqTqDB9JUnWGjySpOsNHklRdV16jHREfAy4qi1/NzP8UEZ8HTgeeK/VrMvPOiDgbuA6YBdyemcvLPhYCq4DZwD3AlZm5MyLmAWuBI4AEFmXmtlrHJknav+ojnxImbwX+LbAQ6IuIC4CTgDMzc2H5d2dEzAJWA+cDxwMnR8Q5ZVdrgaszcz7QAywp9ZXAysw8DngQWFHr2CRJ7enGZbctwIcy8/nM3AH8IzCv/FsdEY9ExDURMQ04BXgiM5/MzJ00gXNhRBwNzMrM+8s+15T6DOBM4I7Weq0DkyS1p/plt8z84dDniPh1mstvZwBnAcuAfwG+Arwf2EYTVkO2AEcBc/dSPwzYWoKqtS5JmkS6cs8HICL+DfBV4MOZmcAFLetuAhbTjGB2t2zWA+yiGbG1U6fU2zYwMDCa5i/R19c3ru01dfX393e7C56f2qva52e3JhycBqwDfj8z/zoiTgDmZ+a60qQH2AFsAua0bHoksHkf9aeBQyJiema+UNpsHk3fFixYQG9v7xiOSto3f/FrMhvr+Tk4ODimP9q7MeHgXwP/Dbg0M/+6lHuAGyLiVeW+zVLgTuCBZpM4NiKmA5cC6zNzI7C9hBjAZaW+A9gAXFzqi4H1VQ5MktS2box8/iNwIHBdRAzVPgN8HPgOMANYl5m3AUTE5TSjpAOBu9gzmWARcEtEzAa+D9xY6suAL0TEcuAnwCUdPh5J0ih1Y8LB7wG/t5fVK0dofzdw4gj1h2lmww2vb6SZvCBJmqR8woEkqTrDR5JUneEjSarO8JEkVWf4SJKqM3wkSdUZPpKk6gwfSVJ1ho8kqTrDR5JUneEjSarO8JEkVWf4SJKqM3wkSdUZPpKk6gwfSVJ1ho8kqTrDR5JUneEjSarugG53oBMi4lJgOTADuCEzb+5ylyRJLabcyCciXgv8GXA6sBBYGhFv6G6vJEmtpuLI52zgm5n5LEBE3AG8C/jT/Ww3HeD5558ffw9eedD496EpZXBwsNtdeJFnp4Ybz/nZ8jtz+mi2m4rhMxfY0rK8BTilje3mADz++OPj7kDPuy4Z9z40tQwMDHS7Cy+6pLen213QJDNB5+cc4EftNp6K4TMN2N2y3APsamO77wFn0ITVCx3olyRNRdNpgud7o9loKobPJpoQGXIksHl/G/X19Q0C93aqU5I0hbU94hkyFcPn74A/iYjDgeeAdwJLu9slSVKrKTfbLTN/CnwU+BbwEHBrZn63u72SJLXq2b179/5bSZI0gabcyEeSNPkZPpKk6gwfSVJ1ho8kqbqpONVale3vQa4RsRBYBcwG7gGuzMyd1Tuql6WImA3cB5ybmU8NW+e52SWOfDQubT7IdS1wdWbOp3nixJK6vdTLVUScSvPl8fl7aeK52SWGj8brxQe5ZuZzwNCDXAGIiKOBWZl5fymtAS6s3ku9XC0BrmKEp5x4bnaXl900Xvt7kOtI64+q0C+JzLwCICJGWu252UWOfDRe+3uQ61gf9Cp1mudmFxk+Gq9NlNdRFMMf5Lq/9VK3eG52keGj8fo74C0RcXhEvILmQa5fG1qZmRuB7RFxWildBqyv303pl3ludpfho3HZ24NcI+KuiDipNFsEXB8Rj9G8SPPG7vRWAs/NycEHi0qSqnPkI0mqzvCRJFVn+EiSqjN8JEnVGT6SpOoMH0lSdT7bTZogEXE58CngyVLqoXlU/wZgaWZuj4g1wG8Bzwzb/O3AW4F3Zea5w/b7beDTZfEW4ITM3NSy/hPAqcC/A34EDAK/oHl0zEzgG8CHMnNXRDzVsr7Vssy8LyJ+l+Z7Wz8r9f+bmWeM5v+D1A7DR5pYG1rDIyIOpHmk/3uBz5by9Zn5l8M33MvDL1+UmXdExDnA5yPirZm5OyLOABYDb8zMF8o+FmXmg2WfM4G/B5axJ8BeXD+CNwN/mJm3tne40th42U3qrH8FHAI8O0H7+yAwD1gWEQfRvAbg8swc8Zlkmfk8zcjruDb3/2ZgUUQ8EhFfj4gTJqDP0ks48pEm1hkR8RBwIE3wPA78ZWZ+uaXNH0TEe1qWP52Zq4Zt3+rYoQ+Z+VxEXEJzKe0s4EuZ+TX2IiLmAr9N86bZIV+MiNbLboOZeWpEvBJ4DPjzzLwnIi4C1kfEcZm5rY1jl9pm+EgTa0NmnhsR02h+4V8KfHlYmxEvu7Vu31oo93xelJnfj4iVwAXAJSPsYyhcpgE7gFWZua5l/YiX3crLAN/WsvyliFgBnEzz7D5pwhg+Ugdk5i7gTyPizTSXxs6b4B/xY2BjZu4cYd2+7unsVXmz53mZeVNLuYcmwKQJ5T0fqbOuAt4aEed3uyNteA64NiJOAYiItwOvAL7b1V5pSnLkI3VQZv4oIv6c5rH9X+92f4rh93yg3Hcq93k+W2bJbQUuKJMWpAnlKxUkSdU58pFGIZov0ty+l9WZmRfX7I/0q8qRjySpOiccSJKqM3wkSdUZPpKk6gwfSVJ1ho8kqbr/D8kACngSf9+aAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot showing that Hypertension group frequencies are balanced after SMOTe\n",
    "\n",
    "sns.countplot(x='_RFHYPE5', data=os_y_hypertension, palette='hls')\n",
    "plt.show\n",
    "plt.savefig('Hypertension_Count_Plot_After_SMOTe')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Part 1G: Standardize continuous predictors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 257,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating dataframes of the standardized, continuous variables only\n",
    "# Standardizing X_train and X_test for each disease dataframe\n",
    "\n",
    "# Diabetes\n",
    "\n",
    "diabetes_cols_to_standardize = [col_name for col_name in os_X_diabetes.columns if col_name in countinuous_features_17]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 307,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler_train = preprocessing.StandardScaler()\n",
    "diabetes_scaled_X_train = scaler_train.fit_transform(os_X_diabetes[diabetes_cols_to_standardize])\n",
    "diabetes_scaled_df_X_train = pd.DataFrame(diabetes_scaled_X_train, columns=diabetes_cols_to_standardize, index=os_X_diabetes.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 308,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler_test = preprocessing.StandardScaler()\n",
    "diabetes_scaled_X_test = scaler_test.fit_transform(X_test_diabetes[diabetes_cols_to_standardize])\n",
    "diabetes_scaled_df_X_test = pd.DataFrame(diabetes_scaled_X_test, columns=diabetes_cols_to_standardize, index=X_test_diabetes.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 310,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NUMADULT</th>\n",
       "      <th>NUMMEN</th>\n",
       "      <th>NUMWOMEN</th>\n",
       "      <th>HHADULT</th>\n",
       "      <th>PHYSHLTH</th>\n",
       "      <th>MENTHLTH</th>\n",
       "      <th>POORHLTH</th>\n",
       "      <th>CHILDREN</th>\n",
       "      <th>DRNK3GE5</th>\n",
       "      <th>MAXDRNKS</th>\n",
       "      <th>...</th>\n",
       "      <th>PAFREQ2_</th>\n",
       "      <th>_MINAC11</th>\n",
       "      <th>_MINAC21</th>\n",
       "      <th>STRFREQ_</th>\n",
       "      <th>PAMIN11_</th>\n",
       "      <th>PAMIN21_</th>\n",
       "      <th>PA1MIN_</th>\n",
       "      <th>PAVIG11_</th>\n",
       "      <th>PAVIG21_</th>\n",
       "      <th>PA1VIGM_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.239580</td>\n",
       "      <td>0.271672</td>\n",
       "      <td>0.038185</td>\n",
       "      <td>-0.078883</td>\n",
       "      <td>-0.412726</td>\n",
       "      <td>2.083042</td>\n",
       "      <td>-0.090665</td>\n",
       "      <td>-0.457995</td>\n",
       "      <td>-0.180634</td>\n",
       "      <td>-0.371271</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.396751</td>\n",
       "      <td>0.422779</td>\n",
       "      <td>0.081755</td>\n",
       "      <td>-0.461431</td>\n",
       "      <td>0.174773</td>\n",
       "      <td>-0.011084</td>\n",
       "      <td>0.108908</td>\n",
       "      <td>-0.253613</td>\n",
       "      <td>-0.171797</td>\n",
       "      <td>-0.285183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.239580</td>\n",
       "      <td>0.271672</td>\n",
       "      <td>0.038185</td>\n",
       "      <td>1.465644</td>\n",
       "      <td>-0.628638</td>\n",
       "      <td>3.379311</td>\n",
       "      <td>2.128224</td>\n",
       "      <td>1.718230</td>\n",
       "      <td>-0.180634</td>\n",
       "      <td>0.083050</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.219681</td>\n",
       "      <td>0.250380</td>\n",
       "      <td>-0.237674</td>\n",
       "      <td>-0.122131</td>\n",
       "      <td>0.526441</td>\n",
       "      <td>-0.236915</td>\n",
       "      <td>0.197554</td>\n",
       "      <td>0.804137</td>\n",
       "      <td>-0.171797</td>\n",
       "      <td>0.455363</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.122326</td>\n",
       "      <td>2.653157</td>\n",
       "      <td>0.038185</td>\n",
       "      <td>-0.078883</td>\n",
       "      <td>-0.412726</td>\n",
       "      <td>-0.509498</td>\n",
       "      <td>-0.482234</td>\n",
       "      <td>-0.457995</td>\n",
       "      <td>-0.180634</td>\n",
       "      <td>-0.371271</td>\n",
       "      <td>...</td>\n",
       "      <td>2.192715</td>\n",
       "      <td>0.250380</td>\n",
       "      <td>0.218653</td>\n",
       "      <td>2.931571</td>\n",
       "      <td>0.526441</td>\n",
       "      <td>0.408318</td>\n",
       "      <td>0.611238</td>\n",
       "      <td>0.804137</td>\n",
       "      <td>0.616268</td>\n",
       "      <td>0.949060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-1.643167</td>\n",
       "      <td>-2.109812</td>\n",
       "      <td>0.038185</td>\n",
       "      <td>-0.078883</td>\n",
       "      <td>2.610046</td>\n",
       "      <td>-0.509498</td>\n",
       "      <td>3.433453</td>\n",
       "      <td>0.630118</td>\n",
       "      <td>-0.180634</td>\n",
       "      <td>-0.371271</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.219681</td>\n",
       "      <td>-0.445847</td>\n",
       "      <td>-0.237674</td>\n",
       "      <td>-0.461431</td>\n",
       "      <td>-0.408372</td>\n",
       "      <td>-0.236915</td>\n",
       "      <td>-0.422971</td>\n",
       "      <td>-0.253613</td>\n",
       "      <td>-0.171797</td>\n",
       "      <td>-0.285183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-1.643167</td>\n",
       "      <td>-2.109812</td>\n",
       "      <td>0.038185</td>\n",
       "      <td>-0.078883</td>\n",
       "      <td>-0.628638</td>\n",
       "      <td>-0.509498</td>\n",
       "      <td>-0.482234</td>\n",
       "      <td>-0.457995</td>\n",
       "      <td>-0.180634</td>\n",
       "      <td>-0.371271</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.219681</td>\n",
       "      <td>-0.445847</td>\n",
       "      <td>-0.237674</td>\n",
       "      <td>-0.461431</td>\n",
       "      <td>-0.408372</td>\n",
       "      <td>-0.236915</td>\n",
       "      <td>-0.422971</td>\n",
       "      <td>-0.253613</td>\n",
       "      <td>-0.171797</td>\n",
       "      <td>-0.285183</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 42 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   NUMADULT    NUMMEN  NUMWOMEN   HHADULT  PHYSHLTH  MENTHLTH  POORHLTH  \\\n",
       "0  0.239580  0.271672  0.038185 -0.078883 -0.412726  2.083042 -0.090665   \n",
       "1  0.239580  0.271672  0.038185  1.465644 -0.628638  3.379311  2.128224   \n",
       "2  2.122326  2.653157  0.038185 -0.078883 -0.412726 -0.509498 -0.482234   \n",
       "3 -1.643167 -2.109812  0.038185 -0.078883  2.610046 -0.509498  3.433453   \n",
       "4 -1.643167 -2.109812  0.038185 -0.078883 -0.628638 -0.509498 -0.482234   \n",
       "\n",
       "   CHILDREN  DRNK3GE5  MAXDRNKS    ...     PAFREQ2_  _MINAC11  _MINAC21  \\\n",
       "0 -0.457995 -0.180634 -0.371271    ...    -0.396751  0.422779  0.081755   \n",
       "1  1.718230 -0.180634  0.083050    ...    -0.219681  0.250380 -0.237674   \n",
       "2 -0.457995 -0.180634 -0.371271    ...     2.192715  0.250380  0.218653   \n",
       "3  0.630118 -0.180634 -0.371271    ...    -0.219681 -0.445847 -0.237674   \n",
       "4 -0.457995 -0.180634 -0.371271    ...    -0.219681 -0.445847 -0.237674   \n",
       "\n",
       "   STRFREQ_  PAMIN11_  PAMIN21_   PA1MIN_  PAVIG11_  PAVIG21_  PA1VIGM_  \n",
       "0 -0.461431  0.174773 -0.011084  0.108908 -0.253613 -0.171797 -0.285183  \n",
       "1 -0.122131  0.526441 -0.236915  0.197554  0.804137 -0.171797  0.455363  \n",
       "2  2.931571  0.526441  0.408318  0.611238  0.804137  0.616268  0.949060  \n",
       "3 -0.461431 -0.408372 -0.236915 -0.422971 -0.253613 -0.171797 -0.285183  \n",
       "4 -0.461431 -0.408372 -0.236915 -0.422971 -0.253613 -0.171797 -0.285183  \n",
       "\n",
       "[5 rows x 42 columns]"
      ]
     },
     "execution_count": 310,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View on Diabetes standardized dataframes\n",
    "\n",
    "diabetes_scaled_df_X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 311,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NUMADULT</th>\n",
       "      <th>NUMMEN</th>\n",
       "      <th>NUMWOMEN</th>\n",
       "      <th>HHADULT</th>\n",
       "      <th>PHYSHLTH</th>\n",
       "      <th>MENTHLTH</th>\n",
       "      <th>POORHLTH</th>\n",
       "      <th>CHILDREN</th>\n",
       "      <th>DRNK3GE5</th>\n",
       "      <th>MAXDRNKS</th>\n",
       "      <th>...</th>\n",
       "      <th>PAFREQ2_</th>\n",
       "      <th>_MINAC11</th>\n",
       "      <th>_MINAC21</th>\n",
       "      <th>STRFREQ_</th>\n",
       "      <th>PAMIN11_</th>\n",
       "      <th>PAMIN21_</th>\n",
       "      <th>PA1MIN_</th>\n",
       "      <th>PAVIG11_</th>\n",
       "      <th>PAVIG21_</th>\n",
       "      <th>PA1VIGM_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>63205</th>\n",
       "      <td>0.196005</td>\n",
       "      <td>0.221056</td>\n",
       "      <td>0.032129</td>\n",
       "      <td>-0.092653</td>\n",
       "      <td>-0.489325</td>\n",
       "      <td>-0.449559</td>\n",
       "      <td>-0.380048</td>\n",
       "      <td>3.392356</td>\n",
       "      <td>-0.204189</td>\n",
       "      <td>-0.414822</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.230059</td>\n",
       "      <td>0.066080</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>-0.510207</td>\n",
       "      <td>-0.048119</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.193663</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>153317</th>\n",
       "      <td>0.196005</td>\n",
       "      <td>0.221056</td>\n",
       "      <td>0.032129</td>\n",
       "      <td>-0.092653</td>\n",
       "      <td>2.911358</td>\n",
       "      <td>-0.449559</td>\n",
       "      <td>-0.380048</td>\n",
       "      <td>-0.495660</td>\n",
       "      <td>-0.204189</td>\n",
       "      <td>-0.414822</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.230059</td>\n",
       "      <td>-0.396371</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>-0.510207</td>\n",
       "      <td>-0.380168</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.442049</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>209908</th>\n",
       "      <td>0.196005</td>\n",
       "      <td>0.221056</td>\n",
       "      <td>0.032129</td>\n",
       "      <td>-0.092653</td>\n",
       "      <td>0.304168</td>\n",
       "      <td>1.325720</td>\n",
       "      <td>0.588401</td>\n",
       "      <td>-0.495660</td>\n",
       "      <td>-0.204189</td>\n",
       "      <td>-0.414822</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.230059</td>\n",
       "      <td>-0.396371</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>-0.510207</td>\n",
       "      <td>-0.380168</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.442049</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60268</th>\n",
       "      <td>0.196005</td>\n",
       "      <td>0.221056</td>\n",
       "      <td>0.032129</td>\n",
       "      <td>1.258168</td>\n",
       "      <td>0.190812</td>\n",
       "      <td>-0.449559</td>\n",
       "      <td>0.035001</td>\n",
       "      <td>-0.495660</td>\n",
       "      <td>-0.204189</td>\n",
       "      <td>-0.414822</td>\n",
       "      <td>...</td>\n",
       "      <td>0.707573</td>\n",
       "      <td>-0.396371</td>\n",
       "      <td>0.311587</td>\n",
       "      <td>0.403819</td>\n",
       "      <td>-0.380168</td>\n",
       "      <td>0.160581</td>\n",
       "      <td>-0.193663</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119979</th>\n",
       "      <td>0.196005</td>\n",
       "      <td>0.221056</td>\n",
       "      <td>0.032129</td>\n",
       "      <td>-0.092653</td>\n",
       "      <td>2.911358</td>\n",
       "      <td>-0.449559</td>\n",
       "      <td>-0.380048</td>\n",
       "      <td>1.448348</td>\n",
       "      <td>-0.204189</td>\n",
       "      <td>-0.414822</td>\n",
       "      <td>...</td>\n",
       "      <td>1.020274</td>\n",
       "      <td>-0.036686</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>1.622672</td>\n",
       "      <td>-0.121908</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.248860</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 42 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        NUMADULT    NUMMEN  NUMWOMEN   HHADULT  PHYSHLTH  MENTHLTH  POORHLTH  \\\n",
       "63205   0.196005  0.221056  0.032129 -0.092653 -0.489325 -0.449559 -0.380048   \n",
       "153317  0.196005  0.221056  0.032129 -0.092653  2.911358 -0.449559 -0.380048   \n",
       "209908  0.196005  0.221056  0.032129 -0.092653  0.304168  1.325720  0.588401   \n",
       "60268   0.196005  0.221056  0.032129  1.258168  0.190812 -0.449559  0.035001   \n",
       "119979  0.196005  0.221056  0.032129 -0.092653  2.911358 -0.449559 -0.380048   \n",
       "\n",
       "        CHILDREN  DRNK3GE5  MAXDRNKS    ...     PAFREQ2_  _MINAC11  _MINAC21  \\\n",
       "63205   3.392356 -0.204189 -0.414822    ...    -0.230059  0.066080 -0.265828   \n",
       "153317 -0.495660 -0.204189 -0.414822    ...    -0.230059 -0.396371 -0.265828   \n",
       "209908 -0.495660 -0.204189 -0.414822    ...    -0.230059 -0.396371 -0.265828   \n",
       "60268  -0.495660 -0.204189 -0.414822    ...     0.707573 -0.396371  0.311587   \n",
       "119979  1.448348 -0.204189 -0.414822    ...     1.020274 -0.036686 -0.265828   \n",
       "\n",
       "        STRFREQ_  PAMIN11_  PAMIN21_   PA1MIN_  PAVIG11_  PAVIG21_  PA1VIGM_  \n",
       "63205  -0.510207 -0.048119 -0.279088 -0.193663 -0.237113 -0.205763 -0.297649  \n",
       "153317 -0.510207 -0.380168 -0.279088 -0.442049 -0.237113 -0.205763 -0.297649  \n",
       "209908 -0.510207 -0.380168 -0.279088 -0.442049 -0.237113 -0.205763 -0.297649  \n",
       "60268   0.403819 -0.380168  0.160581 -0.193663 -0.237113 -0.205763 -0.297649  \n",
       "119979  1.622672 -0.121908 -0.279088 -0.248860 -0.237113 -0.205763 -0.297649  \n",
       "\n",
       "[5 rows x 42 columns]"
      ]
     },
     "execution_count": 311,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diabetes_scaled_df_X_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 297,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop unstandardized versions of continuous features for X train and X test dataframes\n",
    "\n",
    "os_X_diabetes_drop = os_X_diabetes.drop(columns = diabetes_cols_to_standardize)\n",
    "X_test_diabetes_drop = X_test_diabetes.drop(columns = diabetes_cols_to_standardize)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 298,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Join standardized versions of continuous features\n",
    "\n",
    "diabetes_scaled_df_X_train_final = os_X_diabetes_drop.join(diabetes_scaled_df_X_train)\n",
    "diabetes_scaled_df_X_test_final = X_test_diabetes_drop.join(diabetes_scaled_df_X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 315,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(540712, 522)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CTELENM1</th>\n",
       "      <th>STATERE1</th>\n",
       "      <th>CTELNUM1</th>\n",
       "      <th>CELLFON5</th>\n",
       "      <th>CVDSTRK3</th>\n",
       "      <th>CHCSCNCR</th>\n",
       "      <th>CHCOCNCR</th>\n",
       "      <th>CHCCOPD1</th>\n",
       "      <th>ADDEPEV2</th>\n",
       "      <th>CHCKIDNY</th>\n",
       "      <th>...</th>\n",
       "      <th>PAFREQ2_</th>\n",
       "      <th>_MINAC11</th>\n",
       "      <th>_MINAC21</th>\n",
       "      <th>STRFREQ_</th>\n",
       "      <th>PAMIN11_</th>\n",
       "      <th>PAMIN21_</th>\n",
       "      <th>PA1MIN_</th>\n",
       "      <th>PAVIG11_</th>\n",
       "      <th>PAVIG21_</th>\n",
       "      <th>PA1VIGM_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.396751</td>\n",
       "      <td>0.422779</td>\n",
       "      <td>0.081755</td>\n",
       "      <td>-0.461431</td>\n",
       "      <td>0.174773</td>\n",
       "      <td>-0.011084</td>\n",
       "      <td>0.108908</td>\n",
       "      <td>-0.253613</td>\n",
       "      <td>-0.171797</td>\n",
       "      <td>-0.285183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.219681</td>\n",
       "      <td>0.250380</td>\n",
       "      <td>-0.237674</td>\n",
       "      <td>-0.122131</td>\n",
       "      <td>0.526441</td>\n",
       "      <td>-0.236915</td>\n",
       "      <td>0.197554</td>\n",
       "      <td>0.804137</td>\n",
       "      <td>-0.171797</td>\n",
       "      <td>0.455363</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.192715</td>\n",
       "      <td>0.250380</td>\n",
       "      <td>0.218653</td>\n",
       "      <td>2.931571</td>\n",
       "      <td>0.526441</td>\n",
       "      <td>0.408318</td>\n",
       "      <td>0.611238</td>\n",
       "      <td>0.804137</td>\n",
       "      <td>0.616268</td>\n",
       "      <td>0.949060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.219681</td>\n",
       "      <td>-0.445847</td>\n",
       "      <td>-0.237674</td>\n",
       "      <td>-0.461431</td>\n",
       "      <td>-0.408372</td>\n",
       "      <td>-0.236915</td>\n",
       "      <td>-0.422971</td>\n",
       "      <td>-0.253613</td>\n",
       "      <td>-0.171797</td>\n",
       "      <td>-0.285183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.219681</td>\n",
       "      <td>-0.445847</td>\n",
       "      <td>-0.237674</td>\n",
       "      <td>-0.461431</td>\n",
       "      <td>-0.408372</td>\n",
       "      <td>-0.236915</td>\n",
       "      <td>-0.422971</td>\n",
       "      <td>-0.253613</td>\n",
       "      <td>-0.171797</td>\n",
       "      <td>-0.285183</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 522 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   CTELENM1  STATERE1  CTELNUM1  CELLFON5  CVDSTRK3  CHCSCNCR  CHCOCNCR  \\\n",
       "0       1.0       1.0       1.0       1.0       0.0       0.0       0.0   \n",
       "1       1.0       1.0       1.0       1.0       0.0       0.0       0.0   \n",
       "2       1.0       1.0       1.0       1.0       0.0       0.0       1.0   \n",
       "3       1.0       1.0       1.0       1.0       1.0       0.0       0.0   \n",
       "4       1.0       1.0       1.0       1.0       0.0       0.0       0.0   \n",
       "\n",
       "   CHCCOPD1  ADDEPEV2  CHCKIDNY    ...     PAFREQ2_  _MINAC11  _MINAC21  \\\n",
       "0       0.0       1.0       0.0    ...    -0.396751  0.422779  0.081755   \n",
       "1       0.0       1.0       0.0    ...    -0.219681  0.250380 -0.237674   \n",
       "2       0.0       0.0       1.0    ...     2.192715  0.250380  0.218653   \n",
       "3       0.0       0.0       0.0    ...    -0.219681 -0.445847 -0.237674   \n",
       "4       1.0       0.0       0.0    ...    -0.219681 -0.445847 -0.237674   \n",
       "\n",
       "   STRFREQ_  PAMIN11_  PAMIN21_   PA1MIN_  PAVIG11_  PAVIG21_  PA1VIGM_  \n",
       "0 -0.461431  0.174773 -0.011084  0.108908 -0.253613 -0.171797 -0.285183  \n",
       "1 -0.122131  0.526441 -0.236915  0.197554  0.804137 -0.171797  0.455363  \n",
       "2  2.931571  0.526441  0.408318  0.611238  0.804137  0.616268  0.949060  \n",
       "3 -0.461431 -0.408372 -0.236915 -0.422971 -0.253613 -0.171797 -0.285183  \n",
       "4 -0.461431 -0.408372 -0.236915 -0.422971 -0.253613 -0.171797 -0.285183  \n",
       "\n",
       "[5 rows x 522 columns]"
      ]
     },
     "execution_count": 315,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View on final X train and X test datasets - dummy coded & standardized\n",
    "\n",
    "print(diabetes_scaled_df_X_train_final.shape)\n",
    "#print(diabetes_scaled_df_X_train_final.isna().sum())\n",
    "diabetes_scaled_df_X_train_final.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 314,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(135005, 522)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CTELENM1</th>\n",
       "      <th>STATERE1</th>\n",
       "      <th>CTELNUM1</th>\n",
       "      <th>CELLFON5</th>\n",
       "      <th>CVDSTRK3</th>\n",
       "      <th>CHCSCNCR</th>\n",
       "      <th>CHCOCNCR</th>\n",
       "      <th>CHCCOPD1</th>\n",
       "      <th>ADDEPEV2</th>\n",
       "      <th>CHCKIDNY</th>\n",
       "      <th>...</th>\n",
       "      <th>PAFREQ2_</th>\n",
       "      <th>_MINAC11</th>\n",
       "      <th>_MINAC21</th>\n",
       "      <th>STRFREQ_</th>\n",
       "      <th>PAMIN11_</th>\n",
       "      <th>PAMIN21_</th>\n",
       "      <th>PA1MIN_</th>\n",
       "      <th>PAVIG11_</th>\n",
       "      <th>PAVIG21_</th>\n",
       "      <th>PA1VIGM_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>63205</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.230059</td>\n",
       "      <td>0.066080</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>-0.510207</td>\n",
       "      <td>-0.048119</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.193663</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>153317</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.230059</td>\n",
       "      <td>-0.396371</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>-0.510207</td>\n",
       "      <td>-0.380168</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.442049</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>209908</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.230059</td>\n",
       "      <td>-0.396371</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>-0.510207</td>\n",
       "      <td>-0.380168</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.442049</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60268</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.707573</td>\n",
       "      <td>-0.396371</td>\n",
       "      <td>0.311587</td>\n",
       "      <td>0.403819</td>\n",
       "      <td>-0.380168</td>\n",
       "      <td>0.160581</td>\n",
       "      <td>-0.193663</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119979</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.020274</td>\n",
       "      <td>-0.036686</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>1.622672</td>\n",
       "      <td>-0.121908</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.248860</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 522 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        CTELENM1  STATERE1  CTELNUM1  CELLFON5  CVDSTRK3  CHCSCNCR  CHCOCNCR  \\\n",
       "63205        1.0       1.0       1.0       1.0       0.0       0.0       0.0   \n",
       "153317       1.0       1.0       1.0       1.0       0.0       1.0       1.0   \n",
       "209908       1.0       1.0       1.0       1.0       0.0       0.0       0.0   \n",
       "60268        1.0       1.0       1.0       1.0       1.0       0.0       0.0   \n",
       "119979       1.0       1.0       1.0       1.0       0.0       0.0       0.0   \n",
       "\n",
       "        CHCCOPD1  ADDEPEV2  CHCKIDNY    ...     PAFREQ2_  _MINAC11  _MINAC21  \\\n",
       "63205        0.0       0.0       0.0    ...    -0.230059  0.066080 -0.265828   \n",
       "153317       1.0       0.0       0.0    ...    -0.230059 -0.396371 -0.265828   \n",
       "209908       0.0       1.0       0.0    ...    -0.230059 -0.396371 -0.265828   \n",
       "60268        0.0       1.0       0.0    ...     0.707573 -0.396371  0.311587   \n",
       "119979       0.0       0.0       0.0    ...     1.020274 -0.036686 -0.265828   \n",
       "\n",
       "        STRFREQ_  PAMIN11_  PAMIN21_   PA1MIN_  PAVIG11_  PAVIG21_  PA1VIGM_  \n",
       "63205  -0.510207 -0.048119 -0.279088 -0.193663 -0.237113 -0.205763 -0.297649  \n",
       "153317 -0.510207 -0.380168 -0.279088 -0.442049 -0.237113 -0.205763 -0.297649  \n",
       "209908 -0.510207 -0.380168 -0.279088 -0.442049 -0.237113 -0.205763 -0.297649  \n",
       "60268   0.403819 -0.380168  0.160581 -0.193663 -0.237113 -0.205763 -0.297649  \n",
       "119979  1.622672 -0.121908 -0.279088 -0.248860 -0.237113 -0.205763 -0.297649  \n",
       "\n",
       "[5 rows x 522 columns]"
      ]
     },
     "execution_count": 314,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(diabetes_scaled_df_X_test_final.shape)\n",
    "#print(diabetes_scaled_df_X_test_final.isna().sum())\n",
    "diabetes_scaled_df_X_test_final.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 316,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hypertension\n",
    "\n",
    "hypertension_cols_to_standardize = [col_name for col_name in os_X_hypertension.columns if col_name in countinuous_features_17]\n",
    "\n",
    "scaler_train = preprocessing.StandardScaler()\n",
    "hypertension_scaled_X_train = scaler_train.fit_transform(os_X_hypertension[hypertension_cols_to_standardize])\n",
    "hypertension_scaled_df_X_train = pd.DataFrame(hypertension_scaled_X_train, columns=hypertension_cols_to_standardize, index=os_X_hypertension.index)\n",
    "\n",
    "scaler_test = preprocessing.StandardScaler()\n",
    "hypertension_scaled_X_test = scaler_test.fit_transform(X_test_hypertension[hypertension_cols_to_standardize])\n",
    "hypertension_scaled_df_X_test = pd.DataFrame(hypertension_scaled_X_test, columns=hypertension_cols_to_standardize, index=X_test_hypertension.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 317,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NUMADULT</th>\n",
       "      <th>NUMMEN</th>\n",
       "      <th>NUMWOMEN</th>\n",
       "      <th>HHADULT</th>\n",
       "      <th>PHYSHLTH</th>\n",
       "      <th>MENTHLTH</th>\n",
       "      <th>POORHLTH</th>\n",
       "      <th>CHILDREN</th>\n",
       "      <th>DRNK3GE5</th>\n",
       "      <th>MAXDRNKS</th>\n",
       "      <th>...</th>\n",
       "      <th>PAFREQ2_</th>\n",
       "      <th>_MINAC11</th>\n",
       "      <th>_MINAC21</th>\n",
       "      <th>STRFREQ_</th>\n",
       "      <th>PAMIN11_</th>\n",
       "      <th>PAMIN21_</th>\n",
       "      <th>PA1MIN_</th>\n",
       "      <th>PAVIG11_</th>\n",
       "      <th>PAVIG21_</th>\n",
       "      <th>PA1VIGM_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.213470</td>\n",
       "      <td>0.242765</td>\n",
       "      <td>0.033966</td>\n",
       "      <td>-0.084923</td>\n",
       "      <td>-0.295156</td>\n",
       "      <td>2.129153</td>\n",
       "      <td>0.008536</td>\n",
       "      <td>-0.473809</td>\n",
       "      <td>-0.203588</td>\n",
       "      <td>-0.415286</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.397997</td>\n",
       "      <td>0.340852</td>\n",
       "      <td>0.055056</td>\n",
       "      <td>-0.498281</td>\n",
       "      <td>0.122166</td>\n",
       "      <td>-0.030886</td>\n",
       "      <td>0.063706</td>\n",
       "      <td>-0.251007</td>\n",
       "      <td>-0.174378</td>\n",
       "      <td>-0.287073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.213470</td>\n",
       "      <td>0.242765</td>\n",
       "      <td>0.033966</td>\n",
       "      <td>1.343546</td>\n",
       "      <td>-0.522346</td>\n",
       "      <td>3.425834</td>\n",
       "      <td>2.353060</td>\n",
       "      <td>1.545861</td>\n",
       "      <td>-0.203588</td>\n",
       "      <td>-0.013595</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.226111</td>\n",
       "      <td>0.183302</td>\n",
       "      <td>-0.249146</td>\n",
       "      <td>-0.172802</td>\n",
       "      <td>0.445008</td>\n",
       "      <td>-0.247397</td>\n",
       "      <td>0.147148</td>\n",
       "      <td>0.718013</td>\n",
       "      <td>-0.174378</td>\n",
       "      <td>0.406465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.084743</td>\n",
       "      <td>2.614122</td>\n",
       "      <td>0.033966</td>\n",
       "      <td>-0.084923</td>\n",
       "      <td>-0.295156</td>\n",
       "      <td>-0.464211</td>\n",
       "      <td>-0.405203</td>\n",
       "      <td>-0.473809</td>\n",
       "      <td>-0.203588</td>\n",
       "      <td>-0.415286</td>\n",
       "      <td>...</td>\n",
       "      <td>2.115670</td>\n",
       "      <td>0.183302</td>\n",
       "      <td>0.185428</td>\n",
       "      <td>2.756505</td>\n",
       "      <td>0.445008</td>\n",
       "      <td>0.371207</td>\n",
       "      <td>0.536545</td>\n",
       "      <td>0.718013</td>\n",
       "      <td>0.576150</td>\n",
       "      <td>0.868824</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-1.657803</td>\n",
       "      <td>-2.128593</td>\n",
       "      <td>0.033966</td>\n",
       "      <td>-0.084923</td>\n",
       "      <td>2.885508</td>\n",
       "      <td>-0.464211</td>\n",
       "      <td>3.732191</td>\n",
       "      <td>0.536026</td>\n",
       "      <td>-0.203588</td>\n",
       "      <td>-0.415286</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.226111</td>\n",
       "      <td>-0.452958</td>\n",
       "      <td>-0.249146</td>\n",
       "      <td>-0.498281</td>\n",
       "      <td>-0.413180</td>\n",
       "      <td>-0.247397</td>\n",
       "      <td>-0.436947</td>\n",
       "      <td>-0.251007</td>\n",
       "      <td>-0.174378</td>\n",
       "      <td>-0.287073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-1.657803</td>\n",
       "      <td>-2.128593</td>\n",
       "      <td>0.033966</td>\n",
       "      <td>-0.084923</td>\n",
       "      <td>-0.522346</td>\n",
       "      <td>-0.464211</td>\n",
       "      <td>-0.405203</td>\n",
       "      <td>-0.473809</td>\n",
       "      <td>-0.203588</td>\n",
       "      <td>-0.415286</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.226111</td>\n",
       "      <td>-0.452958</td>\n",
       "      <td>-0.249146</td>\n",
       "      <td>-0.498281</td>\n",
       "      <td>-0.413180</td>\n",
       "      <td>-0.247397</td>\n",
       "      <td>-0.436947</td>\n",
       "      <td>-0.251007</td>\n",
       "      <td>-0.174378</td>\n",
       "      <td>-0.287073</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 42 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   NUMADULT    NUMMEN  NUMWOMEN   HHADULT  PHYSHLTH  MENTHLTH  POORHLTH  \\\n",
       "0  0.213470  0.242765  0.033966 -0.084923 -0.295156  2.129153  0.008536   \n",
       "1  0.213470  0.242765  0.033966  1.343546 -0.522346  3.425834  2.353060   \n",
       "2  2.084743  2.614122  0.033966 -0.084923 -0.295156 -0.464211 -0.405203   \n",
       "3 -1.657803 -2.128593  0.033966 -0.084923  2.885508 -0.464211  3.732191   \n",
       "4 -1.657803 -2.128593  0.033966 -0.084923 -0.522346 -0.464211 -0.405203   \n",
       "\n",
       "   CHILDREN  DRNK3GE5  MAXDRNKS    ...     PAFREQ2_  _MINAC11  _MINAC21  \\\n",
       "0 -0.473809 -0.203588 -0.415286    ...    -0.397997  0.340852  0.055056   \n",
       "1  1.545861 -0.203588 -0.013595    ...    -0.226111  0.183302 -0.249146   \n",
       "2 -0.473809 -0.203588 -0.415286    ...     2.115670  0.183302  0.185428   \n",
       "3  0.536026 -0.203588 -0.415286    ...    -0.226111 -0.452958 -0.249146   \n",
       "4 -0.473809 -0.203588 -0.415286    ...    -0.226111 -0.452958 -0.249146   \n",
       "\n",
       "   STRFREQ_  PAMIN11_  PAMIN21_   PA1MIN_  PAVIG11_  PAVIG21_  PA1VIGM_  \n",
       "0 -0.498281  0.122166 -0.030886  0.063706 -0.251007 -0.174378 -0.287073  \n",
       "1 -0.172802  0.445008 -0.247397  0.147148  0.718013 -0.174378  0.406465  \n",
       "2  2.756505  0.445008  0.371207  0.536545  0.718013  0.576150  0.868824  \n",
       "3 -0.498281 -0.413180 -0.247397 -0.436947 -0.251007 -0.174378 -0.287073  \n",
       "4 -0.498281 -0.413180 -0.247397 -0.436947 -0.251007 -0.174378 -0.287073  \n",
       "\n",
       "[5 rows x 42 columns]"
      ]
     },
     "execution_count": 317,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View on Hypertension standardized dataframes\n",
    "\n",
    "hypertension_scaled_df_X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 318,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NUMADULT</th>\n",
       "      <th>NUMMEN</th>\n",
       "      <th>NUMWOMEN</th>\n",
       "      <th>HHADULT</th>\n",
       "      <th>PHYSHLTH</th>\n",
       "      <th>MENTHLTH</th>\n",
       "      <th>POORHLTH</th>\n",
       "      <th>CHILDREN</th>\n",
       "      <th>DRNK3GE5</th>\n",
       "      <th>MAXDRNKS</th>\n",
       "      <th>...</th>\n",
       "      <th>PAFREQ2_</th>\n",
       "      <th>_MINAC11</th>\n",
       "      <th>_MINAC21</th>\n",
       "      <th>STRFREQ_</th>\n",
       "      <th>PAMIN11_</th>\n",
       "      <th>PAMIN21_</th>\n",
       "      <th>PA1MIN_</th>\n",
       "      <th>PAVIG11_</th>\n",
       "      <th>PAVIG21_</th>\n",
       "      <th>PA1VIGM_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>63205</th>\n",
       "      <td>0.196005</td>\n",
       "      <td>0.221056</td>\n",
       "      <td>0.032129</td>\n",
       "      <td>-0.092653</td>\n",
       "      <td>-0.489325</td>\n",
       "      <td>-0.449559</td>\n",
       "      <td>-0.380048</td>\n",
       "      <td>3.392356</td>\n",
       "      <td>-0.204189</td>\n",
       "      <td>-0.414822</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.230059</td>\n",
       "      <td>0.066080</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>-0.510207</td>\n",
       "      <td>-0.048119</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.193663</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>153317</th>\n",
       "      <td>0.196005</td>\n",
       "      <td>0.221056</td>\n",
       "      <td>0.032129</td>\n",
       "      <td>-0.092653</td>\n",
       "      <td>2.911358</td>\n",
       "      <td>-0.449559</td>\n",
       "      <td>-0.380048</td>\n",
       "      <td>-0.495660</td>\n",
       "      <td>-0.204189</td>\n",
       "      <td>-0.414822</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.230059</td>\n",
       "      <td>-0.396371</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>-0.510207</td>\n",
       "      <td>-0.380168</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.442049</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>209908</th>\n",
       "      <td>0.196005</td>\n",
       "      <td>0.221056</td>\n",
       "      <td>0.032129</td>\n",
       "      <td>-0.092653</td>\n",
       "      <td>0.304168</td>\n",
       "      <td>1.325720</td>\n",
       "      <td>0.588401</td>\n",
       "      <td>-0.495660</td>\n",
       "      <td>-0.204189</td>\n",
       "      <td>-0.414822</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.230059</td>\n",
       "      <td>-0.396371</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>-0.510207</td>\n",
       "      <td>-0.380168</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.442049</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60268</th>\n",
       "      <td>0.196005</td>\n",
       "      <td>0.221056</td>\n",
       "      <td>0.032129</td>\n",
       "      <td>1.258168</td>\n",
       "      <td>0.190812</td>\n",
       "      <td>-0.449559</td>\n",
       "      <td>0.035001</td>\n",
       "      <td>-0.495660</td>\n",
       "      <td>-0.204189</td>\n",
       "      <td>-0.414822</td>\n",
       "      <td>...</td>\n",
       "      <td>0.707573</td>\n",
       "      <td>-0.396371</td>\n",
       "      <td>0.311587</td>\n",
       "      <td>0.403819</td>\n",
       "      <td>-0.380168</td>\n",
       "      <td>0.160581</td>\n",
       "      <td>-0.193663</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119979</th>\n",
       "      <td>0.196005</td>\n",
       "      <td>0.221056</td>\n",
       "      <td>0.032129</td>\n",
       "      <td>-0.092653</td>\n",
       "      <td>2.911358</td>\n",
       "      <td>-0.449559</td>\n",
       "      <td>-0.380048</td>\n",
       "      <td>1.448348</td>\n",
       "      <td>-0.204189</td>\n",
       "      <td>-0.414822</td>\n",
       "      <td>...</td>\n",
       "      <td>1.020274</td>\n",
       "      <td>-0.036686</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>1.622672</td>\n",
       "      <td>-0.121908</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.248860</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 42 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        NUMADULT    NUMMEN  NUMWOMEN   HHADULT  PHYSHLTH  MENTHLTH  POORHLTH  \\\n",
       "63205   0.196005  0.221056  0.032129 -0.092653 -0.489325 -0.449559 -0.380048   \n",
       "153317  0.196005  0.221056  0.032129 -0.092653  2.911358 -0.449559 -0.380048   \n",
       "209908  0.196005  0.221056  0.032129 -0.092653  0.304168  1.325720  0.588401   \n",
       "60268   0.196005  0.221056  0.032129  1.258168  0.190812 -0.449559  0.035001   \n",
       "119979  0.196005  0.221056  0.032129 -0.092653  2.911358 -0.449559 -0.380048   \n",
       "\n",
       "        CHILDREN  DRNK3GE5  MAXDRNKS    ...     PAFREQ2_  _MINAC11  _MINAC21  \\\n",
       "63205   3.392356 -0.204189 -0.414822    ...    -0.230059  0.066080 -0.265828   \n",
       "153317 -0.495660 -0.204189 -0.414822    ...    -0.230059 -0.396371 -0.265828   \n",
       "209908 -0.495660 -0.204189 -0.414822    ...    -0.230059 -0.396371 -0.265828   \n",
       "60268  -0.495660 -0.204189 -0.414822    ...     0.707573 -0.396371  0.311587   \n",
       "119979  1.448348 -0.204189 -0.414822    ...     1.020274 -0.036686 -0.265828   \n",
       "\n",
       "        STRFREQ_  PAMIN11_  PAMIN21_   PA1MIN_  PAVIG11_  PAVIG21_  PA1VIGM_  \n",
       "63205  -0.510207 -0.048119 -0.279088 -0.193663 -0.237113 -0.205763 -0.297649  \n",
       "153317 -0.510207 -0.380168 -0.279088 -0.442049 -0.237113 -0.205763 -0.297649  \n",
       "209908 -0.510207 -0.380168 -0.279088 -0.442049 -0.237113 -0.205763 -0.297649  \n",
       "60268   0.403819 -0.380168  0.160581 -0.193663 -0.237113 -0.205763 -0.297649  \n",
       "119979  1.622672 -0.121908 -0.279088 -0.248860 -0.237113 -0.205763 -0.297649  \n",
       "\n",
       "[5 rows x 42 columns]"
      ]
     },
     "execution_count": 318,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hypertension_scaled_df_X_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 322,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop unstandardized versions of continuous features for X train and X test dataframes\n",
    "\n",
    "os_X_hypertension_drop = os_X_hypertension.drop(columns = hypertension_cols_to_standardize)\n",
    "X_test_hypertension_drop = X_test_hypertension.drop(columns = hypertension_cols_to_standardize)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 323,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Join standardized versions of continuous features\n",
    "\n",
    "hypertension_scaled_df_X_train_final = os_X_hypertension_drop.join(hypertension_scaled_df_X_train)\n",
    "hypertension_scaled_df_X_test_final = X_test_hypertension_drop.join(hypertension_scaled_df_X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 327,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(376392, 521)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CTELENM1</th>\n",
       "      <th>STATERE1</th>\n",
       "      <th>CTELNUM1</th>\n",
       "      <th>CELLFON5</th>\n",
       "      <th>CVDSTRK3</th>\n",
       "      <th>CHCSCNCR</th>\n",
       "      <th>CHCOCNCR</th>\n",
       "      <th>CHCCOPD1</th>\n",
       "      <th>ADDEPEV2</th>\n",
       "      <th>CHCKIDNY</th>\n",
       "      <th>...</th>\n",
       "      <th>PAFREQ2_</th>\n",
       "      <th>_MINAC11</th>\n",
       "      <th>_MINAC21</th>\n",
       "      <th>STRFREQ_</th>\n",
       "      <th>PAMIN11_</th>\n",
       "      <th>PAMIN21_</th>\n",
       "      <th>PA1MIN_</th>\n",
       "      <th>PAVIG11_</th>\n",
       "      <th>PAVIG21_</th>\n",
       "      <th>PA1VIGM_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.397997</td>\n",
       "      <td>0.340852</td>\n",
       "      <td>0.055056</td>\n",
       "      <td>-0.498281</td>\n",
       "      <td>0.122166</td>\n",
       "      <td>-0.030886</td>\n",
       "      <td>0.063706</td>\n",
       "      <td>-0.251007</td>\n",
       "      <td>-0.174378</td>\n",
       "      <td>-0.287073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.226111</td>\n",
       "      <td>0.183302</td>\n",
       "      <td>-0.249146</td>\n",
       "      <td>-0.172802</td>\n",
       "      <td>0.445008</td>\n",
       "      <td>-0.247397</td>\n",
       "      <td>0.147148</td>\n",
       "      <td>0.718013</td>\n",
       "      <td>-0.174378</td>\n",
       "      <td>0.406465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.115670</td>\n",
       "      <td>0.183302</td>\n",
       "      <td>0.185428</td>\n",
       "      <td>2.756505</td>\n",
       "      <td>0.445008</td>\n",
       "      <td>0.371207</td>\n",
       "      <td>0.536545</td>\n",
       "      <td>0.718013</td>\n",
       "      <td>0.576150</td>\n",
       "      <td>0.868824</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.226111</td>\n",
       "      <td>-0.452958</td>\n",
       "      <td>-0.249146</td>\n",
       "      <td>-0.498281</td>\n",
       "      <td>-0.413180</td>\n",
       "      <td>-0.247397</td>\n",
       "      <td>-0.436947</td>\n",
       "      <td>-0.251007</td>\n",
       "      <td>-0.174378</td>\n",
       "      <td>-0.287073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.226111</td>\n",
       "      <td>-0.452958</td>\n",
       "      <td>-0.249146</td>\n",
       "      <td>-0.498281</td>\n",
       "      <td>-0.413180</td>\n",
       "      <td>-0.247397</td>\n",
       "      <td>-0.436947</td>\n",
       "      <td>-0.251007</td>\n",
       "      <td>-0.174378</td>\n",
       "      <td>-0.287073</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 521 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   CTELENM1  STATERE1  CTELNUM1  CELLFON5  CVDSTRK3  CHCSCNCR  CHCOCNCR  \\\n",
       "0       1.0       1.0       1.0       1.0       0.0       0.0       0.0   \n",
       "1       1.0       1.0       1.0       1.0       0.0       0.0       0.0   \n",
       "2       1.0       1.0       1.0       1.0       0.0       0.0       1.0   \n",
       "3       1.0       1.0       1.0       1.0       1.0       0.0       0.0   \n",
       "4       1.0       1.0       1.0       1.0       0.0       0.0       0.0   \n",
       "\n",
       "   CHCCOPD1  ADDEPEV2  CHCKIDNY    ...     PAFREQ2_  _MINAC11  _MINAC21  \\\n",
       "0       0.0       1.0       0.0    ...    -0.397997  0.340852  0.055056   \n",
       "1       0.0       1.0       0.0    ...    -0.226111  0.183302 -0.249146   \n",
       "2       0.0       0.0       1.0    ...     2.115670  0.183302  0.185428   \n",
       "3       0.0       0.0       0.0    ...    -0.226111 -0.452958 -0.249146   \n",
       "4       1.0       0.0       0.0    ...    -0.226111 -0.452958 -0.249146   \n",
       "\n",
       "   STRFREQ_  PAMIN11_  PAMIN21_   PA1MIN_  PAVIG11_  PAVIG21_  PA1VIGM_  \n",
       "0 -0.498281  0.122166 -0.030886  0.063706 -0.251007 -0.174378 -0.287073  \n",
       "1 -0.172802  0.445008 -0.247397  0.147148  0.718013 -0.174378  0.406465  \n",
       "2  2.756505  0.445008  0.371207  0.536545  0.718013  0.576150  0.868824  \n",
       "3 -0.498281 -0.413180 -0.247397 -0.436947 -0.251007 -0.174378 -0.287073  \n",
       "4 -0.498281 -0.413180 -0.247397 -0.436947 -0.251007 -0.174378 -0.287073  \n",
       "\n",
       "[5 rows x 521 columns]"
      ]
     },
     "execution_count": 327,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View on final X train and X test datasets - dummy coded & standardized\n",
    "\n",
    "print(hypertension_scaled_df_X_train_final.shape)\n",
    "#print(hypertension_scaled_df_X_train_final.isna().sum())\n",
    "hypertension_scaled_df_X_train_final.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 329,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(135005, 521)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CTELENM1</th>\n",
       "      <th>STATERE1</th>\n",
       "      <th>CTELNUM1</th>\n",
       "      <th>CELLFON5</th>\n",
       "      <th>CVDSTRK3</th>\n",
       "      <th>CHCSCNCR</th>\n",
       "      <th>CHCOCNCR</th>\n",
       "      <th>CHCCOPD1</th>\n",
       "      <th>ADDEPEV2</th>\n",
       "      <th>CHCKIDNY</th>\n",
       "      <th>...</th>\n",
       "      <th>PAFREQ2_</th>\n",
       "      <th>_MINAC11</th>\n",
       "      <th>_MINAC21</th>\n",
       "      <th>STRFREQ_</th>\n",
       "      <th>PAMIN11_</th>\n",
       "      <th>PAMIN21_</th>\n",
       "      <th>PA1MIN_</th>\n",
       "      <th>PAVIG11_</th>\n",
       "      <th>PAVIG21_</th>\n",
       "      <th>PA1VIGM_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>63205</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.230059</td>\n",
       "      <td>0.066080</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>-0.510207</td>\n",
       "      <td>-0.048119</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.193663</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>153317</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.230059</td>\n",
       "      <td>-0.396371</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>-0.510207</td>\n",
       "      <td>-0.380168</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.442049</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>209908</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.230059</td>\n",
       "      <td>-0.396371</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>-0.510207</td>\n",
       "      <td>-0.380168</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.442049</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60268</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.707573</td>\n",
       "      <td>-0.396371</td>\n",
       "      <td>0.311587</td>\n",
       "      <td>0.403819</td>\n",
       "      <td>-0.380168</td>\n",
       "      <td>0.160581</td>\n",
       "      <td>-0.193663</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119979</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.020274</td>\n",
       "      <td>-0.036686</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>1.622672</td>\n",
       "      <td>-0.121908</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.248860</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 521 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        CTELENM1  STATERE1  CTELNUM1  CELLFON5  CVDSTRK3  CHCSCNCR  CHCOCNCR  \\\n",
       "63205        1.0       1.0       1.0       1.0       0.0       0.0       0.0   \n",
       "153317       1.0       1.0       1.0       1.0       0.0       1.0       1.0   \n",
       "209908       1.0       1.0       1.0       1.0       0.0       0.0       0.0   \n",
       "60268        1.0       1.0       1.0       1.0       1.0       0.0       0.0   \n",
       "119979       1.0       1.0       1.0       1.0       0.0       0.0       0.0   \n",
       "\n",
       "        CHCCOPD1  ADDEPEV2  CHCKIDNY    ...     PAFREQ2_  _MINAC11  _MINAC21  \\\n",
       "63205        0.0       0.0       0.0    ...    -0.230059  0.066080 -0.265828   \n",
       "153317       1.0       0.0       0.0    ...    -0.230059 -0.396371 -0.265828   \n",
       "209908       0.0       1.0       0.0    ...    -0.230059 -0.396371 -0.265828   \n",
       "60268        0.0       1.0       0.0    ...     0.707573 -0.396371  0.311587   \n",
       "119979       0.0       0.0       0.0    ...     1.020274 -0.036686 -0.265828   \n",
       "\n",
       "        STRFREQ_  PAMIN11_  PAMIN21_   PA1MIN_  PAVIG11_  PAVIG21_  PA1VIGM_  \n",
       "63205  -0.510207 -0.048119 -0.279088 -0.193663 -0.237113 -0.205763 -0.297649  \n",
       "153317 -0.510207 -0.380168 -0.279088 -0.442049 -0.237113 -0.205763 -0.297649  \n",
       "209908 -0.510207 -0.380168 -0.279088 -0.442049 -0.237113 -0.205763 -0.297649  \n",
       "60268   0.403819 -0.380168  0.160581 -0.193663 -0.237113 -0.205763 -0.297649  \n",
       "119979  1.622672 -0.121908 -0.279088 -0.248860 -0.237113 -0.205763 -0.297649  \n",
       "\n",
       "[5 rows x 521 columns]"
      ]
     },
     "execution_count": 329,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(hypertension_scaled_df_X_test_final.shape)\n",
    "#print(hypertension_scaled_df_X_test_final.isna().sum())\n",
    "hypertension_scaled_df_X_test_final.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 330,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Heart Disease/Attack\n",
    "\n",
    "heart_cols_to_standardize = [col_name for col_name in os_X_heart.columns if col_name in countinuous_features_17]\n",
    "\n",
    "scaler_train = preprocessing.StandardScaler()\n",
    "heart_scaled_X_train = scaler_train.fit_transform(os_X_heart[heart_cols_to_standardize])\n",
    "heart_scaled_df_X_train = pd.DataFrame(heart_scaled_X_train, columns=heart_cols_to_standardize, index=os_X_heart.index)\n",
    "\n",
    "scaler_test = preprocessing.StandardScaler()\n",
    "heart_scaled_X_test = scaler_test.fit_transform(X_test_heart[heart_cols_to_standardize])\n",
    "heart_scaled_df_X_test = pd.DataFrame(heart_scaled_X_test, columns=heart_cols_to_standardize, index=X_test_heart.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 331,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NUMADULT</th>\n",
       "      <th>NUMMEN</th>\n",
       "      <th>NUMWOMEN</th>\n",
       "      <th>HHADULT</th>\n",
       "      <th>PHYSHLTH</th>\n",
       "      <th>MENTHLTH</th>\n",
       "      <th>POORHLTH</th>\n",
       "      <th>CHILDREN</th>\n",
       "      <th>DRNK3GE5</th>\n",
       "      <th>MAXDRNKS</th>\n",
       "      <th>...</th>\n",
       "      <th>PAFREQ2_</th>\n",
       "      <th>_MINAC11</th>\n",
       "      <th>_MINAC21</th>\n",
       "      <th>STRFREQ_</th>\n",
       "      <th>PAMIN11_</th>\n",
       "      <th>PAMIN21_</th>\n",
       "      <th>PA1MIN_</th>\n",
       "      <th>PAVIG11_</th>\n",
       "      <th>PAVIG21_</th>\n",
       "      <th>PA1VIGM_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.269311</td>\n",
       "      <td>0.273869</td>\n",
       "      <td>0.085981</td>\n",
       "      <td>-0.056818</td>\n",
       "      <td>-0.473336</td>\n",
       "      <td>2.011979</td>\n",
       "      <td>-0.155505</td>\n",
       "      <td>-0.406738</td>\n",
       "      <td>-0.181731</td>\n",
       "      <td>-0.374786</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.406148</td>\n",
       "      <td>0.385478</td>\n",
       "      <td>0.078189</td>\n",
       "      <td>-0.469827</td>\n",
       "      <td>0.136135</td>\n",
       "      <td>-0.027178</td>\n",
       "      <td>0.075181</td>\n",
       "      <td>-0.280466</td>\n",
       "      <td>-0.190263</td>\n",
       "      <td>-0.313216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.269311</td>\n",
       "      <td>0.273869</td>\n",
       "      <td>0.085981</td>\n",
       "      <td>1.522208</td>\n",
       "      <td>-0.678868</td>\n",
       "      <td>3.278965</td>\n",
       "      <td>1.932805</td>\n",
       "      <td>1.854720</td>\n",
       "      <td>-0.181731</td>\n",
       "      <td>0.070129</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.229223</td>\n",
       "      <td>0.221934</td>\n",
       "      <td>-0.252176</td>\n",
       "      <td>-0.135063</td>\n",
       "      <td>0.467820</td>\n",
       "      <td>-0.250775</td>\n",
       "      <td>0.160393</td>\n",
       "      <td>0.717252</td>\n",
       "      <td>-0.190263</td>\n",
       "      <td>0.380778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.123414</td>\n",
       "      <td>2.678065</td>\n",
       "      <td>0.085981</td>\n",
       "      <td>-0.056818</td>\n",
       "      <td>-0.473336</td>\n",
       "      <td>-0.521991</td>\n",
       "      <td>-0.524030</td>\n",
       "      <td>-0.406738</td>\n",
       "      <td>-0.181731</td>\n",
       "      <td>-0.374786</td>\n",
       "      <td>...</td>\n",
       "      <td>2.181192</td>\n",
       "      <td>0.221934</td>\n",
       "      <td>0.219774</td>\n",
       "      <td>2.877810</td>\n",
       "      <td>0.467820</td>\n",
       "      <td>0.388073</td>\n",
       "      <td>0.558048</td>\n",
       "      <td>0.717252</td>\n",
       "      <td>0.554917</td>\n",
       "      <td>0.843440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-1.584792</td>\n",
       "      <td>-2.130327</td>\n",
       "      <td>0.085981</td>\n",
       "      <td>-0.056818</td>\n",
       "      <td>2.404113</td>\n",
       "      <td>-0.521991</td>\n",
       "      <td>3.161222</td>\n",
       "      <td>0.723991</td>\n",
       "      <td>-0.181731</td>\n",
       "      <td>-0.374786</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.229223</td>\n",
       "      <td>-0.438534</td>\n",
       "      <td>-0.252176</td>\n",
       "      <td>-0.469827</td>\n",
       "      <td>-0.413873</td>\n",
       "      <td>-0.250775</td>\n",
       "      <td>-0.436090</td>\n",
       "      <td>-0.280466</td>\n",
       "      <td>-0.190263</td>\n",
       "      <td>-0.313216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-1.584792</td>\n",
       "      <td>-2.130327</td>\n",
       "      <td>0.085981</td>\n",
       "      <td>-0.056818</td>\n",
       "      <td>-0.678868</td>\n",
       "      <td>-0.521991</td>\n",
       "      <td>-0.524030</td>\n",
       "      <td>-0.406738</td>\n",
       "      <td>-0.181731</td>\n",
       "      <td>-0.374786</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.229223</td>\n",
       "      <td>-0.438534</td>\n",
       "      <td>-0.252176</td>\n",
       "      <td>-0.469827</td>\n",
       "      <td>-0.413873</td>\n",
       "      <td>-0.250775</td>\n",
       "      <td>-0.436090</td>\n",
       "      <td>-0.280466</td>\n",
       "      <td>-0.190263</td>\n",
       "      <td>-0.313216</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 42 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   NUMADULT    NUMMEN  NUMWOMEN   HHADULT  PHYSHLTH  MENTHLTH  POORHLTH  \\\n",
       "0  0.269311  0.273869  0.085981 -0.056818 -0.473336  2.011979 -0.155505   \n",
       "1  0.269311  0.273869  0.085981  1.522208 -0.678868  3.278965  1.932805   \n",
       "2  2.123414  2.678065  0.085981 -0.056818 -0.473336 -0.521991 -0.524030   \n",
       "3 -1.584792 -2.130327  0.085981 -0.056818  2.404113 -0.521991  3.161222   \n",
       "4 -1.584792 -2.130327  0.085981 -0.056818 -0.678868 -0.521991 -0.524030   \n",
       "\n",
       "   CHILDREN  DRNK3GE5  MAXDRNKS    ...     PAFREQ2_  _MINAC11  _MINAC21  \\\n",
       "0 -0.406738 -0.181731 -0.374786    ...    -0.406148  0.385478  0.078189   \n",
       "1  1.854720 -0.181731  0.070129    ...    -0.229223  0.221934 -0.252176   \n",
       "2 -0.406738 -0.181731 -0.374786    ...     2.181192  0.221934  0.219774   \n",
       "3  0.723991 -0.181731 -0.374786    ...    -0.229223 -0.438534 -0.252176   \n",
       "4 -0.406738 -0.181731 -0.374786    ...    -0.229223 -0.438534 -0.252176   \n",
       "\n",
       "   STRFREQ_  PAMIN11_  PAMIN21_   PA1MIN_  PAVIG11_  PAVIG21_  PA1VIGM_  \n",
       "0 -0.469827  0.136135 -0.027178  0.075181 -0.280466 -0.190263 -0.313216  \n",
       "1 -0.135063  0.467820 -0.250775  0.160393  0.717252 -0.190263  0.380778  \n",
       "2  2.877810  0.467820  0.388073  0.558048  0.717252  0.554917  0.843440  \n",
       "3 -0.469827 -0.413873 -0.250775 -0.436090 -0.280466 -0.190263 -0.313216  \n",
       "4 -0.469827 -0.413873 -0.250775 -0.436090 -0.280466 -0.190263 -0.313216  \n",
       "\n",
       "[5 rows x 42 columns]"
      ]
     },
     "execution_count": 331,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View on Hypertension standardized dataframes\n",
    "\n",
    "heart_scaled_df_X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 332,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NUMADULT</th>\n",
       "      <th>NUMMEN</th>\n",
       "      <th>NUMWOMEN</th>\n",
       "      <th>HHADULT</th>\n",
       "      <th>PHYSHLTH</th>\n",
       "      <th>MENTHLTH</th>\n",
       "      <th>POORHLTH</th>\n",
       "      <th>CHILDREN</th>\n",
       "      <th>DRNK3GE5</th>\n",
       "      <th>MAXDRNKS</th>\n",
       "      <th>...</th>\n",
       "      <th>PAFREQ2_</th>\n",
       "      <th>_MINAC11</th>\n",
       "      <th>_MINAC21</th>\n",
       "      <th>STRFREQ_</th>\n",
       "      <th>PAMIN11_</th>\n",
       "      <th>PAMIN21_</th>\n",
       "      <th>PA1MIN_</th>\n",
       "      <th>PAVIG11_</th>\n",
       "      <th>PAVIG21_</th>\n",
       "      <th>PA1VIGM_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>63205</th>\n",
       "      <td>0.196005</td>\n",
       "      <td>0.221056</td>\n",
       "      <td>0.032129</td>\n",
       "      <td>-0.092653</td>\n",
       "      <td>-0.489325</td>\n",
       "      <td>-0.449559</td>\n",
       "      <td>-0.380048</td>\n",
       "      <td>3.392356</td>\n",
       "      <td>-0.204189</td>\n",
       "      <td>-0.414822</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.230059</td>\n",
       "      <td>0.066080</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>-0.510207</td>\n",
       "      <td>-0.048119</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.193663</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>153317</th>\n",
       "      <td>0.196005</td>\n",
       "      <td>0.221056</td>\n",
       "      <td>0.032129</td>\n",
       "      <td>-0.092653</td>\n",
       "      <td>2.911358</td>\n",
       "      <td>-0.449559</td>\n",
       "      <td>-0.380048</td>\n",
       "      <td>-0.495660</td>\n",
       "      <td>-0.204189</td>\n",
       "      <td>-0.414822</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.230059</td>\n",
       "      <td>-0.396371</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>-0.510207</td>\n",
       "      <td>-0.380168</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.442049</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>209908</th>\n",
       "      <td>0.196005</td>\n",
       "      <td>0.221056</td>\n",
       "      <td>0.032129</td>\n",
       "      <td>-0.092653</td>\n",
       "      <td>0.304168</td>\n",
       "      <td>1.325720</td>\n",
       "      <td>0.588401</td>\n",
       "      <td>-0.495660</td>\n",
       "      <td>-0.204189</td>\n",
       "      <td>-0.414822</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.230059</td>\n",
       "      <td>-0.396371</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>-0.510207</td>\n",
       "      <td>-0.380168</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.442049</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60268</th>\n",
       "      <td>0.196005</td>\n",
       "      <td>0.221056</td>\n",
       "      <td>0.032129</td>\n",
       "      <td>1.258168</td>\n",
       "      <td>0.190812</td>\n",
       "      <td>-0.449559</td>\n",
       "      <td>0.035001</td>\n",
       "      <td>-0.495660</td>\n",
       "      <td>-0.204189</td>\n",
       "      <td>-0.414822</td>\n",
       "      <td>...</td>\n",
       "      <td>0.707573</td>\n",
       "      <td>-0.396371</td>\n",
       "      <td>0.311587</td>\n",
       "      <td>0.403819</td>\n",
       "      <td>-0.380168</td>\n",
       "      <td>0.160581</td>\n",
       "      <td>-0.193663</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119979</th>\n",
       "      <td>0.196005</td>\n",
       "      <td>0.221056</td>\n",
       "      <td>0.032129</td>\n",
       "      <td>-0.092653</td>\n",
       "      <td>2.911358</td>\n",
       "      <td>-0.449559</td>\n",
       "      <td>-0.380048</td>\n",
       "      <td>1.448348</td>\n",
       "      <td>-0.204189</td>\n",
       "      <td>-0.414822</td>\n",
       "      <td>...</td>\n",
       "      <td>1.020274</td>\n",
       "      <td>-0.036686</td>\n",
       "      <td>-0.265828</td>\n",
       "      <td>1.622672</td>\n",
       "      <td>-0.121908</td>\n",
       "      <td>-0.279088</td>\n",
       "      <td>-0.248860</td>\n",
       "      <td>-0.237113</td>\n",
       "      <td>-0.205763</td>\n",
       "      <td>-0.297649</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 42 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        NUMADULT    NUMMEN  NUMWOMEN   HHADULT  PHYSHLTH  MENTHLTH  POORHLTH  \\\n",
       "63205   0.196005  0.221056  0.032129 -0.092653 -0.489325 -0.449559 -0.380048   \n",
       "153317  0.196005  0.221056  0.032129 -0.092653  2.911358 -0.449559 -0.380048   \n",
       "209908  0.196005  0.221056  0.032129 -0.092653  0.304168  1.325720  0.588401   \n",
       "60268   0.196005  0.221056  0.032129  1.258168  0.190812 -0.449559  0.035001   \n",
       "119979  0.196005  0.221056  0.032129 -0.092653  2.911358 -0.449559 -0.380048   \n",
       "\n",
       "        CHILDREN  DRNK3GE5  MAXDRNKS    ...     PAFREQ2_  _MINAC11  _MINAC21  \\\n",
       "63205   3.392356 -0.204189 -0.414822    ...    -0.230059  0.066080 -0.265828   \n",
       "153317 -0.495660 -0.204189 -0.414822    ...    -0.230059 -0.396371 -0.265828   \n",
       "209908 -0.495660 -0.204189 -0.414822    ...    -0.230059 -0.396371 -0.265828   \n",
       "60268  -0.495660 -0.204189 -0.414822    ...     0.707573 -0.396371  0.311587   \n",
       "119979  1.448348 -0.204189 -0.414822    ...     1.020274 -0.036686 -0.265828   \n",
       "\n",
       "        STRFREQ_  PAMIN11_  PAMIN21_   PA1MIN_  PAVIG11_  PAVIG21_  PA1VIGM_  \n",
       "63205  -0.510207 -0.048119 -0.279088 -0.193663 -0.237113 -0.205763 -0.297649  \n",
       "153317 -0.510207 -0.380168 -0.279088 -0.442049 -0.237113 -0.205763 -0.297649  \n",
       "209908 -0.510207 -0.380168 -0.279088 -0.442049 -0.237113 -0.205763 -0.297649  \n",
       "60268   0.403819 -0.380168  0.160581 -0.193663 -0.237113 -0.205763 -0.297649  \n",
       "119979  1.622672 -0.121908 -0.279088 -0.248860 -0.237113 -0.205763 -0.297649  \n",
       "\n",
       "[5 rows x 42 columns]"
      ]
     },
     "execution_count": 332,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hypertension_scaled_df_X_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 333,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop unstandardized versions of continuous features for X train and X test dataframes\n",
    "\n",
    "os_X_heart_drop = os_X_heart.drop(columns = heart_cols_to_standardize)\n",
    "X_test_heart_drop = X_test_heart.drop(columns = heart_cols_to_standardize)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 334,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Join standardized versions of continuous features\n",
    "\n",
    "heart_scaled_df_X_train_final = os_X_heart_drop.join(heart_scaled_df_X_train)\n",
    "heart_scaled_df_X_test_final = X_test_heart_drop.join(heart_scaled_df_X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 336,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(135005, 522)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CTELENM1</th>\n",
       "      <th>STATERE1</th>\n",
       "      <th>CTELNUM1</th>\n",
       "      <th>CELLFON5</th>\n",
       "      <th>CVDSTRK3</th>\n",
       "      <th>CHCSCNCR</th>\n",
       "      <th>CHCOCNCR</th>\n",
       "      <th>CHCCOPD1</th>\n",
       "      <th>ADDEPEV2</th>\n",
       "      <th>CHCKIDNY</th>\n",
       "      <th>...</th>\n",
       "      <th>PAFREQ2_</th>\n",
       "      <th>_MINAC11</th>\n",
       "      <th>_MINAC21</th>\n",
       "      <th>STRFREQ_</th>\n",
       "      <th>PAMIN11_</th>\n",
       "      <th>PAMIN21_</th>\n",
       "      <th>PA1MIN_</th>\n",
       "      <th>PAVIG11_</th>\n",
       "      <th>PAVIG21_</th>\n",
       "      <th>PA1VIGM_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.406148</td>\n",
       "      <td>0.385478</td>\n",
       "      <td>0.078189</td>\n",
       "      <td>-0.469827</td>\n",
       "      <td>0.136135</td>\n",
       "      <td>-0.027178</td>\n",
       "      <td>0.075181</td>\n",
       "      <td>-0.280466</td>\n",
       "      <td>-0.190263</td>\n",
       "      <td>-0.313216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.229223</td>\n",
       "      <td>0.221934</td>\n",
       "      <td>-0.252176</td>\n",
       "      <td>-0.135063</td>\n",
       "      <td>0.467820</td>\n",
       "      <td>-0.250775</td>\n",
       "      <td>0.160393</td>\n",
       "      <td>0.717252</td>\n",
       "      <td>-0.190263</td>\n",
       "      <td>0.380778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.181192</td>\n",
       "      <td>0.221934</td>\n",
       "      <td>0.219774</td>\n",
       "      <td>2.877810</td>\n",
       "      <td>0.467820</td>\n",
       "      <td>0.388073</td>\n",
       "      <td>0.558048</td>\n",
       "      <td>0.717252</td>\n",
       "      <td>0.554917</td>\n",
       "      <td>0.843440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.229223</td>\n",
       "      <td>-0.438534</td>\n",
       "      <td>-0.252176</td>\n",
       "      <td>-0.469827</td>\n",
       "      <td>-0.413873</td>\n",
       "      <td>-0.250775</td>\n",
       "      <td>-0.436090</td>\n",
       "      <td>-0.280466</td>\n",
       "      <td>-0.190263</td>\n",
       "      <td>-0.313216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.229223</td>\n",
       "      <td>-0.438534</td>\n",
       "      <td>-0.252176</td>\n",
       "      <td>-0.469827</td>\n",
       "      <td>-0.413873</td>\n",
       "      <td>-0.250775</td>\n",
       "      <td>-0.436090</td>\n",
       "      <td>-0.280466</td>\n",
       "      <td>-0.190263</td>\n",
       "      <td>-0.313216</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 522 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   CTELENM1  STATERE1  CTELNUM1  CELLFON5  CVDSTRK3  CHCSCNCR  CHCOCNCR  \\\n",
       "0       1.0       1.0       1.0       1.0       0.0       0.0       0.0   \n",
       "1       1.0       1.0       1.0       1.0       0.0       0.0       0.0   \n",
       "2       1.0       1.0       1.0       1.0       0.0       0.0       1.0   \n",
       "3       1.0       1.0       1.0       1.0       1.0       0.0       0.0   \n",
       "4       1.0       1.0       1.0       1.0       0.0       0.0       0.0   \n",
       "\n",
       "   CHCCOPD1  ADDEPEV2  CHCKIDNY    ...     PAFREQ2_  _MINAC11  _MINAC21  \\\n",
       "0       0.0       1.0       0.0    ...    -0.406148  0.385478  0.078189   \n",
       "1       0.0       1.0       0.0    ...    -0.229223  0.221934 -0.252176   \n",
       "2       0.0       0.0       1.0    ...     2.181192  0.221934  0.219774   \n",
       "3       0.0       0.0       0.0    ...    -0.229223 -0.438534 -0.252176   \n",
       "4       1.0       0.0       0.0    ...    -0.229223 -0.438534 -0.252176   \n",
       "\n",
       "   STRFREQ_  PAMIN11_  PAMIN21_   PA1MIN_  PAVIG11_  PAVIG21_  PA1VIGM_  \n",
       "0 -0.469827  0.136135 -0.027178  0.075181 -0.280466 -0.190263 -0.313216  \n",
       "1 -0.135063  0.467820 -0.250775  0.160393  0.717252 -0.190263  0.380778  \n",
       "2  2.877810  0.467820  0.388073  0.558048  0.717252  0.554917  0.843440  \n",
       "3 -0.469827 -0.413873 -0.250775 -0.436090 -0.280466 -0.190263 -0.313216  \n",
       "4 -0.469827 -0.413873 -0.250775 -0.436090 -0.280466 -0.190263 -0.313216  \n",
       "\n",
       "[5 rows x 522 columns]"
      ]
     },
     "execution_count": 336,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View on final X train and X test datasets - dummy coded & standardized\n",
    "\n",
    "print(heart_scaled_df_X_test_final.shape)\n",
    "#print(heart_scaled_df_X_test_final.isna().sum())\n",
    "heart_scaled_df_X_train_final.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 338,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(574900, 522)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CTELENM1</th>\n",
       "      <th>STATERE1</th>\n",
       "      <th>CTELNUM1</th>\n",
       "      <th>CELLFON5</th>\n",
       "      <th>CVDSTRK3</th>\n",
       "      <th>CHCSCNCR</th>\n",
       "      <th>CHCOCNCR</th>\n",
       "      <th>CHCCOPD1</th>\n",
       "      <th>ADDEPEV2</th>\n",
       "      <th>CHCKIDNY</th>\n",
       "      <th>...</th>\n",
       "      <th>PAFREQ2_</th>\n",
       "      <th>_MINAC11</th>\n",
       "      <th>_MINAC21</th>\n",
       "      <th>STRFREQ_</th>\n",
       "      <th>PAMIN11_</th>\n",
       "      <th>PAMIN21_</th>\n",
       "      <th>PA1MIN_</th>\n",
       "      <th>PAVIG11_</th>\n",
       "      <th>PAVIG21_</th>\n",
       "      <th>PA1VIGM_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.406148</td>\n",
       "      <td>0.385478</td>\n",
       "      <td>0.078189</td>\n",
       "      <td>-0.469827</td>\n",
       "      <td>0.136135</td>\n",
       "      <td>-0.027178</td>\n",
       "      <td>0.075181</td>\n",
       "      <td>-0.280466</td>\n",
       "      <td>-0.190263</td>\n",
       "      <td>-0.313216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.229223</td>\n",
       "      <td>0.221934</td>\n",
       "      <td>-0.252176</td>\n",
       "      <td>-0.135063</td>\n",
       "      <td>0.467820</td>\n",
       "      <td>-0.250775</td>\n",
       "      <td>0.160393</td>\n",
       "      <td>0.717252</td>\n",
       "      <td>-0.190263</td>\n",
       "      <td>0.380778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.181192</td>\n",
       "      <td>0.221934</td>\n",
       "      <td>0.219774</td>\n",
       "      <td>2.877810</td>\n",
       "      <td>0.467820</td>\n",
       "      <td>0.388073</td>\n",
       "      <td>0.558048</td>\n",
       "      <td>0.717252</td>\n",
       "      <td>0.554917</td>\n",
       "      <td>0.843440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.229223</td>\n",
       "      <td>-0.438534</td>\n",
       "      <td>-0.252176</td>\n",
       "      <td>-0.469827</td>\n",
       "      <td>-0.413873</td>\n",
       "      <td>-0.250775</td>\n",
       "      <td>-0.436090</td>\n",
       "      <td>-0.280466</td>\n",
       "      <td>-0.190263</td>\n",
       "      <td>-0.313216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.229223</td>\n",
       "      <td>-0.438534</td>\n",
       "      <td>-0.252176</td>\n",
       "      <td>-0.469827</td>\n",
       "      <td>-0.413873</td>\n",
       "      <td>-0.250775</td>\n",
       "      <td>-0.436090</td>\n",
       "      <td>-0.280466</td>\n",
       "      <td>-0.190263</td>\n",
       "      <td>-0.313216</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 522 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   CTELENM1  STATERE1  CTELNUM1  CELLFON5  CVDSTRK3  CHCSCNCR  CHCOCNCR  \\\n",
       "0       1.0       1.0       1.0       1.0       0.0       0.0       0.0   \n",
       "1       1.0       1.0       1.0       1.0       0.0       0.0       0.0   \n",
       "2       1.0       1.0       1.0       1.0       0.0       0.0       1.0   \n",
       "3       1.0       1.0       1.0       1.0       1.0       0.0       0.0   \n",
       "4       1.0       1.0       1.0       1.0       0.0       0.0       0.0   \n",
       "\n",
       "   CHCCOPD1  ADDEPEV2  CHCKIDNY    ...     PAFREQ2_  _MINAC11  _MINAC21  \\\n",
       "0       0.0       1.0       0.0    ...    -0.406148  0.385478  0.078189   \n",
       "1       0.0       1.0       0.0    ...    -0.229223  0.221934 -0.252176   \n",
       "2       0.0       0.0       1.0    ...     2.181192  0.221934  0.219774   \n",
       "3       0.0       0.0       0.0    ...    -0.229223 -0.438534 -0.252176   \n",
       "4       1.0       0.0       0.0    ...    -0.229223 -0.438534 -0.252176   \n",
       "\n",
       "   STRFREQ_  PAMIN11_  PAMIN21_   PA1MIN_  PAVIG11_  PAVIG21_  PA1VIGM_  \n",
       "0 -0.469827  0.136135 -0.027178  0.075181 -0.280466 -0.190263 -0.313216  \n",
       "1 -0.135063  0.467820 -0.250775  0.160393  0.717252 -0.190263  0.380778  \n",
       "2  2.877810  0.467820  0.388073  0.558048  0.717252  0.554917  0.843440  \n",
       "3 -0.469827 -0.413873 -0.250775 -0.436090 -0.280466 -0.190263 -0.313216  \n",
       "4 -0.469827 -0.413873 -0.250775 -0.436090 -0.280466 -0.190263 -0.313216  \n",
       "\n",
       "[5 rows x 522 columns]"
      ]
     },
     "execution_count": 338,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(heart_scaled_df_X_train_final.shape)\n",
    "#print(heart_scaled_df_X_train_final.isna().sum())\n",
    "heart_scaled_df_X_train_final.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Part 1H: Pickle Fully Preprocessed Dataframes</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 347,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Pickling Diabetes dataframes\n",
    "\n",
    "diabetes_scaled_df_X_train_final.to_pickle(\"./os_X_diabetes.pkl\")\n",
    "os_y_diabetes.to_pickle(\"./os_y_diabetes.pkl\")\n",
    "diabetes_scaled_df_X_test_final.to_pickle(\"./X_test_diabetes.pkl\")\n",
    "y_test_diabetes.to_pickle(\"./y_test_diabetes.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 348,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Pickling Heart Disease/Heart Attack dataframes\n",
    "\n",
    "heart_scaled_df_X_train_final.to_pickle(\"./os_X_heart.pkl\")\n",
    "os_y_heart.to_pickle(\"./os_y_heart.pkl\")\n",
    "heart_scaled_df_X_test_final.to_pickle(\"./X_test_heart.pkl\")\n",
    "y_test_heart.to_pickle(\"./y_test_heart.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 349,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Pickling Hypertension dataframes\n",
    "\n",
    "hypertension_scaled_df_X_train_final.to_pickle(\"./os_X_hypertension.pkl\")\n",
    "os_y_hypertension.to_pickle(\"./os_y_hypertension.pkl\")\n",
    "hypertension_scaled_df_X_test_final.to_pickle(\"./X_test_hypertension.pkl\")\n",
    "y_test_hypertension.to_pickle(\"./y_test_hypertension.pkl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Preprocessed Dataframes for Modeling by Outcome (2017):</b>\n",
    "    \n",
    "The following dataframes should be used for modeling:\n",
    "- Diabetes: <i>Train</i> = os_X_diabetes, os_y_diabetes; <i>Test</i> = X_test_diabetes, y_test_diabetes\n",
    "- Heart Disease/Heart Attack: <i>Train</i> = os_X_heart, os_y_heart; <i>Test</i> = X_test_heart, y_test_heart\n",
    "- Hypertension:<i>Train</i> = os_X_hypertension, os_y_hypertension; <i>Test</i> = X_test_hypertension, y_test_hypertension\n",
    "\n",
    "Pickled versions of these dataframes are provided. To unpickle, run the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 350,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Unpickling Diabetes dataframes\n",
    "\n",
    "unpick_os_X_diabetes = pd.read_pickle(\"./os_X_diabetes.pkl\")\n",
    "unpick_os_y_diabetes = pd.read_pickle(\"./os_y_diabetes.pkl\")\n",
    "unpick_X_test_diabetes = pd.read_pickle(\"./X_test_diabetes.pkl\")\n",
    "unpick_y_test_diabetes = pd.read_pickle(\"./y_test_diabetes.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 351,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Unpickling Heart Disease/Heart Attack dataframes\n",
    "\n",
    "unpick_os_X_heart = pd.read_pickle(\"./os_X_heart.pkl\")\n",
    "unpick_os_y_heart = pd.read_pickle(\"./os_y_heart.pkl\")\n",
    "unpick_X_test_heart = pd.read_pickle(\"./X_test_heart.pkl\")\n",
    "unpick_y_test_heart = pd.read_pickle(\"./y_test_heart.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 352,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Unpickling Hypertension dataframes\n",
    "\n",
    "unpick_os_X_hypertension = pd.read_pickle(\"./os_X_hypertension.pkl\")\n",
    "unpick_os_y_hypertension = pd.read_pickle(\"./os_y_hypertension.pkl\")\n",
    "unpick_X_test_hypertension = pd.read_pickle(\"./X_test_hypertension.pkl\")\n",
    "unpick_y_test_hypertension = pd.read_pickle(\"./y_test_hypertension.pkl\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}