1 - Data preprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.patches as mpatches\n",
    "plt.style.use('ggplot')\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this first part, we will process our two datasets :\n",
    "- The first one is related to the **mRNA**, provided by the *Bälherlab (ULC)*.\n",
    "- The second one is related to the **proteins**, provided by the *Sismanis-lab (EPFL)*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to merge these 2 dataset into a coherent one, before further studies. We will proceed in 3 parts :\n",
    "+ First we will describe our datasets;\n",
    "+ Then we will process them independently;\n",
    "+ Finally we will merge them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0 - Loading data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# set data path\n",
    "path_data_prot = \"data/wtratioall.csv\"\n",
    "path_data_mRNA = \"data/pat1_average_modified.txt\"\n",
    "\n",
    "# read data from files\n",
    "raw_data_prot = pd.read_csv(path_data_prot, index_col=0)\n",
    "raw_data_mRNA = pd.read_csv(path_data_mRNA, sep='\\t', header=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 - Datasets description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### a) Proteins data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2969, 33)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A_00</th>\n",
       "      <th>A_01</th>\n",
       "      <th>A_02</th>\n",
       "      <th>A_03</th>\n",
       "      <th>A_04</th>\n",
       "      <th>A_05</th>\n",
       "      <th>A_06</th>\n",
       "      <th>A_07</th>\n",
       "      <th>A_08</th>\n",
       "      <th>A_09</th>\n",
       "      <th>...</th>\n",
       "      <th>C_01</th>\n",
       "      <th>C_02</th>\n",
       "      <th>C_03</th>\n",
       "      <th>C_04</th>\n",
       "      <th>C_05</th>\n",
       "      <th>C_06</th>\n",
       "      <th>C_07</th>\n",
       "      <th>C_08</th>\n",
       "      <th>C_09</th>\n",
       "      <th>C_10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SPAC1002.02</th>\n",
       "      <td>1</td>\n",
       "      <td>1.451188</td>\n",
       "      <td>3.734827</td>\n",
       "      <td>6.181905</td>\n",
       "      <td>7.607555</td>\n",
       "      <td>5.215707</td>\n",
       "      <td>3.439898</td>\n",
       "      <td>2.598756</td>\n",
       "      <td>2.183230</td>\n",
       "      <td>1.811290</td>\n",
       "      <td>...</td>\n",
       "      <td>1.555395</td>\n",
       "      <td>3.676540</td>\n",
       "      <td>6.030714</td>\n",
       "      <td>6.635780</td>\n",
       "      <td>5.153358</td>\n",
       "      <td>3.210663</td>\n",
       "      <td>2.511956</td>\n",
       "      <td>1.853247</td>\n",
       "      <td>1.525539</td>\n",
       "      <td>1.332680</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.03c</th>\n",
       "      <td>1</td>\n",
       "      <td>0.922745</td>\n",
       "      <td>0.763064</td>\n",
       "      <td>0.749450</td>\n",
       "      <td>0.742595</td>\n",
       "      <td>0.771641</td>\n",
       "      <td>0.819817</td>\n",
       "      <td>0.859544</td>\n",
       "      <td>0.849914</td>\n",
       "      <td>0.838128</td>\n",
       "      <td>...</td>\n",
       "      <td>0.901151</td>\n",
       "      <td>0.828585</td>\n",
       "      <td>0.779982</td>\n",
       "      <td>0.767321</td>\n",
       "      <td>0.813157</td>\n",
       "      <td>0.830826</td>\n",
       "      <td>0.875965</td>\n",
       "      <td>0.938763</td>\n",
       "      <td>0.819587</td>\n",
       "      <td>0.850455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.04c</th>\n",
       "      <td>1</td>\n",
       "      <td>1.063774</td>\n",
       "      <td>1.109541</td>\n",
       "      <td>1.105289</td>\n",
       "      <td>1.256196</td>\n",
       "      <td>1.193269</td>\n",
       "      <td>0.834514</td>\n",
       "      <td>0.698355</td>\n",
       "      <td>0.714280</td>\n",
       "      <td>0.716728</td>\n",
       "      <td>...</td>\n",
       "      <td>0.834014</td>\n",
       "      <td>0.829090</td>\n",
       "      <td>0.911089</td>\n",
       "      <td>1.179868</td>\n",
       "      <td>1.106116</td>\n",
       "      <td>0.824800</td>\n",
       "      <td>0.652223</td>\n",
       "      <td>0.654474</td>\n",
       "      <td>0.618405</td>\n",
       "      <td>0.651751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.07c</th>\n",
       "      <td>1</td>\n",
       "      <td>0.946289</td>\n",
       "      <td>1.270676</td>\n",
       "      <td>1.577819</td>\n",
       "      <td>2.059521</td>\n",
       "      <td>2.079450</td>\n",
       "      <td>2.026755</td>\n",
       "      <td>1.934475</td>\n",
       "      <td>1.886784</td>\n",
       "      <td>1.820797</td>\n",
       "      <td>...</td>\n",
       "      <td>1.145764</td>\n",
       "      <td>1.396319</td>\n",
       "      <td>1.819543</td>\n",
       "      <td>2.141494</td>\n",
       "      <td>2.219369</td>\n",
       "      <td>2.134959</td>\n",
       "      <td>2.000173</td>\n",
       "      <td>1.983480</td>\n",
       "      <td>1.959289</td>\n",
       "      <td>1.902660</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.09c</th>\n",
       "      <td>1</td>\n",
       "      <td>0.909058</td>\n",
       "      <td>0.871595</td>\n",
       "      <td>0.907030</td>\n",
       "      <td>0.941867</td>\n",
       "      <td>0.963464</td>\n",
       "      <td>0.963832</td>\n",
       "      <td>0.978117</td>\n",
       "      <td>1.026633</td>\n",
       "      <td>1.043288</td>\n",
       "      <td>...</td>\n",
       "      <td>0.917147</td>\n",
       "      <td>0.879789</td>\n",
       "      <td>0.881237</td>\n",
       "      <td>0.897882</td>\n",
       "      <td>0.903383</td>\n",
       "      <td>0.907655</td>\n",
       "      <td>0.899867</td>\n",
       "      <td>0.909359</td>\n",
       "      <td>0.945886</td>\n",
       "      <td>0.975173</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 33 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              A_00      A_01      A_02      A_03      A_04      A_05  \\\n",
       "SPAC1002.02      1  1.451188  3.734827  6.181905  7.607555  5.215707   \n",
       "SPAC1002.03c     1  0.922745  0.763064  0.749450  0.742595  0.771641   \n",
       "SPAC1002.04c     1  1.063774  1.109541  1.105289  1.256196  1.193269   \n",
       "SPAC1002.07c     1  0.946289  1.270676  1.577819  2.059521  2.079450   \n",
       "SPAC1002.09c     1  0.909058  0.871595  0.907030  0.941867  0.963464   \n",
       "\n",
       "                  A_06      A_07      A_08      A_09    ...         C_01  \\\n",
       "SPAC1002.02   3.439898  2.598756  2.183230  1.811290    ...     1.555395   \n",
       "SPAC1002.03c  0.819817  0.859544  0.849914  0.838128    ...     0.901151   \n",
       "SPAC1002.04c  0.834514  0.698355  0.714280  0.716728    ...     0.834014   \n",
       "SPAC1002.07c  2.026755  1.934475  1.886784  1.820797    ...     1.145764   \n",
       "SPAC1002.09c  0.963832  0.978117  1.026633  1.043288    ...     0.917147   \n",
       "\n",
       "                  C_02      C_03      C_04      C_05      C_06      C_07  \\\n",
       "SPAC1002.02   3.676540  6.030714  6.635780  5.153358  3.210663  2.511956   \n",
       "SPAC1002.03c  0.828585  0.779982  0.767321  0.813157  0.830826  0.875965   \n",
       "SPAC1002.04c  0.829090  0.911089  1.179868  1.106116  0.824800  0.652223   \n",
       "SPAC1002.07c  1.396319  1.819543  2.141494  2.219369  2.134959  2.000173   \n",
       "SPAC1002.09c  0.879789  0.881237  0.897882  0.903383  0.907655  0.899867   \n",
       "\n",
       "                  C_08      C_09      C_10  \n",
       "SPAC1002.02   1.853247  1.525539  1.332680  \n",
       "SPAC1002.03c  0.938763  0.819587  0.850455  \n",
       "SPAC1002.04c  0.654474  0.618405  0.651751  \n",
       "SPAC1002.07c  1.983480  1.959289  1.902660  \n",
       "SPAC1002.09c  0.909359  0.945886  0.975173  \n",
       "\n",
       "[5 rows x 33 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# preview of the raw protein data\n",
    "print(raw_data_prot.shape)\n",
    "raw_data_prot.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The dataset counts **2969 proteins**. <br/>\n",
    "- For each protein, a relative concentration is measured for **11 time points** in **triplicate** (the 10 time points have been measured in 3 independent experiments).<br/>\n",
    "- These 11 concentrations are measured for 0h, 1h, 2h, 3h, 4h, 5h, 6h, 7h, 8h, 9h, 10h.<br/>\n",
    "- The 3 measures can differ (sometimes a lot in terms of order of magnitude), but the concentration's evolution seems to be the same (Averaged Pearson correlation coefficient = **0.8** between each measures A, B, C)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### b) mRNA data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(5113, 15)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>aap1</td>\n",
       "      <td>AAP1</td>\n",
       "      <td>SPBC1652.02 SPBC16A3.20C</td>\n",
       "      <td>1.0</td>\n",
       "      <td>22.703531</td>\n",
       "      <td>0.960360</td>\n",
       "      <td>0.544460</td>\n",
       "      <td>0.896232</td>\n",
       "      <td>1.160251</td>\n",
       "      <td>1.252143</td>\n",
       "      <td>2.309153</td>\n",
       "      <td>2.310750</td>\n",
       "      <td>2.449327</td>\n",
       "      <td>2.706837</td>\n",
       "      <td>3.064384</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>abc1: C2D10.18</td>\n",
       "      <td>ABC1</td>\n",
       "      <td>SPBC2D10.18</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.043491</td>\n",
       "      <td>0.622283</td>\n",
       "      <td>0.663191</td>\n",
       "      <td>0.619409</td>\n",
       "      <td>0.648403</td>\n",
       "      <td>0.662236</td>\n",
       "      <td>0.655953</td>\n",
       "      <td>0.580761</td>\n",
       "      <td>0.606205</td>\n",
       "      <td>0.752322</td>\n",
       "      <td>0.798264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>abc1: C9E9.12c</td>\n",
       "      <td>ABC1</td>\n",
       "      <td>SPAC9E9.12C</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.112546</td>\n",
       "      <td>0.906822</td>\n",
       "      <td>0.565746</td>\n",
       "      <td>0.535220</td>\n",
       "      <td>1.224398</td>\n",
       "      <td>10.170195</td>\n",
       "      <td>7.380023</td>\n",
       "      <td>4.455168</td>\n",
       "      <td>2.804587</td>\n",
       "      <td>1.657471</td>\n",
       "      <td>1.310630</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>abp1</td>\n",
       "      <td>CBP1 ABP1</td>\n",
       "      <td>SPBC1105.04C</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.351277</td>\n",
       "      <td>0.884985</td>\n",
       "      <td>0.705397</td>\n",
       "      <td>0.401055</td>\n",
       "      <td>0.202862</td>\n",
       "      <td>0.175360</td>\n",
       "      <td>0.829920</td>\n",
       "      <td>0.959749</td>\n",
       "      <td>1.121036</td>\n",
       "      <td>1.122949</td>\n",
       "      <td>1.103435</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>abp2</td>\n",
       "      <td>ABP2</td>\n",
       "      <td>SPBC1861.02</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.816301</td>\n",
       "      <td>1.969788</td>\n",
       "      <td>1.736116</td>\n",
       "      <td>1.057102</td>\n",
       "      <td>0.494822</td>\n",
       "      <td>0.372226</td>\n",
       "      <td>0.533438</td>\n",
       "      <td>0.851486</td>\n",
       "      <td>0.989702</td>\n",
       "      <td>1.115665</td>\n",
       "      <td>1.159631</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               0          1                         2    3          4   \\\n",
       "0            aap1       AAP1  SPBC1652.02 SPBC16A3.20C  1.0  22.703531   \n",
       "1  abc1: C2D10.18       ABC1               SPBC2D10.18  1.0   1.043491   \n",
       "2  abc1: C9E9.12c       ABC1               SPAC9E9.12C  1.0   2.112546   \n",
       "3            abp1  CBP1 ABP1              SPBC1105.04C  1.0   1.351277   \n",
       "4            abp2       ABP2               SPBC1861.02  1.0   0.816301   \n",
       "\n",
       "         5         6         7         8          9         10        11  \\\n",
       "0  0.960360  0.544460  0.896232  1.160251   1.252143  2.309153  2.310750   \n",
       "1  0.622283  0.663191  0.619409  0.648403   0.662236  0.655953  0.580761   \n",
       "2  0.906822  0.565746  0.535220  1.224398  10.170195  7.380023  4.455168   \n",
       "3  0.884985  0.705397  0.401055  0.202862   0.175360  0.829920  0.959749   \n",
       "4  1.969788  1.736116  1.057102  0.494822   0.372226  0.533438  0.851486   \n",
       "\n",
       "         12        13        14  \n",
       "0  2.449327  2.706837  3.064384  \n",
       "1  0.606205  0.752322  0.798264  \n",
       "2  2.804587  1.657471  1.310630  \n",
       "3  1.121036  1.122949  1.103435  \n",
       "4  0.989702  1.115665  1.159631  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# preview of the raw mRNA data\n",
    "print(raw_data_mRNA.shape)\n",
    "raw_data_mRNA.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The data from the Bählerlab (ULC) is a table of measures for **5121 mRNAs**.<br/>\n",
    "- For each mRNA a relative concentration is measured for **11 time points** (12 with t0, in which we are not so interested in). These relative concentrations are an **average** of 4 independent experiments.<br/>\n",
    "- These 11 concentrations are measured for 0h, 1h, 2h, 3h, 4h, 5h, 6h, 7h, 8h, 10h, 12h<br/>\n",
    "- As in the protein's dataset, most values are located between 1 and 2, but some values can be about several hundreds.<br/>\n",
    "- The first 3 columns are respectively : the primer name, the common name and the systematic name of the mRNA. Only the systematic name corresponds to the name given in our first dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For both datasets, **t0** is used as a ratio (thus, always equal to 1). In the proteins dataset, it is the time point at 0h (vegetative state), and in the mRNA dataset, it expresses another ratio, independent from time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 - Datasets preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### a) Proteins data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our first preprocessing consists in averaging the triplicates at each time point. <br/>\n",
    "Indead, it would be more simple to compute corralation between mRNA et proteins concentrations if we have a single value at each time point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# split protein data into 3 sets\n",
    "data_prot_A = pd.DataFrame(raw_data_prot.ix[:,0:11].values)\n",
    "data_prot_B = pd.DataFrame(raw_data_prot.ix[:,11:22].values)\n",
    "data_prot_C = pd.DataFrame(raw_data_prot.ix[:,22:33].values)\n",
    "\n",
    "# compute average of the 3 data sets\n",
    "data_prot = (data_prot_A + data_prot_B + data_prot_C).copy()/3.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to get measures at the same time points in both datasets, we remove the 9th measure (= 9h, absent from the mRNA dataset)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_prot = data_prot.drop(9, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we choose to use the protein's name as index, and name the features as the time of the measurement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2969, 10)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>h0</th>\n",
       "      <th>h1</th>\n",
       "      <th>h2</th>\n",
       "      <th>h3</th>\n",
       "      <th>h4</th>\n",
       "      <th>h5</th>\n",
       "      <th>h6</th>\n",
       "      <th>h7</th>\n",
       "      <th>h8</th>\n",
       "      <th>h10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SPAC1002.02</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.393371</td>\n",
       "      <td>3.257203</td>\n",
       "      <td>5.591645</td>\n",
       "      <td>6.932151</td>\n",
       "      <td>5.479337</td>\n",
       "      <td>3.574659</td>\n",
       "      <td>2.726113</td>\n",
       "      <td>2.114409</td>\n",
       "      <td>1.439884</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.03c</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.910152</td>\n",
       "      <td>0.785974</td>\n",
       "      <td>0.750362</td>\n",
       "      <td>0.746000</td>\n",
       "      <td>0.770242</td>\n",
       "      <td>0.800074</td>\n",
       "      <td>0.843706</td>\n",
       "      <td>0.872157</td>\n",
       "      <td>0.839864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.04c</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.990121</td>\n",
       "      <td>0.980455</td>\n",
       "      <td>1.050549</td>\n",
       "      <td>1.248047</td>\n",
       "      <td>1.217455</td>\n",
       "      <td>0.880529</td>\n",
       "      <td>0.704999</td>\n",
       "      <td>0.712103</td>\n",
       "      <td>0.795477</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.07c</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.046894</td>\n",
       "      <td>1.307253</td>\n",
       "      <td>1.640936</td>\n",
       "      <td>2.070811</td>\n",
       "      <td>2.125540</td>\n",
       "      <td>2.078482</td>\n",
       "      <td>1.997900</td>\n",
       "      <td>1.900363</td>\n",
       "      <td>1.780422</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.09c</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.904997</td>\n",
       "      <td>0.855320</td>\n",
       "      <td>0.872414</td>\n",
       "      <td>0.904201</td>\n",
       "      <td>0.923575</td>\n",
       "      <td>0.932110</td>\n",
       "      <td>0.939830</td>\n",
       "      <td>0.961271</td>\n",
       "      <td>1.018891</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               h0        h1        h2        h3        h4        h5        h6  \\\n",
       "SPAC1002.02   1.0  1.393371  3.257203  5.591645  6.932151  5.479337  3.574659   \n",
       "SPAC1002.03c  1.0  0.910152  0.785974  0.750362  0.746000  0.770242  0.800074   \n",
       "SPAC1002.04c  1.0  0.990121  0.980455  1.050549  1.248047  1.217455  0.880529   \n",
       "SPAC1002.07c  1.0  1.046894  1.307253  1.640936  2.070811  2.125540  2.078482   \n",
       "SPAC1002.09c  1.0  0.904997  0.855320  0.872414  0.904201  0.923575  0.932110   \n",
       "\n",
       "                    h7        h8       h10  \n",
       "SPAC1002.02   2.726113  2.114409  1.439884  \n",
       "SPAC1002.03c  0.843706  0.872157  0.839864  \n",
       "SPAC1002.04c  0.704999  0.712103  0.795477  \n",
       "SPAC1002.07c  1.997900  1.900363  1.780422  \n",
       "SPAC1002.09c  0.939830  0.961271  1.018891  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# set columns and rows indexes\n",
    "data_prot.columns = ['h'+str(k) for k in range(0,11) if k != 9]\n",
    "data_prot.index = raw_data_prot.index\n",
    "\n",
    "# show begining of dataframe\n",
    "print(data_prot.shape)\n",
    "data_prot.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### b) mRNA data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, we remove the first 2 columns (the primer name and the common name) in which we are not interested."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# remove first 2 columns\n",
    "data_mRNA = raw_data_mRNA.drop([0,1], axis=1).copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to get measures at the same time points in both datasets, we also remove t0 (which is meaningless for us) and t11 (=12h, which is missing in the proteins dataset)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_mRNA = data_mRNA.drop([3,14], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The remaining dataset counts 2434 missing values, involving 327 NaN entries. We chose to drop them :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_mRNA.dropna(how='any', inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, as for the protein's dataset, we use the name as index and timing as columns.<br/>\n",
    "Notice that some names are duplicates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4786, 10)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>h0</th>\n",
       "      <th>h1</th>\n",
       "      <th>h2</th>\n",
       "      <th>h3</th>\n",
       "      <th>h4</th>\n",
       "      <th>h5</th>\n",
       "      <th>h6</th>\n",
       "      <th>h7</th>\n",
       "      <th>h8</th>\n",
       "      <th>h10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SPBC1652.02 SPBC16A3.20C</th>\n",
       "      <td>22.703531</td>\n",
       "      <td>0.960360</td>\n",
       "      <td>0.544460</td>\n",
       "      <td>0.896232</td>\n",
       "      <td>1.160251</td>\n",
       "      <td>1.252143</td>\n",
       "      <td>2.309153</td>\n",
       "      <td>2.310750</td>\n",
       "      <td>2.449327</td>\n",
       "      <td>2.706837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPBC2D10.18</th>\n",
       "      <td>1.043491</td>\n",
       "      <td>0.622283</td>\n",
       "      <td>0.663191</td>\n",
       "      <td>0.619409</td>\n",
       "      <td>0.648403</td>\n",
       "      <td>0.662236</td>\n",
       "      <td>0.655953</td>\n",
       "      <td>0.580761</td>\n",
       "      <td>0.606205</td>\n",
       "      <td>0.752322</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC9E9.12C</th>\n",
       "      <td>2.112546</td>\n",
       "      <td>0.906822</td>\n",
       "      <td>0.565746</td>\n",
       "      <td>0.535220</td>\n",
       "      <td>1.224398</td>\n",
       "      <td>10.170195</td>\n",
       "      <td>7.380023</td>\n",
       "      <td>4.455168</td>\n",
       "      <td>2.804587</td>\n",
       "      <td>1.657471</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPBC1105.04C</th>\n",
       "      <td>1.351277</td>\n",
       "      <td>0.884985</td>\n",
       "      <td>0.705397</td>\n",
       "      <td>0.401055</td>\n",
       "      <td>0.202862</td>\n",
       "      <td>0.175360</td>\n",
       "      <td>0.829920</td>\n",
       "      <td>0.959749</td>\n",
       "      <td>1.121036</td>\n",
       "      <td>1.122949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPBC1861.02</th>\n",
       "      <td>0.816301</td>\n",
       "      <td>1.969788</td>\n",
       "      <td>1.736116</td>\n",
       "      <td>1.057102</td>\n",
       "      <td>0.494822</td>\n",
       "      <td>0.372226</td>\n",
       "      <td>0.533438</td>\n",
       "      <td>0.851486</td>\n",
       "      <td>0.989702</td>\n",
       "      <td>1.115665</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 h0        h1        h2        h3        h4  \\\n",
       "SPBC1652.02 SPBC16A3.20C  22.703531  0.960360  0.544460  0.896232  1.160251   \n",
       "SPBC2D10.18                1.043491  0.622283  0.663191  0.619409  0.648403   \n",
       "SPAC9E9.12C                2.112546  0.906822  0.565746  0.535220  1.224398   \n",
       "SPBC1105.04C               1.351277  0.884985  0.705397  0.401055  0.202862   \n",
       "SPBC1861.02                0.816301  1.969788  1.736116  1.057102  0.494822   \n",
       "\n",
       "                                 h5        h6        h7        h8       h10  \n",
       "SPBC1652.02 SPBC16A3.20C   1.252143  2.309153  2.310750  2.449327  2.706837  \n",
       "SPBC2D10.18                0.662236  0.655953  0.580761  0.606205  0.752322  \n",
       "SPAC9E9.12C               10.170195  7.380023  4.455168  2.804587  1.657471  \n",
       "SPBC1105.04C               0.175360  0.829920  0.959749  1.121036  1.122949  \n",
       "SPBC1861.02                0.372226  0.533438  0.851486  0.989702  1.115665  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# set columns indexes\n",
    "data_mRNA.set_index([2], inplace=True) # Some names are duplicates\n",
    "data_mRNA.index.name = None\n",
    "data_mRNA.columns = ['h'+str(k) for k in range(0,11) if k != 9]\n",
    "\n",
    "print(data_mRNA.shape)\n",
    "data_mRNA.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 - Merging the mRNA and protein data sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've got 2969 entries in our proteins dataset and 4786 entries in the mRNA one.<br/>\n",
    "We'll first reduce the 2 datasets to their common entries, and then join the 2 tables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### a) Filtering mRNA data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mRNA names can be composed of several names, we extract the mRNA entries which name contains the one of our proteins."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2874, 10)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>h0</th>\n",
       "      <th>h1</th>\n",
       "      <th>h2</th>\n",
       "      <th>h3</th>\n",
       "      <th>h4</th>\n",
       "      <th>h5</th>\n",
       "      <th>h6</th>\n",
       "      <th>h7</th>\n",
       "      <th>h8</th>\n",
       "      <th>h10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SPBC2D10.18</th>\n",
       "      <td>1.043491</td>\n",
       "      <td>0.622283</td>\n",
       "      <td>0.663191</td>\n",
       "      <td>0.619409</td>\n",
       "      <td>0.648403</td>\n",
       "      <td>0.662236</td>\n",
       "      <td>0.655953</td>\n",
       "      <td>0.580761</td>\n",
       "      <td>0.606205</td>\n",
       "      <td>0.752322</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPBC1105.04C</th>\n",
       "      <td>1.351277</td>\n",
       "      <td>0.884985</td>\n",
       "      <td>0.705397</td>\n",
       "      <td>0.401055</td>\n",
       "      <td>0.202862</td>\n",
       "      <td>0.175360</td>\n",
       "      <td>0.829920</td>\n",
       "      <td>0.959749</td>\n",
       "      <td>1.121036</td>\n",
       "      <td>1.122949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPBC32H8.12C</th>\n",
       "      <td>0.621846</td>\n",
       "      <td>0.325697</td>\n",
       "      <td>0.390388</td>\n",
       "      <td>0.419048</td>\n",
       "      <td>0.435039</td>\n",
       "      <td>0.539601</td>\n",
       "      <td>0.403898</td>\n",
       "      <td>0.337072</td>\n",
       "      <td>0.281851</td>\n",
       "      <td>0.265122</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC630.03</th>\n",
       "      <td>0.945575</td>\n",
       "      <td>0.576106</td>\n",
       "      <td>0.698916</td>\n",
       "      <td>0.638058</td>\n",
       "      <td>0.423130</td>\n",
       "      <td>0.352959</td>\n",
       "      <td>0.746192</td>\n",
       "      <td>0.829190</td>\n",
       "      <td>0.843786</td>\n",
       "      <td>0.889201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPBC106.04</th>\n",
       "      <td>0.871751</td>\n",
       "      <td>0.869876</td>\n",
       "      <td>1.023361</td>\n",
       "      <td>0.727945</td>\n",
       "      <td>0.261486</td>\n",
       "      <td>0.143920</td>\n",
       "      <td>0.437443</td>\n",
       "      <td>0.805628</td>\n",
       "      <td>0.984534</td>\n",
       "      <td>0.974699</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    h0        h1        h2        h3        h4        h5  \\\n",
       "SPBC2D10.18   1.043491  0.622283  0.663191  0.619409  0.648403  0.662236   \n",
       "SPBC1105.04C  1.351277  0.884985  0.705397  0.401055  0.202862  0.175360   \n",
       "SPBC32H8.12C  0.621846  0.325697  0.390388  0.419048  0.435039  0.539601   \n",
       "SPAC630.03    0.945575  0.576106  0.698916  0.638058  0.423130  0.352959   \n",
       "SPBC106.04    0.871751  0.869876  1.023361  0.727945  0.261486  0.143920   \n",
       "\n",
       "                    h6        h7        h8       h10  \n",
       "SPBC2D10.18   0.655953  0.580761  0.606205  0.752322  \n",
       "SPBC1105.04C  0.829920  0.959749  1.121036  1.122949  \n",
       "SPBC32H8.12C  0.403898  0.337072  0.281851  0.265122  \n",
       "SPAC630.03    0.746192  0.829190  0.843786  0.889201  \n",
       "SPBC106.04    0.437443  0.805628  0.984534  0.974699  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create the pattern and the filter\n",
    "pattern = '|'.join(data_prot.index.values)\n",
    "filter_ = data_mRNA.index.str.contains(pattern, case=False, na=False)\n",
    "\n",
    "# extract mRNA data with corresponding protein name\n",
    "data_mRNA = data_mRNA[filter_]\n",
    "\n",
    "# show preview\n",
    "print(data_mRNA.shape)\n",
    "data_mRNA.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've lost 95 entries, and still have 56 **duplicates**, representing 28 proteins."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(56, 10)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>h0</th>\n",
       "      <th>h1</th>\n",
       "      <th>h2</th>\n",
       "      <th>h3</th>\n",
       "      <th>h4</th>\n",
       "      <th>h5</th>\n",
       "      <th>h6</th>\n",
       "      <th>h7</th>\n",
       "      <th>h8</th>\n",
       "      <th>h10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SPAC13G7.02C</th>\n",
       "      <td>1.139014</td>\n",
       "      <td>5.825920</td>\n",
       "      <td>7.595293</td>\n",
       "      <td>19.737550</td>\n",
       "      <td>24.382723</td>\n",
       "      <td>27.602160</td>\n",
       "      <td>7.971517</td>\n",
       "      <td>6.919817</td>\n",
       "      <td>9.305873</td>\n",
       "      <td>12.082448</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC13G7.02C</th>\n",
       "      <td>0.480851</td>\n",
       "      <td>1.821550</td>\n",
       "      <td>2.200407</td>\n",
       "      <td>2.647362</td>\n",
       "      <td>3.204608</td>\n",
       "      <td>2.962069</td>\n",
       "      <td>0.816963</td>\n",
       "      <td>0.817025</td>\n",
       "      <td>0.790903</td>\n",
       "      <td>1.049258</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC144.03</th>\n",
       "      <td>0.597784</td>\n",
       "      <td>0.552436</td>\n",
       "      <td>0.423964</td>\n",
       "      <td>0.333494</td>\n",
       "      <td>0.287265</td>\n",
       "      <td>0.176862</td>\n",
       "      <td>0.230313</td>\n",
       "      <td>0.223476</td>\n",
       "      <td>0.261122</td>\n",
       "      <td>0.277014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC144.03</th>\n",
       "      <td>1.247429</td>\n",
       "      <td>2.754731</td>\n",
       "      <td>2.002274</td>\n",
       "      <td>1.818740</td>\n",
       "      <td>1.673060</td>\n",
       "      <td>2.102656</td>\n",
       "      <td>2.313404</td>\n",
       "      <td>2.398567</td>\n",
       "      <td>3.019608</td>\n",
       "      <td>3.698596</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC16E8.15</th>\n",
       "      <td>0.300177</td>\n",
       "      <td>1.893358</td>\n",
       "      <td>0.955042</td>\n",
       "      <td>0.580376</td>\n",
       "      <td>0.673911</td>\n",
       "      <td>1.094835</td>\n",
       "      <td>0.801017</td>\n",
       "      <td>0.614706</td>\n",
       "      <td>0.535658</td>\n",
       "      <td>0.551441</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    h0        h1        h2         h3         h4         h5  \\\n",
       "SPAC13G7.02C  1.139014  5.825920  7.595293  19.737550  24.382723  27.602160   \n",
       "SPAC13G7.02C  0.480851  1.821550  2.200407   2.647362   3.204608   2.962069   \n",
       "SPAC144.03    0.597784  0.552436  0.423964   0.333494   0.287265   0.176862   \n",
       "SPAC144.03    1.247429  2.754731  2.002274   1.818740   1.673060   2.102656   \n",
       "SPAC16E8.15   0.300177  1.893358  0.955042   0.580376   0.673911   1.094835   \n",
       "\n",
       "                    h6        h7        h8        h10  \n",
       "SPAC13G7.02C  7.971517  6.919817  9.305873  12.082448  \n",
       "SPAC13G7.02C  0.816963  0.817025  0.790903   1.049258  \n",
       "SPAC144.03    0.230313  0.223476  0.261122   0.277014  \n",
       "SPAC144.03    2.313404  2.398567  3.019608   3.698596  \n",
       "SPAC16E8.15   0.801017  0.614706  0.535658   0.551441  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "duplicated_data_mRNA = data_mRNA.loc[data_mRNA[data_mRNA.index.duplicated()].index.sort_values()]\n",
    "print(duplicated_data_mRNA.shape)\n",
    "duplicated_data_mRNA.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To keep things simple, we chose to add them. <br/>\n",
    "(As the duplicates are coding for the same protein, the sum of their concentration should be correlated with the proteins production)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/pierre/anaconda3/lib/python3.5/site-packages/ipykernel/__main__.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  if __name__ == '__main__':\n"
     ]
    }
   ],
   "source": [
    "duplicated_data_mRNA['name'] = duplicated_data_mRNA.index.values\n",
    "duplicated_data_mRNA_sum = duplicated_data_mRNA.groupby(['name']).sum()\n",
    "duplicated_data_mRNA_sum.index.name = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we replace them in our dataset :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data_mRNA_without_duplicates = data_mRNA.drop(duplicated_data_mRNA_sum.index)\n",
    "data_mRNA = pd.concat([data_mRNA_without_duplicates, duplicated_data_mRNA_sum])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### b) Ending with mRNA duplicates and Filtering proteins data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we have more proteins (2969) than we have mRNA (2858). Before extracting the common entries from the protein dataset, let's check if we have remaining duplicates :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['SPAC145.04 SPAC20H4.10', 'SPAC20H4.10'], dtype='object') SPAC20H4.10\n"
     ]
    }
   ],
   "source": [
    "duplicates = np.array([], dtype=object)\n",
    "\n",
    "for name in data_prot.index.values:\n",
    "    filter_ = data_mRNA.index.str.contains(name, case=False, na=False)\n",
    "    \n",
    "    if len(data_mRNA[filter_].index)>1:\n",
    "        print(data_mRNA[filter_].index, name)\n",
    "        duplicates = np.append(duplicates, data_mRNA[filter_].index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We still have one case of protein potentially produced by 2 distinct mRNA. Let's aggregate it as we did previously :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "last_duplicate = data_mRNA.loc[duplicates]\n",
    "last_duplicate_sum = last_duplicate.sum()\n",
    "last_duplicate_sum.name = 'SPAC20H4.10'\n",
    "\n",
    "data_mRNA = data_mRNA.drop(duplicates)\n",
    "data_mRNA = data_mRNA.append(last_duplicate_sum)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can extract the common entries (the ARNm positions corresponding to proteins names) :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# get list of indexes and names to sort\n",
    "idxname = np.array([], dtype=int)\n",
    "listname = np.array([], dtype=object)\n",
    "\n",
    "for name in data_prot.index.values:\n",
    "    filter_ = data_mRNA.index.str.contains(name, case=False, na=False)\n",
    "    \n",
    "    if len(data_mRNA[filter_].index)>0:\n",
    "        idxname = np.append(idxname, data_mRNA.index.get_loc(data_mRNA[filter_].index[0])) # mRNA numeric positions\n",
    "        listname = np.append(listname, name) # Protein index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's filter the proteins dataset :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2845, 10)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>h0</th>\n",
       "      <th>h1</th>\n",
       "      <th>h2</th>\n",
       "      <th>h3</th>\n",
       "      <th>h4</th>\n",
       "      <th>h5</th>\n",
       "      <th>h6</th>\n",
       "      <th>h7</th>\n",
       "      <th>h8</th>\n",
       "      <th>h10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SPAC1002.02</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.393371</td>\n",
       "      <td>3.257203</td>\n",
       "      <td>5.591645</td>\n",
       "      <td>6.932151</td>\n",
       "      <td>5.479337</td>\n",
       "      <td>3.574659</td>\n",
       "      <td>2.726113</td>\n",
       "      <td>2.114409</td>\n",
       "      <td>1.439884</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.03c</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.910152</td>\n",
       "      <td>0.785974</td>\n",
       "      <td>0.750362</td>\n",
       "      <td>0.746000</td>\n",
       "      <td>0.770242</td>\n",
       "      <td>0.800074</td>\n",
       "      <td>0.843706</td>\n",
       "      <td>0.872157</td>\n",
       "      <td>0.839864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.04c</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.990121</td>\n",
       "      <td>0.980455</td>\n",
       "      <td>1.050549</td>\n",
       "      <td>1.248047</td>\n",
       "      <td>1.217455</td>\n",
       "      <td>0.880529</td>\n",
       "      <td>0.704999</td>\n",
       "      <td>0.712103</td>\n",
       "      <td>0.795477</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.07c</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.046894</td>\n",
       "      <td>1.307253</td>\n",
       "      <td>1.640936</td>\n",
       "      <td>2.070811</td>\n",
       "      <td>2.125540</td>\n",
       "      <td>2.078482</td>\n",
       "      <td>1.997900</td>\n",
       "      <td>1.900363</td>\n",
       "      <td>1.780422</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.09c</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.904997</td>\n",
       "      <td>0.855320</td>\n",
       "      <td>0.872414</td>\n",
       "      <td>0.904201</td>\n",
       "      <td>0.923575</td>\n",
       "      <td>0.932110</td>\n",
       "      <td>0.939830</td>\n",
       "      <td>0.961271</td>\n",
       "      <td>1.018891</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               h0        h1        h2        h3        h4        h5        h6  \\\n",
       "SPAC1002.02   1.0  1.393371  3.257203  5.591645  6.932151  5.479337  3.574659   \n",
       "SPAC1002.03c  1.0  0.910152  0.785974  0.750362  0.746000  0.770242  0.800074   \n",
       "SPAC1002.04c  1.0  0.990121  0.980455  1.050549  1.248047  1.217455  0.880529   \n",
       "SPAC1002.07c  1.0  1.046894  1.307253  1.640936  2.070811  2.125540  2.078482   \n",
       "SPAC1002.09c  1.0  0.904997  0.855320  0.872414  0.904201  0.923575  0.932110   \n",
       "\n",
       "                    h7        h8       h10  \n",
       "SPAC1002.02   2.726113  2.114409  1.439884  \n",
       "SPAC1002.03c  0.843706  0.872157  0.839864  \n",
       "SPAC1002.04c  0.704999  0.712103  0.795477  \n",
       "SPAC1002.07c  1.997900  1.900363  1.780422  \n",
       "SPAC1002.09c  0.939830  0.961271  1.018891  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# removing non existant keys in protein data\n",
    "data_prot = data_prot.loc[listname]\n",
    "print(data_prot.shape)\n",
    "data_prot.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've lost 124 entries, and now we've got the same number of entries in both datasets (2845)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### c) Joining the 2 datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before merging the 2 datasets, we have to sort the mRNA dataset in the same order than the protein one :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>h0</th>\n",
       "      <th>h1</th>\n",
       "      <th>h2</th>\n",
       "      <th>h3</th>\n",
       "      <th>h4</th>\n",
       "      <th>h5</th>\n",
       "      <th>h6</th>\n",
       "      <th>h7</th>\n",
       "      <th>h8</th>\n",
       "      <th>h10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SPAC1002.02</th>\n",
       "      <td>1.035300</td>\n",
       "      <td>2.905877</td>\n",
       "      <td>4.153496</td>\n",
       "      <td>4.783120</td>\n",
       "      <td>2.603015</td>\n",
       "      <td>1.647060</td>\n",
       "      <td>1.481230</td>\n",
       "      <td>1.616345</td>\n",
       "      <td>1.347246</td>\n",
       "      <td>1.122512</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.03c</th>\n",
       "      <td>0.958027</td>\n",
       "      <td>0.676712</td>\n",
       "      <td>0.703215</td>\n",
       "      <td>0.747138</td>\n",
       "      <td>0.861940</td>\n",
       "      <td>1.257074</td>\n",
       "      <td>1.235414</td>\n",
       "      <td>0.950157</td>\n",
       "      <td>0.988842</td>\n",
       "      <td>1.161624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.04c</th>\n",
       "      <td>1.133477</td>\n",
       "      <td>1.140642</td>\n",
       "      <td>1.189899</td>\n",
       "      <td>1.622498</td>\n",
       "      <td>2.308321</td>\n",
       "      <td>3.020603</td>\n",
       "      <td>1.853997</td>\n",
       "      <td>1.714919</td>\n",
       "      <td>1.682955</td>\n",
       "      <td>1.530989</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.07c</th>\n",
       "      <td>0.783431</td>\n",
       "      <td>1.362667</td>\n",
       "      <td>1.602399</td>\n",
       "      <td>1.836938</td>\n",
       "      <td>1.938045</td>\n",
       "      <td>1.969526</td>\n",
       "      <td>0.570085</td>\n",
       "      <td>0.593492</td>\n",
       "      <td>0.524142</td>\n",
       "      <td>0.437590</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.09c</th>\n",
       "      <td>0.728322</td>\n",
       "      <td>0.701878</td>\n",
       "      <td>1.354810</td>\n",
       "      <td>1.140213</td>\n",
       "      <td>0.637736</td>\n",
       "      <td>0.368306</td>\n",
       "      <td>0.276762</td>\n",
       "      <td>0.236933</td>\n",
       "      <td>0.258554</td>\n",
       "      <td>0.260154</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    h0        h1        h2        h3        h4        h5  \\\n",
       "SPAC1002.02   1.035300  2.905877  4.153496  4.783120  2.603015  1.647060   \n",
       "SPAC1002.03c  0.958027  0.676712  0.703215  0.747138  0.861940  1.257074   \n",
       "SPAC1002.04c  1.133477  1.140642  1.189899  1.622498  2.308321  3.020603   \n",
       "SPAC1002.07c  0.783431  1.362667  1.602399  1.836938  1.938045  1.969526   \n",
       "SPAC1002.09c  0.728322  0.701878  1.354810  1.140213  0.637736  0.368306   \n",
       "\n",
       "                    h6        h7        h8       h10  \n",
       "SPAC1002.02   1.481230  1.616345  1.347246  1.122512  \n",
       "SPAC1002.03c  1.235414  0.950157  0.988842  1.161624  \n",
       "SPAC1002.04c  1.853997  1.714919  1.682955  1.530989  \n",
       "SPAC1002.07c  0.570085  0.593492  0.524142  0.437590  \n",
       "SPAC1002.09c  0.276762  0.236933  0.258554  0.260154  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_mRNA = data_mRNA.iloc[list(idxname)]\n",
    "data_mRNA.set_index([listname.tolist()], inplace=True)\n",
    "data_mRNA.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can join the 2 datasets :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of joined data: (2845, 20)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"10\" halign=\"left\">prot</th>\n",
       "      <th colspan=\"10\" halign=\"left\">mRNA</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>h0</th>\n",
       "      <th>h1</th>\n",
       "      <th>h2</th>\n",
       "      <th>h3</th>\n",
       "      <th>h4</th>\n",
       "      <th>h5</th>\n",
       "      <th>h6</th>\n",
       "      <th>h7</th>\n",
       "      <th>h8</th>\n",
       "      <th>h10</th>\n",
       "      <th>h0</th>\n",
       "      <th>h1</th>\n",
       "      <th>h2</th>\n",
       "      <th>h3</th>\n",
       "      <th>h4</th>\n",
       "      <th>h5</th>\n",
       "      <th>h6</th>\n",
       "      <th>h7</th>\n",
       "      <th>h8</th>\n",
       "      <th>h10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SPAC1002.02</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.393371</td>\n",
       "      <td>3.257203</td>\n",
       "      <td>5.591645</td>\n",
       "      <td>6.932151</td>\n",
       "      <td>5.479337</td>\n",
       "      <td>3.574659</td>\n",
       "      <td>2.726113</td>\n",
       "      <td>2.114409</td>\n",
       "      <td>1.439884</td>\n",
       "      <td>1.035300</td>\n",
       "      <td>2.905877</td>\n",
       "      <td>4.153496</td>\n",
       "      <td>4.783120</td>\n",
       "      <td>2.603015</td>\n",
       "      <td>1.647060</td>\n",
       "      <td>1.481230</td>\n",
       "      <td>1.616345</td>\n",
       "      <td>1.347246</td>\n",
       "      <td>1.122512</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.03c</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.910152</td>\n",
       "      <td>0.785974</td>\n",
       "      <td>0.750362</td>\n",
       "      <td>0.746000</td>\n",
       "      <td>0.770242</td>\n",
       "      <td>0.800074</td>\n",
       "      <td>0.843706</td>\n",
       "      <td>0.872157</td>\n",
       "      <td>0.839864</td>\n",
       "      <td>0.958027</td>\n",
       "      <td>0.676712</td>\n",
       "      <td>0.703215</td>\n",
       "      <td>0.747138</td>\n",
       "      <td>0.861940</td>\n",
       "      <td>1.257074</td>\n",
       "      <td>1.235414</td>\n",
       "      <td>0.950157</td>\n",
       "      <td>0.988842</td>\n",
       "      <td>1.161624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.04c</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.990121</td>\n",
       "      <td>0.980455</td>\n",
       "      <td>1.050549</td>\n",
       "      <td>1.248047</td>\n",
       "      <td>1.217455</td>\n",
       "      <td>0.880529</td>\n",
       "      <td>0.704999</td>\n",
       "      <td>0.712103</td>\n",
       "      <td>0.795477</td>\n",
       "      <td>1.133477</td>\n",
       "      <td>1.140642</td>\n",
       "      <td>1.189899</td>\n",
       "      <td>1.622498</td>\n",
       "      <td>2.308321</td>\n",
       "      <td>3.020603</td>\n",
       "      <td>1.853997</td>\n",
       "      <td>1.714919</td>\n",
       "      <td>1.682955</td>\n",
       "      <td>1.530989</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.07c</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.046894</td>\n",
       "      <td>1.307253</td>\n",
       "      <td>1.640936</td>\n",
       "      <td>2.070811</td>\n",
       "      <td>2.125540</td>\n",
       "      <td>2.078482</td>\n",
       "      <td>1.997900</td>\n",
       "      <td>1.900363</td>\n",
       "      <td>1.780422</td>\n",
       "      <td>0.783431</td>\n",
       "      <td>1.362667</td>\n",
       "      <td>1.602399</td>\n",
       "      <td>1.836938</td>\n",
       "      <td>1.938045</td>\n",
       "      <td>1.969526</td>\n",
       "      <td>0.570085</td>\n",
       "      <td>0.593492</td>\n",
       "      <td>0.524142</td>\n",
       "      <td>0.437590</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SPAC1002.09c</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.904997</td>\n",
       "      <td>0.855320</td>\n",
       "      <td>0.872414</td>\n",
       "      <td>0.904201</td>\n",
       "      <td>0.923575</td>\n",
       "      <td>0.932110</td>\n",
       "      <td>0.939830</td>\n",
       "      <td>0.961271</td>\n",
       "      <td>1.018891</td>\n",
       "      <td>0.728322</td>\n",
       "      <td>0.701878</td>\n",
       "      <td>1.354810</td>\n",
       "      <td>1.140213</td>\n",
       "      <td>0.637736</td>\n",
       "      <td>0.368306</td>\n",
       "      <td>0.276762</td>\n",
       "      <td>0.236933</td>\n",
       "      <td>0.258554</td>\n",
       "      <td>0.260154</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             prot                                                              \\\n",
       "               h0        h1        h2        h3        h4        h5        h6   \n",
       "SPAC1002.02   1.0  1.393371  3.257203  5.591645  6.932151  5.479337  3.574659   \n",
       "SPAC1002.03c  1.0  0.910152  0.785974  0.750362  0.746000  0.770242  0.800074   \n",
       "SPAC1002.04c  1.0  0.990121  0.980455  1.050549  1.248047  1.217455  0.880529   \n",
       "SPAC1002.07c  1.0  1.046894  1.307253  1.640936  2.070811  2.125540  2.078482   \n",
       "SPAC1002.09c  1.0  0.904997  0.855320  0.872414  0.904201  0.923575  0.932110   \n",
       "\n",
       "                                                mRNA                      \\\n",
       "                    h7        h8       h10        h0        h1        h2   \n",
       "SPAC1002.02   2.726113  2.114409  1.439884  1.035300  2.905877  4.153496   \n",
       "SPAC1002.03c  0.843706  0.872157  0.839864  0.958027  0.676712  0.703215   \n",
       "SPAC1002.04c  0.704999  0.712103  0.795477  1.133477  1.140642  1.189899   \n",
       "SPAC1002.07c  1.997900  1.900363  1.780422  0.783431  1.362667  1.602399   \n",
       "SPAC1002.09c  0.939830  0.961271  1.018891  0.728322  0.701878  1.354810   \n",
       "\n",
       "                                                                          \\\n",
       "                    h3        h4        h5        h6        h7        h8   \n",
       "SPAC1002.02   4.783120  2.603015  1.647060  1.481230  1.616345  1.347246   \n",
       "SPAC1002.03c  0.747138  0.861940  1.257074  1.235414  0.950157  0.988842   \n",
       "SPAC1002.04c  1.622498  2.308321  3.020603  1.853997  1.714919  1.682955   \n",
       "SPAC1002.07c  1.836938  1.938045  1.969526  0.570085  0.593492  0.524142   \n",
       "SPAC1002.09c  1.140213  0.637736  0.368306  0.276762  0.236933  0.258554   \n",
       "\n",
       "                        \n",
       "                   h10  \n",
       "SPAC1002.02   1.122512  \n",
       "SPAC1002.03c  1.161624  \n",
       "SPAC1002.04c  1.530989  \n",
       "SPAC1002.07c  0.437590  \n",
       "SPAC1002.09c  0.260154  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#We add a hierarchical column name\n",
    "data_mRNA = pd.concat({\"mRNA\": pd.DataFrame(data_mRNA)}, axis=1)\n",
    "data_prot = pd.concat({\"prot\": pd.DataFrame(data_prot)}, axis=1)\n",
    "\n",
    "#And join\n",
    "data = data_prot.join(data_mRNA)\n",
    "print('Size of joined data:', data.shape)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4 - Data Standardization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we will need to cluster our data, it could be useful to standardize them.<br/>\n",
    "First of all, we define a suitable standization function :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def standardize_by_row(df):\n",
    "    mean_rows = df.mean(axis=1)\n",
    "    std_rows = df.std(axis=1)\n",
    "    \n",
    "    df = df.sub(mean_rows, axis=0)\n",
    "    df = df.div(std_rows, axis=0)\n",
    "\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We standardize the 10 measures of each mRNA and proteins independently :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data_prot_norm = standardize_by_row(data_prot)\n",
    "data_mRNA_norm = standardize_by_row(data_mRNA)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally join the standardized datasets :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data_norm = data_prot_norm.join(data_mRNA_norm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Unstandardized data\n",
    "data_prot.to_csv('data/data_prot.csv')\n",
    "data_mRNA.to_csv('data/data_mRNA.csv')\n",
    "data.to_csv('data/data.csv')\n",
    "\n",
    "# Standardized data\n",
    "data_prot_norm.to_csv('data/data_prot_norm.csv')\n",
    "data_mRNA_norm.to_csv('data/data_mRNA_norm.csv')\n",
    "data_norm.to_csv('data/data_norm.csv')"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}