0306_first_data_overview.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ecotox Data Documentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Essa é uma documentação inicial sobre a base de dados Ecotox.\n",
    "\n",
    "Com sua manipulação, é esperado entender mais sobre:\n",
    "\n",
    "- Quais combinações usadas em testes causam malefícios a determinadas famílias (ou outro ranking taxonomico)\n",
    "- Prever os efeitos com base nos materias e procedimentos\n",
    "\n",
    "O objetivo deste estudo é evitar ou minimizar a necessidade de teste em animais."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fonte dos dados:\n",
    "\n",
    "https://cfpub.epa.gov/ecotox/\n",
    "\n",
    "\n",
    "ECOTOX Support\n",
    "\n",
    "U.S. Environmental Protection Agency\n",
    "\n",
    "Office of Research and Development\n",
    "\n",
    "National Health and Environmental Effects Research Laboratory Mid-Continent Ecology Division (MED)\n",
    "\n",
    "6201 Congdon Boulevard\n",
    "\n",
    "Duluth, Minnesota 55804\n",
    "\n",
    "Telephone: 218-529-5225\n",
    "\n",
    "Fax: 218-529-5003\n",
    "\n",
    "E-mail: ecotox.support@epa.gov"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tabela de dados\n",
    "\n",
    "- TESTS\n",
    "\n",
    "o Information pertaining to the experimental design.\n",
    "\n",
    "- CHEMICAL_CARRIERS\n",
    "\n",
    "o Information pertaining to the carrier and/or positive control chemicals\n",
    "reported for the test. \n",
    "\n",
    "- RESULTS\n",
    "\n",
    "o Information pertaining to the endpoint or non-endpoint result or dose- response summary (also NR endpoint).\n",
    "\n",
    "- MEDIA_CHARACTERISTICS\n",
    "\n",
    "o Water chemistry and media characteristics parameters. 1-1 relation with\n",
    "RESULTS table. \n",
    "\n",
    "- DOSES\n",
    "\n",
    "o Information pertaining to the dose-response dose. \n",
    "\n",
    "- DOSE_RESPONSES\n",
    "\n",
    "o Parent dose response record containing sample size, effect measurement, response site, observation duration, etc.\n",
    "\n",
    "- DOSE_RESPONSE_DETAILS\n",
    "\n",
    "o Detail dose response record, one for each response value by dose.\n",
    "\n",
    "- DOSE_RESPONSE_LINKS\n",
    "\n",
    "o Ties dose response to its NR endpoint result summary record.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A tabela ```doses``` é ligada à de testes pelo ```test_id```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test_id</th>\n",
       "      <th>dose_id</th>\n",
       "      <th>dose_conc_unit</th>\n",
       "      <th>dose1_mean</th>\n",
       "      <th>dose2_mean</th>\n",
       "      <th>dose3_mean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>mg/kg</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>mg/kg</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>mg/kg</td>\n",
       "      <td>25</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>mg/kg</td>\n",
       "      <td>125</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>mg/kg</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>mg/kg</td>\n",
       "      <td>150</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  test_id dose_id dose_conc_unit dose1_mean dose2_mean dose3_mean\n",
       "0       1       1          mg/kg          0        NaN        NaN\n",
       "1       1       2          mg/kg          5        NaN        NaN\n",
       "2       1       3          mg/kg         25        NaN        NaN\n",
       "3       1       4          mg/kg        125        NaN        NaN\n",
       "4       2       5          mg/kg          0        NaN        NaN\n",
       "5       2       6          mg/kg        150        NaN        NaN"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_dose = pd.read_csv(\"../data/ecotox_ascii_12_13_2018/doses.txt\",sep=\"|\", dtype='unicode')\n",
    "df_dose.loc[:5, ['test_id', 'dose_id', 'dose_conc_unit', 'dose1_mean', 'dose2_mean', 'dose3_mean']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A tabela de ```dose_respose``` apresenta os efeitos dos testes e os códigos são explicados por ```effect_codes```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test_id</th>\n",
       "      <th>effect_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>ENZ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>ENZ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>HRM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>ENZ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>ENZ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>ENZ</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  test_id effect_code\n",
       "0       1         ENZ\n",
       "1       1         ENZ\n",
       "2       1         HRM\n",
       "3       2         ENZ\n",
       "4       2         ENZ\n",
       "5       2         ENZ"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_dose_responses = pd.read_csv(\"../data/ecotox_ascii_12_13_2018/dose_responses.txt\",sep=\"|\",dtype='unicode')\n",
    "df_dose_responses.loc[:5, ['test_id', 'effect_code']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>code</th>\n",
       "      <th>description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>--</td>\n",
       "      <td>Unspecified</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ACC</td>\n",
       "      <td>Accumulation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AVO</td>\n",
       "      <td>Avoidance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>BCM</td>\n",
       "      <td>Biochemistry</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>BEH</td>\n",
       "      <td>Behavior</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>CEL</td>\n",
       "      <td>Cell(s)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  code   description\n",
       "0   --   Unspecified\n",
       "1  ACC  Accumulation\n",
       "2  AVO     Avoidance\n",
       "3  BCM  Biochemistry\n",
       "4  BEH      Behavior\n",
       "5  CEL       Cell(s)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_effect_codes = pd.read_csv(\"../data/ecotox_ascii_12_13_2018/validation/effect_codes.txt\",sep=\"|\",dtype='unicode')\n",
    "df_effect_codes.loc[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A tabela de ```results``` apresenta os efeitos dos testes de acordo com o ```test_id```, que é a chave principal da base de dados e é ligada a família dos animais por ```tests```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test_id</th>\n",
       "      <th>effect</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1143197</td>\n",
       "      <td>MOR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1047376</td>\n",
       "      <td>POP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1152742</td>\n",
       "      <td>POP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1101244</td>\n",
       "      <td>ITX</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1210976</td>\n",
       "      <td>CEL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1035811</td>\n",
       "      <td>MLT</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   test_id effect\n",
       "0  1143197    MOR\n",
       "1  1047376    POP\n",
       "2  1152742    POP\n",
       "3  1101244    ITX\n",
       "4  1210976    CEL\n",
       "5  1035811    MLT"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_results = pd.read_csv(\"../data/ecotox_ascii_12_13_2018/results.txt\",sep=\"|\", dtype='unicode')\n",
    "df_results.loc[:5, ['test_id', 'effect']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test_id</th>\n",
       "      <th>species_number</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>4906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>4906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>4510</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>4435</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>4435</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>4435</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  test_id species_number\n",
       "0       1           4906\n",
       "1       2           4906\n",
       "2       3           4510\n",
       "3       4           4435\n",
       "4       5           4435\n",
       "5       6           4435"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_tests = pd.read_csv(\"../data/ecotox_ascii_12_13_2018/tests.txt\",sep=\"|\", dtype='unicode')\n",
    "df_tests.loc[:5, ['test_id', 'species_number']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Um exemplo de manipulação simples feita para gerar os dados de forma mais consistente e consica é dado abaixo:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ds = pd.read_csv(\"dataset_dose.csv\")\n",
    "df_ds.loc[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}