tutorial/pyam_first_steps.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# First steps with the ``pyam`` package \n",
    "\n",
    "## *An open-source Python package for IAM scenario analysis and visualization*\n",
    "\n",
    "<img style=\"float: right; height: 100px; margin-top: 10px;\" src=\"_static/IIASA_logo.png\">\n",
    "<img style=\"float: right; height: 80px;\" src=\"_static/IAMC_logo.jpg\">\n",
    "\n",
    "## Scope and feature overview\n",
    "\n",
    "The ``pyam`` package provides a range of diagnostic tools and functions  \n",
    "for analyzing and working with scenario data following the IAMC template format.\n",
    "A comprehensive documentation of the package is available\n",
    "at [software.ene.iiasa.ac.at/pyam/](http://software.ene.iiasa.ac.at/pyam/)\n",
    "\n",
    "An illustrative example of the IAMC template is shown below;\n",
    "see [data.ene.iiasa.ac.at/database/](http://data.ene.iiasa.ac.at/database/) for more information.\n",
    "\n",
    "\n",
    "| **Model**           | **Scenario**  | **Region** | **Variable**   | **Unit** | **2005** | **2010** | **2015** |\n",
    "|---------------------|---------------|------------|----------------|----------|----------|----------|----------|\n",
    "| MESSAGE V.4         | AMPERE3-Base  | World      | Primary Energy | EJ/y     | 454.5    |\t479.6    | ...      |\n",
    "| ...                 | ...           | ...        | ...            | ...      | ...      | ...      | ...      |\n",
    "\n",
    "This notebook illustrates some basic functionality of the ``pyam`` package\n",
    "and the ``IamDataFrame`` class:\n",
    "\n",
    "0. Importing timeseries data from `xlsx` or `csv` files.\n",
    "0. Listing models, scenarios and variables included in the data.\n",
    "0. Display of timeseries data \n",
    "   as [`pd.DataFrame`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html).\n",
    "0. Visualization tools for timeseries data using the [`matplotlib`](https://matplotlib.org/) package.\n",
    "0. Evaluating the model data and executing a range of diagnostic checks for identifying outliers.\n",
    "0. Categorization of scenarios according to timeseries data values or checks on required variables.\n",
    "0. Exporting data to `xlsx` using the IAMC template.\n",
    "\n",
    "\n",
    "## Tutorial data\n",
    "\n",
    "The timeseries data used in this tutorial is a partial snapshot of the scenario database \n",
    "compiled for the IPCC's Fifth Assessment Report (AR5):\n",
    "\n",
    "> Krey V., O. Masera, G. Blanford, T. Bruckner, R. Cooke, K. Fisher-Vanden, H. Haberl, E. Hertwich, E. Kriegler, D. Mueller, S. Paltsev, L. Price, S. Schlömer, D. Ürge-Vorsatz, D. van Vuuren, and T. Zwickel, 2014: *Annex II: Metrics & Methodology*.   \n",
    "> In: *Climate Change 2014: Mitigation of Climate Change. Contribution of Working Group III to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change* [Edenhofer, O., R. Pichs-Madruga, Y. Sokona, E. Farahani, S. Kadner, K. Seyboth, A. Adler, I. Baum, S. Brunner, P. Eickemeier, B. Kriemann, J. Savolainen, S. Schlömer, C. von Stechow, T. Zwickel and J.C. Minx (eds.)]. Cambridge University Press, Cambridge, United Kingdom and New York, NY, USA. [Link](https://www.ipcc.ch/report/ar5/wg3/)\n",
    "\n",
    "The complete database is publicly available at [tntcat.iiasa.ac.at/AR5DB/](https://tntcat.iiasa.ac.at/AR5DB/).\n",
    "\n",
    "<img style=\"float: right; height: 100px;\" src=\"_static/AMPERE-Logo.png\">\n",
    "<img style=\"float: right; height: 40px; margin-top: 50px; margin-right: 20px;\" src=\"_static/EMF-Logo_v2.1.png\">\n",
    "\n",
    "The data snapshot used for this tutorial consists of selected data from two model intercomparison projects:\n",
    " - Energy Modeling Forum Round 27 \n",
    "   ([**EMF27**](https://emf.stanford.edu/projects/emf-27-global-model-comparison-exercise)),\n",
    "   see the Special Issue in [*Climatic Change* 3-4, 2014](https://link.springer.com/journal/10584/123/3/page/1).\n",
    " - EU FP7 project [**AMPERE**](https://tntcat.iiasa.ac.at/AMPEREDB/), \n",
    "   see the following scientific publications:\n",
    "   > - Riahi, K., et al. (2015). \"Locked into Copenhagen pledges — Implications of short-term emission targets \n",
    "   >   for the cost and feasibility of long-term climate goals.\" \n",
    "   >   *Technological Forecasting and Social Change* 90(Part A): 8-23.  \n",
    "   >   [DOI: 10.1016/j.techfore.2013.09.016](https://doi.org/10.1016/j.techfore.2013.09.016)\n",
    "   > - Kriegler, E., et al. (2015). \"Making or breaking climate targets: The AMPERE study on \n",
    "   >   staged accession scenarios for climate policy.\"\n",
    "   >   *Technological Forecasting and Social Change* 90(Part A): 24-44.  \n",
    "   >   [DOI: 10.1016/j.techfore.2013.09.021](https://doi.org/10.1016/j.techfore.2013.09.021)\n",
    "\n",
    "<div style=\"text-align: center; padding: 10px; border: 2px solid red; width: 700px\">\n",
    "*The data used in this tutorial is ONLY a partial snapshot of the IPCC AR5 scenario database!*  \n",
    "*This tutorial is only intended for an illustration of the ``pyam`` package.*\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import package and load data from the AR5 tutorial snapshot\n",
    "\n",
    "We import the snapshot timeseries data from the file ``tutorial_AR5_data.csv`` in the ``tutorial`` folder.\n",
    "\n",
    "As a first step, we show lists of all models, scenarios, regions, and the variables and units included in the snapshot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyam\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pyam.IamDataFrame(data='tutorial_AR5_data.csv', encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.models()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.scenarios()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.regions()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.variables(include_units=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tutorial on data filtering\n",
    "A selection of the timeseries data  of an ``IamDataFrame`` can be obtained by applying the ``filter()`` funtion,\n",
    "which takes a dictionary of filter criteria as argument.\n",
    "The function `filter()` returns a filtered clone of the ``IamDataFrame``.\n",
    "\n",
    "### Filtering by model names, scenarios and regions\n",
    "\n",
    "The feature for filtering by **model, scenario or region** \n",
    "are implemented using exact string matching, where ``*`` can be used as a wildcard:\n",
    "\n",
    "> Applying the keyword argument filter ``model='MESSAGE'`` to the ``IamDataFrame``\n",
    "  will return an empty array.\n",
    "\n",
    "> Filtering for ``model='MESSAGE*'`` will return all scenarios from the *\"MESSAGE\"* family,\n",
    "> identified by the model name \"MESSAGE\" and a version identifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.filter(model='MESSAGE').models()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.filter(model='MESSAGE*')[['model', 'scenario']].drop_duplicates()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering by variables and hierarchy levels\n",
    "\n",
    "Filtering for **variable** strings works in an identical way as above,\n",
    "with ``*`` available as a wildcard.\n",
    "\n",
    "> Filtering for _Primary Energy_ will return only exactly those data.\n",
    "\n",
    "> Filtering for _Primary Energy|*_ will return all sub-categories of \n",
    "> primary-energy level (and only the sub-categories).\n",
    "\n",
    "In additon, IAM variables can be filtered by the **level**,\n",
    "i.e., the \"depth\" of the variable in a hierarchical reading of the string separated by `'|'`.\n",
    "That is, the variable _Primary Energy_ has level 0, while _Primary Energy|Coal_ has level 1.\n",
    "Filtering by both **variables** and **level** will search for the hierarchical depth \n",
    "_following the variable string_, so filter arguments ``'variable': 'Primary Energy|*'`` and ``'level': 0``\n",
    "will return all variables immediately below ``'Primary Energy'``.\n",
    "Filtering by **level** only will return all variables at that depth.\n",
    "\n",
    "To illustrate the functionality of the filters, we first show all sub-categories of the ``'Emissions'`` variable.  \n",
    "Then, we reduce variables to only one hierarchical levels below ``'Emissions|'``; the list returned by the function call will include only ``'Emissions|CO2|Fossil Fuels and Industry'``, because the level argument (by default) only shows variables at exactly the hierarchical levels below ``'Emissions|...'``.\n",
    "\n",
    "The third example illustrates another use case of the level argument - filtering by `'1-'` instead of `1` will return all variables *up to* the specified depth.\n",
    "\n",
    "The last cell shows how to filter only by hierarchical level, without providing a variable string.\n",
    "The function returns all variables that are at the top hierarchical level (i.e., ``'Primary Energy'``) and those at the first sub-category level.\n",
    "Keep in mind that there are no variables ``'Emissions'`` or ``'Price'`` (no top level)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.filter(variable='Emissions|*').variables()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.filter(variable='Emissions|*', level=1).variables()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.filter(variable='Emissions|*', level='1-').variables()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.filter(level='1-').variables()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering by year\n",
    "\n",
    "Filtering for **years** can be done by integer number, a list of integers, or the Python class ``range``.  \n",
    "Note that the last year of a range is not included, so ``range(2010,2015)``\n",
    "is interpreted as ``[2010, 2011, 2012, 2013, 2014]``."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting help\n",
    "\n",
    "When in doubt, you can look at the help for any function by appending it with a ``?``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.filter?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Displaying timeseries data\n",
    "\n",
    "As a next step, we want to view a selection of the data in the tutorial snapshot using the IAMC standard.\n",
    "The `timeseries()` function returns the data in the standard IAMC format as a `pd.DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(df\n",
    " .filter(scenario='AMPERE3-450', variable='Primary Energy|Coal', region='World')\n",
    " .timeseries()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For displaying data in a different format, the class ``IamDataFrame`` has a wrapper of the ``pd.DataFrame.pivot_table()`` function. It allows to flexibly specify the columns and rows.\n",
    "The function automatically aggregates by summation or counting (specified by the parameter `aggfunc`) \n",
    "over all timeseries data identifiers ('model', 'scenario', 'variable', 'region', 'unit', 'year')\n",
    "which are not used as `index` or `columns`.\n",
    "\n",
    "In the example below, the filter of the timeseries data is set for all subcategories of 'Primary Energy', \n",
    "which are then summed up in the displayed table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(df\n",
    " .filter(variable='Primary Energy', region='World')\n",
    " .pivot_table(index=['year'], columns=['scenario'], values='value', aggfunc='sum')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are familiar with the `python` package `pandas`, you can use many of usual functions directly on the ``IamDataFrame``. The function ``head()``, for example, will show the first `n` rows of the data in long form \n",
    "(columns are in year/value format)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization of timeseries\n",
    "\n",
    "This section provides one illustrative example of the plotting features of the ``pyam`` package.  \n",
    "Please see the `plotting.ipynb` notebook for a full tutorial on plotting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.filter(variable='Emissions|CO2', region='World').line_plot(legend=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validation and diagnostic assessment of timeseries data\n",
    "\n",
    "When analyzing scenario results, it is often useful to check whether certain timeseries exist or the values are within a specific range. For example, it may make sense to ensure that reported data for historical periods are close to established reference data or that near-term developments are reasonable.\n",
    "\n",
    "The following section provides three illustrations:\n",
    "1. Check whether a timeseries `'Primary Energy'` exists in each scenario (in at least one year).\n",
    "2. Check for every scenario whether the value for `'Primary Energy'` at the global level exceeds 515 EJ/y \n",
    "   in the reference year 2010\n",
    "   (the value must satisfy an upper bound of 515 EJ/y in this notation).\n",
    "3. Check for every scenario from the `AMPERE` project\n",
    "   whether the value for `'Primary Energy|Coal'` exceeds 400 EJ/y in mid-century.\n",
    "\n",
    "The `validate()` function takes a `filters` dictionary to perform the checks on a selection of models/scenarios\n",
    "similar to the functions introduced above.  \n",
    "The ``criteria`` argument can specify a valid range by an upper and lower bound (``up``, ``lo``) for a variable and a subset of years to which the validation is applied - all scenarios with a value in at least one year outside that range are considered to *not satisfy* the validation.\n",
    "\n",
    "By setting the argument ``exclude=True``, all scenarios failing the validation will be categorized as ``exclude`` in the metadata. This allows to remove these scenarios from subsequent analysis or figures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.require_variable(variable='Primary Energy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.validate(criteria={'Primary Energy': {'up': 515, 'year': 2010}})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pyam.validate(df, \n",
    "              filters={'region': 'World', 'scenario': 'AMPERE*'}, \n",
    "              criteria={'Primary Energy|Coal': {'up': 400, 'year': 2050}}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categorization of scenarios by timeseries characteristics\n",
    "\n",
    "It is often useful to apply categorization to classes of scenarios according to specific characteristics of the timeseries data. In the following example, we use the temperature change assessment by MAGICC 6 to group scenarios by the median global warming by the end of the century (year 2100).\n",
    "\n",
    "We proceed in the following steps:\n",
    "\n",
    "0. Plot the timeseries data of the variable that we want to use. \n",
    "   This provides some insights on useful thresholds for the categorization.\n",
    "0. Use the function ``categorize()`` to apply a categorization (and colour code for later use) \n",
    "   to all scenarios that satisfy a number of specific criteria.\n",
    "0. Use the categorization of scenarios for analysis of other timeseries data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v = 'Temperature|Global Mean|MAGICC6|MED'\n",
    "df.filter(region='World', variable=v).line_plot(legend=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now use the categorization feature of the ``pyam`` package to group scenarios by temperature outcome by the end of the century.\n",
    "\n",
    "The first cell sets the ``'Temperature'`` categorization to the default `\"uncategorized\"`.\n",
    "This is not necessary per se (setting a meta column via the categorization will mark all non-assigned rows as `\"uncategorized\"` (if the value is a string) or `np.nan`.\n",
    "Still, having this cell may be helpful in this tutorial if you are going back and forth between cells to reset the assignment.\n",
    "\n",
    "The function `categorize()` takes `color` and similar arguments, which can then be used by the plotting library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.set_meta(meta='uncategorized', name='Temperature')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.categorize(\n",
    "    'Temperature', 'Below 1.6C',\n",
    "    criteria={v: {'up': 1.6, 'year': 2100}},\n",
    "    color='cornflowerblue'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.categorize(\n",
    "    'Temperature', 'Below 2.0C',\n",
    "    criteria={'Temperature|Global Mean|MAGICC6|MED': {'up': 2.0, 'lo': 1.6, 'year': 2100}},\n",
    "    color='forestgreen'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.categorize(\n",
    "    'Temperature', 'Below 2.5C',\n",
    "    criteria={v: {'up': 2.5, 'lo': 2.0, 'year': 2100}},\n",
    "    color='gold'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.categorize(\n",
    "    'Temperature', 'Below 3.5C',\n",
    "     criteria={v: {'up': 3.5, 'lo': 2.5, 'year': 2100}},\n",
    "     color='firebrick'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.categorize(\n",
    "    'Temperature', 'Above 3.5C',\n",
    "    criteria={v: {'lo': 3.5, 'year': 2100}},\n",
    "    color='magenta'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two models included in the snapshot have not been assessed by MAGICC6 regarding their long-term climate and warming impact. Therefore, the timeseries ``'Temperature|Global Mean|MAGICC6|MED'`` does not exist, and they have not been categorized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.require_variable(variable=v, exclude_on_fail=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we again display the median global temperature increase for all scenarios, but we use the colouring by category to illustrate the common charateristics across scenarios."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.filter(variable='Temperature*').line_plot(color='Temperature', legend=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a last step, we display the aggregate CO2 emissions by category. This allows to highlight alternative pathways within the same category. \n",
    "\n",
    "In this step, we also export this figure as a png using the option ``savefig``. The figure will be saved in the tutorials folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "(df\n",
    " .filter(variable='Emissions|CO2', region='World')\n",
    " .line_plot(ax=ax, color='Temperature', legend=True)\n",
    ")\n",
    "fig.savefig('co2_emissions.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exporting timeseries data for further analysis\n",
    "\n",
    "The IamDataFrame can be directly exported to `xlsx` and `csv` in the standard IAMC format.  \n",
    "This feature is based on [pd.DataFrame.to_excel()](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_excel.html) and can use any keyword arguments of that function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_excel('tutorial_export.xlsx')\n",
    "df.export_metadata('tutorial_metadata.xlsx')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}