examples/rossmann/rossmann-store-sales-pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-09T23:34:00.158056Z",
     "iopub.status.busy": "2020-12-09T23:34:00.157495Z",
     "iopub.status.idle": "2020-12-09T23:34:00.159833Z",
     "shell.execute_reply": "2020-12-09T23:34:00.159365Z"
    }
   },
   "outputs": [],
   "source": [
    "# Copyright 2020 NVIDIA Corporation. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "# =============================================================================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://developer.download.nvidia.com/compute/machine-learning/frameworks/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# NVTabular demo on Rossmann data - PyTorch\n",
    "\n",
    "## Overview\n",
    "\n",
    "NVTabular is a feature engineering and preprocessing library for tabular data designed to quickly and easily manipulate terabyte scale datasets used to train deep learning based recommender systems.  It provides a high level abstraction to simplify code and accelerates computation on the GPU using the RAPIDS cuDF library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning objectives\n",
    "\n",
    "In the previous notebooks ([rossmann-store-sales-preproc.ipynb](https://github.com/NVIDIA/NVTabular/blob/main/examples/rossmann/rossmann-store-sales-preproc.ipynb) and [rossmann-store-sales-feature-engineering.ipynb](https://github.com/NVIDIA/NVTabular/blob/main/examples/rossmann/rossmann-store-sales-feature-engineering.ipynb)), we downloaded, preprocessed and created features for the dataset. Now, we are ready to train our deep learning model on the dataset. In this notebook, we use **PyTorch** with the NVTabular data loader for PyTorch to accelereate the training pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-09T23:34:00.163145Z",
     "iopub.status.busy": "2020-12-09T23:34:00.162689Z",
     "iopub.status.idle": "2020-12-09T23:34:02.057533Z",
     "shell.execute_reply": "2020-12-09T23:34:02.057909Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import math\n",
    "import json\n",
    "import nvtabular as nvt\n",
    "import glob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading NVTabular workflow\n",
    "This time, we only need to define our data directories. We can load the data schema from the NVTabular workflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-09T23:34:02.062672Z",
     "iopub.status.busy": "2020-12-09T23:34:02.062080Z",
     "iopub.status.idle": "2020-12-09T23:34:02.064319Z",
     "shell.execute_reply": "2020-12-09T23:34:02.063911Z"
    }
   },
   "outputs": [],
   "source": [
    "DATA_DIR = os.environ.get(\"OUTPUT_DATA_DIR\", \"./data\")\n",
    "INPUT_DATA_DIR = os.environ.get(\"INPUT_DATA_DIR\", \"./data\")\n",
    "PREPROCESS_DIR = os.path.join(INPUT_DATA_DIR, 'ross_pre/')\n",
    "PREPROCESS_DIR_TRAIN = os.path.join(PREPROCESS_DIR, 'train')\n",
    "PREPROCESS_DIR_VALID = os.path.join(PREPROCESS_DIR, 'valid')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What files are available to train on in our directories?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-09T23:34:02.068105Z",
     "iopub.status.busy": "2020-12-09T23:34:02.067594Z",
     "iopub.status.idle": "2020-12-09T23:34:02.619485Z",
     "shell.execute_reply": "2020-12-09T23:34:02.619836Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stats.json  train  valid\r\n"
     ]
    }
   ],
   "source": [
    "!ls $PREPROCESS_DIR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-09T23:34:02.627413Z",
     "iopub.status.busy": "2020-12-09T23:34:02.623677Z",
     "iopub.status.idle": "2020-12-09T23:34:03.158154Z",
     "shell.execute_reply": "2020-12-09T23:34:03.158496Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.274616a5186f4ae6b9688cad2a1ac1e5.parquet  _metadata\r\n",
      "_file_list.txt\t\t\t\t    _metadata.json\r\n"
     ]
    }
   ],
   "source": [
    "!ls $PREPROCESS_DIR_TRAIN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-09T23:34:03.164912Z",
     "iopub.status.busy": "2020-12-09T23:34:03.161637Z",
     "iopub.status.idle": "2020-12-09T23:34:03.700836Z",
     "shell.execute_reply": "2020-12-09T23:34:03.701213Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_metadata  part.0.parquet\n"
     ]
    }
   ],
   "source": [
    "!ls $PREPROCESS_DIR_VALID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load the data schema and statistic information from `stats.json`. We created the file in the previous notebook `rossmann-store-sales-feature-engineering`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-09T23:34:03.724731Z",
     "iopub.status.busy": "2020-12-09T23:34:03.715161Z",
     "iopub.status.idle": "2020-12-09T23:34:03.727038Z",
     "shell.execute_reply": "2020-12-09T23:34:03.726599Z"
    }
   },
   "outputs": [],
   "source": [
    "stats = json.load(open(PREPROCESS_DIR + \"/stats.json\", \"r\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-09T23:34:03.730696Z",
     "iopub.status.busy": "2020-12-09T23:34:03.730253Z",
     "iopub.status.idle": "2020-12-09T23:34:03.732468Z",
     "shell.execute_reply": "2020-12-09T23:34:03.732102Z"
    }
   },
   "outputs": [],
   "source": [
    "CATEGORICAL_COLUMNS = stats['CATEGORICAL_COLUMNS']\n",
    "CONTINUOUS_COLUMNS = stats['CONTINUOUS_COLUMNS']\n",
    "LABEL_COLUMNS = stats['LABEL_COLUMNS']\n",
    "COLUMNS = CATEGORICAL_COLUMNS + CONTINUOUS_COLUMNS + LABEL_COLUMNS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The embedding table shows the cardinality of each categorical variable along with its associated embedding size. Each entry is of the form `(cardinality, embedding_size)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-09T23:34:03.736225Z",
     "iopub.status.busy": "2020-12-09T23:34:03.735788Z",
     "iopub.status.idle": "2020-12-09T23:34:03.757466Z",
     "shell.execute_reply": "2020-12-09T23:34:03.757088Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Assortment': [4, 16],\n",
       " 'CompetitionMonthsOpen': [26, 16],\n",
       " 'CompetitionOpenSinceYear': [24, 16],\n",
       " 'Day': [32, 16],\n",
       " 'DayOfWeek': [8, 16],\n",
       " 'Events': [22, 16],\n",
       " 'Month': [13, 16],\n",
       " 'Promo2SinceYear': [9, 16],\n",
       " 'Promo2Weeks': [27, 16],\n",
       " 'PromoInterval': [4, 16],\n",
       " 'Promo_bw': [7, 16],\n",
       " 'Promo_fw': [7, 16],\n",
       " 'SchoolHoliday_bw': [9, 16],\n",
       " 'SchoolHoliday_fw': [9, 16],\n",
       " 'State': [13, 16],\n",
       " 'StateHoliday': [3, 16],\n",
       " 'StateHoliday_bw': [4, 16],\n",
       " 'StateHoliday_fw': [4, 16],\n",
       " 'Store': [1116, 81],\n",
       " 'StoreType': [5, 16],\n",
       " 'Week': [53, 16],\n",
       " 'Year': [4, 16]}"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "EMBEDDING_TABLE_SHAPES = stats['EMBEDDING_TABLE_SHAPES']\n",
    "EMBEDDING_TABLE_SHAPES"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training a Network\n",
    "\n",
    "Now that our data is preprocessed and saved out, we can leverage `dataset`s to read through the preprocessed parquet files in an online fashion to train neural networks.\n",
    "\n",
    "We'll start by setting some universal hyperparameters for our model and optimizer. These settings will be the same across all of the frameworks that we explore in the different notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "EMBEDDING_DROPOUT_RATE = 0.04\n",
    "DROPOUT_RATES = [0.001, 0.01]\n",
    "HIDDEN_DIMS = [1000, 500]\n",
    "BATCH_SIZE = 65536\n",
    "LEARNING_RATE = 0.001\n",
    "EPOCHS = 25\n",
    "\n",
    "# TODO: Calculate on the fly rather than recalling from previous analysis.\n",
    "MAX_SALES_IN_TRAINING_SET = 38722.0\n",
    "MAX_LOG_SALES_PREDICTION = 1.2 * math.log(MAX_SALES_IN_TRAINING_SET + 1.0)\n",
    "\n",
    "TRAIN_PATHS = sorted(glob.glob(os.path.join(PREPROCESS_DIR_TRAIN, '*.parquet')))\n",
    "VALID_PATHS = sorted(glob.glob(os.path.join(PREPROCESS_DIR_VALID, '*.parquet')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PyTorch<a id=\"PyTorch\"></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PyTorch: Preparing Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from nvtabular.loader.torch import TorchAsyncItr, DLDataLoader\n",
    "from nvtabular.framework_utils.torch.models import Model\n",
    "from nvtabular.framework_utils.torch.utils import process_epoch\n",
    "\n",
    "# TensorItrDataset returns a single batch of x_cat, x_cont, y.\n",
    "collate_fn = lambda x: x\n",
    "\n",
    "train_dataset = TorchAsyncItr(nvt.Dataset(TRAIN_PATHS), batch_size=BATCH_SIZE, cats=CATEGORICAL_COLUMNS, conts=CONTINUOUS_COLUMNS, labels=LABEL_COLUMNS)\n",
    "train_loader = DLDataLoader(train_dataset, batch_size=None, collate_fn=collate_fn, pin_memory=False, num_workers=0)\n",
    "\n",
    "valid_dataset = TorchAsyncItr(nvt.Dataset(VALID_PATHS), batch_size=BATCH_SIZE, cats=CATEGORICAL_COLUMNS, conts=CONTINUOUS_COLUMNS, labels=LABEL_COLUMNS)\n",
    "valid_loader = DLDataLoader(valid_dataset, batch_size=None, collate_fn=collate_fn, pin_memory=False, num_workers=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PyTorch: Defining a Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Model(\n",
    "    embedding_table_shapes=EMBEDDING_TABLE_SHAPES,\n",
    "    num_continuous=len(CONTINUOUS_COLUMNS),\n",
    "    emb_dropout=EMBEDDING_DROPOUT_RATE,\n",
    "    layer_hidden_dims=HIDDEN_DIMS,\n",
    "    layer_dropout_rates=DROPOUT_RATES,\n",
    "    max_output=MAX_LOG_SALES_PREDICTION\n",
    ").to('cuda')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PyTorch: Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rmspe_func(y_pred, y):\n",
    "    \"Return y_pred and y to non-log space and compute RMSPE\"\n",
    "    y_pred, y = torch.exp(y_pred) - 1, torch.exp(y) - 1\n",
    "    pct_var = (y_pred - y) / y\n",
    "    return (pct_var**2).mean().pow(0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 00. Train loss: 11.4002. Train RMSPE: inf. Valid loss: 4.4211. Valid RMSPE: 0.8606.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 01. Train loss: 4.7419. Train RMSPE: 0.8401. Valid loss: 3.0792. Valid RMSPE: 0.8065.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 02. Train loss: 3.2193. Train RMSPE: 0.8018. Valid loss: 2.4229. Valid RMSPE: 0.7645.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 03. Train loss: 1.8061. Train RMSPE: 0.6998. Valid loss: 1.1669. Valid RMSPE: 0.6192.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 04. Train loss: 0.6868. Train RMSPE: 0.5317. Valid loss: 0.3846. Valid RMSPE: 0.4372.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 05. Train loss: 0.2363. Train RMSPE: inf. Valid loss: 0.1892. Valid RMSPE: 0.4733.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 06. Train loss: 0.1798. Train RMSPE: inf. Valid loss: 0.1634. Valid RMSPE: 0.4897.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 07. Train loss: 0.1573. Train RMSPE: inf. Valid loss: 0.1355. Valid RMSPE: 0.4112.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 08. Train loss: 0.1357. Train RMSPE: inf. Valid loss: 0.1203. Valid RMSPE: 0.3619.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 09. Train loss: 0.1238. Train RMSPE: 0.4115. Valid loss: 0.1102. Valid RMSPE: 0.3440.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 10. Train loss: 0.1157. Train RMSPE: 0.4002. Valid loss: 0.0973. Valid RMSPE: 0.3385.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 11. Train loss: 0.1098. Train RMSPE: 0.3974. Valid loss: 0.0931. Valid RMSPE: 0.3348.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 12. Train loss: 0.1035. Train RMSPE: 0.3957. Valid loss: 0.0870. Valid RMSPE: 0.3189.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 13. Train loss: 0.0978. Train RMSPE: 0.3750. Valid loss: 0.0830. Valid RMSPE: 0.3020.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 14. Train loss: 0.0928. Train RMSPE: 0.3546. Valid loss: 0.0806. Valid RMSPE: 0.2970.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 15. Train loss: 0.0894. Train RMSPE: 0.3564. Valid loss: 0.0744. Valid RMSPE: 0.2939.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 16. Train loss: 0.0856. Train RMSPE: 0.3462. Valid loss: 0.0713. Valid RMSPE: 0.2839.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 17. Train loss: 0.0830. Train RMSPE: 0.3418. Valid loss: 0.0714. Valid RMSPE: 0.2800.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 18. Train loss: 0.0806. Train RMSPE: inf. Valid loss: 0.0656. Valid RMSPE: 0.2833.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 19. Train loss: 0.0781. Train RMSPE: 0.3273. Valid loss: 0.0750. Valid RMSPE: 0.2765.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 20. Train loss: 0.0784. Train RMSPE: 0.3318. Valid loss: 0.0600. Valid RMSPE: 0.2655.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 21. Train loss: 0.0783. Train RMSPE: 0.3350. Valid loss: 0.0587. Valid RMSPE: 0.2637.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 22. Train loss: 0.0735. Train RMSPE: inf. Valid loss: 0.0604. Valid RMSPE: 0.2583.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 23. Train loss: 0.0725. Train RMSPE: 0.3154. Valid loss: 0.0560. Valid RMSPE: 0.2567.\n",
      "Total batches: 12\n",
      "Total batches: 0\n",
      "Epoch 24. Train loss: 0.0721. Train RMSPE: 0.3209. Valid loss: 0.0553. Valid RMSPE: 0.2619.\n",
      "CPU times: user 41.7 s, sys: 1.38 s, total: 43.1 s\n",
      "Wall time: 43.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from time import time\n",
    "\n",
    "for epoch in range(EPOCHS):\n",
    "    start = time()\n",
    "    train_loss, y_pred, y = process_epoch(train_loader, model, train=True, optimizer=optimizer)\n",
    "    train_rmspe = rmspe_func(y_pred, y)\n",
    "    valid_loss, y_pred, y = process_epoch(valid_loader, model, train=False)\n",
    "    valid_rmspe = rmspe_func(y_pred, y)\n",
    "    print(f'Epoch {epoch:02d}. Train loss: {train_loss:.4f}. Train RMSPE: {train_rmspe:.4f}. Valid loss: {valid_loss:.4f}. Valid RMSPE: {valid_rmspe:.4f}.')\n",
    "    t_final = time() - start\n",
    "    total_rows = train_dataset.num_rows_processed + valid_dataset.num_rows_processed\n",
    "    print(f\"run_time: {t_final} - rows: {total_rows} - epochs: {EPOCHS} - dl_thru: {total_rows / t_final}\")"
   ]
  }
 ],
 "metadata": {
  "file_extension": ".py",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  },
  "mimetype": "text/x-python",
  "name": "python",
  "npconvert_exporter": "python",
  "pygments_lexer": "ipython3",
  "version": 3
 },
 "nbformat": 4,
 "nbformat_minor": 4
}