notebook/getting-started.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started\n",
    "## Prerequisites\n",
    "An understanding of the following packages is required to better understand the workings of SageRender.\n",
    "* [Phiera](https://github.com/nike-inc/phiera)\n",
    "* [SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/)\n",
    "\n",
    "## Step 1: Phiera Configuration\n",
    "Create a `hiera.yaml` with the following sections:\n",
    "* backends\n",
    "  Define the supported backends for the existing hierarchical configuration. Phiera supports, yaml, yaml.enc (YAML files encrypted by SOPS), and json backends.\n",
    "* context\n",
    "  Define context variables that used to define the hierarchy at runtime. These can be looked up in the configuration files using `%{variable_name}` syntax where variable_name is the context variable.\n",
    "* hierarchy\n",
    "  Define the hierarchy to adhere to in the order of precedence. *The values defined in the current path overrides the values of all the paths defined below it.*\n",
    "* datadir definition for each backend\n",
    "  Define the source of the files for each backend. This is defined using the following structure:\n",
    "  ```yaml\n",
    "  yaml: # Name of the backend\n",
    "    datadir: path/to/config/folder  # path to the configuration folder from pwd of execution environment\n",
    "  ```\n",
    "\n",
    "![Simple Hierarchy](../extras/simple-hierarchy.png)\n",
    "Let's consider a simple example using the above hierarchy defined in [hiera.yaml](../examples/hiera.yaml).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Configure a simple single step pipeline\n",
    "\n",
    "The pipeline is configured in [common.yaml](../examples/config/common.yaml) between lines 15-30. The environment specific configuration is then configured in the environment-specific YAML files.\n",
    "\n",
    "In addition to the context variables defined in hiera.yaml, we can pass in additional context variables which can then be referenced in the YAML file using `%{variable_name}` syntax. In the example, `team` is configured as an additional context variable and this is passed in as part of the CLI arguments.\n",
    "\n",
    "Each step of the pipeline is configured in two parts:\n",
    "* Processor<br/>\n",
    "Defines the execution environment used for the step. Follows the method signature defined in SageMaker Python SDK.\n",
    "\n",
    "* Step<br/>\n",
    "Defines the I/O and script used for executing the step. Follows the method signature defined in SageMaker Python SDK."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Upsert Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Change pwd to examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.chdir(\"../examples\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Required only if hiera.yaml is defined elsewhere\n",
    "If your hiera.yaml is not defined in the same location as the execution enviroment, then set `SAGERENDER_HIERA_FILE`\n",
    "to point to the path of hiera.yaml file.\n",
    "\n",
    "To set env var,\n",
    "`%env SAGERENDER_HIERA_FILE path/to/hiera.yaml`\n",
    "\n",
    "### Run Upsert Pipeline\n",
    "In this example, `team` is the additional context variable used. Please set it to your specific team name when running this example notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!sagerender upsert-pipeline --env test --pipeline-name local-pipeline --team <team-name>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dry Run Mode\n",
    "Use dry-run to determine the generated config is inline with expected pipeline definition and that there are no syntax errors. When `--dry-run` flag is set, pipeline definition is not upserted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!sagerender upsert-pipeline --env test --pipeline-name local-pipeline --team <team-name> --dry-run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Local Mode\n",
    "There is support for local mode for processors that can be run in local. Simply set `--local` flag to run in local mode. This does not upsert a pipeline to SageMaker and uses local execution environment to run the pipeline. Information about SageMaker pipelines local mode can be found [here](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-local-mode.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!sagerender upsert-pipeline --env test --pipeline-name local-pipeline --team <team-name> --local"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Run Pipeline\n",
    "There are a few different ways to run a pipeline on SageMaker.\n",
    "* Studio UI<br/>\n",
    "Search the pipeline in the Studio UI by going to the pipeline explorer on the left. Select the pipeline -> Click on Create Execution -> Run the execution.\n",
    "* AWS CLI<br/>\n",
    "Using the following command from local or Studio notebook, `aws sagemaker start-pipeline-execution --pipeline-name <pipeline-name> --region <region-name>`\n",
    "* SageRender CLI<br/>\n",
    "Using the following command from local or Studio notebook, `sagerender run-pipeline --pipeline-name <pipeline-name>`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}