workshop/1-Personalization/1.1-Personalize.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Retail Demo Store - Personalization Workshop\n",
    "\n",
    "Welcome to the Retail Demo Store Personalization Workshop. In this module we're going to be adding three core personalization features powered by [Amazon Personalize](https://aws.amazon.com/personalize/): related product recommendations on the product detail page, personalized recommendations on the Retail Demo Store homepage, and personalized ranking of items on the featured product page and product search results. This will allow us to give our users targeted recommendations based on their activity.\n",
    "We also illustrate another user case of Amazon Personalize:\n",
    "Selecting a product to discount from a list of products for a user, using the \"contextual metadata\" facility of Amazon Personalize.\n",
    "\n",
    "Recommended Time: 2 Hours"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "To get started, we need to perform a bit of setup.\n",
    "Walk through each of the following steps to configure your environment to\n",
    "interact with the Amazon Personalize Service."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import Dependencies and Setup Boto3 Python Clients\n",
    "\n",
    "Throughout this workshop we will need access to some common libraries and clients for connecting to AWS services.\n",
    "We also have to retrieve Uid from a SageMaker notebook instance tag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Dependencies\n",
    "\n",
    "import boto3\n",
    "import json\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import time\n",
    "import requests\n",
    "import csv\n",
    "import sys\n",
    "import os\n",
    "import botocore\n",
    "import uuid\n",
    "from collections import defaultdict\n",
    "import random\n",
    "import numpy as np\n",
    "\n",
    "from packaging import version\n",
    "from botocore.exceptions import ClientError\n",
    "from pathlib import Path\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "# Setup Clients\n",
    "\n",
    "personalize = boto3.client('personalize')\n",
    "personalize_runtime = boto3.client('personalize-runtime')\n",
    "personalize_events = boto3.client('personalize-events')\n",
    "\n",
    "servicediscovery = boto3.client('servicediscovery')\n",
    "ssm = boto3.client('ssm')\n",
    "\n",
    "\n",
    "# The Uid is a unique ID and we need it to find the role made by CloudFormation\n",
    "with open('/opt/ml/metadata/resource-metadata.json') as f:\n",
    "    data = json.load(f)\n",
    "sagemaker = boto3.client('sagemaker')\n",
    "sagemakerResponce = sagemaker.list_tags(ResourceArn=data[\"ResourceArn\"])\n",
    "for tag in sagemakerResponce[\"Tags\"]:\n",
    "    if tag['Key'] == 'Uid':\n",
    "        Uid = tag['Value']\n",
    "        break\n",
    "\n",
    "print('Uid:', Uid)\n",
    "print('Region:', ssm.meta.region_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Bucket and Data Output Location"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be configuring some variables that will store the location of our source data. When the Retail Demo Store stack was deployed in this account, an S3 bucket was created for you and the name of this bucket was stored in Systems Manager Parameter Store. Using the Boto3 client we can get the name of this bucket for use within our Notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bucketresponse = ssm.get_parameter(\n",
    "    Name='retaildemostore-stack-bucket'\n",
    ")\n",
    "\n",
    "# We will use this bucket to store our training data:\n",
    "bucket = bucketresponse['Parameter']['Value']     # Do Not Change\n",
    "\n",
    "# We will upload our training data in these files:\n",
    "items_filename = \"items.csv\"                # Do Not Change\n",
    "users_filename = \"users.csv\"                # Do Not Change\n",
    "interactions_filename = \"interactions.csv\"  # Do Not Change\n",
    "\n",
    "print('Bucket: {}'.format(bucket))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get, Prepare, and Upload User, Product, and Interaction Data\n",
    "\n",
    "Amazon Personalize provides predefined recipes, based on common use cases, for training models. A recipe is a machine learning algorithm that you use with settings, or hyperparameters, and the data you provide to train an Amazon Personalize model. The data you provide to train a model are organized into separate datasets by the type of data being provided. A collection of datasets are organized into a dataset group. The three dataset types supported by Personalize are items, users, and interactions. Depending on the recipe type you choose, a different combination of dataset types are required. For all recipe types, an interactions dataset is required. Interactions represent how users interact with items. For example, viewing a product, watching a video, listening to a recording, or reading an article. For this workshop, we will be using a recipe that supports all three dataset types.\n",
    "\n",
    "When we deployed the Retail Demo Store, it was deployed with an initial seed of fictitious User and Product data. We will use this data to train three models, or solutions, in the Amazon Personalize service which will be used to serve product recommendations, related items,  and to rerank product lists for our users. The User and Product data can be accessed from the Retail Demo Store's [Users](https://github.com/aws-samples/retail-demo-store/tree/master/src/users) and [Products](https://github.com/aws-samples/retail-demo-store/tree/master/src/products) microservices, respectively. We will access our data through microservice data APIs, process the data, and upload them as CSVs to S3. Once our datasets are in S3, we can import them into the Amazon Personalize service.\n",
    "\n",
    "Let's get started."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Products Service Instance\n",
    "\n",
    "We will be pulling our Product data from the [Products Service](https://github.com/aws-samples/retail-demo-store/tree/master/src/products) that was deployed in Amazon Elastic Container Service as part of the Retail Demo Store. To connect to this service we will use [AWS Cloud Map](https://aws.amazon.com/cloud-map/)'s Service Discovery to discover an instance of the Product Service running in ECS, and then connect directly to that service instances to access our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = servicediscovery.discover_instances(\n",
    "    NamespaceName='retaildemostore.local',\n",
    "    ServiceName='products',\n",
    "    MaxResults=1,\n",
    "    HealthStatus='HEALTHY'\n",
    ")\n",
    "\n",
    "assert len(response['Instances']) > 0, 'Products service instance not found; check ECS to ensure it launched cleanly'\n",
    "\n",
    "products_service_instance = response['Instances'][0]['Attributes']['AWS_INSTANCE_IPV4']\n",
    "print('Products Service Instance IP: {}'.format(products_service_instance))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download and Explore the Products Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get('http://{}/products/all'.format(products_service_instance))\n",
    "products = response.json()\n",
    "products_df = pd.DataFrame(products)\n",
    "pd.set_option('display.max_rows', 5)\n",
    "\n",
    "products_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prepare and Upload Data\n",
    "\n",
    "When training models in Amazon Personalize, we can provide structured and unstructured metadata about our items. This data helps improve the relevance of recommendations and is particularly useful when recommending new/cold items added to your catalog. For this workshop we will add each product's category and style as structured metadata and the product description will be added as textual/unstructured metadata. Personalize will automatically apply a natural language processing (NLP) machine learning model to the textual column to extract features from the text. The product's unique identifier is required. Then we will rename the columns in our dataset to match our schema (defined later) and those expected by Personalize. Finally, we will save our dataset as a CSV and copy it to our S3 bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "products_dataset_df = products_df[['id','category','style','description']]\n",
    "products_dataset_df = products_dataset_df.rename(columns = {'id':'ITEM_ID',\n",
    "                                                            'category':'CATEGORY',\n",
    "                                                            'style':'STYLE',\n",
    "                                                            'description':'DESCRIPTION'}) \n",
    "\n",
    "products_dataset_df.to_csv(items_filename, index=False)\n",
    "boto3.Session().resource('s3').Bucket(bucket).Object(items_filename).upload_file(items_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Users Service Instance\n",
    "\n",
    "We will be pulling our User data from the [Users Service](https://github.com/aws-samples/retail-demo-store/tree/master/src/users) that is deployed as part of the Retail Demo Store. To connect to this service we will use Service Discovery to discover an instance of the User Service, and then connect directly to that service instance to access our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = servicediscovery.discover_instances(\n",
    "    NamespaceName='retaildemostore.local',\n",
    "    ServiceName='users',\n",
    "    MaxResults=1,\n",
    "    HealthStatus='HEALTHY'\n",
    ")\n",
    "\n",
    "assert len(response['Instances']) > 0, 'Users service instance not found; check ECS to ensure it launched cleanly'\n",
    "\n",
    "users_service_instance = response['Instances'][0]['Attributes']['AWS_INSTANCE_IPV4']\n",
    "print('Users Service Instance IP: {}'.format(users_service_instance))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download and Explore the Users Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get('http://{}/users/all?count=10000'.format(users_service_instance))\n",
    "users = response.json()\n",
    "users_df = pd.DataFrame(users)\n",
    "# Remove any users without a persona or gender (i.e. created in web UI)\n",
    "users_df = users_df[(users_df['persona'].str.strip().astype(bool)) | (users_df['gender'].str.strip().astype(bool))]\n",
    "pd.set_option('display.max_rows', 5)\n",
    "\n",
    "users_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prepare and Upload Data\n",
    "\n",
    "Similar to the items dataset we created above, we can provide metadata on our users when training models in Personalize. For this workshop we will include each user's age and gender. As before, we will name the columns to match our schema, save the data as a CSV, and upload to our S3 bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "users_dataset_df = users_df[['id','age','gender']]\n",
    "users_dataset_df = users_dataset_df.rename(columns = {'id':'USER_ID','age':'AGE','gender':'GENDER'}) \n",
    "\n",
    "users_dataset_df.to_csv(users_filename, index=False)\n",
    "boto3.Session().resource('s3').Bucket(bucket).Object(users_filename).upload_file(users_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create User-Items Interactions Dataset\n",
    "\n",
    "To mimic user behavior, we will be generating a new dataset that represents user interactions with items.\n",
    "To make the interactions more realistic, we will use the pre-defined shopper persona for each user to generate event\n",
    "types for products matching that persona. We will create events for viewing products, adding products to a cart,\n",
    "checking out, and completing orders. \n",
    "The effect of discounts on interactions depends on a separate \"discount persona\" stored against the user.\n",
    "The script also makes an effort to keep interactions balanced between and within categories and products.\n",
    "\n",
    "For more information about how this script works, navigate to the script file at [generate_interactions_personalize.py](../../generators/generate_interactions_personalize.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sys.path.insert(0, os.path.abspath('../../generators'))\n",
    "\n",
    "import generate_interactions_personalize as gi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Where to put the generated data\n",
    "gi.GENERATED_DATA_ROOT = \"./\"\n",
    "\n",
    "# Let us keep things deterministic so we can replicate the data.\n",
    "gi.RANDOM_SEED = 0\n",
    "\n",
    "# Interactions will be generated between these dates \n",
    "# (we keep things deterministic by keeping fixed dates)\n",
    "gi.FIRST_TIMESTAMP = 1591803782  # 2020-06-10, 18:43:02\n",
    "gi.LAST_TIMESTAMP = 1599579782  # 2020-09-08, 18:43:02\n",
    "\n",
    "# Minimum number of interactions to generate\n",
    "gi.min_interactions = 675000\n",
    "\n",
    "# Users are set up with 3 product categories on their personas, \n",
    "# such as beauty_electronics_outdoors\n",
    "# [. If [0.6, 0.25, 0.15] it means 60% of the time they'll want to\n",
    "# choose a product from the first category (beauty in this example),\n",
    "# 25% from 2nd, etc.\n",
    "gi.CATEGORY_AFFINITY_PROBS = [0.6, 0.25, 0.15]\n",
    "\n",
    "# With this probability a product interaction will be with the product discounted\n",
    "# Here we go the other way - what is the probability that a product that a user is already interacting\n",
    "# with is discounted - (depending on whether user likes discounts or not,\n",
    "# a feature available on their simulated profile as discount_persona)\n",
    "gi.DISCOUNT_PROBABILITY = 0.2\n",
    "gi.DISCOUNT_PROBABILITY_WITH_PREFERENCE = 0.5\n",
    "\n",
    "# After interacting with a product, there are this many products within \n",
    "# the category that a user is likely to jump on next.\n",
    "# The purpose of this constant is to keep recommendations focused\n",
    "# if there are too many products in a category. \n",
    "gi.PRODUCT_AFFINITY_N = 4\n",
    "\n",
    "# from 0 to 1. If 0 then products in busy categories get represented less. If 1 then all products same amount.\n",
    "gi.NORMALISE_PER_PRODUCT_WEIGHT = 1.0\n",
    "\n",
    "# Show progress every 30 seconds. The script takes some time to complete.\n",
    "gi.PROGRESS_MONITOR_SECONDS_UPDATE = 30\n",
    "\n",
    "# Percentages of each event type to generate\n",
    "gi.product_added_percent = .08\n",
    "gi.cart_viewed_percent = .05\n",
    "gi.checkout_started_percent = .02\n",
    "gi.order_completed_percent = .01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "gi.generate_interactions(interactions_filename, users_df, products_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Open and Explore the Simulated Interactions Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let us see a few lines of the raw CSV data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!head -n 5 $interactions_filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us load it as a Pandas dataframe. Note:\n",
    "\n",
    "- An EVENT_TYPE column which can be used to train different Personalize campaigns and also to filter on recommendations.\n",
    "- The custom DISCOUNT column which is a contextual metadata field, that Personalize reranking and user recommendation campaigns can take into account to guess on the best next product."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_df = pd.read_csv(interactions_filename)\n",
    "interactions_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chart the counts of each `EVENT_TYPE` generated for the interactions dataset. We're simulating a site where visitors heavily view/browse products and to a lesser degree add products to their cart and checkout."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_attributes = interactions_df.select_dtypes(include = ['object'])\n",
    "\n",
    "plt.figure(figsize=(16,3))\n",
    "chart = sns.countplot(data = categorical_attributes, x = 'EVENT_TYPE')\n",
    "plt.xticks(rotation=90, horizontalalignment='right')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us plot product persona vs product category distribution - so that you can see what categories get assigned to which types of persona. Note that because of attempts to reach balance between categories and products in the generation script, the proportions do not exactly match those configured in "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merged_df = interactions_df[['USER_ID', 'ITEM_ID']].astype({'USER_ID':str, 'ITEM_ID':str})\n",
    "merged_df = merged_df.merge(users_df[['id', 'persona']], left_on='USER_ID', right_on='id').drop(columns=['id', 'USER_ID'])\n",
    "merged_df = merged_df.merge(products_df[['id', 'category']], left_on='ITEM_ID', right_on='id').drop(columns=['id', 'ITEM_ID'])\n",
    "merged_df\n",
    "plot_df = merged_df.groupby(['persona', 'category'])['category'].count().unstack()\n",
    "sns.heatmap(plot_df, annot=True, fmt=\"g\", cmap='viridis')\n",
    "plt.title('Heatmap of user persona vs product category')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discount persona vs event type distribution\n",
    "\n",
    "Let us see how the event distribution came out. We should see a different takeup of discounts between users with different discount personas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merged_df = interactions_df.loc[interactions_df.EVENT_TYPE == 'ProductAdded'][['USER_ID', 'DISCOUNT']]\n",
    "merged_df = merged_df[['USER_ID', 'DISCOUNT']].astype({'USER_ID':str}).merge(users_df, left_on='USER_ID', right_on='id')\n",
    "merged_df \n",
    "plot_df = merged_df.groupby(['discount_persona', 'DISCOUNT'])[['id']].count().unstack()\n",
    "plot_df = plot_df.droplevel(axis='columns', level=0)\n",
    "plot_df.plot.bar()\n",
    "plt.title('Event types according to discount persona')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Balance over products I\n",
    "\n",
    "Let us have a careful look at product and category distributions.\n",
    "The interactions generation script ensures that there are small groups of products users tend to interact with, to maintain\n",
    "strong training signals. If you look at the script you will see that although we choose\n",
    "products randomly within a category, they are interacted with in small random groups in the category.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "merged_df = interactions_df[['ITEM_ID', 'USER_ID']].astype({'ITEM_ID': str}).merge(products_df, left_on='ITEM_ID', right_on='id')\n",
    "plot_df = merged_df.groupby(['USER_ID', 'category']).id.apply(set)\n",
    "plot_df.apply(len).value_counts().sort_index().plot.bar()\n",
    "plt.xlabel('Number of different products examined by user.')\n",
    "plt.title(f'We should have reduced users to a small number of products\\n'\n",
    "          f'maximum size should be {gi.PRODUCT_AFFINITY_N+1}')\n",
    "# The peak at 1 is the male jewelry - there is only one product\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Balance over products II\n",
    "Ensuring that users interact with only overlapping subsets of products is not sufficient. We also ensure there is some balance in interactions between and within\n",
    "categories explicitly by using balance information when sampling category and to add\n",
    "interactions depending on the product simulated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "interactions_df.groupby('ITEM_ID')['USER_ID'].count().plot.bar(figsize=[20,4])\n",
    "plt.title('Balance over products. \\nIf you selected NORMALISE_PER_PRODUCT_WEIGHT=1\\nthis should be flatter, otherwise catgories with\\n'\n",
    "          'higher products should be having less per product')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "merged_df = interactions_df[['ITEM_ID', 'USER_ID']].astype({'ITEM_ID': str}).merge(products_df, left_on='ITEM_ID', right_on='id')[['ITEM_ID', 'category', 'USER_ID']]\n",
    "plot_df = merged_df.groupby(['ITEM_ID', 'category'])['USER_ID'].count()\n",
    "plot_df.unstack().plot.barh(figsize=[8, 16])\n",
    "plt.title('Balance over products with category coded.')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prepare and Upload Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us send our generated interactions data to S3 to be picked up by Amazon Personalize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "boto3.Session().resource('s3').Bucket(bucket).Object(interactions_filename).upload_file(interactions_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure Amazon Personalize\n",
    "\n",
    "Now that we've prepared our three datasets and uploaded them to S3 we'll need to configure the Amazon Personalize service to understand our data so that it can be used to train models for generating recommendations.\n",
    "\n",
    "If Personalize was enabled when you deployed the demo, the below logic was run by polling AWS Lambda function whose code is in the file `src/aws-lambda/personalize-pre-create-campaigns/personalize-pre-create-campaigns.py`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Schemas for Datasets\n",
    "\n",
    "Amazon Personalize requires a schema for each dataset so it can map the columns in our CSVs to fields for model training. Each schema is declared in JSON using the [Apache Avro](https://avro.apache.org/) format.\n",
    "\n",
    "Let's define and create schemas in Personalize for our datasets.\n",
    "\n",
    "Note that categorical fields include an additional attribute of `\"categorical\": true` and the textual field has an additional attribute of `\"textual\": true`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Items Datsaset Schema"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "items_schema = {\n",
    "    \"type\": \"record\",\n",
    "    \"name\": \"Items\",\n",
    "    \"namespace\": \"com.amazonaws.personalize.schema\",\n",
    "    \"fields\": [\n",
    "        {\n",
    "            \"name\": \"ITEM_ID\",\n",
    "            \"type\": \"string\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"CATEGORY\",\n",
    "            \"type\": \"string\",\n",
    "            \"categorical\": True,\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"STYLE\",\n",
    "            \"type\": \"string\",\n",
    "            \"categorical\": True,\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"DESCRIPTION\",\n",
    "            \"type\": \"string\",\n",
    "            \"textual\": True\n",
    "        }\n",
    "    ],\n",
    "    \"version\": \"1.0\"\n",
    "}\n",
    "\n",
    "try:\n",
    "    create_schema_response = personalize.create_schema(\n",
    "        name = \"retaildemostore-schema-items\",\n",
    "        schema = json.dumps(items_schema)\n",
    "    )\n",
    "    items_schema_arn = create_schema_response['schemaArn']\n",
    "    print(json.dumps(create_schema_response, indent=2))\n",
    "except personalize.exceptions.ResourceAlreadyExistsException:\n",
    "    print('You aready created this schema, seemingly')\n",
    "    schemas = personalize.list_schemas(maxResults=100)['schemas']\n",
    "    for schema_response in schemas:\n",
    "        if schema_response['name'] == \"retaildemostore-schema-items\":\n",
    "            items_schema_arn = schema_response['schemaArn']\n",
    "            print(f\"Using existing schema: {items_schema_arn}\")\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Users Dataset Schema"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "users_schema = {\n",
    "    \"type\": \"record\",\n",
    "    \"name\": \"Users\",\n",
    "    \"namespace\": \"com.amazonaws.personalize.schema\",\n",
    "    \"fields\": [\n",
    "        {\n",
    "            \"name\": \"USER_ID\",\n",
    "            \"type\": \"string\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"AGE\",\n",
    "            \"type\": \"int\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"GENDER\",\n",
    "            \"type\": \"string\",\n",
    "            \"categorical\": True,\n",
    "        }\n",
    "    ],\n",
    "    \"version\": \"1.0\"\n",
    "}\n",
    "\n",
    "try:\n",
    "    create_schema_response = personalize.create_schema(\n",
    "        name = \"retaildemostore-schema-users\",\n",
    "        schema = json.dumps(users_schema)\n",
    "    )\n",
    "    print(json.dumps(create_schema_response, indent=2))\n",
    "    users_schema_arn = create_schema_response['schemaArn']\n",
    "except personalize.exceptions.ResourceAlreadyExistsException:\n",
    "    print('You aready created this schema, seemingly')\n",
    "    schemas = personalize.list_schemas(maxResults=100)['schemas']\n",
    "    for schema_response in schemas:\n",
    "        if schema_response['name'] == \"retaildemostore-schema-users\":\n",
    "            users_schema_arn = schema_response['schemaArn']\n",
    "            print(f\"Using existing schema: {users_schema_arn}\")\n",
    "    \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Interactions Dataset Schema"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_schema = {\n",
    "    \"type\": \"record\",\n",
    "    \"name\": \"Interactions\",\n",
    "    \"namespace\": \"com.amazonaws.personalize.schema\",\n",
    "    \"fields\": [\n",
    "        {\n",
    "            \"name\": \"ITEM_ID\",\n",
    "            \"type\": \"string\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"USER_ID\",\n",
    "            \"type\": \"string\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"EVENT_TYPE\",  # \"ProductViewed\", \"OrderCompleted\", etc.\n",
    "            \"type\": \"string\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"TIMESTAMP\",\n",
    "            \"type\": \"long\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"DISCOUNT\",  # This is the contextual metadata - \"Yes\" or null.\n",
    "            \"type\": \"string\"\n",
    "        },\n",
    "    ],\n",
    "    \"version\": \"1.0\"\n",
    "}\n",
    "\n",
    "try:\n",
    "    create_schema_response = personalize.create_schema(\n",
    "        name = \"retaildemostore-schema-interactions\",\n",
    "        schema = json.dumps(interactions_schema)\n",
    "    )\n",
    "    print(json.dumps(create_schema_response, indent=2))\n",
    "    interactions_schema_arn = create_schema_response['schemaArn']\n",
    "except personalize.exceptions.ResourceAlreadyExistsException:\n",
    "    print('You aready created this schema, seemingly')\n",
    "    schemas = personalize.list_schemas(maxResults=100)['schemas']\n",
    "    for schema_response in schemas:\n",
    "        if schema_response['name'] == \"retaildemostore-schema-interactions\":\n",
    "            interactions_schema_arn = schema_response['schemaArn']\n",
    "            print(f\"Using existing schema: {interactions_schema_arn}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create and Wait for Dataset Group\n",
    "\n",
    "Next we need to create the dataset group that will contain our three datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Dataset Group"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_dataset_group_response = personalize.create_dataset_group(\n",
    "    name = 'retaildemostore'\n",
    ")\n",
    "dataset_group_arn = create_dataset_group_response['datasetGroupArn']\n",
    "print(json.dumps(create_dataset_group_response, indent=2))\n",
    "\n",
    "print(f'DatasetGroupArn = {dataset_group_arn}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Wait for Dataset Group to Have ACTIVE Status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "status = None\n",
    "max_time = time.time() + 3*60*60 # 3 hours\n",
    "while time.time() < max_time:\n",
    "    describe_dataset_group_response = personalize.describe_dataset_group(\n",
    "        datasetGroupArn = dataset_group_arn\n",
    "    )\n",
    "    status = describe_dataset_group_response[\"datasetGroup\"][\"status\"]\n",
    "    print(\"DatasetGroup: {}\".format(status))\n",
    "    \n",
    "    if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n",
    "        break\n",
    "        \n",
    "    time.sleep(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Items Dataset\n",
    "\n",
    "Next we will create the datasets in Personalize for our three dataset types. Let's start with the items dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_type = \"ITEMS\"\n",
    "create_dataset_response = personalize.create_dataset(\n",
    "    name = \"retaildemostore-dataset-items\",\n",
    "    datasetType = dataset_type,\n",
    "    datasetGroupArn = dataset_group_arn,\n",
    "    schemaArn = items_schema_arn\n",
    ")\n",
    "\n",
    "items_dataset_arn = create_dataset_response['datasetArn']\n",
    "print(json.dumps(create_dataset_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Users Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_type = \"USERS\"\n",
    "create_dataset_response = personalize.create_dataset(\n",
    "    name = \"retaildemostore-dataset-users\",\n",
    "    datasetType = dataset_type,\n",
    "    datasetGroupArn = dataset_group_arn,\n",
    "    schemaArn = users_schema_arn\n",
    ")\n",
    "\n",
    "users_dataset_arn = create_dataset_response['datasetArn']\n",
    "print(json.dumps(create_dataset_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Interactions Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_type = \"INTERACTIONS\"\n",
    "create_dataset_response = personalize.create_dataset(\n",
    "    name = \"retaildemostore-dataset-interactions\",\n",
    "    datasetType = dataset_type,\n",
    "    datasetGroupArn = dataset_group_arn,\n",
    "    schemaArn = interactions_schema_arn\n",
    ")\n",
    "\n",
    "interactions_dataset_arn = create_dataset_response['datasetArn']\n",
    "print(json.dumps(create_dataset_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Datasets to Personalize\n",
    "\n",
    "Up to this point we have generated CSVs containing data for our users, items, and interactions and staged them in an S3 bucket. We also created schemas in Personalize that define the columns in our CSVs. Then we created a datset group and three datasets in Personalize that will receive our data. In the following steps we will create import jobs with Personalize that will import the datasets from our S3 bucket into the service.\n",
    "\n",
    "### Setup Permissions\n",
    "\n",
    "By default, the Personalize service does not have permission to acccess the data we uploaded into the S3 bucket in our account. In order to grant access to the  Personalize service to read our CSVs, we need to set a Bucket Policy and create an IAM role that the Amazon Personalize service will assume."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Attach policy to S3 bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3 = boto3.client(\"s3\")\n",
    "\n",
    "policy = {\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Id\": \"PersonalizeS3BucketAccessPolicy\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "            \"Sid\": \"PersonalizeS3BucketAccessPolicy\",\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Principal\": {\n",
    "                \"Service\": \"personalize.amazonaws.com\"\n",
    "            },\n",
    "            \"Action\": [\n",
    "                \"s3:GetObject\",\n",
    "                \"s3:ListBucket\"\n",
    "            ],\n",
    "            \"Resource\": [\n",
    "                \"arn:aws:s3:::{}\".format(bucket),\n",
    "                \"arn:aws:s3:::{}/*\".format(bucket)\n",
    "            ]\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "s3.put_bucket_policy(Bucket=bucket, Policy=json.dumps(policy));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create S3 Read Only Access Role"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iam = boto3.client(\"iam\")\n",
    "\n",
    "role_name = Uid+\"-PersonalizeS3\"\n",
    "assume_role_policy_document = {\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "          \"Effect\": \"Allow\",\n",
    "          \"Principal\": {\n",
    "            \"Service\": \"personalize.amazonaws.com\"\n",
    "          },\n",
    "          \"Action\": \"sts:AssumeRole\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "try:\n",
    "    create_role_response = iam.create_role(\n",
    "        RoleName = role_name,\n",
    "        AssumeRolePolicyDocument = json.dumps(assume_role_policy_document)\n",
    "    );\n",
    "    \n",
    "except iam.exceptions.EntityAlreadyExistsException as e:\n",
    "    print('Warning: role already exists:', e)\n",
    "    create_role_response = iam.get_role(\n",
    "        RoleName = role_name\n",
    "    );\n",
    "\n",
    "role_arn = create_role_response[\"Role\"][\"Arn\"]\n",
    "    \n",
    "print('IAM Role: {}'.format(role_arn))\n",
    "    \n",
    "attach_response = iam.attach_role_policy(\n",
    "    RoleName = role_name,\n",
    "    PolicyArn = \"arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess\"\n",
    ");\n",
    "\n",
    "role_arn = create_role_response[\"Role\"][\"Arn\"]\n",
    "\n",
    "# Pause to allow role to be fully consistent\n",
    "time.sleep(30)\n",
    "print('Done.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Import Jobs\n",
    "\n",
    "With the permissions in place to allow Personalize to access our CSV files, let's create three import jobs to import each file into its respective dataset. Each import job can take several minutes to complete so we'll create all three and then wait for them all to complete."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Items Dataset Import Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "items_create_dataset_import_job_response = personalize.create_dataset_import_job(\n",
    "    jobName = \"retaildemostore-dataset-items-import\",\n",
    "    datasetArn = items_dataset_arn,\n",
    "    dataSource = {\n",
    "        \"dataLocation\": \"s3://{}/{}\".format(bucket, items_filename)\n",
    "    },\n",
    "    roleArn = role_arn\n",
    ")\n",
    "\n",
    "items_dataset_import_job_arn = items_create_dataset_import_job_response['datasetImportJobArn']\n",
    "print(json.dumps(items_create_dataset_import_job_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Users Dataset Import Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "users_create_dataset_import_job_response = personalize.create_dataset_import_job(\n",
    "    jobName = \"retaildemostore-dataset-users-import\",\n",
    "    datasetArn = users_dataset_arn,\n",
    "    dataSource = {\n",
    "        \"dataLocation\": \"s3://{}/{}\".format(bucket, users_filename)\n",
    "    },\n",
    "    roleArn = role_arn\n",
    ")\n",
    "\n",
    "users_dataset_import_job_arn = users_create_dataset_import_job_response['datasetImportJobArn']\n",
    "print(json.dumps(users_create_dataset_import_job_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Interactions Dataset Import Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_create_dataset_import_job_response = personalize.create_dataset_import_job(\n",
    "    jobName = \"retaildemostore-dataset-interactions-import\",\n",
    "    datasetArn = interactions_dataset_arn,\n",
    "    dataSource = {\n",
    "        \"dataLocation\": \"s3://{}/{}\".format(bucket, interactions_filename)\n",
    "    },\n",
    "    roleArn = role_arn\n",
    ")\n",
    "\n",
    "interactions_dataset_import_job_arn = interactions_create_dataset_import_job_response['datasetImportJobArn']\n",
    "print(json.dumps(interactions_create_dataset_import_job_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wait for Import Jobs to Complete\n",
    "\n",
    "It will take 10-15 minutes for the import jobs to complete, while you're waiting you can learn more about Datasets and Schemas here: https://docs.aws.amazon.com/personalize/latest/dg/how-it-works-dataset-schema.html\n",
    "\n",
    "We will wait for all three jobs to finish."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Wait for Items Import Job to Complete"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "import_job_arns = [ items_dataset_import_job_arn, users_dataset_import_job_arn, interactions_dataset_import_job_arn ]\n",
    "\n",
    "max_time = time.time() + 3*60*60 # 3 hours\n",
    "while time.time() < max_time:\n",
    "    for job_arn in reversed(import_job_arns):\n",
    "        import_job_response = personalize.describe_dataset_import_job(\n",
    "            datasetImportJobArn = job_arn\n",
    "        )\n",
    "        status = import_job_response[\"datasetImportJob\"]['status']\n",
    "\n",
    "        if status == \"ACTIVE\":\n",
    "            print(f'Import job {job_arn} successfully completed')\n",
    "            import_job_arns.remove(job_arn)\n",
    "        elif status == \"CREATE FAILED\":\n",
    "            print(f'Import job {job_arn} failed')\n",
    "            if import_job_response.get('failureReason'):\n",
    "                print('   Reason: ' + import_job_response['failureReason'])\n",
    "            import_job_arns.remove(job_arn)\n",
    "\n",
    "    if len(import_job_arns) > 0:\n",
    "        print('At least one dataset import job still in progress')\n",
    "        time.sleep(60)\n",
    "    else:\n",
    "        print(\"All import jobs have ended\")\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Solutions\n",
    "\n",
    "With our three datasets imported into our dataset group, we can now turn to training models. As a reminder, we will be training three models in this workshop to support four different personalization use-cases wth three different models. One model will be used to make related product recommendations on the product detail view/page, another model will be used to make personalized product recommendations to users on the homepage, and the last model will be used to rerank product lists on the category and featured products page. This last model will be repurposed to offer discounts to users. In Amazon Personalize, training a model involves creating a Solution and Solution Version. So when we are finished we will have three solutions and a solution version for each solution. \n",
    "\n",
    "When creating a solution, you provide your dataset group and the recipe for training. Let's declare the recipes that we will need for our solutions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List Recipes\n",
    "\n",
    "First, let's list all available recipes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_recipes_response = personalize.list_recipes()\n",
    "list_recipes_response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see above, there are several recipes to choose from. Let's declare the recipes for each Solution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Declare Personalize Recipe for Related Products\n",
    "\n",
    "On the product detail page we want to display related products so we'll create a campaign using the [SIMS](https://docs.aws.amazon.com/personalize/latest/dg/native-recipe-sims.html) recipe.\n",
    "\n",
    "> The Item-to-item similarities (SIMS) recipe is based on the concept of collaborative filtering. A SIMS model leverages user-item interaction data to recommend items similar to a given item. In the absence of sufficient user behavior data for an item, this recipe recommends popular items."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "related_recipe_arn = \"arn:aws:personalize:::recipe/aws-sims\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Declare Personalize Recipe for Product Recommendations\n",
    "\n",
    "Since we are providing metadata for users and items, we will be using the [User-Personalization](https://docs.aws.amazon.com/personalize/latest/dg/native-recipe-new-item-USER_PERSONALIZATION.html) recipe for our product recommendations solution.\n",
    "\n",
    "> The User-Personalization (aws-user-personalization) recipe is optimized for all personalized recommendation scenarios. It predicts the items that a user will interact with based on Interactions, Items, and Users datasets. When recommending items, it uses automatic item exploration. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "recommend_recipe_arn = \"arn:aws:personalize:::recipe/aws-user-personalization\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Declare Personalize Recipe for Personalized Ranking\n",
    "\n",
    "In use-cases where we have a curated list of products, we can use the [Personalized-Ranking](https://docs.aws.amazon.com/personalize/latest/dg/native-recipe-search.html) recipe to reorder the products for the current user.\n",
    "\n",
    "> The Personalized-Ranking recipe generates personalized rankings. A personalized ranking is a list of recommended items that are re-ranked for a specific user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ranking_recipe_arn = \"arn:aws:personalize:::recipe/aws-personalized-ranking\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Solutions and Solution Versions\n",
    "\n",
    "With our recipes defined, we can now create our solutions and solution versions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Related Products Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_solution_response = personalize.create_solution(\n",
    "    name = \"retaildemostore-related-products\",\n",
    "    datasetGroupArn = dataset_group_arn,\n",
    "    recipeArn = related_recipe_arn\n",
    ")\n",
    "\n",
    "related_solution_arn = create_solution_response['solutionArn']\n",
    "print(json.dumps(create_solution_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Related Products Solution Version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_solution_version_response = personalize.create_solution_version(\n",
    "    solutionArn = related_solution_arn\n",
    ")\n",
    "\n",
    "related_solution_version_arn = create_solution_version_response['solutionVersionArn']\n",
    "print(json.dumps(create_solution_version_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Product Recommendation Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_solution_response = personalize.create_solution(\n",
    "    name = \"retaildemostore-product-personalization\",\n",
    "    datasetGroupArn = dataset_group_arn,\n",
    "    recipeArn = recommend_recipe_arn\n",
    ")\n",
    "\n",
    "recommend_solution_arn = create_solution_response['solutionArn']\n",
    "print(json.dumps(create_solution_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Product Recommendation Solution Version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_solution_version_response = personalize.create_solution_version(\n",
    "    solutionArn = recommend_solution_arn\n",
    ")\n",
    "\n",
    "recommend_solution_version_arn = create_solution_version_response['solutionVersionArn']\n",
    "print(json.dumps(create_solution_version_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Personalized Ranking Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_solution_response = personalize.create_solution(\n",
    "    name = \"retaildemostore-personalized-ranking\",\n",
    "    datasetGroupArn = dataset_group_arn,\n",
    "    recipeArn = ranking_recipe_arn\n",
    ")\n",
    "\n",
    "ranking_solution_arn = create_solution_response['solutionArn']\n",
    "print(json.dumps(create_solution_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Personalized Ranking Solution Version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_solution_version_response = personalize.create_solution_version(\n",
    "    solutionArn = ranking_solution_arn\n",
    ")\n",
    "\n",
    "ranking_solution_version_arn = create_solution_version_response['solutionVersionArn']\n",
    "print(json.dumps(create_solution_version_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wait for Solution Versions to Complete\n",
    "\n",
    "It can take 40-60 minutes for all solution versions to be created. During this process a model is being trained and tested with the data contained within your datasets. The duration of training jobs can increase based on the size of the dataset, training parameters and a selected recipe. We submitted requests for all three solutions and versions at once so they are trained in parallel and then below we will wait for all three to finish.\n",
    "\n",
    "While you are waiting for this process to complete you can learn more about solutions here: https://docs.aws.amazon.com/personalize/latest/dg/training-deploying-solutions.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Wait for Related Products Solution Version to Have ACTIVE Status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "soln_ver_arns = [ related_solution_version_arn, recommend_solution_version_arn, ranking_solution_version_arn ]\n",
    "\n",
    "max_time = time.time() + 3*60*60 # 3 hours\n",
    "while time.time() < max_time:\n",
    "    for soln_ver_arn in reversed(soln_ver_arns):\n",
    "        soln_ver_response = personalize.describe_solution_version(\n",
    "            solutionVersionArn = soln_ver_arn\n",
    "        )\n",
    "        status = soln_ver_response[\"solutionVersion\"][\"status\"]\n",
    "\n",
    "        if status == \"ACTIVE\":\n",
    "            print(f'Solution version {soln_ver_arn} successfully completed')\n",
    "            soln_ver_arns.remove(soln_ver_arn)\n",
    "        elif status == \"CREATE FAILED\":\n",
    "            print(f'Solution version {soln_ver_arn} failed')\n",
    "            if soln_ver_response.get('failureReason'):\n",
    "                print('   Reason: ' + soln_ver_response['failureReason'])\n",
    "            soln_ver_arns.remove(soln_ver_arn)\n",
    "\n",
    "    if len(soln_ver_arns) > 0:\n",
    "        print('At least one solution version is still in progress')\n",
    "        time.sleep(60)\n",
    "    else:\n",
    "        print(\"All solution versions have completed\")\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate Offline Metrics for Solution Versions\n",
    "\n",
    "Amazon Personalize provides [offline metrics](https://docs.aws.amazon.com/personalize/latest/dg/working-with-training-metrics.html#working-with-training-metrics-metrics) that allow you to evaluate the performance of the solution version before you deploy the model in your application. Metrics can also be used to view the effects of modifying a Solution's hyperparameters or to compare the metrics between solutions that use the same training data but created with different recipes.\n",
    "\n",
    "Let's retrieve the metrics for the solution versions we just created."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Related Products Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_solution_metrics_response = personalize.get_solution_metrics(\n",
    "    solutionVersionArn = related_solution_version_arn\n",
    ")\n",
    "\n",
    "print(json.dumps(get_solution_metrics_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Product Recommendations Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_solution_metrics_response = personalize.get_solution_metrics(\n",
    "    solutionVersionArn = recommend_solution_version_arn\n",
    ")\n",
    "\n",
    "print(json.dumps(get_solution_metrics_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Personalized Ranking Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_solution_metrics_response = personalize.get_solution_metrics(\n",
    "    solutionVersionArn = ranking_solution_version_arn\n",
    ")\n",
    "\n",
    "print(json.dumps(get_solution_metrics_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Campaigns\n",
    "\n",
    "Once we're satisfied with our solution versions, we need to create Campaigns for each solution version. When creating a campaign you specify the minimum transactions per second (`minProvisionedTPS`) that you expect to make against the service for this campaign. Personalize will automatically scale the inference endpoint up and down for the campaign to match demand but will never scale below `minProvisionedTPS`.\n",
    "\n",
    "Let's create campaigns for our three solution versions with each set at `minProvisionedTPS` of 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Related Products Campaign"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_campaign_response = personalize.create_campaign(\n",
    "    name = \"retaildemostore-related-products\",\n",
    "    solutionVersionArn = related_solution_version_arn,\n",
    "    minProvisionedTPS = 1\n",
    ")\n",
    "\n",
    "related_campaign_arn = create_campaign_response['campaignArn']\n",
    "print(json.dumps(create_campaign_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Product Recommendation Campaign"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_campaign_response = personalize.create_campaign(\n",
    "    name = \"retaildemostore-product-personalization\",\n",
    "    solutionVersionArn = recommend_solution_version_arn,\n",
    "    minProvisionedTPS = 1\n",
    ")\n",
    "\n",
    "recommend_campaign_arn = create_campaign_response['campaignArn']\n",
    "print(json.dumps(create_campaign_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Personalized Ranking Campaign"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_campaign_response = personalize.create_campaign(\n",
    "    name = \"retaildemostore-personalized-ranking\",\n",
    "    solutionVersionArn = ranking_solution_version_arn,\n",
    "    minProvisionedTPS = 1\n",
    ")\n",
    "\n",
    "ranking_campaign_arn = create_campaign_response['campaignArn']\n",
    "print(json.dumps(create_campaign_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Wait for Related Products Campaign to Have ACTIVE Status\n",
    "\n",
    "It can take 20-30 minutes for the campaigns to be fully created. \n",
    "\n",
    "While you are waiting for this to complete you can learn more about campaigns here: https://docs.aws.amazon.com/personalize/latest/dg/campaigns.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "campaign_arns = [ related_campaign_arn, recommend_campaign_arn, ranking_campaign_arn ]\n",
    "\n",
    "max_time = time.time() + 3*60*60 # 3 hours\n",
    "while time.time() < max_time:\n",
    "    for campaign_arn in reversed(campaign_arns):\n",
    "        campaign_response = personalize.describe_campaign(\n",
    "            campaignArn = campaign_arn\n",
    "        )\n",
    "        status = campaign_response[\"campaign\"][\"status\"]\n",
    "\n",
    "        if status == \"ACTIVE\":\n",
    "            print(f'Campaign {campaign_arn} successfully completed')\n",
    "            campaign_arns.remove(campaign_arn)\n",
    "        elif status == \"CREATE FAILED\":\n",
    "            print(f'Campaign {campaign_arn} failed')\n",
    "            if campaign_response.get('failureReason'):\n",
    "                print('   Reason: ' + campaign_response['failureReason'])\n",
    "            campaign_arns.remove(campaign_arn)\n",
    "\n",
    "    if len(campaign_arns) > 0:\n",
    "        print('At least one campaign is still in progress')\n",
    "        time.sleep(60)\n",
    "    else:\n",
    "        print(\"All campaigns have completed\")\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test Campaigns\n",
    "\n",
    "Now that our campaigns have been fully created, let's test each campaign and evaluate the results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Related Product Recommendations Campaign\n",
    "\n",
    "Let's test the recommendations made by the related items/products campaign by selecting a product from the Retail Demo Store's [Products](https://github.com/aws-samples/retail-demo-store/tree/master/src/products) microservice and requesting related item recommendations for that product."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Select a Product\n",
    "\n",
    "We'll just pick a random product for simplicity. Feel free to change the `product_id` below and execute the following cells with a different product to get a sense for how the recommendations change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "product_id = '020a5afe-fb13-4499-a1fa-8594d326eaa0'\n",
    "\n",
    "response = requests.get('http://{}/products/id/{}'.format(products_service_instance, product_id))\n",
    "product = response.json()\n",
    "print(json.dumps(product, indent=4, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get Related Product Recommendations for Product\n",
    "\n",
    "Now let's call Amazon Personalize to get related item/product recommendations for our product from the related item campaign."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_recommendations_response = personalize_runtime.get_recommendations(\n",
    "    campaignArn = related_campaign_arn,\n",
    "    itemId = str(product_id),\n",
    "    numResults = 10\n",
    ")\n",
    "\n",
    "item_list = get_recommendations_response['itemList']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(json.dumps(item_list, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the `itemId`'s in the above response don't tell us much about the products being recommended, let's get detailed information for each item ID from the Products microservice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "for item in item_list:\n",
    "    response = requests.get('http://{}/products/id/{}'.format(products_service_instance, item['itemId']))\n",
    "    print(json.dumps(response.json(), indent = 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the random product selected above, do the similar item recommendations from Personalize make sense? Keep in mind that the similar item recommendations from the SIMS recipe are based on the interactions we generated as input into the solution creation process above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Product Recommendations Campaign\n",
    "\n",
    "Let's test the recommendations made by the product recommendations campaign by selecting a user from the Retail Demo Store's Users microservice and requesting item recommendations for that user."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Select a User\n",
    "\n",
    "We'll just pick a random user for simplicity. Feel free to change the `user_id` below and execute the following cells with a different user to get a sense for how the recommendations change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_id = 256\n",
    "\n",
    "response = requests.get('http://{}/users/id/{}'.format(users_service_instance, user_id))\n",
    "user = response.json()\n",
    "persona = user['persona']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(json.dumps(user, indent=4, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Take note of the `persona` value for the user above. We should see recommendations for products consistent with this persona since we generated historical interactions for products in the categories represented in the persona.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Shopper persona for user {} is {}'.format(user_id, persona))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get Product Recommendations for User\n",
    "\n",
    "Now let's call Amazon Personalize to get recommendations for our user from the product recommendations campaign."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_recommendations_response = personalize_runtime.get_recommendations(\n",
    "    campaignArn = recommend_campaign_arn,\n",
    "    userId = str(user_id),\n",
    "    numResults = 10\n",
    ")\n",
    "\n",
    "item_list = get_recommendations_response['itemList']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(json.dumps(item_list, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that in this response we have a `score` field returned with each `itemId`. For all recipes except SIMS and Popularity-Count, Personalize [calculates a score](https://docs.aws.amazon.com/personalize/latest/dg/getting-real-time-recommendations.html) for each recommended item. Score values are between 0.0 and 1.0 and the sum of all scores across all items in your interactions and items datasets will total to 1.0. Therefore, the absolute value of scores will be smaller for larger item catalogs. We'll see how scores are calculated a bit differently for the personalized-ranking recipe below.\n",
    "\n",
    "Since the `itemId`'s in the above response don't tell us much about the products being recommended, let's retrieve product details from the Products microservice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "print('User persona: ' + persona)\n",
    "\n",
    "for item in item_list:\n",
    "    response = requests.get('http://{}/products/id/{}'.format(products_service_instance, item['itemId']))\n",
    "    print(json.dumps(response.json(), indent = 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Are the recommended products consistent with the persona? Note that this is a rather contrived example using a limited amount of generated interaction data without model parameter tuning. The purpose is to give you hands on experience building models and retrieving inferences from Amazon Personalize. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Personalized Ranking Campaign\n",
    "\n",
    "Next let's evaluate the results of the personalized ranking campaign. As a reminder, given a list of items and a user, this campaign will rerank the items based on the preferences of the user. For the Retail Demo Store, we will use this campaign to rerank the products listed for each category and the featured products list as well as reranking catalog search results displayed in the search widget."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get Featured Products List\n",
    "\n",
    "First let's get the list of featured products from the Products microservice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "response = requests.get('http://{}/products/featured'.format(products_service_instance))\n",
    "featured_products = response.json()\n",
    "print(json.dumps(featured_products, indent = 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ReRank Featured Products\n",
    "\n",
    "Using the featured products list just retrieved, first we'll create a list of item IDs that we want to rerank for a specific user. This reranking will allow us to provide ranked products based on the user's behavior. These behaviors should be consistent the same persona that was mentioned above (since we're going to use the same `user_id`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "unranked_product_ids = []\n",
    "\n",
    "for product in featured_products:\n",
    "    unranked_product_ids.append(product['id'])\n",
    "    \n",
    "print(', '.join(unranked_product_ids))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's have Personalize rank the featured product IDs based on our random user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "response = personalize_runtime.get_personalized_ranking(\n",
    "    campaignArn=ranking_campaign_arn,\n",
    "    inputList=unranked_product_ids,\n",
    "    userId=str(user_id)\n",
    ")\n",
    "reranked = response['personalizedRanking']\n",
    "print(json.dumps(response['personalizedRanking'], indent = 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Are the reranked results different than the original results from the Search service? Notice that we are also given a score for each item but this time the score values are larger. This is because scores for personalized-ranking results are calculated just across the items being reranked. Experiment with a different `user_id` in the cells above to see how the item ranking changes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pick products for discount\n",
    "\n",
    "Using the featured products list we'll pick some products for discount from the featured products.\n",
    "\n",
    "We'll get the ranking when discount context is applied for comparison. This is a using the \"contextual metadata\" feature of Amazon Personalize.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = personalize_runtime.get_personalized_ranking(\n",
    "    campaignArn=ranking_campaign_arn,\n",
    "    inputList=unranked_product_ids,\n",
    "    userId=str(user_id),\n",
    "    context={'DISCOUNT': 'Yes'} # Here we provide the context for the ranking\n",
    ")\n",
    "disount_reranked = response['personalizedRanking']\n",
    "print('Discount context ranking:', json.dumps(disount_reranked, indent = 4))\n",
    "print('Discount:', [item['itemId'] for item in disount_reranked[:2]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "We could use the discount-context ranking directly, but what we might be more interested in seeing is those products that\n",
    "benefit from having a discout shown. In our simulated data, certain products are more likely to see \n",
    "purchases with discount (to be precise, the cheaper ones). Let us find out which products benefit most. We also make use of the scores returned by Personalize when it returns the ranking."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "eps = 0.00001 #  \"epsilon\" - a number slightly more than zero so we don't get division by zero\n",
    "non_discount_rerank_scores = {item['itemId']: max(item['score'], eps) for item in reranked}\n",
    "discount_rerank_scores = {item['itemId']: item['score'] for item in disount_reranked}\n",
    "score_increases_with_discount = {item_id: discount_rerank_scores[item_id]/non_discount_rerank_scores[item_id]\n",
    "                                 for item_id in discount_rerank_scores}\n",
    "# Let us get the sorted items:\n",
    "discount_improve_sorted_items = sorted(score_increases_with_discount.keys(),\n",
    "                                       key=lambda key: score_increases_with_discount[key])\n",
    "\n",
    "print('Improvement ranking:', discount_improve_sorted_items)\n",
    "# Let us pick the two items that respond best to discounts\n",
    "print('Discount:', discount_improve_sorted_items[:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Has the ranking changed?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enable Campaigns in Retail Demo Store Recommendations Service\n",
    "\n",
    "Now that we've tested our campaigns and can get related product, product recommendations, and reranked items for our users, we need to enable the campaigns in the Retail Demo Store's [Recommendations service](https://github.com/aws-samples/retail-demo-store/tree/master/src/recommendations). The Recommendations service is called by the Retail Demo Store Web UI when a signed in user visits a page with personalized content capabilities (home page, product detail page, and category page). The Recommendations service checks Systems Manager Parameters values to determine the Personalize campaign ARNs to use for each of our three personalization use-cases.\n",
    "\n",
    "Let's set the campaign ARNs for our campaigns in the expected parameter names."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Update SSM Parameter To Enable Related Products"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = ssm.put_parameter(\n",
    "    Name='retaildemostore-related-products-campaign-arn',\n",
    "    Description='Retail Demo Store Related Products Campaign Arn Parameter',\n",
    "    Value='{}'.format(related_campaign_arn),\n",
    "    Type='String',\n",
    "    Overwrite=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Update SSM Parameter To Enable Product Recommendations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = ssm.put_parameter(\n",
    "    Name='retaildemostore-product-recommendation-campaign-arn',\n",
    "    Description='Retail Demo Store Product Recommendation Campaign Arn Parameter',\n",
    "    Value='{}'.format(recommend_campaign_arn),\n",
    "    Type='String',\n",
    "    Overwrite=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Update SSM Parameter To Enable Search Personalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = ssm.put_parameter(\n",
    "    Name='retaildemostore-personalized-ranking-campaign-arn',\n",
    "    Description='Retail Demo Store Personalized Ranking Campaign Arn Parameter',\n",
    "    Value='{}'.format(ranking_campaign_arn),\n",
    "    Type='String',\n",
    "    Overwrite=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate Personalization in Retail Demo Store's Web UI\n",
    "\n",
    "Now that you've enabled each personalization feature by setting the respective campaign ARN, you can test these\n",
    "personalization features through the Web App UI. If you haven't already opened a browser window/tab to the Retail Demo Store Web UI, navigate to the CloudFormation console in this AWS account and check the Outputs section of the stack used to launch the Retail Demo Store. Make sure you're checking the base/root stack and not the nested stacks that were created. In the Outputs section look for the output named: WebURL and browse to the URL provided.\n",
    "\n",
    "![CloudFormation Outputs](../images/cfn-webui-outputs.png)\n",
    "\n",
    "If you haven't already created a user account in your Retail Demo Store instance, let's create one now. When you access the Retail Demo Store Web UI for the first time, you will be prompted to create an account or sign in to an existing account. Click the \"**Create an account**\" button. If you skipped the account creation process, click the \"**Sign In**\" button and then click the \"**No account? Create account**\" link to create an account. Follow the prompts and enter the required data. You will need to provide a valid email address in order to receive an email with the confirmation code to validate your account.\n",
    "\n",
    "Once you've created and validated your account, click on the Sign In button again and sign in with the account you created."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Emulate Shopper\n",
    "\n",
    "To confirm product recommendations are personalized, you can emulate one of the many ficticious shoppers loaded into the system. You can also switch between shoppers by clicking the shopper profile name and details in the top navigation. You can have a shopper auto-selected for you or you can choose your own. In the shopper selection window, specify an age range and a primary shopping interest. Click Submit and a closely matching shopper is shown, confirm your choice or try again. Product recommendations should match the persona of the shopper you've selected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Viewing Related Product Recommendations\n",
    "\n",
    "Let's start with the Related Product Recommendations use-case.\n",
    "This campaign for this use-case is based on the [SIMS](https://docs.aws.amazon.com/personalize/latest/dg/native-recipe-sims.html)\n",
    "recipe which uses item-to-item collaborative filtering at its core to derive an understanding of how users interact with similar items.\n",
    "\n",
    "Browse to a [product detail page](https://github.com/aws-samples/retail-demo-store/blob/master/src/web-ui/src/public/ProductDetail.vue)\n",
    "and evaluate the products listed in the **Compare similar items** section.\n",
    "You should see the Personalize service icon and the SIMS recipe name displayed to the right of the section header.\n",
    "This tells you that results are actually coming from the related item campaign.\n",
    "If you don't see the Personalize service icon and recipe name, the page is using default behavior of displaying products from the same category\n",
    "(verify that the campaign was created successfully above **and** the campaign ARN is set as an SSM parameter).\n",
    "\n",
    "![Related Product Recommendations](./images/retaildemostore-related-products.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Viewing Product Recommendations\n",
    "\n",
    "With the user emulation saved, browse to the Retail Demo Store\n",
    "[home page](https://github.com/aws-samples/retail-demo-store/blob/master/src/web-ui/src/public/Main.vue) and evaluate\n",
    "the products listed in the **Inspired by your shopping trends** section.\n",
    "Do they appear consistent with the shopping persona you're emulating? For the screenshots listed here,\n",
    "the user was trained with historical data based primarily on products from the \"Footwear\" category, then to a lesser degree on products from the \"Jewelry\" category, and slightly on products from the \"Furniture\" category.\n",
    "\n",
    "![Personalized Product Recommendations](./images/retaildemostore-product-recs.jpg)\n",
    "\n",
    "Note that if the section is titled **Featured** or you don't see the Personalize service icon and recipe name displayed, this indicates that either you are not signed in as a user or the campaign ARN is not set as the appropriate SSM parameter. Double check that the campaign was created successfully above and that the campaign ARN is set in SSM."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Personalized Ranking\n",
    "\n",
    "Finally, let's evaluate the personalizated ranking use-case.\n",
    "There are two places where personalized ranking is implemented in the Retail Demo Store.\n",
    "With a user emulated, browse to the featured product category list by clicking on \"Featured\" from the Retail Demo Store home page.\n",
    "Note how for the emulated user with a persona of \"footwear_outdoors\" that the shoe is moved to the first product.\n",
    "(See [CategoryDetail.vue](https://github.com/aws-samples/retail-demo-store/blob/master/src/web-ui/src/public/CategoryDetail.vue)).\n",
    "\n",
    "![Personalized Product Ranking](./images/retaildemostore-personalized-ranking.jpg)\n",
    "\n",
    "The other feature where personalized ranking is implemented is in\n",
    "[search results](https://github.com/aws-samples/retail-demo-store/blob/master/src/web-ui/src/public/Search.vue).\n",
    "Start typing a word in the search box and a search result widget will be displayed.\n",
    "If the results were reranked by Personalize, you will see a \"Personalize Ranking\" annotation in the search box.\n",
    "For the emulated user with a historical affinity for footwear and outdoors,\n",
    "notice that a search for product keywords starting with \"s\" will move shoes to the top of the results.\n",
    "\n",
    "![Personalized Search Results](./images/retaildemostore-personalized-search.jpg)\n",
    "\n",
    "If the search functionality is not working at all for you, make sure that you\n",
    "completed the [Search workshop](../0-StartHere/Search.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Personalized Discounts\n",
    "\n",
    "The Personalized Discounts are enabled against the IVS demo tab (\"Live\") in the navigation bar where discounts are chosen over the\n",
    "current set of products streamed from the IVS live stream:\n",
    "\n",
    "![Personalized Discounts](./images/retaildemostore-personalized-discounts.png)\n",
    "\n",
    "Currently 2 products are selected for each video to be offered a discount.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Event Tracking - Keeping up with evolving user intent\n",
    "\n",
    "Up to this point we have trained and deployed three Amazon Personalize campaigns based on historical data that we\n",
    " generated in this workshop. This allows us to make related product, user recommendations, and rerank product\n",
    "  lists based on already observed behavior of our users. However, user intent often changes in real-time such\n",
    "  that what products the user is interested in now may be different than what they were interested in a week ago,\n",
    "  a day ago, or even a few minutes ago. Making recommendations that keep up with evolving user intent is one of the\n",
    "   more difficult challenges with personalization. Fortunately, Amazon Personalize has a mechanism for this exact issue.\n",
    "\n",
    "Amazon Personalize supports the ability to send real-time user events (i.e. clickstream) data into the service.\n",
    "Personalize uses this event data to improve recommendations. It will also save these events and automatically\n",
    "include them when solutions for the same dataset group are re-created (i.e. model retraining).\n",
    "\n",
    "The Retail Demo Store's Web UI already has\n",
    "[logic to send events](https://github.com/aws-samples/retail-demo-store/blob/master/src/web-ui/src/analytics/AnalyticsHandler.js)\n",
    "such as 'ProductViewed', 'ProductAdded', 'OrderCompleted', and others as they occur in real-time to a Personalize Event Tracker.\n",
    "These are the same event types we used to initially create the solutions and campaigns for our three use-cases.\n",
    "All we need to do is create an event tracker in Personalize, set the tracking Id for the tracker in an SSM parameter,\n",
    "and rebuild the Web UI service to pick up the change."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Personalize Event Tracker\n",
    "\n",
    "Let's start by creating an event tracker for our dataset group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "event_tracker_response = personalize.create_event_tracker(\n",
    "    datasetGroupArn=dataset_group_arn,\n",
    "    name='retaildemostore-event-tracker'\n",
    ")\n",
    "\n",
    "event_tracker_arn = event_tracker_response['eventTrackerArn']\n",
    "event_tracking_id = event_tracker_response['trackingId']\n",
    "\n",
    "print('Event Tracker ARN: ' + event_tracker_arn)\n",
    "print('Event Tracking ID: ' + event_tracking_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wait for Event Tracker Status to Become ACTIVE\n",
    "\n",
    "The event tracker should take a minute or so to become active."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "status = None\n",
    "max_time = time.time() + 60*60 # 1 hours\n",
    "while time.time() < max_time:\n",
    "    describe_event_tracker_response = personalize.describe_event_tracker(\n",
    "        eventTrackerArn = event_tracker_arn\n",
    "    )\n",
    "    status = describe_event_tracker_response[\"eventTracker\"][\"status\"]\n",
    "    print(\"EventTracker: {}\".format(status))\n",
    "    \n",
    "    if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n",
    "        break\n",
    "        \n",
    "    time.sleep(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Update SSM Parameter To Enable Event Tracking\n",
    "\n",
    "The Retail Demo Store's Web UI service just needs a Personalize Event Tracking Id to be able to send events to Personalize. The CodeBuild configuration for the Web UI service will pull the event tracking ID from an SSM parameter. \n",
    "\n",
    "Let's set our tracking ID in an SSM parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = ssm.put_parameter(\n",
    "    Name='retaildemostore-personalize-event-tracker-id',\n",
    "    Description='Retail Demo Store Personalize Event Tracker ID Parameter',\n",
    "    Value='{}'.format(event_tracking_id),\n",
    "    Type='String',\n",
    "    Overwrite=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trigger Web UI Service Release\n",
    "\n",
    "Next let's trigger a new release of the Retail Demo Store's Web UI service so that it will pick up our SSM parameter change.\n",
    "\n",
    "In the AWS console, browse to the AWS Code Pipeline service. Find the pipeline with **WebUIPipeline** in the name. Click on the pipeline name.\n",
    "\n",
    "![AWS CodePipeline](./images/retaildemostore-codepipeline.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Trigger Release\n",
    "\n",
    "To manually trigger a new release, click the **Release change** button, click the **Release** button on the popup dialog window, and then wait for the pipeline to build and deploy. This will rebuild the web app, deploy it to the web UI S3 bucket, and invalidate the CloudFront distribution to force browsers to load from the origin rather than from their local cache.\n",
    "\n",
    "![AWS CodePipeline Release](./images/retaildemostore-codepipeline-release.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify Event Tracking\n",
    "\n",
    "Return to your web browser tab/window where the Retail Demo Store Web UI is loaded and **reload the web app/page**. Reloading the page is important so that the web app is reloaded in your browser and the new event tracking configuration is loaded as well.\n",
    "\n",
    "There are a couple ways to verify that events are being sent to the Event Tracker. First, you can use your browser's Developer Tools to monitor the network calls made by the Retail Demo Store Web UI when you're browsing to product detail pages, adding items to carts, and completing orders. The other way you can verify that events are being received by the event tracker is in CloudWatch metrics for Personalize.\n",
    "\n",
    "1. If you have done so, **reload the web app by refreshing/reloading your browser page.** This is important so you browser session picks up the Event Tracker change released above.\n",
    "2. If not already signed in as a storefront user, sign in as (or create) a user. \n",
    "3. In the Retail Demo Store Web app, view product detail pages, add items to your cart, complete an order.\n",
    "4. Verify that the Web UI is making \"events\" calls to the Personalize Event Tracker.\n",
    "5. In the AWS console, browse to CloudWatch and then Metrics.\n",
    "\n",
    "![Personalize CloudWatch Metrics](./images/retaildemostore-eventtracker-cw.png)\n",
    "\n",
    "If events are not being sent to the event tracker, make sure that the WebUIPipeline pipeline was built and deployed successfully and that you reloaded the web app in your browser. Note that it make take a minute or so before events are reflected in CloudWatch.\n",
    "\n",
    "To assess the impact of real-time event tracking in recommendations made by the user recommendations on the home page, follow these steps.\n",
    "\n",
    "1. Sign in as (or create) a storefront user.\n",
    "2. View the product recommendations displayed on the home page under the \"Inspired by your shopping trends\" header. Take note of the products being recommended.\n",
    "3. View products from categories that are not being recommended by clicking on their product images to take you to the product detail view. When you view the details for a product, an event is fired and sent to the Personalize event tracker.\n",
    "4. Return to the home page and you should see product recommendations subtly changing to reflect the products you've engaged with. Repeat this process with other products and return to home page to see how recommendations are shifting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cold User Recommendations\n",
    "\n",
    "One of the key features of Personalize is being able to cold start users. Cold users are typically those who are new to your site or application and cold starting a user is getting from no personalization to making personalized recommendations in real-time. \n",
    "\n",
    "Personalize accomplishes cold starting users via the Event Tracker, just as we saw above with existing users. However, since new users are typically anonymous for a period of time before they create an account or may choose to transact as a guest, personalization is a valuable tool to help convert those anonymous users to transacting users. \n",
    "\n",
    "The challenge here is that Personalize needs a `userId` for anonymous users before it can make personalized recommendations. The Retail Demo Store solves this challenge by creating a provisional user ID the moment an anonymous user first hits the site. This provisional user ID is then used when streaming events to the Event Tracker and when retrieving recommendations from the Recommendations service. This allows the Retail Demo Store to start serving personalized recommendations after the first couple events are streamed to Personalize. Before recommendations can be personalized, Personalize will provide recommendations for popular items as a fallback.\n",
    "\n",
    "To see this behavior in action, browse to the Retail Demo Store storefront using a different browser, an Incognito/Private window, or sign out of your existing account. What you should see on the home page is that instead of **\"Inspired by your shopping behavior\"**, the section is **\"Trending products\"**. After you click on a couple provide detail pages, return to the home page and see that the section title and recommendations have changed. This indicates that recommendations are now being personalized and will continue to become more relevant as you engage with products.\n",
    "\n",
    "Similarly, the category pages will rerank products at first based on popularity and then become more and more personalized.\n",
    "\n",
    "There are some challenges with this approach, though. First is the question of what to do with the provisional user ID when the user creates an account. To maintain continuity of the user's interaction history, the Retail Demo Store passes the provisional user ID to the Users microservice when creating a new user account. The Users service then uses this ID as the user's ID going forward. Another challenge is how to handle a user that anonymously browses the site using multiple devices such as on the mobile device and then on a desktop/laptop. In this case, separate provisional user IDs are generated for sessions on each device. However, once the user creates an account on one device and then signs in with that account on the other device, both devices will starting using the same user ID going forward. A side effect here is that the interaction history from one of the devices will be orphaned. This is an acceptable tradeoff given the benefit of cold starting users earlier and is functionally the same UX without this scheme. Additional logic could be added to merge the interaction history from both prior anonymous sessions when the user creates an account. Also, customer data platforms can be used to help manage this for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Purchased Products Filter\n",
    "\n",
    "Amazon Personalize supports the ability to create [filters](https://docs.aws.amazon.com/personalize/latest/dg/filter.html) that can be used to exclude items from being recommended that meet a filter expression. Since it's a poor user experience to recommend products that a user has already purchased, we will create a filter that excludes recently purchased products. We'll do this by creating a filter expression that excludes items that have an interaction with an event type of `OrderCompleted` for the user.\n",
    "\n",
    "> As noted above, the Retail Demo Store web application streams clickstream events to Personalize when the user performs various actions such as viewing and purchasing products. The filter created below allows us to use those events as exclusion criteria. See the [AnalyticsHandler.js](https://github.com/aws-samples/retail-demo-store/blob/master/src/web-ui/src/analytics/AnalyticsHandler.js) file for the code that sends clickstream events."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = personalize.create_filter(\n",
    "    name = 'retaildemostore-filter-purchased-products',\n",
    "    datasetGroupArn = dataset_group_arn,\n",
    "    filterExpression = 'EXCLUDE itemId WHERE INTERACTIONS.event_type in (\"OrderCompleted\")'\n",
    ")\n",
    " \n",
    "filter_arn = response['filterArn']\n",
    "print(f'Filter ARN: {filter_arn}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wait for Filter Status to Become ACTIVE\n",
    "\n",
    "The filter should take a minute or so to become active."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "status = None\n",
    "max_time = time.time() + 60*60 # 1 hours\n",
    "while time.time() < max_time:\n",
    "    describe_filter_response = personalize.describe_filter(\n",
    "        filterArn = filter_arn\n",
    "    )\n",
    "    status = describe_filter_response[\"filter\"][\"status\"]\n",
    "    print(\"Filter: {}\".format(status))\n",
    "    \n",
    "    if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n",
    "        break\n",
    "        \n",
    "    time.sleep(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Purchased Products Filter\n",
    "\n",
    "To test our purchased products filter, we will request recommendations for a random user. Then we will send an `OrderCompleted` event for one of the recommended products to Personalize using the event tracker created above. Finally, we will request recommendations again for the same user but this time specify our filter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pick a user ID in the range of test users and fetch 5 recommendations.\n",
    "user_id = '456'\n",
    "get_recommendations_response = personalize_runtime.get_recommendations(\n",
    "    campaignArn = recommend_campaign_arn,\n",
    "    userId = user_id,\n",
    "    numResults = 5\n",
    ")\n",
    "\n",
    "item_list = get_recommendations_response['itemList']\n",
    "print(json.dumps(item_list, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next let's randomly select an item from the returned list of recommendations to be our product to purchase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "product_id_to_purchase = random.choice(item_list)['itemId']\n",
    "print(f'Product to simulate purchasing: {product_id_to_purchase}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next let's send an `OrderCompleted` event to Personalize to simulate that the product was just purchased.\n",
    "This will match the criteria for our filter.\n",
    "In the Retail Demo Store web application, this event is sent for each product in the order after the order is completed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = personalize_events.put_events(\n",
    "    trackingId = event_tracking_id,\n",
    "    userId = user_id,\n",
    "    sessionId = str(uuid.uuid4()),\n",
    "    eventList = [\n",
    "        {\n",
    "            'eventId': str(uuid.uuid4()),\n",
    "            'eventType': 'OrderCompleted',\n",
    "            'itemId': str(product_id_to_purchase),\n",
    "            'sentAt': int(time.time()),\n",
    "            'properties': '{\"discount\": \"No\"}'\n",
    "        }\n",
    "    ]\n",
    ")\n",
    "\n",
    "# Wait for OrderCompleted event to become consistent.\n",
    "time.sleep(5)\n",
    "\n",
    "print(json.dumps(response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's retrieve recommendations for the user again but this time specifying the filter to exclude recently\n",
    "purchased items. We do this by passing the filter's ARN via the `filterArn` parameter.\n",
    "In the Retail Demo Store, this is done in the\n",
    "[Recommendations](https://github.com/aws-samples/retail-demo-store/tree/master/src/recommendations) service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_recommendations_response = personalize_runtime.get_recommendations(\n",
    "    campaignArn = recommend_campaign_arn,\n",
    "    userId = user_id,\n",
    "    numResults = 5,\n",
    "    filterArn = filter_arn\n",
    ")\n",
    "\n",
    "item_list = get_recommendations_response['itemList']\n",
    "print(json.dumps(item_list, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code will raise an assertion error if the product we just purchased is still recommended."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "found_item = next((item for item in item_list if item['itemId'] == product_id_to_purchase), None)\n",
    "if found_item:\n",
    "    assert found_item == False, 'Purchased item found unexpectedly in recommendations'\n",
    "else:\n",
    "    print('Purchased item filtered from recommendations for user!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Update Filter SSM Parameter\n",
    "\n",
    "With our filter created and tested, the last step is to update the SSM parameter that is used throughout the\n",
    "Retail Demo Store project to detect the filter ARN.\n",
    "\n",
    "The [Recommendations](https://github.com/aws-samples/retail-demo-store/tree/master/src/recommendations) service already has logic to look for the purchased products filter ARN in SSM and use it when fetching recommendations. All we have to do is set the filter's ARN in SSM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = ssm.put_parameter(\n",
    "    Name='retaildemostore-personalize-filter-purchased-arn',\n",
    "    Description='Retail Demo Store Personalize Filter Purchased Products Arn Parameter',\n",
    "    Value='{}'.format(filter_arn),\n",
    "    Type='String',\n",
    "    Overwrite=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now if you test completing an order for one or more items in the Retail Demo Store web application for a user,\n",
    "those products should no longer be included in recommendations for that user.\n",
    "Test it out by purchasing a recommended product from the \"Inspired by your shopping trends\"\n",
    "section of the home page and then verifying that the product is no longer recommended."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Workshop Complete\n",
    "\n",
    "Congratulations! You have completed the Retail Demo Store Personalization Workshop.\n",
    "\n",
    "### Cleanup\n",
    "\n",
    "If you launched the Retail Demo Store in your personal AWS account **AND** you're done with all workshops, you can follow the [Personalize workshop cleanup](./1.3-Personalize-Cleanup.ipynb) notebook to delete all of the Amazon Personalize resources created by this workshop. **IMPORTANT: since the Personalize resources were created by this notebook and not CloudFormation, deleting the CloudFormation stack for the Retail Demo Store will not remove the Personalize resources. You MUST run the [Personalize workshop cleanup](./1.3-Personalize-Cleanup.ipynb) notebook or manually clean up these resources.**\n",
    "\n",
    "If you are participating in an AWS managed event such as a workshop and using an AWS provided temporary account, you can skip the cleanup workshop unless otherwise instructed."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}