Course_OpenAIBaselines_Unit11.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using OpenAI with ROS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/robotignite_logo_text.png\" width=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unit 5: Save and Load the trained policy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:green;color:white;\">SUMMARY</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time to completion: <b>1 hour</b><br><br>\n",
    "In this unit, you are going to see how to save the learned policy and how to load it to apply what the agent has learned."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:green;color:white;\">END OF SUMMARY</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, in the previous Units, you have been learning how to create the whole environments structure in order to get your robot training, and how this structure works. But now... what do you do when the training has finished? What do you do with all the new knowledge acquired? Well... you save it, of course!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Artificial Intellegence, we usually reffer to all this knowledge obtained during a training session as the **learned policy**. In this Unit, you are going to learn how to save this policy, and also how to load it afterwards so that your robot can use this knowledge to decide what to do."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding the save and load functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing we are going to do is to do an small modification to the **qlearn.py** script that you've using to train the robots until now. Basically, we are going to add 2 new functions to the script: \n",
    "* **save() function**: It will save the trained policy at the end of a training session to a new file.\n",
    "\n",
    "\n",
    "* **load() function**: It will load the trained policy from an existing file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, at the end of the **qlearn.py** file, add the following 2 new functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def save(self, filename):\n",
    "    np.save(filename, self.q)\n",
    "\n",
    "def load(self, filename):\n",
    "    self.q = np.load(filename).item()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, it's quite self-explanatory. We are basically using the **numpy** save and load methods to save and load the learned policy. So, at the end, you should have a file like this one:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:green;color:white;\">**qlearn.py**</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import random\n",
    "import numpy as np\n",
    "\n",
    "class QLearn:\n",
    "    def __init__(self, actions, epsilon, alpha, gamma):\n",
    "        self.q = dict()\n",
    "        self.epsilon = epsilon  # exploration constant\n",
    "        self.alpha = alpha      # discount constant\n",
    "        self.gamma = gamma      # discount factor\n",
    "        self.actions = actions\n",
    "\n",
    "    def getQ(self, state, action):\n",
    "        return self.q.get((state, action), 0.0)\n",
    "\n",
    "    def learnQ(self, state, action, reward, value):\n",
    "        '''\n",
    "        Q-learning:\n",
    "            Q(s, a) += alpha * (reward(s,a) + max(Q(s') - Q(s,a))            \n",
    "        '''\n",
    "        oldv = self.q.get((state, action), None)\n",
    "        if oldv is None:\n",
    "            self.q[(state, action)] = reward\n",
    "        else:\n",
    "            self.q[(state, action)] = oldv + self.alpha * (value - oldv)\n",
    "\n",
    "    def chooseAction(self, state, return_q=False):\n",
    "        q = [self.getQ(state, a) for a in self.actions]\n",
    "        maxQ = max(q)\n",
    "\n",
    "        if random.random() < self.epsilon:\n",
    "            minQ = min(q); mag = max(abs(minQ), abs(maxQ))\n",
    "            # add random values to all the actions, recalculate maxQ\n",
    "            q = [q[i] + random.random() * mag - .5 * mag for i in range(len(self.actions))] \n",
    "            maxQ = max(q)\n",
    "\n",
    "        count = q.count(maxQ)\n",
    "        # In case there're several state-action max values \n",
    "        # we select a random one among them\n",
    "        if count > 1:\n",
    "            best = [i for i in range(len(self.actions)) if q[i] == maxQ]\n",
    "            i = random.choice(best)\n",
    "        else:\n",
    "            i = q.index(maxQ)\n",
    "\n",
    "        action = self.actions[i]        \n",
    "        if return_q: # if they want it, give it!\n",
    "            return action, q\n",
    "        return action\n",
    "\n",
    "    def learn(self, state1, action1, reward, state2):\n",
    "        maxqnew = max([self.getQ(state2, a) for a in self.actions])\n",
    "        self.learnQ(state1, action1, reward, reward + self.gamma*maxqnew)\n",
    "\n",
    "    def save(self, filename):\n",
    "        np.save(filename, self.q)\n",
    "\n",
    "    def load(self, filename):\n",
    "        self.q = np.load(filename).item()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:green;color:white;\">**qlearn.py**</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modify the training scripts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! So now that we've added these 2 new functionalities to our Qlearn algorithm, let's use them in our training scripts. First of all, let's make sure that we save the trained policy whenever a training session has finished."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For that, let's add to our **start_training.py** script the code line in order to save the learned policy. The line will be like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "qlearn.save(\"file_name.npy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And where will we add it? Well, the best option is at the end of the script, right before we close the environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "qlearn.saveQ(\"file_name.npy\")\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Excellent! So we are now saving our training before it finishes, which it is a great improvement already. Now let's see how we can do it in order to load an already trained policy, in the case we already have one. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are actually many ways to do this. For instance, you could create a totally new script, which just loads a trained policy and uses it in order to decide what to do. For this case, though, we are going to integrate into the same script. Have a look at the following code snippet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "qfile = \"qlearn_states.npy\"\n",
    "if (os.path.exists(qfile)):\n",
    "    print(\"Loading from file:\",qfile)\n",
    "    qlearn.loadQ(qfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's quite simple, actually. We define a file name for our trained policy, in this case, *qlearn_states.npy*. Then, we check if this file already exists. If it already exists, then we load it. If it doesn't exist, then it will continue as usual, and save the file at the end of the training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, at the end, you would have something similar to this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:green;color:white;\">**start_training.py**</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#!/usr/bin/env python\n",
    "\n",
    "import gym\n",
    "import time\n",
    "import numpy\n",
    "import random\n",
    "import time\n",
    "import qlearn\n",
    "from gym import wrappers\n",
    "\n",
    "# ROS packages required\n",
    "import rospy\n",
    "import rospkg\n",
    "import os\n",
    "\n",
    "# import our training environment\n",
    "from openai_ros.task_envs.cartpole_stay_up import stay_up\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    \n",
    "    rospy.init_node('cartpole_gym', anonymous=True, log_level=rospy.WARN)\n",
    "\n",
    "    # Create the Gym environment\n",
    "    env = gym.make('CartPoleStayUp-v0')\n",
    "    rospy.loginfo ( \"Gym environment done\")\n",
    "        \n",
    "    # Set the logging system\n",
    "    rospack = rospkg.RosPack()\n",
    "    pkg_path = rospack.get_path('cartpole_v0_training')\n",
    "    outdir = pkg_path + '/training_results'\n",
    "    env = wrappers.Monitor(env, outdir, force=True) \n",
    "    rospy.loginfo ( \"Monitor Wrapper started\")\n",
    "\n",
    "    last_time_steps = numpy.ndarray(0)\n",
    "\n",
    "    # Loads parameters from the ROS param server\n",
    "    # Parameters are stored in a yaml file inside the config directory\n",
    "    # They are loaded at runtime by the launch file\n",
    "    Alpha = rospy.get_param(\"/cartpole_v0/alpha\")\n",
    "    Epsilon = rospy.get_param(\"/cartpole_v0/epsilon\")\n",
    "    Gamma = rospy.get_param(\"/cartpole_v0/gamma\")\n",
    "    epsilon_discount = rospy.get_param(\"/cartpole_v0/epsilon_discount\")\n",
    "    nepisodes = rospy.get_param(\"/cartpole_v0/nepisodes\")\n",
    "    nsteps = rospy.get_param(\"/cartpole_v0/nsteps\")\n",
    "    running_step = rospy.get_param(\"/cartpole_v0/running_step\")\n",
    "\n",
    "    # Initialises the algorithm that we are going to use for learning\n",
    "    qlearn = qlearn.QLearn(actions=range(env.action_space.n),\n",
    "                    alpha=Alpha, gamma=Gamma, epsilon=Epsilon)\n",
    "    initial_epsilon = qlearn.epsilon\n",
    "\n",
    "    start_time = time.time()\n",
    "    highest_reward = 0\n",
    "    \n",
    "    # Check if it already exists a training policy, and load it if so\n",
    "    qfile = \"qlearn_states.npy\"\n",
    "    if (os.path.exists(qfile)):\n",
    "        print(\"Loading from file:\",qfile)\n",
    "        qlearn.loadQ(qfile)\n",
    "\n",
    "    # Starts the main training loop: the one about the episodes to do\n",
    "    for x in range(nepisodes):\n",
    "        rospy.logdebug(\"############### START EPISODE => \" + str(x))\n",
    "        \n",
    "        cumulated_reward = 0  \n",
    "        done = False\n",
    "        if qlearn.epsilon > 0.05:\n",
    "            qlearn.epsilon *= epsilon_discount\n",
    "        \n",
    "        # Initialize the environment and get first state of the robot\n",
    "        \n",
    "        observation = env.reset()\n",
    "        state = ''.join(map(str, observation))\n",
    "        \n",
    "        # Show on screen the actual situation of the robot\n",
    "        # for each episode, we test the robot for nsteps\n",
    "        for i in range(nsteps):\n",
    "            \n",
    "            rospy.loginfo(\"############### Start Step => \"+str(i))\n",
    "            # Pick an action based on the current state\n",
    "            action = qlearn.chooseAction(state)\n",
    "            rospy.loginfo (\"Next action is: %d\", action)\n",
    "            # Execute the action in the environment and get feedback\n",
    "            observation, reward, done, info = env.step(action)\n",
    "            rospy.loginfo(str(observation) + \" \" + str(reward))\n",
    "            cumulated_reward += reward\n",
    "            if highest_reward < cumulated_reward:\n",
    "                highest_reward = cumulated_reward\n",
    "\n",
    "            nextState = ''.join(map(str, observation))\n",
    "\n",
    "            # Make the algorithm learn based on the results\n",
    "            #rospy.logwarn(\"############### State we were => \" + str(state))\n",
    "            #rospy.logwarn(\"############### Action that we took => \" + str(action))\n",
    "            #rospy.logwarn(\"############### Reward that action gave => \" + str(reward))\n",
    "            #rospy.logwarn(\"############### State in which we will start next step => \" + str(nextState))\n",
    "            qlearn.learn(state, action, reward, nextState)\n",
    "\n",
    "            if not(done):\n",
    "                state = nextState\n",
    "            else:\n",
    "                rospy.loginfo (\"DONE\")\n",
    "                last_time_steps = numpy.append(last_time_steps, [int(i + 1)])\n",
    "                break\n",
    "            rospy.loginfo(\"############### END Step =>\" + str(i))\n",
    "            #raw_input(\"Next Step...PRESS KEY\")\n",
    "            #rospy.sleep(2.0)\n",
    "        m, s = divmod(int(time.time() - start_time), 60)\n",
    "        h, m = divmod(m, 60)\n",
    "        rospy.logwarn ( (\"EP: \"+str(x+1)+\" - [alpha: \"+str(round(qlearn.alpha,2))+\" - gamma: \"+str(round(qlearn.gamma,2))+\" - epsilon: \"+str(round(qlearn.epsilon,2))+\"] - Reward: \"+str(cumulated_reward)+\"     Time: %d:%02d:%02d\" % (h, m, s)))\n",
    "\n",
    "    \n",
    "    rospy.loginfo ( (\"\\n|\"+str(nepisodes)+\"|\"+str(qlearn.alpha)+\"|\"+str(qlearn.gamma)+\"|\"+str(initial_epsilon)+\"*\"+str(epsilon_discount)+\"|\"+str(highest_reward)+\"| PICTURE |\"))\n",
    "\n",
    "    l = last_time_steps.tolist()\n",
    "    l.sort()\n",
    "\n",
    "    #print(\"Parameters: a=\"+str)\n",
    "    rospy.loginfo(\"Overall score: {:0.2f}\".format(last_time_steps.mean()))\n",
    "    rospy.loginfo(\"Best 100 score: {:0.2f}\".format(reduce(lambda x, y: x + y, l[-100:]) / len(l[-100:])))\n",
    "    \n",
    "    #Save the trained policy\n",
    "    qlearn.save(qfile)\n",
    "\n",
    "    env.close()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:green;color:white;\">**start_training.py**</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:#EE9023;color:white;\">**Exercise 5.1**</p>\n",
    "<br>\n",
    "\n",
    "a) With your new modified files, start a new training session and save the learned policy into a file.\n",
    "\n",
    "b) Then, load the trained policy and compare the results with the previous training session. Have they improved?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:#EE9023;color:white;\">**End of Exercise 5.1**</p>"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}