normApprox.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:754f4084f73583611e276f751c37f0ca952785a6f7d8a7a03e13a2d30b010e65"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<p class=\"title\"> Confidence intervals based on the normal approximation</p>\n",
      "\n",
      "One of the first things most statisticians would consider is to use confidence based on the normal approximation, especially if it were possible to draw a moderately large sample (e.g., 100 or more units).\n",
      "\n",
      "The procedure would be:\n",
      "+ choose the desired coverage probability $1-\\alpha$ and the sample size $n$\n",
      "+ draw a simple random sample or a random sample with replacement of size $n$ from the population\n",
      "+ compute the sample mean $\\bar{X}$ and sample standard deviation $s$\n",
      "+ find $\\hat{SE} = f \\times s/\\sqrt{n}$, the estimated standard error of the sample mean, where $f=1$ for sampling with replacement, and $f = \\sqrt{\\frac{N-n}{N-1}}$ for sampling without replacement ($N$ is the\n",
      "population size)\n",
      "+ find $t$, the appropriate critical value of Student's t distribution with $n-1$ degrees of freedom\n",
      "+ the approximate 2-sided confidence interval is $[\\bar{X} - t \\hat{SE}, \\bar{X} + t \\hat{SE}]$.\n",
      "\n",
      "### Formally, this gives an approximate confidence interval, but there's no guarantee that the actual coverage probability is close to the nominal coverage probability\n",
      "\n",
      "+ Normal approximation is common, but can be grossly inaccurate, especially with:\n",
      "  - small sample sizes\n",
      "  - skewed populations\n",
      "  \n",
      "In auditing situations, typical for the distributions to be very skewed: point mass at zero, mixed with something else.\n",
      "\n",
      "+ Examine scenarios by simulation.\n",
      "+ First scenario is a population of zeros and ones, in various proportions\n",
      "+ Second scenario is \"nonstandard\" mixture of pointmass at 0 and a Uniform\n",
      "\n",
      "We are estimating the coverage by simulation. \n",
      "\n",
      "For simulations:\n",
      "\n",
      "+ it's important to know which pseudo-random number generator (PRNG) you are using (Python uses the Mersenne Twister by default)\n",
      "+ it's important to assess whether that PRNG is adequate  (the Mersenne twister is pretty good for statistical simulations, but not good enough for cryptographic applications)\n",
      "+ it's important to remember that your estimates will have sampling error\n",
      "+ if the result matters, you should quantify the sampling error, for instance, by a confidence interval for the estimated result.\n",
      "\n",
      "Here, we will use 1000 or 10,000 iterations in the estimates. For estimating probabilities, the standard error of the result is no larger than $1/(2\\sqrt{n})$: 1.58% for $n=1000$ and 0.5% for $n=10,000$.\n",
      "\n",
      "For estimating things like expected length, guarantees of accuracy are harder, but the kinds of techniques we will see over this short course can be applied to _that_ problem too."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# This is the first cell with code: set up the Python environment\n",
      "%matplotlib inline\n",
      "import matplotlib.pyplot as plt\n",
      "import math\n",
      "import numpy as np\n",
      "import scipy as sp\n",
      "import scipy.stats\n",
      "from scipy.stats import binom\n",
      "import pandas as pd\n",
      "# For interactive widgets\n",
      "from IPython.html.widgets import interact, interactive, fixed\n",
      "from IPython.html import widgets\n",
      "from IPython.display import clear_output, display, HTML"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 6
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Population of two values, {0, 1}, in various proportions.  Amounts to Binomial random variable\n",
      "ns = np.array([25, 50, 100, 400])  # sample sizes\n",
      "ps = np.array([.001, .01, 0.1])    # mixture fractions, proportion of 1s in the population\n",
      "alpha = 0.05  # 1- (confidence level)\n",
      "reps = 1.0e4   # just for demonstration\n",
      "vals = [0, 1]\n",
      "\n",
      "simTable = pd.DataFrame(columns=('fraction of 1s', 'sample size', 'Student-t cov', 'Student-t len'))\n",
      "for p in ps:\n",
      "    popMean = p\n",
      "    for n in ns:\n",
      "        tCrit = sp.stats.t.ppf(q=1-alpha/2, df=n-1)\n",
      "        samMean = sp.stats.binom.rvs(n, p, size=reps)/float(n)\n",
      "        samSD = np.sqrt(samMean*(1-samMean)/(n-1))\n",
      "        cover = ( np.fabs(samMean-popMean) < tCrit*samSD).sum()\n",
      "        aveLen = 2*(tCrit*samSD).mean()\n",
      "        simTable.loc[len(simTable)] =  p, n, str(100*float(cover)/float(reps)) + '%', str(round(aveLen,4))\n",
      "#\n",
      "ansStr =  '<h3>Simulated coverage probability and expected length of Student-t confidence intervals for a {0, 1} population</h3>' +\\\n",
      "          '<strong>Nominal coverage probability ' + str(100*(1-alpha)) +\\\n",
      "          '%</strong>.<br /><strong>Estimated from ' + str(reps) + ' replications.</strong>'\n",
      "\n",
      "display(HTML(ansStr))\n",
      "display(simTable)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<h3>Simulated coverage probability and expected length of Student-t confidence intervals for a {0, 1} population</h3><strong>Nominal coverage probability 95.0%</strong>.<br /><strong>Estimated from 10000.0 replications.</strong>"
       ],
       "metadata": {},
       "output_type": "display_data",
       "text": [
        "<IPython.core.display.HTML at 0x10da89690>"
       ]
      },
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>fraction of 1s</th>\n",
        "      <th>sample size</th>\n",
        "      <th>Student-t cov</th>\n",
        "      <th>Student-t len</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0 </th>\n",
        "      <td> 0.001</td>\n",
        "      <td>  25</td>\n",
        "      <td>  2.39%</td>\n",
        "      <td> 0.0039</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1 </th>\n",
        "      <td> 0.001</td>\n",
        "      <td>  50</td>\n",
        "      <td>  4.82%</td>\n",
        "      <td> 0.0039</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2 </th>\n",
        "      <td> 0.001</td>\n",
        "      <td> 100</td>\n",
        "      <td>  9.37%</td>\n",
        "      <td> 0.0038</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3 </th>\n",
        "      <td> 0.001</td>\n",
        "      <td> 400</td>\n",
        "      <td>  33.0%</td>\n",
        "      <td> 0.0035</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4 </th>\n",
        "      <td>  0.01</td>\n",
        "      <td>  25</td>\n",
        "      <td> 22.43%</td>\n",
        "      <td> 0.0388</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5 </th>\n",
        "      <td>  0.01</td>\n",
        "      <td>  50</td>\n",
        "      <td> 38.65%</td>\n",
        "      <td> 0.0342</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>6 </th>\n",
        "      <td>  0.01</td>\n",
        "      <td> 100</td>\n",
        "      <td> 63.41%</td>\n",
        "      <td> 0.0307</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>7 </th>\n",
        "      <td>  0.01</td>\n",
        "      <td> 400</td>\n",
        "      <td> 89.82%</td>\n",
        "      <td> 0.0187</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>8 </th>\n",
        "      <td>   0.1</td>\n",
        "      <td>  25</td>\n",
        "      <td> 92.47%</td>\n",
        "      <td> 0.2319</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>9 </th>\n",
        "      <td>   0.1</td>\n",
        "      <td>  50</td>\n",
        "      <td>  87.9%</td>\n",
        "      <td> 0.1672</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>10</th>\n",
        "      <td>   0.1</td>\n",
        "      <td> 100</td>\n",
        "      <td> 92.92%</td>\n",
        "      <td> 0.1178</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>11</th>\n",
        "      <td>   0.1</td>\n",
        "      <td> 400</td>\n",
        "      <td> 95.19%</td>\n",
        "      <td> 0.0589</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "<p>12 rows \u00d7 4 columns</p>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "display_data",
       "text": [
        "   fraction of 1s sample size Student-t cov Student-t len\n",
        "0           0.001          25         2.39%        0.0039\n",
        "1           0.001          50         4.82%        0.0039\n",
        "2           0.001         100         9.37%        0.0038\n",
        "3           0.001         400         33.0%        0.0035\n",
        "4            0.01          25        22.43%        0.0388\n",
        "5            0.01          50        38.65%        0.0342\n",
        "6            0.01         100        63.41%        0.0307\n",
        "7            0.01         400        89.82%        0.0187\n",
        "8             0.1          25        92.47%        0.2319\n",
        "9             0.1          50         87.9%        0.1672\n",
        "10            0.1         100        92.92%        0.1178\n",
        "11            0.1         400        95.19%        0.0589\n",
        "\n",
        "[12 rows x 4 columns]"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Why/When is the coverage bad?\n",
      "\n",
      "+ Because of the high population mass at 0, the sample will often contain _only_ 0s. Then the SD will be zero, the length of the interval will be 0, and the interval won't cover the mean&mdash;which is bigger than zero.\n",
      "+ In general, coverage will be bad when sampling distribution of the sample mean is strongly skewed\n",
      "\n",
      "E.g., Binomial with very small or very large p."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def plotBinomialPmf(n, p):\n",
      "    '''\n",
      "       Plots the sampling distribution of the sample mean of a sample of size n\n",
      "       drawn with replacement from a {0, 1} population; i.e., a scaled binomial.\n",
      "       Plots a vertical red line at the expected value.\n",
      "    '''\n",
      "    fig, ax = plt.subplots(nrows=1, ncols=1)\n",
      "    x = np.arange(n+1)\n",
      "    xFrac = x.astype(float)/ x.max()\n",
      "    width = 1.0/n\n",
      "    ax.bar(xFrac, binom.pmf(x, n, p), width, align=\"center\")\n",
      "    plt.xlim([-width,1+width])\n",
      "    plt.axvline(x=p, color='r')\n",
      "\n",
      "interact(plotBinomialPmf, n=widgets.IntSliderWidget(min=5, max=300, step=5, value=30),\\\n",
      "         p=widgets.FloatSliderWidget(min=0.001, max=1, step=0.001, value=.001))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEACAYAAAC9Gb03AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAG5lJREFUeJzt3X9UW/X9x/FXHPtudhPWGGbdTosLZZL4o3CgSV0Ro+so\nyul+iWeN27HqthO7Y6nt8A88O0fmH84ddyY16/Fk/rPjgHl2xs5acY7BOUvT6khgqz/WwjyjZdMd\nPQeojczKBu39/oHcNkKTUBJCL8/HOTnm5n5C3/mkvnr53Hs/H5thGIYAAJZ1Sa4LAABkF0EPABZH\n0AOAxRH0AGBxBD0AWBxBDwAWlzLoI5GIXC6XSkpKFAwG52zT1NQkp9OpiooKDQ4Omq8//fTT+sIX\nvqCKigo98MADmasaAJC2lEG/c+dOhUIh9fT0aO/evRodHU3YH4vFdPDgQfX396uxsVGNjY2SpBMn\nTujRRx9Vd3e3+vr69Prrr6urqys7nwIAcF5Jgz4ej0uSqqurVVRUpJqaGkWj0YQ20WhU9fX1stvt\n8vv9GhgYkCRdeumlMgxD8Xhc77//vk6dOqWVK1dm6WMAAM4nadD39fWptLTU3Ha73ert7U1oE4vF\n5Ha7ze3CwkINDQ3p0ksv1VNPPaWrrrpKq1at0saNG+XxeDJcPgAglQWfjDUMQx+eRcFms2lkZETb\nt2/X0aNHNTw8rD//+c96/vnnF/rHAQDmKWnQr1+/PuHk6pEjR7Rhw4aENl6vV0ePHjW3R0ZG5HQ6\nFYvFtGHDBq1du1aXX3657rjjDkUikVl/RllZmWw2Gw8ePHjwmMejrKws/aQ3UigrKzMOHDhgHD9+\n3Lj66quNkZGRhP3RaNTYuHGjMTo6arS1tRl1dXWGYRjGyZMnjeLiYmNsbMyYmJgwtmzZYvT09Mz6\n+eeW8PDDD6cqZ1nIeT+k/muxKHLeD0sE/TCNfpg20w9pxLcpL9U/BC0tLQoEApqcnFRDQ4McDodC\noZAkKRAIyOPxqKqqSpWVlbLb7WptbZUkFRQU6Ac/+IG+9rWv6dSpU6qtrdXNN9+c/r9AAICMSBn0\nN910k3klzYxAIJCw/dhjj+mxxx6b9d67775bd99998IqBAAsyJK6M9bn8+W6hCWBfphGP0yjH6bR\nD9MupB9sH4z15IzNZpt11Q5yzGaT+E6AJW0+2bmkjugBAJlH0AOAxRH0AGBxBD0AWBxBjznZbDbl\n59tzXQaADOCqG8xms8kmQxLfDbBUcdUN5i0/327OoQHAWjiihyR9EPDT34MhjuiBpY4jegCAiaAH\nAIsj6AHA4gh6JJFnnqDlUkvg4sXJWEg6/8nYmdc4MQssLZyMBQCYCHoAsDiCfhnjJilgeWCMfhk7\nd1z+3PF4xuiBpS+jY/SRSEQul0slJSUKBoNztmlqapLT6VRFRYUGBwclSX//+99VXl5uPgoKCvTk\nk0/O42MAADIh5RF9eXm59uzZo6KiIm3evFmHDh2Sw+Ew98diMe3evVv79+9XV1eX2tra1NnZmfAz\nzpw5o89+9rOKxWJavXp1YgEc0ecMR/TAxStjR/TxeFySVF1draKiItXU1CgajSa0iUajqq+vl91u\nl9/v18DAwKyf09PTo+Li4lkhDwDIvqRB39fXp9LSUnPb7Xart7c3oU0sFpPb7Ta3CwsLNTQ0lNDm\n2Wef1Z133pmJegEA87Tgq24Mw5j168O5V3H873//03PPPac77rhjoX8UAOAC5CXbuX79ej344IPm\n9pEjR1RbW5vQxuv16ujRo9q8ebMkaWRkRE6n09z/wgsvqKKiQoWFhef9c5qbm83nPp9PPp9vPp8B\nACwvHA4rHA5f0HvTPhm7Zs0a1dbWnvdk7L59+9TV1aX29vaEk7Fbt27Vrbfeqm3bts1dACdjc4aT\nscDFaz7ZmfSIXpJaWloUCAQ0OTmphoYGORwOhUIhSVIgEJDH41FVVZUqKytlt9vV2tpqvve9995T\nT0+Pnn766Qv8KFg6pic4u+yylXr33RO5LgbAPHDD1DI23yN6seoUsGQwqRkAwETQA4DFEfQAYHEE\nPQBYHEG/DM1MTwxgeeCqm2Xo7NU2XHUDXKy46gYAYCLoAcDiCHoAsDiCHgAsjqAHAIsj6AHA4gh6\nALA4gh4ALI6gBwCLI+gBwOIIeszT9EpTNptN+fn2XBcDIA0plxIEEk1pZv6b8XEmRgMuBhzRA4DF\npQz6SCQil8ulkpISBYPBOds0NTXJ6XSqoqJCg4OD5uvvvfeetm3bps9//vNyu93q7e3NXOUAgLSk\nDPqdO3cqFAqpp6dHe/fu1ejoaML+WCymgwcPqr+/X42NjWpsbDT3Pfzww1qzZo1effVVvfrqq3K5\nXJn/BACApJIGfTwelyRVV1erqKhINTU1ikajCW2i0ajq6+tlt9vl9/s1MDBg7uvp6dFDDz2kj3/8\n48rLy1NBQUEWPgIAIJmkQd/X16fS0lJze67hl1gsJrfbbW4XFhbq2LFjevPNNzUxMaHt27fL6/Xq\nxz/+sSYmJjJcPgAglQVfdWMYxpyrnExMTOj111/X448/rk2bNikQCOjXv/617rrrrlltm5ubzec+\nn08+n2+hZQGApYTDYYXD4Qt6b9KlBOPxuHw+nw4fPixJ2rFjh2pra1VXV2e2CQaDmpqa0q5duyRJ\nxcXFGhoakiS5XC5zKOeFF17QM888o1/96leJBbCU4KJb6FKC577GdwfkRsaWEpwZU49EIhoeHlZ3\nd7e8Xm9CG6/Xq46ODo2Njam9vT3hhGtJSYmi0ajOnDmj559/Xps2bZrvZ0GGzCwIzqLgwPKTcuim\npaVFgUBAk5OTamhokMPhUCgUkiQFAgF5PB5VVVWpsrJSdrtdra2t5nt/8pOf6K677tLExIQ2bdqk\nrVu3Zu+TIKnx8XeUeHQOYLlIOnSzKAUwdLMozg7XSOcbhmHoBrh4ZGzoBgBw8SPoAcDiCHoAsDiC\nHgAsjqAHAIsj6AHA4gh6ALA4gh4ALI6gBwCLI+gBwOIIegCwOIIeC5Anm82m/Hx7rgsBkMSCFx7B\ncjYlydD4OLNhAksZR/QAYHEEPQBYHEEPABZH0AOAxRH0AGBxBD0AWFzKoI9EInK5XCopKVEwGJyz\nTVNTk5xOpyoqKjQ4OGi+ftVVV+n6669XeXm5PB5P5qoGAKQt5XX0O3fuVCgUUlFRkTZv3iy/3y+H\nw2Huj8ViOnjwoPr7+9XV1aXGxkZ1dnZKml68NhwOy27nhhoAyJWkR/TxeFySVF1draKiItXU1Cga\njSa0iUajqq+vl91ul9/v18DAQML+dFcpR3bk59tls3FDE7CcJQ36vr4+lZaWmttut1u9vb0JbWKx\nmNxut7ldWFioY8eOSZo+or/lllv01a9+Vfv3789k3UjT+Pg7kvjHFljOFjwFgmEY5z1qf/HFF3Xl\nlVdqYGBAW7Zskcfj0apVq2a1a25uNp/7fD75fL6FlgUAlhIOhxUOhy/ovTYjydhKPB6Xz+fT4cOH\nJUk7duxQbW2t6urqzDbBYFBTU1PatWuXJKm4uFhDQ0Ozftbu3bvlcrn03e9+N7EAm43hnSyaHrYx\nJM38V+c8n+s1yZBNtnm9h+8QWGzzyc6kQzcFBQWSpq+8GR4eVnd3t7xeb0Ibr9erjo4OjY2Nqb29\nXS6XS5J06tQpjY+PS5JGRkbU1dWl2traeX8YAMDCpBy6aWlpUSAQ0OTkpBoaGuRwOBQKhSRJgUBA\nHo9HVVVVqqyslN1uV2trqyTp7bff1te//nVJ0uWXX67vf//7Wr16dRY/CgBgLkmHbhalAIZusoqh\nG8CaMjZ0AwC4+BH0AGBxBD0AWBxBDwAWR9ADgMUR9ABgcQQ9MiBPNptNNptN+fnMVAosNQue6waQ\npjRzbf34ODNlAksNR/QAYHEEPQBYHEEPABZH0AOAxRH0AGBxBD0AWBxBDwAWR9ADgMUR9ABgcQQ9\nAFgcQW9B+fl2c+4ZAEgZ9JFIRC6XSyUlJQoGg3O2aWpqktPpVEVFhQYHBxP2nT59WuXl5dqyZUtm\nKkZK4+PvaHruGdZxBZBG0O/cuVOhUEg9PT3au3evRkdHE/bHYjEdPHhQ/f39amxsVGNjY8L+PXv2\nyO12c3QJADmSNOjj8bgkqbq6WkVFRaqpqVE0Gk1oE41GVV9fL7vdLr/fr4GBAXPfm2++qd///vf6\nzne+k/Zq5QCAzEoa9H19fSotLTW33W63ent7E9rEYjG53W5zu7CwUMeOHZMk7dq1S48//rguuYRT\nAQCQKwuej94wjDmP1js7O/XpT39a5eXlCofDSX9Gc3Oz+dzn88nn8y20LACwlHA4nDJLz8dmJBlT\nicfj8vl8Onz4sCRpx44dqq2tVV1dndkmGAxqampKu3btkiQVFxdraGhIDz30kH75y18qLy9PExMT\nevfdd3X77bfrmWeeSSzAZmNYJ8Omz4fM9OnM83RfkwzZZJvXexJf4/sEsm8+2Zl0TKWgoEDS9JU3\nw8PD6u7ultfrTWjj9XrV0dGhsbExtbe3y+VySZIeffRRvfHGGzp+/LieffZZ3XLLLbNCHgCQfSmH\nblpaWhQIBDQ5OamGhgY5HA6FQiFJUiAQkMfjUVVVlSorK2W329Xa2jrnz+GqGwDIjaRDN4tSAEM3\nGcfQDWB9GRu6AQBc/Ah6ALA4gh4Zliebzab8fHuuCwHwgQVfRw8kmpJkaHyck+/AUsERPQBYHEEP\nABZH0AOAxRH0AGBxBD0AWBxBDwAWR9ADgMUR9ABgcQQ9AFgcQQ8AFkfQA4DFEfQWkp9vZ4EXALMQ\n9BYyPv6Ozi4AAgDTCHoAsDiCHgAsLmXQRyIRuVwulZSUKBgMztmmqalJTqdTFRUVGhwclCRNTEzI\n6/WqrKxMGzZs0BNPPJHZygEAaUm5OHh5ebn27NmjoqIibd68WYcOHZLD4TD3x2Ix7d69W/v371dX\nV5fa2trU2dkpSTp16pRWrFih//73v6qoqNDvfvc7rV27NrEAFgfPmLOLgud2cfCZ1/hegezJ2OLg\n8XhcklRdXa2ioiLV1NQoGo0mtIlGo6qvr5fdbpff79fAwIC5b8WKFZKk//znP5qamtLHPvaxeX0Q\nAMDCJQ36vr4+lZaWmttut1u9vb0JbWKxmNxut7ldWFiooaEhSdLp06e1bt06XXHFFbr//vu1evXq\nTNYOAEjDgteMNQxj1q8PM9dyf+QjH9Err7yi4eFh3Xbbbdq4caPKy8tn/Yzm5mbzuc/nk8/nW2hZ\nAGAp4XBY4XD4gt6bdIw+Ho/L5/Pp8OHDkqQdO3aotrZWdXV1ZptgMKipqSnt2rVLklRcXGwe0Z+r\nsbFRa9eu1X333ZdYAGP0GbO0xug/qumFwqXLLlupd989kbHPCSCDY/QFBQWSpq+8GR4eVnd3t7xe\nb0Ibr9erjo4OjY2Nqb29XS6XS5I0OjqqkydPSpLGxsb0xz/+UV/5ylfm/WFwsZrSdOAbH9zIBSBX\nUg7dtLS0KBAIaHJyUg0NDXI4HAqFQpKkQCAgj8ejqqoqVVZWym63q7W1VZL01ltvadu2bTp9+rRW\nrVqlxsZGXXnlldn9NACAWVJeXpn1Ahi6yZilNXST+BrfMZBZGRu6AQBc/Ah6ALA4gh4ALI6gBwCL\nI+gBwOIIegCwOIIeACyOoAcAiyPoAcDiCPqLXH6+XTabzZwxFAA+jKC/yE1PGGbo7HQDAJCIoAcA\niyPoAcDiCHoAsDiCHgAsjqAHAIsj6LEI8mSz2ZSfb891IcCylHIpQWDhptePHR/nWn8gFziiBwCL\nSyvoI5GIXC6XSkpKFAwG52zT1NQkp9OpiooKDQ4OSpLeeOMN3Xzzzbrmmmvk8/nU3t6eucoBAGlJ\na3Hw8vJy7dmzR0VFRdq8ebMOHTokh8Nh7o/FYtq9e7f279+vrq4utbW1qbOzU2+//bbefvttlZWV\naXR0VB6PR6+88oouu+yyswWwOPiCnF0QXFroAt7ZXBx85jW+ayAzMro4eDwelyRVV1erqKhINTU1\nikajCW2i0ajq6+tlt9vl9/s1MDAgSVq1apXKysokSQ6HQ9dcc436+/vn9WEAAAuTMuj7+vpUWlpq\nbrvdbvX29ia0icVicrvd5nZhYaGGhoYS2vzjH//QkSNH5PF4FlozAGAeMnLVjWEYs36FOHc2xfHx\ncX3jG9/QE088oU984hOz3t/c3Gw+9/l88vl8mSgLACwjHA4rHA5f0HtTjtHH43H5fD4dPnxYkrRj\nxw7V1taqrq7ObBMMBjU1NaVdu3ZJkoqLi80j+snJSdXV1em2227TAw88MLsAxugXhDF6YHnK6Bh9\nQUGBpOkrb4aHh9Xd3S2v15vQxuv1qqOjQ2NjY2pvb5fL5ZI0faT/7W9/W9dee+2cIQ8AyL60hm5a\nWloUCAQ0OTmphoYGORwOhUIhSVIgEJDH41FVVZUqKytlt9vV2toqSXrxxRfV2tqq66+/XuXl5ZKk\nH/3oR6qtrc3SxwEAfFhal1dmtQCGbhaEoRtgecro0A2WppklBAEgFYL+InV2CUEASI6gBwCLI+gB\nwOIIeiyi6XnpmZseWFzMR49FND0vvSTmpgcWEUf0AGBxBD0AWBxBDwAWR9ADgMUR9ABgcQQ9AFgc\nQQ8AFkfQA4DFEfQAYHEEPQBYHEF/EZmZg5556AHMB0F/ETk7Bz3z0ANIX1pBH4lE5HK5VFJSomAw\nOGebpqYmOZ1OVVRUaHBw0Hz93nvv1RVXXKHrrrsuMxXDIvKYxRJYJGkF/c6dOxUKhdTT06O9e/dq\ndHQ0YX8sFtPBgwfV39+vxsZGNTY2mvvuuece/eEPf8hs1bCA6Zksp39LAZBNKYM+Ho9Lkqqrq1VU\nVKSamhpFo9GENtFoVPX19bLb7fL7/RoYGDD33XjjjVq5cmWGywYApCtl0Pf19am0tNTcdrvd6u3t\nTWgTi8XkdrvN7cLCQg0NDWWwTADAhcrIyVjDMGQYiScIuTIEAJaGlCtMrV+/Xg8++KC5feTIEdXW\n1ia08Xq9Onr0qDZv3ixJGhkZkdPpTLuI5uZm87nP55PP50v7vQCwHITDYYXD4Qt6r8348KH4HMrL\ny7Vnzx6tWbNGtbW1OnTokBwOh7k/Fotp9+7d2rdvn7q6utTe3q7Ozk5z//DwsLZs2aLXXnttdgE2\n26zfBjC36d+SZvpq5nmmX5MM2WTLys+e+zW+f2D+5pOdaa0Z29LSokAgoMnJSTU0NMjhcCgUCkmS\nAoGAPB6PqqqqVFlZKbvdrtbWVvO9fr9fBw4c0NjYmFavXq1HHnlE99xzzwV8LADAhUjriD6rBXBE\nnzaO6AHMmE92cmfsRWBm6gMAuBAE/UXg7NQHVpRnzt/DXbJAdqQ1Rg9kz/QdspI0Ps5vLUA2cEQP\nABZH0AOAxRH0AGBxBD0AWBxBDwAWR9BjCWExEiAbCPolanmuD8tiJEA2EPRLFOvDAsgUgh4ALI6g\nBwCLI+gBwOIIeixBTHQGZBJBv8QwJbF0dqIzrsABMoGgX2KsPSUxgFwg6AHA4gh6LHGM1wMLlTLo\nI5GIXC6XSkpKFAwG52zT1NQkp9OpiooKDQ4Ozuu9WK53waaL8XpgwYwUysrKjAMHDhjDw8PG1Vdf\nbYyMjCTsj0ajxsaNG42xsTGjvb3dqKurS/u9HyxMbj7/05/+lKocS5JkSMYHDxnSn+Z4ba522XjN\nMIxF/fOSvfbhfsgzJBmXXbYy11/Zolqu/198GP0wbaYf0ohvU9Ij+ng8Lkmqrq5WUVGRampqFI1G\nE9pEo1HV19fLbrfL7/drYGAg7fd+WDgcTrrfSpIfxYcXu5wlKvyh7eU5F85y+v8iGfph2oX0Q9Kg\n7+vrU2lpqbntdrvV29ub0CYWi8ntdpvbhYWFGhoaSuu9y9FMwDOXzUIwbg/Mx4JPxhqGMf3L/jkY\na5529qj9/8xg4vLJTDh33H58Vh/PPOcfAeADycZ1Tp48aZSVlZnb999/v9HZ2ZnQ5sknnzR++tOf\nmttOp9MwDMN45513Ur7XMAxj3bp1M4e1PHjw4MEjzce6devSHqPPUxIFBQWSpq+eWbNmjbq7u/Xw\nww8ntPF6vdq9e7fuuusudXV1yeVySZI+9alPpXyvJL388svJSgAALFDSoJeklpYWBQIBTU5OqqGh\nQQ6HQ6FQSJIUCATk8XhUVVWlyspK2e12tba2Jn0vAGBx2YwPD7ADACwlJ3fGLuQmLCtJ1Q9tbW1a\nt26d1q1bpzvvvFOvv/56DqrMrnRvquvr61NeXp5++9vfLmJ1iyedfujr69P69evlcrnk8/kWt8BF\nkqof3n//fW3btk3l5eW66aabtG/fvhxUmX333nuvrrjiCl133XXnbTOvjEx7ND+DFnITlpWk6oeX\nXnrJOHnypGEYhvGLX/zC+Na3vpWLMrMqnZvqpqamjJtvvtmoq6szfvOb3+SgyuxL1Q9nzpwxrr32\nWqO7u9swDGPOfrKCVP3w1FNPGdu3bzcMwzCGh4cNp9NpnDlzJhelZlUkEjH++te/Gtdee+2c++eb\nkYt+RL+Qm7CsJJ1+uOGGG8wT4nV1dTpw4MCi15lN6d5UFwwGVV9fr8LCwsUucVGk0w/9/f26/vrr\ntWnTJkmy5PmudPqhoKBA4+Pjmpyc1IkTJ7RixQpLXs594403auXKlefdP9+MXPSgX8hNWFYy3xvK\nfv7zn2vLli2LUdqiSacP/v3vf2vfvn3avn27JGveo5FOP3R1dclms+nGG2/Uli1b1NXVtdhlZl06\n/eD3+3X69Gk5HA5VVVWpra1tsctcEuabkSmvuskFg5uwEvT09Ki1tVUvvfRSrktZdA888IAee+wx\n2Wy2Of9eLBcTExN6+eWX1dPTo1OnTulLX/qS/va3v+nSSy/NdWmL6mc/+5ny8vL01ltv6bXXXlNd\nXZ3++c9/6pJLltdEvPPNyEXvnfXr1yecODhy5Ig2bNiQ0Mbr9ero0aPm9sjIiJxO56LVuBjS6QdJ\nevXVV3Xfffdp//795r0JVpFOH/zlL3/R1q1b9bnPfU4dHR363ve+p/379y92qVmVTj/ccMMNuvXW\nW7Vq1So5nU5VVlYqEoksdqlZlU4/RCIRffOb39SKFSvk9Xr1mc98xpIXKaQy34xc9KA/9yas4eFh\ndXd3y+v1JrTxer3q6OjQ2NiY2tvbzZuwrCSdfvjXv/6l22+/XW1tbVq7dm0uysyqdPrg2LFjOn78\nuI4fP676+no99dRT+vKXv5yLcrMmnX7YsGGDDhw4oFOnTunEiRM6fPiwNm7cmItysyadfvjiF7+o\n5557TmfOnNGxY8d04sSJhOGe5WK+GZmToZuF3IRlJan64ZFHHtGJEyd03333SZI++tGPKhaL5bLk\njEvVB8tFqn64/PLLdc8996iyslKFhYV65JFH9MlPfjLHVWdeqn7YunWrjh49avbDnj17clxxdvj9\nfh04cECjo6NavXq1fvjDH2pyclLShWUkN0wBgMUtrzMYALAMEfQAYHEEPQBYHEEPABZH0AOAxRH0\nAGBxBD0AWBxBDwAW9//n5XS3wTxw/QAAAABJRU5ErkJggg==\n",
       "text": [
        "<matplotlib.figure.Figure at 0x10da7de10>"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Nonstandard mixtures of distributions\n",
      "\n",
      "+ Common in statistical study of auditing to consider \"nonstandard mixtures\" of a pointmass and continuous distribution \n",
      "+ E.g., NRC Monograph and [1989 Statistical Science article](http://projecteuclid.org/download/pdf_1/euclid.ss/1177012665)\n",
      "+ Let's look at a mixture of uniform(0,1) and a pointmass at zero"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Nonstandard mixture: a pointmass at zero and a uniform[0,1]\n",
      "ns = np.array([25, 50, 100, 400])  # sample sizes\n",
      "ps = np.array([0.9, 0.99, 0.999])    # mixture fraction, weight of pointmass\n",
      "alpha = 0.05  # 1- (confidence level)\n",
      "reps = 1.0e4   # just for demonstration\n",
      "\n",
      "simTable = pd.DataFrame(columns=('mass at 0', 'sample size', 'Student-t cov', 'Student-t len'))\n",
      "for p in ps:\n",
      "    popMean = (1-p)*0.5  #  p*0 + (1-p)*.5\n",
      "    for n in ns:\n",
      "        tCrit = sp.stats.t.ppf(q=1-alpha/2, df=n-1)\n",
      "        cover = 0\n",
      "        totLen = 0.0\n",
      "        samMean = np.zeros(reps)\n",
      "        for rep in range(int(reps)):\n",
      "            sam = np.random.uniform(size=n)\n",
      "            ptMass = np.random.uniform(size=n)\n",
      "            sam[ptMass < p] = 0.0\n",
      "            samMean[rep] = np.mean(sam)\n",
      "            samSD = np.std(sam, ddof=1)\n",
      "            cover += ( math.fabs(samMean[rep]-popMean) < tCrit*samSD )\n",
      "            totLen += 2*tCrit*samSD\n",
      "        simTable.loc[len(simTable)] =  p, n, str(100*float(cover)/float(reps)) + '%', str(round(totLen/reps,4))\n",
      "#\n",
      "ansStr =  '<h3>Simulated coverage probability and expected length of Student-t confidence intervals for mixture of U[0,1] and ' +\\\n",
      "          'pointmass at 0</h3>' +\\\n",
      "          '<strong>Nominal coverage probability ' + str(100*(1-alpha)) +\\\n",
      "          '%</strong>. <br /><strong>Estimated from ' + str(reps) + ' replications.</strong>'\n",
      "\n",
      "display(HTML(ansStr))\n",
      "display(simTable)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<h3>Simulated coverage probability and expected length of Student-t confidence intervals for mixture of U[0,1] and pointmass at 0</h3><strong>Nominal coverage probability 95.0%</strong>. <br /><strong>Estimated from 10000.0 replications.</strong>"
       ],
       "metadata": {},
       "output_type": "display_data",
       "text": [
        "<IPython.core.display.HTML at 0x10dd7e490>"
       ]
      },
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>mass at 0</th>\n",
        "      <th>sample size</th>\n",
        "      <th>Student-t cov</th>\n",
        "      <th>Student-t len</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0 </th>\n",
        "      <td>   0.9</td>\n",
        "      <td>  25</td>\n",
        "      <td> 90.52%</td>\n",
        "      <td> 0.6477</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1 </th>\n",
        "      <td>   0.9</td>\n",
        "      <td>  50</td>\n",
        "      <td> 98.69%</td>\n",
        "      <td> 0.6714</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2 </th>\n",
        "      <td>   0.9</td>\n",
        "      <td> 100</td>\n",
        "      <td> 99.99%</td>\n",
        "      <td> 0.6829</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3 </th>\n",
        "      <td>   0.9</td>\n",
        "      <td> 400</td>\n",
        "      <td> 100.0%</td>\n",
        "      <td> 0.6865</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4 </th>\n",
        "      <td>  0.99</td>\n",
        "      <td>  25</td>\n",
        "      <td> 22.61%</td>\n",
        "      <td> 0.1006</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5 </th>\n",
        "      <td>  0.99</td>\n",
        "      <td>  50</td>\n",
        "      <td> 38.35%</td>\n",
        "      <td>  0.124</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>6 </th>\n",
        "      <td>  0.99</td>\n",
        "      <td> 100</td>\n",
        "      <td> 63.07%</td>\n",
        "      <td> 0.1631</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>7 </th>\n",
        "      <td>  0.99</td>\n",
        "      <td> 400</td>\n",
        "      <td>  98.1%</td>\n",
        "      <td> 0.2107</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>8 </th>\n",
        "      <td> 0.999</td>\n",
        "      <td>  25</td>\n",
        "      <td>  2.75%</td>\n",
        "      <td> 0.0119</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>9 </th>\n",
        "      <td> 0.999</td>\n",
        "      <td>  50</td>\n",
        "      <td>  5.15%</td>\n",
        "      <td>  0.015</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>10</th>\n",
        "      <td> 0.999</td>\n",
        "      <td> 100</td>\n",
        "      <td>  9.44%</td>\n",
        "      <td> 0.0188</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>11</th>\n",
        "      <td> 0.999</td>\n",
        "      <td> 400</td>\n",
        "      <td> 32.92%</td>\n",
        "      <td> 0.0357</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "<p>12 rows \u00d7 4 columns</p>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "display_data",
       "text": [
        "   mass at 0 sample size Student-t cov Student-t len\n",
        "0        0.9          25        90.52%        0.6477\n",
        "1        0.9          50        98.69%        0.6714\n",
        "2        0.9         100        99.99%        0.6829\n",
        "3        0.9         400        100.0%        0.6865\n",
        "4       0.99          25        22.61%        0.1006\n",
        "5       0.99          50        38.35%         0.124\n",
        "6       0.99         100        63.07%        0.1631\n",
        "7       0.99         400         98.1%        0.2107\n",
        "8      0.999          25         2.75%        0.0119\n",
        "9      0.999          50         5.15%         0.015\n",
        "10     0.999         100         9.44%        0.0188\n",
        "11     0.999         400        32.92%        0.0357\n",
        "\n",
        "[12 rows x 4 columns]"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Again the distribution of the sample mean is skewed, which hurts the coverage of Student-t intervals. \n",
      "\n",
      "When the mass at 0 gets large enough, there is again a large chance that the sample will contain _only_ 0s, in which case the sample SD will be zero, the length of the interval will be zero, and the interval won't cover."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Summary\n",
      "\n",
      "* The actual coverage probability of confidence intervals based on the normal approximation can be *much* less than the nominal coverage probability, even for large-ish sample sizes, for skewed population distributions.\n",
      "* In many applications, the population _is_ very skewed: \"Nonstandard mixture distributions\"\n",
      "* For example, accounting overstatements might be mostly 0, and occasionally large. Then:\n",
      "   * Can be very likely that the sample variance is zero\n",
      "   * Leads to apparent certainty that the result is correct."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## What's next?\n",
      "We will consider some conservative methods for constructing confidence bounds by inverting hypothesis tests.\n",
      "\n",
      "- [Next: Duality between hypothesis tests and confidence sets](duality.ipynb)\n",
      "- [Previous: Canonical examples of real-world problems we will consider](canonical.ipynb)\n",
      "- [Index](index.ipynb)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%run talkTools.py"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<style>\n",
        "\n",
        "font-family: Helvetica\n",
        "\n",
        ".MathJax_Display {\n",
        "  padding: 10px;\n",
        "}\n",
        "\n",
        "div.callout {\n",
        "  color: #000000;\n",
        "  background-color: #DDDDDD;\n",
        "  margin: 20px 20px 20px 20px;\n",
        "  border-style: solid;\n",
        "  border-width: 2px;\n",
        "  padding: 10px 10px;\n",
        "}\n",
        ".rendered_html {\n",
        "  color: #2C5494;\n",
        "  font-family: Helvetica;\n",
        "  font-size: 140%;\n",
        "  line-height: 1.1;\n",
        "  margin: 0.5em 0;\n",
        "}\n",
        "\n",
        ".title {\n",
        "  color: #498AF3;\n",
        "  font-size: 250%;\n",
        "  font-weight:bold;\n",
        "  line-height: 1.2;\n",
        "  margin: 10px 50px 10px;\n",
        "}\n",
        "\n",
        ".subtitle {\n",
        "  color: #386BBC;\n",
        "  font-size: 180%;\n",
        "  font-weight:bold;\n",
        "  line-height: 1.2;\n",
        "  margin: 20px 50px 20px;\n",
        "}\n",
        "\n",
        ".slide-header, p.slide-header {\n",
        "  color: #498AF3;\n",
        "  font-size: 200%;\n",
        "  font-weight:bold;\n",
        "  margin: 0px 20px 10px;\n",
        "  page-break-before: always;\n",
        "  text-align: center;\n",
        "}\n",
        "\n",
        ".rendered_html .code {\n",
        "  background-color: #999999;\n",
        "}\n",
        "\n",
        ".rendered_html h1 {\n",
        "  color: #498AF3;\n",
        "  line-height: 1.2;\n",
        "  margin: 0.15em 0em 0.5em;\n",
        "  page-break-before: always;\n",
        "  text-align: center;\n",
        "}\n",
        "\n",
        "\n",
        ".rendered_html h2 {\n",
        "  color: #386BBC;\n",
        "  line-height: 1.2;\n",
        "  margin: 1.1em 0em 0.5em;\n",
        "}\n",
        "\n",
        ".rendered_html h3 {\n",
        "  font-size: 100%;\n",
        "  line-height: 1.2;\n",
        "  margin: 1.1em 0em 0.5em;\n",
        "}\n",
        "\n",
        ".rendered_html .definition, .proposition, .proof, .theorem {\n",
        "    padding-top: 20px;\n",
        "    color: #222299;\n",
        "    font-size: 120%;\n",
        "    font-style: italic;\n",
        "}\n",
        "\n",
        ".definition, .proposition, .theorem {\n",
        "  background-color: #EEEEEE;\n",
        "  border-style: solid;\n",
        "  border-width: 2px;\n",
        "  padding-left: 20px;\n",
        "  padding-right: 20px;\n",
        "}\n",
        "\n",
        ".rendered_html .definition::before{\n",
        "    content: \"Definition:\";\n",
        "    background-color: #DDDDDD;\n",
        "    color: #222299;\n",
        "    font-size: 110%;\n",
        "    font-weight: bold;\n",
        "    font-style: normal;\n",
        "}\n",
        "\n",
        ".rendered_html .proposition::before{\n",
        "    content: \"Proposition:\";\n",
        "    background-color: #DDDDDD;\n",
        "    color: #222299;\n",
        "    font-size: 110%;\n",
        "    font-weight: bold;\n",
        "    font-style: normal;\n",
        "}\n",
        "\n",
        ".rendered_html .proof::before{\n",
        "    content: \"Proof:\";\n",
        "    background-color: #DDDDDD;\n",
        "    color: #222299;\n",
        "    font-size: 110%;\n",
        "    font-weight: bold;\n",
        "    font-style: normal;\n",
        "}\n",
        "\n",
        ".rendered_html .theorem::before{\n",
        "    content: \"Theorem:\";\n",
        "    background-color: #DDDDDD;\n",
        "    color: #222299;\n",
        "    font-size: 110%;\n",
        "    font-weight: bold;\n",
        "    font-style: normal;\n",
        "}\n",
        "\n",
        "\n",
        ".rendered_html li {\n",
        "  line-height: 1.8;\n",
        "}\n",
        "\n",
        ".input_prompt, .CodeMirror-lines, .output_area {\n",
        "  font-family: Consolas;\n",
        "  font-size: 120%;\n",
        "}\n",
        "\n",
        ".gap-above {\n",
        "  padding-top: 200px;\n",
        "}\n",
        "\n",
        ".gap01 {\n",
        "  padding-top: 10px;\n",
        "}\n",
        "\n",
        ".gap05 {\n",
        "  padding-top: 50px;\n",
        "}\n",
        "\n",
        ".gap1 {\n",
        "  padding-top: 100px;\n",
        "}\n",
        "\n",
        ".gap2 {\n",
        "  padding-top: 200px;\n",
        "}\n",
        "\n",
        ".gap3 {\n",
        "  padding-top: 300px;\n",
        "}\n",
        "\n",
        ".emph {\n",
        "  color: #386BBC;\n",
        "}\n",
        "\n",
        ".warn {\n",
        "  color: red;\n",
        "}\n",
        "\n",
        ".center {\n",
        "  text-align: center;\n",
        "}\n",
        "\n",
        ".nb_link {\n",
        "    padding-bottom: 0.5em;\n",
        "}\n",
        "\n",
        "</style>"
       ],
       "metadata": {},
       "output_type": "display_data",
       "text": [
        "<IPython.core.display.HTML at 0x10da411d0>"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 10
    }
   ],
   "metadata": {}
  }
 ]
}