More IO tutorial (dmlc#20)

* io tutorial

* adv io

python/basic/record_io.ipynb

@@ -0,0 +1,329 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Python Record IO\n",
+    "\n",
+    "In [image_io](image_io.ipynb) we already learned how to pack image into standard recordio format and load it with ImageRecordIter. This tutorial will walk through the python interface for reading and writing record io files. It can be useful when you need more more control over the details of data pipeline. For example, when you need to augument image and label together for detection and segmentation, or when you need a custom data iterator for triplet sampling and negative sampling.\n",
+    "\n",
+    "Setup environment first:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 26,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "%matplotlib inline\n",
+    "from __future__ import print_function\n",
+    "import mxnet as mx\n",
+    "import numpy as np\n",
+    "import matplotlib.pyplot as plt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The relevent code is under `mx.recordio`. There are two classes: `MXRecordIO`, which supports sequential read and write, and `MXIndexedRecordIO`, which supports random read and sequential write.\n",
+    "\n",
+    "## MXRecordIO\n",
+    "First let's take a look at `MXRecordIO`. We open a file `tmp.rec` and write 5 strings to it:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "record = mx.recordio.MXRecordIO('tmp.rec', 'w')\n",
+    "for i in range(5):\n",
+    "    record.write('record_%d'%i)\n",
+    "record.close()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Then we can read it back by opening the same file with 'r':"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "record_0\n",
+      "record_1\n",
+      "record_2\n",
+      "record_3\n",
+      "record_4\n"
+     ]
+    }
+   ],
+   "source": [
+    "record = mx.recordio.MXRecordIO('tmp.rec', 'r')\n",
+    "while True:\n",
+    "    item = record.read()\n",
+    "    if not item:\n",
+    "        break\n",
+    "    print item\n",
+    "record.close()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "collapsed": true
+   },
+   "source": [
+    "## MXIndexedRecordIO\n",
+    "Some times you need random access for more complex tasks. `MXIndexedRecordIO` is designed for this. Here we create a indexed record `tmp.rec` and a corresponding index file `tmp.idx`:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "record = mx.recordio.MXIndexedRecordIO('tmp.idx', 'tmp.rec', 'w')\n",
+    "for i in range(5):\n",
+    "    record.write_idx(i, 'record_%d'%i)\n",
+    "record.close()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can then access records with keys:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'record_3'"
+      ]
+     },
+     "execution_count": 17,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "record = mx.recordio.MXIndexedRecordIO('tmp.idx', 'tmp.rec', 'r')\n",
+    "record.read_idx(3)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can list all keys with:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[0, 1, 2, 3, 4]"
+      ]
+     },
+     "execution_count": 18,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "record.keys"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Packing and Unpacking Data\n",
+    "\n",
+    "Each record in a .rec file can contain arbitrary binary data, but machine learning data typically has a label/data structure. `mx.recordio` also contains a few utility functions for packing such data, namely: `pack`, `unpack`, `pack_img`, and `unpack_img`.\n",
+    "\n",
+    "### Binary Data\n",
+    "`pack` and `unpack` are used for storing float (or 1d array of float) label and binary data:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 27,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "float label '\\x00\\x00\\x00\\x00\\x00\\x00\\x80?\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00data'\n",
+      "array label '\\x03\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x80?\\x00\\x00\\x00@\\x00\\x00@@data'\n"
+     ]
+    }
+   ],
+   "source": [
+    "# pack\n",
+    "data = 'data'\n",
+    "label1 = 1.0\n",
+    "header1 = mx.recordio.IRHeader(flag=0, label=label1, id=1, id2=0)\n",
+    "s1 = mx.recordio.pack(header1, data)\n",
+    "print('float label:', repr(s1))\n",
+    "label2 = [1.0, 2.0, 3.0]\n",
+    "header2 = mx.recordio.IRHeader(flag=0, label=label2, id=2, id2=0)\n",
+    "s2 = mx.recordio.pack(header2, data)\n",
+    "print('array label:', repr(s2))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 30,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "(HEADER(flag=0, label=1.0, id=1, id2=0), 'data')\n",
+      "(HEADER(flag=3, label=array([ 1.,  2.,  3.], dtype=float32), id=2, id2=0), 'data')\n"
+     ]
+    }
+   ],
+   "source": [
+    "# unpack\n",
+    "print(*mx.recordio.unpack(s1))\n",
+    "print(*mx.recordio.unpack(s2))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Image Data\n",
+    "\n",
+    "`pack_img` and `unpack_img` are used for packing image data. Records packed by `pack_img` can be loaded by `mx.io.ImageRecordIter`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 36,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "'\\x00\\x00\\x00\\x00\\x00\\x00\\x80?\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00\\x00\\x01\\x00\\x01\\x00\\x00\\xff\\xdb\\x00C\\x00\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\xff\\xc0\\x00\\x0b\\x08\\x00\\x03\\x00\\x03\\x01\\x01\\x11\\x00\\xff\\xc4\\x00\\x1f\\x00\\x00\\x01\\x05\\x01\\x01\\x01\\x01\\x01\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x02\\x03\\x04\\x05\\x06\\x07\\x08\\t\\n\\x0b\\xff\\xc4\\x00\\xb5\\x10\\x00\\x02\\x01\\x03\\x03\\x02\\x04\\x03\\x05\\x05\\x04\\x04\\x00\\x00\\x01}\\x01\\x02\\x03\\x00\\x04\\x11\\x05\\x12!1A\\x06\\x13Qa\\x07\"q\\x142\\x81\\x91\\xa1\\x08#B\\xb1\\xc1\\x15R\\xd1\\xf0$3br\\x82\\t\\n\\x16\\x17\\x18\\x19\\x1a%&\\'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz\\x83\\x84\\x85\\x86\\x87\\x88\\x89\\x8a\\x92\\x93\\x94\\x95\\x96\\x97\\x98\\x99\\x9a\\xa2\\xa3\\xa4\\xa5\\xa6\\xa7\\xa8\\xa9\\xaa\\xb2\\xb3\\xb4\\xb5\\xb6\\xb7\\xb8\\xb9\\xba\\xc2\\xc3\\xc4\\xc5\\xc6\\xc7\\xc8\\xc9\\xca\\xd2\\xd3\\xd4\\xd5\\xd6\\xd7\\xd8\\xd9\\xda\\xe1\\xe2\\xe3\\xe4\\xe5\\xe6\\xe7\\xe8\\xe9\\xea\\xf1\\xf2\\xf3\\xf4\\xf5\\xf6\\xf7\\xf8\\xf9\\xfa\\xff\\xda\\x00\\x08\\x01\\x01\\x00\\x00?\\x00\\xfe\\x01\\xeb\\xff\\xd9'\n"
+     ]
+    }
+   ],
+   "source": [
+    "# pack_img\n",
+    "data = np.ones((3,3,1), dtype=np.uint8)\n",
+    "label = 1.0\n",
+    "header = mx.recordio.IRHeader(flag=0, label=label, id=0, id2=0)\n",
+    "s = mx.recordio.pack_img(header, data, quality=100, img_fmt='.jpg')\n",
+    "print(repr(s))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 37,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "HEADER(flag=0, label=1.0, id=0, id2=0) [[1 1 1]\n",
+      " [1 1 1]\n",
+      " [1 1 1]]\n"
+     ]
+    }
+   ],
+   "source": [
+    "# unpack_img\n",
+    "print(*mx.recordio.unpack_img(s))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Next Step\n",
+    "- [Advanced Image IO](advanced_img_io.ipynb) Advanced image IO for detection, segmentation, etc..."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python [Root]",
+   "language": "python",
+   "name": "Python [Root]"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}