Torch_Dataset_Dataloader.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Torch_Dataset_Dataloader.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LlF8Xr3XmRuM"
      },
      "source": [
        "PyTorch(or Torch) Dataset and Dataloader are used for easy accessing(=retrieving, obtaining, acquiring, reading, examining) of data for machine learning and deep learning models. A lot of effort in solving any machine learning problem goes in to preparing(=preprocessing) the data. PyTorch provides many tools to make data loading (= activation) easy (if data loading is easy, feeding the data into the model is easy), and to make the code more readable."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vYQM32S3QduK"
      },
      "source": [
        "#Mounting Google Drive"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P3R8YTXWh4Nk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0b2f76bf-729f-4223-b4aa-cbd93e25a62b"
      },
      "source": [
        "'''\n",
        "Mounting => Before your computer can use any kind of storage device (such as hard drive, Google drive), you or your operating system must make it\n",
        "accessible through the computer’s file system. This process is called mounting. You can only access files on mounted media.\n",
        "In Computers, to mount is to make a group of files in a file system structure accessible to a user or user group. In some usages, it means to make a\n",
        "device physically accessible. Mounting a file system (Google drive) attaches that Google drive to a directory (mount  point) and makes it available to the\n",
        "system. In simple words, with mounting a Google drive, user and operating system can access to all the files present in the Google drive. A mounted disk \n",
        "(a mounted drive) is available to the operating system as a file system, for reading, writing, or both.\n",
        "'''\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M7ZiwBqlUZWE"
      },
      "source": [
        "#Importing the Required Libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZRXOUoT9SV3J"
      },
      "source": [
        "'''\n",
        "Python library = python library is the collection of modules(= python files) and this python library is the reusable(=able to use again) chunk(= part, section\n",
        ", block) of code we want to include in our python programs or projects to make the implementation easier and faster.\n",
        "\n",
        "os module in python provides functions for interacting with the operating system. os module in Python provides functions for creating and removing a \n",
        "directory(folder), fetching its contents, os module used for changing and identifying the current directory, etc. Basically os module allows source code\n",
        "to communicate (interact) with operating system.\n",
        "\n",
        "Pickle in Python is primarily used in serializing and deserializing a Python object structure. In other words, it's the process of converting a python\n",
        "object into a byte stream to store it in a file/database. Basically pickle library is used to dump (store) all the files of a directory (folder) into \n",
        "single combined file (pickle(.pkz) file) for easy fetching and fast retrieval of data. pickle.dump() is used to create a pickle file, it is used to dump\n",
        "(store) data in a pickle file. pickle.load() is used to load(= start, activate) pickle file\n",
        "\n",
        "Torch (or PyTorch) is an open-source library for machine learning and deep learning. Torch library provides a wide range of algorithms for machine learning and \n",
        "deep learning. The core package of Torch is “torch”, torch package provides a flexible (= adaptable, adjustable, alterable, changeable) N-dimensional array or \n",
        "Tensor, which supports basic routines for indexing, slicing, transposing, type-casting, resizing, sharing storage and cloning. The Tensor also supports \n",
        "mathematical operations like max, min, sum, statistical distributions like uniform, normal and multinomial, and BLAS operations like dot product, matrix-vector \n",
        "multiplication, matrix-matrix multiplication, matrix-vector product and matrix product. The torch package also simplifies object oriented programming and \n",
        "serialization by providing various convenience functions which are used throughout its packages \n",
        "\n",
        "Code for processing data samples can get messy and hard to maintain; we ideally want our dataset code to be decoupled (=separated) from our model training code \n",
        "for better readability and modularity. PyTorch provides two data primitives: torch.utils.data.DataLoader and torch.utils.data.Dataset that allow us to use \n",
        "pre-loaded datasets as well as our own data. DataLoader wraps an iterable around the Dataset to enable easy access to the samples\n",
        "\n",
        "torch.utils.data.Dataset is an abstract class representing a dataset. custom dataset (torch dataset) should inherit Dataset and apply the following methods:\n",
        "(1)__init__ , (2)__len__ , (3)__getitem__ , and (4) an optional argument transform. Dataset stores the samples and their corresponding labels\n",
        "\n",
        "torch.utils.data.DataLoader wraps an iterable around the Dataset to enable easy access to the samples\n",
        "\n",
        "numpy module allows us to work with numerical data. numpy provides an object called numpy array. numpy supports large multi-dimensional arrays & matrices. \n",
        "Basically numpy is a python library used for working with arrays. numpy used for arithmetic operations, statistical operations, bitwise operations, copying \n",
        "and viewing arrays, stacking, matrix operations, linear algebra, mathematical operations, searching, sorting, and counting.\n",
        " \n",
        "as keyword is used as alias (AKA, also called as)\n",
        "'''\n",
        "import os\n",
        "import pickle as pkl\n",
        "import torch\n",
        "from torch.utils.data import Dataset\n",
        "from torch.utils.data import DataLoader\n",
        "import numpy as np"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ODHldSa8o1Or"
      },
      "source": [
        "#Helper Functions\n",
        "Helper Function = A helper function is a function that performs part of the computation (= operation, calculation, estimation, guess) of another function. Helper functions are used to make our programs (= codes) easier to read by giving names to computations. Helper functions also let you reuse computations, just as with functions in general. A helper function is a function we write because we need that particular functionality (= purpose, operation) of a function in multiple places in a code, and because it makes the code more readable. Instead of defining a particular functionality many times, insert(=put, embed) the functionality which we required many times in a helper function, so that we can use that particular functionality as many times we required without defining again"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C6K8Z8Lg7at_"
      },
      "source": [
        "#Loading Data from Pickle file"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lixyHvNclzp-"
      },
      "source": [
        "'''\n",
        "#Load Data from Pickle File\n",
        "\n",
        "def keyword is used to define (= create) a function or method in python\n",
        "\n",
        "using def keyword we defined (= created) a function load_data_from_pfile and passed an argument file_path. Argument is the value to be passed in a function. \n",
        "Here file_path is the location of all pickle files. i.e. file_path = /content/drive/MyDrive/Colab Notebooks \n",
        "\n",
        "with keyword => automatically releases memory after allocation. Whenever we open the file with open() function, it allocates some resources and memory to the \n",
        "file. And we should use close() function to release or delete that memory from the file otherwise errors will come. Sometimes we forget to close() the file and \n",
        "we couldn’t find that we didn’t closed the file, so even the whole code is correct we might get errors and we may not be able to correct it. So it’s better to \n",
        "use “with” keyword along with open() function as “with” keyword automatically releases or deletes memory after process completion\n",
        "\n",
        "The open() function opens a file in text format by default. To open a file in binary format, add 'b' to the mode parameter. Hence the \"rb\" mode opens the file \n",
        "in binary format for reading, while the \"wb\" mode opens the file in binary format for writing. (Note: there are 2 basic mode parameters (r = read mode,w = write \n",
        "mode)). Unlike text files, binary files are not human-readable\n",
        "\n",
        "as = The as keyword is used to create an alias (= aka, also known as, also called). In the code, we created an alias pfile when opening the file_path, and now \n",
        "we can refer to the file_path (or we can access the file_path) by using pfile instead of file_path.\n",
        "\n",
        "Basically The pickle(.pkz) file is created using Python pickle and the dump() method and is loaded (=started, activated) using Python pickle and the load() \n",
        "method. we imported(=send) pickle module as pkl in the code. Therefore pkl.dump() is used to create pickle(.pkz) file and pkl.load() is used to load(=start, \n",
        "activate) pickle file. Here, pfile is the pickle file. Pkl.load(pfile) loads (= starts, activates) the pfile in rb (read binary) mode. We stored the loaded \n",
        "pfile in sample_data variable.\n",
        "\n",
        "return keyword = the return keyword is used to exit (= come out from) a function and return a value. Return sample_data => returns value of sample_data and \n",
        "                 exits a function.\n",
        "\n",
        "'''\n",
        "def load_data_from_pfile(file_path):\n",
        "    with open(file_path, 'rb') as pfile:\n",
        "        sample_data = pkl.load(pfile)\n",
        "    return sample_data"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-Qd5uxQHc4S0"
      },
      "source": [
        "#Pickle(.pkz) Files Path"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dO5zvUCmc17i"
      },
      "source": [
        "'''\n",
        "created pkzfiles_path variable, and assigned(= allocate, allot, set) the path where the pickle(.pkz) files were saved in the pc or in the google drive.\n",
        "'''\n",
        "pkzfiles_path = '/content/drive/MyDrive/Colab Notebooks/'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iwPC-ywwcMaE",
        "outputId": "51be15b7-9b5a-4c83-b643-12870340c284"
      },
      "source": [
        "'''\n",
        "This code cell is just for representation purpose\n",
        "\n",
        "sample_file = example file, created sample_file variable and assigned bearing1_1_train_data.pkz file along with it's path\n",
        "\n",
        "sample_data = created sample_data variable and stored load_data_from_pfile() function and passed sample_file as an argument in the function. argument is the \n",
        "              data or value passed in the function. arguments should be passed inside the parenthesis. load_data_from_pfile() is the function which we created\n",
        "              previously, this functions loads(= activates, starts) the data from pickle files and returns pickle files(data from pickle files). Here, we\n",
        "              passed sample_file as an argument in load_data_from_pfile() function, hence it activates the bearing1_1_train_data.pkz file and returns that \n",
        "              file data\n",
        "\n",
        "printing sample_data['x'].shape and sample_data['y'].shape. \n",
        "shape => https://www.w3schools.com/python/numpy/numpy_array_shape.asp\n",
        "shape in python displays output in tuple format. The shape of an array is the number of elements in each dimension. sample_data['x'].shape gives = no. of horiz \n",
        "accel & vert accel feature elements in array, no. of rows(here, 1st row represents horiz accel feature images, 2nd row represents vert accel feature images), \n",
        "and each feature image size (i,e. no. of pixels(128 x 128)) as an ouptut. sample_data['y'].shape gives = no. of failure probabilities as an output.\n",
        "sample_data['x'] array contains 4 dimensions, and sample_data['y'] array contains 1 dimension.\n",
        "basically, computer stores an image in 0's and 1's.\n",
        "128 x 128 pixels => represents that each feature image is divided into 128 small parts(i.e. each image is represented as 128 rows and 128 columnns with\n",
        "0's and 1's)\n",
        "'''\n",
        "sample_file = pkzfiles_path + 'bearing1_1_train_data.pkz'\n",
        "sample_data = load_data_from_pfile(sample_file)\n",
        "print(sample_data['x'].shape, sample_data['y'].shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(2523, 2, 128, 128) (2523,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "klcn6_II18tk"
      },
      "source": [
        "#Dataset class\n",
        "\n",
        "(Converting numpy array (nd array) into tensor)\n",
        "\n",
        "https://pytorch.org/tutorials/beginner/data_loading_tutorial.html\n",
        "\n",
        "torch.utils.data is a data loading (=activation) class of Torch (or PyTorch)\n",
        "\n",
        "torch.utils.data.Dataset is an abstract class representing(= considered as) a dataset. custom dataset (= if the dataset can be modified (=changed) according to our (user’s) requirements, then dataset is considered as custom dataset, for example we are modifying torch dataset in the code, so torch dataset is the custom dataset) should inherit (= acquire, obtain, derive, get, receive)  Dataset and apply (= use) the following methods:\n",
        "\n",
        "(1) `__len__` so that len(dataset) returns the size of the dataset\n",
        "\n",
        "(2) `__getitem__`to support the indexing such that dataset[i] can be used to get ith sample (data)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jxOU74bN1k0K"
      },
      "source": [
        "'''\n",
        "Python classes and objects => https://www.w3schools.com/python/python_classes.asp\n",
        "A Class is like an object constructor, or a blueprint for creating objects. and objects of the class is used as references to access (= read, get, obtain,\n",
        "examine, acquire) the class properties (properties = variables, arrays, lists, etc.)\n",
        "\n",
        "Python Inheritance => https://www.w3schools.com/python/python_inheritance.asp\n",
        "Inheritance allows us to define a class that inherits (= acquires, obtains, gets, recieves) all the methods and properties from another class.\n",
        "\n",
        "Created PHMDataset class using class keyword. we send Dataset as the parameter in the class. To create a class that inherits the functionality (=methods and \n",
        "properties) from another class, send the parent class as a parameter when creating the child class. Here, parameter Dataset is the parent class and PHMDataset \n",
        "is the child class. Parent class is a class from which a class inherits, parent class also called as base class or super class. Child class is a class which \n",
        "inherits from another class (parent class), child class also called as derived class or sub class. class PHMDatset(Dataset) => this means PHMDataset class \n",
        "inherits the functionality (= methods and propperties) from Dataset class\n",
        "\n",
        "Using def keyword, we defined (= created) __init__ function. __init__ function is used to assign values to class object properties or other operations that are \n",
        "necessary to do when the object is being created. The __init__() function is called automatically every time the class is being used to create a new object. We \n",
        "added __init__() function to the child class, therefore the child class will no longer inherit the parent's __init__() function. The child's __init__() \n",
        "function overrides (= removes) the inheritance of the parent's __init__() function. \n",
        "The self parameter is a reference to the current (= present, ongoing) instance (= object) of the class, and is used to access (= read, obtain, examine, acquire) \n",
        "variables that belongs to the class. (reference in python = a python program accesses (= reads, obtains, gets) data values through (= via) references. A \n",
        "reference is a name that refers to the specific (= certain, particular) location in memory of a value (object))\n",
        "created pfiles list and passed as a parameter in __init__ function and assigned an empty list to the pfiles. __init__ function helps in assigning the values.\n",
        "\n",
        "self.data = {'x': [], 'y': []} => here, self acts as a class (PHMDataset() class) object and data is the dictionary with keys x and y present in PHMDataset() \n",
        "class. x key referred to an empty list. y key also referred to an empty list. since self is the object of PHMDataset() class, we can access data dictionary of\n",
        "PHMDataset class using self parameter i.e. self.data\n",
        "Total 6 different bearing training dataset pickle files are there, therefore used for loop to access one training pickle file at a time from 6 different pickle\n",
        "files\n",
        "underscore can be used as variable name in python. created _data variable, _data is the single leading underscore variable. created _data variable and assigned\n",
        "(= stored) load_data_from_pfile() function which we created previously and passed pfile as a parameter inside the function. this load_data_from_pfile() function\n",
        "returns the data of pickle file as an output.\n",
        "The append() method in python adds a single item to the existing list. It doesn’t return a new list of items but will modify the original list by adding the \n",
        "item to the end of the list. After executing the append method on the list the size of the list increases by one. \n",
        "self.data['x'].append(_data['x']) => _data['x'] contains the list of horiz accel feature values and vert accel feature values, using append method added \n",
        "_data['x'] values to the data dictionary with x key (i.e. data['x']). In data dictionary (data['x']), x key represents an empty list therefore values of _data[x] \n",
        "added at the end of x empty list. self parameter is used to access.\n",
        "Therefore, self.data['x'] contains the list of horiz accel feature values and vert accel feature values of pfile (single pickle file)\n",
        "self.data['y'].append(_data['y']) => _data['y'] contains the list of failure probability values, using append method added _data['y'] values to the data \n",
        "dictionary with y key (i.e. data['y']). In data dictionary (data['y']), y key represents an empty list therefore values of _data[y] added at the end of y empty \n",
        "list. self parameter is used to access.\n",
        "Therefore, self.data['y'] contains the list of failure probability values of pfile (single pickle file)\n",
        "\n",
        "numpy.concatenate() => https://www.tutorialspoint.com/numpy/numpy_concatenate.htm\n",
        "Concatenation refers to joining. The concatenate() function is used to join two or more arrays of the same shape along a specified axis. The function takes the \n",
        "following parameters. general syntax => numpy.concatenate((a1, a2, ...), axis), here a1, a2, ... => represents Sequence (= order, arrangement) of arrays of the \n",
        "same type and axis => represents Axis along which arrays have to be joined. Default is 0.\n",
        "self.data['x'] = np.concatenate(self.data['x']) => joining list of horiz accel feature values and vert accel feature values of all 6 different bearing\n",
        "training pickle files as one. Therefore, self.data['x'] now contains the list of horiz accel feature values and vert accel feature values of all pfiles (6 \n",
        "different bearing training pickle files)\n",
        "self.data['y'] = np.concatenate(self.data['y']) => joining list of failure probability values of all 6 different bearing training pickle files as one. Therefore\n",
        "self.data['y'] now contains the list of failure probability values of all pfiles (6 different bearing training pickle files)\n",
        "np.newaxis is used to add the new axis (= new dimension). Hence, self.data['y'] values are stored in new dimension (= new axis)\n",
        "\n",
        "def keyword is used to create a function. using def keyword defined (= created) __len__() function and passed self parameter as an argument. __len__() function\n",
        "returns the size of the dataset. we passed self parameter in __len__() function, hence we can access all self parameter or self object properties inside the\n",
        "__len__() function.\n",
        "return self.data['x'].shape[0] => here, __len__() function returns the shape of data['x'] in 0th dimension. shape in python returns the no. of elements in each\n",
        "dimension. shape returns the output in tuple format. shape[0] => returns the no. of elements in 0th dimension. in data['x'] in 0th dimension(data['x'].shape[0]) \n",
        "=> total no. of training pickle data files are present i.e. total no. of horiz accel feature elements and vert accel feature elements are present\n",
        "\n",
        "def keyword is used to create a function. using def keyword defined (= created) __getitem__() function and passed self parameter and i as argmuments. arguments\n",
        "are the values that can be passed inside the function. arguments should be passed inside the parenthesis of function name. __getitem__() is used to support the \n",
        "indexing such that dataset[i] can be used to get ith sample (ith data). we passed self parameter in __getitem__() function, hence we can access all self \n",
        "parameter or self object properties inside the __getitem__() function. i is the index\n",
        "the function torch.from_numpy() => converts a numpy array into a tensor \n",
        "numpy array => https://www.javatpoint.com/numpy-array, \n",
        "tensor => https://www.javatpoint.com/pytorch-tensors#:~:text=A%20tensor%20is%20an%20n,computation%20with%20strong%20GPU%20acceleration. \n",
        "tensor = torch.from_numpy(numpy.array) => general syntax to Create a Tensor from numpy array\n",
        "i is used to access the ith data in tensor\n",
        "created sample dictionary with x and y keys. x key contains tensor of self.data['x'][i] i.e. contains the tensor of all horiz accel feature values and vert\n",
        "accel feature values. y key contains tensor of self.data['y'][i] i.e. contains the tensor of all failure probability values\n",
        "return sample => __getitem__() function return the sample dictionary as an output and exits the function\n",
        "\n",
        "'''\n",
        "class PHMDataset(Dataset):\n",
        "    '''\n",
        "    PHM IEEE 2012 Data Challenge Training data set (6 different Mechanical Bearings data)\n",
        "    '''\n",
        "    def __init__(self, pfiles=[]):\n",
        "        self.data = {'x': [], 'y': []}\n",
        "        for pfile in pfiles:\n",
        "            _data = load_data_from_pfile(pfile)\n",
        "            self.data['x'].append(_data['x'])\n",
        "            self.data['y'].append(_data['y'])\n",
        "        self.data['x'] = np.concatenate(self.data['x'])\n",
        "        self.data['y'] = np.concatenate(self.data['y'])[:,np.newaxis]\n",
        "\n",
        "    def __len__(self):\n",
        "        return self.data['x'].shape[0]\n",
        "    \n",
        "    def __getitem__(self, i):\n",
        "        sample = {'x': torch.from_numpy(self.data['x'][i]), 'y': torch.from_numpy(self.data['y'][i])}\n",
        "        return sample"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nCbqJPrs_RiD"
      },
      "source": [
        "(storing 6 different bearings train data pickle files in a single list (train_pfiles))"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QJmHXC-2hr1Y"
      },
      "source": [
        "'''\n",
        "created pkzfiles_path variable, and assigned(= allocate, allot, set) the path where the pickle(.pkz) files were saved in the pc or in the google drive.\n",
        "\n",
        "Python lists => https://www.w3schools.com/python/python_lists.asp (lists are used to store multiple items in a single variable)\n",
        "created train_pfiles list and stored 6 different bearings train data pickle files (bearing1_1_train_data.pkz, bearing1_2_train_data.pkz, \n",
        "bearing2_1_train_data.pkz, bearing2_2_train_data.pkz, bearing3_1_train_data.pkz, bearing3_2_train_data.pkz)\n",
        "\n",
        "'''\n",
        "pkzfiles_path = '/content/drive/MyDrive/Colab Notebooks/'\n",
        "train_pfiles = [pkzfiles_path+'bearing1_1_train_data.pkz', pkzfiles_path+'bearing1_2_train_data.pkz', \\\n",
        "                pkzfiles_path+'bearing2_1_train_data.pkz', pkzfiles_path+'bearing2_2_train_data.pkz', \\\n",
        "                pkzfiles_path+'bearing3_1_train_data.pkz', pkzfiles_path+'bearing3_2_train_data.pkz']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rgcoz17mBOAx"
      },
      "source": [
        "(Total length (= total no. data files present in train_dataset))"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kaTT-2gh9KP1",
        "outputId": "edc67c13-f7eb-4716-9977-6e5e04fdeb42"
      },
      "source": [
        "'''\n",
        "created train_dataset variable and called PHMDataset class and send pfiles list as a parameter inside the class and assigned train_pfiles list to the pfiles.\n",
        "therefore PHMDataset() class now can access to all the train dataset pickle files, hence with train_dataset, can access all train dataset pickle files\n",
        "'''\n",
        "train_dataset = PHMDataset(pfiles=train_pfiles)\n",
        "print(len(train_dataset))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "6783\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WBV8_o3iCJzM"
      },
      "source": [
        "#DataLoader\n",
        "\n",
        "https://pytorch.org/tutorials/beginner/data_loading_tutorial.html\n",
        "\n",
        "Following features are essential when loading(= activating, starting) data for training the models\n",
        "1. Batching data => splitting(= dividing) the dataset into a collection(= batch) of smaller data chunks(= portions, blocks) that are send into the model one at a time. (Basically, batching data means dividing the dataset into batches of data and feed that batches as input to the model one at a time\n",
        "\n",
        "2. Shuffling data => rearranging data or reorganizing data. data shuffling is important because by shuffling the data, one can make sure that each data point creates a separate change on the model, without being affected(= altered, changed) by the same data points before them \n",
        "\n",
        "3. (optionally) load tha data in parallel using multiprocessing workers\n",
        "\n",
        "torch.utils.data.DataLoader is an iterator which provides all these features\n",
        "\n",
        "(iterator in python is an object that is used to iterate(= repeat, loop) over iterable objects like lists, tuples, dictionaries, and sets. the iterator object is initialized using the iter() method. iterator uses the next() method for iteration)\n",
        "\n",
        "(iterable is an object, which one can iterate(=repeat, loop) over. examples: lists, tuples, dictionaries, sets. iterable generates an iterator when passed(=moved) to iter() method. iterator is an object, which is used to iterate over an iterable object using __next__() method)\n",
        "\n",
        "(every iterator is also an iterable, but not every iterable is an iterator)\n",
        "\n",
        "Note: if for loop is used to iterate(= repeat, loop) over the data, then these features(Batching data, shuffling data) are not included which makes data loading somewhat difficult and requires more processing time"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ky2IA8yCDJ2P"
      },
      "source": [
        "sample dataloader(train_dataloader) that loads batch of 4 samples(data files) at a time from train_dataset (without any multiprocessing (= simultaneously process two or more batches of data)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qA3p0TNPB5GV"
      },
      "source": [
        "'''\n",
        "DataLoader is an iterator which is used to iterate over train_dataset\n",
        "train_dataset is our train dataset, train_dataset contains all the train data pickle files\n",
        "batch_size => refers to the no. of samples in each batch\n",
        "shuffle => whether we want the data to be shuffled(= rearranged or reorganized) or not. shuffle is a boolean value, either true or false. shuffle=true \n",
        "represents data to be shuffled. shuffle=false represents data not to be shuffled\n",
        "num_workers => no. of processes needed for loading(= activating, starting) the data\n",
        "'''\n",
        "train_dataloader = DataLoader(train_dataset, batch_size=4, shuffle=True, num_workers=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vhzGviLBh3Pk",
        "outputId": "b7dc2ebe-fdf8-4f6f-abd6-2ee4b170233e"
      },
      "source": [
        "'''\n",
        "representing torch dataset files for 5 samples\n",
        "for loop => https://www.w3schools.com/python/python_for_loops.asp\n",
        "A for loop is used for iterating over a sequence(iterable) (that is either a list, a tuple, a dictionary, a set, or a string)\n",
        "i is the index of for loop. i acts as an iterator of for loop\n",
        "batch = collection of 4 samples(4 datafiles) of learning/training dataset. batch acts as an iterator of for loop\n",
        "the enumerate() function takes an iterable(list or tuple) as an input and returns it as an enumerate object. the enumerate() method adds counter to an iterable\n",
        "and returns the enumerate object. (counter is the variable that automatically increments(=increases) it's value).\n",
        "syntax of enumerate() => enumerate(iterable, start = 0)\n",
        "parameters of enumerate() => enumerate method takes 2 parameters. (1) iterable => is a python object which supports iteration (iterable is an object, which one \n",
        "can iterate(=repeat, loop) over. examples: lists, tuples, dictionaries, sets.), (2) start(optional) => this start parameter is considered as counter in \n",
        "enumerate() method. enumerate() starts counting from this start variable value. since this start is an optional parameter for enumerate(), we can exclude start.\n",
        "if start is excluded, zero(0) is taken as start value.\n",
        "printing indices (each ith index), size of batch['x'] (each batch['x'] contains 4 horiz accel & vert accel feature elements), size of batch['y']\n",
        "(each batch['y']) contains 4 failure probability values)\n",
        "size() method in python => count the no. of elements along a given axis(= dimension) \n",
        "batch['x'].size() => gives 4 horiz accel & vert accel feature elements, no. of rows(here, 1st row represents horiz accel feature images, 2nd row \n",
        "                     represents vert accel feature images), and each feature image size (i,e. no. of pixels(128 x 128)) as an ouptut\n",
        "batch['x'] has 4 dimensions\n",
        "batch['y'].size() => gives 4 failure probability values as an output \n",
        "batch size = 4, that means each batch contains 4 samples.\n",
        "i==4, therefore displays output for 5 samples\n",
        "break => With the break statement we can stop the loop before it has looped(=iterated, repeated) through all the items\n",
        "\n",
        "'''\n",
        "for i, batch in enumerate(train_dataloader):\n",
        "    print(i, batch['x'].size(), batch['y'].size())\n",
        "    if i==4:\n",
        "        break"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0 torch.Size([4, 2, 128, 128]) torch.Size([4, 1])\n",
            "1 torch.Size([4, 2, 128, 128]) torch.Size([4, 1])\n",
            "2 torch.Size([4, 2, 128, 128]) torch.Size([4, 1])\n",
            "3 torch.Size([4, 2, 128, 128]) torch.Size([4, 1])\n",
            "4 torch.Size([4, 2, 128, 128]) torch.Size([4, 1])\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}