SEMI_SUPERVISED_LEARNING_OVERHEAD_MNIST_DATASET.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# **SEMI-SUPERVISED LEARNING ON OVERHEAD MNIST DATASET**"
      ],
      "metadata": {
        "id": "D1lk__33oJkH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install ipython-autotime --quiet\n",
        "%load_ext autotime"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9FWMADFnGt5r",
        "outputId": "113a117c-d2e1-4697-d9f8-23c895d25044"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The autotime extension is already loaded. To reload it, use:\n",
            "  %reload_ext autotime\n",
            "time: 6.95 s (started: 2024-04-21 16:12:31 +00:00)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#  Import Necessary Models"
      ],
      "metadata": {
        "id": "GdCdiiW8pyzh"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aPge5ZKY-cdW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "08d6558b-b663-470c-c4f8-4ea6568acc56"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "time: 993 µs (started: 2024-04-21 16:12:38 +00:00)\n"
          ]
        }
      ],
      "source": [
        "# import necessary models\n",
        "from tqdm import tqdm\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "from tensorflow import keras\n",
        "from sklearn.cluster import KMeans, MiniBatchKMeans\n",
        "from sklearn.linear_model import LogisticRegression"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Load Overhead MNIST data"
      ],
      "metadata": {
        "id": "GbSOMJX_p3BM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Save files as dataframes\n",
        "\n",
        "train = pd.read_csv('train.csv')\n",
        "test = pd.read_csv('test.csv')"
      ],
      "metadata": {
        "id": "W4GfBTgzBDDv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7ccc328b-7aea-40f4-e385-be5931a3d225"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "time: 909 ms (started: 2024-04-21 16:12:38 +00:00)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Data Preprocessing"
      ],
      "metadata": {
        "id": "-stWhaCbqIeX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "X_train = train.iloc[:, 1:].to_numpy()\n",
        "y_train = train.iloc[:, 0].to_numpy()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QvAZ_BOJLKU5",
        "outputId": "7bb751eb-e3d3-45d6-eb51-f1858a74b78a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "time: 2.18 ms (started: 2024-04-21 16:12:39 +00:00)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "X_test = test.iloc[:, 1:].to_numpy()\n",
        "y_test = test.iloc[:, 0].to_numpy()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uSaLia_2Mrnr",
        "outputId": "4abc0af2-df85-43a9-aa82-0b3383ea0931"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "time: 1.88 ms (started: 2024-04-21 16:12:39 +00:00)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "X_train = X_train.astype('float32') / 255.0\n",
        "X_test = X_test.astype('float32') / 255.0"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9rhw1D2_NRa2",
        "outputId": "ab17e2cc-7526-4f07-db48-6feabbd63823"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "time: 53.8 ms (started: 2024-04-21 16:12:39 +00:00)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# A function to print a particular image"
      ],
      "metadata": {
        "id": "CG_sCk0sqOZe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "get_original_array = lambda arr: np.array([int(x * 255) for x in arr], dtype=np.uint8)\n",
        "\n",
        "def print_image(i, data='train'):\n",
        "  if data == 'train':\n",
        "    return get_original_array(X_train[i]).reshape(28,28)\n",
        "  else:\n",
        "    return get_original_array(X_test[i]).reshape(28,28)\n",
        "\n",
        "#example\n",
        "print_image(0)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        },
        "id": "0yg9vVMvM3Nk",
        "outputId": "106e2bc5-5b87-4c73-8bcb-34346b3c97e5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 86,  91,  98,  97,  96,  89,  76,  84,  87,  96,  85,  78,  60,\n",
              "         71,  78,  85,  91,  96, 106,  97,  74, 108, 126, 100,  54,  72,\n",
              "         68,  93],\n",
              "       [ 81,  87,  89, 100, 104,  91, 107,  97,  91,  92,  90,  88,  94,\n",
              "         87,  76,  96,  99,  99,  95,  94, 112, 127, 109, 114, 132,  95,\n",
              "         92, 113],\n",
              "       [ 81,  67,  81,  86,  89,  91,  96,  87,  89,  83, 107, 105,  83,\n",
              "        103, 145, 107,  93, 102, 105,  99,  82,  56,  31,  42,  61,  88,\n",
              "        101, 120],\n",
              "       [ 38,  48,  80,  98,  92,  86,  86,  80,  72,  78, 143, 124,  75,\n",
              "        103, 135,  88,  78,  85,  70,  88,  80,  25,  21,  60,  97,  87,\n",
              "         93, 130],\n",
              "       [ 81,  72,  96,  86,  88,  78,  54,  53,  60,  75,  61,  62,  79,\n",
              "         74,  73,  73,  79, 110, 137, 155, 118,  53,  42,  48,  88,  80,\n",
              "         85, 121],\n",
              "       [ 90, 103,  92,  79,  91,  73, 120, 136,  79,  64,  73,  80,  72,\n",
              "         84,  90,  98,  93, 122, 174, 161, 112, 104, 107,  80,  70, 200,\n",
              "        181, 144],\n",
              "       [ 99, 100,  82,  75,  90,  88, 119, 133,  83,  90,  80,  76,  72,\n",
              "         79,  87,  84,  92,  81,  84,  71,  77,  90,  88,  83,  63, 246,\n",
              "        241, 169],\n",
              "       [107, 104,  75,  87,  84,  84,  77,  70,  78,  76,  77,  76,  76,\n",
              "         77,  76,  81,  79,  74,  85,  87,  84,  81,  98, 102,  49, 206,\n",
              "        255, 178],\n",
              "       [ 99, 102,  76,  84,  81,  76,  83,  79,  81,  76,  72,  75,  79,\n",
              "         77,  77,  82,  78,  80,  83,  78,  83,  81,  90,  96,  54, 168,\n",
              "        250, 188],\n",
              "       [100, 105,  81,  84,  84,  81,  88,  85,  88,  83,  77,  77,  80,\n",
              "         79,  82,  90,  88,  88,  89,  84,  86,  81,  87,  93,  87,  87,\n",
              "        135, 130],\n",
              "       [ 98, 107,  84,  79,  82,  82,  86,  86,  87,  86,  83,  83,  84,\n",
              "         82,  83,  89,  90,  90,  91,  87,  88,  82,  86,  94,  87,  42,\n",
              "         71,  81],\n",
              "       [ 95, 109,  90,  76,  80,  82,  83,  85,  86,  86,  85,  85,  86,\n",
              "         83,  82,  85,  86,  86,  86,  87,  87,  81,  84,  93,  81,  46,\n",
              "         81, 100],\n",
              "       [ 93, 109,  98,  78,  81,  84,  83,  86,  90,  89,  84,  82,  83,\n",
              "         83,  83,  86,  85,  85,  83,  86,  84,  80,  80,  89,  89,  53,\n",
              "         74,  87],\n",
              "       [ 85, 103, 103,  80,  80,  83,  82,  85,  88,  88,  85,  82,  84,\n",
              "         84,  82,  82,  85,  85,  81,  85,  83,  82,  81,  87,  89,  49,\n",
              "         50,  57],\n",
              "       [ 80,  97, 108,  84,  79,  80,  79,  82,  83,  87,  87,  85,  85,\n",
              "         83,  80,  79,  84,  85,  79,  83,  82,  85,  82,  87,  99,  57,\n",
              "         40,  60],\n",
              "       [ 82,  98, 115,  90,  82,  82,  79,  82,  86,  90,  88,  83,  82,\n",
              "         82,  82,  84,  85,  86,  79,  81,  80,  85,  81,  83,  89,  67,\n",
              "         35,  55],\n",
              "       [ 84,  93, 112,  95,  72,  85,  82,  80,  86,  86,  86,  83,  81,\n",
              "         83,  84,  81,  85,  84,  80,  77,  78,  78,  78,  82,  84,  76,\n",
              "         62,  69],\n",
              "       [ 77,  80, 106, 103,  77,  86,  81,  84,  84,  88,  87,  85,  85,\n",
              "         82,  78,  79,  84,  85,  82,  80,  82,  82,  82,  84,  99,  95,\n",
              "         79, 110],\n",
              "       [ 80,  88, 108, 105,  78,  89,  78,  83,  80,  88,  84,  77,  81,\n",
              "         81,  79,  84,  82,  83,  80,  76,  76,  76,  75,  76,  87,  86,\n",
              "         75,  70],\n",
              "       [ 65,  86, 103, 104,  81,  97,  79,  75,  80,  90,  90,  83,  81,\n",
              "         82,  79,  77,  80,  83,  79,  74,  74,  76,  76,  76,  87,  82,\n",
              "         89,  33],\n",
              "       [ 68,  73,  89, 106,  80,  91,  86,  80,  79,  79,  81,  80,  74,\n",
              "         72,  74,  72,  73,  76,  75,  74,  76,  79,  79,  79,  82,  72,\n",
              "         84,  33],\n",
              "       [ 79,  67,  76, 111,  84,  76,  85,  82,  77,  73,  74,  76,  71,\n",
              "         67,  69,  72,  72,  74,  74,  74,  75,  72,  69,  69,  80,  76,\n",
              "         69,  46],\n",
              "       [ 73,  90,  81, 106,  99,  75,  76,  79,  77,  88,  82,  73,  78,\n",
              "         81,  77,  79,  78,  78,  76,  76,  75,  70,  67,  69,  77,  82,\n",
              "         73,  60],\n",
              "       [ 96, 152, 105,  88, 105,  82,  75,  90, 190, 204, 160, 101,  92,\n",
              "         95,  82,  79,  74,  71,  69,  71,  74,  71,  73,  79,  73,  75,\n",
              "         87,  72],\n",
              "       [ 89, 107,  93,  80,  92,  95,  87,  80, 113, 106,  93,  87,  81,\n",
              "         77,  78,  77,  71,  83,  72,  71,  79,  82,  78,  54,  57,  52,\n",
              "         59,  79],\n",
              "       [ 50,  67,  63,  67,  86,  88,  84,  87,  75,  80,  83,  83,  75,\n",
              "         69,  72,  77,  84, 100,  86,  73,  71,  73,  67,  42,  27,  28,\n",
              "         51,  61],\n",
              "       [ 59,  86,  82,  73,  88,  96,  88,  80,  89,  87,  85,  82,  78,\n",
              "         72,  69,  67,  65,  77,  73,  75,  78,  78,  83,  79,  52,  43,\n",
              "         66,  56],\n",
              "       [101, 116, 103,  92,  98,  92,  79,  75,  62,  51,  49,  53,  71,\n",
              "         78,  78,  74,  74,  78,  76,  81,  80,  72,  77,  83,  71,  44,\n",
              "         53,  33]], dtype=uint8)"
            ],
            "text/html": [
              "<style>\n",
              "      .ndarray_repr .ndarray_raw_data {\n",
              "        display: none;\n",
              "      }\n",
              "      .ndarray_repr.show_array .ndarray_raw_data {\n",
              "        display: block;\n",
              "      }\n",
              "      .ndarray_repr.show_array .ndarray_image_preview {\n",
              "        display: none;\n",
              "      }\n",
              "      </style>\n",
              "      <div id=\"id-86023dd2-59e4-4743-935f-71ce03cfe991\" class=\"ndarray_repr\"><pre>ndarray (28, 28) <button style=\"padding: 0 2px;\">show data</button></pre><img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAACYUlEQVR4nAXBy24cRRQA0Puoqu7p6ZnxzJjYiRdRkCOhLJBQNhFL/iAbJPZ8CT+UT8gqyjqQh7EFiEgODorHjNv9qMe9l3PwR1+Zxa7mWR9386oKvhk/f7359Oum3MfngeauOxgSqobYHzIU26x2fnMhXzn4shSmfqR0rP8efn9+Owvp46d/5tuTRUXvrq3lPMSbfLqdYLje3/zR69/D5tSmBp+PxwMNV1atbnGdDcLDk83Fyz58M/lTN7vXxzb/cPB631ZTG+rcPu7SU77c/XJ94WYyFFuZuAdDgcy39YfzLiyU47f5ztUS2wTvYx0RnPa+y/MDGdfuz7PvZjQpikxhxpCNAdGCjgZT8ue5ccgeGDwxISBxrYBIKvLoxV/ZKXFmVDQOgIpGgIxoCL/91zihMCEJARCgWUEANFNLrtmSQk1IgI5NBBDNxMRKqavYuSl5MjNTQEDQwqBkZErl6oZy8s5EJBsxmoqBqjFCUw9MhFVgAmBGFSOPCKpgZWhyJsriB2aFomBIAopFo1ienSxdLIvaozl1RiQZDQkIgFZzfui4JkOeSJxEB+gVtQAYspUz2ms2rnECCgEQooKpAgKU5YXjjteWKrSeLClpYQNigDwQ0Dx13u2NNMUiOYOpgcVhULdfuHQ05tU4MqzSWDWFDBRMlSTI0h09e7NbV3H+8/Ltq5E7KaLEGELSpeFPm9uu5Sk+il1KBS21vdVcrjhXh+73dXGBpDr3DCmxXxxwN1G62+PRzs22ZaR6mpaqwiz5Lhby0LSX42pwT1ZdH9JYdr26TQUFkL1NFraX3Zf/AQw1eqqo7cJ/AAAAAElFTkSuQmCC\" class=\"ndarray_image_preview\" /><pre class=\"ndarray_raw_data\">array([[ 86,  91,  98,  97,  96,  89,  76,  84,  87,  96,  85,  78,  60,\n",
              "         71,  78,  85,  91,  96, 106,  97,  74, 108, 126, 100,  54,  72,\n",
              "         68,  93],\n",
              "       [ 81,  87,  89, 100, 104,  91, 107,  97,  91,  92,  90,  88,  94,\n",
              "         87,  76,  96,  99,  99,  95,  94, 112, 127, 109, 114, 132,  95,\n",
              "         92, 113],\n",
              "       [ 81,  67,  81,  86,  89,  91,  96,  87,  89,  83, 107, 105,  83,\n",
              "        103, 145, 107,  93, 102, 105,  99,  82,  56,  31,  42,  61,  88,\n",
              "        101, 120],\n",
              "       [ 38,  48,  80,  98,  92,  86,  86,  80,  72,  78, 143, 124,  75,\n",
              "        103, 135,  88,  78,  85,  70,  88,  80,  25,  21,  60,  97,  87,\n",
              "         93, 130],\n",
              "       [ 81,  72,  96,  86,  88,  78,  54,  53,  60,  75,  61,  62,  79,\n",
              "         74,  73,  73,  79, 110, 137, 155, 118,  53,  42,  48,  88,  80,\n",
              "         85, 121],\n",
              "       [ 90, 103,  92,  79,  91,  73, 120, 136,  79,  64,  73,  80,  72,\n",
              "         84,  90,  98,  93, 122, 174, 161, 112, 104, 107,  80,  70, 200,\n",
              "        181, 144],\n",
              "       [ 99, 100,  82,  75,  90,  88, 119, 133,  83,  90,  80,  76,  72,\n",
              "         79,  87,  84,  92,  81,  84,  71,  77,  90,  88,  83,  63, 246,\n",
              "        241, 169],\n",
              "       [107, 104,  75,  87,  84,  84,  77,  70,  78,  76,  77,  76,  76,\n",
              "         77,  76,  81,  79,  74,  85,  87,  84,  81,  98, 102,  49, 206,\n",
              "        255, 178],\n",
              "       [ 99, 102,  76,  84,  81,  76,  83,  79,  81,  76,  72,  75,  79,\n",
              "         77,  77,  82,  78,  80,  83,  78,  83,  81,  90,  96,  54, 168,\n",
              "        250, 188],\n",
              "       [100, 105,  81,  84,  84,  81,  88,  85,  88,  83,  77,  77,  80,\n",
              "         79,  82,  90,  88,  88,  89,  84,  86,  81,  87,  93,  87,  87,\n",
              "        135, 130],\n",
              "       [ 98, 107,  84,  79,  82,  82,  86,  86,  87,  86,  83,  83,  84,\n",
              "         82,  83,  89,  90,  90,  91,  87,  88,  82,  86,  94,  87,  42,\n",
              "         71,  81],\n",
              "       [ 95, 109,  90,  76,  80,  82,  83,  85,  86,  86,  85,  85,  86,\n",
              "         83,  82,  85,  86,  86,  86,  87,  87,  81,  84,  93,  81,  46,\n",
              "         81, 100],\n",
              "       [ 93, 109,  98,  78,  81,  84,  83,  86,  90,  89,  84,  82,  83,\n",
              "         83,  83,  86,  85,  85,  83,  86,  84,  80,  80,  89,  89,  53,\n",
              "         74,  87],\n",
              "       [ 85, 103, 103,  80,  80,  83,  82,  85,  88,  88,  85,  82,  84,\n",
              "         84,  82,  82,  85,  85,  81,  85,  83,  82,  81,  87,  89,  49,\n",
              "         50,  57],\n",
              "       [ 80,  97, 108,  84,  79,  80,  79,  82,  83,  87,  87,  85,  85,\n",
              "         83,  80,  79,  84,  85,  79,  83,  82,  85,  82,  87,  99,  57,\n",
              "         40,  60],\n",
              "       [ 82,  98, 115,  90,  82,  82,  79,  82,  86,  90,  88,  83,  82,\n",
              "         82,  82,  84,  85,  86,  79,  81,  80,  85,  81,  83,  89,  67,\n",
              "         35,  55],\n",
              "       [ 84,  93, 112,  95,  72,  85,  82,  80,  86,  86,  86,  83,  81,\n",
              "         83,  84,  81,  85,  84,  80,  77,  78,  78,  78,  82,  84,  76,\n",
              "         62,  69],\n",
              "       [ 77,  80, 106, 103,  77,  86,  81,  84,  84,  88,  87,  85,  85,\n",
              "         82,  78,  79,  84,  85,  82,  80,  82,  82,  82,  84,  99,  95,\n",
              "         79, 110],\n",
              "       [ 80,  88, 108, 105,  78,  89,  78,  83,  80,  88,  84,  77,  81,\n",
              "         81,  79,  84,  82,  83,  80,  76,  76,  76,  75,  76,  87,  86,\n",
              "         75,  70],\n",
              "       [ 65,  86, 103, 104,  81,  97,  79,  75,  80,  90,  90,  83,  81,\n",
              "         82,  79,  77,  80,  83,  79,  74,  74,  76,  76,  76,  87,  82,\n",
              "         89,  33],\n",
              "       [ 68,  73,  89, 106,  80,  91,  86,  80,  79,  79,  81,  80,  74,\n",
              "         72,  74,  72,  73,  76,  75,  74,  76,  79,  79,  79,  82,  72,\n",
              "         84,  33],\n",
              "       [ 79,  67,  76, 111,  84,  76,  85,  82,  77,  73,  74,  76,  71,\n",
              "         67,  69,  72,  72,  74,  74,  74,  75,  72,  69,  69,  80,  76,\n",
              "         69,  46],\n",
              "       [ 73,  90,  81, 106,  99,  75,  76,  79,  77,  88,  82,  73,  78,\n",
              "         81,  77,  79,  78,  78,  76,  76,  75,  70,  67,  69,  77,  82,\n",
              "         73,  60],\n",
              "       [ 96, 152, 105,  88, 105,  82,  75,  90, 190, 204, 160, 101,  92,\n",
              "         95,  82,  79,  74,  71,  69,  71,  74,  71,  73,  79,  73,  75,\n",
              "         87,  72],\n",
              "       [ 89, 107,  93,  80,  92,  95,  87,  80, 113, 106,  93,  87,  81,\n",
              "         77,  78,  77,  71,  83,  72,  71,  79,  82,  78,  54,  57,  52,\n",
              "         59,  79],\n",
              "       [ 50,  67,  63,  67,  86,  88,  84,  87,  75,  80,  83,  83,  75,\n",
              "         69,  72,  77,  84, 100,  86,  73,  71,  73,  67,  42,  27,  28,\n",
              "         51,  61],\n",
              "       [ 59,  86,  82,  73,  88,  96,  88,  80,  89,  87,  85,  82,  78,\n",
              "         72,  69,  67,  65,  77,  73,  75,  78,  78,  83,  79,  52,  43,\n",
              "         66,  56],\n",
              "       [101, 116, 103,  92,  98,  92,  79,  75,  62,  51,  49,  53,  71,\n",
              "         78,  78,  74,  74,  78,  76,  81,  80,  72,  77,  83,  71,  44,\n",
              "         53,  33]], dtype=uint8)</pre></div><script>\n",
              "      (() => {\n",
              "      const titles = ['show data', 'hide data'];\n",
              "      let index = 0\n",
              "      document.querySelector('#id-86023dd2-59e4-4743-935f-71ce03cfe991 button').onclick = (e) => {\n",
              "        document.querySelector('#id-86023dd2-59e4-4743-935f-71ce03cfe991').classList.toggle('show_array');\n",
              "        index = (++index) % 2;\n",
              "        document.querySelector('#id-86023dd2-59e4-4743-935f-71ce03cfe991 button').textContent = titles[index];\n",
              "        e.preventDefault();\n",
              "        e.stopPropagation();\n",
              "      }\n",
              "      })();\n",
              "    </script>"
            ]
          },
          "metadata": {},
          "execution_count": 96
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "time: 15.5 ms (started: 2024-04-21 16:12:44 +00:00)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Standard Logistic Regression"
      ],
      "metadata": {
        "id": "htKEZYydqVcQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Standard logistic model has a very high accuracy for the dataset.\n",
        "\n",
        "logistic_regressor_clf = LogisticRegression(random_state=69, max_iter=10000).fit(X_train, y_train)\n",
        "logistic_regressor_clf.score(X_test, y_test)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZwV7t4mdQu4N",
        "outputId": "25910e8f-af62-4fc5-f63a-ae2abc67c60d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.4676056338028169"
            ]
          },
          "metadata": {},
          "execution_count": 97
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "time: 1min 22s (started: 2024-04-21 16:12:54 +00:00)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The standard logistic regression model gives an accuracy score of 0.4676 which is pretty less."
      ],
      "metadata": {
        "id": "4GVLWphcn1Vh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Now, we will perform semi-supervised learning on the Fashion MNIST dataset and compare the accuracy scores.**\n",
        "\n",
        "1.   First, we will take 40 random samples and do the logistic regression.\n",
        "2.   Next, we will use k-means clustering to get 40 clusters and take the nearest points as centroids. Then we will use those 40 data-points to train the logistic regression classifier.\n",
        "3. Then we will propagate the labelling to the whole cluster and do the same.\n",
        "4. Repeat the same, but only propagate to the 20% of the dataset now.\n",
        "\n"
      ],
      "metadata": {
        "id": "Qd4QsRXeQ1lo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Perform Logistic Regression On Randomly Selected 40 Samples"
      ],
      "metadata": {
        "id": "TRY52W7zsHt6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Randomly select 40 samples from the training data\n",
        "idx = np.random.choice(X_train.shape[0], 40, replace=False)\n",
        "\n",
        "# Get the selected samples and their corresponding labels\n",
        "X_sample = X_train[idx]\n",
        "y_sample = y_train[idx]\n",
        "\n",
        "logistic_regressor_clf = LogisticRegression(random_state=69, max_iter=10000).fit(X_sample, y_sample)\n",
        "logistic_regressor_clf.score(X_test, y_test)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wzBpZVNmQ7gu",
        "outputId": "2ee8b926-e1ec-4610-8e67-cb3c4c9a1cc5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.20093896713615023"
            ]
          },
          "metadata": {},
          "execution_count": 145
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "time: 975 ms (started: 2024-04-21 16:29:25 +00:00)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note that a pretty low accuracy score of 0.2009 is obtained."
      ],
      "metadata": {
        "id": "DNYQiHxXnjJ2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Perform Logistic Regression On 40 Cluster Representative Points"
      ],
      "metadata": {
        "id": "r-kQ2eh1slM3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Use KMeans to select 40 representative samples and then train the classifier\n",
        "\n",
        "kmeans = MiniBatchKMeans(n_clusters=40, random_state=42)\n",
        "kmeans.fit(X_train)\n",
        "\n",
        "cluster_centers = kmeans.cluster_centers_\n",
        "\n",
        "nearest_points_to_cluster = [-1]*40\n",
        "distance_from_nearest_points_to_cluster = [10e9]*40\n",
        "\n",
        "# Select those representative points\n",
        "for i, cluster_center in enumerate(cluster_centers):\n",
        "  for j, data_point in enumerate(X_train):\n",
        "    distance = np.linalg.norm(cluster_center-data_point)\n",
        "    if distance < distance_from_nearest_points_to_cluster[i]:\n",
        "      distance_from_nearest_points_to_cluster[i] = distance\n",
        "      nearest_points_to_cluster[i] = j\n",
        "\n",
        "X_sample_ = X_train[nearest_points_to_cluster]\n",
        "y_sample_ = y_train[nearest_points_to_cluster]\n",
        "\n",
        "# Train and test the logistic regression classifier\n",
        "logistic_regressor_clf = LogisticRegression(random_state=69, max_iter=10000).fit(X_sample_, y_sample_)\n",
        "logistic_regressor_clf.score(X_test, y_test)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ldu6VqfuRG6V",
        "outputId": "8acc282f-b222-428e-ba5c-4ca8666ecbb4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 3 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.2516431924882629"
            ]
          },
          "metadata": {},
          "execution_count": 151
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "time: 18.4 s (started: 2024-04-21 16:34:36 +00:00)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Perform Logistic Regression On 40 Cluster Representative Points After Propagating the Labels"
      ],
      "metadata": {
        "id": "J38WoH47sttW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Use KMeans to select 40 representative samples, propagate the labels and then train the classifier\n",
        "\n",
        "kmeans = MiniBatchKMeans(n_clusters=40, random_state=42)\n",
        "kmeans.fit(X_train)\n",
        "\n",
        "cluster_centers = kmeans.cluster_centers_\n",
        "datapoints_labels = kmeans.labels_\n",
        "\n",
        "# Nearest points of each cluster\n",
        "nearest_points_to_cluster = [-1]*40\n",
        "distance_from_nearest_points_to_cluster = [10e9]*40\n",
        "\n",
        "# Mapping between labels and nearest points\n",
        "nearest_point_of_a_label = [-1]*40\n",
        "\n",
        "# Select those representative points\n",
        "for i, cluster_center in enumerate(cluster_centers):\n",
        "  for j, data_point in enumerate(X_train):\n",
        "    distance = np.linalg.norm(cluster_center-data_point)\n",
        "    if distance < distance_from_nearest_points_to_cluster[i]:\n",
        "      distance_from_nearest_points_to_cluster[i] = distance\n",
        "      nearest_points_to_cluster[i] = j\n",
        "      nearest_point_of_a_label[datapoints_labels[j]] = j\n",
        "\n",
        "y_propagated = [-1]*(X_train.shape[0])\n",
        "for i, datapoint in enumerate(X_train):\n",
        "  cluster_label = datapoints_labels[i]\n",
        "  y_propagated[i] = y_train[nearest_point_of_a_label[cluster_label]]\n",
        "\n",
        "# Train and test the logistic regression classifier\n",
        "logistic_regressor_clf = LogisticRegression(random_state=69, max_iter=10000).fit(X_train, y_propagated)\n",
        "logistic_regressor_clf.score(X_test, y_test)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0H1dQqfSRoip",
        "outputId": "d2b59f18-a147-4f0c-8129-6ed3ebd8d803"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 3 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.27417840375586855"
            ]
          },
          "metadata": {},
          "execution_count": 147
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "time: 2min 54s (started: 2024-04-21 16:30:19 +00:00)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Perform Logistic Regression On 40 Cluster Representative Points After Propagating the Labels Partially"
      ],
      "metadata": {
        "id": "mmMH08W2swEq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Use KMeans to select 40 representative samples, propagate the labels partially and then train the classifier\n",
        "\n",
        "kmeans = MiniBatchKMeans(n_clusters=40, random_state=42)\n",
        "kmeans.fit(X_train)\n",
        "\n",
        "cluster_centers = kmeans.cluster_centers_\n",
        "datapoints_labels = kmeans.labels_\n",
        "\n",
        "# Nearest points of each cluster\n",
        "nearest_points_to_cluster = [-1]*40\n",
        "distance_from_nearest_points_to_cluster = [10e9]*40\n",
        "\n",
        "# Mapping between labels and nearest points\n",
        "nearest_point_of_a_label = [-1]*40\n",
        "\n",
        "# Select those representative points\n",
        "for i in tqdm(range(len(cluster_centers))):\n",
        "  for j, data_point in enumerate(X_train):\n",
        "    distance = np.linalg.norm(cluster_centers[i]-data_point)\n",
        "    if distance < distance_from_nearest_points_to_cluster[i]:\n",
        "      distance_from_nearest_points_to_cluster[i] = distance\n",
        "      nearest_points_to_cluster[i] = j\n",
        "      nearest_point_of_a_label[datapoints_labels[j]] = j\n",
        "\n",
        "closest_points_from_centroid = np.array([[1]*40]*40)\n",
        "\n",
        "for i in tqdm(range(len(nearest_points_to_cluster))):\n",
        "  distance_point_pair = np.empty((0, 2))\n",
        "\n",
        "  for j, data_point in enumerate(X_train):\n",
        "\n",
        "    dist = np.linalg.norm(data_point-X_train[nearest_points_to_cluster[i]])\n",
        "    to_append = np.array([dist, j])\n",
        "    distance_point_pair = np.concatenate((distance_point_pair, [to_append]))\n",
        "\n",
        "  sorted_indices = np.argsort(distance_point_pair[:, 0])\n",
        "  distance_point_pair = (distance_point_pair[sorted_indices])[:40]\n",
        "\n",
        "  closest_points_idx = distance_point_pair[:, 1]\n",
        "  closest_points_from_centroid[i] = closest_points_idx\n",
        "\n",
        "X_sample = []\n",
        "y_propagated = []\n",
        "\n",
        "for i, id in enumerate(closest_points_from_centroid):\n",
        "  for j in id:\n",
        "    X_sample.append(X_train[j])\n",
        "    y_propagated.append(y_train[nearest_points_to_cluster[i]])\n",
        "\n",
        "# Train and test the logistic regression classifier\n",
        "logistic_regressor_clf = LogisticRegression(random_state=37, max_iter=10000).fit(X_sample, y_propagated)\n",
        "logistic_regressor_clf.score(X_test, y_test)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xoG15KQ8R2-2",
        "outputId": "a867bc8a-4af8-4efe-f0e1-f16e4cdbb66b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 3 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning\n",
            "  warnings.warn(\n",
            "100%|██████████| 40/40 [00:06<00:00,  5.74it/s]\n",
            "100%|██████████| 40/40 [00:10<00:00,  3.97it/s]\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.26291079812206575"
            ]
          },
          "metadata": {},
          "execution_count": 152
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "time: 46.7 s (started: 2024-04-21 16:51:02 +00:00)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Report:\n",
        "\n",
        "- NOTE THAT THE NUMBER OF CLUSTERS TO BE MADE WAS CHOSEN BASED ON THE SIZE OF THE DATASET.\n",
        "- Standard logistic model has a moderate accuracy for the dataset. (`0.4676056338028169`)\n",
        "- Now, we choose randomly 40 datapoints (out of 60000) and use standard logistic regression to get (`0.20093896713615023`). The performance was poor.\n",
        "- Then, instead of choosing randomly, we cluster the dataset into 40 clusters and used the nearest points to the centroids as the representative points to train the standard logistic model. The accuracy was `0.2516431924882629`. The accuracy is greatly improved as we are now using the best of its kind to train.\n",
        "- Next, we propagate the labels to every cluster point to get the accuracy as `0.27417840375586855`. Performance improves as there are more training points.\n",
        "- Now, instead of propagating to all the points, we propagate to the 20% data (`0.26291079812206575`). Surprisingly, the performance is still moderate implying labelling small dataset accordingly (semi-supervised learning) performs greatly."
      ],
      "metadata": {
        "id": "bvboMWZjtYzk"
      }
    }
  ]
}