Text_Data_Preprocessing.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/RakeshSharma21/Sessions_Notebook/blob/main/Text_Data_Preprocessing.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SQc1X3PvJN5T"
      },
      "source": [
        "# \"Text Preprocessing\"\n",
        "> \"Vairous preprocessing steps required to clean and prepare your data in NLP\"\n",
        "\n",
        "- toc: true\n",
        "- branch: master\n",
        "- badges: true\n",
        "- comments: true\n",
        "- categories: [Text Preprocessing, Natural Language Processing]\n",
        "- image: images/posts/text_preprocessing.jpg\n",
        "- hide: false\n",
        "- search_exclude: false"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Anxi7YqvJN5o"
      },
      "source": [
        "In any machine learning task, cleaning and pre-processing of the data is a very important step. The better we can represent our data, the better the model training and prediction can be expected.\n",
        "\n",
        "Specially in the domain of Natural Language Processing (NLP) the data is unstructured. It become crucial to clean and properly format it based on the task at hand. There are various pre-processing steps that can be performed but not necessary to perform all. These steps should be applied based on the problem statement.\n",
        "\n",
        "Example: Sentiment analysis on twitter data can required to remove hashtags, emoticons, etc. but this may not be the case if we are doing the same analysis on customer feedback data.\n",
        "\n",
        "Here we are using the twitter_sample dataset from the nltk library."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install demoji\n",
        "!pip install emoji\n",
        "!pip install contractions\n",
        "!pip install unidecode\n",
        "!pip install num2words\n",
        "# !pip install spellchecker\n",
        "!pip install pyspellchecker"
      ],
      "metadata": {
        "id": "Aa59aee2JtDy",
        "outputId": "de013641-3ec0-4a54-99a7-5acfc1acee9c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting demoji\n",
            "  Downloading demoji-1.1.0-py3-none-any.whl (42 kB)\n",
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/42.9 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m42.9/42.9 kB\u001b[0m \u001b[31m1.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: demoji\n",
            "Successfully installed demoji-1.1.0\n",
            "Collecting emoji\n",
            "  Downloading emoji-2.10.1-py2.py3-none-any.whl (421 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m421.5/421.5 kB\u001b[0m \u001b[31m1.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: emoji\n",
            "Successfully installed emoji-2.10.1\n",
            "Collecting contractions\n",
            "  Downloading contractions-0.1.73-py2.py3-none-any.whl (8.7 kB)\n",
            "Collecting textsearch>=0.0.21 (from contractions)\n",
            "  Downloading textsearch-0.0.24-py2.py3-none-any.whl (7.6 kB)\n",
            "Collecting anyascii (from textsearch>=0.0.21->contractions)\n",
            "  Downloading anyascii-0.3.2-py3-none-any.whl (289 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m289.9/289.9 kB\u001b[0m \u001b[31m7.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pyahocorasick (from textsearch>=0.0.21->contractions)\n",
            "  Downloading pyahocorasick-2.0.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (110 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m110.8/110.8 kB\u001b[0m \u001b[31m15.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: pyahocorasick, anyascii, textsearch, contractions\n",
            "Successfully installed anyascii-0.3.2 contractions-0.1.73 pyahocorasick-2.0.0 textsearch-0.0.24\n",
            "Collecting unidecode\n",
            "  Downloading Unidecode-1.3.8-py3-none-any.whl (235 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m235.5/235.5 kB\u001b[0m \u001b[31m4.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: unidecode\n",
            "Successfully installed unidecode-1.3.8\n",
            "Collecting num2words\n",
            "  Downloading num2words-0.5.13-py3-none-any.whl (143 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m143.3/143.3 kB\u001b[0m \u001b[31m3.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting docopt>=0.6.2 (from num2words)\n",
            "  Downloading docopt-0.6.2.tar.gz (25 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Building wheels for collected packages: docopt\n",
            "  Building wheel for docopt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13706 sha256=41d72587ce83dad65aff75064939053f5025a287e11f2974e7bc72bb1a1c78bc\n",
            "  Stored in directory: /root/.cache/pip/wheels/fc/ab/d4/5da2067ac95b36618c629a5f93f809425700506f72c9732fac\n",
            "Successfully built docopt\n",
            "Installing collected packages: docopt, num2words\n",
            "Successfully installed docopt-0.6.2 num2words-0.5.13\n",
            "Collecting pyspellchecker\n",
            "  Downloading pyspellchecker-0.8.1-py3-none-any.whl (6.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.8/6.8 MB\u001b[0m \u001b[31m55.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: pyspellchecker\n",
            "Successfully installed pyspellchecker-0.8.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 117,
      "metadata": {
        "id": "xy3bKU7EJN5q",
        "outputId": "2b2482ae-a8e1-48ab-a8e8-b1b34805d261",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " - negative_tweets.json\n",
            " - positive_tweets.json\n",
            " - tweets.20150430-223406.json\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package twitter_samples to /root/nltk_data...\n",
            "[nltk_data]   Package twitter_samples is already up-to-date!\n"
          ]
        }
      ],
      "source": [
        "#collapse-hide\n",
        "# Import libraries and load the data\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import re\n",
        "import nltk\n",
        "import string\n",
        "import demoji\n",
        "import contractions\n",
        "import unidecode\n",
        "from num2words import num2words\n",
        "from nltk.corpus import twitter_samples\n",
        "from nltk.corpus import stopwords\n",
        "from nltk.stem.porter import PorterStemmer #driving ==> driv\n",
        "from nltk.stem.snowball import SnowballStemmer\n",
        "from nltk.stem import WordNetLemmatizer ## driving ==> drive\n",
        "from bs4 import BeautifulSoup\n",
        "from spellchecker import SpellChecker\n",
        "\n",
        "pd.options.display.max_columns=None\n",
        "pd.options.display.max_rows=None\n",
        "pd.options.display.max_colwidth=None\n",
        "\n",
        "nltk.download('twitter_samples') # Download the dataset\n",
        "\n",
        "# We are going to use the Negative and Positive Tweets file which each contains 5000 tweets.\n",
        "for name in twitter_samples.fileids():\n",
        "    print(f' - {name}')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 118,
      "metadata": {
        "id": "YX7DZbYoJN5x"
      },
      "outputs": [],
      "source": [
        "#collapse-hide\n",
        "\n",
        "# Load the negative tweets file and assign label as 0 for negative\n",
        "negative_tweets = twitter_samples.strings(\"negative_tweets.json\")\n",
        "df_neg = pd.DataFrame(negative_tweets, columns=['text'])\n",
        "df_neg['label'] = 0\n",
        "\n",
        "# Load the positive tweets file and assign label as 1 for positive\n",
        "positive_tweets = twitter_samples.strings(\"positive_tweets.json\")\n",
        "df_pos = pd.DataFrame(positive_tweets, columns=['text'])\n",
        "df_pos['label'] = 1\n",
        "\n",
        "# df = pd.concat([df_pos, df_neg]) # Concatenate both the files\n",
        "# df = df.sample(frac=1).reset_index(drop=True) # Shuffle the data to mix negative and positive tweets"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_neg.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GHv_0xf-8L7u",
        "outputId": "448444c7-8d96-48aa-849b-2f21f1ce3efd"
      },
      "execution_count": 120,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(5000, 2)"
            ]
          },
          "metadata": {},
          "execution_count": 120
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_neg.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "NVPFt4Y2dNeo",
        "outputId": "ee4052f0-2a86-47ce-9619-6f11bdf529ff"
      },
      "execution_count": 121,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                    text  \\\n",
              "0                                                                    hopeless for tmr :(   \n",
              "1  Everything in the kids section of IKEA is so cute. Shame I'm nearly 19 in 2 months :(   \n",
              "2                                 @Hegelbon That heart sliding into the waste basket. :(   \n",
              "3                       “@ketchBurning: I hate Japanese call him \"bani\" :( :(”\\n\\nMe too   \n",
              "4                                               Dang starting next week I have \"work\" :(   \n",
              "\n",
              "   label  \n",
              "0      0  \n",
              "1      0  \n",
              "2      0  \n",
              "3      0  \n",
              "4      0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-91825b38-3bcc-4ca1-ace8-54f2b3d7cb68\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>hopeless for tmr :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Everything in the kids section of IKEA is so cute. Shame I'm nearly 19 in 2 months :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>@Hegelbon That heart sliding into the waste basket. :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>“@ketchBurning: I hate Japanese call him \"bani\" :( :(”\\n\\nMe too</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Dang starting next week I have \"work\" :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-91825b38-3bcc-4ca1-ace8-54f2b3d7cb68')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-91825b38-3bcc-4ca1-ace8-54f2b3d7cb68 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-91825b38-3bcc-4ca1-ace8-54f2b3d7cb68');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-4da567e3-e9d0-4c7f-81a5-8b400b156473\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-4da567e3-e9d0-4c7f-81a5-8b400b156473')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-4da567e3-e9d0-4c7f-81a5-8b400b156473 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df_neg",
              "summary": "{\n  \"name\": \"df_neg\",\n  \"rows\": 5000,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5000,\n        \"samples\": [\n          \"im so tempted to stream to screenshot sjkaos :(((( no\",\n          \"@yearsandyears I wanted to go but I don't live near any of these places at all :((\",\n          \"I wish my Parents would live forever. :(\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 0,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 121
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_pos.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hRXTmeKLogv1",
        "outputId": "908ab4af-5d20-450f-874e-530b54bd56d3"
      },
      "execution_count": 122,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(5000, 2)"
            ]
          },
          "metadata": {},
          "execution_count": 122
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_pos.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "FveSWyQ4omgO",
        "outputId": "1281c13e-ec33-4ba0-f152-d942c34292df"
      },
      "execution_count": 123,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                             text  \\\n",
              "0                 #FollowFriday @France_Inte @PKuchly57 @Milipol_Paris for being top engaged members in my community this week :)   \n",
              "1  @Lamb2ja Hey James! How odd :/ Please call our Contact Centre on 02392441234 and we will be able to assist you :) Many thanks!   \n",
              "2                     @DespiteOfficial we had a listen last night :) As You Bleed is an amazing track. When are you in Scotland?!   \n",
              "3                                                                                                            @97sides CONGRATS :)   \n",
              "4                      yeaaaah yippppy!!!  my accnt verified rqst has succeed got a blue tick mark on my fb profile :) in 15 days   \n",
              "\n",
              "   label  \n",
              "0      1  \n",
              "1      1  \n",
              "2      1  \n",
              "3      1  \n",
              "4      1  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-f9a18c29-7b5a-4d5d-8e4d-ab7239c95be4\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>#FollowFriday @France_Inte @PKuchly57 @Milipol_Paris for being top engaged members in my community this week :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>@Lamb2ja Hey James! How odd :/ Please call our Contact Centre on 02392441234 and we will be able to assist you :) Many thanks!</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>@DespiteOfficial we had a listen last night :) As You Bleed is an amazing track. When are you in Scotland?!</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>@97sides CONGRATS :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>yeaaaah yippppy!!!  my accnt verified rqst has succeed got a blue tick mark on my fb profile :) in 15 days</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f9a18c29-7b5a-4d5d-8e4d-ab7239c95be4')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-f9a18c29-7b5a-4d5d-8e4d-ab7239c95be4 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-f9a18c29-7b5a-4d5d-8e4d-ab7239c95be4');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-599e0254-11cc-4376-9e9b-9a4455312876\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-599e0254-11cc-4376-9e9b-9a4455312876')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-599e0254-11cc-4376-9e9b-9a4455312876 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df_pos",
              "summary": "{\n  \"name\": \"df_pos\",\n  \"rows\": 5000,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5000,\n        \"samples\": [\n          \"@Sanza_T @LeratoTmohale Does the bag do the job Sanza?  :-)\",\n          \"@WeAlwaysChillin This sounds nice :D\",\n          \"@wishinguponstar We couldn't have said it better ourselves, Kelly! So glad you're enjoying your nibbles :)\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 1,\n        \"max\": 1,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 123
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df = pd.concat([df_pos, df_neg])"
      ],
      "metadata": {
        "id": "bSSoVethLNZQ"
      },
      "execution_count": 124,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XFZGB6eNTJZu",
        "outputId": "a2e6d24c-1290-4791-d500-2c38abec7ffa"
      },
      "execution_count": 125,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(10000, 2)"
            ]
          },
          "metadata": {},
          "execution_count": 125
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df[5000:].head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "72RLBF8T8jwl",
        "outputId": "9122a494-0033-4a12-f602-7aacc93722a6"
      },
      "execution_count": 127,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                    text  \\\n",
              "0                                                                    hopeless for tmr :(   \n",
              "1  Everything in the kids section of IKEA is so cute. Shame I'm nearly 19 in 2 months :(   \n",
              "2                                 @Hegelbon That heart sliding into the waste basket. :(   \n",
              "3                       “@ketchBurning: I hate Japanese call him \"bani\" :( :(”\\n\\nMe too   \n",
              "4                                               Dang starting next week I have \"work\" :(   \n",
              "\n",
              "   label  \n",
              "0      0  \n",
              "1      0  \n",
              "2      0  \n",
              "3      0  \n",
              "4      0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-4225a57e-4f3a-4afd-a05c-882fc4e947cb\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>hopeless for tmr :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Everything in the kids section of IKEA is so cute. Shame I'm nearly 19 in 2 months :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>@Hegelbon That heart sliding into the waste basket. :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>“@ketchBurning: I hate Japanese call him \"bani\" :( :(”\\n\\nMe too</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Dang starting next week I have \"work\" :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-4225a57e-4f3a-4afd-a05c-882fc4e947cb')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-4225a57e-4f3a-4afd-a05c-882fc4e947cb button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-4225a57e-4f3a-4afd-a05c-882fc4e947cb');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-3323ea7f-ad6f-4ad2-8783-41553383588f\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-3323ea7f-ad6f-4ad2-8783-41553383588f')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-3323ea7f-ad6f-4ad2-8783-41553383588f button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"df[5000:]\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"Everything in the kids section of IKEA is so cute. Shame I'm nearly 19 in 2 months :(\",\n          \"Dang starting next week I have \\\"work\\\" :(\",\n          \"@Hegelbon That heart sliding into the waste basket. :(\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 0,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 127
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df = df.sample(frac=1).reset_index(drop=True)"
      ],
      "metadata": {
        "id": "YpguaYYxVBgL"
      },
      "execution_count": 128,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df.tail()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "6L8anEcyVUKt",
        "outputId": "35e82b0a-fc26-48db-9890-bafbf5969eb8"
      },
      "execution_count": 129,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                        text  \\\n",
              "9995                                                                                                  @luvsgngrhd ME TOO :((( wHERE IS MARGO   \n",
              "9996             @markbowthai @twomeanyoung it's 6:15pm rn!! And it's currently 70 degrees Fahrenheit!! Which is really warm so it's good :)   \n",
              "9997                                                            @Doomcrew_Glen awwwww i'm so sorry. :( i bet she's looking down on you&lt;33   \n",
              "9998                                                        @BOCAGIRLSLAYED FOLLOWED ME THANKS, AND\\n@justinbieber PLEASE FOLLOWED ME TOO :(   \n",
              "9999  i miss being a kid. i can't believe I'm doing adult stuff like paying rent and bills and doing a degree and shit like . the fuck??!?:(   \n",
              "\n",
              "      label  \n",
              "9995      0  \n",
              "9996      1  \n",
              "9997      0  \n",
              "9998      0  \n",
              "9999      0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-9adf7d48-0fd4-44d7-a2da-f52f8cc0ab8a\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>9995</th>\n",
              "      <td>@luvsgngrhd ME TOO :((( wHERE IS MARGO</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9996</th>\n",
              "      <td>@markbowthai @twomeanyoung it's 6:15pm rn!! And it's currently 70 degrees Fahrenheit!! Which is really warm so it's good :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9997</th>\n",
              "      <td>@Doomcrew_Glen awwwww i'm so sorry. :( i bet she's looking down on you&amp;lt;33</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9998</th>\n",
              "      <td>@BOCAGIRLSLAYED FOLLOWED ME THANKS, AND\\n@justinbieber PLEASE FOLLOWED ME TOO :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9999</th>\n",
              "      <td>i miss being a kid. i can't believe I'm doing adult stuff like paying rent and bills and doing a degree and shit like . the fuck??!?:(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9adf7d48-0fd4-44d7-a2da-f52f8cc0ab8a')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-9adf7d48-0fd4-44d7-a2da-f52f8cc0ab8a button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-9adf7d48-0fd4-44d7-a2da-f52f8cc0ab8a');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-5ba5b39c-6335-43c8-9a2f-2931043eaa03\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-5ba5b39c-6335-43c8-9a2f-2931043eaa03')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-5ba5b39c-6335-43c8-9a2f-2931043eaa03 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"@markbowthai @twomeanyoung it's 6:15pm rn!! And it's currently 70 degrees Fahrenheit!! Which is really warm so it's good :)\",\n          \"i miss being a kid. i can't believe I'm doing adult stuff like paying rent and bills and doing a degree and shit like . the fuck??!?:(\",\n          \"@Doomcrew_Glen awwwww i'm so sorry. :( i bet she's looking down on you&lt;33\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 129
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 130,
      "metadata": {
        "id": "ZObIlpTJJN5y",
        "outputId": "4413010d-1a5c-4cb0-ed4b-014db62bda04",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Shape of the whole data is: 10000 rows and 2 columns\n"
          ]
        }
      ],
      "source": [
        "print(f'Shape of the whole data is: {df.shape[0]} rows and {df.shape[1]} columns')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 133,
      "metadata": {
        "id": "qgglAuTGJN50",
        "outputId": "1ac192dd-0151-4ecb-a51d-adfee6f13da3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                              text  \\\n",
              "9960                                                                                                         @JackJackJohnson ended up happy. :)))   \n",
              "9961                                             Who even started this trend? I wanna know if there is a jot of truth :( #ZaynIsComingBackOnJuly26   \n",
              "9962       Here it is, our #crowdfunding campaign! If we get our goal, we can walk on with this dream. Will you help us? :) http://t.co/tY9qr7g4my   \n",
              "9963                                                                      @HoldyHoldy Cotton Candy is my favourite too! :( Time to stock up again!   \n",
              "9964                                                                                                                      @CraziestMocha I wish :(   \n",
              "9965                                                                              Happy Birthday Nawazuddin Siddique...\\nBig, Big fan of yours. :)   \n",
              "9966                                             Stats for the week have arrived. 1 new follower and NO unfollowers :) via http://t.co/4QiS2e3DaM.   \n",
              "9967  @samm_amberr Then there will be a few \"Wtf this never happened?!!\" moments that you may or may not like. But there's a surprise thrown in :D   \n",
              "9968                                                                                               @emis_nug Indeed. Thanks for link to article :)   \n",
              "9969                                                                                                   NOT AN APOLOGY ME ENCANTA VALE OSEA BEA :-(   \n",
              "9970                    I added a video to a @YouTube playlist http://t.co/09GuSuMrNb im back on twitch and today it going to be league :) - 1 / 3   \n",
              "9971        I love it how you tell every crowd that they're the loudest. We all are the loudest fans in the world aren't we? :) x @Real_Liam_Payne   \n",
              "9972                                                                                                                     @jsjnete dd leave lagi?:(   \n",
              "9973                                                                                                                             Vidcon :(((((((((   \n",
              "9974                                                                                                    @Opera_North love it! Have a #FabFriday :)   \n",
              "9975                                                      @awfullyariana WELCOME!! AND AW :( I HAVE TO GO BACK NEXT MONTH BUT RN TRYNA GET HEALTHY   \n",
              "9976                                                                                                             @Little_Nells happy birthday :)🎉🎈   \n",
              "9977   so I guess blocking number on phones spam SMS feature wont work. Nor will using specific words since these guys write Sinhala in English :(   \n",
              "9978                                                                                            You need to come back to England....:(  @MacHarmon   \n",
              "9979           @magicmoon_rp I wanna watch them again too :( maybe when they do comeback shows in Korea... or in Japan. I wanna visit your country   \n",
              "9980                                                                                 @Y_A_T_E_S Sorry about this. It could be a schedule change :(   \n",
              "9981                                                                                           @ranaPTX_ BUT WHO WILL LEAD US WHEN YOU'RE GONE :((   \n",
              "9982                                                                                                                             still awake :))))   \n",
              "9983                                                                                                                 @howdydollie I need you by :(   \n",
              "9984                                                                                              @StepintolifeYvl Amazing news! Enjoy your toy :)   \n",
              "9985                                                                @kendrahatesu FOLLOWED ME THANKS, AND\\n@justinbieber PLEASE FOLLOWED ME TOO :(   \n",
              "9986                                                                                         @johnsonef184562  You're very welcome and than You :)   \n",
              "9987                                         @ribster13 She can walk. Barely, according to her that is :) All good here. Hope T got home all ok :)   \n",
              "9988                                                   @LauraAlyceH @weimingkam And by turning up, to be fair! It meant a lot that people came. :)   \n",
              "9989                                                                             @kayceecreates @ArtieCraftie you guys have a great weekend too!:)   \n",
              "9990      \"I know you said this is Michael Jackson, but this is obviously a little boy\" - my little brother while listening to a Jackson 5 song :(   \n",
              "9991                                                                                 My face is so skinny in that picture omg :( why did i get fAt   \n",
              "9992                                                                      Agree?? :)  #Phone #WiFi #LifeStyle \\n\\n#QatarDay http://t.co/VnItXo8eZ0   \n",
              "9993                                                                                             @heyclaireee is back! thnx God!!! i'm so happy :)   \n",
              "9994                                                                                       always fall asleep so early when I'm texting someone :(   \n",
              "9995                                                                                                        @luvsgngrhd ME TOO :((( wHERE IS MARGO   \n",
              "9996                   @markbowthai @twomeanyoung it's 6:15pm rn!! And it's currently 70 degrees Fahrenheit!! Which is really warm so it's good :)   \n",
              "9997                                                                  @Doomcrew_Glen awwwww i'm so sorry. :( i bet she's looking down on you&lt;33   \n",
              "9998                                                              @BOCAGIRLSLAYED FOLLOWED ME THANKS, AND\\n@justinbieber PLEASE FOLLOWED ME TOO :(   \n",
              "9999        i miss being a kid. i can't believe I'm doing adult stuff like paying rent and bills and doing a degree and shit like . the fuck??!?:(   \n",
              "\n",
              "      label  \n",
              "9960      1  \n",
              "9961      0  \n",
              "9962      1  \n",
              "9963      0  \n",
              "9964      0  \n",
              "9965      1  \n",
              "9966      1  \n",
              "9967      1  \n",
              "9968      1  \n",
              "9969      0  \n",
              "9970      1  \n",
              "9971      1  \n",
              "9972      0  \n",
              "9973      0  \n",
              "9974      1  \n",
              "9975      0  \n",
              "9976      1  \n",
              "9977      0  \n",
              "9978      0  \n",
              "9979      0  \n",
              "9980      0  \n",
              "9981      0  \n",
              "9982      1  \n",
              "9983      0  \n",
              "9984      1  \n",
              "9985      0  \n",
              "9986      1  \n",
              "9987      1  \n",
              "9988      1  \n",
              "9989      1  \n",
              "9990      0  \n",
              "9991      0  \n",
              "9992      1  \n",
              "9993      1  \n",
              "9994      0  \n",
              "9995      0  \n",
              "9996      1  \n",
              "9997      0  \n",
              "9998      0  \n",
              "9999      0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-ec9234d3-793e-4046-b247-d2697bacd890\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>9960</th>\n",
              "      <td>@JackJackJohnson ended up happy. :)))</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9961</th>\n",
              "      <td>Who even started this trend? I wanna know if there is a jot of truth :( #ZaynIsComingBackOnJuly26</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9962</th>\n",
              "      <td>Here it is, our #crowdfunding campaign! If we get our goal, we can walk on with this dream. Will you help us? :) http://t.co/tY9qr7g4my</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9963</th>\n",
              "      <td>@HoldyHoldy Cotton Candy is my favourite too! :( Time to stock up again!</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9964</th>\n",
              "      <td>@CraziestMocha I wish :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9965</th>\n",
              "      <td>Happy Birthday Nawazuddin Siddique...\\nBig, Big fan of yours. :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9966</th>\n",
              "      <td>Stats for the week have arrived. 1 new follower and NO unfollowers :) via http://t.co/4QiS2e3DaM.</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9967</th>\n",
              "      <td>@samm_amberr Then there will be a few \"Wtf this never happened?!!\" moments that you may or may not like. But there's a surprise thrown in :D</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9968</th>\n",
              "      <td>@emis_nug Indeed. Thanks for link to article :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9969</th>\n",
              "      <td>NOT AN APOLOGY ME ENCANTA VALE OSEA BEA :-(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9970</th>\n",
              "      <td>I added a video to a @YouTube playlist http://t.co/09GuSuMrNb im back on twitch and today it going to be league :) - 1 / 3</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9971</th>\n",
              "      <td>I love it how you tell every crowd that they're the loudest. We all are the loudest fans in the world aren't we? :) x @Real_Liam_Payne</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9972</th>\n",
              "      <td>@jsjnete dd leave lagi?:(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9973</th>\n",
              "      <td>Vidcon :(((((((((</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9974</th>\n",
              "      <td>@Opera_North love it! Have a #FabFriday :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9975</th>\n",
              "      <td>@awfullyariana WELCOME!! AND AW :( I HAVE TO GO BACK NEXT MONTH BUT RN TRYNA GET HEALTHY</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9976</th>\n",
              "      <td>@Little_Nells happy birthday :)🎉🎈</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9977</th>\n",
              "      <td>so I guess blocking number on phones spam SMS feature wont work. Nor will using specific words since these guys write Sinhala in English :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9978</th>\n",
              "      <td>You need to come back to England....:(  @MacHarmon</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9979</th>\n",
              "      <td>@magicmoon_rp I wanna watch them again too :( maybe when they do comeback shows in Korea... or in Japan. I wanna visit your country</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9980</th>\n",
              "      <td>@Y_A_T_E_S Sorry about this. It could be a schedule change :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9981</th>\n",
              "      <td>@ranaPTX_ BUT WHO WILL LEAD US WHEN YOU'RE GONE :((</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9982</th>\n",
              "      <td>still awake :))))</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9983</th>\n",
              "      <td>@howdydollie I need you by :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9984</th>\n",
              "      <td>@StepintolifeYvl Amazing news! Enjoy your toy :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9985</th>\n",
              "      <td>@kendrahatesu FOLLOWED ME THANKS, AND\\n@justinbieber PLEASE FOLLOWED ME TOO :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9986</th>\n",
              "      <td>@johnsonef184562  You're very welcome and than You :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9987</th>\n",
              "      <td>@ribster13 She can walk. Barely, according to her that is :) All good here. Hope T got home all ok :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9988</th>\n",
              "      <td>@LauraAlyceH @weimingkam And by turning up, to be fair! It meant a lot that people came. :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9989</th>\n",
              "      <td>@kayceecreates @ArtieCraftie you guys have a great weekend too!:)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9990</th>\n",
              "      <td>\"I know you said this is Michael Jackson, but this is obviously a little boy\" - my little brother while listening to a Jackson 5 song :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9991</th>\n",
              "      <td>My face is so skinny in that picture omg :( why did i get fAt</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9992</th>\n",
              "      <td>Agree?? :)  #Phone #WiFi #LifeStyle \\n\\n#QatarDay http://t.co/VnItXo8eZ0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9993</th>\n",
              "      <td>@heyclaireee is back! thnx God!!! i'm so happy :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9994</th>\n",
              "      <td>always fall asleep so early when I'm texting someone :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9995</th>\n",
              "      <td>@luvsgngrhd ME TOO :((( wHERE IS MARGO</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9996</th>\n",
              "      <td>@markbowthai @twomeanyoung it's 6:15pm rn!! And it's currently 70 degrees Fahrenheit!! Which is really warm so it's good :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9997</th>\n",
              "      <td>@Doomcrew_Glen awwwww i'm so sorry. :( i bet she's looking down on you&amp;lt;33</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9998</th>\n",
              "      <td>@BOCAGIRLSLAYED FOLLOWED ME THANKS, AND\\n@justinbieber PLEASE FOLLOWED ME TOO :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9999</th>\n",
              "      <td>i miss being a kid. i can't believe I'm doing adult stuff like paying rent and bills and doing a degree and shit like . the fuck??!?:(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ec9234d3-793e-4046-b247-d2697bacd890')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-ec9234d3-793e-4046-b247-d2697bacd890 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-ec9234d3-793e-4046-b247-d2697bacd890');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-f10a467b-678a-4aee-babf-ea50c841a2c9\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-f10a467b-678a-4aee-babf-ea50c841a2c9')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-f10a467b-678a-4aee-babf-ea50c841a2c9 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 40,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 40,\n        \"samples\": [\n          \"@magicmoon_rp I wanna watch them again too :( maybe when they do comeback shows in Korea... or in Japan. I wanna visit your country\",\n          \"@Little_Nells happy birthday :)\\ud83c\\udf89\\ud83c\\udf88\",\n          \"@awfullyariana WELCOME!! AND AW :( I HAVE TO GO BACK NEXT MONTH BUT RN TRYNA GET HEALTHY\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 133
        }
      ],
      "source": [
        "# Look at the head of the dataframe\n",
        "df.tail(40)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6hMQCCOQJN51"
      },
      "source": [
        "> Note: Always make it a practice to first skim the dataset before performing any text pre-processing steps. It is important because text data can be very noisy eg. dates are written in different formats, present of accented characters, etc. These are stuff we can easily miss if we don't go through the dataset properly."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KTiC3iEmJN53"
      },
      "source": [
        "## Lower Casing\n",
        "\n",
        "Lowercasing is a common text preprocessing technique. It helps to transform all the text in same case. <br>\n",
        "Examples 'The', 'the', 'ThE' -> 'the'\n",
        "\n",
        "This is also useful to find all the duplicates since words in different cases are treated as separate words and becomes difficult for us to remove redundant words in all different case combination.\n",
        "\n",
        "This may not be helpful when we do tasks like Part of Speech tagging (where proper casing gives some information about Nouns and so on) and Sentiment Analysis (where upper casing refers to anger and so on)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 134,
      "metadata": {
        "id": "ohipSKqOJN54",
        "outputId": "fbea80f2-0d3d-4a9a-917f-5bd2ae5d5c99",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 677
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                            text  \\\n",
              "0            aww the poor thing :( hope it'okay and in good health. luckely it has been freed from those rocks #orcalove https://t.co/toszmoafv6   \n",
              "1       @batesm0t3l hi there, i've spoken with the store who have advised they do have a paypoint and that card can be topped up :) thanks, beth   \n",
              "2                                                                      @danielle_isla i could never go there ever after seeing that, so cruel :(   \n",
              "3                                                                           when the bus is late and you have no time to get food before work :(   \n",
              "4                                                                                                                 another ridiculous headache :(   \n",
              "5                      my kik : oulive70748 #kik #kikmenow #photo #babe #loveofmylife #brasileirao #viernesderolenahot :( http://t.co/gmmwd4prhu   \n",
              "6                       @dylanobrien @mazerunnermovie it is just beyond words, sooooooo freaking good! i'm going to die before it comes out!! :d   \n",
              "7                                                                                                            @jaymcguiness  :-( please notice me   \n",
              "8                                                                                                                                        babe :(   \n",
              "9                                                                                                     @barbaranadel @kirkdalebooks definitely :)   \n",
              "10  ♛♛♛\\n》》》》 \\ni love you so much.\\ni beli̇eve that he wi̇ll follow.\\nplease follow me please justi̇n @justinbieber :( x15.331\\n》》》》ｓｅｅ ｍｅ\\n♛♛♛   \n",
              "11                                                                                                     @forlodge1 it is indeed - many thanks :-)   \n",
              "12                                  @sweetnuskie i just did, i don't get it tbh :p but ellie with superpowers is what i live for so i'm happy :p   \n",
              "13                                                                can anyone give me money for the train so that i can see @littlemix today?! :(   \n",
              "14                                                                                             but i'm still hoping :( #zayniscomingbackonjuly26   \n",
              "15              photoset: katmtan: for the swan queen fandom :) sorry the others are blurry! i was so far away and was... http://t.co/uvaz6ca1ig   \n",
              "16                                                   #@thewinesociety: .@vin_dourthe will be very pleased to here it! :)… http://t.co/i9zotnyyec   \n",
              "17      hey guys! i'll be live in 30 mins! do you want to do some fun low ranked games or subgames or faceit today? :)... http://t.co/e4h5imxzmh   \n",
              "18                                                                                                        @huskar_nana who would you like? :) xo   \n",
              "19                                                                                                           @streetfighter still not working :(   \n",
              "\n",
              "    label  \n",
              "0       0  \n",
              "1       1  \n",
              "2       0  \n",
              "3       0  \n",
              "4       0  \n",
              "5       0  \n",
              "6       1  \n",
              "7       0  \n",
              "8       0  \n",
              "9       1  \n",
              "10      0  \n",
              "11      1  \n",
              "12      1  \n",
              "13      0  \n",
              "14      0  \n",
              "15      1  \n",
              "16      1  \n",
              "17      1  \n",
              "18      1  \n",
              "19      0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-eaf4b523-63d8-4860-8e81-3fe7df4ed78b\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>aww the poor thing :( hope it'okay and in good health. luckely it has been freed from those rocks #orcalove https://t.co/toszmoafv6</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>@batesm0t3l hi there, i've spoken with the store who have advised they do have a paypoint and that card can be topped up :) thanks, beth</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>@danielle_isla i could never go there ever after seeing that, so cruel :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>when the bus is late and you have no time to get food before work :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>another ridiculous headache :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>my kik : oulive70748 #kik #kikmenow #photo #babe #loveofmylife #brasileirao #viernesderolenahot :( http://t.co/gmmwd4prhu</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>@dylanobrien @mazerunnermovie it is just beyond words, sooooooo freaking good! i'm going to die before it comes out!! :d</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>@jaymcguiness  :-( please notice me</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>babe :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>@barbaranadel @kirkdalebooks definitely :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>♛♛♛\\n》》》》 \\ni love you so much.\\ni beli̇eve that he wi̇ll follow.\\nplease follow me please justi̇n @justinbieber :( x15.331\\n》》》》ｓｅｅ ｍｅ\\n♛♛♛</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>@forlodge1 it is indeed - many thanks :-)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>@sweetnuskie i just did, i don't get it tbh :p but ellie with superpowers is what i live for so i'm happy :p</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>can anyone give me money for the train so that i can see @littlemix today?! :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>but i'm still hoping :( #zayniscomingbackonjuly26</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>photoset: katmtan: for the swan queen fandom :) sorry the others are blurry! i was so far away and was... http://t.co/uvaz6ca1ig</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>#@thewinesociety: .@vin_dourthe will be very pleased to here it! :)… http://t.co/i9zotnyyec</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>hey guys! i'll be live in 30 mins! do you want to do some fun low ranked games or subgames or faceit today? :)... http://t.co/e4h5imxzmh</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>@huskar_nana who would you like? :) xo</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>@streetfighter still not working :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-eaf4b523-63d8-4860-8e81-3fe7df4ed78b')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-eaf4b523-63d8-4860-8e81-3fe7df4ed78b button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-eaf4b523-63d8-4860-8e81-3fe7df4ed78b');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-36f2bab6-b402-48c7-b5d3-c029b0de0758\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-36f2bab6-b402-48c7-b5d3-c029b0de0758')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-36f2bab6-b402-48c7-b5d3-c029b0de0758 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 10000,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 9993,\n        \"samples\": [\n          \"@pussy_lilac no problem :) it's a pleasure \\u2764\\ufe0f\",\n          \"@andyfales @gustopizzadm i wish i could have pizza delivered to me at 5am... :(\",\n          \"@slaydrey where is this?? \\u2665\\u2665\\u2665 looks so relaxing :(\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 134
        }
      ],
      "source": [
        "df.text = df.text.str.lower()\n",
        "df.head(20)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ZRYmuVWJN56"
      },
      "source": [
        "## Remove"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z7rmNmfhJN57"
      },
      "source": [
        "### URL's"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q6GfbXOOJN57"
      },
      "source": [
        "URL stands for Uniform Resource Locator. If present in a text, it represents the location of another website.\n",
        "\n",
        "If we are performing any websites backlink analysis, in that case URL's are useful to keep. Otherwise, they don't provide any information. So we can remove them from our text."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 135,
      "metadata": {
        "id": "s4AJVaFFJN58",
        "outputId": "175ce9c0-fb96-4fc3-d113-2a4831b0bf4c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 677
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                            text  \\\n",
              "0                                   aww the poor thing :( hope it'okay and in good health. luckely it has been freed from those rocks #orcalove    \n",
              "1       @batesm0t3l hi there, i've spoken with the store who have advised they do have a paypoint and that card can be topped up :) thanks, beth   \n",
              "2                                                                      @danielle_isla i could never go there ever after seeing that, so cruel :(   \n",
              "3                                                                           when the bus is late and you have no time to get food before work :(   \n",
              "4                                                                                                                 another ridiculous headache :(   \n",
              "5                                            my kik : oulive70748 #kik #kikmenow #photo #babe #loveofmylife #brasileirao #viernesderolenahot :(    \n",
              "6                       @dylanobrien @mazerunnermovie it is just beyond words, sooooooo freaking good! i'm going to die before it comes out!! :d   \n",
              "7                                                                                                            @jaymcguiness  :-( please notice me   \n",
              "8                                                                                                                                        babe :(   \n",
              "9                                                                                                     @barbaranadel @kirkdalebooks definitely :)   \n",
              "10  ♛♛♛\\n》》》》 \\ni love you so much.\\ni beli̇eve that he wi̇ll follow.\\nplease follow me please justi̇n @justinbieber :( x15.331\\n》》》》ｓｅｅ ｍｅ\\n♛♛♛   \n",
              "11                                                                                                     @forlodge1 it is indeed - many thanks :-)   \n",
              "12                                  @sweetnuskie i just did, i don't get it tbh :p but ellie with superpowers is what i live for so i'm happy :p   \n",
              "13                                                                can anyone give me money for the train so that i can see @littlemix today?! :(   \n",
              "14                                                                                             but i'm still hoping :( #zayniscomingbackonjuly26   \n",
              "15                                    photoset: katmtan: for the swan queen fandom :) sorry the others are blurry! i was so far away and was...    \n",
              "16                                                                         #@thewinesociety: .@vin_dourthe will be very pleased to here it! :)…    \n",
              "17                            hey guys! i'll be live in 30 mins! do you want to do some fun low ranked games or subgames or faceit today? :)...    \n",
              "18                                                                                                        @huskar_nana who would you like? :) xo   \n",
              "19                                                                                                           @streetfighter still not working :(   \n",
              "\n",
              "    label  \n",
              "0       0  \n",
              "1       1  \n",
              "2       0  \n",
              "3       0  \n",
              "4       0  \n",
              "5       0  \n",
              "6       1  \n",
              "7       0  \n",
              "8       0  \n",
              "9       1  \n",
              "10      0  \n",
              "11      1  \n",
              "12      1  \n",
              "13      0  \n",
              "14      0  \n",
              "15      1  \n",
              "16      1  \n",
              "17      1  \n",
              "18      1  \n",
              "19      0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a027841c-57a0-43b4-af7b-f660f999aa10\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>aww the poor thing :( hope it'okay and in good health. luckely it has been freed from those rocks #orcalove</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>@batesm0t3l hi there, i've spoken with the store who have advised they do have a paypoint and that card can be topped up :) thanks, beth</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>@danielle_isla i could never go there ever after seeing that, so cruel :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>when the bus is late and you have no time to get food before work :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>another ridiculous headache :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>my kik : oulive70748 #kik #kikmenow #photo #babe #loveofmylife #brasileirao #viernesderolenahot :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>@dylanobrien @mazerunnermovie it is just beyond words, sooooooo freaking good! i'm going to die before it comes out!! :d</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>@jaymcguiness  :-( please notice me</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>babe :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>@barbaranadel @kirkdalebooks definitely :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>♛♛♛\\n》》》》 \\ni love you so much.\\ni beli̇eve that he wi̇ll follow.\\nplease follow me please justi̇n @justinbieber :( x15.331\\n》》》》ｓｅｅ ｍｅ\\n♛♛♛</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>@forlodge1 it is indeed - many thanks :-)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>@sweetnuskie i just did, i don't get it tbh :p but ellie with superpowers is what i live for so i'm happy :p</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>can anyone give me money for the train so that i can see @littlemix today?! :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>but i'm still hoping :( #zayniscomingbackonjuly26</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>photoset: katmtan: for the swan queen fandom :) sorry the others are blurry! i was so far away and was...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>#@thewinesociety: .@vin_dourthe will be very pleased to here it! :)…</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>hey guys! i'll be live in 30 mins! do you want to do some fun low ranked games or subgames or faceit today? :)...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>@huskar_nana who would you like? :) xo</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>@streetfighter still not working :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a027841c-57a0-43b4-af7b-f660f999aa10')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-a027841c-57a0-43b4-af7b-f660f999aa10 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-a027841c-57a0-43b4-af7b-f660f999aa10');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-343bc806-f447-4f9f-85dd-f0b1a5f3e6ce\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-343bc806-f447-4f9f-85dd-f0b1a5f3e6ce')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-343bc806-f447-4f9f-85dd-f0b1a5f3e6ce button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 10000,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 9880,\n        \"samples\": [\n          \"@bahihontera y not , sure .. :)\",\n          \"@zouiriaii good luck for ur exam :)\",\n          \"|| i want cereal.\\n\\ntoo bad we don't have any. &gt;:(\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 135
        }
      ],
      "source": [
        "df.text = df.text.str.replace(r'https?://\\S+|www\\.\\S+', '', regex=True)\n",
        "df.head(20)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qBgweUUMJN59"
      },
      "source": [
        "### E-mail"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FLzUla3JJN5-"
      },
      "source": [
        "E-mail id's are common in customer feedback data and they do not provide any useful information. So we remove them from the text.\n",
        "\n",
        "Twitter data that we are using does not contain any email id's. Hence, please find the code snipper with an dummy example to remove e-mail id's."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 136,
      "metadata": {
        "id": "Haj8Y-5tJN5_",
        "outputId": "3b378f0b-3f34-45b2-fb82-b3aeaf153e74",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'I have being trying to contact xyz via email to  but there is no response.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 136
        }
      ],
      "source": [
        "text = 'I have being trying to contact xyz via email to xyz@abc.co.in but there is no response.'\n",
        "re.sub(r'\\S+@\\S+', '', text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-D24xhOSJN6A"
      },
      "source": [
        "### Date"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d_0bkJyEJN6A"
      },
      "source": [
        "Dates can be represented in various formats and can be difficult at times to remove them. They are unlikely to contain any useful information for predicting the labels.\n",
        "\n",
        "Below I have used dummy text to showcase the following task."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 137,
      "metadata": {
        "id": "H31Bj7qMJN6B",
        "outputId": "80033e86-9f68-4fd8-8a28-87cdd4271995",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Today is  and after two days on  our vacation starts until '"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 137
        }
      ],
      "source": [
        "text = \"Today is 22/12/2020 and after two days on 24-12-2020 our vacation starts until 25th.09.2021\"\n",
        "\n",
        "# 1. Remove date formats like: dd/mm/yy(yy), dd-mm-yy(yy), dd(st|nd|rd).mm/yy(yy)\n",
        "re.sub(r'\\d{1,2}(st|nd|rd|th)?[-./]\\d{1,2}[-./]\\d{2,4}', '', text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 138,
      "metadata": {
        "id": "SH-6He7tJN6C",
        "outputId": "60499909-944c-4dcc-dad8-711e8d1ff9cd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Today is  when I am writing this post. I hope to post this byor max to max by or '"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 138
        }
      ],
      "source": [
        "text = \"Today is 11th of January, 2021 when I am writing this post. I hope to post this by February 15th or max to max by 20 may 21 or 20th-December-21\"\n",
        "\n",
        "# 2. Remove date formats like: 20 apr 21, April 15th, 11th of April, 2021\n",
        "pattern = re.compile(r'(\\d{1,2})?(st|nd|rd|th)?[-./,]?\\s?(of)?\\s?([J|j]an(uary)?|[F|f]eb(ruary)?|[Mm]ar(ch)?|[Aa]pr(il)?|[Mm]ay|[Jj]un(e)?|[Jj]ul(y)?|[Aa]ug(ust)?|[Ss]ep(tember)?|[Oo]ct(ober)?|[Nn]ov(ember)?|[Dd]ec(ember)?)\\s?(\\d{1,2})?(st|nd|rd|th)?\\s?[-./,]?\\s?(\\d{2,4})?')\n",
        "pattern.sub(r'', text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "50IfdvJWJN6D"
      },
      "source": [
        "There are various formats in which dates are represented and the above regex can be customized in many ways. Above, \"byor\" got combined cause we are trying multiple format in single regex pattern. You can customize the above expression accordingly to your need."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZGlQtdzDJN6D"
      },
      "source": [
        "### HTML Tags"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XDLH4ihDJN6E"
      },
      "source": [
        "If we are extracting data from various websites, it is possible that the data also contains HTML tags. These tags does not provide any information and should be removed. These tags can be removed using regex or by using BeautifulSoup library."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 141,
      "metadata": {
        "id": "ufUeX9N0JN6E"
      },
      "outputs": [],
      "source": [
        "# Dummy text\n",
        "text = \"\"\"\n",
        "<title>Below is a dummy html code.</title>\n",
        "<body>\n",
        "    <p>All the html opening and closing brackets should be remove.</p>\n",
        "    <a href=\"https://www.abc.com\">Company Site</a>\n",
        "</body>\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 142,
      "metadata": {
        "id": "x4xcz6pqJN6F",
        "outputId": "5190d42a-34ac-403c-f492-be58a9b23b6b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'\\nBelow is a dummy html code.\\n\\n    All the html opening and closing brackets should be remove.\\n    Company Site\\n\\n'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 142
        }
      ],
      "source": [
        "# Using regex to remove html tags\n",
        "pattern = re.compile('<.*?>')\n",
        "pattern.sub('', text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 143,
      "metadata": {
        "id": "W3vxjhEUJN6G"
      },
      "outputs": [],
      "source": [
        "# Using Beautiful Soup\n",
        "def remove_html(text):\n",
        "    clean_text = BeautifulSoup(text).get_text()\n",
        "    return clean_text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 144,
      "metadata": {
        "id": "uENxJZp-JN6G",
        "outputId": "0df79c22-447f-474f-e93c-8104dccc29f6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Below is a dummy html code.\\n\\nAll the html opening and closing brackets should be remove.\\nCompany Site\\n\\n'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 144
        }
      ],
      "source": [
        "remove_html(text)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.tail(50)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "ORcVKf3eJbH4",
        "outputId": "95b77af4-8df4-4337-9885-145f720dac24"
      },
      "execution_count": 147,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                              text  \\\n",
              "9950                                                                            @lawrenceispichu oh gosh what did you say? and aw hun :( *cuddles*   \n",
              "9951                                                                                           @diongzons  follow @jnlazts &amp;  follow u back :)   \n",
              "9952                                                                 @cath_tyldesley @yourstylist it is always good to stand out from the crowd :)   \n",
              "9953                                                                                                    so much editing to do. so little time. :-(   \n",
              "9954                                                                              @iloveksoo we almost got to see his cute ankles but his socks :(   \n",
              "9955                             @lukebryanonline yayyyy!!! i hope it's not while i am knocked out by anesthesia. i will be so sad if i miss it :(   \n",
              "9956                                                                                            @tmhsposey that was the account that got hacked :(   \n",
              "9957             shadowplaylouis we recently got this mutual and i'm really loving your tweets and everything your about so yay that's exciting :)   \n",
              "9958                                                                                                                           @aangelayap easy :d   \n",
              "9959                                                                                                phone fell out of my hand and it woke me up :(   \n",
              "9960                                                                                                         @jackjackjohnson ended up happy. :)))   \n",
              "9961                                             who even started this trend? i wanna know if there is a jot of truth :( #zayniscomingbackonjuly26   \n",
              "9962                             here it is, our #crowdfunding campaign! if we get our goal, we can walk on with this dream. will you help us? :)    \n",
              "9963                                                                      @holdyholdy cotton candy is my favourite too! :( time to stock up again!   \n",
              "9964                                                                                                                      @craziestmocha i wish :(   \n",
              "9965                                                                              happy birthday nawazuddin siddique...\\nbig, big fan of yours. :)   \n",
              "9966                                                                    stats for the week have arrived. 1 new follower and no unfollowers :) via    \n",
              "9967  @samm_amberr then there will be a few \"wtf this never happened?!!\" moments that you may or may not like. but there's a surprise thrown in :d   \n",
              "9968                                                                                               @emis_nug indeed. thanks for link to article :)   \n",
              "9969                                                                                                   not an apology me encanta vale osea bea :-(   \n",
              "9970                                          i added a video to a @youtube playlist  im back on twitch and today it going to be league :) - 1 / 3   \n",
              "9971        i love it how you tell every crowd that they're the loudest. we all are the loudest fans in the world aren't we? :) x @real_liam_payne   \n",
              "9972                                                                                                                     @jsjnete dd leave lagi?:(   \n",
              "9973                                                                                                                             vidcon :(((((((((   \n",
              "9974                                                                                                    @opera_north love it! have a #fabfriday :)   \n",
              "9975                                                      @awfullyariana welcome!! and aw :( i have to go back next month but rn tryna get healthy   \n",
              "9976                                                                                                             @little_nells happy birthday :)🎉🎈   \n",
              "9977   so i guess blocking number on phones spam sms feature wont work. nor will using specific words since these guys write sinhala in english :(   \n",
              "9978                                                                                            you need to come back to england....:(  @macharmon   \n",
              "9979           @magicmoon_rp i wanna watch them again too :( maybe when they do comeback shows in korea... or in japan. i wanna visit your country   \n",
              "9980                                                                                 @y_a_t_e_s sorry about this. it could be a schedule change :(   \n",
              "9981                                                                                           @ranaptx_ but who will lead us when you're gone :((   \n",
              "9982                                                                                                                             still awake :))))   \n",
              "9983                                                                                                                 @howdydollie i need you by :(   \n",
              "9984                                                                                              @stepintolifeyvl amazing news! enjoy your toy :)   \n",
              "9985                                                                @kendrahatesu followed me thanks, and\\n@justinbieber please followed me too :(   \n",
              "9986                                                                                         @johnsonef184562  you're very welcome and than you :)   \n",
              "9987                                         @ribster13 she can walk. barely, according to her that is :) all good here. hope t got home all ok :)   \n",
              "9988                                                   @lauraalyceh @weimingkam and by turning up, to be fair! it meant a lot that people came. :)   \n",
              "9989                                                                             @kayceecreates @artiecraftie you guys have a great weekend too!:)   \n",
              "9990      \"i know you said this is michael jackson, but this is obviously a little boy\" - my little brother while listening to a jackson 5 song :(   \n",
              "9991                                                                                 my face is so skinny in that picture omg :( why did i get fat   \n",
              "9992                                                                                            agree?? :)  #phone #wifi #lifestyle \\n\\n#qatarday    \n",
              "9993                                                                                             @heyclaireee is back! thnx god!!! i'm so happy :)   \n",
              "9994                                                                                       always fall asleep so early when i'm texting someone :(   \n",
              "9995                                                                                                        @luvsgngrhd me too :((( where is margo   \n",
              "9996                   @markbowthai @twomeanyoung it's 6:15pm rn!! and it's currently 70 degrees fahrenheit!! which is really warm so it's good :)   \n",
              "9997                                                                  @doomcrew_glen awwwww i'm so sorry. :( i bet she's looking down on you&lt;33   \n",
              "9998                                                              @bocagirlslayed followed me thanks, and\\n@justinbieber please followed me too :(   \n",
              "9999        i miss being a kid. i can't believe i'm doing adult stuff like paying rent and bills and doing a degree and shit like . the fuck??!?:(   \n",
              "\n",
              "      label  \n",
              "9950      0  \n",
              "9951      1  \n",
              "9952      1  \n",
              "9953      0  \n",
              "9954      0  \n",
              "9955      0  \n",
              "9956      0  \n",
              "9957      1  \n",
              "9958      1  \n",
              "9959      0  \n",
              "9960      1  \n",
              "9961      0  \n",
              "9962      1  \n",
              "9963      0  \n",
              "9964      0  \n",
              "9965      1  \n",
              "9966      1  \n",
              "9967      1  \n",
              "9968      1  \n",
              "9969      0  \n",
              "9970      1  \n",
              "9971      1  \n",
              "9972      0  \n",
              "9973      0  \n",
              "9974      1  \n",
              "9975      0  \n",
              "9976      1  \n",
              "9977      0  \n",
              "9978      0  \n",
              "9979      0  \n",
              "9980      0  \n",
              "9981      0  \n",
              "9982      1  \n",
              "9983      0  \n",
              "9984      1  \n",
              "9985      0  \n",
              "9986      1  \n",
              "9987      1  \n",
              "9988      1  \n",
              "9989      1  \n",
              "9990      0  \n",
              "9991      0  \n",
              "9992      1  \n",
              "9993      1  \n",
              "9994      0  \n",
              "9995      0  \n",
              "9996      1  \n",
              "9997      0  \n",
              "9998      0  \n",
              "9999      0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a4d69545-65ef-4047-820a-fa1269c7ce16\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>9950</th>\n",
              "      <td>@lawrenceispichu oh gosh what did you say? and aw hun :( *cuddles*</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9951</th>\n",
              "      <td>@diongzons  follow @jnlazts &amp;amp;  follow u back :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9952</th>\n",
              "      <td>@cath_tyldesley @yourstylist it is always good to stand out from the crowd :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9953</th>\n",
              "      <td>so much editing to do. so little time. :-(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9954</th>\n",
              "      <td>@iloveksoo we almost got to see his cute ankles but his socks :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9955</th>\n",
              "      <td>@lukebryanonline yayyyy!!! i hope it's not while i am knocked out by anesthesia. i will be so sad if i miss it :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9956</th>\n",
              "      <td>@tmhsposey that was the account that got hacked :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9957</th>\n",
              "      <td>shadowplaylouis we recently got this mutual and i'm really loving your tweets and everything your about so yay that's exciting :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9958</th>\n",
              "      <td>@aangelayap easy :d</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9959</th>\n",
              "      <td>phone fell out of my hand and it woke me up :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9960</th>\n",
              "      <td>@jackjackjohnson ended up happy. :)))</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9961</th>\n",
              "      <td>who even started this trend? i wanna know if there is a jot of truth :( #zayniscomingbackonjuly26</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9962</th>\n",
              "      <td>here it is, our #crowdfunding campaign! if we get our goal, we can walk on with this dream. will you help us? :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9963</th>\n",
              "      <td>@holdyholdy cotton candy is my favourite too! :( time to stock up again!</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9964</th>\n",
              "      <td>@craziestmocha i wish :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9965</th>\n",
              "      <td>happy birthday nawazuddin siddique...\\nbig, big fan of yours. :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9966</th>\n",
              "      <td>stats for the week have arrived. 1 new follower and no unfollowers :) via</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9967</th>\n",
              "      <td>@samm_amberr then there will be a few \"wtf this never happened?!!\" moments that you may or may not like. but there's a surprise thrown in :d</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9968</th>\n",
              "      <td>@emis_nug indeed. thanks for link to article :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9969</th>\n",
              "      <td>not an apology me encanta vale osea bea :-(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9970</th>\n",
              "      <td>i added a video to a @youtube playlist  im back on twitch and today it going to be league :) - 1 / 3</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9971</th>\n",
              "      <td>i love it how you tell every crowd that they're the loudest. we all are the loudest fans in the world aren't we? :) x @real_liam_payne</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9972</th>\n",
              "      <td>@jsjnete dd leave lagi?:(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9973</th>\n",
              "      <td>vidcon :(((((((((</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9974</th>\n",
              "      <td>@opera_north love it! have a #fabfriday :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9975</th>\n",
              "      <td>@awfullyariana welcome!! and aw :( i have to go back next month but rn tryna get healthy</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9976</th>\n",
              "      <td>@little_nells happy birthday :)🎉🎈</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9977</th>\n",
              "      <td>so i guess blocking number on phones spam sms feature wont work. nor will using specific words since these guys write sinhala in english :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9978</th>\n",
              "      <td>you need to come back to england....:(  @macharmon</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9979</th>\n",
              "      <td>@magicmoon_rp i wanna watch them again too :( maybe when they do comeback shows in korea... or in japan. i wanna visit your country</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9980</th>\n",
              "      <td>@y_a_t_e_s sorry about this. it could be a schedule change :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9981</th>\n",
              "      <td>@ranaptx_ but who will lead us when you're gone :((</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9982</th>\n",
              "      <td>still awake :))))</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9983</th>\n",
              "      <td>@howdydollie i need you by :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9984</th>\n",
              "      <td>@stepintolifeyvl amazing news! enjoy your toy :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9985</th>\n",
              "      <td>@kendrahatesu followed me thanks, and\\n@justinbieber please followed me too :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9986</th>\n",
              "      <td>@johnsonef184562  you're very welcome and than you :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9987</th>\n",
              "      <td>@ribster13 she can walk. barely, according to her that is :) all good here. hope t got home all ok :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9988</th>\n",
              "      <td>@lauraalyceh @weimingkam and by turning up, to be fair! it meant a lot that people came. :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9989</th>\n",
              "      <td>@kayceecreates @artiecraftie you guys have a great weekend too!:)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9990</th>\n",
              "      <td>\"i know you said this is michael jackson, but this is obviously a little boy\" - my little brother while listening to a jackson 5 song :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9991</th>\n",
              "      <td>my face is so skinny in that picture omg :( why did i get fat</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9992</th>\n",
              "      <td>agree?? :)  #phone #wifi #lifestyle \\n\\n#qatarday</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9993</th>\n",
              "      <td>@heyclaireee is back! thnx god!!! i'm so happy :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9994</th>\n",
              "      <td>always fall asleep so early when i'm texting someone :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9995</th>\n",
              "      <td>@luvsgngrhd me too :((( where is margo</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9996</th>\n",
              "      <td>@markbowthai @twomeanyoung it's 6:15pm rn!! and it's currently 70 degrees fahrenheit!! which is really warm so it's good :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9997</th>\n",
              "      <td>@doomcrew_glen awwwww i'm so sorry. :( i bet she's looking down on you&amp;lt;33</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9998</th>\n",
              "      <td>@bocagirlslayed followed me thanks, and\\n@justinbieber please followed me too :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9999</th>\n",
              "      <td>i miss being a kid. i can't believe i'm doing adult stuff like paying rent and bills and doing a degree and shit like . the fuck??!?:(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a4d69545-65ef-4047-820a-fa1269c7ce16')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-a4d69545-65ef-4047-820a-fa1269c7ce16 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-a4d69545-65ef-4047-820a-fa1269c7ce16');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-3c361450-3d19-459c-95a7-c84fdc6ba512\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-3c361450-3d19-459c-95a7-c84fdc6ba512')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-3c361450-3d19-459c-95a7-c84fdc6ba512 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 50,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 50,\n        \"samples\": [\n          \"@holdyholdy cotton candy is my favourite too! :( time to stock up again!\",\n          \"@kayceecreates @artiecraftie you guys have a great weekend too!:)\",\n          \"@y_a_t_e_s sorry about this. it could be a schedule change :(\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 147
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Emoji handling with emoji library"
      ],
      "metadata": {
        "id": "l8UWZjTDUmCO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install emoji"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GlBgOvCuAkjC",
        "outputId": "fd500ade-a13e-4ef1-dfdd-279bfe985fe5"
      },
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: emoji in /usr/local/lib/python3.10/dist-packages (2.10.1)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import emoji"
      ],
      "metadata": {
        "id": "fX-XxaKCA854"
      },
      "execution_count": 148,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "emoji.replace_emoji(\"game is on 🔥🔥. Hilarious😂\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "L4VC2zhPBOCl",
        "outputId": "cf59732b-0c63-4d34-f13c-1498984177ca"
      },
      "execution_count": 149,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'game is on . Hilarious'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 149
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "emoji.demojize(\"game is on 🔥🔥. Hilarious😂\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "xtQxdWvoBsIY",
        "outputId": "ffd01c81-b719-4115-ac38-35fee0bf35ca"
      },
      "execution_count": 150,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'game is on :fire::fire:. Hilarious:face_with_tears_of_joy:'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 150
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HyUBPxrzJN6H"
      },
      "source": [
        "### Emojis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KOaJ10HuJN6H"
      },
      "source": [
        "As more and more people have started using social media emoji's play a very crucial role. Emoji's are used to express emotions that are universally understood.\n",
        "\n",
        "In some analysis such as sentiment analysis emoji's can be useful. We can convert them to words or create some new features based on them. For some analysis we need to remove them. Find the below code snippet used to remove the emoji's."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 151,
      "metadata": {
        "id": "OreSDxRAJN6I"
      },
      "outputs": [],
      "source": [
        "#collapse-hide\n",
        "# Reference: https://gist.github.com/slowkow/7a7f61f495e3dbb7e3d767f97bd7304b\n",
        "\n",
        "def remove_emoji(text):\n",
        "    emoji_pattern = re.compile(\"[\"\n",
        "                               u\"\\U0001F600-\\U0001F64F\"  # emoticons\n",
        "                               u\"\\U0001F300-\\U0001F5FF\"  # symbols & pictographs\n",
        "                               u\"\\U0001F680-\\U0001F6FF\"  # transport & map symbols\n",
        "                               u\"\\U0001F1E0-\\U0001F1FF\"  # flags (iOS)\n",
        "                               u\"\\U00002500-\\U00002BEF\"  # chinese char\n",
        "                               u\"\\U00002702-\\U000027B0\"\n",
        "                               u\"\\U00002702-\\U000027B0\"\n",
        "                               u\"\\U000024C2-\\U0001F251\"\n",
        "                               u\"\\U0001f926-\\U0001f937\"\n",
        "                               u\"\\U00010000-\\U0010ffff\"\n",
        "                               u\"\\u2640-\\u2642\"\n",
        "                               u\"\\u2600-\\u2B55\"\n",
        "                               u\"\\u200d\"\n",
        "                               u\"\\u23cf\"\n",
        "                               u\"\\u23e9\"\n",
        "                               u\"\\u231a\"\n",
        "                               u\"\\ufe0f\"  # dingbats\n",
        "                               u\"\\u3030\"\n",
        "                               \"]+\", flags=re.UNICODE)\n",
        "    return emoji_pattern.sub(r'', text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 152,
      "metadata": {
        "id": "6TwNIc1SJN6J",
        "outputId": "8854ab50-8849-43cb-9ce4-17c54d091b3a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'game is on . Hilarious'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 152
        }
      ],
      "source": [
        "text = \"game is on 🔥🔥. Hilarious😂\"\n",
        "remove_emoji(text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 154,
      "metadata": {
        "id": "cZX7Tcu9JN6K"
      },
      "outputs": [],
      "source": [
        "# Remove emoji's from text\n",
        "df.text = df.text.apply(lambda x: emoji.replace_emoji(x))"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.tail(50)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "rpWQ-Syfh6ay",
        "outputId": "d0745564-2a40-49f4-fd5d-c88341c7b928"
      },
      "execution_count": 155,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                              text  \\\n",
              "9950                                                                            @lawrenceispichu oh gosh what did you say? and aw hun :( *cuddles*   \n",
              "9951                                                                                           @diongzons  follow @jnlazts &amp;  follow u back :)   \n",
              "9952                                                                 @cath_tyldesley @yourstylist it is always good to stand out from the crowd :)   \n",
              "9953                                                                                                    so much editing to do. so little time. :-(   \n",
              "9954                                                                              @iloveksoo we almost got to see his cute ankles but his socks :(   \n",
              "9955                             @lukebryanonline yayyyy!!! i hope it's not while i am knocked out by anesthesia. i will be so sad if i miss it :(   \n",
              "9956                                                                                            @tmhsposey that was the account that got hacked :(   \n",
              "9957             shadowplaylouis we recently got this mutual and i'm really loving your tweets and everything your about so yay that's exciting :)   \n",
              "9958                                                                                                                           @aangelayap easy :d   \n",
              "9959                                                                                                phone fell out of my hand and it woke me up :(   \n",
              "9960                                                                                                         @jackjackjohnson ended up happy. :)))   \n",
              "9961                                             who even started this trend? i wanna know if there is a jot of truth :( #zayniscomingbackonjuly26   \n",
              "9962                             here it is, our #crowdfunding campaign! if we get our goal, we can walk on with this dream. will you help us? :)    \n",
              "9963                                                                      @holdyholdy cotton candy is my favourite too! :( time to stock up again!   \n",
              "9964                                                                                                                      @craziestmocha i wish :(   \n",
              "9965                                                                              happy birthday nawazuddin siddique...\\nbig, big fan of yours. :)   \n",
              "9966                                                                    stats for the week have arrived. 1 new follower and no unfollowers :) via    \n",
              "9967  @samm_amberr then there will be a few \"wtf this never happened?!!\" moments that you may or may not like. but there's a surprise thrown in :d   \n",
              "9968                                                                                               @emis_nug indeed. thanks for link to article :)   \n",
              "9969                                                                                                   not an apology me encanta vale osea bea :-(   \n",
              "9970                                          i added a video to a @youtube playlist  im back on twitch and today it going to be league :) - 1 / 3   \n",
              "9971        i love it how you tell every crowd that they're the loudest. we all are the loudest fans in the world aren't we? :) x @real_liam_payne   \n",
              "9972                                                                                                                     @jsjnete dd leave lagi?:(   \n",
              "9973                                                                                                                             vidcon :(((((((((   \n",
              "9974                                                                                                    @opera_north love it! have a #fabfriday :)   \n",
              "9975                                                      @awfullyariana welcome!! and aw :( i have to go back next month but rn tryna get healthy   \n",
              "9976                                                                                                               @little_nells happy birthday :)   \n",
              "9977   so i guess blocking number on phones spam sms feature wont work. nor will using specific words since these guys write sinhala in english :(   \n",
              "9978                                                                                            you need to come back to england....:(  @macharmon   \n",
              "9979           @magicmoon_rp i wanna watch them again too :( maybe when they do comeback shows in korea... or in japan. i wanna visit your country   \n",
              "9980                                                                                 @y_a_t_e_s sorry about this. it could be a schedule change :(   \n",
              "9981                                                                                           @ranaptx_ but who will lead us when you're gone :((   \n",
              "9982                                                                                                                             still awake :))))   \n",
              "9983                                                                                                                 @howdydollie i need you by :(   \n",
              "9984                                                                                              @stepintolifeyvl amazing news! enjoy your toy :)   \n",
              "9985                                                                @kendrahatesu followed me thanks, and\\n@justinbieber please followed me too :(   \n",
              "9986                                                                                         @johnsonef184562  you're very welcome and than you :)   \n",
              "9987                                         @ribster13 she can walk. barely, according to her that is :) all good here. hope t got home all ok :)   \n",
              "9988                                                   @lauraalyceh @weimingkam and by turning up, to be fair! it meant a lot that people came. :)   \n",
              "9989                                                                             @kayceecreates @artiecraftie you guys have a great weekend too!:)   \n",
              "9990      \"i know you said this is michael jackson, but this is obviously a little boy\" - my little brother while listening to a jackson 5 song :(   \n",
              "9991                                                                                 my face is so skinny in that picture omg :( why did i get fat   \n",
              "9992                                                                                            agree?? :)  #phone #wifi #lifestyle \\n\\n#qatarday    \n",
              "9993                                                                                             @heyclaireee is back! thnx god!!! i'm so happy :)   \n",
              "9994                                                                                       always fall asleep so early when i'm texting someone :(   \n",
              "9995                                                                                                        @luvsgngrhd me too :((( where is margo   \n",
              "9996                   @markbowthai @twomeanyoung it's 6:15pm rn!! and it's currently 70 degrees fahrenheit!! which is really warm so it's good :)   \n",
              "9997                                                                  @doomcrew_glen awwwww i'm so sorry. :( i bet she's looking down on you&lt;33   \n",
              "9998                                                              @bocagirlslayed followed me thanks, and\\n@justinbieber please followed me too :(   \n",
              "9999        i miss being a kid. i can't believe i'm doing adult stuff like paying rent and bills and doing a degree and shit like . the fuck??!?:(   \n",
              "\n",
              "      label  \n",
              "9950      0  \n",
              "9951      1  \n",
              "9952      1  \n",
              "9953      0  \n",
              "9954      0  \n",
              "9955      0  \n",
              "9956      0  \n",
              "9957      1  \n",
              "9958      1  \n",
              "9959      0  \n",
              "9960      1  \n",
              "9961      0  \n",
              "9962      1  \n",
              "9963      0  \n",
              "9964      0  \n",
              "9965      1  \n",
              "9966      1  \n",
              "9967      1  \n",
              "9968      1  \n",
              "9969      0  \n",
              "9970      1  \n",
              "9971      1  \n",
              "9972      0  \n",
              "9973      0  \n",
              "9974      1  \n",
              "9975      0  \n",
              "9976      1  \n",
              "9977      0  \n",
              "9978      0  \n",
              "9979      0  \n",
              "9980      0  \n",
              "9981      0  \n",
              "9982      1  \n",
              "9983      0  \n",
              "9984      1  \n",
              "9985      0  \n",
              "9986      1  \n",
              "9987      1  \n",
              "9988      1  \n",
              "9989      1  \n",
              "9990      0  \n",
              "9991      0  \n",
              "9992      1  \n",
              "9993      1  \n",
              "9994      0  \n",
              "9995      0  \n",
              "9996      1  \n",
              "9997      0  \n",
              "9998      0  \n",
              "9999      0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-d496773d-7edd-4de5-8c46-2f2d650a57ce\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>9950</th>\n",
              "      <td>@lawrenceispichu oh gosh what did you say? and aw hun :( *cuddles*</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9951</th>\n",
              "      <td>@diongzons  follow @jnlazts &amp;amp;  follow u back :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9952</th>\n",
              "      <td>@cath_tyldesley @yourstylist it is always good to stand out from the crowd :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9953</th>\n",
              "      <td>so much editing to do. so little time. :-(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9954</th>\n",
              "      <td>@iloveksoo we almost got to see his cute ankles but his socks :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9955</th>\n",
              "      <td>@lukebryanonline yayyyy!!! i hope it's not while i am knocked out by anesthesia. i will be so sad if i miss it :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9956</th>\n",
              "      <td>@tmhsposey that was the account that got hacked :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9957</th>\n",
              "      <td>shadowplaylouis we recently got this mutual and i'm really loving your tweets and everything your about so yay that's exciting :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9958</th>\n",
              "      <td>@aangelayap easy :d</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9959</th>\n",
              "      <td>phone fell out of my hand and it woke me up :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9960</th>\n",
              "      <td>@jackjackjohnson ended up happy. :)))</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9961</th>\n",
              "      <td>who even started this trend? i wanna know if there is a jot of truth :( #zayniscomingbackonjuly26</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9962</th>\n",
              "      <td>here it is, our #crowdfunding campaign! if we get our goal, we can walk on with this dream. will you help us? :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9963</th>\n",
              "      <td>@holdyholdy cotton candy is my favourite too! :( time to stock up again!</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9964</th>\n",
              "      <td>@craziestmocha i wish :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9965</th>\n",
              "      <td>happy birthday nawazuddin siddique...\\nbig, big fan of yours. :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9966</th>\n",
              "      <td>stats for the week have arrived. 1 new follower and no unfollowers :) via</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9967</th>\n",
              "      <td>@samm_amberr then there will be a few \"wtf this never happened?!!\" moments that you may or may not like. but there's a surprise thrown in :d</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9968</th>\n",
              "      <td>@emis_nug indeed. thanks for link to article :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9969</th>\n",
              "      <td>not an apology me encanta vale osea bea :-(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9970</th>\n",
              "      <td>i added a video to a @youtube playlist  im back on twitch and today it going to be league :) - 1 / 3</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9971</th>\n",
              "      <td>i love it how you tell every crowd that they're the loudest. we all are the loudest fans in the world aren't we? :) x @real_liam_payne</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9972</th>\n",
              "      <td>@jsjnete dd leave lagi?:(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9973</th>\n",
              "      <td>vidcon :(((((((((</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9974</th>\n",
              "      <td>@opera_north love it! have a #fabfriday :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9975</th>\n",
              "      <td>@awfullyariana welcome!! and aw :( i have to go back next month but rn tryna get healthy</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9976</th>\n",
              "      <td>@little_nells happy birthday :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9977</th>\n",
              "      <td>so i guess blocking number on phones spam sms feature wont work. nor will using specific words since these guys write sinhala in english :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9978</th>\n",
              "      <td>you need to come back to england....:(  @macharmon</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9979</th>\n",
              "      <td>@magicmoon_rp i wanna watch them again too :( maybe when they do comeback shows in korea... or in japan. i wanna visit your country</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9980</th>\n",
              "      <td>@y_a_t_e_s sorry about this. it could be a schedule change :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9981</th>\n",
              "      <td>@ranaptx_ but who will lead us when you're gone :((</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9982</th>\n",
              "      <td>still awake :))))</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9983</th>\n",
              "      <td>@howdydollie i need you by :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9984</th>\n",
              "      <td>@stepintolifeyvl amazing news! enjoy your toy :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9985</th>\n",
              "      <td>@kendrahatesu followed me thanks, and\\n@justinbieber please followed me too :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9986</th>\n",
              "      <td>@johnsonef184562  you're very welcome and than you :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9987</th>\n",
              "      <td>@ribster13 she can walk. barely, according to her that is :) all good here. hope t got home all ok :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9988</th>\n",
              "      <td>@lauraalyceh @weimingkam and by turning up, to be fair! it meant a lot that people came. :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9989</th>\n",
              "      <td>@kayceecreates @artiecraftie you guys have a great weekend too!:)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9990</th>\n",
              "      <td>\"i know you said this is michael jackson, but this is obviously a little boy\" - my little brother while listening to a jackson 5 song :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9991</th>\n",
              "      <td>my face is so skinny in that picture omg :( why did i get fat</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9992</th>\n",
              "      <td>agree?? :)  #phone #wifi #lifestyle \\n\\n#qatarday</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9993</th>\n",
              "      <td>@heyclaireee is back! thnx god!!! i'm so happy :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9994</th>\n",
              "      <td>always fall asleep so early when i'm texting someone :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9995</th>\n",
              "      <td>@luvsgngrhd me too :((( where is margo</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9996</th>\n",
              "      <td>@markbowthai @twomeanyoung it's 6:15pm rn!! and it's currently 70 degrees fahrenheit!! which is really warm so it's good :)</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9997</th>\n",
              "      <td>@doomcrew_glen awwwww i'm so sorry. :( i bet she's looking down on you&amp;lt;33</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9998</th>\n",
              "      <td>@bocagirlslayed followed me thanks, and\\n@justinbieber please followed me too :(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9999</th>\n",
              "      <td>i miss being a kid. i can't believe i'm doing adult stuff like paying rent and bills and doing a degree and shit like . the fuck??!?:(</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d496773d-7edd-4de5-8c46-2f2d650a57ce')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-d496773d-7edd-4de5-8c46-2f2d650a57ce button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-d496773d-7edd-4de5-8c46-2f2d650a57ce');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-f565b33f-1626-4bb6-9b18-a1463372fcd3\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-f565b33f-1626-4bb6-9b18-a1463372fcd3')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-f565b33f-1626-4bb6-9b18-a1463372fcd3 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 50,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 50,\n        \"samples\": [\n          \"@holdyholdy cotton candy is my favourite too! :( time to stock up again!\",\n          \"@kayceecreates @artiecraftie you guys have a great weekend too!:)\",\n          \"@y_a_t_e_s sorry about this. it could be a schedule change :(\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 155
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QljXrNUQJN6K"
      },
      "source": [
        "### Emoticons"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XkTzNKHLJN6L"
      },
      "source": [
        "Emoji's and Emoticons are different. Yes!!<br>\n",
        "Emoticons are used to express facial expressions using keyboard characters such as letters, numbers, and pucntuation marks. Where emjoi's are small images.\n",
        "\n",
        "Thanks to [Neel Shah](https://github.com/NeelShah18/emot/blob/master/emot/emo_unicode.py) for curating a dictionary of emoticons and their description. We shall use this dictionary and remove the emoticons from our text."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 156,
      "metadata": {
        "id": "gJwdypCVJN6L"
      },
      "outputs": [],
      "source": [
        "#collapse-hide\n",
        "\n",
        "EMOTICONS = {\n",
        "    u\":‑\\)\":\"Happy face or smiley\",\n",
        "    u\":\\)\":\"Happy face or smiley\",\n",
        "    u\":-\\]\":\"Happy face or smiley\",\n",
        "    u\":\\]\":\"Happy face or smiley\",\n",
        "    u\":-3\":\"Happy face smiley\",\n",
        "    u\":3\":\"Happy face smiley\",\n",
        "    u\":->\":\"Happy face smiley\",\n",
        "    u\":>\":\"Happy face smiley\",\n",
        "    u\"8-\\)\":\"Happy face smiley\",\n",
        "    u\":o\\)\":\"Happy face smiley\",\n",
        "    u\":-\\}\":\"Happy face smiley\",\n",
        "    u\":\\}\":\"Happy face smiley\",\n",
        "    u\":-\\)\":\"Happy face smiley\",\n",
        "    u\":c\\)\":\"Happy face smiley\",\n",
        "    u\":\\^\\)\":\"Happy face smiley\",\n",
        "    u\"=\\]\":\"Happy face smiley\",\n",
        "    u\"=\\)\":\"Happy face smiley\",\n",
        "    u\":‑D\":\"Laughing, big grin or laugh with glasses\",\n",
        "    u\":D\":\"Laughing, big grin or laugh with glasses\",\n",
        "    u\":d\":\"Laughing, big grin or laugh with glasses\",\n",
        "    u\"8‑D\":\"Laughing, big grin or laugh with glasses\",\n",
        "    u\"8D\":\"Laughing, big grin or laugh with glasses\",\n",
        "    u\"X‑D\":\"Laughing, big grin or laugh with glasses\",\n",
        "    u\"XD\":\"Laughing, big grin or laugh with glasses\",\n",
        "    u\"=D\":\"Laughing, big grin or laugh with glasses\",\n",
        "    u\"=3\":\"Laughing, big grin or laugh with glasses\",\n",
        "    u\"B\\^D\":\"Laughing, big grin or laugh with glasses\",\n",
        "    u\":-\\)\\)\":\"Very happy\",\n",
        "    u\":‑\\(\":\"Frown, sad, andry or pouting\",\n",
        "    u\":-\\(\":\"Frown, sad, andry or pouting\",\n",
        "    u\":\\(\":\"Frown, sad, andry or pouting\",\n",
        "    u\":‑c\":\"Frown, sad, andry or pouting\",\n",
        "    u\":c\":\"Frown, sad, andry or pouting\",\n",
        "    u\":‑<\":\"Frown, sad, andry or pouting\",\n",
        "    u\":<\":\"Frown, sad, andry or pouting\",\n",
        "    u\":‑\\[\":\"Frown, sad, andry or pouting\",\n",
        "    u\":\\[\":\"Frown, sad, andry or pouting\",\n",
        "    u\":-\\|\\|\":\"Frown, sad, andry or pouting\",\n",
        "    u\">:\\[\":\"Frown, sad, andry or pouting\",\n",
        "    u\":\\{\":\"Frown, sad, andry or pouting\",\n",
        "    u\":@\":\"Frown, sad, andry or pouting\",\n",
        "    u\">:\\(\":\"Frown, sad, andry or pouting\",\n",
        "    u\":'‑\\(\":\"Crying\",\n",
        "    u\":'\\(\":\"Crying\",\n",
        "    u\":'‑\\)\":\"Tears of happiness\",\n",
        "    u\":'\\)\":\"Tears of happiness\",\n",
        "    u\"D‑':\":\"Horror\",\n",
        "    u\"D:<\":\"Disgust\",\n",
        "    u\"D:\":\"Sadness\",\n",
        "    u\"D8\":\"Great dismay\",\n",
        "    u\"D;\":\"Great dismay\",\n",
        "    u\"D=\":\"Great dismay\",\n",
        "    u\"DX\":\"Great dismay\",\n",
        "    u\":‑O\":\"Surprise\",\n",
        "    u\":O\":\"Surprise\",\n",
        "    u\":‑o\":\"Surprise\",\n",
        "    u\":o\":\"Surprise\",\n",
        "    u\":-0\":\"Shock\",\n",
        "    u\"8‑0\":\"Yawn\",\n",
        "    u\">:O\":\"Yawn\",\n",
        "    u\":-\\*\":\"Kiss\",\n",
        "    u\":\\*\":\"Kiss\",\n",
        "    u\":X\":\"Kiss\",\n",
        "    u\";‑\\)\":\"Wink or smirk\",\n",
        "    u\";\\)\":\"Wink or smirk\",\n",
        "    u\"\\*-\\)\":\"Wink or smirk\",\n",
        "    u\"\\*\\)\":\"Wink or smirk\",\n",
        "    u\";‑\\]\":\"Wink or smirk\",\n",
        "    u\";\\]\":\"Wink or smirk\",\n",
        "    u\";\\^\\)\":\"Wink or smirk\",\n",
        "    u\":‑,\":\"Wink or smirk\",\n",
        "    u\";D\":\"Wink or smirk\",\n",
        "    u\":‑P\":\"Tongue sticking out, cheeky, playful or blowing a raspberry\",\n",
        "    u\":P\":\"Tongue sticking out, cheeky, playful or blowing a raspberry\",\n",
        "    u\"X‑P\":\"Tongue sticking out, cheeky, playful or blowing a raspberry\",\n",
        "    u\"XP\":\"Tongue sticking out, cheeky, playful or blowing a raspberry\",\n",
        "    u\":‑Þ\":\"Tongue sticking out, cheeky, playful or blowing a raspberry\",\n",
        "    u\":Þ\":\"Tongue sticking out, cheeky, playful or blowing a raspberry\",\n",
        "    u\":b\":\"Tongue sticking out, cheeky, playful or blowing a raspberry\",\n",
        "    u\"d:\":\"Tongue sticking out, cheeky, playful or blowing a raspberry\",\n",
        "    u\"=p\":\"Tongue sticking out, cheeky, playful or blowing a raspberry\",\n",
        "    u\">:P\":\"Tongue sticking out, cheeky, playful or blowing a raspberry\",\n",
        "    u\":‑/\":\"Skeptical, annoyed, undecided, uneasy or hesitant\",\n",
        "    u\":/\":\"Skeptical, annoyed, undecided, uneasy or hesitant\",\n",
        "    u\":-[.]\":\"Skeptical, annoyed, undecided, uneasy or hesitant\",\n",
        "    u\">:[(\\\\\\)]\":\"Skeptical, annoyed, undecided, uneasy or hesitant\",\n",
        "    u\">:/\":\"Skeptical, annoyed, undecided, uneasy or hesitant\",\n",
        "    u\":[(\\\\\\)]\":\"Skeptical, annoyed, undecided, uneasy or hesitant\",\n",
        "    u\"=/\":\"Skeptical, annoyed, undecided, uneasy or hesitant\",\n",
        "    u\"=[(\\\\\\)]\":\"Skeptical, annoyed, undecided, uneasy or hesitant\",\n",
        "    u\":L\":\"Skeptical, annoyed, undecided, uneasy or hesitant\",\n",
        "    u\"=L\":\"Skeptical, annoyed, undecided, uneasy or hesitant\",\n",
        "    u\":S\":\"Skeptical, annoyed, undecided, uneasy or hesitant\",\n",
        "    u\":‑\\|\":\"Straight face\",\n",
        "    u\":\\|\":\"Straight face\",\n",
        "    u\":$\":\"Embarrassed or blushing\",\n",
        "    u\":‑x\":\"Sealed lips or wearing braces or tongue-tied\",\n",
        "    u\":x\":\"Sealed lips or wearing braces or tongue-tied\",\n",
        "    u\":‑#\":\"Sealed lips or wearing braces or tongue-tied\",\n",
        "    u\":#\":\"Sealed lips or wearing braces or tongue-tied\",\n",
        "    u\":‑&\":\"Sealed lips or wearing braces or tongue-tied\",\n",
        "    u\":&\":\"Sealed lips or wearing braces or tongue-tied\",\n",
        "    u\"O:‑\\)\":\"Angel, saint or innocent\",\n",
        "    u\"O:\\)\":\"Angel, saint or innocent\",\n",
        "    u\"0:‑3\":\"Angel, saint or innocent\",\n",
        "    u\"0:3\":\"Angel, saint or innocent\",\n",
        "    u\"0:‑\\)\":\"Angel, saint or innocent\",\n",
        "    u\"0:\\)\":\"Angel, saint or innocent\",\n",
        "    u\":‑b\":\"Tongue sticking out, cheeky, playful or blowing a raspberry\",\n",
        "    u\"0;\\^\\)\":\"Angel, saint or innocent\",\n",
        "    u\">:‑\\)\":\"Evil or devilish\",\n",
        "    u\">:\\)\":\"Evil or devilish\",\n",
        "    u\"\\}:‑\\)\":\"Evil or devilish\",\n",
        "    u\"\\}:\\)\":\"Evil or devilish\",\n",
        "    u\"3:‑\\)\":\"Evil or devilish\",\n",
        "    u\"3:\\)\":\"Evil or devilish\",\n",
        "    u\">;\\)\":\"Evil or devilish\",\n",
        "    u\"\\|;‑\\)\":\"Cool\",\n",
        "    u\"\\|‑O\":\"Bored\",\n",
        "    u\":‑J\":\"Tongue-in-cheek\",\n",
        "    u\"#‑\\)\":\"Party all night\",\n",
        "    u\"%‑\\)\":\"Drunk or confused\",\n",
        "    u\"%\\)\":\"Drunk or confused\",\n",
        "    u\":-###..\":\"Being sick\",\n",
        "    u\":###..\":\"Being sick\",\n",
        "    u\"<:‑\\|\":\"Dump\",\n",
        "    u\"\\(>_<\\)\":\"Troubled\",\n",
        "    u\"\\(>_<\\)>\":\"Troubled\",\n",
        "    u\"\\(';'\\)\":\"Baby\",\n",
        "    u\"\\(\\^\\^>``\":\"Nervous or Embarrassed or Troubled or Shy or Sweat drop\",\n",
        "    u\"\\(\\^_\\^;\\)\":\"Nervous or Embarrassed or Troubled or Shy or Sweat drop\",\n",
        "    u\"\\(-_-;\\)\":\"Nervous or Embarrassed or Troubled or Shy or Sweat drop\",\n",
        "    u\"\\(~_~;\\) \\(・\\.・;\\)\":\"Nervous or Embarrassed or Troubled or Shy or Sweat drop\",\n",
        "    u\"\\(-_-\\)zzz\":\"Sleeping\",\n",
        "    u\"\\(\\^_-\\)\":\"Wink\",\n",
        "    u\"\\(\\(\\+_\\+\\)\\)\":\"Confused\",\n",
        "    u\"\\(\\+o\\+\\)\":\"Confused\",\n",
        "    u\"\\(o\\|o\\)\":\"Ultraman\",\n",
        "    u\"\\^_\\^\":\"Joyful\",\n",
        "    u\"\\(\\^_\\^\\)/\":\"Joyful\",\n",
        "    u\"\\(\\^O\\^\\)／\":\"Joyful\",\n",
        "    u\"\\(\\^o\\^\\)／\":\"Joyful\",\n",
        "    u\"\\(__\\)\":\"Kowtow as a sign of respect, or dogeza for apology\",\n",
        "    u\"_\\(\\._\\.\\)_\":\"Kowtow as a sign of respect, or dogeza for apology\",\n",
        "    u\"<\\(_ _\\)>\":\"Kowtow as a sign of respect, or dogeza for apology\",\n",
        "    u\"<m\\(__\\)m>\":\"Kowtow as a sign of respect, or dogeza for apology\",\n",
        "    u\"m\\(__\\)m\":\"Kowtow as a sign of respect, or dogeza for apology\",\n",
        "    u\"m\\(_ _\\)m\":\"Kowtow as a sign of respect, or dogeza for apology\",\n",
        "    u\"\\('_'\\)\":\"Sad or Crying\",\n",
        "    u\"\\(/_;\\)\":\"Sad or Crying\",\n",
        "    u\"\\(T_T\\) \\(;_;\\)\":\"Sad or Crying\",\n",
        "    u\"\\(;_;\":\"Sad of Crying\",\n",
        "    u\"\\(;_:\\)\":\"Sad or Crying\",\n",
        "    u\"\\(;O;\\)\":\"Sad or Crying\",\n",
        "    u\"\\(:_;\\)\":\"Sad or Crying\",\n",
        "    u\"\\(ToT\\)\":\"Sad or Crying\",\n",
        "    u\";_;\":\"Sad or Crying\",\n",
        "    u\";-;\":\"Sad or Crying\",\n",
        "    u\";n;\":\"Sad or Crying\",\n",
        "    u\";;\":\"Sad or Crying\",\n",
        "    u\"Q\\.Q\":\"Sad or Crying\",\n",
        "    u\"T\\.T\":\"Sad or Crying\",\n",
        "    u\"QQ\":\"Sad or Crying\",\n",
        "    u\"Q_Q\":\"Sad or Crying\",\n",
        "    u\"\\(-\\.-\\)\":\"Shame\",\n",
        "    u\"\\(-_-\\)\":\"Shame\",\n",
        "    u\"\\(一一\\)\":\"Shame\",\n",
        "    u\"\\(；一_一\\)\":\"Shame\",\n",
        "    u\"\\(=_=\\)\":\"Tired\",\n",
        "    u\"\\(=\\^\\·\\^=\\)\":\"cat\",\n",
        "    u\"\\(=\\^\\·\\·\\^=\\)\":\"cat\",\n",
        "    u\"=_\\^=\t\":\"cat\",\n",
        "    u\"\\(\\.\\.\\)\":\"Looking down\",\n",
        "    u\"\\(\\._\\.\\)\":\"Looking down\",\n",
        "    u\"\\^m\\^\":\"Giggling with hand covering mouth\",\n",
        "    u\"\\(\\・\\・?\":\"Confusion\",\n",
        "    u\"\\(?_?\\)\":\"Confusion\",\n",
        "    u\">\\^_\\^<\":\"Normal Laugh\",\n",
        "    u\"<\\^!\\^>\":\"Normal Laugh\",\n",
        "    u\"\\^/\\^\":\"Normal Laugh\",\n",
        "    u\"\\（\\*\\^_\\^\\*）\" :\"Normal Laugh\",\n",
        "    u\"\\(\\^<\\^\\) \\(\\^\\.\\^\\)\":\"Normal Laugh\",\n",
        "    u\"\\(^\\^\\)\":\"Normal Laugh\",\n",
        "    u\"\\(\\^\\.\\^\\)\":\"Normal Laugh\",\n",
        "    u\"\\(\\^_\\^\\.\\)\":\"Normal Laugh\",\n",
        "    u\"\\(\\^_\\^\\)\":\"Normal Laugh\",\n",
        "    u\"\\(\\^\\^\\)\":\"Normal Laugh\",\n",
        "    u\"\\(\\^J\\^\\)\":\"Normal Laugh\",\n",
        "    u\"\\(\\*\\^\\.\\^\\*\\)\":\"Normal Laugh\",\n",
        "    u\"\\(\\^—\\^\\）\":\"Normal Laugh\",\n",
        "    u\"\\(#\\^\\.\\^#\\)\":\"Normal Laugh\",\n",
        "    u\"\\（\\^—\\^\\）\":\"Waving\",\n",
        "    u\"\\(;_;\\)/~~~\":\"Waving\",\n",
        "    u\"\\(\\^\\.\\^\\)/~~~\":\"Waving\",\n",
        "    u\"\\(-_-\\)/~~~ \\($\\·\\·\\)/~~~\":\"Waving\",\n",
        "    u\"\\(T_T\\)/~~~\":\"Waving\",\n",
        "    u\"\\(ToT\\)/~~~\":\"Waving\",\n",
        "    u\"\\(\\*\\^0\\^\\*\\)\":\"Excited\",\n",
        "    u\"\\(\\*_\\*\\)\":\"Amazed\",\n",
        "    u\"\\(\\*_\\*;\":\"Amazed\",\n",
        "    u\"\\(\\+_\\+\\) \\(@_@\\)\":\"Amazed\",\n",
        "    u\"\\(\\*\\^\\^\\)v\":\"Laughing,Cheerful\",\n",
        "    u\"\\(\\^_\\^\\)v\":\"Laughing,Cheerful\",\n",
        "    u\"\\(\\(d[-_-]b\\)\\)\":\"Headphones,Listening to music\",\n",
        "    u'\\(-\"-\\)':\"Worried\",\n",
        "    u\"\\(ーー;\\)\":\"Worried\",\n",
        "    u\"\\(\\^0_0\\^\\)\":\"Eyeglasses\",\n",
        "    u\"\\(\\＾ｖ\\＾\\)\":\"Happy\",\n",
        "    u\"\\(\\＾ｕ\\＾\\)\":\"Happy\",\n",
        "    u\"\\(\\^\\)o\\(\\^\\)\":\"Happy\",\n",
        "    u\"\\(\\^O\\^\\)\":\"Happy\",\n",
        "    u\"\\(\\^o\\^\\)\":\"Happy\",\n",
        "    u\"\\)\\^o\\^\\(\":\"Happy\",\n",
        "    u\":O o_O\":\"Surprised\",\n",
        "    u\"o_0\":\"Surprised\",\n",
        "    u\"o\\.O\":\"Surpised\",\n",
        "    u\"\\(o\\.o\\)\":\"Surprised\",\n",
        "    u\"oO\":\"Surprised\",\n",
        "    u\"\\(\\*￣m￣\\)\":\"Dissatisfied\",\n",
        "    u\"\\(‘A`\\)\":\"Snubbed or Deflated\"\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 157,
      "metadata": {
        "id": "rKcvebwmJN6P"
      },
      "outputs": [],
      "source": [
        "def remove_emoticons(text):\n",
        "    emoticons_pattern = re.compile(u'(' + u'|'.join(emo for emo in EMOTICONS) + u')')\n",
        "    return emoticons_pattern.sub(r'', text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 158,
      "metadata": {
        "id": "INrDCoDnJN6Q",
        "outputId": "78c80594-5899-4270-e20b-fd097409296c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Hello '"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 158
        }
      ],
      "source": [
        "remove_emoticons(\"Hello :->\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 159,
      "metadata": {
        "id": "CiP6kvZ4JN6R"
      },
      "outputs": [],
      "source": [
        "# Remove emoticons from text\n",
        "df.text = df.text.apply(lambda x: remove_emoticons(x))"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.tail(50)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "SB4WZFs9aYhG",
        "outputId": "0d97b340-cfad-4dda-879b-d985f11c95e5"
      },
      "execution_count": 160,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                            text  \\\n",
              "9950                                                                            @lawrenceispichu oh gosh what did you say? and aw hun  *cuddles*   \n",
              "9951                                                                                           @diongzons  follow @jnlazts &amp;  follow u back    \n",
              "9952                                                                 @cath_tyldesley @yourstylist it is always good to stand out from the crowd    \n",
              "9953                                                                                                     so much editing to do. so little time.    \n",
              "9954                                                                              @iloveksoo we almost got to see his cute ankles but his socks    \n",
              "9955                             @lukebryanonline yayyyy!!! i hope it's not while i am knocked out by anesthesia. i will be so sad if i miss it    \n",
              "9956                                                                                            @tmhsposey that was the account that got hacked    \n",
              "9957             shadowplaylouis we recently got this mutual and i'm really loving your tweets and everything your about so yay that's exciting    \n",
              "9958                                                                                                                           @aangelayap easy    \n",
              "9959                                                                                                phone fell out of my hand and it woke me up    \n",
              "9960                                                                                                           @jackjackjohnson ended up happy.    \n",
              "9961                                             who even started this trend? i wanna know if there is a jot of truth  #zayniscomingbackonjuly26   \n",
              "9962                             here it is, our #crowdfunding campaign! if we get our goal, we can walk on with this dream. will you help us?     \n",
              "9963                                                                      @holdyholdy cotton candy is my favourite too!  time to stock up again!   \n",
              "9964                                                                                                                      @craziestmocha i wish    \n",
              "9965                                                                              happy birthday nawazuddin siddique...\\nbig, big fan of yours.    \n",
              "9966                                                                    stats for the week have arrived. 1 new follower and no unfollowers  via    \n",
              "9967  @samm_amberr then there will be a few \"wtf this never happened?!!\" moments that you may or may not like. but there's a surprise thrown in    \n",
              "9968                                                                                               @emis_nug indeed. thanks for link to article    \n",
              "9969                                                                                                    not an apology me encanta vale osea bea    \n",
              "9970                                          i added a video to a @youtube playlist  im back on twitch and today it going to be league  - 1 / 3   \n",
              "9971        i love it how you tell every crowd that they're the loudest. we all are the loudest fans in the world aren't we?  x @real_liam_payne   \n",
              "9972                                                                                                                     @jsjnete dd leave lagi?   \n",
              "9973                                                                                                                             vidcon ((((((((   \n",
              "9974                                                                                                    @opera_north love it! have a #fabfriday    \n",
              "9975                                                      @awfullyariana welcome!! and aw  i have to go back next month but rn tryna get healthy   \n",
              "9976                                                                                                               @little_nells happy birthday    \n",
              "9977   so i guess blocking number on phones spam sms feature wont work. nor will using specific words since these guys write sinhala in english    \n",
              "9978                                                                                            you need to come back to england....  @macharmon   \n",
              "9979           @magicmoon_rp i wanna watch them again too  maybe when they do comeback shows in korea... or in japan. i wanna visit your country   \n",
              "9980                                                                                 @y_a_t_e_s sorry about this. it could be a schedule change    \n",
              "9981                                                                                           @ranaptx_ but who will lead us when you're gone (   \n",
              "9982                                                                                                                                still awake    \n",
              "9983                                                                                                                 @howdydollie i need you by    \n",
              "9984                                                                                              @stepintolifeyvl amazing news! enjoy your toy    \n",
              "9985                                                                @kendrahatesu followed me thanks, and\\n@justinbieber please followed me too    \n",
              "9986                                                                                         @johnsonef184562  you're very welcome and than you    \n",
              "9987                                           @ribster13 she can walk. barely, according to her that is  all good here. hope t got home all ok    \n",
              "9988                                                   @lauraalyceh @weimingkam and by turning up, to be fair! it meant a lot that people came.    \n",
              "9989                                                                             @kayceecreates @artiecraftie you guys have a great weekend too!   \n",
              "9990      \"i know you said this is michael jackson, but this is obviously a little boy\" - my little brother while listening to a jackson 5 song    \n",
              "9991                                                                                 my face is so skinny in that picture omg  why did i get fat   \n",
              "9992                                                                                            agree??   #phone #wifi #lifestyle \\n\\n#qatarday    \n",
              "9993                                                                                             @heyclaireee is back! thnx god!!! i'm so happy    \n",
              "9994                                                                                       always fall asleep so early when i'm texting someone    \n",
              "9995                                                                                                        @luvsgngrhd me too (( where is margo   \n",
              "9996                   @markbowthai @twomeanyoung it's 6:15pm rn!! and it's currently 70 degrees fahrenheit!! which is really warm so it's good    \n",
              "9997                                                                  @doomcrew_glen awwwww i'm so sorry.  i bet she's looking down on you&lt;33   \n",
              "9998                                                              @bocagirlslayed followed me thanks, and\\n@justinbieber please followed me too    \n",
              "9999        i miss being a kid. i can't believe i'm doing adult stuff like paying rent and bills and doing a degree and shit like . the fuck??!?   \n",
              "\n",
              "      label  \n",
              "9950      0  \n",
              "9951      1  \n",
              "9952      1  \n",
              "9953      0  \n",
              "9954      0  \n",
              "9955      0  \n",
              "9956      0  \n",
              "9957      1  \n",
              "9958      1  \n",
              "9959      0  \n",
              "9960      1  \n",
              "9961      0  \n",
              "9962      1  \n",
              "9963      0  \n",
              "9964      0  \n",
              "9965      1  \n",
              "9966      1  \n",
              "9967      1  \n",
              "9968      1  \n",
              "9969      0  \n",
              "9970      1  \n",
              "9971      1  \n",
              "9972      0  \n",
              "9973      0  \n",
              "9974      1  \n",
              "9975      0  \n",
              "9976      1  \n",
              "9977      0  \n",
              "9978      0  \n",
              "9979      0  \n",
              "9980      0  \n",
              "9981      0  \n",
              "9982      1  \n",
              "9983      0  \n",
              "9984      1  \n",
              "9985      0  \n",
              "9986      1  \n",
              "9987      1  \n",
              "9988      1  \n",
              "9989      1  \n",
              "9990      0  \n",
              "9991      0  \n",
              "9992      1  \n",
              "9993      1  \n",
              "9994      0  \n",
              "9995      0  \n",
              "9996      1  \n",
              "9997      0  \n",
              "9998      0  \n",
              "9999      0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-d0110bca-53a1-4b54-aba2-99e1580d4b46\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>9950</th>\n",
              "      <td>@lawrenceispichu oh gosh what did you say? and aw hun  *cuddles*</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9951</th>\n",
              "      <td>@diongzons  follow @jnlazts &amp;amp;  follow u back</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9952</th>\n",
              "      <td>@cath_tyldesley @yourstylist it is always good to stand out from the crowd</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9953</th>\n",
              "      <td>so much editing to do. so little time.</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9954</th>\n",
              "      <td>@iloveksoo we almost got to see his cute ankles but his socks</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9955</th>\n",
              "      <td>@lukebryanonline yayyyy!!! i hope it's not while i am knocked out by anesthesia. i will be so sad if i miss it</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9956</th>\n",
              "      <td>@tmhsposey that was the account that got hacked</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9957</th>\n",
              "      <td>shadowplaylouis we recently got this mutual and i'm really loving your tweets and everything your about so yay that's exciting</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9958</th>\n",
              "      <td>@aangelayap easy</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9959</th>\n",
              "      <td>phone fell out of my hand and it woke me up</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9960</th>\n",
              "      <td>@jackjackjohnson ended up happy.</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9961</th>\n",
              "      <td>who even started this trend? i wanna know if there is a jot of truth  #zayniscomingbackonjuly26</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9962</th>\n",
              "      <td>here it is, our #crowdfunding campaign! if we get our goal, we can walk on with this dream. will you help us?</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9963</th>\n",
              "      <td>@holdyholdy cotton candy is my favourite too!  time to stock up again!</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9964</th>\n",
              "      <td>@craziestmocha i wish</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9965</th>\n",
              "      <td>happy birthday nawazuddin siddique...\\nbig, big fan of yours.</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9966</th>\n",
              "      <td>stats for the week have arrived. 1 new follower and no unfollowers  via</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9967</th>\n",
              "      <td>@samm_amberr then there will be a few \"wtf this never happened?!!\" moments that you may or may not like. but there's a surprise thrown in</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9968</th>\n",
              "      <td>@emis_nug indeed. thanks for link to article</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9969</th>\n",
              "      <td>not an apology me encanta vale osea bea</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9970</th>\n",
              "      <td>i added a video to a @youtube playlist  im back on twitch and today it going to be league  - 1 / 3</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9971</th>\n",
              "      <td>i love it how you tell every crowd that they're the loudest. we all are the loudest fans in the world aren't we?  x @real_liam_payne</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9972</th>\n",
              "      <td>@jsjnete dd leave lagi?</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9973</th>\n",
              "      <td>vidcon ((((((((</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9974</th>\n",
              "      <td>@opera_north love it! have a #fabfriday</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9975</th>\n",
              "      <td>@awfullyariana welcome!! and aw  i have to go back next month but rn tryna get healthy</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9976</th>\n",
              "      <td>@little_nells happy birthday</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9977</th>\n",
              "      <td>so i guess blocking number on phones spam sms feature wont work. nor will using specific words since these guys write sinhala in english</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9978</th>\n",
              "      <td>you need to come back to england....  @macharmon</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9979</th>\n",
              "      <td>@magicmoon_rp i wanna watch them again too  maybe when they do comeback shows in korea... or in japan. i wanna visit your country</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9980</th>\n",
              "      <td>@y_a_t_e_s sorry about this. it could be a schedule change</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9981</th>\n",
              "      <td>@ranaptx_ but who will lead us when you're gone (</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9982</th>\n",
              "      <td>still awake</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9983</th>\n",
              "      <td>@howdydollie i need you by</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9984</th>\n",
              "      <td>@stepintolifeyvl amazing news! enjoy your toy</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9985</th>\n",
              "      <td>@kendrahatesu followed me thanks, and\\n@justinbieber please followed me too</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9986</th>\n",
              "      <td>@johnsonef184562  you're very welcome and than you</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9987</th>\n",
              "      <td>@ribster13 she can walk. barely, according to her that is  all good here. hope t got home all ok</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9988</th>\n",
              "      <td>@lauraalyceh @weimingkam and by turning up, to be fair! it meant a lot that people came.</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9989</th>\n",
              "      <td>@kayceecreates @artiecraftie you guys have a great weekend too!</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9990</th>\n",
              "      <td>\"i know you said this is michael jackson, but this is obviously a little boy\" - my little brother while listening to a jackson 5 song</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9991</th>\n",
              "      <td>my face is so skinny in that picture omg  why did i get fat</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9992</th>\n",
              "      <td>agree??   #phone #wifi #lifestyle \\n\\n#qatarday</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9993</th>\n",
              "      <td>@heyclaireee is back! thnx god!!! i'm so happy</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9994</th>\n",
              "      <td>always fall asleep so early when i'm texting someone</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9995</th>\n",
              "      <td>@luvsgngrhd me too (( where is margo</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9996</th>\n",
              "      <td>@markbowthai @twomeanyoung it's 6:15pm rn!! and it's currently 70 degrees fahrenheit!! which is really warm so it's good</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9997</th>\n",
              "      <td>@doomcrew_glen awwwww i'm so sorry.  i bet she's looking down on you&amp;lt;33</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9998</th>\n",
              "      <td>@bocagirlslayed followed me thanks, and\\n@justinbieber please followed me too</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9999</th>\n",
              "      <td>i miss being a kid. i can't believe i'm doing adult stuff like paying rent and bills and doing a degree and shit like . the fuck??!?</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d0110bca-53a1-4b54-aba2-99e1580d4b46')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-d0110bca-53a1-4b54-aba2-99e1580d4b46 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-d0110bca-53a1-4b54-aba2-99e1580d4b46');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-3cdf29f5-cc0b-4ab0-97f5-e77f88f2413d\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-3cdf29f5-cc0b-4ab0-97f5-e77f88f2413d')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-3cdf29f5-cc0b-4ab0-97f5-e77f88f2413d button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 50,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 50,\n        \"samples\": [\n          \"@holdyholdy cotton candy is my favourite too!  time to stock up again!\",\n          \"@kayceecreates @artiecraftie you guys have a great weekend too!\",\n          \"@y_a_t_e_s sorry about this. it could be a schedule change \"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 160
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sD_0midkJN6R"
      },
      "source": [
        "### Hashtags and Mentions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HOKHQkszJN6S"
      },
      "source": [
        "We are habituated to use hashtags and mentions in our tweet either to indicate the context or bring attention to an individual. Hashtags can be used to extract features, to see what's trending and in various other applications.\n",
        "\n",
        "Since, we don't require them we'll remove them."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 161,
      "metadata": {
        "id": "L6XAXZS_JN6S"
      },
      "outputs": [],
      "source": [
        "def remove_tags_mentions(text):\n",
        "    pattern = re.compile(r'(@\\S+|#\\S+)')\n",
        "    return pattern.sub('', text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 162,
      "metadata": {
        "id": "M4AAyF7-JN6S",
        "outputId": "1f9bbc76-5598-4f42-93fc-254f6cad1741",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'live  on  - jonah and jareddddd'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 162
        }
      ],
      "source": [
        "text = \"live @flippinginja on #younow - jonah and jareddddd\"\n",
        "remove_tags_mentions(text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 163,
      "metadata": {
        "id": "lImccD57JN6T"
      },
      "outputs": [],
      "source": [
        "# Remove hashtags and mentions\n",
        "df.text = df.text.apply(lambda x: remove_tags_mentions(x))"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.tail(50)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "kcxf56o4bm0Z",
        "outputId": "b5f58eaa-2457-46b8-8446-5a18cbf30bb1"
      },
      "execution_count": 164,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                           text  \\\n",
              "9950                                                                                            oh gosh what did you say? and aw hun  *cuddles*   \n",
              "9951                                                                                                              follow  &amp;  follow u back    \n",
              "9952                                                                                             it is always good to stand out from the crowd    \n",
              "9953                                                                                                    so much editing to do. so little time.    \n",
              "9954                                                                                        we almost got to see his cute ankles but his socks    \n",
              "9955                                             yayyyy!!! i hope it's not while i am knocked out by anesthesia. i will be so sad if i miss it    \n",
              "9956                                                                                                      that was the account that got hacked    \n",
              "9957            shadowplaylouis we recently got this mutual and i'm really loving your tweets and everything your about so yay that's exciting    \n",
              "9958                                                                                                                                      easy    \n",
              "9959                                                                                               phone fell out of my hand and it woke me up    \n",
              "9960                                                                                                                           ended up happy.    \n",
              "9961                                                                     who even started this trend? i wanna know if there is a jot of truth     \n",
              "9962                                         here it is, our  campaign! if we get our goal, we can walk on with this dream. will you help us?     \n",
              "9963                                                                                 cotton candy is my favourite too!  time to stock up again!   \n",
              "9964                                                                                                                                    i wish    \n",
              "9965                                                                             happy birthday nawazuddin siddique...\\nbig, big fan of yours.    \n",
              "9966                                                                   stats for the week have arrived. 1 new follower and no unfollowers  via    \n",
              "9967              then there will be a few \"wtf this never happened?!!\" moments that you may or may not like. but there's a surprise thrown in    \n",
              "9968                                                                                                        indeed. thanks for link to article    \n",
              "9969                                                                                                   not an apology me encanta vale osea bea    \n",
              "9970                                                 i added a video to a  playlist  im back on twitch and today it going to be league  - 1 / 3   \n",
              "9971                       i love it how you tell every crowd that they're the loudest. we all are the loudest fans in the world aren't we?  x    \n",
              "9972                                                                                                                             dd leave lagi?   \n",
              "9973                                                                                                                            vidcon ((((((((   \n",
              "9974                                                                                                                          love it! have a     \n",
              "9975                                                                    welcome!! and aw  i have to go back next month but rn tryna get healthy   \n",
              "9976                                                                                                                            happy birthday    \n",
              "9977  so i guess blocking number on phones spam sms feature wont work. nor will using specific words since these guys write sinhala in english    \n",
              "9978                                                                                                     you need to come back to england....     \n",
              "9979                        i wanna watch them again too  maybe when they do comeback shows in korea... or in japan. i wanna visit your country   \n",
              "9980                                                                                           sorry about this. it could be a schedule change    \n",
              "9981                                                                                                    but who will lead us when you're gone (   \n",
              "9982                                                                                                                               still awake    \n",
              "9983                                                                                                                             i need you by    \n",
              "9984                                                                                                              amazing news! enjoy your toy    \n",
              "9985                                                                                          followed me thanks, and\\n please followed me too    \n",
              "9986                                                                                                          you're very welcome and than you    \n",
              "9987                                                     she can walk. barely, according to her that is  all good here. hope t got home all ok    \n",
              "9988                                                                           and by turning up, to be fair! it meant a lot that people came.    \n",
              "9989                                                                                                         you guys have a great weekend too!   \n",
              "9990     \"i know you said this is michael jackson, but this is obviously a little boy\" - my little brother while listening to a jackson 5 song    \n",
              "9991                                                                                my face is so skinny in that picture omg  why did i get fat   \n",
              "9992                                                                                                                         agree??      \\n\\n    \n",
              "9993                                                                                                         is back! thnx god!!! i'm so happy    \n",
              "9994                                                                                      always fall asleep so early when i'm texting someone    \n",
              "9995                                                                                                                   me too (( where is margo   \n",
              "9996                                             it's 6:15pm rn!! and it's currently 70 degrees fahrenheit!! which is really warm so it's good    \n",
              "9997                                                                                awwwww i'm so sorry.  i bet she's looking down on you&lt;33   \n",
              "9998                                                                                          followed me thanks, and\\n please followed me too    \n",
              "9999       i miss being a kid. i can't believe i'm doing adult stuff like paying rent and bills and doing a degree and shit like . the fuck??!?   \n",
              "\n",
              "      label  \n",
              "9950      0  \n",
              "9951      1  \n",
              "9952      1  \n",
              "9953      0  \n",
              "9954      0  \n",
              "9955      0  \n",
              "9956      0  \n",
              "9957      1  \n",
              "9958      1  \n",
              "9959      0  \n",
              "9960      1  \n",
              "9961      0  \n",
              "9962      1  \n",
              "9963      0  \n",
              "9964      0  \n",
              "9965      1  \n",
              "9966      1  \n",
              "9967      1  \n",
              "9968      1  \n",
              "9969      0  \n",
              "9970      1  \n",
              "9971      1  \n",
              "9972      0  \n",
              "9973      0  \n",
              "9974      1  \n",
              "9975      0  \n",
              "9976      1  \n",
              "9977      0  \n",
              "9978      0  \n",
              "9979      0  \n",
              "9980      0  \n",
              "9981      0  \n",
              "9982      1  \n",
              "9983      0  \n",
              "9984      1  \n",
              "9985      0  \n",
              "9986      1  \n",
              "9987      1  \n",
              "9988      1  \n",
              "9989      1  \n",
              "9990      0  \n",
              "9991      0  \n",
              "9992      1  \n",
              "9993      1  \n",
              "9994      0  \n",
              "9995      0  \n",
              "9996      1  \n",
              "9997      0  \n",
              "9998      0  \n",
              "9999      0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-9fd2a1d9-3897-4085-8c68-f3add181bd76\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>9950</th>\n",
              "      <td>oh gosh what did you say? and aw hun  *cuddles*</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9951</th>\n",
              "      <td>follow  &amp;amp;  follow u back</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9952</th>\n",
              "      <td>it is always good to stand out from the crowd</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9953</th>\n",
              "      <td>so much editing to do. so little time.</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9954</th>\n",
              "      <td>we almost got to see his cute ankles but his socks</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9955</th>\n",
              "      <td>yayyyy!!! i hope it's not while i am knocked out by anesthesia. i will be so sad if i miss it</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9956</th>\n",
              "      <td>that was the account that got hacked</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9957</th>\n",
              "      <td>shadowplaylouis we recently got this mutual and i'm really loving your tweets and everything your about so yay that's exciting</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9958</th>\n",
              "      <td>easy</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9959</th>\n",
              "      <td>phone fell out of my hand and it woke me up</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9960</th>\n",
              "      <td>ended up happy.</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9961</th>\n",
              "      <td>who even started this trend? i wanna know if there is a jot of truth</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9962</th>\n",
              "      <td>here it is, our  campaign! if we get our goal, we can walk on with this dream. will you help us?</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9963</th>\n",
              "      <td>cotton candy is my favourite too!  time to stock up again!</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9964</th>\n",
              "      <td>i wish</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9965</th>\n",
              "      <td>happy birthday nawazuddin siddique...\\nbig, big fan of yours.</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9966</th>\n",
              "      <td>stats for the week have arrived. 1 new follower and no unfollowers  via</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9967</th>\n",
              "      <td>then there will be a few \"wtf this never happened?!!\" moments that you may or may not like. but there's a surprise thrown in</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9968</th>\n",
              "      <td>indeed. thanks for link to article</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9969</th>\n",
              "      <td>not an apology me encanta vale osea bea</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9970</th>\n",
              "      <td>i added a video to a  playlist  im back on twitch and today it going to be league  - 1 / 3</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9971</th>\n",
              "      <td>i love it how you tell every crowd that they're the loudest. we all are the loudest fans in the world aren't we?  x</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9972</th>\n",
              "      <td>dd leave lagi?</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9973</th>\n",
              "      <td>vidcon ((((((((</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9974</th>\n",
              "      <td>love it! have a</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9975</th>\n",
              "      <td>welcome!! and aw  i have to go back next month but rn tryna get healthy</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9976</th>\n",
              "      <td>happy birthday</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9977</th>\n",
              "      <td>so i guess blocking number on phones spam sms feature wont work. nor will using specific words since these guys write sinhala in english</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9978</th>\n",
              "      <td>you need to come back to england....</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9979</th>\n",
              "      <td>i wanna watch them again too  maybe when they do comeback shows in korea... or in japan. i wanna visit your country</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9980</th>\n",
              "      <td>sorry about this. it could be a schedule change</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9981</th>\n",
              "      <td>but who will lead us when you're gone (</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9982</th>\n",
              "      <td>still awake</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9983</th>\n",
              "      <td>i need you by</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9984</th>\n",
              "      <td>amazing news! enjoy your toy</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9985</th>\n",
              "      <td>followed me thanks, and\\n please followed me too</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9986</th>\n",
              "      <td>you're very welcome and than you</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9987</th>\n",
              "      <td>she can walk. barely, according to her that is  all good here. hope t got home all ok</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9988</th>\n",
              "      <td>and by turning up, to be fair! it meant a lot that people came.</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9989</th>\n",
              "      <td>you guys have a great weekend too!</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9990</th>\n",
              "      <td>\"i know you said this is michael jackson, but this is obviously a little boy\" - my little brother while listening to a jackson 5 song</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9991</th>\n",
              "      <td>my face is so skinny in that picture omg  why did i get fat</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9992</th>\n",
              "      <td>agree??      \\n\\n</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9993</th>\n",
              "      <td>is back! thnx god!!! i'm so happy</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9994</th>\n",
              "      <td>always fall asleep so early when i'm texting someone</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9995</th>\n",
              "      <td>me too (( where is margo</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9996</th>\n",
              "      <td>it's 6:15pm rn!! and it's currently 70 degrees fahrenheit!! which is really warm so it's good</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9997</th>\n",
              "      <td>awwwww i'm so sorry.  i bet she's looking down on you&amp;lt;33</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9998</th>\n",
              "      <td>followed me thanks, and\\n please followed me too</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9999</th>\n",
              "      <td>i miss being a kid. i can't believe i'm doing adult stuff like paying rent and bills and doing a degree and shit like . the fuck??!?</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9fd2a1d9-3897-4085-8c68-f3add181bd76')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-9fd2a1d9-3897-4085-8c68-f3add181bd76 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-9fd2a1d9-3897-4085-8c68-f3add181bd76');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-ba4d1bd0-a5b6-40ef-a08e-da5b4fee7fe8\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-ba4d1bd0-a5b6-40ef-a08e-da5b4fee7fe8')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-ba4d1bd0-a5b6-40ef-a08e-da5b4fee7fe8 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 50,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 49,\n        \"samples\": [\n          \" cotton candy is my favourite too!  time to stock up again!\",\n          \" me too (( where is margo\",\n          \" awwwww i'm so sorry.  i bet she's looking down on you&lt;33\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 164
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jrqY21SJJN6T"
      },
      "source": [
        "### Punctuations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nLr5jMeRJN6U"
      },
      "source": [
        "Punctuations are character other than alphaters and digits. These include [!\"#$%&\\'()*+,-./:;<=>?@\\\\^_`{|}~]\n",
        "\n",
        "It is better remove or convert emoticons before removing the punctuations, since if we do the other we around we might loose the emoticons from the text. Another example, if the text contains $10.50 then we'll remove the .(dot) and the value will loose it's meaning."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 165,
      "metadata": {
        "id": "oHzOQl_5JN6X"
      },
      "outputs": [],
      "source": [
        "PUNCTUATIONS = string.punctuation\n",
        "\n",
        "def remove_punctuation(text):\n",
        "    return text.translate(str.maketrans('', '', PUNCTUATIONS))"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "PUNCTUATIONS"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "cJwzA2J0cPrc",
        "outputId": "bf146eca-4dfc-4687-e19f-156509eb5019"
      },
      "execution_count": 166,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'!\"#$%&\\'()*+,-./:;<=>?@[\\\\]^_`{|}~'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 166
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 167,
      "metadata": {
        "id": "Zm-25C7fJN6Y"
      },
      "outputs": [],
      "source": [
        "df.text = df[\"text\"].apply(lambda text: remove_punctuation(text))"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.tail(50)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "2EEYu0EmcY0c",
        "outputId": "0eb6aa4e-4f36-489a-9107-2453f5e54b0f"
      },
      "execution_count": 168,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                          text  \\\n",
              "9950                                                                                              oh gosh what did you say and aw hun  cuddles   \n",
              "9951                                                                                                               follow  amp  follow u back    \n",
              "9952                                                                                            it is always good to stand out from the crowd    \n",
              "9953                                                                                                     so much editing to do so little time    \n",
              "9954                                                                                       we almost got to see his cute ankles but his socks    \n",
              "9955                                                 yayyyy i hope its not while i am knocked out by anesthesia i will be so sad if i miss it    \n",
              "9956                                                                                                     that was the account that got hacked    \n",
              "9957             shadowplaylouis we recently got this mutual and im really loving your tweets and everything your about so yay thats exciting    \n",
              "9958                                                                                                                                     easy    \n",
              "9959                                                                                              phone fell out of my hand and it woke me up    \n",
              "9960                                                                                                                           ended up happy    \n",
              "9961                                                                     who even started this trend i wanna know if there is a jot of truth     \n",
              "9962                                             here it is our  campaign if we get our goal we can walk on with this dream will you help us     \n",
              "9963                                                                                  cotton candy is my favourite too  time to stock up again   \n",
              "9964                                                                                                                                   i wish    \n",
              "9965                                                                                 happy birthday nawazuddin siddique\\nbig big fan of yours    \n",
              "9966                                                                   stats for the week have arrived 1 new follower and no unfollowers  via    \n",
              "9967                    then there will be a few wtf this never happened moments that you may or may not like but theres a surprise thrown in    \n",
              "9968                                                                                                        indeed thanks for link to article    \n",
              "9969                                                                                                  not an apology me encanta vale osea bea    \n",
              "9970                                                  i added a video to a  playlist  im back on twitch and today it going to be league   1  3   \n",
              "9971                          i love it how you tell every crowd that theyre the loudest we all are the loudest fans in the world arent we  x    \n",
              "9972                                                                                                                             dd leave lagi   \n",
              "9973                                                                                                                                   vidcon    \n",
              "9974                                                                                                                          love it have a     \n",
              "9975                                                                     welcome and aw  i have to go back next month but rn tryna get healthy   \n",
              "9976                                                                                                                           happy birthday    \n",
              "9977  so i guess blocking number on phones spam sms feature wont work nor will using specific words since these guys write sinhala in english    \n",
              "9978                                                                                                        you need to come back to england     \n",
              "9979                           i wanna watch them again too  maybe when they do comeback shows in korea or in japan i wanna visit your country   \n",
              "9980                                                                                           sorry about this it could be a schedule change    \n",
              "9981                                                                                                     but who will lead us when youre gone    \n",
              "9982                                                                                                                              still awake    \n",
              "9983                                                                                                                            i need you by    \n",
              "9984                                                                                                              amazing news enjoy your toy    \n",
              "9985                                                                                          followed me thanks and\\n please followed me too    \n",
              "9986                                                                                                          youre very welcome and than you    \n",
              "9987                                                       she can walk barely according to her that is  all good here hope t got home all ok    \n",
              "9988                                                                             and by turning up to be fair it meant a lot that people came    \n",
              "9989                                                                                                         you guys have a great weekend too   \n",
              "9990        i know you said this is michael jackson but this is obviously a little boy  my little brother while listening to a jackson 5 song    \n",
              "9991                                                                               my face is so skinny in that picture omg  why did i get fat   \n",
              "9992                                                                                                                          agree      \\n\\n    \n",
              "9993                                                                                                             is back thnx god im so happy    \n",
              "9994                                                                                      always fall asleep so early when im texting someone    \n",
              "9995                                                                                                                    me too  where is margo   \n",
              "9996                                                    its 615pm rn and its currently 70 degrees fahrenheit which is really warm so its good    \n",
              "9997                                                                                    awwwww im so sorry  i bet shes looking down on yoult33   \n",
              "9998                                                                                          followed me thanks and\\n please followed me too    \n",
              "9999              i miss being a kid i cant believe im doing adult stuff like paying rent and bills and doing a degree and shit like  the fuck   \n",
              "\n",
              "      label  \n",
              "9950      0  \n",
              "9951      1  \n",
              "9952      1  \n",
              "9953      0  \n",
              "9954      0  \n",
              "9955      0  \n",
              "9956      0  \n",
              "9957      1  \n",
              "9958      1  \n",
              "9959      0  \n",
              "9960      1  \n",
              "9961      0  \n",
              "9962      1  \n",
              "9963      0  \n",
              "9964      0  \n",
              "9965      1  \n",
              "9966      1  \n",
              "9967      1  \n",
              "9968      1  \n",
              "9969      0  \n",
              "9970      1  \n",
              "9971      1  \n",
              "9972      0  \n",
              "9973      0  \n",
              "9974      1  \n",
              "9975      0  \n",
              "9976      1  \n",
              "9977      0  \n",
              "9978      0  \n",
              "9979      0  \n",
              "9980      0  \n",
              "9981      0  \n",
              "9982      1  \n",
              "9983      0  \n",
              "9984      1  \n",
              "9985      0  \n",
              "9986      1  \n",
              "9987      1  \n",
              "9988      1  \n",
              "9989      1  \n",
              "9990      0  \n",
              "9991      0  \n",
              "9992      1  \n",
              "9993      1  \n",
              "9994      0  \n",
              "9995      0  \n",
              "9996      1  \n",
              "9997      0  \n",
              "9998      0  \n",
              "9999      0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-706a22d2-73dd-471a-96fa-2f8fc3d4a4ab\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>9950</th>\n",
              "      <td>oh gosh what did you say and aw hun  cuddles</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9951</th>\n",
              "      <td>follow  amp  follow u back</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9952</th>\n",
              "      <td>it is always good to stand out from the crowd</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9953</th>\n",
              "      <td>so much editing to do so little time</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9954</th>\n",
              "      <td>we almost got to see his cute ankles but his socks</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9955</th>\n",
              "      <td>yayyyy i hope its not while i am knocked out by anesthesia i will be so sad if i miss it</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9956</th>\n",
              "      <td>that was the account that got hacked</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9957</th>\n",
              "      <td>shadowplaylouis we recently got this mutual and im really loving your tweets and everything your about so yay thats exciting</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9958</th>\n",
              "      <td>easy</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9959</th>\n",
              "      <td>phone fell out of my hand and it woke me up</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9960</th>\n",
              "      <td>ended up happy</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9961</th>\n",
              "      <td>who even started this trend i wanna know if there is a jot of truth</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9962</th>\n",
              "      <td>here it is our  campaign if we get our goal we can walk on with this dream will you help us</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9963</th>\n",
              "      <td>cotton candy is my favourite too  time to stock up again</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9964</th>\n",
              "      <td>i wish</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9965</th>\n",
              "      <td>happy birthday nawazuddin siddique\\nbig big fan of yours</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9966</th>\n",
              "      <td>stats for the week have arrived 1 new follower and no unfollowers  via</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9967</th>\n",
              "      <td>then there will be a few wtf this never happened moments that you may or may not like but theres a surprise thrown in</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9968</th>\n",
              "      <td>indeed thanks for link to article</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9969</th>\n",
              "      <td>not an apology me encanta vale osea bea</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9970</th>\n",
              "      <td>i added a video to a  playlist  im back on twitch and today it going to be league   1  3</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9971</th>\n",
              "      <td>i love it how you tell every crowd that theyre the loudest we all are the loudest fans in the world arent we  x</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9972</th>\n",
              "      <td>dd leave lagi</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9973</th>\n",
              "      <td>vidcon</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9974</th>\n",
              "      <td>love it have a</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9975</th>\n",
              "      <td>welcome and aw  i have to go back next month but rn tryna get healthy</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9976</th>\n",
              "      <td>happy birthday</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9977</th>\n",
              "      <td>so i guess blocking number on phones spam sms feature wont work nor will using specific words since these guys write sinhala in english</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9978</th>\n",
              "      <td>you need to come back to england</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9979</th>\n",
              "      <td>i wanna watch them again too  maybe when they do comeback shows in korea or in japan i wanna visit your country</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9980</th>\n",
              "      <td>sorry about this it could be a schedule change</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9981</th>\n",
              "      <td>but who will lead us when youre gone</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9982</th>\n",
              "      <td>still awake</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9983</th>\n",
              "      <td>i need you by</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9984</th>\n",
              "      <td>amazing news enjoy your toy</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9985</th>\n",
              "      <td>followed me thanks and\\n please followed me too</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9986</th>\n",
              "      <td>youre very welcome and than you</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9987</th>\n",
              "      <td>she can walk barely according to her that is  all good here hope t got home all ok</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9988</th>\n",
              "      <td>and by turning up to be fair it meant a lot that people came</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9989</th>\n",
              "      <td>you guys have a great weekend too</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9990</th>\n",
              "      <td>i know you said this is michael jackson but this is obviously a little boy  my little brother while listening to a jackson 5 song</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9991</th>\n",
              "      <td>my face is so skinny in that picture omg  why did i get fat</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9992</th>\n",
              "      <td>agree      \\n\\n</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9993</th>\n",
              "      <td>is back thnx god im so happy</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9994</th>\n",
              "      <td>always fall asleep so early when im texting someone</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9995</th>\n",
              "      <td>me too  where is margo</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9996</th>\n",
              "      <td>its 615pm rn and its currently 70 degrees fahrenheit which is really warm so its good</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9997</th>\n",
              "      <td>awwwww im so sorry  i bet shes looking down on yoult33</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9998</th>\n",
              "      <td>followed me thanks and\\n please followed me too</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9999</th>\n",
              "      <td>i miss being a kid i cant believe im doing adult stuff like paying rent and bills and doing a degree and shit like  the fuck</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-706a22d2-73dd-471a-96fa-2f8fc3d4a4ab')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-706a22d2-73dd-471a-96fa-2f8fc3d4a4ab button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-706a22d2-73dd-471a-96fa-2f8fc3d4a4ab');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-cb79ed2f-23ed-44e2-8311-f8720128187a\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-cb79ed2f-23ed-44e2-8311-f8720128187a')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-cb79ed2f-23ed-44e2-8311-f8720128187a button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 50,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 49,\n        \"samples\": [\n          \" cotton candy is my favourite too  time to stock up again\",\n          \" me too  where is margo\",\n          \" awwwww im so sorry  i bet shes looking down on yoult33\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 168
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Y7OhzhjJN6Y"
      },
      "source": [
        "### Stopwords"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HpdxJAZ3JN6Y"
      },
      "source": [
        "Stopwords are commonly occuring words in any language. Such as, in english these words are 'the', 'a', 'an', & many more. They are in most cases not useful and should be removed.\n",
        "\n",
        "There are certain tasks in which these words are useful such as Part-of-Speech(POS) tagging, language translation. Stopwords are compiled for many languages, for english language we can use the list from the nltk package."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 170,
      "metadata": {
        "id": "3kAG4Y68JN6Z",
        "outputId": "74a987f6-2bf6-427e-c210-9c962cfee1f5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n"
          ]
        }
      ],
      "source": [
        "nltk.download('stopwords')\n",
        "STOPWORDS = set(stopwords.words('english'))\n",
        "\n",
        "def remove_stopwords(text):\n",
        "    return ' '.join([word for word in text.split() if word not in STOPWORDS])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 171,
      "metadata": {
        "id": "dbfDaHV4JN6Z",
        "outputId": "5154597e-c52e-4766-a335-6d1ecd6285e8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 677
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                           text  \\\n",
              "0                                    aww poor thing hope itokay good health luckely freed rocks   \n",
              "1                                  hi ive spoken store advised paypoint card topped thanks beth   \n",
              "2                                                              could never go ever seeing cruel   \n",
              "3                                                                   bus late time get food work   \n",
              "4                                                                   another ridiculous headache   \n",
              "5                                                                               kik oulive70748   \n",
              "6                                        beyond words sooooooo freaking good im going die comes   \n",
              "7                                                                                 please notice   \n",
              "8                                                                                          babe   \n",
              "9                                                                                    definitely   \n",
              "10  ♛♛♛ 》》》》 love much beli̇eve wi̇ll follow please follow please justi̇n x15331 》》》》ｓｅｅ ｍｅ ♛♛♛   \n",
              "11                                                                           indeed many thanks   \n",
              "12                                             dont get tbh p ellie superpowers live im happy p   \n",
              "13                                                            anyone give money train see today   \n",
              "14                                                                              im still hoping   \n",
              "15                              photoset katmtan swan queen fandom sorry others blurry far away   \n",
              "16                                                                                    pleased …   \n",
              "17                    hey guys ill live 30 mins want fun low ranked games subgames faceit today   \n",
              "18                                                                                would like xo   \n",
              "19                                                                                still working   \n",
              "\n",
              "    label  \n",
              "0       0  \n",
              "1       1  \n",
              "2       0  \n",
              "3       0  \n",
              "4       0  \n",
              "5       0  \n",
              "6       1  \n",
              "7       0  \n",
              "8       0  \n",
              "9       1  \n",
              "10      0  \n",
              "11      1  \n",
              "12      1  \n",
              "13      0  \n",
              "14      0  \n",
              "15      1  \n",
              "16      1  \n",
              "17      1  \n",
              "18      1  \n",
              "19      0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-3f998ab6-6010-4253-bfb2-03dc0a675020\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>aww poor thing hope itokay good health luckely freed rocks</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>hi ive spoken store advised paypoint card topped thanks beth</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>could never go ever seeing cruel</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>bus late time get food work</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>another ridiculous headache</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>kik oulive70748</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>beyond words sooooooo freaking good im going die comes</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>please notice</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>babe</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>definitely</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>♛♛♛ 》》》》 love much beli̇eve wi̇ll follow please follow please justi̇n x15331 》》》》ｓｅｅ ｍｅ ♛♛♛</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>indeed many thanks</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>dont get tbh p ellie superpowers live im happy p</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>anyone give money train see today</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>im still hoping</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>photoset katmtan swan queen fandom sorry others blurry far away</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>pleased …</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>hey guys ill live 30 mins want fun low ranked games subgames faceit today</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>would like xo</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>still working</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3f998ab6-6010-4253-bfb2-03dc0a675020')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-3f998ab6-6010-4253-bfb2-03dc0a675020 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-3f998ab6-6010-4253-bfb2-03dc0a675020');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-8b4e6e41-1b55-46b4-abe3-85a70a8bc4dd\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-8b4e6e41-1b55-46b4-abe3-85a70a8bc4dd')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-8b4e6e41-1b55-46b4-abe3-85a70a8bc4dd button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 10000,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 9030,\n        \"samples\": [\n          \"missed 1111 btw wish follow\",\n          \"found amp flow thanks sharing already passed like\",\n          \"unable login last 4 hours response support team either support case 00128835 bad day\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 171
        }
      ],
      "source": [
        "# Remove stopwords\n",
        "df.text = df.text.apply(lambda text: remove_stopwords(text))\n",
        "df.head(20)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.tail(20)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 677
        },
        "id": "8ZNtdAHesPwm",
        "outputId": "7723778f-b061-42d1-d3b3-2de9796b9a80"
      },
      "execution_count": 172,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                        text  \\\n",
              "9980                                                             sorry could schedule change   \n",
              "9981                                                                      lead us youre gone   \n",
              "9982                                                                             still awake   \n",
              "9983                                                                                    need   \n",
              "9984                                                                  amazing news enjoy toy   \n",
              "9985                                                         followed thanks please followed   \n",
              "9986                                                                           youre welcome   \n",
              "9987                                             walk barely according good hope got home ok   \n",
              "9988                                                      turning fair meant lot people came   \n",
              "9989                                                                      guys great weekend   \n",
              "9990  know said michael jackson obviously little boy little brother listening jackson 5 song   \n",
              "9991                                                         face skinny picture omg get fat   \n",
              "9992                                                                                   agree   \n",
              "9993                                                                  back thnx god im happy   \n",
              "9994                                             always fall asleep early im texting someone   \n",
              "9995                                                                                   margo   \n",
              "9996                               615pm rn currently 70 degrees fahrenheit really warm good   \n",
              "9997                                                awwwww im sorry bet shes looking yoult33   \n",
              "9998                                                         followed thanks please followed   \n",
              "9999       miss kid cant believe im adult stuff like paying rent bills degree shit like fuck   \n",
              "\n",
              "      label  \n",
              "9980      0  \n",
              "9981      0  \n",
              "9982      1  \n",
              "9983      0  \n",
              "9984      1  \n",
              "9985      0  \n",
              "9986      1  \n",
              "9987      1  \n",
              "9988      1  \n",
              "9989      1  \n",
              "9990      0  \n",
              "9991      0  \n",
              "9992      1  \n",
              "9993      1  \n",
              "9994      0  \n",
              "9995      0  \n",
              "9996      1  \n",
              "9997      0  \n",
              "9998      0  \n",
              "9999      0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-9840f899-6b5f-40f7-b03b-9c1c2a937053\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>9980</th>\n",
              "      <td>sorry could schedule change</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9981</th>\n",
              "      <td>lead us youre gone</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9982</th>\n",
              "      <td>still awake</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9983</th>\n",
              "      <td>need</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9984</th>\n",
              "      <td>amazing news enjoy toy</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9985</th>\n",
              "      <td>followed thanks please followed</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9986</th>\n",
              "      <td>youre welcome</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9987</th>\n",
              "      <td>walk barely according good hope got home ok</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9988</th>\n",
              "      <td>turning fair meant lot people came</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9989</th>\n",
              "      <td>guys great weekend</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9990</th>\n",
              "      <td>know said michael jackson obviously little boy little brother listening jackson 5 song</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9991</th>\n",
              "      <td>face skinny picture omg get fat</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9992</th>\n",
              "      <td>agree</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9993</th>\n",
              "      <td>back thnx god im happy</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9994</th>\n",
              "      <td>always fall asleep early im texting someone</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9995</th>\n",
              "      <td>margo</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9996</th>\n",
              "      <td>615pm rn currently 70 degrees fahrenheit really warm good</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9997</th>\n",
              "      <td>awwwww im sorry bet shes looking yoult33</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9998</th>\n",
              "      <td>followed thanks please followed</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9999</th>\n",
              "      <td>miss kid cant believe im adult stuff like paying rent bills degree shit like fuck</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9840f899-6b5f-40f7-b03b-9c1c2a937053')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-9840f899-6b5f-40f7-b03b-9c1c2a937053 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-9840f899-6b5f-40f7-b03b-9c1c2a937053');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-0ed1d8c4-f6fd-4873-bb0f-8e3cd325814b\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-0ed1d8c4-f6fd-4873-bb0f-8e3cd325814b')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-0ed1d8c4-f6fd-4873-bb0f-8e3cd325814b button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 20,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 19,\n        \"samples\": [\n          \"sorry could schedule change\",\n          \"followed thanks please followed\",\n          \"face skinny picture omg get fat\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 172
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ugwj7-RbJN6Z"
      },
      "source": [
        "### Numbers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mGoyAS06JN6a"
      },
      "source": [
        "We may remove numbers if they are not useful in our analysis. But analysis in the financial domain, numbers are very useful."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 173,
      "metadata": {
        "id": "AEqGzsjUJN6a"
      },
      "outputs": [],
      "source": [
        "df.text = df.text.str.replace(r'\\d+', '', regex=True)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.tail(50)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "R7FykpofdsJY",
        "outputId": "bfeefb0a-0e03-4520-fcdb-7a17ca1b4c44"
      },
      "execution_count": 175,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                               text  \\\n",
              "9950                                                                                     oh gosh say aw hun cuddles   \n",
              "9951                                                                                       follow amp follow u back   \n",
              "9952                                                                                        always good stand crowd   \n",
              "9953                                                                                       much editing little time   \n",
              "9954                                                                               almost got see cute ankles socks   \n",
              "9955                                                                        yayyyy hope knocked anesthesia sad miss   \n",
              "9956                                                                                             account got hacked   \n",
              "9957                      shadowplaylouis recently got mutual im really loving tweets everything yay thats exciting   \n",
              "9958                                                                                                           easy   \n",
              "9959                                                                                           phone fell hand woke   \n",
              "9960                                                                                                    ended happy   \n",
              "9961                                                                        even started trend wanna know jot truth   \n",
              "9962                                                                           campaign get goal walk dream help us   \n",
              "9963                                                                              cotton candy favourite time stock   \n",
              "9964                                                                                                           wish   \n",
              "9965                                                                 happy birthday nawazuddin siddique big big fan   \n",
              "9966                                                               stats week arrived  new follower unfollowers via   \n",
              "9967                                                 wtf never happened moments may may like theres surprise thrown   \n",
              "9968                                                                                     indeed thanks link article   \n",
              "9969                                                                                  apology encanta vale osea bea   \n",
              "9970                                                       added video playlist im back twitch today going league     \n",
              "9971                                                love tell every crowd theyre loudest loudest fans world arent x   \n",
              "9972                                                                                                  dd leave lagi   \n",
              "9973                                                                                                         vidcon   \n",
              "9974                                                                                                           love   \n",
              "9975                                                             welcome aw go back next month rn tryna get healthy   \n",
              "9976                                                                                                 happy birthday   \n",
              "9977  guess blocking number phones spam sms feature wont work using specific words since guys write sinhala english   \n",
              "9978                                                                                         need come back england   \n",
              "9979                                               wanna watch maybe comeback shows korea japan wanna visit country   \n",
              "9980                                                                                    sorry could schedule change   \n",
              "9981                                                                                             lead us youre gone   \n",
              "9982                                                                                                    still awake   \n",
              "9983                                                                                                           need   \n",
              "9984                                                                                         amazing news enjoy toy   \n",
              "9985                                                                                followed thanks please followed   \n",
              "9986                                                                                                  youre welcome   \n",
              "9987                                                                    walk barely according good hope got home ok   \n",
              "9988                                                                             turning fair meant lot people came   \n",
              "9989                                                                                             guys great weekend   \n",
              "9990                          know said michael jackson obviously little boy little brother listening jackson  song   \n",
              "9991                                                                                face skinny picture omg get fat   \n",
              "9992                                                                                                          agree   \n",
              "9993                                                                                         back thnx god im happy   \n",
              "9994                                                                    always fall asleep early im texting someone   \n",
              "9995                                                                                                          margo   \n",
              "9996                                                           pm rn currently  degrees fahrenheit really warm good   \n",
              "9997                                                                         awwwww im sorry bet shes looking yoult   \n",
              "9998                                                                                followed thanks please followed   \n",
              "9999                              miss kid cant believe im adult stuff like paying rent bills degree shit like fuck   \n",
              "\n",
              "      label  \n",
              "9950      0  \n",
              "9951      1  \n",
              "9952      1  \n",
              "9953      0  \n",
              "9954      0  \n",
              "9955      0  \n",
              "9956      0  \n",
              "9957      1  \n",
              "9958      1  \n",
              "9959      0  \n",
              "9960      1  \n",
              "9961      0  \n",
              "9962      1  \n",
              "9963      0  \n",
              "9964      0  \n",
              "9965      1  \n",
              "9966      1  \n",
              "9967      1  \n",
              "9968      1  \n",
              "9969      0  \n",
              "9970      1  \n",
              "9971      1  \n",
              "9972      0  \n",
              "9973      0  \n",
              "9974      1  \n",
              "9975      0  \n",
              "9976      1  \n",
              "9977      0  \n",
              "9978      0  \n",
              "9979      0  \n",
              "9980      0  \n",
              "9981      0  \n",
              "9982      1  \n",
              "9983      0  \n",
              "9984      1  \n",
              "9985      0  \n",
              "9986      1  \n",
              "9987      1  \n",
              "9988      1  \n",
              "9989      1  \n",
              "9990      0  \n",
              "9991      0  \n",
              "9992      1  \n",
              "9993      1  \n",
              "9994      0  \n",
              "9995      0  \n",
              "9996      1  \n",
              "9997      0  \n",
              "9998      0  \n",
              "9999      0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-7417d503-4cf2-49b5-a204-d11a6598c48e\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>9950</th>\n",
              "      <td>oh gosh say aw hun cuddles</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9951</th>\n",
              "      <td>follow amp follow u back</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9952</th>\n",
              "      <td>always good stand crowd</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9953</th>\n",
              "      <td>much editing little time</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9954</th>\n",
              "      <td>almost got see cute ankles socks</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9955</th>\n",
              "      <td>yayyyy hope knocked anesthesia sad miss</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9956</th>\n",
              "      <td>account got hacked</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9957</th>\n",
              "      <td>shadowplaylouis recently got mutual im really loving tweets everything yay thats exciting</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9958</th>\n",
              "      <td>easy</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9959</th>\n",
              "      <td>phone fell hand woke</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9960</th>\n",
              "      <td>ended happy</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9961</th>\n",
              "      <td>even started trend wanna know jot truth</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9962</th>\n",
              "      <td>campaign get goal walk dream help us</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9963</th>\n",
              "      <td>cotton candy favourite time stock</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9964</th>\n",
              "      <td>wish</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9965</th>\n",
              "      <td>happy birthday nawazuddin siddique big big fan</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9966</th>\n",
              "      <td>stats week arrived  new follower unfollowers via</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9967</th>\n",
              "      <td>wtf never happened moments may may like theres surprise thrown</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9968</th>\n",
              "      <td>indeed thanks link article</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9969</th>\n",
              "      <td>apology encanta vale osea bea</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9970</th>\n",
              "      <td>added video playlist im back twitch today going league</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9971</th>\n",
              "      <td>love tell every crowd theyre loudest loudest fans world arent x</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9972</th>\n",
              "      <td>dd leave lagi</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9973</th>\n",
              "      <td>vidcon</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9974</th>\n",
              "      <td>love</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9975</th>\n",
              "      <td>welcome aw go back next month rn tryna get healthy</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9976</th>\n",
              "      <td>happy birthday</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9977</th>\n",
              "      <td>guess blocking number phones spam sms feature wont work using specific words since guys write sinhala english</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9978</th>\n",
              "      <td>need come back england</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9979</th>\n",
              "      <td>wanna watch maybe comeback shows korea japan wanna visit country</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9980</th>\n",
              "      <td>sorry could schedule change</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9981</th>\n",
              "      <td>lead us youre gone</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9982</th>\n",
              "      <td>still awake</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9983</th>\n",
              "      <td>need</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9984</th>\n",
              "      <td>amazing news enjoy toy</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9985</th>\n",
              "      <td>followed thanks please followed</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9986</th>\n",
              "      <td>youre welcome</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9987</th>\n",
              "      <td>walk barely according good hope got home ok</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9988</th>\n",
              "      <td>turning fair meant lot people came</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9989</th>\n",
              "      <td>guys great weekend</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9990</th>\n",
              "      <td>know said michael jackson obviously little boy little brother listening jackson  song</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9991</th>\n",
              "      <td>face skinny picture omg get fat</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9992</th>\n",
              "      <td>agree</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9993</th>\n",
              "      <td>back thnx god im happy</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9994</th>\n",
              "      <td>always fall asleep early im texting someone</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9995</th>\n",
              "      <td>margo</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9996</th>\n",
              "      <td>pm rn currently  degrees fahrenheit really warm good</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9997</th>\n",
              "      <td>awwwww im sorry bet shes looking yoult</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9998</th>\n",
              "      <td>followed thanks please followed</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9999</th>\n",
              "      <td>miss kid cant believe im adult stuff like paying rent bills degree shit like fuck</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7417d503-4cf2-49b5-a204-d11a6598c48e')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-7417d503-4cf2-49b5-a204-d11a6598c48e button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-7417d503-4cf2-49b5-a204-d11a6598c48e');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-7e806cc1-1d87-4357-a599-956ce02647cb\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-7e806cc1-1d87-4357-a599-956ce02647cb')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-7e806cc1-1d87-4357-a599-956ce02647cb button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 50,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 49,\n        \"samples\": [\n          \"cotton candy favourite time stock\",\n          \"margo\",\n          \"awwwww im sorry bet shes looking yoult\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 175
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y9ylXI6xJN6a"
      },
      "source": [
        "### Extra whitespaces"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QVjRc-r3JN6a"
      },
      "source": [
        "After usually after preprocessing the text there might be extra whitespaces that might be created after transforming, removing various characters. Also, there is a need to remove all the new line, tab characters as well from our text."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 176,
      "metadata": {
        "id": "H94CIbbeJN6b"
      },
      "outputs": [],
      "source": [
        "def remove_whitespaces(text):\n",
        "    return \" \".join(text.split())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 177,
      "metadata": {
        "id": "mPLbHrehJN6b",
        "outputId": "9ff7a15c-9bc1-4be5-c878-fbce2bc3f035",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Whitespaces in the beginning are removed as well as in between the text'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 177
        }
      ],
      "source": [
        "text = \"  Whitespaces in the beginning are removed  \\t as well \\n  as in between  the text   \"\n",
        "\n",
        "clean_text = \" \".join(text.split())\n",
        "clean_text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 178,
      "metadata": {
        "id": "GGbIi0JmJN6b"
      },
      "outputs": [],
      "source": [
        "df.text = df.text.apply(lambda x: remove_whitespaces(x))"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.tail(30)"
      ],
      "metadata": {
        "id": "R0-CJXifeao-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 990
        },
        "outputId": "9ed40803-25de-4858-e585-c4e786991178"
      },
      "execution_count": 179,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                               text  \\\n",
              "9970                                                         added video playlist im back twitch today going league   \n",
              "9971                                                love tell every crowd theyre loudest loudest fans world arent x   \n",
              "9972                                                                                                  dd leave lagi   \n",
              "9973                                                                                                         vidcon   \n",
              "9974                                                                                                           love   \n",
              "9975                                                             welcome aw go back next month rn tryna get healthy   \n",
              "9976                                                                                                 happy birthday   \n",
              "9977  guess blocking number phones spam sms feature wont work using specific words since guys write sinhala english   \n",
              "9978                                                                                         need come back england   \n",
              "9979                                               wanna watch maybe comeback shows korea japan wanna visit country   \n",
              "9980                                                                                    sorry could schedule change   \n",
              "9981                                                                                             lead us youre gone   \n",
              "9982                                                                                                    still awake   \n",
              "9983                                                                                                           need   \n",
              "9984                                                                                         amazing news enjoy toy   \n",
              "9985                                                                                followed thanks please followed   \n",
              "9986                                                                                                  youre welcome   \n",
              "9987                                                                    walk barely according good hope got home ok   \n",
              "9988                                                                             turning fair meant lot people came   \n",
              "9989                                                                                             guys great weekend   \n",
              "9990                           know said michael jackson obviously little boy little brother listening jackson song   \n",
              "9991                                                                                face skinny picture omg get fat   \n",
              "9992                                                                                                          agree   \n",
              "9993                                                                                         back thnx god im happy   \n",
              "9994                                                                    always fall asleep early im texting someone   \n",
              "9995                                                                                                          margo   \n",
              "9996                                                            pm rn currently degrees fahrenheit really warm good   \n",
              "9997                                                                         awwwww im sorry bet shes looking yoult   \n",
              "9998                                                                                followed thanks please followed   \n",
              "9999                              miss kid cant believe im adult stuff like paying rent bills degree shit like fuck   \n",
              "\n",
              "      label  \n",
              "9970      1  \n",
              "9971      1  \n",
              "9972      0  \n",
              "9973      0  \n",
              "9974      1  \n",
              "9975      0  \n",
              "9976      1  \n",
              "9977      0  \n",
              "9978      0  \n",
              "9979      0  \n",
              "9980      0  \n",
              "9981      0  \n",
              "9982      1  \n",
              "9983      0  \n",
              "9984      1  \n",
              "9985      0  \n",
              "9986      1  \n",
              "9987      1  \n",
              "9988      1  \n",
              "9989      1  \n",
              "9990      0  \n",
              "9991      0  \n",
              "9992      1  \n",
              "9993      1  \n",
              "9994      0  \n",
              "9995      0  \n",
              "9996      1  \n",
              "9997      0  \n",
              "9998      0  \n",
              "9999      0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-260995bf-e3ca-47e6-af65-444cee85a111\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>9970</th>\n",
              "      <td>added video playlist im back twitch today going league</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9971</th>\n",
              "      <td>love tell every crowd theyre loudest loudest fans world arent x</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9972</th>\n",
              "      <td>dd leave lagi</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9973</th>\n",
              "      <td>vidcon</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9974</th>\n",
              "      <td>love</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9975</th>\n",
              "      <td>welcome aw go back next month rn tryna get healthy</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9976</th>\n",
              "      <td>happy birthday</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9977</th>\n",
              "      <td>guess blocking number phones spam sms feature wont work using specific words since guys write sinhala english</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9978</th>\n",
              "      <td>need come back england</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9979</th>\n",
              "      <td>wanna watch maybe comeback shows korea japan wanna visit country</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9980</th>\n",
              "      <td>sorry could schedule change</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9981</th>\n",
              "      <td>lead us youre gone</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9982</th>\n",
              "      <td>still awake</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9983</th>\n",
              "      <td>need</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9984</th>\n",
              "      <td>amazing news enjoy toy</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9985</th>\n",
              "      <td>followed thanks please followed</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9986</th>\n",
              "      <td>youre welcome</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9987</th>\n",
              "      <td>walk barely according good hope got home ok</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9988</th>\n",
              "      <td>turning fair meant lot people came</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9989</th>\n",
              "      <td>guys great weekend</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9990</th>\n",
              "      <td>know said michael jackson obviously little boy little brother listening jackson song</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9991</th>\n",
              "      <td>face skinny picture omg get fat</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9992</th>\n",
              "      <td>agree</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9993</th>\n",
              "      <td>back thnx god im happy</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9994</th>\n",
              "      <td>always fall asleep early im texting someone</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9995</th>\n",
              "      <td>margo</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9996</th>\n",
              "      <td>pm rn currently degrees fahrenheit really warm good</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9997</th>\n",
              "      <td>awwwww im sorry bet shes looking yoult</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9998</th>\n",
              "      <td>followed thanks please followed</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9999</th>\n",
              "      <td>miss kid cant believe im adult stuff like paying rent bills degree shit like fuck</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-260995bf-e3ca-47e6-af65-444cee85a111')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-260995bf-e3ca-47e6-af65-444cee85a111 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-260995bf-e3ca-47e6-af65-444cee85a111');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-eb186ca6-ba24-4785-9562-38f416211261\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-eb186ca6-ba24-4785-9562-38f416211261')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-eb186ca6-ba24-4785-9562-38f416211261 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 30,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 29,\n        \"samples\": [\n          \"awwwww im sorry bet shes looking yoult\",\n          \"youre welcome\",\n          \"still awake\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 179
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# spacy.word_tokenize"
      ],
      "metadata": {
        "id": "exXUqXwAOZiG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y6_dBEdMJN6c"
      },
      "source": [
        "### Frequent words"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "odKZM9NqJN6c"
      },
      "source": [
        "Previously we have removed stopwords which are common in any language. If we are working in any domain, we can also remove the common words used in that domain which don't provide us with much information."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 180,
      "metadata": {
        "id": "OMoYKfEjJN6c"
      },
      "outputs": [],
      "source": [
        "from collections import Counter\n",
        "def freq_words(text):\n",
        "    tokens = nltk.word_tokenize(text)\n",
        "    FrequentWords = []\n",
        "\n",
        "    for word in tokens:\n",
        "        counter[word] += 1\n",
        "\n",
        "    for (word, word_count) in counter.most_common(10):\n",
        "        FrequentWords.append(word)\n",
        "    return FrequentWords\n",
        "\n",
        "def remove_fw(text, FrequentWords):\n",
        "    tokens = nltk.word_tokenize(text)\n",
        "    without_fw = []\n",
        "    for word in tokens:\n",
        "        if word not in FrequentWords:\n",
        "            without_fw.append(word)\n",
        "\n",
        "    without_fw = ' '.join(without_fw)\n",
        "    return without_fw\n",
        "\n",
        "counter = Counter()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 181,
      "metadata": {
        "id": "_8Cw936EJN6d"
      },
      "outputs": [],
      "source": [
        "text = \"\"\"\n",
        "Natural Language Processing is the technology used to aid computers to understand the human’s natural language. It’s not an easy task teaching machines to understand how we communicate. Leand Romaf, an experienced software engineer who is passionate at teaching people how artificial intelligence systems work, says that “in recent years, there have been significant breakthroughs in empowering computers to understand language just as we do.” This article will give a simple introduction to Natural Language Processing and how it can be achieved. Natural Language Processing, usually shortened as NLP, is a branch of artificial intelligence that deals with the interaction between computers and humans using the natural language. The ultimate objective of NLP is to read, decipher, understand, and make sense of the human languages in a manner that is valuable. Most NLP techniques rely on machine learning to derive meaning from human languages.\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 182,
      "metadata": {
        "id": "RUFsNR7RJN6d",
        "outputId": "45a6db43-0041-4385-bfaa-ff4b4ed0ce83",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[',', 'to', '.', 'is', 'the', 'understand', 'Natural', 'Language', 'Processing', 'computers']\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n"
          ]
        }
      ],
      "source": [
        "nltk.download('punkt')\n",
        "FrequentWords = freq_words(text)\n",
        "print(FrequentWords)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 183,
      "metadata": {
        "id": "URwL18kDJN6e",
        "outputId": "ea78ac8c-9dc0-4aaa-cd7d-230ebc7d24bb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'technology used aid human ’ s natural language It ’ s not an easy task teaching machines how we communicate Leand Romaf an experienced software engineer who passionate at teaching people how artificial intelligence systems work says that “ in recent years there have been significant breakthroughs in empowering language just as we do. ” This article will give a simple introduction and how it can be achieved usually shortened as NLP a branch of artificial intelligence that deals with interaction between and humans using natural language The ultimate objective of NLP read decipher and make sense of human languages in a manner that valuable Most NLP techniques rely on machine learning derive meaning from human languages'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 183
        }
      ],
      "source": [
        "fw_result = remove_fw(text, FrequentWords)\n",
        "fw_result"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "khSZ1McSJN6e"
      },
      "source": [
        "### Rare words"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LOLrHEnaJN6f"
      },
      "source": [
        "Rare words are similar to frequent words. We can remove them because they are so less that they cannot add any value to the purpose."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 184,
      "metadata": {
        "id": "kNsiJl_wJN6f"
      },
      "outputs": [],
      "source": [
        "def rare_words(text):\n",
        "    # tokenization\n",
        "    tokens = nltk.word_tokenize(text)\n",
        "    for word in tokens:\n",
        "        counter[word]= +1\n",
        "\n",
        "    RareWords = []\n",
        "    number_rare_words = 10\n",
        "    # take top 10 frequent words\n",
        "    frequentWords = counter.most_common()\n",
        "    for (word, word_count) in frequentWords[:-number_rare_words:-1]:\n",
        "        RareWords.append(word)\n",
        "\n",
        "    return RareWords\n",
        "\n",
        "def remove_rw(text, RareWords):\n",
        "    tokens = nltk.word_tokenize(text)\n",
        "    without_rw = []\n",
        "    for word in tokens:\n",
        "        if word not in RareWords:\n",
        "            without_rw.append(word)\n",
        "\n",
        "    without_rw = ' '.join(without_fw)\n",
        "    return without_rw\n",
        "\n",
        "counter = Counter()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 185,
      "metadata": {
        "id": "YWkOBua-JN6g"
      },
      "outputs": [],
      "source": [
        "text = \"\"\"\n",
        "Natural Language Processing is the technology used to aid computers to understand the human’s natural language. It’s not an easy task teaching machines to understand how we communicate. Leand Romaf, an experienced software engineer who is passionate at teaching people how artificial intelligence systems work, says that “in recent years, there have been significant breakthroughs in empowering computers to understand language just as we do.” This article will give a simple introduction to Natural Language Processing and how it can be achieved. Natural Language Processing, usually shortened as NLP, is a branch of artificial intelligence that deals with the interaction between computers and humans using the natural language. The ultimate objective of NLP is to read, decipher, understand, and make sense of the human languages in a manner that is valuable. Most NLP techniques rely on machine learning to derive meaning from human languages.\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 186,
      "metadata": {
        "id": "IPb51jZVJN6g",
        "outputId": "81dd31f0-e933-4fd6-fd80-512270e36301",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['from',\n",
              " 'meaning',\n",
              " 'derive',\n",
              " 'learning',\n",
              " 'machine',\n",
              " 'on',\n",
              " 'rely',\n",
              " 'techniques',\n",
              " 'Most']"
            ]
          },
          "metadata": {},
          "execution_count": 186
        }
      ],
      "source": [
        "RareWords = rare_words(text)\n",
        "RareWords"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 187,
      "metadata": {
        "id": "f7gN2pQ2JN6h",
        "outputId": "d19b3697-2429-401b-99a8-9a1314b3d8db",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 140
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Natural Language Processing is the technology used to aid computers to understand the human ’ s natural language . It ’ s not an easy task teaching machines to understand how we communicate . Leand Romaf , an experienced software engineer who is passionate at teaching people how artificial intelligence systems work , says that “ in recent years , there have been significant breakthroughs in empowering computers to understand language just as we do. ” This article will give a simple introduction to Natural Language Processing and how it can be achieved . Natural Language Processing , usually shortened as NLP , is a branch of artificial intelligence that deals with the interaction between computers and humans using the natural language . The ultimate objective of NLP is to read , decipher , understand , and make sense of the human languages in a manner that is valuable . NLP to human languages .'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 187
        }
      ],
      "source": [
        "rw_result = remove_fw(text, RareWords)\n",
        "rw_result"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bw9QWjkUJN6i"
      },
      "source": [
        "## Conversion of Emoji to Words"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XV5A-4JnJN6i"
      },
      "source": [
        "To remove or not is done based on the purpose of the application. Example if we are building a sentiment analysis system emoji's can be useful.\n",
        "\n",
        "\"The movie was 🔥\"\n",
        "or\n",
        "\"The movie was 💩\"\n",
        "\n",
        "If we remove the emoji's the meaning of the sentence changes completely. In these cases we can convert emoji's to words."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "On487oTrJN6j"
      },
      "source": [
        "demoji requires an initial data download from the Unicode Consortium's [emoji code repository](http://unicode.org/Public/emoji/12.0/emoji-test.txt).\n",
        "\n",
        "On first use of the package, call download_codes().<br>\n",
        "This will store the Unicode hex-notated symbols at ~/.demoji/codes.json for future use.\n",
        "\n",
        "Read more about demoji on [pypi.org](https://pypi.org/project/demoji/)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zZ8uqekgJN6j",
        "outputId": "f93430ba-7c74-4ce0-f386-e0954bf77983",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-62-b6fe2259445e>:1: FutureWarning: The demoji.download_codes attribute is deprecated and will be removed from demoji in a future version. It is an unused attribute as emoji codes are now distributed directly with the demoji package.\n",
            "  demoji.download_codes()\n"
          ]
        }
      ],
      "source": [
        "demoji.download_codes()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 188,
      "metadata": {
        "id": "WT_xhd_zJN6k"
      },
      "outputs": [],
      "source": [
        "def emoji_to_words(text):\n",
        "    return demoji.replace_with_desc(text, sep=\"__\")"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "r7YKOxVCMNdj"
      },
      "execution_count": 79,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 189,
      "metadata": {
        "id": "p7iUxgUdJN6k",
        "outputId": "9423c4fb-0b28-4163-8061-20849bcea456",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'game is on __fire__ __person rowing boat: medium-light skin tone__'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 189
        }
      ],
      "source": [
        "text = \"game is on 🔥 🚣🏼\"\n",
        "emoji_to_words(text)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "text = \"Hey there!! :-)\"\n",
        "emoji_to_words(text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "YTUNfNcgGYqD",
        "outputId": "76791a62-f82a-403e-ab26-36a7076929ae"
      },
      "execution_count": 190,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Hey there!! :-)'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 190
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GinqiNESJN6k"
      },
      "source": [
        "## Conversion of Emoticons to Words"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2H-4LAyqJN6k"
      },
      "source": [
        "As we did for emoji's, we convert emoticons to words for the same purpose."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 191,
      "metadata": {
        "id": "IhmxhS0xJN6k"
      },
      "outputs": [],
      "source": [
        "def emoticons_to_words(text):\n",
        "    for emot in EMOTICONS:\n",
        "        text = re.sub(u'('+emot+')', \"_\".join(EMOTICONS[emot].replace(\",\",\"\").replace(\":\",\"\").split()), text)\n",
        "    return text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 192,
      "metadata": {
        "id": "37G7w4JjJN6k",
        "outputId": "f3d9be0f-4706-492d-cc4c-aa224e2ce308",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Hey there!! Happy_face_smiley'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 192
        }
      ],
      "source": [
        "text = \"Hey there!! :-)\"\n",
        "emoticons_to_words(text)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install emot"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vcOKPD26HVQy",
        "outputId": "91a4e212-e7b2-417e-f21d-e599b07fbb82"
      },
      "execution_count": 85,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting emot\n",
            "  Downloading emot-3.1-py3-none-any.whl (61 kB)\n",
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/61.5 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.5/61.5 kB\u001b[0m \u001b[31m1.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: emot\n",
            "Successfully installed emot-3.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import emot"
      ],
      "metadata": {
        "id": "0iCU7DCwHarH"
      },
      "execution_count": 193,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "emot_obj = emot.core.emot()"
      ],
      "metadata": {
        "id": "7RhojusUHjld"
      },
      "execution_count": 194,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "text = \"I love python ☮ 🙂 ❤ :-) :-( :-)))\""
      ],
      "metadata": {
        "id": "zVjU2eDdHmxj"
      },
      "execution_count": 195,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "emot_obj.emoji(text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4941RwGOHoX9",
        "outputId": "52a03cbd-1d76-47a1-e545-f6d0cd1ee908"
      },
      "execution_count": 196,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'value': ['☮', '🙂', '❤'],\n",
              " 'location': [[14, 15], [16, 17], [18, 19]],\n",
              " 'mean': [':peace_symbol:', ':slightly_smiling_face:', ':red_heart:'],\n",
              " 'flag': True}"
            ]
          },
          "metadata": {},
          "execution_count": 196
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "emot_obj.emoticons(text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ULCiDVdtH2lP",
        "outputId": "293b9bff-4618-4607-d601-d0199ebbdc14"
      },
      "execution_count": 197,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'value': [':-)', ':-(', ':-)))'],\n",
              " 'location': [[20, 23], [24, 27], [28, 33]],\n",
              " 'mean': ['Happy face smiley',\n",
              "  'Frown, sad, andry or pouting',\n",
              "  'Very very Happy face or smiley'],\n",
              " 'flag': True}"
            ]
          },
          "metadata": {},
          "execution_count": 197
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ecoJFKAbJN6l"
      },
      "source": [
        "## Converting Numbers to Words"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H523PUfjJN6l"
      },
      "source": [
        "If our analysis require us to use information based on the numbers in the text, we can convert them to words.\n",
        "\n",
        "Read more about num2words on [github](https://github.com/savoirfairelinux/num2words)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 198,
      "metadata": {
        "id": "3jydW4YZJN6l"
      },
      "outputs": [],
      "source": [
        "def nums_to_words(text):\n",
        "    new_text = []\n",
        "    for word in text.split():\n",
        "        if word.isdigit():\n",
        "            new_text.append(num2words(word))\n",
        "        else:\n",
        "            new_text.append(word)\n",
        "    return \" \".join(new_text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 199,
      "metadata": {
        "id": "tzzhKZpqJN6l",
        "outputId": "03947bf6-559e-4784-cef2-80fb460d878f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'I ran this track thirty times'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 199
        }
      ],
      "source": [
        "text = \"I ran this track 30 times\"\n",
        "nums_to_words(text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QpRiYHEAJN6l"
      },
      "source": [
        "## Chat words Conversion"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bJPtXavuJN6l"
      },
      "source": [
        "The more we use social media, we have become lazy to type the whole phrase or word. Due to which slang words came into existance such as \"omg\" which represents \"Oh my god\". Such slang words don't provide much information and if we need to use them we have to convert them.\n",
        "\n",
        "Thank you: [GitHub repo](https://github.com/rishabhverma17/sms_slang_translator/blob/master/slang.txt) for the list of slang words"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 200,
      "metadata": {
        "id": "stQgzPSyJN6m"
      },
      "outputs": [],
      "source": [
        "#collapse-hide\n",
        "chat_words = \"\"\"\n",
        "AFAIK=As Far As I Know\n",
        "AFK=Away From Keyboard\n",
        "ASAP=As Soon As Possible\n",
        "ATK=At The Keyboard\n",
        "ATM=At The Moment\n",
        "A3=Anytime, Anywhere, Anyplace\n",
        "BAK=Back At Keyboard\n",
        "BBL=Be Back Later\n",
        "BBS=Be Back Soon\n",
        "BFN=Bye For Now\n",
        "B4N=Bye For Now\n",
        "BRB=Be Right Back\n",
        "BRT=Be Right There\n",
        "BTW=By The Way\n",
        "B4=Before\n",
        "B4N=Bye For Now\n",
        "CU=See You\n",
        "CUL8R=See You Later\n",
        "CYA=See You\n",
        "FAQ=Frequently Asked Questions\n",
        "FC=Fingers Crossed\n",
        "FWIW=For What It's Worth\n",
        "FYI=For Your Information\n",
        "GAL=Get A Life\n",
        "GG=Good Game\n",
        "GN=Good Night\n",
        "GMTA=Great Minds Think Alike\n",
        "GR8=Great!\n",
        "G9=Genius\n",
        "IC=I See\n",
        "ICQ=I Seek you (also a chat program)\n",
        "ILU=ILU: I Love You\n",
        "IMHO=In My Honest/Humble Opinion\n",
        "IMO=In My Opinion\n",
        "IOW=In Other Words\n",
        "IRL=In Real Life\n",
        "KISS=Keep It Simple, Stupid\n",
        "LDR=Long Distance Relationship\n",
        "LMAO=Laugh My A.. Off\n",
        "LOL=Laughing Out Loud\n",
        "LTNS=Long Time No See\n",
        "L8R=Later\n",
        "MTE=My Thoughts Exactly\n",
        "M8=Mate\n",
        "NRN=No Reply Necessary\n",
        "OIC=Oh I See\n",
        "PITA=Pain In The A..\n",
        "PRT=Party\n",
        "PRW=Parents Are Watching\n",
        "QPSA?=Que Pasa?\n",
        "ROFL=Rolling On The Floor Laughing\n",
        "ROFLOL=Rolling On The Floor Laughing Out Loud\n",
        "ROTFLMAO=Rolling On The Floor Laughing My A.. Off\n",
        "SK8=Skate\n",
        "STATS=Your sex and age\n",
        "ASL=Age, Sex, Location\n",
        "THX=Thank You\n",
        "TTFN=Ta-Ta For Now!\n",
        "TTYL=Talk To You Later\n",
        "U=You\n",
        "U2=You Too\n",
        "U4E=Yours For Ever\n",
        "WB=Welcome Back\n",
        "WTF=What The F...\n",
        "WTG=Way To Go!\n",
        "WUF=Where Are You From?\n",
        "W8=Wait...\n",
        "7K=Sick:-D Laugher\n",
        "OMG=Oh my god\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 201,
      "metadata": {
        "id": "EUszFh2dJN6m"
      },
      "outputs": [],
      "source": [
        "chat_words_dict = dict()\n",
        "chat_words_set = set()\n",
        "\n",
        "def cw_conversion(text):\n",
        "    new_text = []\n",
        "    for word in text.split():\n",
        "        if word.upper() in chat_words_set:\n",
        "            new_text.append(chat_words_dict[word.upper()])\n",
        "        else:\n",
        "            new_text.append(word)\n",
        "    return \" \".join(new_text)\n",
        "\n",
        "for line in chat_words.split('\\n'):\n",
        "    if line != '':\n",
        "        cw, cw_expanded = line.split('=')[0], line.split('=')[1]\n",
        "\n",
        "        chat_words_set.add(cw)\n",
        "        chat_words_dict[cw] = cw_expanded"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 202,
      "metadata": {
        "id": "0jzmU0pFJN6n",
        "outputId": "62845c58-1984-4370-9bd9-a129a1fec621",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"Oh my god that's awesome.\""
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 202
        }
      ],
      "source": [
        "text = \"omg that's awesome.\"\n",
        "cw_conversion(text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pm7oV-cOJN6n"
      },
      "source": [
        "## Expanding Contractions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gTmEPTOyJN6n"
      },
      "source": [
        "Contractions are words or combinations of words created by dropping a few letters and replacing those letters by an apostrophe.\n",
        "\n",
        "Example:\n",
        "- don't: do not\n",
        "- we'll: we will\n",
        "\n",
        "Our nlp model don't understand these contractions i.e. they don't understand that \"don't\" and \"do not\" are the same thing. If our problem statement requires them then we can expand them or else leave it as it is."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 203,
      "metadata": {
        "id": "vsWwwYMqJN6o"
      },
      "outputs": [],
      "source": [
        "def expand_contractions(text):\n",
        "    expanded_text = []\n",
        "    for line in text:\n",
        "        expanded_text.append(contractions.fix(line))\n",
        "    return expanded_text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 204,
      "metadata": {
        "id": "68CUO_GxJN6o",
        "outputId": "edb9ddaa-7efc-4d28-ec27-4b83293a6662",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['I will be there within 15 minutes.',\n",
              " 'It is awesome to meet your new friends.']"
            ]
          },
          "metadata": {},
          "execution_count": 204
        }
      ],
      "source": [
        "text = [\"I'll be there within 15 minutes.\", \"It's awesome to meet your new friends.\"]\n",
        "expand_contractions(text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zUuYCIINJN6o"
      },
      "source": [
        "## Stemming"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QeMFCZm0JN6p"
      },
      "source": [
        "In stemming we reduce the word to it's base or root form by removing the suffix characters from the word. It is one of the technique to normalize text.\n",
        "\n",
        "Stemming for root word \"like\" include:\n",
        "- \"likes\"\n",
        "- \"liked\"\n",
        "- \"likely\"\n",
        "- \"liking\"\n",
        "\n",
        "Stemmed word doesn't always match the words in our dictionary such as:\n",
        "- console -> consol\n",
        "- company -> compani\n",
        "- welcome -> welcom\n",
        "\n",
        "Due to which stemming is not performed in all nlp tasks.\n",
        "\n",
        "There are various algorithms used for stemming but the most widely used is PorterStemmer. In this post we have used the PorterStemmer as well."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 205,
      "metadata": {
        "id": "tgZkwRZUJN6p"
      },
      "outputs": [],
      "source": [
        "stemmer = PorterStemmer()\n",
        "\n",
        "def stem_words(text):\n",
        "    return ' '.join([stemmer.stem(word) for word in text.split()])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 206,
      "metadata": {
        "id": "A4udHi4nJN6p",
        "outputId": "c1e7d493-9e11-4148-a41f-0dd8b55a54b4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                           text  \\\n",
              "0    aww poor thing hope itokay good health luckely freed rocks   \n",
              "1  hi ive spoken store advised paypoint card topped thanks beth   \n",
              "2                              could never go ever seeing cruel   \n",
              "3                                   bus late time get food work   \n",
              "4                                   another ridiculous headache   \n",
              "\n",
              "                                             text_stemmed  \n",
              "0  aww poor thing hope itokay good health luck freed rock  \n",
              "1  hi ive spoken store advis paypoint card top thank beth  \n",
              "2                           could never go ever see cruel  \n",
              "3                              bu late time get food work  \n",
              "4                                   anoth ridicul headach  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-5c85fc25-efbf-447a-b268-b74477128f1e\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>text_stemmed</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>aww poor thing hope itokay good health luckely freed rocks</td>\n",
              "      <td>aww poor thing hope itokay good health luck freed rock</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>hi ive spoken store advised paypoint card topped thanks beth</td>\n",
              "      <td>hi ive spoken store advis paypoint card top thank beth</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>could never go ever seeing cruel</td>\n",
              "      <td>could never go ever see cruel</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>bus late time get food work</td>\n",
              "      <td>bu late time get food work</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>another ridiculous headache</td>\n",
              "      <td>anoth ridicul headach</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5c85fc25-efbf-447a-b268-b74477128f1e')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-5c85fc25-efbf-447a-b268-b74477128f1e button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-5c85fc25-efbf-447a-b268-b74477128f1e');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-a9556220-cce6-4289-90c6-ad4d0e6a5404\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-a9556220-cce6-4289-90c6-ad4d0e6a5404')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-a9556220-cce6-4289-90c6-ad4d0e6a5404 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"df[['text', 'text_stemmed']]\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"hi ive spoken store advised paypoint card topped thanks beth\",\n          \"another ridiculous headache\",\n          \"could never go ever seeing cruel\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"text_stemmed\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"hi ive spoken store advis paypoint card top thank beth\",\n          \"anoth ridicul headach\",\n          \"could never go ever see cruel\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 206
        }
      ],
      "source": [
        "df['text_stemmed'] = df.text.apply(lambda text: stem_words(text))\n",
        "df[['text', 'text_stemmed']].head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DXbu0nmPJN6q"
      },
      "source": [
        "PorterStemmer can be used only for english. If we are working with other than english then we can use SnowballStemmer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 207,
      "metadata": {
        "id": "SwYG7b3RJN6q",
        "outputId": "41b3bae2-bbdc-40d3-820f-4327744b8c9d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "('arabic',\n",
              " 'danish',\n",
              " 'dutch',\n",
              " 'english',\n",
              " 'finnish',\n",
              " 'french',\n",
              " 'german',\n",
              " 'hungarian',\n",
              " 'italian',\n",
              " 'norwegian',\n",
              " 'porter',\n",
              " 'portuguese',\n",
              " 'romanian',\n",
              " 'russian',\n",
              " 'spanish',\n",
              " 'swedish')"
            ]
          },
          "metadata": {},
          "execution_count": 207
        }
      ],
      "source": [
        "SnowballStemmer.languages"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kRtEgnvOJN6q"
      },
      "source": [
        "## Lemmatization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j3HqcocDJN6q"
      },
      "source": [
        "Lemmatization tried to perform the similar task as that of stemming i.e. trying to reduce the inflection words to it's base form. But lemmatization does it by using a different approach.\n",
        "\n",
        "Lemmatizations takes into consideration of the morphological analysis of the word. It tries to reduce to words to it's dictionary form which is known as lemma."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 208,
      "metadata": {
        "id": "y5wvv1McJN6r"
      },
      "outputs": [],
      "source": [
        "lemmatizer = WordNetLemmatizer()\n",
        "\n",
        "def text_lemmatize(text):\n",
        "    return ' '.join([lemmatizer.lemmatize(word) for word in text.split()])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 209,
      "metadata": {
        "id": "JpvWd0hwJN6r",
        "outputId": "d80a1cb8-ef40-453e-b34c-64916524cb5e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 276
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                           text  \\\n",
              "0    aww poor thing hope itokay good health luckely freed rocks   \n",
              "1  hi ive spoken store advised paypoint card topped thanks beth   \n",
              "2                              could never go ever seeing cruel   \n",
              "3                                   bus late time get food work   \n",
              "4                                   another ridiculous headache   \n",
              "\n",
              "                                             text_stemmed  \\\n",
              "0  aww poor thing hope itokay good health luck freed rock   \n",
              "1  hi ive spoken store advis paypoint card top thank beth   \n",
              "2                           could never go ever see cruel   \n",
              "3                              bu late time get food work   \n",
              "4                                   anoth ridicul headach   \n",
              "\n",
              "                                                text_lemmatized  \n",
              "0     aww poor thing hope itokay good health luckely freed rock  \n",
              "1  hi ive spoken store advised paypoint card topped thanks beth  \n",
              "2                              could never go ever seeing cruel  \n",
              "3                                   bus late time get food work  \n",
              "4                                   another ridiculous headache  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-d52c3bf8-87d1-431f-a649-647fb372339f\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>text_stemmed</th>\n",
              "      <th>text_lemmatized</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>aww poor thing hope itokay good health luckely freed rocks</td>\n",
              "      <td>aww poor thing hope itokay good health luck freed rock</td>\n",
              "      <td>aww poor thing hope itokay good health luckely freed rock</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>hi ive spoken store advised paypoint card topped thanks beth</td>\n",
              "      <td>hi ive spoken store advis paypoint card top thank beth</td>\n",
              "      <td>hi ive spoken store advised paypoint card topped thanks beth</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>could never go ever seeing cruel</td>\n",
              "      <td>could never go ever see cruel</td>\n",
              "      <td>could never go ever seeing cruel</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>bus late time get food work</td>\n",
              "      <td>bu late time get food work</td>\n",
              "      <td>bus late time get food work</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>another ridiculous headache</td>\n",
              "      <td>anoth ridicul headach</td>\n",
              "      <td>another ridiculous headache</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d52c3bf8-87d1-431f-a649-647fb372339f')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-d52c3bf8-87d1-431f-a649-647fb372339f button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-d52c3bf8-87d1-431f-a649-647fb372339f');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-66763993-0271-4d97-9b3c-b7466775ef32\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-66763993-0271-4d97-9b3c-b7466775ef32')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-66763993-0271-4d97-9b3c-b7466775ef32 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"df[['text', 'text_stemmed', 'text_lemmatized']]\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"hi ive spoken store advised paypoint card topped thanks beth\",\n          \"another ridiculous headache\",\n          \"could never go ever seeing cruel\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"text_stemmed\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"hi ive spoken store advis paypoint card top thank beth\",\n          \"anoth ridicul headach\",\n          \"could never go ever see cruel\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"text_lemmatized\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"hi ive spoken store advised paypoint card topped thanks beth\",\n          \"another ridiculous headache\",\n          \"could never go ever seeing cruel\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 209
        }
      ],
      "source": [
        "nltk.download('wordnet')\n",
        "df['text_lemmatized'] = df.text.apply(lambda text: text_lemmatize(text))\n",
        "df[['text', 'text_stemmed', 'text_lemmatized']].head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "osZ4YPRcJN6r"
      },
      "source": [
        "Difference between Stemming and Lemmatization:\n",
        "\n",
        "|Stemming | Lemmatization |\n",
        "| ---- | ---- |\n",
        "| Fast compared to lemmatization | Slow compared to stemming |\n",
        "| Reduces the word to it's base form by removing the suffix | Uses lexical knowledge to get the base form of the word |\n",
        "| Does not always provide meaning or dictionary form of the original word | Resulting words are always meaningful and dictionary words |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UBQeNtRMJN6s"
      },
      "source": [
        "## Spelling Correction"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ix2DQw2XJN6s"
      },
      "source": [
        "We as human always make mistake. Normally incorrect spelling in text are know as typos.\n",
        "\n",
        "Since the NLP model doesn't know the difference between a correct and an incorrect word. For the model \"thanks\" and \"thnks\" are two different words. Therefore, spelling correction is an important step to bring the incorrect words in the correct format."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 210,
      "metadata": {
        "id": "CvO0BWdrJN6s"
      },
      "outputs": [],
      "source": [
        "spell = SpellChecker()\n",
        "\n",
        "def correct_spelling(text):\n",
        "    correct_text = []\n",
        "    misspelled_words = spell.unknown(text.split())\n",
        "    for word in text.split():\n",
        "        if word in misspelled_words:\n",
        "            correct_text.append(spell.correction(word))\n",
        "        else:\n",
        "            correct_text.append(word)\n",
        "    return \" \".join(correct_text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 211,
      "metadata": {
        "id": "LYuskNq6JN6s",
        "outputId": "792fa87a-35b3-4f5e-ed73-778b89ecd70f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"Hi, how are you doing I'm good thanks for asking\""
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 211
        }
      ],
      "source": [
        "text = \"Hi, hwo are you doin? I'm good thnks for asking\"\n",
        "correct_spelling(text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 215,
      "metadata": {
        "id": "y91-N-qAJN6t",
        "outputId": "f392ac3d-dcb4-449f-b4c2-5130225b453b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"how are you doing I'm god thanks\""
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 215
        }
      ],
      "source": [
        "text = \"hwo are you doin? I'm god thnks\"\n",
        "correct_spelling(text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RQyRwqTVJN6t"
      },
      "source": [
        "## Convert accented characters to ASCII characters"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xDrxISlPJN6t"
      },
      "source": [
        "Accent marks (also referred to as diacritics or diacriticals) usually appear above a character when we press the character for a long time. These need to be remove cause the model cannot distinguish between \"dèèp\" and \"deep\". It will consider them as two different words."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 216,
      "metadata": {
        "id": "EpTwVGi-JN6u"
      },
      "outputs": [],
      "source": [
        "def accented_to_ascii(text):\n",
        "    return unidecode.unidecode(text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 217,
      "metadata": {
        "id": "-2z7LPVzJN6u",
        "outputId": "52a2baf9-cb1b-46b8-eeec-95f474465519",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'This is an example text with accented characters like deep learning and computer vision etc.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 217
        }
      ],
      "source": [
        "text = \"This is an example text with accented characters like dèèp lèarning ánd cömputer vísíön etc.\"\n",
        "accented_to_ascii(text)"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "Iqnk2lfvZKSR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer"
      ],
      "metadata": {
        "id": "brqGOO36ZKXV"
      },
      "execution_count": 218,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "vectorizer = TfidfVectorizer()"
      ],
      "metadata": {
        "id": "vIhRWC60ZKcX"
      },
      "execution_count": 219,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "X = vectorizer.fit_transform(df['text_lemmatized'])"
      ],
      "metadata": {
        "id": "Omltm-L1ZQ3D"
      },
      "execution_count": 220,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "y=df['label']"
      ],
      "metadata": {
        "id": "2kPaTfAiqhVv"
      },
      "execution_count": 221,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# prompt: split the X and y data, thereafter fit a logistic regression algorithm\n",
        "\n",
        "from sklearn.model_selection import train_test_split\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\n",
        "\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "logistic_model = LogisticRegression()\n",
        "logistic_model.fit(X_train, y_train)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 75
        },
        "id": "qgFlnqedxr9O",
        "outputId": "8822c56a-7563-4bc6-cee8-26f986773a6d"
      },
      "execution_count": 222,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "LogisticRegression()"
            ],
            "text/html": [
              "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression()</pre></div></div></div></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 222
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# prompt: test the predictions of the model on X_test and give the accuracy ,recall, precision, f1-score and confusion metrics\n",
        "\n",
        "from sklearn.metrics import confusion_matrix\n",
        "from sklearn.metrics import accuracy_score, recall_score, precision_score, f1_score\n",
        "\n",
        "y_pred = logistic_model.predict(X_test)\n",
        "\n",
        "accuracy = accuracy_score(y_test, y_pred)\n",
        "recall = recall_score(y_test, y_pred, average='weighted')\n",
        "precision = precision_score(y_test, y_pred, average='weighted')\n",
        "f1 = f1_score(y_test, y_pred, average='weighted')\n",
        "\n",
        "print(\"Accuracy:\", accuracy)\n",
        "print(\"Recall:\", recall)\n",
        "print(\"Precision:\", precision)\n",
        "print(\"F1 Score:\", f1)\n",
        "\n",
        "conf_matrix = confusion_matrix(y_test, y_pred)\n",
        "print(\"Confusion Matrix:\\n\", conf_matrix)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BgTTFAhGx8Nb",
        "outputId": "697cea0e-3149-455f-ae4b-7aac63d5f135"
      },
      "execution_count": 223,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy: 0.7404\n",
            "Recall: 0.7404\n",
            "Precision: 0.7422980869565218\n",
            "F1 Score: 0.7399861024960781\n",
            "Confusion Matrix:\n",
            " [[972 271]\n",
            " [378 879]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZiyKqEfoJN6u"
      },
      "source": [
        "## Conclusion"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YrOHFpJAJN6u"
      },
      "source": [
        "In this article, most of the text pre-processing techniques are explanied. I'll update this post as I learn more techniques to pre-process text.\n",
        "\n",
        "Share if you liked it, comment if you loved it. Hope to see you guys in the next one. Peace!"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.7"
    },
    "toc-autonumbering": false,
    "toc-showcode": false,
    "toc-showmarkdowntxt": false,
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}