Text_Classification.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "b6ff46c8",
      "metadata": {
        "id": "b6ff46c8"
      },
      "source": [
        "# Text Classification of Biomimicry Papers"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "48d375f9",
      "metadata": {
        "id": "48d375f9"
      },
      "source": [
        "The goal of this jupyter notebook is to classify biomimicry paper into the given catetories based on its full abstract and title."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ff05e1dc",
      "metadata": {
        "id": "ff05e1dc"
      },
      "source": [
        "## 1. Import basic packages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "id": "a51b033f",
      "metadata": {
        "id": "a51b033f"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import os\n",
        "import string\n",
        "import sys\n",
        "import re"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "58886de7",
      "metadata": {
        "id": "58886de7"
      },
      "source": [
        "## 2. Get dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 45,
      "id": "2aeab490",
      "metadata": {
        "id": "2aeab490"
      },
      "outputs": [],
      "source": [
        "url = 'https://raw.githubusercontent.com/nasa-petal/search-engine/main/golden.json'\n",
        "df = pd.read_json(url, orient='columns')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f69ad415",
      "metadata": {
        "id": "f69ad415"
      },
      "source": [
        "### 2.1 Initial Dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 46,
      "id": "f431a922",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 389
        },
        "id": "f431a922",
        "outputId": "9a749d3e-df23-4916-a989-1b9870761406"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total Number of data: 11084\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>paper</th>\n",
              "      <th>mag</th>\n",
              "      <th>venue_mag</th>\n",
              "      <th>author</th>\n",
              "      <th>reference</th>\n",
              "      <th>title</th>\n",
              "      <th>abstract</th>\n",
              "      <th>petalID</th>\n",
              "      <th>doi</th>\n",
              "      <th>venue</th>\n",
              "      <th>level1</th>\n",
              "      <th>level2</th>\n",
              "      <th>level3</th>\n",
              "      <th>url</th>\n",
              "      <th>isBiomimicry</th>\n",
              "      <th>fullDocLink</th>\n",
              "      <th>isOpenAccess</th>\n",
              "      <th>abstract_full</th>\n",
              "      <th>title_full</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2103410568</td>\n",
              "      <td>['bubble nest', 'nest', 'mixing', 'bubble', 'p...</td>\n",
              "      <td>['Biology Letters']</td>\n",
              "      <td>[2346835213, 2098042950]</td>\n",
              "      <td>[2130285640, 2066345165, 2054319467, 204771406...</td>\n",
              "      <td>['building', 'home', 'foam', 'tungara', 'frog'...</td>\n",
              "      <td>['frogs', 'build', 'foam', 'nests', 'floating'...</td>\n",
              "      <td>0</td>\n",
              "      <td>10.1098/RSBL.2009.0934</td>\n",
              "      <td>[\"Weird Nature: An Astonishing Exploration of ...</td>\n",
              "      <td>['physically_assemble/disassemble', 'protect_f...</td>\n",
              "      <td>['physically_assemble_structure', 'protect_fro...</td>\n",
              "      <td>['protect_from_animals', 'protect_from_loss_of...</td>\n",
              "      <td>https://royalsocietypublishing.org/doi/10.1098...</td>\n",
              "      <td>Y</td>\n",
              "      <td>https://royalsocietypublishing.org/doi/10.1098...</td>\n",
              "      <td>True</td>\n",
              "      <td>Frogs that build foam nests floating on water ...</td>\n",
              "      <td>Building a home from foam—túngara frog foam ne...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2138292607</td>\n",
              "      <td>['sunset', 'earth s magnetic field', 'compass'...</td>\n",
              "      <td>['Proceedings of the National Academy of Scien...</td>\n",
              "      <td>[2132083079, 2425702268, 2552946098]</td>\n",
              "      <td>[1493129647, 2037761037, 1984592609, 213699427...</td>\n",
              "      <td>['nocturnal', 'mammal', 'greater', 'mouse', 'e...</td>\n",
              "      <td>['evidence', 'suggests', 'bats', 'detect', 'ge...</td>\n",
              "      <td>1</td>\n",
              "      <td>10.1073/PNAS.0912477107</td>\n",
              "      <td>['Proceedings of the National Academy of Scien...</td>\n",
              "      <td>['sense_send_or_process_information']</td>\n",
              "      <td>['sense_signals/environmental_cues']</td>\n",
              "      <td>['sense_spatial_awareness/balance/orientation']</td>\n",
              "      <td>https://www.pnas.org/content/107/15/6941</td>\n",
              "      <td>Y</td>\n",
              "      <td>https://www.pnas.org/content/107/15/6941.full.pdf</td>\n",
              "      <td>True</td>\n",
              "      <td>Recent evidence suggests that bats can detect ...</td>\n",
              "      <td>A nocturnal mammal, the greater mouse-eared ba...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2005539166</td>\n",
              "      <td>['sepia mestus', 'optomotor response', 'cuttle...</td>\n",
              "      <td>['The Journal of Experimental Biology']</td>\n",
              "      <td>[2163942483, 3088803717]</td>\n",
              "      <td>[2035108601, 2155571491, 2159857711, 207521876...</td>\n",
              "      <td>['polarization', 'sensitivity', 'two', 'specie...</td>\n",
              "      <td>['existence', 'polarization', 'sensitivity', '...</td>\n",
              "      <td>2</td>\n",
              "      <td>10.1242/JEB.042937</td>\n",
              "      <td>['The Journal of Experimental Biology', 'Curre...</td>\n",
              "      <td>['sense_send_or_process_information']</td>\n",
              "      <td>['sense_signals/environmental_cues']</td>\n",
              "      <td>['sense_light_in_the_non-visible_spectrum', 's...</td>\n",
              "      <td>https://jeb.biologists.org/content/213/19/3364</td>\n",
              "      <td>Y</td>\n",
              "      <td>https://journals.biologists.com/jeb/article-pd...</td>\n",
              "      <td>True</td>\n",
              "      <td>SUMMARY The existence of polarization sensitiv...</td>\n",
              "      <td>Polarization sensitivity in two species of cut...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        paper  ...                                         title_full\n",
              "0  2103410568  ...  Building a home from foam—túngara frog foam ne...\n",
              "1  2138292607  ...  A nocturnal mammal, the greater mouse-eared ba...\n",
              "2  2005539166  ...  Polarization sensitivity in two species of cut...\n",
              "\n",
              "[3 rows x 19 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 46
        }
      ],
      "source": [
        "print(\"Total Number of data:\", len(df))\n",
        "df.head(3)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e83c6c39",
      "metadata": {
        "id": "e83c6c39"
      },
      "source": [
        "## 3. Data Cleaning"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3.1 Drop unnecessary columns"
      ],
      "metadata": {
        "id": "qnvghc3bO5dw"
      },
      "id": "qnvghc3bO5dw"
    },
    {
      "cell_type": "code",
      "source": [
        "df = df.drop(columns=['mag', 'venue_mag', 'author', 'reference', 'title', 'abstract', 'petalID', 'doi', 'venue', 'level2', 'level3', 'url', 'isBiomimicry', 'fullDocLink', 'isOpenAccess'])\n",
        "df.head(3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "id": "ltIN5T-8PDET",
        "outputId": "704492d8-3592-429b-b912-ed1d3eec893e"
      },
      "id": "ltIN5T-8PDET",
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>paper</th>\n",
              "      <th>level1</th>\n",
              "      <th>abstract_full</th>\n",
              "      <th>title_full</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2103410568</td>\n",
              "      <td>['physically_assemble/disassemble', 'protect_f...</td>\n",
              "      <td>Frogs that build foam nests floating on water ...</td>\n",
              "      <td>Building a home from foam—túngara frog foam ne...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2138292607</td>\n",
              "      <td>['sense_send_or_process_information']</td>\n",
              "      <td>Recent evidence suggests that bats can detect ...</td>\n",
              "      <td>A nocturnal mammal, the greater mouse-eared ba...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2005539166</td>\n",
              "      <td>['sense_send_or_process_information']</td>\n",
              "      <td>SUMMARY The existence of polarization sensitiv...</td>\n",
              "      <td>Polarization sensitivity in two species of cut...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        paper  ...                                         title_full\n",
              "0  2103410568  ...  Building a home from foam—túngara frog foam ne...\n",
              "1  2138292607  ...  A nocturnal mammal, the greater mouse-eared ba...\n",
              "2  2005539166  ...  Polarization sensitivity in two species of cut...\n",
              "\n",
              "[3 rows x 4 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "markdown",
      "id": "45acf8e1",
      "metadata": {
        "id": "45acf8e1"
      },
      "source": [
        "### 3.2 Remove N/A values"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 48,
      "id": "0a4a263b",
      "metadata": {
        "id": "0a4a263b"
      },
      "outputs": [],
      "source": [
        "df.fillna('[]', inplace = True)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bf4846ad",
      "metadata": {
        "id": "bf4846ad"
      },
      "source": [
        "### 3.3 Remove data that contains missing value"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 49,
      "id": "2a61d680",
      "metadata": {
        "id": "2a61d680"
      },
      "outputs": [],
      "source": [
        "df = df[df['level1'] != '[]']\n",
        "df = df[df['level1'] != \"['']\"]\n",
        "df = df[df['paper'] != '']\n",
        "df = df[df['title_full'] != '']\n",
        "df = df[df['title_full'] != '[]']\n",
        "df = df[df['title_full'] != \"['']\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "050f3c14",
      "metadata": {
        "id": "050f3c14"
      },
      "source": [
        "### 3.4 Rename `level1` label to make it more consistent"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 50,
      "id": "e8c99f63",
      "metadata": {
        "id": "e8c99f63"
      },
      "outputs": [],
      "source": [
        "df.level1 = df.level1.replace(\n",
        "    {'physically_assemble/disassemble' : 'physically_assemble_or_disassemble',\n",
        "     'sense,_send,_or_process_information': 'sense_send_or_process_information',\n",
        "     'maintain_ecological_community':'sustain_ecological_community',\n",
        "     'manipulate_solids,_liquids,_gases,_or_energy':'manipulate_solids_liquids_gases_or_energy'},\n",
        "    regex=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "03591ddd",
      "metadata": {
        "id": "03591ddd"
      },
      "source": [
        "### 3.5 Convert `level1` type to list from string"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 51,
      "id": "f0c60573",
      "metadata": {
        "id": "f0c60573"
      },
      "outputs": [],
      "source": [
        "from ast import literal_eval\n",
        "df['level1'] = df['level1'].apply(literal_eval)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3.6 Make `level1` have only one category"
      ],
      "metadata": {
        "id": "U8EeT4DGSszc"
      },
      "id": "U8EeT4DGSszc"
    },
    {
      "cell_type": "markdown",
      "source": [
        "In order to make this model simple, I will choose only one category out of the given list."
      ],
      "metadata": {
        "id": "S-LoSawxTJb-"
      },
      "id": "S-LoSawxTJb-"
    },
    {
      "cell_type": "code",
      "source": [
        "for index, row in df.iterrows():\n",
        "  df['level1'][index] = df['level1'][index][0]"
      ],
      "metadata": {
        "id": "x9K6OgwISsJH"
      },
      "id": "x9K6OgwISsJH",
      "execution_count": 52,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "b2ac640e",
      "metadata": {
        "id": "b2ac640e"
      },
      "source": [
        "### 3.6 Clean Text (Text Pre-processing)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 53,
      "id": "7a408f9e",
      "metadata": {
        "id": "7a408f9e"
      },
      "outputs": [],
      "source": [
        "import pickle\n",
        "import nltk\n",
        "from nltk import word_tokenize\n",
        "from nltk.corpus import stopwords\n",
        "from nltk.stem import WordNetLemmatizer"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "nltk.download('punkt')\n",
        "nltk.download('stopwords')\n",
        "nltk.download('wordnet')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A6v0S_01M03k",
        "outputId": "d292e0cc-79cd-4fd9-b378-958729e12372"
      },
      "id": "A6v0S_01M03k",
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "wordnet = WordNetLemmatizer()\n",
        "\n",
        "def clean_text(text):\n",
        "    \"\"\"Clean raw text using different methods :\n",
        "       1. tokenize text\n",
        "       2. lower text\n",
        "       3. remove punctuation\n",
        "       4. remove non-alphabetics char\n",
        "       5. remove stopwords\n",
        "       6. lemmatize\n",
        "    \n",
        "    Arguments:\n",
        "        text {string} -- raw text\n",
        "    \n",
        "    Returns:\n",
        "        [string] -- clean text\n",
        "    \"\"\"\n",
        "\n",
        "    # split into words\n",
        "    tokens = word_tokenize(text)\n",
        "    # convert to lower case\n",
        "    tokens = [w.lower() for w in tokens]\n",
        "    # remove punctuation from each word\n",
        "    table = str.maketrans('', '', string.punctuation)\n",
        "    stripped = [w.translate(table) for w in tokens]\n",
        "    # remove remaining tokens that are not alphabetic\n",
        "    words = [word for word in stripped if word.isalpha()]\n",
        "    # filter out stop words\n",
        "    stop_words = set(stopwords.words('english'))\n",
        "    words = [w for w in words if not w in stop_words]\n",
        "\n",
        "    stemmed = [wordnet.lemmatize(word) for word in words]\n",
        "\n",
        "    return ' '.join(stemmed)"
      ],
      "metadata": {
        "id": "HpiDyp0pL9IS"
      },
      "id": "HpiDyp0pL9IS",
      "execution_count": 55,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "title_clean = []\n",
        "abstract_clean = []\n",
        "\n",
        "for index, row in df.iterrows():\n",
        "  title_cleaned = clean_text(row['title_full'])\n",
        "  abstract_cleaned = clean_text(row['abstract_full'])\n",
        "\n",
        "  title_clean.append(title_cleaned)\n",
        "  abstract_clean.append(abstract_cleaned)\n",
        "\n",
        "df['title_clean'] = title_clean\n",
        "df['abstract_clean'] = abstract_clean\n",
        "df.head(3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 196
        },
        "id": "1LHC36kWN0qp",
        "outputId": "6298b034-c1c6-48a8-ce03-da9ca40aaeb7"
      },
      "id": "1LHC36kWN0qp",
      "execution_count": 56,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>paper</th>\n",
              "      <th>level1</th>\n",
              "      <th>abstract_full</th>\n",
              "      <th>title_full</th>\n",
              "      <th>title_clean</th>\n",
              "      <th>abstract_clean</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2103410568</td>\n",
              "      <td>physically_assemble_or_disassemble</td>\n",
              "      <td>Frogs that build foam nests floating on water ...</td>\n",
              "      <td>Building a home from foam—túngara frog foam ne...</td>\n",
              "      <td>building home frog foam nest architecture thre...</td>\n",
              "      <td>frog build foam nest floating water face probl...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2138292607</td>\n",
              "      <td>sense_send_or_process_information</td>\n",
              "      <td>Recent evidence suggests that bats can detect ...</td>\n",
              "      <td>A nocturnal mammal, the greater mouse-eared ba...</td>\n",
              "      <td>nocturnal mammal greater mouseeared bat calibr...</td>\n",
              "      <td>recent evidence suggests bat detect geomagneti...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2005539166</td>\n",
              "      <td>sense_send_or_process_information</td>\n",
              "      <td>SUMMARY The existence of polarization sensitiv...</td>\n",
              "      <td>Polarization sensitivity in two species of cut...</td>\n",
              "      <td>polarization sensitivity two specie cuttlefish...</td>\n",
              "      <td>summary existence polarization sensitivity p l...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        paper  ...                                     abstract_clean\n",
              "0  2103410568  ...  frog build foam nest floating water face probl...\n",
              "1  2138292607  ...  recent evidence suggests bat detect geomagneti...\n",
              "2  2005539166  ...  summary existence polarization sensitivity p l...\n",
              "\n",
              "[3 rows x 6 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 56
        }
      ]
    },
    {
      "cell_type": "markdown",
      "id": "44fd14cf",
      "metadata": {
        "id": "44fd14cf"
      },
      "source": [
        "### 3.7 Data cleaning result"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1b9eed07",
      "metadata": {
        "id": "1b9eed07"
      },
      "source": [
        "#### 3.7.1 Unique categories in `level1`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 57,
      "id": "3cee9036",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3cee9036",
        "outputId": "d994c917-2389-495f-bbe0-5649eabc2abf"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "move\n",
            "attach\n",
            "chemically_modify_or_change_energy_state\n",
            "sustain_ecological_community\n",
            "physically_assemble_or_disassemble\n",
            "maintain_structural_integrity\n",
            "change_size_or_color\n",
            "protect_from_harm\n",
            "process_resources\n",
            "sense_send_or_process_information\n"
          ]
        }
      ],
      "source": [
        "# Find unique category in Level1\n",
        "candidate_labels = []\n",
        "\n",
        "for category in df[\"level1\"]:\n",
        "    candidate_labels.append(category)\n",
        "\n",
        "candidate_labels = list(set(candidate_labels))\n",
        "        \n",
        "for category in candidate_labels:\n",
        "    print(category)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "37c2b01a",
      "metadata": {
        "id": "37c2b01a"
      },
      "source": [
        "#### 3.7.2 Total number of data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 58,
      "id": "f6ba4914",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 213
        },
        "id": "f6ba4914",
        "outputId": "88d06af6-fd8c-42d7-8351-1f373e85aae1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total number of data after cleaning: 1058\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>paper</th>\n",
              "      <th>level1</th>\n",
              "      <th>abstract_full</th>\n",
              "      <th>title_full</th>\n",
              "      <th>title_clean</th>\n",
              "      <th>abstract_clean</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2103410568</td>\n",
              "      <td>physically_assemble_or_disassemble</td>\n",
              "      <td>Frogs that build foam nests floating on water ...</td>\n",
              "      <td>Building a home from foam—túngara frog foam ne...</td>\n",
              "      <td>building home frog foam nest architecture thre...</td>\n",
              "      <td>frog build foam nest floating water face probl...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2138292607</td>\n",
              "      <td>sense_send_or_process_information</td>\n",
              "      <td>Recent evidence suggests that bats can detect ...</td>\n",
              "      <td>A nocturnal mammal, the greater mouse-eared ba...</td>\n",
              "      <td>nocturnal mammal greater mouseeared bat calibr...</td>\n",
              "      <td>recent evidence suggests bat detect geomagneti...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2005539166</td>\n",
              "      <td>sense_send_or_process_information</td>\n",
              "      <td>SUMMARY The existence of polarization sensitiv...</td>\n",
              "      <td>Polarization sensitivity in two species of cut...</td>\n",
              "      <td>polarization sensitivity two specie cuttlefish...</td>\n",
              "      <td>summary existence polarization sensitivity p l...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        paper  ...                                     abstract_clean\n",
              "0  2103410568  ...  frog build foam nest floating water face probl...\n",
              "1  2138292607  ...  recent evidence suggests bat detect geomagneti...\n",
              "2  2005539166  ...  summary existence polarization sensitivity p l...\n",
              "\n",
              "[3 rows x 6 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 58
        }
      ],
      "source": [
        "print(\"Total number of data after cleaning:\", len(df))\n",
        "df.head(3)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "47d70be1",
      "metadata": {
        "id": "47d70be1"
      },
      "source": [
        "## 4. Training"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1ae20b14",
      "metadata": {
        "id": "1ae20b14"
      },
      "source": [
        "After cleaning data, we found that there are 10 different catergory (3.5.1). Here, we are going to classify each biomimicry paper into these category."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4.1 Import packages"
      ],
      "metadata": {
        "id": "GLZaeRxLQnFj"
      },
      "id": "GLZaeRxLQnFj"
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn import model_selection, preprocessing, linear_model, naive_bayes, metrics, svm\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\n",
        "from sklearn import decomposition, ensemble\n",
        "\n",
        "import pandas, xgboost, numpy, textblob, string\n",
        "from keras.preprocessing import text, sequence\n",
        "from keras import layers, models, optimizers"
      ],
      "metadata": {
        "id": "-uEykdoaRqZ-"
      },
      "id": "-uEykdoaRqZ-",
      "execution_count": 59,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4.2 Create train and test data"
      ],
      "metadata": {
        "id": "giCF-B8VR-mn"
      },
      "id": "giCF-B8VR-mn"
    },
    {
      "cell_type": "code",
      "source": [
        "df_train, df_test = train_test_split(df, test_size=0.2)\n",
        "X_train, X_test, y_train, y_test = train_test_split(df_train[\"abstract_clean\"], df_train['level1'])\n",
        "\n",
        "# label encode the target variable \n",
        "encoder = preprocessing.LabelEncoder()\n",
        "y_train = encoder.fit_transform(y_train)\n",
        "y_test = encoder.fit_transform(y_test)"
      ],
      "metadata": {
        "id": "cO0I0L9mSFPI"
      },
      "id": "cO0I0L9mSFPI",
      "execution_count": 60,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4.3 Word2Vec"
      ],
      "metadata": {
        "id": "lGsnHFMhV9rN"
      },
      "id": "lGsnHFMhV9rN"
    },
    {
      "cell_type": "code",
      "source": [
        "# create a count vectorizer object \n",
        "count_vect = CountVectorizer(analyzer='word', token_pattern=r'\\w{1,}')\n",
        "count_vect.fit(df_train['abstract_clean'])\n",
        "\n",
        "# transform the training and validation data using count vectorizer object\n",
        "xtrain_count =  count_vect.transform(X_train)\n",
        "xvalid_count =  count_vect.transform(X_test)"
      ],
      "metadata": {
        "id": "isWexdboYqFO"
      },
      "id": "isWexdboYqFO",
      "execution_count": 61,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4.4 Term Frequency and Inverse Document Frequency"
      ],
      "metadata": {
        "id": "zg3qXPqpWGwe"
      },
      "id": "zg3qXPqpWGwe"
    },
    {
      "cell_type": "code",
      "source": [
        "# word level tf-idf\n",
        "tfidf_vect = TfidfVectorizer(analyzer='word', token_pattern=r'\\w{1,}', max_features=5000)\n",
        "tfidf_vect.fit(df_train['abstract_clean'])\n",
        "xtrain_tfidf =  tfidf_vect.transform(X_train)\n",
        "xvalid_tfidf =  tfidf_vect.transform(X_test)\n",
        "\n",
        "# ngram level tf-idf \n",
        "tfidf_vect_ngram = TfidfVectorizer(analyzer='word', token_pattern=r'\\w{1,}', ngram_range=(2,3), max_features=5000)\n",
        "tfidf_vect_ngram.fit(df_train['abstract_clean'])\n",
        "xtrain_tfidf_ngram =  tfidf_vect_ngram.transform(X_train)\n",
        "xvalid_tfidf_ngram =  tfidf_vect_ngram.transform(X_test)"
      ],
      "metadata": {
        "id": "xDDj5sDyZNTW"
      },
      "id": "xDDj5sDyZNTW",
      "execution_count": 62,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4.5 Training Function"
      ],
      "metadata": {
        "id": "Ker_iAw0nfm-"
      },
      "id": "Ker_iAw0nfm-"
    },
    {
      "cell_type": "code",
      "source": [
        "def train_model(classifier, feature_vector_train, label, feature_vector_valid, is_neural_net=False):\n",
        "    # fit the training dataset on the classifier\n",
        "    classifier.fit(feature_vector_train, label)\n",
        "    \n",
        "    # predict the labels on validation dataset\n",
        "    predictions = classifier.predict(feature_vector_valid)\n",
        "    \n",
        "    if is_neural_net:\n",
        "        predictions = predictions.argmax(axis=-1)\n",
        "    \n",
        "    return metrics.accuracy_score(predictions, y_test)"
      ],
      "metadata": {
        "id": "wwc5PVrHZekJ"
      },
      "id": "wwc5PVrHZekJ",
      "execution_count": 63,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 5 Machine Learning ModeL"
      ],
      "metadata": {
        "id": "DX0PByKRnk3e"
      },
      "id": "DX0PByKRnk3e"
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 5.1 Naive Bayes Model"
      ],
      "metadata": {
        "id": "R_LgV8dZn3dq"
      },
      "id": "R_LgV8dZn3dq"
    },
    {
      "cell_type": "code",
      "source": [
        "# Naive Bayes on Count Vectors\n",
        "accuracy = train_model(naive_bayes.MultinomialNB(), xtrain_count, y_train, xvalid_count)\n",
        "print (\"NB, Count Vectors: \", accuracy)\n",
        "\n",
        "# Naive Bayes on Word Level TF IDF Vectors\n",
        "accuracy = train_model(naive_bayes.MultinomialNB(), xtrain_tfidf, y_train, xvalid_tfidf)\n",
        "print (\"NB, WordLevel TF-IDF: \", accuracy)\n",
        "\n",
        "# Naive Bayes on Ngram Level TF IDF Vectors\n",
        "accuracy = train_model(naive_bayes.MultinomialNB(), xtrain_tfidf_ngram, y_train, xvalid_tfidf_ngram)\n",
        "print (\"NB, N-Gram Vectors: \", accuracy)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AZDPz59UnEhc",
        "outputId": "c6bd13d9-4098-4401-bb0e-04af41eed7e3"
      },
      "id": "AZDPz59UnEhc",
      "execution_count": 64,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "NB, Count Vectors:  0.5754716981132075\n",
            "NB, WordLevel TF-IDF:  0.3490566037735849\n",
            "NB, N-Gram Vectors:  0.29245283018867924\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 5.2 Linear Classifier"
      ],
      "metadata": {
        "id": "Inam0689n5_L"
      },
      "id": "Inam0689n5_L"
    },
    {
      "cell_type": "code",
      "source": [
        "# Linear Classifier on Count Vectors\n",
        "accuracy = train_model(linear_model.LogisticRegression(), xtrain_count, y_train, xvalid_count)\n",
        "print (\"LR, Count Vectors: \", accuracy)\n",
        "\n",
        "# Linear Classifier on Word Level TF IDF Vectors\n",
        "accuracy = train_model(linear_model.LogisticRegression(), xtrain_tfidf, y_train, xvalid_tfidf)\n",
        "print (\"LR, WordLevel TF-IDF: \", accuracy)\n",
        "\n",
        "# Linear Classifier on Ngram Level TF IDF Vectors\n",
        "accuracy = train_model(linear_model.LogisticRegression(), xtrain_tfidf_ngram, y_train, xvalid_tfidf_ngram)\n",
        "print (\"LR, N-Gram Vectors: \", accuracy)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-DTnwTCALchN",
        "outputId": "133b8e85-62b6-4dae-b3cf-098b20266243"
      },
      "id": "-DTnwTCALchN",
      "execution_count": 65,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "LR, Count Vectors:  0.5377358490566038\n",
            "LR, WordLevel TF-IDF:  0.4481132075471698\n",
            "LR, N-Gram Vectors:  0.28773584905660377\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 6. Result"
      ],
      "metadata": {
        "id": "f--Ln6bxokDS"
      },
      "id": "f--Ln6bxokDS"
    },
    {
      "cell_type": "markdown",
      "source": [
        "The best model is Naive Bayes model using count vectors. Since TF-IDF is widely used in NLP, I expected TF-IDF would help model to have a higher accuracy, but it doesn't. TF-IDF cares about how many times a word appears in a document and the inverse document frequency of the word across a set of documents. It penalizes too frequent words in the document and gives more weights to the rare words in general."
      ],
      "metadata": {
        "id": "HRIUxfmToqfU"
      },
      "id": "HRIUxfmToqfU"
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 7. Future work"
      ],
      "metadata": {
        "id": "GtsFCvg2TRn-"
      },
      "id": "GtsFCvg2TRn-"
    },
    {
      "cell_type": "markdown",
      "source": [
        "The original dataset has a list of categories for each paper but I only keep only one to make this model simple. This model can be modified to have more than one expected values for each paper. I strongly believe the precision could be higher as there are more than one expected value, which lead the model to have a broader coverage for the predicted value.\n",
        "\n",
        "Moreover, I'd like to use convolutional neural network and recurrent neural network (LSTM) with word embeddings in future."
      ],
      "metadata": {
        "id": "de0fUr0sTZZG"
      },
      "id": "de0fUr0sTZZG"
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.8"
    },
    "colab": {
      "name": "Text Classification.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 5
}