Part 2 - Classification with BoW and TF-IDF.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2 - Classification with Bag of Words and TF-IDF\n",
    "\n",
    "This is part 2 of the **Natural Language Processing** series. If you haven't already done or watched part 1, you can find the [notebook and presentation here](https://github.com/IBMDeveloperUK/Analysing-Jokes-with-Natural-Language-Processing). \n",
    "\n",
    "In this notebook we will look at the task of classification. We will continue to work with the dataset of jokes that were scraped from the web but this time we're interested in how we can automatically categorize jokes.\n",
    "\n",
    "The libraries we'll be using are:\n",
    "- [NLTK (the Natural Language Toolkit)](https://www.nltk.org/) a very easy Python library for working with text data and NLP.\n",
    "- [scikit-learn](https://scikit-learn.org/stable/) a library dedicated to machine learning (and other statistics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-processing\n",
    "\n",
    "To make things easier for this notebook, we're going to define one big function that will handle all of the pre-processing steps we want to do. We covered how all of these steps work in part 1 but let's quickly remind ourselves.\n",
    "\n",
    "1. First up, we take every word and make it **lowercase**. This avoids case-sensitive duplicates. \n",
    "2. Next, we expand some very common **contractions** in the english language (e.g. we'll = we will).\n",
    "3. Thirdly we strip all other **punctuation** out of the text.\n",
    "4. We now turn our jokes in to lists of words (**tokens**)\n",
    "5. Remove any tokens that are included in our **stopwords**.\n",
    "6. Tag the **part of speech** of each word.\n",
    "7. Use the part of speech to reduce each token to it's **lemma**.\n",
    "\n",
    "By doing all of this we drastically reduce the number of unique words we have to work with whilst still maintaining almost all of the meaning. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\n",
    "import re\n",
    "from nltk.corpus import wordnet\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "from nltk import pos_tag\n",
    "from nltk.stem import WordNetLemmatizer \n",
    "\n",
    "\n",
    "def preprocess_jokes(jokes):\n",
    "    #Make them all lowercase\n",
    "    jokes_no_empties = [joke.lower() for joke in jokes]\n",
    "    \n",
    "    #Replace english language contractions\n",
    "    jokes_expanded = []\n",
    "    for joke in jokes_no_empties:\n",
    "        # specific\n",
    "        joke = re.sub(r\"won\\'t\", \"will not\", joke)\n",
    "        joke = re.sub(r\"can\\'t\", \"can not\", joke)\n",
    "\n",
    "        # general\n",
    "        joke = re.sub(r\"n\\'t\", \" not\", joke)\n",
    "        joke = re.sub(r\"\\'re\", \" are\", joke)\n",
    "        joke = re.sub(r\"\\'s\", \" is\", joke)\n",
    "        joke = re.sub(r\"\\'d\", \" would\", joke)\n",
    "        joke = re.sub(r\"\\'ll\", \" will\", joke)\n",
    "        joke = re.sub(r\"\\'t\", \" not\", joke)\n",
    "        joke = re.sub(r\"\\'ve\", \" have\", joke)\n",
    "        joke = re.sub(r\"\\'m\", \" am\", joke)\n",
    "        jokes_expanded.append(joke)\n",
    "    \n",
    "    #Remove all other punctuation\n",
    "    jokes_no_punct = [joke.translate(str.maketrans(string.punctuation, ' '*len(string.punctuation))) for joke in jokes_expanded]\n",
    "    \n",
    "    #Tokenize each of the jokes\n",
    "    jokes_tokenized = [word_tokenize(joke) for joke in jokes_no_punct]\n",
    "    \n",
    "    #Remove stopwords\n",
    "    stop_words = set(stopwords.words('english'))\n",
    "    jokes_no_stops = []\n",
    "    for joke in jokes_tokenized:\n",
    "        joke_no_stops = [word for word in joke if word not in stop_words]\n",
    "        jokes_no_stops.append(joke_no_stops)\n",
    "    \n",
    "    #Tag parts of speech using nltk pos_tag and convert to WN format\n",
    "    pos_tagged_jokes = [pos_tag(joke) for joke in jokes_no_stops]\n",
    "    def get_wordnet_pos(treebank_tag):\n",
    "        if treebank_tag.startswith('J'):\n",
    "            return wordnet.ADJ\n",
    "        elif treebank_tag.startswith('V'):\n",
    "            return wordnet.VERB\n",
    "        elif treebank_tag.startswith('N'):\n",
    "            return wordnet.NOUN\n",
    "        elif treebank_tag.startswith('R'):\n",
    "            return wordnet.ADV\n",
    "        else:\n",
    "            return 'n'\n",
    "    wn_pos_jokes = []\n",
    "    for joke in pos_tagged_jokes:\n",
    "        wn_pos_joke = [(val[0], get_wordnet_pos(val[1])) for val in joke]\n",
    "        wn_pos_jokes.append(wn_pos_joke)\n",
    "    \n",
    "    #Lemmatize all the individual words\n",
    "    lemmatizer = WordNetLemmatizer() \n",
    "    lemmatized_jokes = []\n",
    "    for joke in wn_pos_jokes:\n",
    "        lemmatized = [lemmatizer.lemmatize(token[0], pos = token[1]) for token in joke]\n",
    "        lemmatized_jokes.append(lemmatized)\n",
    "\n",
    "    \n",
    "    return lemmatized_jokes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing the data\n",
    "\n",
    "We're using the same file as last time `stupidstuff.json`. We will import it, save it to a variable called `data` and then remove any empty entries from the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "with open('stupidstuff.json') as json_file:\n",
    "    data = json.load(json_file)\n",
    "print(\"Number of data entries {}\".format(len(data)))\n",
    "data = [item for item in data if not item['body'] == '']\n",
    "print(\"Number of entries after removing empties {}\".format(len(data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Looking at the categories\n",
    "\n",
    "Because we are interested in [classifying](https://en.wikipedia.org/wiki/Statistical_classification) our jokes, we need to look at the categories that have been assigned to each of them so that we can design our model. Let's count how many are in each category and then sort them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "categories = [data[i][\"category\"] for i in range(len(data))]\n",
    "categories_counts = dict(Counter(categories))\n",
    "sorted_categories = {k: v for k, v in sorted(categories_counts.items(), key=lambda item: item[1])}\n",
    "sorted_categories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choosing an appropriate subset\n",
    "\n",
    "We can see that there are actually LOTS of categories. On top of this, many of the categories have very few jokes in them at all. These will be particularly difficult for us to work with because we don't have enough training data. \n",
    "\n",
    "There is also one category `Miscellaneous` that is significantly bigger than the others. In order to achieve reasonable results in multiclass classification we ideally want a similar number of training examples for each class. \n",
    "\n",
    "If we ignore the `Miscellaneous` class, let's try taking the next 7 biggest categories and use them to build our classification model. That still gives us a reasonable spread of categories and a decent number (100+) of training examples for each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "category_dict = {'Insults':0, 'Men':1, 'Women':2, 'Yo Mama':3, 'Light Bulbs':4, 'Religious':5, 'Political':6}\n",
    "jokes = []\n",
    "labels = []\n",
    "for entry in data:\n",
    "    if entry['category'] in category_dict:\n",
    "        jokes.append(entry['body'])\n",
    "        labels.append(category_dict[entry['category']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PreProcessing the data subset\n",
    "\n",
    "You'll notice that we also created a `labels` list which will store the category of each joke. This is important for both training and testing, as you'll find out later on.\n",
    "\n",
    "We can now preprocess the data using the function we defined at the beginning of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "jokes_raw = jokes\n",
    "jokes = preprocess_jokes(jokes)\n",
    "jokes[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see from the first joke printed that the tokens look nice and clean, they all contain relevant words to the joke. We can see immediately that this joke is about Bill and Hillary Clinton. Maybe that tells you something about the age of this dataset 😎"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Picking an example to work through\n",
    "\n",
    "Let's choose an individual joke to work with so that we can get a feel for how the bag of words model works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(jokes_raw[6])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here's what the joke looks like after pre-processing as well as it's category:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "joke = jokes[6]\n",
    "print(joke)\n",
    "joke_label = labels[6]\n",
    "print('The joke\\'s category is: '+list(category_dict.keys())[list(category_dict.values()).index(joke_label)] +' which has the label value '+str(joke_label))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building a bag of words dictionary\n",
    "\n",
    "The first thing we need to do with any [bag of words model](https://en.wikipedia.org/wiki/Bag-of-words_model) is to build a **Dictionary**. \n",
    "\n",
    "The **Dictionary** allows us to map the individual tokens to their indices in a vector later on. It involves collecting all of the unique words in the training data and storing an entry for each one in the dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "unique_words = set(joke)\n",
    "index = 0\n",
    "dictionary = {}\n",
    "for word in unique_words:\n",
    "    dictionary[word] = index\n",
    "    index += 1\n",
    "dictionary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compiling a bag of words vector\n",
    "\n",
    "Once we've built our dictionary, we can now create the **Bag of Words Vector** for our joke. This is the important part because it is the vector representation of the joke that we are trying to classify. \n",
    "\n",
    "By representing each joke as a vector (i.e. a series of numbers) we are making it computer friendly. It allows us to run mathematical models and algorithms on the data in ways we couldn't do with data stored as a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bag_of_words = len(dictionary) * [0]\n",
    "for word in joke:\n",
    "    if word in dictionary:\n",
    "        index = dictionary[word]\n",
    "        bag_of_words[index] += 1\n",
    "print(bag_of_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating vectors for new jokes\n",
    "\n",
    "Now that we've created a dictionary and a vector for our joke. Let's try vectorizing another joke with the same dictionary and see what happens..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "joke_2 = jokes[500]\n",
    "print(joke_2)\n",
    "joke_label = labels[500]\n",
    "print('The joke\\'s category is: '+list(category_dict.keys())[list(category_dict.values()).index(joke_label)] +' which has the label value '+str(joke_label))\n",
    "bag_of_words = len(dictionary) * [0]\n",
    "for word in joke_2:\n",
    "    if word in dictionary:\n",
    "        index = dictionary[word]\n",
    "        bag_of_words[index] += 1\n",
    "print(bag_of_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bag of words caveats\n",
    "\n",
    "Oh no! you can see that the bag of words vector is completely empty (all zero values). That's because none of the words from our second joke appear in the first.\n",
    "\n",
    "This is one of the problems with the bag-of-words model - it doesn't handle unseen words very well. This is why it's important to train a bag of words model with lots of examples so that it has a long dictionary and is unlikely to not have seen a word before. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take these two jokes we've used and keep them as examples for later. We can then test our machine learning model with them to see if it works!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#remove joke 1 and 2 from the training data\n",
    "jokes_for_testing = []\n",
    "labels_for_testing = []\n",
    "jokes_for_testing.append(jokes.pop(6));\n",
    "jokes_for_testing.append(jokes.pop(500));\n",
    "labels_for_testing.append(labels.pop(6));\n",
    "labels_for_testing.append(labels.pop(500));\n",
    "labels_for_testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dictionaries at scale\n",
    "\n",
    "Last time we built a bag-of-words dictionary it was only for a single joke. To create our classifier we're going to need a dictionary that represents all of the words in the training set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_bow_dictionary(jokes):\n",
    "    index = 0\n",
    "    dictionary = {}\n",
    "    for joke in jokes:\n",
    "        unique_words = set(joke)\n",
    "        for word in unique_words:\n",
    "            if word not in dictionary:\n",
    "                dictionary[word] = index\n",
    "                index += 1\n",
    "    return dictionary\n",
    "dictionary = build_bow_dictionary(jokes)\n",
    "len(dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bag of words vector function\n",
    "\n",
    "Let's also build a function that will allow us to generate the bag of words vectors for any given dictionary. This will be handy if we want to try out different datasets or a different subset of our joke dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_bow_vector(joke, dictionary):\n",
    "    bow_vector = len(dictionary) * [0]\n",
    "    for word in joke:\n",
    "        if word in dictionary:\n",
    "            index = dictionary[word]\n",
    "            bow_vector[index] += 1\n",
    "    return bow_vector\n",
    "print(build_bow_vector(joke, dictionary))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use this function to genereate vectors for every single one of our jokes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bow_vectors = [build_bow_vector(joke, dictionary) for joke in jokes]\n",
    "print(bow_vectors[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting the data\n",
    "\n",
    "Before we train our model, we want to separate our dataset into two parts: a **training set** and a **test set**.\n",
    "\n",
    "Our **training set** will be used to train the model along with it's corresponding labels and the **test set** will then be evaluated against that model to give us an accuracy value i.e. how many of the test set's label's the classifier got right!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Split to train and test data here\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(bow_vectors, labels, train_size=0.8, random_state=97)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building the model\n",
    "\n",
    "We're going to use the [Multinomail Naive-Bayes](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html) algorithm from the scikit-learn library. This is a good choice for multi-class classification problems around NLP and works particularly well with the bag of words model.\n",
    "\n",
    "We import this and then train it on our **training set** `X_train` and **training labels** `y_train`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "classifier = MultinomialNB()\n",
    "classifier.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making a prediction\n",
    "\n",
    "Now that the model has been built, we can make predictions on our test set and see how well it performs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = classifier.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking the accuracy\n",
    "\n",
    "Finally, we can then compare the predictions the model made with the actual labels for each of our test jokes. This will give us an accuracy value that is indicative of how good our model is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking with our jokes from earlier\n",
    "\n",
    "Let's feed in the two jokes we set aside earlier and see what the classifier predicts!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "unseen_joke_vectors = [build_bow_vector(joke, dictionary) for joke in jokes_for_testing]\n",
    "unseen_preds = classifier.predict(unseen_joke_vectors)\n",
    "for i in range(len(unseen_preds)):\n",
    "    correct_or_not = 'incorrectly'\n",
    "    if unseen_preds[i] == labels_for_testing[i]:\n",
    "        correct_or_not = 'correctly'\n",
    "    label_verbose = list(category_dict.keys())[list(category_dict.values()).index(unseen_preds[i])]\n",
    "    print(\"The classifier {} predicted the category to be: {}\".format(correct_or_not, label_verbose))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Great stuff! It predicted our categories correctly.\n",
    "\n",
    "In future, when you're working with the bag-of-words model, you don't actually have to write out the functions yourself. \n",
    "\n",
    "We did it because it was a good way to understand what's going on but you can use the CountVectorizer() function from scikit-learn to do it much more quickly:\n",
    "```\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vectorizer = CountVectorizer()\n",
    " \n",
    "term_frequencies = vectorizer.fit_transform(jokes)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Term Frequency - Inverse Document Frequency\n",
    "\n",
    "Our model performed pretty well, but can we do any better with a different approach? Let's try building a model using tf-idf representation instead of just bag-of-words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "vectorizer = TfidfVectorizer()\n",
    "joined_jokes = [' '.join(joke) for joke in jokes]\n",
    "tfidf = vectorizer.fit_transform(joined_jokes)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting the data\n",
    "\n",
    "Again, let's separate our data in to **training set** and **test set**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(tfidf, labels, train_size=0.8, random_state=97)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And once again, build the model..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier = MultinomialNB()\n",
    "classifier.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and create predictions and evaluate the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = classifier.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Good stuff! Our model is now even better!\n",
    "\n",
    "TF-IDF has performed slightly better, but only marginally in this case. That's most likely because we did such a great job in pre-processing."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}