NER-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3e89c5b3",
   "metadata": {},
   "source": [
    "This notebook was created by [William Mattingly](https://datascience.si.edu/people/dr-william-mattingly) for the 2022 Text Analysis Pedagogy Institute, with support from the [National Endowment for the Humanities](https://neh.gov), [JSTOR Labs](https://labs.jstor.org/), and [University of Arizona Libraries](https://new.library.arizona.edu/).\n",
    "\n",
    "The exercises in this notebook are based on the notebooks created by [Zoe LeBlanc](https://ischool.illinois.edu/people/zoe-leblanc) for the 2021 Text Analysis Pedagogy Institute, with support from the [National Endowment for the Humanities](https://neh.gov), [JSTOR Labs](https://labs.jstor.org/), and [University of Virginia Libraries](https://library.virginia.edu).\n",
    "\n",
    "\n",
    "This notebook is adapted by Zhuo Chen under [Creative Commons CC BY License](https://creativecommons.org/licenses/by/4.0/).\n",
    "\n",
    "For questions/comments/improvements, email zhuo.chen@ithaka.org or nathan.kelber@ithaka.org<br />\n",
    "____"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68f932d1",
   "metadata": {},
   "source": [
    "# Multilingual NER 2\n",
    "\n",
    "This is lesson 2 of 3 in the educational series on multilingual NER. This notebook is focused on rules-based NER. \n",
    "\n",
    "**Description:** This notebook describes how to:\n",
    "\n",
    "* Use spaCy to do rule-based NER\n",
    "* Create an EntityRuler\n",
    "* Identify Languages of a Corpus\n",
    "\n",
    "**Use case:** For Learners (Detailed explanation, not ideal for researchers)\n",
    "\n",
    "**Difficulty:** Intermediate\n",
    "\n",
    "**Completion time:** 90 minutes\n",
    "\n",
    "**Knowledge Required:** \n",
    "\n",
    "* Python Basics ([start learning Python basics](./python-basics-1.ipynb))\n",
    "* [Python intermediate 4](./python-intermediate-4.ipynb)\n",
    "* [Regular expressions](./regular-expressions.ipynb)\n",
    "\n",
    "**Knowledge Recommended:** None\n",
    "\n",
    "**Data Format**: .csv\n",
    "\n",
    "**Libraries Used**: spaCy\n",
    "\n",
    "**Research Pipeline**: None\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "157c0555",
   "metadata": {},
   "source": [
    "# Install required Python libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2fd0c5a-201c-4247-9182-7c238fcff3b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install spacy # for NLP and NER\n",
    "!pip3 install pandas # for working with tabular data in the exercises of this notebook\n",
    "!python3 -m spacy download en_core_web_sm # for English NER\n",
    "!python3 -m spacy download es_core_news_sm # for Spanish NER"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "faa33f3d-8acb-4e7d-9484-adf1b405fff6",
   "metadata": {},
   "source": [
    "# Introduction to spaCy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99f1f94f-a290-46a4-b3cb-6efb4e7f1f1c",
   "metadata": {},
   "source": [
    "The spaCy (spelled correctly) library is a robust library for Natural Language Processing. It supports a wide variety of languages with statistical models capable of parsing texts, identifying parts-of-speech, and extract entities. \n",
    "\n",
    "Let's see an example of NLP task that spaCy can do for us.\n",
    "\n",
    "## Tokenization\n",
    "Recall that last time we have seen a graph showing the NLP pipelines. A pipeline's purpose is to take input data, perform some sort of operations on that input data, and then output some useful information from the data. On a pipeline, we find the pipes. A pipe is an individual component of a pipeline. Different pipes perform different tasks. After we read in the data from a text file, an essential task of NLP is tokenization. \n",
    "\n",
    "<center><img src='https://ithaka-labs.s3.amazonaws.com/static-files/images/tdm/tdmdocs/NER_NLP_pipeline.png' width=700></center>\n",
    "\n",
    "One form of tokenization is **word tokenization**. When we do word tokenization, we break a text up into individual words and punctuations. Another form of tokenization is **sentence tokenization**. Sentence tokenization is precisely the same as word tokenization, except instead of breaking a text up into individual words and punctuations, we break a text up into individual sentences.\n",
    "\n",
    "If you are an English speaker, you may think you do not need spaCy for sentence tokenization, because in English, the end of a sentence is indicated by a period `.`. Why not just use the the built-in `split()` function which allows us to split a text string by the period `.`? \n",
    "\n",
    "This is a ligit question, but simply splitting a text string by the period `.` will run into problems sometimes and spaCy is actually way more smarter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1972efd0-6796-4f57-84da-d1556bb34e2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# String to be split\n",
    "text = \"Martin J. Thompson is known for his writing skills. He is also good at programming.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2eef0746-500d-4b19-859c-a169f95bf728",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split the string by period\n",
    "sents = text.split(\".\")\n",
    "print(sents)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff03c2e6-ad40-402b-bd56-1a41ec717e3a",
   "metadata": {},
   "source": [
    "We had the unfortunate result of splitting at Martin J. The reason for this is obvious. In English, it is common convention to indicate abbreviation with the same punctuation mark used to indicate the end of a sentence. \n",
    "\n",
    "We can use SpaCy, however, to do sentence tokenization. SpaCy is smart enough to not break at Martin J."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdd1b5d9-916b-4d23-849f-05521020971c",
   "metadata": {},
   "source": [
    "First, let's import the spaCy library. Then, we need to load an NLP model object. To do this, we use the `spacy.load()` function. Here, we load the small English NLP model trained on written web text that includes vocabulary, syntax and entities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74893c3e-8f54-4ce8-a4a3-a9322cb43219",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the small English NLP model\n",
    "import spacy\n",
    "nlp = spacy.load(\"en_core_web_sm\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "361eac93-bbe0-4fab-9ef6-09d8b2199a71",
   "metadata": {},
   "source": [
    "We can use this English NLP model to parse a text and create a Doc object. If you need a quick refresh about what classes and object are, you can refer to [Python intermediate 4](./python-intermediate-4.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3822520c-2982-41e4-9bf8-73502574ac2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the English model to parse the text we created\n",
    "doc = nlp(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05bee644",
   "metadata": {},
   "source": [
    "There is a lot of data stored in the Doc object. For example, we can iterate over the sentences in the Doc object and print them out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92f3b1dd-0d64-488e-b777-a71a5e9f0d27",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the sentence tokens in doc\n",
    "for sent in doc.sents:\n",
    "    print(sent)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ee61835-ba86-4875-b80d-a70c3941da85",
   "metadata": {},
   "source": [
    "# spaCy's built-in NER"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfbbdbc0-64f6-402e-9c8a-05d80476c204",
   "metadata": {},
   "source": [
    "We have seen one example NLP task that spaCy can do for us. Now let's move on to named entity recognition, the NLP task we focus on in this series.\n",
    "\n",
    "SpaCy already has a built-in NER off the shelf for us to use. \n",
    "\n",
    "We will iterate over the doc object as we did above, but instead of iterating over `doc.sents`, we will iterate over `doc.ents`. For our purposes right now, we simply want to get each entity's text (the string itself) and its corresponding label (note the underscore `_` after label)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef33dd7e-e00f-47fb-9df4-2f83b2694111",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print out the entities in the doc object together with their labels\n",
    "for ent in doc.ents: # iterate over the entities \n",
    "    print (ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ead2f1b1-7fd6-4467-9d55-f026e6b2e47e",
   "metadata": {},
   "source": [
    "As we can see the small English model has correctly identified that Martin J. Thompson is an entity and given it the correct label PERSON."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09f5ce23",
   "metadata": {},
   "source": [
    "Of course we have many different kinds of entities. Here is a list of entity labels used by the small English NLP model we loaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd4b1939",
   "metadata": {},
   "outputs": [],
   "source": [
    "# List of labels in the small English model for NER\n",
    "nlp.get_pipe(\"ner\").labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "981c382a",
   "metadata": {},
   "source": [
    "If you would like to know the meaning of a label, you can use the `explain` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02bdd129",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get what a label means\n",
    "spacy.explain('NORP')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b942834f",
   "metadata": {},
   "source": [
    "<h2 style=\"color:red; display:inline\">Coding Challenge! &lt; / &gt; </h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18d68491",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Download the .csv file for this exercise\n",
    "import urllib.request\n",
    "url = 'https://ithaka-labs.s3.amazonaws.com/static-files/images/tdm/tdmdocs/NER_Harry_Potter_1.csv'\n",
    "urllib.request.urlretrieve(url, './data/' + url.rsplit('/', 1)[-1])   \n",
    "print('Sample file ready.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b41aaf8",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Take a look at the first five rows of the table\n",
    "import pandas as pd\n",
    "df = pd.read_csv('./data/NER_Harry_Potter_1.csv', delimiter=';')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccbce6b3",
   "metadata": {},
   "source": [
    "In this table we find the name of the characters speaking and their speech. \n",
    "\n",
    "Can you make two new columns, \"Entities\" and \"Labels\", such that each row of the \"Entities\" column stores a list of entities found in the sentence in the same row and each row of the \"Labels\" column stores a list of labels for the entities?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89a7b4e9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "35545318-8fb3-48ef-bfa1-418f280786b6",
   "metadata": {},
   "source": [
    "# spaCy's EntityRuler"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb707b61-895b-47da-b2ec-a94bbf22d28e",
   "metadata": {},
   "source": [
    "Life would be so easy if we could just grab the ready-to-use built-in NER of spaCy and apply it to the large volume of data we have at hand. However, things are not that easy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11b4c182-a823-4390-bc24-9ff0fd308f30",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Another sample text string\n",
    "text = \"Aars is a small town in Denmark. The town was founded in the 14th century.\"\n",
    "\n",
    "#Create the Doc object\n",
    "doc = nlp(text)\n",
    "\n",
    "#extract entities\n",
    "for ent in doc.ents:\n",
    "    print (ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfc8bfb8",
   "metadata": {},
   "source": [
    "We see that the built-in NER failed to identify Aars as an entity of the GPE type. If we do want to extract 'Aars' from the text and give it a label of GPE, what can we do? "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "365df555",
   "metadata": {},
   "source": [
    "## Add EntityRuler as a new pipe\n",
    "\n",
    "Recall that we have talked about the pipes in a pipeline at the beginning of this lesson. In the case of spaCy, there are a few different pipes that perform different tasks. The tokenizer tokenizes the text into individual tokens; the parser parses the text, and the NER identifies entities and labels them accordingly. When we create a Doc object, all of this data is stored in the Doc object.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22121bd4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Take a look at the current pipes\n",
    "nlp.pipe_names"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9c60267",
   "metadata": {},
   "source": [
    "The EntityRuler is a spaCy factory that allows one to create a set of patterns with corresponding labels. In order to extract the target entities and label them successfully, we can create an EntityRuler, give it some instructions, and then add it to the spaCy pipeline as a new pipe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14c5d665",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the EntityRuler\n",
    "ruler = nlp.add_pipe(\"entity_ruler\")\n",
    "\n",
    "# List of Entities and Patterns\n",
    "patterns = [\n",
    "                {\"label\": \"GPE\", \"pattern\": \"Aars\"}\n",
    "            ]\n",
    "\n",
    "ruler.add_patterns(patterns)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33a5c4fd",
   "metadata": {},
   "source": [
    "After we add the EntityRuler, we can use the new pipeline to do NER. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca606d51",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the new model to parse the text and create a new Doc object\n",
    "doc = nlp(text)\n",
    "\n",
    "# Iterate over the entities and print them out\n",
    "for ent in doc.ents: \n",
    "    print (ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29f10ee0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Take a look at the pipes in the new pipeline\n",
    "nlp.pipe_names"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52bfa7ab",
   "metadata": {},
   "source": [
    "## The importance of order"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "097340de",
   "metadata": {},
   "source": [
    "It is important to remember that pipelines are sequential. This means that components earlier in a pipeline affect what later components receive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "958284e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the new model to parse a new text string\n",
    "text = \"Xiong'an is a satellite city of Beijing.\"\n",
    "nlp1 = spacy.load(\"en_core_web_sm\")\n",
    "doc=nlp1(text)\n",
    "for ent in doc.ents:\n",
    "    print(ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2b17441",
   "metadata": {},
   "source": [
    "Xiong'an is a name of a city. We would want to label it as GPE, not ORG. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71b6359b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the EntityRuler\n",
    "ruler = nlp1.add_pipe(\"entity_ruler\")\n",
    "\n",
    "# List of Entities and Patterns\n",
    "patterns = [\n",
    "                {\"label\": \"GPE\", \"pattern\": \"Xiong'an\"}\n",
    "            ]\n",
    "\n",
    "ruler.add_patterns(patterns)\n",
    "\n",
    "# Get the entities\n",
    "doc = nlp1(text)\n",
    "for ent in doc.ents:\n",
    "    print(ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b274b0b",
   "metadata": {},
   "source": [
    "Why do we still mislabel Xiong'an? This is because when we add the EntityRuler as a new pipe, it gets added at the end of the pipeline automatically. That means the EntityRuler will come after the built-in NER in spaCy. Since NER is a hard classification task, an entity that gets labeled will not be relabeled. If Xiong'an is labeled already by the built-in NER as ORG, it will not be relabeled by the EntityRuler that comes after. In order to give the EntityRuler primacy, we will have to put it in a position before the built-in NER when we add it so that it takes primacy over the built-in NER. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d4d24ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the model\n",
    "nlp2 = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "# Create the EntityRuler and add it to the model\n",
    "ruler = nlp2.add_pipe(\"entity_ruler\", before='ner') # specify that the EntityRuler comes before built-in NER\n",
    "\n",
    "# Add the new patterns to the ruler\n",
    "patterns = [\n",
    "                {\"label\": \"GPE\", \"pattern\": \"Xiong'an\"}\n",
    "            ]\n",
    "\n",
    "ruler.add_patterns(patterns)\n",
    "\n",
    "# Use the new model to parse the text\n",
    "doc = nlp2(text)\n",
    "\n",
    "# Get the entities\n",
    "for ent in doc.ents:\n",
    "    print(ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36cd8c93",
   "metadata": {},
   "outputs": [],
   "source": [
    "# EntityRuler comes before the built in ner in nlp2\n",
    "nlp2.pipe_names"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "291db136",
   "metadata": {},
   "source": [
    "So far, we only add exact strings to our EntityRuler. However, when we talk about patterns, we usually talk about more abstract patterns, not fixed strings. In the following, we will see an example where we write a regular expression pattern and add it to the EntityRuler."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df6041af",
   "metadata": {},
   "source": [
    "## Write a regex pattern"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1e90539",
   "metadata": {},
   "source": [
    "Suppose we have a text written in English, except that the names are written in Latin. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f6d17bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# English text with Latin names\n",
    "text = \"Marius was a consul in Rome. Marie is the vocative form.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17a1a67b",
   "metadata": {},
   "source": [
    "We could write a function that captures the different forms of the name Marius."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b63d65a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write a function that captures the pattern for the Latin name Marius\n",
    "def pattern(root):\n",
    "    endings = [\"us\", \"i\", \"o\", \"um\", \"e\"] # the different endings of the name \n",
    "    patterns = [{\"label\": \"PERSON\", \"pattern\": root+ending} for ending in endings]\n",
    "    return patterns\n",
    "marius = pattern(\"Mari\")\n",
    "marius"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed90e6bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an empty English NLP model\n",
    "nlp_latin = spacy.blank(\"en\")\n",
    "\n",
    "# Add an EntityRuler\n",
    "nlp_latin_ruler = nlp_latin.add_pipe(\"entity_ruler\")\n",
    "\n",
    "# add the pattern for the Latin name Marius to the EntityRuler\n",
    "nlp_latin_ruler.add_patterns(marius)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5a5bb29",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a Doc object\n",
    "doc_latin = nlp_latin(text)\n",
    "\n",
    "# Iterate over the entities in Doc object and print them out\n",
    "for ent in doc_latin.ents:\n",
    "    print (ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "541aa75f",
   "metadata": {},
   "source": [
    "We could also use regex to help us write the pattern. If you would like to have a quick refresh of regular expressions, you can refer to the notebook [Regular Expressions](./regular-expressions.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5cd722c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write a function which returns the pattern for Latin name Marius\n",
    "def latin_roots(root):\n",
    "    return [{\"label\": \"PERSON\", \"pattern\": [{\"TEXT\": {\"REGEX\": \"^\" + root + r\"(us|i|o|um|e)$\"}}]}]\n",
    "\n",
    "# Save the pattern to the variable marius2\n",
    "marius2 = latin_roots(\"Mari\")\n",
    "\n",
    "# Create a blank English NLP model\n",
    "nlp_latin2 = spacy.blank(\"en\")\n",
    "\n",
    "# Add an EntityRuler to the model\n",
    "nlp_latin_ruler2 = nlp_latin2.add_pipe(\"entity_ruler\")\n",
    "\n",
    "# Add the pattern for Latin name Marius to the EntityRuler\n",
    "nlp_latin_ruler2.add_patterns(marius2)\n",
    "\n",
    "# Text to be parsed\n",
    "text = \"Marius was a consul in Rome. Marie is the vocative form. Caesar was a dictator.\"\n",
    "\n",
    "# Create a Doc object using the new model with the regex pattern in EntityRuler\n",
    "doc_latin2 = nlp_latin2(text)\n",
    "\n",
    "# Iterate over the entities and print them out\n",
    "for ent in doc_latin2.ents:\n",
    "    print(ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "918a8819",
   "metadata": {},
   "source": [
    "<h2 style=\"color:red; display:inline\">Coding Challenge! &lt; / &gt; </h2>\n",
    "\n",
    "You have seen in coding challenge one that the off-the-shelf NER of Spacy mislabeled some entities. For example, \"Hagrid\", a person's name, is labeled as ORG. Suppose you have a file with all the characters' names in it. Can you make an EntityRuler and add it to the SpaCy pipeline so that all the person names will be labeled 'PERSON'. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27ccd4e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Download the .csv file for this exercise\n",
    "import urllib.request\n",
    "url = 'https://ithaka-labs.s3.amazonaws.com/static-files/images/tdm/tdmdocs/NER_HarryPotter_Characters.csv'\n",
    "urllib.request.urlretrieve(url, './data/' + url.rsplit('/', 1)[-1])   \n",
    "print('Sample file ready.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a2cdee6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read in the data from the character csv file\n",
    "chars_df = pd.read_csv('./data/NER_HarryPotter_Characters.csv', delimiter=';')\n",
    "\n",
    "# Take a look at the first five rows\n",
    "chars_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1c2f5f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get all parts from a character's name\n",
    "chars_df = chars_df[['Name']] \n",
    "chars_df['split_name'] = chars_df['Name'].str.split(' ')\n",
    "chars_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f7c43f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the first names and last names of the characters\n",
    "chars_df['first_name'] = chars_df['split_name'].str[0]\n",
    "chars_df['last_name'] = chars_df['split_name'].str[-1]\n",
    "\n",
    "first_names = chars_df['first_name'].unique().tolist() # Put all unique first names in a list\n",
    "last_names = chars_df['last_name'].unique().tolist() # Put all unique last names in a list\n",
    "\n",
    "names = list(set(first_names) | set(last_names)) # the vertical bar | gives us the union of the two sets\n",
    "\n",
    "names"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "464976ed",
   "metadata": {},
   "source": [
    "Create an EntityRuler. In the ruler, add all characters' names as pattern and specify the label for them as \"PERSON\". Add the ruler as a new pipe. Last, add two new columns to the dataframe you created from the original NER_Harry_Potter_1.csv file, one storing the entities found in each sentence and one storing the labels for the entities. This time, all characters' names should be correctly labeled as \"PERSON\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5e32c76",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "cbd278cd-abb2-4069-9017-518a2aa90922",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Detecting languages in texts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0335be5f-5a9a-4357-aa82-adcec964ed33",
   "metadata": {},
   "source": [
    "When we work with a multilingual corpus, we will first want to know the different languages used in the corpus. There are different approaches to do this. In this section, I will introduce a third-party library Lingua for language detection. Currently, 75 languages are supported by Lingua. Lingua is an open-source project and the github repository for Lingua is here https://github.com/pemistahl/lingua-py.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90a6a840-d056-486b-9ca9-7e907fc6c48f",
   "metadata": {},
   "source": [
    "## Language detection with Lingua"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33e29cf5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install language detector\n",
    "\n",
    "!pip3 install lingua-language-detector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70d3157a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the language detector builder\n",
    "from lingua import LanguageDetectorBuilder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "543be10a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# build a language detector\n",
    "detector = LanguageDetectorBuilder.from_all_languages().build()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41924cbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the detector to detect the language of a string\n",
    "detector.detect_language_of(\"This is an English text\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06d2c77a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the detector to detect the language of a string\n",
    "detector.detect_language_of(\"Este é um outro texto sem idioma especificado\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d39e860",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the detector to detect the language of a string\n",
    "detector.detect_language_of(\"这是一句中文\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "755b4393",
   "metadata": {},
   "source": [
    "Sometimes you may already know the range of languages in your corpus. You just want to identify the language for each document. In this case, you could narrow down the language detector to only a few languages. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a994e3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# build a language detector\n",
    "from lingua import Language, LanguageDetectorBuilder\n",
    "languages = [Language.ENGLISH, Language.FRENCH, Language.GERMAN, Language.SPANISH]\n",
    "detector = LanguageDetectorBuilder.from_languages(*languages).build()\n",
    "\n",
    "# Use the detector to decide between the given languages \n",
    "detector.compute_language_confidence_values(\"This is an English text\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9519ab65-c5bf-4fc4-8560-4623d6a54b03",
   "metadata": {},
   "source": [
    "## Multiple languages in the same file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d50a90f4-3566-4efd-86bb-3e8910017f15",
   "metadata": {},
   "source": [
    "The examples we go over just now assume that only one language is used in each document. However, the language detector we build cannot reliably detect multiple languages, because it will only output one language for a text by default. What if our text has multiple languages, such as the example below?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2eed8aa5-d63a-48ec-9adb-28ab3c1b6aec",
   "metadata": {},
   "outputs": [],
   "source": [
    "# a text string with multiple languages \n",
    "large_text = '''This is a text where the first line is in English.\n",
    "Maar de tweede regel is in het Nederlands. \n",
    "Dies ist ein deutscher Text.'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43054ff6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# build a language detector\n",
    "languages = [Language.ENGLISH, Language.DUTCH, Language.GERMAN]\n",
    "detector = LanguageDetectorBuilder.from_languages(*languages).build()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc53c55f-eba2-4590-84cf-4f3ec7328cf5",
   "metadata": {},
   "source": [
    "If we run the detector over this text, we get the following output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ff11442-cabf-4304-a3db-7520deb7aca6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the detector to decide the language of the text\n",
    "detector.detect_language_of(large_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "239812ea",
   "metadata": {},
   "source": [
    "By default, Lingua returns the most likely language for a given input text. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51c2f380",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the likelihood of the decision\n",
    "confidence_values = detector.compute_language_confidence_values(large_text)\n",
    "for language, value in confidence_values:\n",
    "    print(f\"{language.name}: {value:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "113331fe-bdc1-4c29-a13b-cbc45c42729c",
   "metadata": {},
   "source": [
    "But this text has multiple languages. In this example text, each sentence is written in a different language. Therefore, we need to get each sentence string and run the detector over it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bb39473-3835-4e1d-b77a-446fe6a131bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a Doc object \n",
    "doc = nlp(large_text)\n",
    "\n",
    "# Iterate over each sentence and run the detector over it\n",
    "for sent in doc.sents:\n",
    "    print(f\"Sentence: {sent.text.strip()}\")\n",
    "    print(detector.detect_language_of(sent.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d126fba",
   "metadata": {},
   "source": [
    "# Bring everything together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0bc8cbd0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# A document that has two languages, English and Spanish\n",
    "multilingual_document = \"\"\"This is a story about Margaret who speaks Spanish. \n",
    "'Juan Miguel es mi amigo y tiene veinte años.' Margeret said to her friend Sarah.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25275e7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# build a language detector\n",
    "from lingua import Language, LanguageDetectorBuilder\n",
    "languages = [Language.ENGLISH, Language.FRENCH, Language.GERMAN, Language.SPANISH]\n",
    "detector = LanguageDetectorBuilder.from_languages(*languages).build()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19b96aad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the relevant models\n",
    "english_nlp = spacy.load(\"en_core_web_sm\") # for English\n",
    "spanish_nlp = spacy.load(\"es_core_news_sm\") # for Spanish"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26ecf359",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an NLP model and create a Doc object\n",
    "multi_nlp = spacy.blank('en')\n",
    "\n",
    "# Add sentencizer\n",
    "multi_nlp.add_pipe('sentencizer')\n",
    "\n",
    "# Create a Doc object\n",
    "multi_doc = multi_nlp(multilingual_document.strip())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a55bb64",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Switching between languages with conditionals\n",
    "\n",
    "for sent in multi_doc.sents:\n",
    "    if detector.detect_language_of(sent.text).name == \"ENGLISH\":\n",
    "        nested_doc = english_nlp(sent.text.strip())\n",
    "    elif detector.detect_language_of(sent.text).name == \"SPANISH\":\n",
    "        nested_doc = spanish_nlp(sent.text.strip())\n",
    "    for ent in nested_doc.ents:\n",
    "        print(ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6720817e",
   "metadata": {},
   "source": [
    "___\n",
    "# Lesson Complete\n",
    "Congratulations! You have completed *NER 2*.\n",
    "\n",
    "## Start Next Lesson: [NER 3](./NER-3.ipynb)\n",
    "\n",
    "## Coding Challenge! Solutions\n",
    "\n",
    "There are often many ways to solve programming problems. Here are a few possible ways to solve the challenges, but there are certainly more!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "892e7680",
   "metadata": {},
   "source": [
    "**Exercise 1** In the code cell below, we add two new columns to the dataframe created from NER_Harry_Potter_1.csv, one storing the entities found in each sentence using the off-the-shelf NER in spaCy, one storing the labels for the entities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5b026ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import libraries\n",
    "import spacy\n",
    "import pandas as pd\n",
    "\n",
    "# Read in the data\n",
    "df = pd.read_csv('./data/NER_Harry_Potter_1.csv', delimiter=';')\n",
    "\n",
    "# Load the English model\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "# Make two new columns to store entities and labels\n",
    "df['Entities'] = df['Sentence'].apply(lambda r: [ent.text for ent in nlp(r).ents])\n",
    "df['Labels'] = df['Sentence'].apply(lambda r: [ent.label_ for ent in nlp(r).ents])\n",
    "\n",
    "# Take a look at the updated df\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f83e6046",
   "metadata": {},
   "source": [
    "**Exercise 2** In the code cell below, we create an EntityRuler. We get all the character names and specify their label as \"PERSON\" in the ruler. We add this EntityRuler to the pipeline. Last, we add two new columns to the dataframe created from NER_Harry_Potter_1.csv, one storing the entities found in each sentence using the updated pipeline, one storing the labels for the entities. This time, all character names are correctly labeled as \"PERSON\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5061a795",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read in the data from the character csv file\n",
    "chars_df = pd.read_csv('./data/NER_HarryPotter_Characters.csv', delimiter=';')\n",
    "\n",
    "# Remove the irrelavant columns and only maintain the 'Name' column\n",
    "chars_df = chars_df[['Name']] \n",
    "\n",
    "# Create a new column storing the parts of each name\n",
    "chars_df['split_name'] = chars_df['Name'].str.split(' ')\n",
    "\n",
    "# Get the first names and last names of the characters\n",
    "chars_df['first_name'] = chars_df['split_name'].str[0]\n",
    "chars_df['last_name'] = chars_df['split_name'].str[-1]\n",
    "first_names = chars_df['first_name'].unique().tolist() # Put all unique first names in a list\n",
    "last_names = chars_df['last_name'].unique().tolist() # Put all unique last names in a list\n",
    "\n",
    "# Get all unique names and put them in a list\n",
    "names = list(set(first_names) | set(last_names)) # the vertical bar | gives us the union of the two sets\n",
    "\n",
    "# Load the English model\n",
    "nlp_ex = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "# Create a new EntityRuler and add it as a new pipe\n",
    "ruler = nlp_ex.add_pipe(\"entity_ruler\", before='ner')\n",
    "patterns = [{\"label\": \"PERSON\", \"pattern\": name} for name in names]\n",
    "ruler.add_patterns(patterns)\n",
    "\n",
    "# Read in the data from the character and speech file\n",
    "df = pd.read_csv('./data/NER_Harry_Potter_1.csv', delimiter=';')\n",
    "\n",
    "# Make two new columns to store entities and labels\n",
    "df['Entities'] = df['Sentence'].apply(lambda r: [ent.text for ent in nlp_ex(r).ents])\n",
    "df['Labels'] = df['Sentence'].apply(lambda r: [ent.label_ for ent in nlp_ex(r).ents])\n",
    "\n",
    "# Take a look at the updated df\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9e9345c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}