![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jTEzoMx6CasV"
+ },
+ "source": [
+ "*Authors:*\n",
+ "*Jason Adleberg [@pixels2patients](https://twitter.com/pixels2patients), [LinkedIn](https://www.linkedin.com/in/jason-adleberg-6b444b52) and\n",
+ "Nicholas Primiano, [LinkedIn](https://www.linkedin.com/in/nicholas-j-primiano-47077792/)*"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "YHK6DyunSbs4"
+ },
+ "source": [
+ "#NIH CXR8 **Classification** Tutorial\n",
+ "**_Estimated completion time: 30 minutes_**\n",
+ "\n",
+ "Welcome!\n",
+ "In this presentation, we will break down the process into five steps:\n",
+ "\n",
+ "1. **Defining a problem**: what do we want our model to look for? For this we'll need to think of pathology we might find in a chest x-ray.\n",
+ "2. **Data preparation**: we'll talk about how to manipulate our data in a way the classification model can understand.\n",
+ "3. **Training a model**: we'll use a technology called [TensorFlow](\n",
+ "https://wwww.tensorflow.org) to create our model.\n",
+ "4. **Evaluating model performace**: we'll talk about some important considerations when creating a model like this.\n",
+ "5. **Deployment**: we'll send our model to a website where we can then test it in the wild and send to others."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "KyPnBhk8ATaL"
+ },
+ "source": [
+ "##i. Using Google Colab\n",
+ "\n",
+ "This demonstration will take place in Google Colab. Please first sign in with your Google account, and click **File -> Save a copy** in Drive.\n",
+ "\n",
+ "When you get into the draft environment, please ensure that you see \"GPU on\" and \"Internet on\" under Settings so you can utilize the cloud GPU for faster model training.\n",
+ "\n",
+ "In this Notebook editing environment, each block of text is referred to as a cell. Cells containing formatted text are Markdown cells, as they use the Markdown formatting language. Similarly, cells containing code are code cells.\n",
+ "\n",
+ "Clicking within a cell will allow you to edit the content of that cell (a.k.a. enter edit mode). You can also navigate between cells with the arrow keys. Note that the appearance of Markdown cells will change when you enter edit mode.\n",
+ "\n",
+ "You can run code cells (and format Markdown cells) as you go along by clicking within the cell and then clicking the blue button with one arrow next to the cell or at the bottom of the window. You can also use the keyboard shortcut SHIFT + ENTER (press both keys at the same time).\n",
+ "\n",
+ "Let's try this out by running the cell below. This will help us load the technologies we need to power our model."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "sfaVRZ1Z3_3M",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "a3dec1b8-e41d-4963-ea45-b59f8d02a79f"
+ },
+ "source": [
+ "%matplotlib inline\n",
+ "\n",
+ "import sklearn.metrics\n",
+ "import random\n",
+ "import tensorflow as tf\n",
+ "import matplotlib.pyplot as plt\n",
+ "import os\n",
+ "import io\n",
+ "import glob\n",
+ "import scipy.misc\n",
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "from six import BytesIO\n",
+ "from PIL import Image, ImageDraw, ImageFont\n",
+ "import shutil\n",
+ "from tensorflow.keras.applications.inception_v3 import InceptionV3\n",
+ "from tensorflow.keras import layers\n",
+ "from tensorflow.keras import Model\n",
+ "import matplotlib\n",
+ "from tensorflow.keras.optimizers import RMSprop\n",
+ "import os\n",
+ "import zipfile\n",
+ "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
+ "import matplotlib.image as mpimg\n",
+ "from matplotlib.ticker import FormatStrFormatter\n",
+ "from tensorflow.keras.utils import plot_model\n",
+ "\n",
+ "\n",
+ "LEARNING_RATE = 0.0001\n",
+ "repo_url = 'https://github.com/adleberg/medical-ai'\n",
+ "IMAGE_HEIGHT, IMAGE_WIDTH = 299, 299\n",
+ "\n",
+ "def load_image_into_numpy_array(image):\n",
+ " image = image.convert('RGB')\n",
+ " (im_width, im_height) = image.size\n",
+ " return np.array(image.getdata()).reshape(\n",
+ " (im_height, im_width, 3)).astype(np.uint8)\n",
+ "\n",
+ "print(\"Welcome! Downloading some things... this will take a minute.\")\n",
+ "\n",
+ "%cd -q /content\n",
+ "repo_dir_path = os.path.abspath(os.path.join('.', os.path.basename(repo_url)))\n",
+ "!git clone {repo_url} --quiet\n",
+ "%cd -q {repo_dir_path}\n",
+ "!git pull -q\n",
+ "\n",
+ "!apt-get install graphviz -y\n",
+ "!pip install pydot\n",
+ "\n",
+ "print(\"Great! You clicked on it correctly. Now let's get started.\")"
+ ],
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Welcome! Downloading some things... this will take a minute.\n",
+ "Reading package lists... Done\n",
+ "Building dependency tree... Done\n",
+ "Reading state information... Done\n",
+ "graphviz is already the newest version (2.42.2-6ubuntu0.1).\n",
+ "0 upgraded, 0 newly installed, 0 to remove and 49 not upgraded.\n",
+ "Requirement already satisfied: pydot in /usr/local/lib/python3.10/dist-packages (3.0.2)\n",
+ "Requirement already satisfied: pyparsing>=3.0.9 in /usr/local/lib/python3.10/dist-packages (from pydot) (3.2.0)\n",
+ "Great! You clicked on it correctly. Now let's get started.\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Kast6Rs7A4bk"
+ },
+ "source": [
+ "##1. Defining a Problem\n",
+ "Great! Let's start by thinking of something we can look for in a chest xray.\n",
+ "\n",
+ "Once we think of something, let's type it in below, in between the quotes `\"\"`, and run the cell. I'll put in `\"atelectasis\"` by default, but let's see if we can think of something else.\n",
+ "\n",
+ "We'll be using a subset of the [NIH CXR8 dataset](https://openaccess.thecvf.com/content_cvpr_2017/papers/Wang_ChestX-ray8_Hospital-Scale_Chest_CVPR_2017_paper.pdf) for this project, but this principles here apply to any project. The NIH CXR8 dataset only has a few selected findings within its dataset, but let's pretend that we're creating a dataset from scratch (using Montage, e.g.). Of note, this dataset also has bounding boxes available to us for another demonstration on object detection.\n",
+ "\n",
+ "We'll need to see how many examples there are of the finding we typed in. Let's run the cell below to see."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "MQTLUPzBEWj6"
+ },
+ "source": [
+ "\n",
+ "\n",
+ "Above are the labels we have available to us in this dataset."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "E1RIh1j6A9m6"
+ },
+ "source": [
+ "finding = \"cardiomegaly\"\n",
+ "finding = finding.capitalize()"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "df = pd.read_csv(\"/content/medical-ai/labels.csv\")\n",
+ "df.head()"
+ ],
+ "metadata": {
+ "id": "w3DUYGd8nVA9",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 206
+ },
+ "outputId": "bad31812-1f1c-4191-90e8-866de7d8dca9"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ " filename height width label xmin ymin xmax ymax \\\n",
+ "0 00013118_008.jpg 2992 2991 Atelectasis 225.0 547.0 311.0 626.0 \n",
+ "1 00014716_007.jpg 3056 2544 Atelectasis 686.0 131.0 871.0 444.0 \n",
+ "2 00029817_009.jpg 3056 2544 Atelectasis 221.0 317.0 376.0 533.0 \n",
+ "3 00014687_001.jpg 2500 2048 Atelectasis 726.0 494.0 867.0 549.0 \n",
+ "4 00017877_001.jpg 2500 2048 Atelectasis 660.0 569.0 860.0 647.0 \n",
+ "\n",
+ " view \n",
+ "0 PA \n",
+ "1 AP \n",
+ "2 AP \n",
+ "3 AP \n",
+ "4 AP "
+ ],
+ "text/html": [
+ "\n",
+ " | \n", + " | filename | \n", + "height | \n", + "width | \n", + "label | \n", + "xmin | \n", + "ymin | \n", + "xmax | \n", + "ymax | \n", + "view | \n", + "
|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "00013118_008.jpg | \n", + "2992 | \n", + "2991 | \n", + "Atelectasis | \n", + "225.0 | \n", + "547.0 | \n", + "311.0 | \n", + "626.0 | \n", + "PA | \n", + "
| 1 | \n", + "00014716_007.jpg | \n", + "3056 | \n", + "2544 | \n", + "Atelectasis | \n", + "686.0 | \n", + "131.0 | \n", + "871.0 | \n", + "444.0 | \n", + "AP | \n", + "
| 2 | \n", + "00029817_009.jpg | \n", + "3056 | \n", + "2544 | \n", + "Atelectasis | \n", + "221.0 | \n", + "317.0 | \n", + "376.0 | \n", + "533.0 | \n", + "AP | \n", + "
| 3 | \n", + "00014687_001.jpg | \n", + "2500 | \n", + "2048 | \n", + "Atelectasis | \n", + "726.0 | \n", + "494.0 | \n", + "867.0 | \n", + "549.0 | \n", + "AP | \n", + "
| 4 | \n", + "00017877_001.jpg | \n", + "2500 | \n", + "2048 | \n", + "Atelectasis | \n", + "660.0 | \n", + "569.0 | \n", + "860.0 | \n", + "647.0 | \n", + "AP | \n", + "
| \n", + " | filepath | \n", + "filename | \n", + "label | \n", + "guess | \n", + "confidence | \n", + "
|---|---|---|---|---|---|
| 0 | \n", + "/content/medical-ai/images/Cardiomegaly/test/p... | \n", + "00013615_052.jpg | \n", + "pos | \n", + "pos | \n", + "0.984043 | \n", + "
| 1 | \n", + "/content/medical-ai/images/Cardiomegaly/test/p... | \n", + "00030206_013.jpg | \n", + "pos | \n", + "pos | \n", + "0.962622 | \n", + "
| 2 | \n", + "/content/medical-ai/images/Cardiomegaly/test/p... | \n", + "00004342_002.jpg | \n", + "pos | \n", + "pos | \n", + "0.947140 | \n", + "
| 3 | \n", + "/content/medical-ai/images/Cardiomegaly/test/p... | \n", + "00004893_085.jpg | \n", + "pos | \n", + "pos | \n", + "0.940779 | \n", + "
| 4 | \n", + "/content/medical-ai/images/Cardiomegaly/test/p... | \n", + "00011557_003.jpg | \n", + "pos | \n", + "pos | \n", + "0.933133 | \n", + "
| \n", + " | filename | \n", + "label | \n", + "guess | \n", + "confidence | \n", + "
|---|---|---|---|---|
| 0 | \n", + "00013615_052.jpg | \n", + "pos | \n", + "pos | \n", + "0.984043 | \n", + "
| 5 | \n", + "00019187_000.jpg | \n", + "pos | \n", + "pos | \n", + "0.921424 | \n", + "
| 10 | \n", + "00004533_014.jpg | \n", + "pos | \n", + "pos | \n", + "0.877944 | \n", + "
| 15 | \n", + "00016414_000.jpg | \n", + "pos | \n", + "pos | \n", + "0.788136 | \n", + "
| 20 | \n", + "00000087_000.jpg | \n", + "neg | \n", + "pos | \n", + "0.693979 | \n", + "
| 25 | \n", + "00000090_002.jpg | \n", + "neg | \n", + "pos | \n", + "0.603835 | \n", + "
| 30 | \n", + "00000093_002.jpg | \n", + "neg | \n", + "pos | \n", + "0.545232 | \n", + "
| 35 | \n", + "00004344_014.jpg | \n", + "pos | \n", + "neg | \n", + "0.470473 | \n", + "
| 40 | \n", + "00030279_000.jpg | \n", + "pos | \n", + "neg | \n", + "0.370510 | \n", + "
| 45 | \n", + "00000091_006.jpg | \n", + "neg | \n", + "neg | \n", + "0.326562 | \n", + "
| 50 | \n", + "00000073_007.jpg | \n", + "neg | \n", + "neg | \n", + "0.169149 | \n", + "
| 55 | \n", + "00000082_000.jpg | \n", + "neg | \n", + "neg | \n", + "0.119252 | \n", + "