simple_audio.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/99-NinetyNine/audio_ai_project/blob/main/simple_audio.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fluF3_oOgkWF"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "AJs7HHFmg1M9"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jYysdyb-CaWM"
      },
      "source": [
        "# Simple audio recognition: Recognizing keywords"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CNbqmZy0gbyE"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/tutorials/audio/simple_audio\">\n",
        "    <img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />\n",
        "    View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/audio/simple_audio.ipynb\">\n",
        "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />\n",
        "    Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/tutorials/audio/simple_audio.ipynb\">\n",
        "    <img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />\n",
        "    View source on GitHub</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/tutorials/audio/simple_audio.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SPfDNFlb66XF"
      },
      "source": [
        "This tutorial demonstrates how to preprocess audio files in the WAV format and build and train a basic [automatic speech recognition](https://en.wikipedia.org/wiki/Speech_recognition) (ASR) model for recognizing ten different words. You will use a portion of the [Speech Commands dataset](https://www.tensorflow.org/datasets/catalog/speech_commands) ([Warden, 2018](https://arxiv.org/abs/1804.03209)), which contains short (one-second or less) audio clips of commands, such as \"down\", \"go\", \"left\", \"no\", \"right\", \"stop\", \"up\" and \"yes\".\n",
        "\n",
        "Real-world speech and audio recognition [systems](https://ai.googleblog.com/search/label/Speech%20Recognition) are complex. But, like [image classification with the MNIST dataset](../quickstart/beginner.ipynb), this tutorial should give you a basic understanding of the techniques involved."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Go9C3uLL8Izc"
      },
      "source": [
        "## Setup\n",
        "\n",
        "Import necessary modules and dependencies. You'll be using `tf.keras.utils.audio_dataset_from_directory` (introduced in TensorFlow 2.10), which helps generate audio classification datasets from directories of `.wav` files. You'll also need [seaborn](https://seaborn.pydata.org) for visualization in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hhNW45sjDEDe"
      },
      "outputs": [],
      "source": [
        "!pip install -U -q tensorflow tensorflow_datasets"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dzLKpmZICaWN"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import pathlib\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import seaborn as sns\n",
        "import tensorflow as tf\n",
        "\n",
        "from tensorflow.keras import layers\n",
        "from tensorflow.keras import models\n",
        "from IPython import display\n",
        "\n",
        "# Set the seed value for experiment reproducibility.\n",
        "seed = 42\n",
        "tf.random.set_seed(seed)\n",
        "np.random.seed(seed)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yR0EdgrLCaWR"
      },
      "source": [
        "## Import the mini Speech Commands dataset\n",
        "\n",
        "To save time with data loading, you will be working with a smaller version of the Speech Commands dataset. The [original dataset](https://www.tensorflow.org/datasets/catalog/speech_commands) consists of over 105,000 audio files in the [WAV (Waveform) audio file format](https://www.aelius.com/njh/wavemetatools/doc/riffmci.pdf) of people saying 35 different words. This data was collected by Google and released under a CC BY license.\n",
        "\n",
        "Download and extract the `mini_speech_commands.zip` file containing the smaller Speech Commands datasets with `tf.keras.utils.get_file`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2-rayb7-3Y0I"
      },
      "outputs": [],
      "source": [
        "DATASET_PATH = 'data/mini_speech_commands'\n",
        "\n",
        "data_dir = pathlib.Path(DATASET_PATH)\n",
        "if not data_dir.exists():\n",
        "  tf.keras.utils.get_file(\n",
        "      'mini_speech_commands.zip',\n",
        "      origin=\"http://storage.googleapis.com/download.tensorflow.org/data/mini_speech_commands.zip\",\n",
        "      extract=True,\n",
        "      cache_dir='.', cache_subdir='data')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BgvFq3uYiS5G"
      },
      "source": [
        "The dataset's audio clips are stored in eight folders corresponding to each speech command: `no`, `yes`, `down`, `go`, `left`, `up`, `right`, and `stop`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "70IBxSKxA1N9"
      },
      "outputs": [],
      "source": [
        "commands = np.array(tf.io.gfile.listdir(str(data_dir)))\n",
        "commands = commands[(commands != 'README.md') & (commands != '.DS_Store')]\n",
        "print('Commands:', commands)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TZ7GJjDvHqtt"
      },
      "source": [
        "Divided into directories this way, you can easily load the data using `keras.utils.audio_dataset_from_directory`.\n",
        "\n",
        "The audio clips are 1 second or less at 16kHz. The `output_sequence_length=16000` pads the short ones to exactly 1 second (and would trim longer ones) so that they can be easily batched."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mFM4c3aMC8Qv"
      },
      "outputs": [],
      "source": [
        "train_ds, val_ds = tf.keras.utils.audio_dataset_from_directory(\n",
        "    directory=data_dir,\n",
        "    batch_size=64,\n",
        "    validation_split=0.2,\n",
        "    seed=0,\n",
        "    output_sequence_length=16000,\n",
        "    subset='both')\n",
        "\n",
        "label_names = np.array(train_ds.class_names)\n",
        "print()\n",
        "print(\"label names:\", label_names)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cestp83qFnU5"
      },
      "source": [
        "The dataset now contains batches of audio clips and integer labels. The audio clips have a shape of `(batch, samples, channels)`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3yU6SQGIFb3H"
      },
      "outputs": [],
      "source": [
        "train_ds.element_spec"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ppG9Dgq2Ex8R"
      },
      "source": [
        "This dataset only contains single channel audio, so use the `tf.squeeze` function to drop the extra axis:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Xl-tnniUIBlM"
      },
      "outputs": [],
      "source": [
        "def squeeze(audio, labels):\n",
        "  audio = tf.squeeze(audio, axis=-1)\n",
        "  return audio, labels\n",
        "\n",
        "train_ds = train_ds.map(squeeze, tf.data.AUTOTUNE)\n",
        "val_ds = val_ds.map(squeeze, tf.data.AUTOTUNE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DtsCSWZN5ILv"
      },
      "source": [
        "The `utils.audio_dataset_from_directory` function only returns up to two splits. It's a good idea to keep a test set separate from your validation set.\n",
        "Ideally you'd keep it in a separate directory, but in this case you can use `Dataset.shard` to split the validation set into two halves. Note that iterating over **any** shard will load **all** the data, and only keep its fraction."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "u5UEGsqM5Gss"
      },
      "outputs": [],
      "source": [
        "test_ds = val_ds.shard(num_shards=2, index=0)\n",
        "val_ds = val_ds.shard(num_shards=2, index=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xIeoJcwJH5h9"
      },
      "outputs": [],
      "source": [
        "for example_audio, example_labels in train_ds.take(1):\n",
        "  print(example_audio.shape)\n",
        "  print(example_labels.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "voxGEwvuh2L7"
      },
      "source": [
        "Let's plot a few audio waveforms:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dYtGq2zYNHuT"
      },
      "outputs": [],
      "source": [
        "label_names[[1,1,3,0]]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8yuX6Nqzf6wT"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(16, 10))\n",
        "rows = 3\n",
        "cols = 3\n",
        "n = rows * cols\n",
        "for i in range(n):\n",
        "  plt.subplot(rows, cols, i+1)\n",
        "  audio_signal = example_audio[i]\n",
        "  plt.plot(audio_signal)\n",
        "  plt.title(label_names[example_labels[i]])\n",
        "  plt.yticks(np.arange(-1.2, 1.2, 0.2))\n",
        "  plt.ylim([-1.1, 1.1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EWXPphxm0B4m"
      },
      "source": [
        "## Convert waveforms to spectrograms\n",
        "\n",
        "The waveforms in the dataset are represented in the time domain. Next, you'll transform the waveforms from the time-domain signals into the time-frequency-domain signals by computing the [short-time Fourier transform (STFT)](https://en.wikipedia.org/wiki/Short-time_Fourier_transform) to convert the waveforms to as [spectrograms](https://en.wikipedia.org/wiki/Spectrogram), which show frequency changes over time and can be represented as 2D images. You will feed the spectrogram images into your neural network to train the model.\n",
        "\n",
        "A Fourier transform (`tf.signal.fft`) converts a signal to its component frequencies, but loses all time information. In comparison, STFT (`tf.signal.stft`) splits the signal into windows of time and runs a Fourier transform on each window, preserving some time information, and returning a 2D tensor that you can run standard convolutions on.\n",
        "\n",
        "Create a utility function for converting waveforms to spectrograms:\n",
        "\n",
        "- The waveforms need to be of the same length, so that when you convert them to spectrograms, the results have similar dimensions. This can be done by simply zero-padding the audio clips that are shorter than one second (using `tf.zeros`).\n",
        "- When calling `tf.signal.stft`, choose the `frame_length` and `frame_step` parameters such that the generated spectrogram \"image\" is almost square. For more information on the STFT parameters choice, refer to [this Coursera video](https://www.coursera.org/lecture/audio-signal-processing/stft-2-tjEQe) on audio signal processing and STFT.\n",
        "- The STFT produces an array of complex numbers representing magnitude and phase. However, in this tutorial you'll only use the magnitude, which you can derive by applying `tf.abs` on the output of `tf.signal.stft`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_4CK75DHz_OR"
      },
      "outputs": [],
      "source": [
        "def get_spectrogram(waveform):\n",
        "  # Convert the waveform to a spectrogram via a STFT.\n",
        "  spectrogram = tf.signal.stft(\n",
        "      waveform, frame_length=255, frame_step=128)\n",
        "  # Obtain the magnitude of the STFT.\n",
        "  spectrogram = tf.abs(spectrogram)\n",
        "  # Add a `channels` dimension, so that the spectrogram can be used\n",
        "  # as image-like input data with convolution layers (which expect\n",
        "  # shape (`batch_size`, `height`, `width`, `channels`).\n",
        "  spectrogram = spectrogram[..., tf.newaxis]\n",
        "  return spectrogram"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5rdPiPYJphs2"
      },
      "source": [
        "Next, start exploring the data. Print the shapes of one example's tensorized waveform and the corresponding spectrogram, and play the original audio:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4Mu6Y7Yz3C-V"
      },
      "outputs": [],
      "source": [
        "for i in range(3):\n",
        "  label = label_names[example_labels[i]]\n",
        "  waveform = example_audio[i]\n",
        "  spectrogram = get_spectrogram(waveform)\n",
        "\n",
        "  print('Label:', label)\n",
        "  print('Waveform shape:', waveform.shape)\n",
        "  print('Spectrogram shape:', spectrogram.shape)\n",
        "  print('Audio playback')\n",
        "  display.display(display.Audio(waveform, rate=16000))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xnSuqyxJ1isF"
      },
      "source": [
        "Now, define a function for displaying a spectrogram:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e62jzb36-Jog"
      },
      "outputs": [],
      "source": [
        "def plot_spectrogram(spectrogram, ax):\n",
        "  if len(spectrogram.shape) > 2:\n",
        "    assert len(spectrogram.shape) == 3\n",
        "    spectrogram = np.squeeze(spectrogram, axis=-1)\n",
        "  # Convert the frequencies to log scale and transpose, so that the time is\n",
        "  # represented on the x-axis (columns).\n",
        "  # Add an epsilon to avoid taking a log of zero.\n",
        "  log_spec = np.log(spectrogram.T + np.finfo(float).eps)\n",
        "  height = log_spec.shape[0]\n",
        "  width = log_spec.shape[1]\n",
        "  X = np.linspace(0, np.size(spectrogram), num=width, dtype=int)\n",
        "  Y = range(height)\n",
        "  ax.pcolormesh(X, Y, log_spec)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "baa5c91e8603"
      },
      "source": [
        "Plot the example's waveform over time and the corresponding spectrogram (frequencies over time):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d2_CikgY1tjv"
      },
      "outputs": [],
      "source": [
        "fig, axes = plt.subplots(2, figsize=(12, 8))\n",
        "timescale = np.arange(waveform.shape[0])\n",
        "axes[0].plot(timescale, waveform.numpy())\n",
        "axes[0].set_title('Waveform')\n",
        "axes[0].set_xlim([0, 16000])\n",
        "\n",
        "plot_spectrogram(spectrogram.numpy(), axes[1])\n",
        "axes[1].set_title('Spectrogram')\n",
        "plt.suptitle(label.title())\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GyYXjW07jCHA"
      },
      "source": [
        "Now, create spectrogram datasets from the audio datasets:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mAD0LpkgqtQo"
      },
      "outputs": [],
      "source": [
        "def make_spec_ds(ds):\n",
        "  return ds.map(\n",
        "      map_func=lambda audio,label: (get_spectrogram(audio), label),\n",
        "      num_parallel_calls=tf.data.AUTOTUNE)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yEVb_oK0oBLQ"
      },
      "outputs": [],
      "source": [
        "train_spectrogram_ds = make_spec_ds(train_ds)\n",
        "val_spectrogram_ds = make_spec_ds(val_ds)\n",
        "test_spectrogram_ds = make_spec_ds(test_ds)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6gQpAAgMnyDi"
      },
      "source": [
        "Examine the spectrograms for different examples of the dataset:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EaM2q5aGis-d"
      },
      "outputs": [],
      "source": [
        "for example_spectrograms, example_spect_labels in train_spectrogram_ds.take(1):\n",
        "  break"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QUbHfTuon4iF"
      },
      "outputs": [],
      "source": [
        "rows = 3\n",
        "cols = 3\n",
        "n = rows*cols\n",
        "fig, axes = plt.subplots(rows, cols, figsize=(16, 9))\n",
        "\n",
        "for i in range(n):\n",
        "    r = i // cols\n",
        "    c = i % cols\n",
        "    ax = axes[r][c]\n",
        "    plot_spectrogram(example_spectrograms[i].numpy(), ax)\n",
        "    ax.set_title(label_names[example_spect_labels[i].numpy()])\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z5KdY8IF8rkt"
      },
      "source": [
        "## Build and train the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GS1uIh6F_TN9"
      },
      "source": [
        "Add `Dataset.cache` and `Dataset.prefetch` operations to reduce read latency while training the model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fdZ6M-F5_QzY"
      },
      "outputs": [],
      "source": [
        "train_spectrogram_ds = train_spectrogram_ds.cache().shuffle(10000).prefetch(tf.data.AUTOTUNE)\n",
        "val_spectrogram_ds = val_spectrogram_ds.cache().prefetch(tf.data.AUTOTUNE)\n",
        "test_spectrogram_ds = test_spectrogram_ds.cache().prefetch(tf.data.AUTOTUNE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rwHkKCQQb5oW"
      },
      "source": [
        "For the model, you'll use a simple convolutional neural network (CNN), since you have transformed the audio files into spectrogram images.\n",
        "\n",
        "Your `tf.keras.Sequential` model will use the following Keras preprocessing layers:\n",
        "\n",
        "- `tf.keras.layers.Resizing`: to downsample the input to enable the model to train faster.\n",
        "- `tf.keras.layers.Normalization`: to normalize each pixel in the image based on its mean and standard deviation.\n",
        "\n",
        "For the `Normalization` layer, its `adapt` method would first need to be called on the training data in order to compute aggregate statistics (that is, the mean and the standard deviation)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ALYz7PFCHblP"
      },
      "outputs": [],
      "source": [
        "input_shape = example_spectrograms.shape[1:]\n",
        "print('Input shape:', input_shape)\n",
        "num_labels = len(label_names)\n",
        "\n",
        "# Instantiate the `tf.keras.layers.Normalization` layer.\n",
        "norm_layer = layers.Normalization()\n",
        "# Fit the state of the layer to the spectrograms\n",
        "# with `Normalization.adapt`.\n",
        "norm_layer.adapt(data=train_spectrogram_ds.map(map_func=lambda spec, label: spec))\n",
        "\n",
        "model = models.Sequential([\n",
        "    layers.Input(shape=input_shape),\n",
        "    # Downsample the input.\n",
        "    layers.Resizing(32, 32),\n",
        "    # Normalize.\n",
        "    norm_layer,\n",
        "    layers.Conv2D(32, 3, activation='relu'),\n",
        "    layers.Conv2D(64, 3, activation='relu'),\n",
        "    layers.MaxPooling2D(),\n",
        "    layers.Dropout(0.25),\n",
        "    layers.Flatten(),\n",
        "    layers.Dense(128, activation='relu'),\n",
        "    layers.Dropout(0.5),\n",
        "    layers.Dense(num_labels),\n",
        "])\n",
        "\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "de52e5afa2f3"
      },
      "source": [
        "Configure the Keras model with the Adam optimizer and the cross-entropy loss:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wFjj7-EmsTD-"
      },
      "outputs": [],
      "source": [
        "model.compile(\n",
        "    optimizer=tf.keras.optimizers.Adam(),\n",
        "    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "    metrics=['accuracy'],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f42b9e3a4705"
      },
      "source": [
        "Train the model over 10 epochs for demonstration purposes:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ttioPJVMcGtq"
      },
      "outputs": [],
      "source": [
        "EPOCHS = 10\n",
        "history = model.fit(\n",
        "    train_spectrogram_ds,\n",
        "    validation_data=val_spectrogram_ds,\n",
        "    epochs=EPOCHS,\n",
        "    callbacks=tf.keras.callbacks.EarlyStopping(verbose=1, patience=2),\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gjpCDeQ4mUfS"
      },
      "source": [
        "Let's plot the training and validation loss curves to check how your model has improved during training:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nzhipg3Gu2AY"
      },
      "outputs": [],
      "source": [
        "metrics = history.history\n",
        "plt.figure(figsize=(16,6))\n",
        "plt.subplot(1,2,1)\n",
        "plt.plot(history.epoch, metrics['loss'], metrics['val_loss'])\n",
        "plt.legend(['loss', 'val_loss'])\n",
        "plt.ylim([0, max(plt.ylim())])\n",
        "plt.xlabel('Epoch')\n",
        "plt.ylabel('Loss [CrossEntropy]')\n",
        "\n",
        "plt.subplot(1,2,2)\n",
        "plt.plot(history.epoch, 100*np.array(metrics['accuracy']), 100*np.array(metrics['val_accuracy']))\n",
        "plt.legend(['accuracy', 'val_accuracy'])\n",
        "plt.ylim([0, 100])\n",
        "plt.xlabel('Epoch')\n",
        "plt.ylabel('Accuracy [%]')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5ZTt3kO3mfm4"
      },
      "source": [
        "## Evaluate the model performance\n",
        "\n",
        "Run the model on the test set and check the model's performance:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FapuRT_SsWGQ"
      },
      "outputs": [],
      "source": [
        "model.evaluate(test_spectrogram_ds, return_dict=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "en9Znt1NOabH"
      },
      "source": [
        "### Display a confusion matrix\n",
        "\n",
        "Use a [confusion matrix](https://developers.google.com/machine-learning/glossary#confusion-matrix) to check how well the model did classifying each of the commands in the test set:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5Y6vmWWQuuT1"
      },
      "outputs": [],
      "source": [
        "y_pred = model.predict(test_spectrogram_ds)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d6F0il82u7lW"
      },
      "outputs": [],
      "source": [
        "y_pred = tf.argmax(y_pred, axis=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vHSNoBYLvX81"
      },
      "outputs": [],
      "source": [
        "y_true = tf.concat(list(test_spectrogram_ds.map(lambda s,lab: lab)), axis=0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LvoSAOiXU3lL"
      },
      "outputs": [],
      "source": [
        "confusion_mtx = tf.math.confusion_matrix(y_true, y_pred)\n",
        "plt.figure(figsize=(10, 8))\n",
        "sns.heatmap(confusion_mtx,\n",
        "            xticklabels=label_names,\n",
        "            yticklabels=label_names,\n",
        "            annot=True, fmt='g')\n",
        "plt.xlabel('Prediction')\n",
        "plt.ylabel('Label')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mQGi_mzPcLvl"
      },
      "source": [
        "## Run inference on an audio file\n",
        "\n",
        "Finally, verify the model's prediction output using an input audio file of someone saying \"no\". How well does your model perform?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zRxauKMdhofU"
      },
      "outputs": [],
      "source": [
        "x = data_dir/'no/01bb6a2a_nohash_0.wav'\n",
        "x = tf.io.read_file(str(x))\n",
        "x, sample_rate = tf.audio.decode_wav(x, desired_channels=1, desired_samples=16000,)\n",
        "x = tf.squeeze(x, axis=-1)\n",
        "waveform = x\n",
        "x = get_spectrogram(x)\n",
        "x = x[tf.newaxis,...]\n",
        "\n",
        "prediction = model(x)\n",
        "x_labels = ['no', 'yes', 'down', 'go', 'left', 'up', 'right', 'stop']\n",
        "plt.bar(x_labels, tf.nn.softmax(prediction[0]))\n",
        "plt.title('No')\n",
        "plt.show()\n",
        "\n",
        "display.display(display.Audio(waveform, rate=16000))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VgWICqdqQNaQ"
      },
      "source": [
        "As the output suggests, your model should have recognized the audio command as \"no\"."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h1icqlM3ISW0"
      },
      "source": [
        "## Export the model with preprocessing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r7HX-MjgIbji"
      },
      "source": [
        "The model's not very easy to use if you have to apply those preprocessing steps before passing data to the model for inference. So build an end-to-end version:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2lIeXdWjIbDE"
      },
      "outputs": [],
      "source": [
        "class ExportModel(tf.Module):\n",
        "  def __init__(self, model):\n",
        "    self.model = model\n",
        "\n",
        "    # Accept either a string-filename or a batch of waveforms.\n",
        "    # YOu could add additional signatures for a single wave, or a ragged-batch.\n",
        "    self.__call__.get_concrete_function(\n",
        "        x=tf.TensorSpec(shape=(), dtype=tf.string))\n",
        "    self.__call__.get_concrete_function(\n",
        "       x=tf.TensorSpec(shape=[None, 16000], dtype=tf.float32))\n",
        "\n",
        "\n",
        "  @tf.function\n",
        "  def __call__(self, x):\n",
        "    # If they pass a string, load the file and decode it.\n",
        "    if x.dtype == tf.string:\n",
        "      x = tf.io.read_file(x)\n",
        "      x, _ = tf.audio.decode_wav(x, desired_channels=1, desired_samples=16000,)\n",
        "      x = tf.squeeze(x, axis=-1)\n",
        "      x = x[tf.newaxis, :]\n",
        "\n",
        "    x = get_spectrogram(x)\n",
        "    result = self.model(x, training=False)\n",
        "\n",
        "    class_ids = tf.argmax(result, axis=-1)\n",
        "    class_names = tf.gather(label_names, class_ids)\n",
        "    return {'predictions':result,\n",
        "            'class_ids': class_ids,\n",
        "            'class_names': class_names}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gtZBmUiB9HGY"
      },
      "source": [
        "Test run the \"export\" model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Z1_8TYaCIRue"
      },
      "outputs": [],
      "source": [
        "export = ExportModel(model)\n",
        "export(tf.constant(str(data_dir/'no/01bb6a2a_nohash_0.wav')))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1J6Iuz829Cxo"
      },
      "source": [
        "Save and reload the model, the reloaded model gives identical output:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wTAg4vsn3oEb"
      },
      "outputs": [],
      "source": [
        "tf.saved_model.save(export, \"saved\")\n",
        "imported = tf.saved_model.load(\"saved\")\n",
        "imported(waveform[tf.newaxis, :])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J3jF933m9z1J"
      },
      "source": [
        "## Next steps\n",
        "\n",
        "This tutorial demonstrated how to carry out simple audio classification/automatic speech recognition using a convolutional neural network with TensorFlow and Python. To learn more, consider the following resources:\n",
        "\n",
        "- The [Sound classification with YAMNet](https://www.tensorflow.org/hub/tutorials/yamnet) tutorial shows how to use transfer learning for audio classification.\n",
        "- The notebooks from [Kaggle's TensorFlow speech recognition challenge](https://www.kaggle.com/c/tensorflow-speech-recognition-challenge/overview).\n",
        "- The\n",
        "[TensorFlow.js - Audio recognition using transfer learning codelab](https://codelabs.developers.google.com/codelabs/tensorflowjs-audio-codelab/index.html#0) teaches how to build your own interactive web app for audio classification.\n",
        "- [A tutorial on deep learning for music information retrieval](https://arxiv.org/abs/1709.04396) (Choi et al., 2017) on arXiv.\n",
        "- TensorFlow also has additional support for [audio data preparation and augmentation](https://www.tensorflow.org/io/tutorials/audio) to help with your own audio-based projects.\n",
        "- Consider using the [librosa](https://librosa.org/) library for music and audio analysis."
      ]
    }
  ],
  "metadata": {
    "accelerator": "CPU",
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}