lab_langchain_evaluation.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "1b00dae2-c271-4a69-a579-742e084a9058",
      "metadata": {
        "id": "1b00dae2-c271-4a69-a579-742e084a9058"
      },
      "source": [
        "# Lab | Langchain Evaluation\n",
        "\n",
        "## Intro\n",
        "\n",
        "Pick different sets of data and re-run this notebook. The point is for you to understand all steps involve and the many different ways one can and should evaluate LLM applications.\n",
        "\n",
        "What did you learn? - Let's discuss that in class"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "52824b89-532a-4e54-87e9-1410813cd39e",
      "metadata": {
        "id": "52824b89-532a-4e54-87e9-1410813cd39e"
      },
      "source": [
        "## LangChain: Evaluation\n",
        "\n",
        "### Outline:\n",
        "\n",
        "* Example generation\n",
        "* Manual evaluation (and debuging)\n",
        "* LLM-assisted evaluation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 51,
      "id": "b7ed03ed-1322-49e3-b2a2-33e94fb592ef",
      "metadata": {
        "height": 98,
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "id": "b7ed03ed-1322-49e3-b2a2-33e94fb592ef",
        "outputId": "4e65165b-5186-4fef-8b4b-442ae5e65926"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"\\nfrom dotenv import load_dotenv, find_dotenv\\nimport os\\n_ = load_dotenv(find_dotenv())\\n\\nOPENAI_API_KEY  = os.getenv('OPENAI_API_KEY') \\n\""
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 51
        }
      ],
      "source": [
        "'''\n",
        "from dotenv import load_dotenv, find_dotenv\n",
        "import os\n",
        "_ = load_dotenv(find_dotenv())\n",
        "\n",
        "OPENAI_API_KEY  = os.getenv('OPENAI_API_KEY')\n",
        "'''"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 52,
      "id": "ce74685f-50e9-48cf-8e2c-fac05e8e3ffd",
      "metadata": {
        "id": "ce74685f-50e9-48cf-8e2c-fac05e8e3ffd"
      },
      "outputs": [],
      "source": [
        "from google.colab import userdata\n",
        "OPENAI_API_KEY = userdata.get('OPENAI_API_KEY')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e7e29d2c-ba67-4cba-8ded-375fe040b9ba",
      "metadata": {
        "id": "e7e29d2c-ba67-4cba-8ded-375fe040b9ba"
      },
      "source": [
        "### Example 1"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install --upgrade langchain\n",
        "!pip install langchain-openai\n",
        "!pip install langchain-community langchain-core\n",
        "!pip install langchain_huggingface\n",
        "!pip install sentence-transformers\n",
        "!pip install docarray"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "56kz8Dk5g8L5",
        "outputId": "c906f2c9-ea07-4598-92de-984c430dbb36"
      },
      "id": "56kz8Dk5g8L5",
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: langchain in /usr/local/lib/python3.10/dist-packages (0.2.7)\n",
            "Requirement already satisfied: PyYAML>=5.3 in /usr/local/lib/python3.10/dist-packages (from langchain) (6.0.1)\n",
            "Requirement already satisfied: SQLAlchemy<3,>=1.4 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.0.31)\n",
            "Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /usr/local/lib/python3.10/dist-packages (from langchain) (3.9.5)\n",
            "Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (4.0.3)\n",
            "Requirement already satisfied: langchain-core<0.3.0,>=0.2.12 in /usr/local/lib/python3.10/dist-packages (from langchain) (0.2.12)\n",
            "Requirement already satisfied: langchain-text-splitters<0.3.0,>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (0.2.2)\n",
            "Requirement already satisfied: langsmith<0.2.0,>=0.1.17 in /usr/local/lib/python3.10/dist-packages (from langchain) (0.1.84)\n",
            "Requirement already satisfied: numpy<2,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain) (1.25.2)\n",
            "Requirement already satisfied: pydantic<3,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.8.0)\n",
            "Requirement already satisfied: requests<3,>=2 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.32.3)\n",
            "Requirement already satisfied: tenacity!=8.4.0,<9.0.0,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (8.4.2)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.3.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (23.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.4.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (6.0.5)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.9.4)\n",
            "Requirement already satisfied: jsonpatch<2.0,>=1.33 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.3.0,>=0.2.12->langchain) (1.33)\n",
            "Requirement already satisfied: packaging<25,>=23.2 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.3.0,>=0.2.12->langchain) (24.1)\n",
            "Requirement already satisfied: orjson<4.0.0,>=3.9.14 in /usr/local/lib/python3.10/dist-packages (from langsmith<0.2.0,>=0.1.17->langchain) (3.10.6)\n",
            "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.20.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain) (2.20.0)\n",
            "Requirement already satisfied: typing-extensions>=4.6.1 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain) (4.12.2)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (2024.6.2)\n",
            "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy<3,>=1.4->langchain) (3.0.3)\n",
            "Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.10/dist-packages (from jsonpatch<2.0,>=1.33->langchain-core<0.3.0,>=0.2.12->langchain) (3.0.0)\n",
            "Requirement already satisfied: langchain-openai in /usr/local/lib/python3.10/dist-packages (0.1.14)\n",
            "Requirement already satisfied: langchain-core<0.3,>=0.2.2 in /usr/local/lib/python3.10/dist-packages (from langchain-openai) (0.2.12)\n",
            "Requirement already satisfied: openai<2.0.0,>=1.32.0 in /usr/local/lib/python3.10/dist-packages (from langchain-openai) (1.35.13)\n",
            "Requirement already satisfied: tiktoken<1,>=0.7 in /usr/local/lib/python3.10/dist-packages (from langchain-openai) (0.7.0)\n",
            "Requirement already satisfied: PyYAML>=5.3 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.3,>=0.2.2->langchain-openai) (6.0.1)\n",
            "Requirement already satisfied: jsonpatch<2.0,>=1.33 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.3,>=0.2.2->langchain-openai) (1.33)\n",
            "Requirement already satisfied: langsmith<0.2.0,>=0.1.75 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.3,>=0.2.2->langchain-openai) (0.1.84)\n",
            "Requirement already satisfied: packaging<25,>=23.2 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.3,>=0.2.2->langchain-openai) (24.1)\n",
            "Requirement already satisfied: pydantic<3,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.3,>=0.2.2->langchain-openai) (2.8.0)\n",
            "Requirement already satisfied: tenacity!=8.4.0,<9.0.0,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.3,>=0.2.2->langchain-openai) (8.4.2)\n",
            "Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from openai<2.0.0,>=1.32.0->langchain-openai) (3.7.1)\n",
            "Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai<2.0.0,>=1.32.0->langchain-openai) (1.7.0)\n",
            "Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from openai<2.0.0,>=1.32.0->langchain-openai) (0.27.0)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from openai<2.0.0,>=1.32.0->langchain-openai) (1.3.1)\n",
            "Requirement already satisfied: tqdm>4 in /usr/local/lib/python3.10/dist-packages (from openai<2.0.0,>=1.32.0->langchain-openai) (4.66.4)\n",
            "Requirement already satisfied: typing-extensions<5,>=4.7 in /usr/local/lib/python3.10/dist-packages (from openai<2.0.0,>=1.32.0->langchain-openai) (4.12.2)\n",
            "Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken<1,>=0.7->langchain-openai) (2024.5.15)\n",
            "Requirement already satisfied: requests>=2.26.0 in /usr/local/lib/python3.10/dist-packages (from tiktoken<1,>=0.7->langchain-openai) (2.32.3)\n",
            "Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai<2.0.0,>=1.32.0->langchain-openai) (3.7)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai<2.0.0,>=1.32.0->langchain-openai) (1.2.1)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai<2.0.0,>=1.32.0->langchain-openai) (2024.6.2)\n",
            "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai<2.0.0,>=1.32.0->langchain-openai) (1.0.5)\n",
            "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai<2.0.0,>=1.32.0->langchain-openai) (0.14.0)\n",
            "Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.10/dist-packages (from jsonpatch<2.0,>=1.33->langchain-core<0.3,>=0.2.2->langchain-openai) (3.0.0)\n",
            "Requirement already satisfied: orjson<4.0.0,>=3.9.14 in /usr/local/lib/python3.10/dist-packages (from langsmith<0.2.0,>=0.1.75->langchain-core<0.3,>=0.2.2->langchain-openai) (3.10.6)\n",
            "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain-core<0.3,>=0.2.2->langchain-openai) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.20.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain-core<0.3,>=0.2.2->langchain-openai) (2.20.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken<1,>=0.7->langchain-openai) (3.3.2)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken<1,>=0.7->langchain-openai) (2.0.7)\n",
            "Requirement already satisfied: langchain-community in /usr/local/lib/python3.10/dist-packages (0.2.7)\n",
            "Requirement already satisfied: langchain-core in /usr/local/lib/python3.10/dist-packages (0.2.12)\n",
            "Requirement already satisfied: PyYAML>=5.3 in /usr/local/lib/python3.10/dist-packages (from langchain-community) (6.0.1)\n",
            "Requirement already satisfied: SQLAlchemy<3,>=1.4 in /usr/local/lib/python3.10/dist-packages (from langchain-community) (2.0.31)\n",
            "Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /usr/local/lib/python3.10/dist-packages (from langchain-community) (3.9.5)\n",
            "Requirement already satisfied: dataclasses-json<0.7,>=0.5.7 in /usr/local/lib/python3.10/dist-packages (from langchain-community) (0.6.7)\n",
            "Requirement already satisfied: langchain<0.3.0,>=0.2.7 in /usr/local/lib/python3.10/dist-packages (from langchain-community) (0.2.7)\n",
            "Requirement already satisfied: langsmith<0.2.0,>=0.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain-community) (0.1.84)\n",
            "Requirement already satisfied: numpy<2,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain-community) (1.25.2)\n",
            "Requirement already satisfied: requests<3,>=2 in /usr/local/lib/python3.10/dist-packages (from langchain-community) (2.32.3)\n",
            "Requirement already satisfied: tenacity!=8.4.0,<9.0.0,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain-community) (8.4.2)\n",
            "Requirement already satisfied: jsonpatch<2.0,>=1.33 in /usr/local/lib/python3.10/dist-packages (from langchain-core) (1.33)\n",
            "Requirement already satisfied: packaging<25,>=23.2 in /usr/local/lib/python3.10/dist-packages (from langchain-core) (24.1)\n",
            "Requirement already satisfied: pydantic<3,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain-core) (2.8.0)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (1.3.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (23.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (1.4.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (6.0.5)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (1.9.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (4.0.3)\n",
            "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /usr/local/lib/python3.10/dist-packages (from dataclasses-json<0.7,>=0.5.7->langchain-community) (3.21.3)\n",
            "Requirement already satisfied: typing-inspect<1,>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from dataclasses-json<0.7,>=0.5.7->langchain-community) (0.9.0)\n",
            "Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.10/dist-packages (from jsonpatch<2.0,>=1.33->langchain-core) (3.0.0)\n",
            "Requirement already satisfied: langchain-text-splitters<0.3.0,>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from langchain<0.3.0,>=0.2.7->langchain-community) (0.2.2)\n",
            "Requirement already satisfied: orjson<4.0.0,>=3.9.14 in /usr/local/lib/python3.10/dist-packages (from langsmith<0.2.0,>=0.1.0->langchain-community) (3.10.6)\n",
            "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain-core) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.20.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain-core) (2.20.0)\n",
            "Requirement already satisfied: typing-extensions>=4.6.1 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain-core) (4.12.2)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain-community) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain-community) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain-community) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain-community) (2024.6.2)\n",
            "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy<3,>=1.4->langchain-community) (3.0.3)\n",
            "Requirement already satisfied: mypy-extensions>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain-community) (1.0.0)\n",
            "Requirement already satisfied: langchain_huggingface in /usr/local/lib/python3.10/dist-packages (0.0.3)\n",
            "Requirement already satisfied: huggingface-hub>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from langchain_huggingface) (0.23.4)\n",
            "Requirement already satisfied: langchain-core<0.3,>=0.1.52 in /usr/local/lib/python3.10/dist-packages (from langchain_huggingface) (0.2.12)\n",
            "Requirement already satisfied: sentence-transformers>=2.6.0 in /usr/local/lib/python3.10/dist-packages (from langchain_huggingface) (3.0.1)\n",
            "Requirement already satisfied: tokenizers>=0.19.1 in /usr/local/lib/python3.10/dist-packages (from langchain_huggingface) (0.19.1)\n",
            "Requirement already satisfied: transformers>=4.39.0 in /usr/local/lib/python3.10/dist-packages (from langchain_huggingface) (4.41.2)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.23.0->langchain_huggingface) (3.15.4)\n",
            "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.23.0->langchain_huggingface) (2023.6.0)\n",
            "Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.23.0->langchain_huggingface) (24.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.23.0->langchain_huggingface) (6.0.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.23.0->langchain_huggingface) (2.32.3)\n",
            "Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.23.0->langchain_huggingface) (4.66.4)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.23.0->langchain_huggingface) (4.12.2)\n",
            "Requirement already satisfied: jsonpatch<2.0,>=1.33 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.3,>=0.1.52->langchain_huggingface) (1.33)\n",
            "Requirement already satisfied: langsmith<0.2.0,>=0.1.75 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.3,>=0.1.52->langchain_huggingface) (0.1.84)\n",
            "Requirement already satisfied: pydantic<3,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.3,>=0.1.52->langchain_huggingface) (2.8.0)\n",
            "Requirement already satisfied: tenacity!=8.4.0,<9.0.0,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.3,>=0.1.52->langchain_huggingface) (8.4.2)\n",
            "Requirement already satisfied: torch>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from sentence-transformers>=2.6.0->langchain_huggingface) (2.3.0+cu121)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from sentence-transformers>=2.6.0->langchain_huggingface) (1.25.2)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from sentence-transformers>=2.6.0->langchain_huggingface) (1.2.2)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from sentence-transformers>=2.6.0->langchain_huggingface) (1.11.4)\n",
            "Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from sentence-transformers>=2.6.0->langchain_huggingface) (9.4.0)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.39.0->langchain_huggingface) (2024.5.15)\n",
            "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.39.0->langchain_huggingface) (0.4.3)\n",
            "Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.10/dist-packages (from jsonpatch<2.0,>=1.33->langchain-core<0.3,>=0.1.52->langchain_huggingface) (3.0.0)\n",
            "Requirement already satisfied: orjson<4.0.0,>=3.9.14 in /usr/local/lib/python3.10/dist-packages (from langsmith<0.2.0,>=0.1.75->langchain-core<0.3,>=0.1.52->langchain_huggingface) (3.10.6)\n",
            "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain-core<0.3,>=0.1.52->langchain_huggingface) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.20.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain-core<0.3,>=0.1.52->langchain_huggingface) (2.20.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.23.0->langchain_huggingface) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.23.0->langchain_huggingface) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.23.0->langchain_huggingface) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.23.0->langchain_huggingface) (2024.6.2)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers>=2.6.0->langchain_huggingface) (1.12.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers>=2.6.0->langchain_huggingface) (3.3)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers>=2.6.0->langchain_huggingface) (3.1.4)\n",
            "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers>=2.6.0->langchain_huggingface) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers>=2.6.0->langchain_huggingface) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers>=2.6.0->langchain_huggingface) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers>=2.6.0->langchain_huggingface) (8.9.2.26)\n",
            "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers>=2.6.0->langchain_huggingface) (12.1.3.1)\n",
            "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers>=2.6.0->langchain_huggingface) (11.0.2.54)\n",
            "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers>=2.6.0->langchain_huggingface) (10.3.2.106)\n",
            "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers>=2.6.0->langchain_huggingface) (11.4.5.107)\n",
            "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers>=2.6.0->langchain_huggingface) (12.1.0.106)\n",
            "Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers>=2.6.0->langchain_huggingface) (2.20.5)\n",
            "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers>=2.6.0->langchain_huggingface) (12.1.105)\n",
            "Requirement already satisfied: triton==2.3.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers>=2.6.0->langchain_huggingface) (2.3.0)\n",
            "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.11.0->sentence-transformers>=2.6.0->langchain_huggingface) (12.5.82)\n",
            "Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sentence-transformers>=2.6.0->langchain_huggingface) (1.4.2)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sentence-transformers>=2.6.0->langchain_huggingface) (3.5.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.11.0->sentence-transformers>=2.6.0->langchain_huggingface) (2.1.5)\n",
            "Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.11.0->sentence-transformers>=2.6.0->langchain_huggingface) (1.3.0)\n",
            "Requirement already satisfied: sentence-transformers in /usr/local/lib/python3.10/dist-packages (3.0.1)\n",
            "Requirement already satisfied: transformers<5.0.0,>=4.34.0 in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (4.41.2)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (4.66.4)\n",
            "Requirement already satisfied: torch>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (2.3.0+cu121)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (1.25.2)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (1.2.2)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (1.11.4)\n",
            "Requirement already satisfied: huggingface-hub>=0.15.1 in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (0.23.4)\n",
            "Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (9.4.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence-transformers) (3.15.4)\n",
            "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence-transformers) (2023.6.0)\n",
            "Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence-transformers) (24.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence-transformers) (6.0.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence-transformers) (2.32.3)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence-transformers) (4.12.2)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (1.12.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (3.3)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (3.1.4)\n",
            "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (8.9.2.26)\n",
            "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.3.1)\n",
            "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (11.0.2.54)\n",
            "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (10.3.2.106)\n",
            "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (11.4.5.107)\n",
            "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.0.106)\n",
            "Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (2.20.5)\n",
            "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.105)\n",
            "Requirement already satisfied: triton==2.3.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (2.3.0)\n",
            "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.11.0->sentence-transformers) (12.5.82)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers<5.0.0,>=4.34.0->sentence-transformers) (2024.5.15)\n",
            "Requirement already satisfied: tokenizers<0.20,>=0.19 in /usr/local/lib/python3.10/dist-packages (from transformers<5.0.0,>=4.34.0->sentence-transformers) (0.19.1)\n",
            "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers<5.0.0,>=4.34.0->sentence-transformers) (0.4.3)\n",
            "Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sentence-transformers) (1.4.2)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sentence-transformers) (3.5.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.11.0->sentence-transformers) (2.1.5)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.15.1->sentence-transformers) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.15.1->sentence-transformers) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.15.1->sentence-transformers) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.15.1->sentence-transformers) (2024.6.2)\n",
            "Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.11.0->sentence-transformers) (1.3.0)\n",
            "Requirement already satisfied: docarray in /usr/local/lib/python3.10/dist-packages (0.40.0)\n",
            "Requirement already satisfied: numpy>=1.17.3 in /usr/local/lib/python3.10/dist-packages (from docarray) (1.25.2)\n",
            "Requirement already satisfied: orjson>=3.8.2 in /usr/local/lib/python3.10/dist-packages (from docarray) (3.10.6)\n",
            "Requirement already satisfied: pydantic>=1.10.8 in /usr/local/lib/python3.10/dist-packages (from docarray) (2.8.0)\n",
            "Requirement already satisfied: rich>=13.1.0 in /usr/local/lib/python3.10/dist-packages (from docarray) (13.7.1)\n",
            "Requirement already satisfied: types-requests>=2.28.11.6 in /usr/local/lib/python3.10/dist-packages (from docarray) (2.32.0.20240622)\n",
            "Requirement already satisfied: typing-inspect>=0.8.0 in /usr/local/lib/python3.10/dist-packages (from docarray) (0.9.0)\n",
            "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic>=1.10.8->docarray) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.20.0 in /usr/local/lib/python3.10/dist-packages (from pydantic>=1.10.8->docarray) (2.20.0)\n",
            "Requirement already satisfied: typing-extensions>=4.6.1 in /usr/local/lib/python3.10/dist-packages (from pydantic>=1.10.8->docarray) (4.12.2)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich>=13.1.0->docarray) (3.0.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich>=13.1.0->docarray) (2.16.1)\n",
            "Requirement already satisfied: urllib3>=2 in /usr/local/lib/python3.10/dist-packages (from types-requests>=2.28.11.6->docarray) (2.0.7)\n",
            "Requirement already satisfied: mypy-extensions>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from typing-inspect>=0.8.0->docarray) (1.0.0)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich>=13.1.0->docarray) (0.1.2)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "id": "28008949",
      "metadata": {
        "id": "28008949"
      },
      "source": [
        "#### Create our QandA application"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 54,
      "id": "974acf8e-8f88-42de-88f8-40a82cb58e8b",
      "metadata": {
        "height": 115,
        "tags": [],
        "id": "974acf8e-8f88-42de-88f8-40a82cb58e8b"
      },
      "outputs": [],
      "source": [
        "from langchain.chains import RetrievalQA\n",
        "from langchain_openai import ChatOpenAI\n",
        "from langchain.llms import OpenAI\n",
        "from langchain_huggingface import HuggingFaceEmbeddings\n",
        "from langchain.document_loaders import CSVLoader, TextLoader\n",
        "from langchain.indexes import VectorstoreIndexCreator\n",
        "from langchain.vectorstores import DocArrayInMemorySearch\n",
        "from langchain.chains import LLMChain\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 55,
      "id": "9ec1106d",
      "metadata": {
        "height": 64,
        "tags": [],
        "id": "9ec1106d"
      },
      "outputs": [],
      "source": [
        "file = '/content/OutdoorClothingCatalog_1000.csv'\n",
        "loader = CSVLoader(file_path=file)\n",
        "data = loader.load()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 56,
      "id": "550eb642-c223-4d78-8f92-0f265ef78b86",
      "metadata": {
        "tags": [],
        "id": "550eb642-c223-4d78-8f92-0f265ef78b86"
      },
      "outputs": [],
      "source": [
        "# !pip install --upgrade --force-reinstall sentence-transformers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 57,
      "id": "b31c218f",
      "metadata": {
        "height": 64,
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b31c218f",
        "outputId": "f1837c0d-28cc-43f7-f77b-18d168140563"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n"
          ]
        }
      ],
      "source": [
        "index = VectorstoreIndexCreator(\n",
        "    vectorstore_cls=DocArrayInMemorySearch,\n",
        "    embedding=HuggingFaceEmbeddings(model_name=\"all-MiniLM-L6-v2\", model_kwargs = {'device': 'cpu'})\n",
        ").from_loaders([loader])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 58,
      "id": "a2006054",
      "metadata": {
        "height": 183,
        "tags": [],
        "id": "a2006054"
      },
      "outputs": [],
      "source": [
        "llm = ChatOpenAI(temperature = 0.0, openai_api_key=OPENAI_API_KEY)\n",
        "qa = RetrievalQA.from_chain_type(\n",
        "    llm=llm,\n",
        "    chain_type=\"stuff\",\n",
        "    retriever=index.vectorstore.as_retriever(),\n",
        "    verbose=True,\n",
        "    chain_type_kwargs = {\n",
        "        \"document_separator\": \"<<<<>>>>>\"\n",
        "    }\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "791ebd73",
      "metadata": {
        "id": "791ebd73"
      },
      "source": [
        "#### Coming up with test datapoints"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 59,
      "id": "fb04a0f9",
      "metadata": {
        "height": 30,
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fb04a0f9",
        "outputId": "7bc12b80-3a15-4dbe-f597-6381de84789d"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Document(metadata={'source': '/content/OutdoorClothingCatalog_1000.csv', 'row': 10}, page_content=\": 10\\nname: Cozy Comfort Pullover Set, Stripe\\ndescription: Perfect for lounging, this striped knit set lives up to its name. We used ultrasoft fabric and an easy design that's as comfortable at bedtime as it is when we have to make a quick run out.\\r\\n\\r\\nSize & Fit\\r\\n- Pants are Favorite Fit: Sits lower on the waist.\\r\\n- Relaxed Fit: Our most generous fit sits farthest from the body.\\r\\n\\r\\nFabric & Care\\r\\n- In the softest blend of 63% polyester, 35% rayon and 2% spandex.\\r\\n\\r\\nAdditional Features\\r\\n- Relaxed fit top with raglan sleeves and rounded hem.\\r\\n- Pull-on pants have a wide elastic waistband and drawstring, side pockets and a modern slim leg.\\r\\n\\r\\nImported.\")"
            ]
          },
          "metadata": {},
          "execution_count": 59
        }
      ],
      "source": [
        "data[10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 60,
      "id": "fe4a88c2",
      "metadata": {
        "height": 30,
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fe4a88c2",
        "outputId": "bb119ef3-cd40-4f6e-b8c1-0e17bfd6a8c6"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Document(metadata={'source': '/content/OutdoorClothingCatalog_1000.csv', 'row': 11}, page_content=': 11\\nname: Ultra-Lofty 850 Stretch Down Hooded Jacket\\ndescription: This technical stretch down jacket from our DownTek collection is sure to keep you warm and comfortable with its full-stretch construction providing exceptional range of motion. With a slightly fitted style that falls at the hip and best with a midweight layer, this jacket is suitable for light activity up to 20° and moderate activity up to -30°. The soft and durable 100% polyester shell offers complete windproof protection and is insulated with warm, lofty goose down. Other features include welded baffles for a no-stitch construction and excellent stretch, an adjustable hood, an interior media port and mesh stash pocket and a hem drawcord. Machine wash and dry. Imported.')"
            ]
          },
          "metadata": {},
          "execution_count": 60
        }
      ],
      "source": [
        "data[11]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8d548aef",
      "metadata": {
        "id": "8d548aef"
      },
      "source": [
        "#### Hard-coded examples"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 61,
      "id": "106fbd91-f7bc-4d6b-b090-54b6a485ce39",
      "metadata": {
        "tags": [],
        "id": "106fbd91-f7bc-4d6b-b090-54b6a485ce39"
      },
      "outputs": [],
      "source": [
        "from langchain.prompts import PromptTemplate"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 62,
      "id": "0c5d5af6-36db-4421-b635-46384e677847",
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0c5d5af6-36db-4421-b635-46384e677847",
        "outputId": "4bae9a12-bdc8-4254-ae4e-949cfd6c7d8c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "answer='Yes, the Cozy Comfort Pullover Set is available in multiple colors such as grey, black, and navy.'\n"
          ]
        }
      ],
      "source": [
        "from langchain.prompts import PromptTemplate\n",
        "from langchain.schema import BaseOutputParser\n",
        "from pydantic import BaseModel, Field\n",
        "\n",
        "examples = [\n",
        "    {\n",
        "        \"query\": \"Do the Cozy Comfort Pullover Set\\\n",
        "        have side pockets?\",\n",
        "        \"answer\": \"Yes\"\n",
        "    },\n",
        "    {\n",
        "        \"query\": \"What collection is the Ultra-Lofty \\\n",
        "        850 Stretch Down Hooded Jacket from?\",\n",
        "        \"answer\": \"The DownTek collection\"\n",
        "    }\n",
        "]\n",
        "\n",
        "# Define the prompt template\n",
        "prompt_template = PromptTemplate(\n",
        "    input_variables=[\"query\"],\n",
        "    template=\"Examples:\\n\"\n",
        "             \"1. Query: Do the Cozy Comfort Pullover Set have side pockets?\\n\"\n",
        "             \"   Answer: Yes\\n\"\n",
        "             \"2. Query: What collection is the Ultra-Lofty 850 Stretch Down Hooded Jacket from?\\n\"\n",
        "             \"   Answer: The DownTek collection\\n\"\n",
        "             \"Query: {query}\\n\"\n",
        "             \"Answer:\"\n",
        ")\n",
        "\n",
        "# Define the output model\n",
        "class Answer(BaseModel):\n",
        "    answer: str = Field(description=\"The answer to the query\")\n",
        "\n",
        "# Create the output parser\n",
        "class AnswerOutputParser(BaseOutputParser):\n",
        "    def parse(self, text: str) -> Answer:\n",
        "        # Split the response to get the answer\n",
        "        answer = text.strip().split(\"Answer:\")[-1].strip()\n",
        "        return Answer(answer=answer)\n",
        "\n",
        "# Initialize the LLM\n",
        "# llm = OpenAI()\n",
        "llm = ChatOpenAI(openai_api_key=OPENAI_API_KEY)\n",
        "\n",
        "# Create the LLMChain\n",
        "llm_chain = LLMChain(\n",
        "    llm=llm,\n",
        "    prompt=prompt_template,\n",
        "    output_parser=AnswerOutputParser()\n",
        ")\n",
        "\n",
        "# Example query\n",
        "query = \"Is the Cozy Comfort Pullover Set available in different colors?\"\n",
        "\n",
        "# Run the chain\n",
        "result = llm_chain.run({\"query\": query})\n",
        "\n",
        "# Print the result\n",
        "print(result)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c7ce3e4f",
      "metadata": {
        "id": "c7ce3e4f"
      },
      "source": [
        "#### LLM-Generated examples"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 63,
      "id": "d44f8376",
      "metadata": {
        "height": 64,
        "tags": [],
        "id": "d44f8376"
      },
      "outputs": [],
      "source": [
        "from langchain.evaluation.qa import QAGenerateChain"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 64,
      "id": "34e87816",
      "metadata": {
        "height": 47,
        "tags": [],
        "id": "34e87816"
      },
      "outputs": [],
      "source": [
        "example_gen_chain = QAGenerateChain.from_llm(ChatOpenAI(openai_api_key=OPENAI_API_KEY))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 65,
      "id": "acb34772-368f-4b5e-b4bd-da9b637cc7e8",
      "metadata": {
        "id": "acb34772-368f-4b5e-b4bd-da9b637cc7e8"
      },
      "outputs": [],
      "source": [
        "llm_chain = LLMChain(llm=llm, prompt=prompt_template)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 66,
      "id": "62abae09",
      "metadata": {
        "height": 64,
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "62abae09",
        "outputId": "c646c93d-2268-40b7-de7f-7206a8b47ca7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/langchain/chains/llm.py:369: UserWarning: The apply_and_parse method is deprecated, instead pass an output parser directly to LLMChain.\n",
            "  warnings.warn(\n"
          ]
        }
      ],
      "source": [
        "new_examples = example_gen_chain.apply_and_parse(\n",
        "    [{\"doc\": t} for t in data[:5]]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 67,
      "id": "97ab28b5",
      "metadata": {
        "height": 30,
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "97ab28b5",
        "outputId": "b38f95ff-121f-4f87-ce1b-4ab0bdd7d2ab"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'qa_pairs': {'query': \"What is the approximate weight of the Women's Campside Oxfords per pair?\",\n",
              "  'answer': \"The approximate weight of the Women's Campside Oxfords is 1 lb. 1 oz. per pair.\"}}"
            ]
          },
          "metadata": {},
          "execution_count": 67
        }
      ],
      "source": [
        "new_examples[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 68,
      "id": "0ebe4228",
      "metadata": {
        "height": 30,
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0ebe4228",
        "outputId": "2a5ab728-cfeb-4ac7-dbe3-5c02a812c3ab"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Document(metadata={'source': '/content/OutdoorClothingCatalog_1000.csv', 'row': 0}, page_content=\": 0\\nname: Women's Campside Oxfords\\ndescription: This ultracomfortable lace-to-toe Oxford boasts a super-soft canvas, thick cushioning, and quality construction for a broken-in feel from the first time you put them on. \\r\\n\\r\\nSize & Fit: Order regular shoe size. For half sizes not offered, order up to next whole size. \\r\\n\\r\\nSpecs: Approx. weight: 1 lb.1 oz. per pair. \\r\\n\\r\\nConstruction: Soft canvas material for a broken-in feel and look. Comfortable EVA innersole with Cleansport NXT® antimicrobial odor control. Vintage hunt, fish and camping motif on innersole. Moderate arch contour of innersole. EVA foam midsole for cushioning and support. Chain-tread-inspired molded rubber outsole with modified chain-tread pattern. Imported. \\r\\n\\r\\nQuestions? Please contact us for any inquiries.\")"
            ]
          },
          "metadata": {},
          "execution_count": 68
        }
      ],
      "source": [
        "data[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 69,
      "id": "7693fe86-feeb-4d73-b400-e66e79315274",
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7693fe86-feeb-4d73-b400-e66e79315274",
        "outputId": "6cb60e41-adac-4a13-f376-0dd52b7ff06c"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'query': \"What is the approximate weight of the Women's Campside Oxfords per pair?\",\n",
              "  'answer': \"The approximate weight of the Women's Campside Oxfords is 1 lb. 1 oz. per pair.\"},\n",
              " {'query': 'What are the dimensions of the small and medium sizes for the Recycled Waterhog Dog Mat, Chevron Weave?',\n",
              "  'answer': 'The small size has dimensions of 18\" x 28\", while the medium size has dimensions of 22.5\" x 34.5\".'},\n",
              " {'query': \"What are some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece as described in the document?\",\n",
              "  'answer': \"Some key features of the swimsuit include bright colors, ruffles, and exclusive whimsical prints. The four-way-stretch and chlorine-resistant fabric keeps its shape and resists snags. The UPF 50+ rated fabric provides the highest rated sun protection possible, blocking 98% of the sun's harmful rays. The swimsuit also has crossover no-slip straps and a fully lined bottom for a secure fit and maximum coverage. It is recommended to machine wash and line dry for best results.\"},\n",
              " {'query': 'What is the fabric composition of the Refresh Swimwear, V-Neck Tankini Contrasts top?',\n",
              "  'answer': 'The body of the swimtop is made of 82% recycled nylon and 18% Lycra® spandex, while the lining is made of 90% recycled nylon and 10% Lycra® spandex.'},\n",
              " {'query': 'What is the fabric composition of the EcoFlex 3L Storm Pants?',\n",
              "  'answer': 'The EcoFlex 3L Storm Pants are made of 100% nylon, exclusive of trim.'}]"
            ]
          },
          "metadata": {},
          "execution_count": 69
        }
      ],
      "source": [
        "d_flattened = [data['qa_pairs'] for data in new_examples]\n",
        "d_flattened"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "faf25f2f",
      "metadata": {
        "id": "faf25f2f"
      },
      "source": [
        "#### Combine examples"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 70,
      "id": "ada2a3fc",
      "metadata": {
        "height": 30,
        "tags": [],
        "id": "ada2a3fc"
      },
      "outputs": [],
      "source": [
        "# examples += new_example\n",
        "examples += d_flattened"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 71,
      "id": "2184b9d7-22ab-43a5-9ba5-b27fef024874",
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2184b9d7-22ab-43a5-9ba5-b27fef024874",
        "outputId": "c16a085d-a336-4bc9-a54c-e87244d8bdf1"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'query': 'Do the Cozy Comfort Pullover Set        have side pockets?',\n",
              " 'answer': 'Yes'}"
            ]
          },
          "metadata": {},
          "execution_count": 71
        }
      ],
      "source": [
        "examples[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 72,
      "id": "9cdf5cf5",
      "metadata": {
        "height": 30,
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9cdf5cf5",
        "outputId": "892d5349-581c-4ee7-fd7c-c0a6457afa51"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'query': 'Do the Cozy Comfort Pullover Set        have side pockets?',\n",
              " 'result': 'Yes, the Cozy Comfort Pullover Set has side seam pockets and a back zip pocket, as well as two elastic mesh water bottle pockets.'}"
            ]
          },
          "metadata": {},
          "execution_count": 72
        }
      ],
      "source": [
        "qa.invoke(examples[0][\"query\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "63f3cb08",
      "metadata": {
        "id": "63f3cb08"
      },
      "source": [
        "### Manual Evaluation - Fun part"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 73,
      "id": "fcaf622e",
      "metadata": {
        "height": 47,
        "tags": [],
        "id": "fcaf622e"
      },
      "outputs": [],
      "source": [
        "import langchain\n",
        "langchain.debug = True"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 74,
      "id": "8a142638",
      "metadata": {
        "height": 30,
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8a142638",
        "outputId": "5dd4d336-1662-4112-e0c2-5accce3fd24f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[32;1m\u001b[1;3m[chain/start]\u001b[0m \u001b[1m[chain:RetrievalQA] Entering Chain run with input:\n",
            "\u001b[0m{\n",
            "  \"query\": \"Do the Cozy Comfort Pullover Set        have side pockets?\"\n",
            "}\n",
            "\u001b[32;1m\u001b[1;3m[chain/start]\u001b[0m \u001b[1m[chain:RetrievalQA > chain:StuffDocumentsChain] Entering Chain run with input:\n",
            "\u001b[0m[inputs]\n",
            "\u001b[32;1m\u001b[1;3m[chain/start]\u001b[0m \u001b[1m[chain:RetrievalQA > chain:StuffDocumentsChain > chain:LLMChain] Entering Chain run with input:\n",
            "\u001b[0m{\n",
            "  \"question\": \"Do the Cozy Comfort Pullover Set        have side pockets?\",\n",
            "  \"context\": \"Side seam pockets and back zip pocket, with mesh insert for quick drainage.<<<<>>>>>Two elastic mesh water bottle pockets.\\r\\nTop compartment includes pocket with double-seal zipper for quick access.\\r\\nSide<<<<>>>>>All pockets have sturdy pocket bags and offer plenty of room for a wallet, cell phone and more.\\r\\n\\r\\nGusseted crotch for ease of movement.\\r\\n\\r\\nImported.<<<<>>>>>Two elastic mesh water bottle pockets.\\r\\nTop compartment includes pocket with double-se\"\n",
            "}\n",
            "\u001b[32;1m\u001b[1;3m[llm/start]\u001b[0m \u001b[1m[chain:RetrievalQA > chain:StuffDocumentsChain > chain:LLMChain > llm:ChatOpenAI] Entering LLM run with input:\n",
            "\u001b[0m{\n",
            "  \"prompts\": [\n",
            "    \"System: Use the following pieces of context to answer the user's question. \\nIf you don't know the answer, just say that you don't know, don't try to make up an answer.\\n----------------\\nSide seam pockets and back zip pocket, with mesh insert for quick drainage.<<<<>>>>>Two elastic mesh water bottle pockets.\\r\\nTop compartment includes pocket with double-seal zipper for quick access.\\r\\nSide<<<<>>>>>All pockets have sturdy pocket bags and offer plenty of room for a wallet, cell phone and more.\\r\\n\\r\\nGusseted crotch for ease of movement.\\r\\n\\r\\nImported.<<<<>>>>>Two elastic mesh water bottle pockets.\\r\\nTop compartment includes pocket with double-se\\nHuman: Do the Cozy Comfort Pullover Set        have side pockets?\"\n",
            "  ]\n",
            "}\n",
            "\u001b[36;1m\u001b[1;3m[llm/end]\u001b[0m \u001b[1m[chain:RetrievalQA > chain:StuffDocumentsChain > chain:LLMChain > llm:ChatOpenAI] [634ms] Exiting LLM run with output:\n",
            "\u001b[0m{\n",
            "  \"generations\": [\n",
            "    [\n",
            "      {\n",
            "        \"text\": \"Yes, the Cozy Comfort Pullover Set has side seam pockets and a back zip pocket, as well as two elastic mesh water bottle pockets.\",\n",
            "        \"generation_info\": {\n",
            "          \"finish_reason\": \"stop\",\n",
            "          \"logprobs\": null\n",
            "        },\n",
            "        \"type\": \"ChatGeneration\",\n",
            "        \"message\": {\n",
            "          \"lc\": 1,\n",
            "          \"type\": \"constructor\",\n",
            "          \"id\": [\n",
            "            \"langchain\",\n",
            "            \"schema\",\n",
            "            \"messages\",\n",
            "            \"AIMessage\"\n",
            "          ],\n",
            "          \"kwargs\": {\n",
            "            \"content\": \"Yes, the Cozy Comfort Pullover Set has side seam pockets and a back zip pocket, as well as two elastic mesh water bottle pockets.\",\n",
            "            \"response_metadata\": {\n",
            "              \"token_usage\": {\n",
            "                \"completion_tokens\": 29,\n",
            "                \"prompt_tokens\": 158,\n",
            "                \"total_tokens\": 187\n",
            "              },\n",
            "              \"model_name\": \"gpt-3.5-turbo-0125\",\n",
            "              \"system_fingerprint\": null,\n",
            "              \"finish_reason\": \"stop\",\n",
            "              \"logprobs\": null\n",
            "            },\n",
            "            \"type\": \"ai\",\n",
            "            \"id\": \"run-46cf8755-b322-4364-a327-02b9548c04e8-0\",\n",
            "            \"usage_metadata\": {\n",
            "              \"input_tokens\": 158,\n",
            "              \"output_tokens\": 29,\n",
            "              \"total_tokens\": 187\n",
            "            },\n",
            "            \"tool_calls\": [],\n",
            "            \"invalid_tool_calls\": []\n",
            "          }\n",
            "        }\n",
            "      }\n",
            "    ]\n",
            "  ],\n",
            "  \"llm_output\": {\n",
            "    \"token_usage\": {\n",
            "      \"completion_tokens\": 29,\n",
            "      \"prompt_tokens\": 158,\n",
            "      \"total_tokens\": 187\n",
            "    },\n",
            "    \"model_name\": \"gpt-3.5-turbo-0125\",\n",
            "    \"system_fingerprint\": null\n",
            "  },\n",
            "  \"run\": null\n",
            "}\n",
            "\u001b[36;1m\u001b[1;3m[chain/end]\u001b[0m \u001b[1m[chain:RetrievalQA > chain:StuffDocumentsChain > chain:LLMChain] [637ms] Exiting Chain run with output:\n",
            "\u001b[0m{\n",
            "  \"text\": \"Yes, the Cozy Comfort Pullover Set has side seam pockets and a back zip pocket, as well as two elastic mesh water bottle pockets.\"\n",
            "}\n",
            "\u001b[36;1m\u001b[1;3m[chain/end]\u001b[0m \u001b[1m[chain:RetrievalQA > chain:StuffDocumentsChain] [641ms] Exiting Chain run with output:\n",
            "\u001b[0m{\n",
            "  \"output_text\": \"Yes, the Cozy Comfort Pullover Set has side seam pockets and a back zip pocket, as well as two elastic mesh water bottle pockets.\"\n",
            "}\n",
            "\u001b[36;1m\u001b[1;3m[chain/end]\u001b[0m \u001b[1m[chain:RetrievalQA] [701ms] Exiting Chain run with output:\n",
            "\u001b[0m{\n",
            "  \"result\": \"Yes, the Cozy Comfort Pullover Set has side seam pockets and a back zip pocket, as well as two elastic mesh water bottle pockets.\"\n",
            "}\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'query': 'Do the Cozy Comfort Pullover Set        have side pockets?',\n",
              " 'result': 'Yes, the Cozy Comfort Pullover Set has side seam pockets and a back zip pocket, as well as two elastic mesh water bottle pockets.'}"
            ]
          },
          "metadata": {},
          "execution_count": 74
        }
      ],
      "source": [
        "qa.invoke(examples[0][\"query\"])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 75,
      "id": "b3d6bef0",
      "metadata": {
        "height": 47,
        "tags": [],
        "id": "b3d6bef0"
      },
      "outputs": [],
      "source": [
        "# Turn off the debug mode\n",
        "langchain.debug = False"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d5bdbdce",
      "metadata": {
        "id": "d5bdbdce"
      },
      "source": [
        "### LLM assisted evaluation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 76,
      "id": "a54769b0-3daf-4cac-b259-89a10dd9b5a2",
      "metadata": {
        "tags": [],
        "id": "a54769b0-3daf-4cac-b259-89a10dd9b5a2"
      },
      "outputs": [],
      "source": [
        "examples += d_flattened"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 77,
      "id": "8ea95385-1b4c-440a-9fea-8500b4cc2154",
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8ea95385-1b4c-440a-9fea-8500b4cc2154",
        "outputId": "58142446-2d4e-4bdf-cbbb-ab2bc9f18d56"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'query': 'Do the Cozy Comfort Pullover Set        have side pockets?',\n",
              "  'answer': 'Yes'},\n",
              " {'query': 'What collection is the Ultra-Lofty         850 Stretch Down Hooded Jacket from?',\n",
              "  'answer': 'The DownTek collection'},\n",
              " {'query': \"What is the approximate weight of the Women's Campside Oxfords per pair?\",\n",
              "  'answer': \"The approximate weight of the Women's Campside Oxfords is 1 lb. 1 oz. per pair.\"},\n",
              " {'query': 'What are the dimensions of the small and medium sizes for the Recycled Waterhog Dog Mat, Chevron Weave?',\n",
              "  'answer': 'The small size has dimensions of 18\" x 28\", while the medium size has dimensions of 22.5\" x 34.5\".'},\n",
              " {'query': \"What are some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece as described in the document?\",\n",
              "  'answer': \"Some key features of the swimsuit include bright colors, ruffles, and exclusive whimsical prints. The four-way-stretch and chlorine-resistant fabric keeps its shape and resists snags. The UPF 50+ rated fabric provides the highest rated sun protection possible, blocking 98% of the sun's harmful rays. The swimsuit also has crossover no-slip straps and a fully lined bottom for a secure fit and maximum coverage. It is recommended to machine wash and line dry for best results.\"},\n",
              " {'query': 'What is the fabric composition of the Refresh Swimwear, V-Neck Tankini Contrasts top?',\n",
              "  'answer': 'The body of the swimtop is made of 82% recycled nylon and 18% Lycra® spandex, while the lining is made of 90% recycled nylon and 10% Lycra® spandex.'},\n",
              " {'query': 'What is the fabric composition of the EcoFlex 3L Storm Pants?',\n",
              "  'answer': 'The EcoFlex 3L Storm Pants are made of 100% nylon, exclusive of trim.'},\n",
              " {'query': \"What is the approximate weight of the Women's Campside Oxfords per pair?\",\n",
              "  'answer': \"The approximate weight of the Women's Campside Oxfords is 1 lb. 1 oz. per pair.\"},\n",
              " {'query': 'What are the dimensions of the small and medium sizes for the Recycled Waterhog Dog Mat, Chevron Weave?',\n",
              "  'answer': 'The small size has dimensions of 18\" x 28\", while the medium size has dimensions of 22.5\" x 34.5\".'},\n",
              " {'query': \"What are some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece as described in the document?\",\n",
              "  'answer': \"Some key features of the swimsuit include bright colors, ruffles, and exclusive whimsical prints. The four-way-stretch and chlorine-resistant fabric keeps its shape and resists snags. The UPF 50+ rated fabric provides the highest rated sun protection possible, blocking 98% of the sun's harmful rays. The swimsuit also has crossover no-slip straps and a fully lined bottom for a secure fit and maximum coverage. It is recommended to machine wash and line dry for best results.\"},\n",
              " {'query': 'What is the fabric composition of the Refresh Swimwear, V-Neck Tankini Contrasts top?',\n",
              "  'answer': 'The body of the swimtop is made of 82% recycled nylon and 18% Lycra® spandex, while the lining is made of 90% recycled nylon and 10% Lycra® spandex.'},\n",
              " {'query': 'What is the fabric composition of the EcoFlex 3L Storm Pants?',\n",
              "  'answer': 'The EcoFlex 3L Storm Pants are made of 100% nylon, exclusive of trim.'}]"
            ]
          },
          "metadata": {},
          "execution_count": 77
        }
      ],
      "source": [
        "examples"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 78,
      "id": "a4dca05a",
      "metadata": {
        "height": 30,
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a4dca05a",
        "outputId": "f1e0082c-05fb-422f-ac67-0eae2e932809"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n",
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n",
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n",
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n",
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n",
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n",
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n"
          ]
        }
      ],
      "source": [
        "predictions = qa.batch(examples)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 79,
      "id": "ae7e8b4e-4468-4048-8544-c9936704ea93",
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ae7e8b4e-4468-4048-8544-c9936704ea93",
        "outputId": "3e8c6b81-d79c-46a8-d2bc-72be23431088"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'query': 'Do the Cozy Comfort Pullover Set        have side pockets?',\n",
              "  'answer': 'Yes',\n",
              "  'result': 'Yes, the Cozy Comfort Pullover Set has side seam pockets and a back zip pocket, as well as two elastic mesh water bottle pockets.'},\n",
              " {'query': 'What collection is the Ultra-Lofty         850 Stretch Down Hooded Jacket from?',\n",
              "  'answer': 'The DownTek collection',\n",
              "  'result': 'The Ultra-Lofty 850 Stretch Down Hooded Jacket is from the DownTek collection.'},\n",
              " {'query': \"What is the approximate weight of the Women's Campside Oxfords per pair?\",\n",
              "  'answer': \"The approximate weight of the Women's Campside Oxfords is 1 lb. 1 oz. per pair.\",\n",
              "  'result': \"The approximate weight of the Women's Campside Oxfords per pair is 1 lb. 1 oz.\"},\n",
              " {'query': 'What are the dimensions of the small and medium sizes for the Recycled Waterhog Dog Mat, Chevron Weave?',\n",
              "  'answer': 'The small size has dimensions of 18\" x 28\", while the medium size has dimensions of 22.5\" x 34.5\".',\n",
              "  'result': 'The dimensions for the small size of the Recycled Waterhog Dog Mat, Chevron Weave are 18\" x 28\", and for the medium size, they are 22.5\" x 34.5\".'},\n",
              " {'query': \"What are some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece as described in the document?\",\n",
              "  'answer': \"Some key features of the swimsuit include bright colors, ruffles, and exclusive whimsical prints. The four-way-stretch and chlorine-resistant fabric keeps its shape and resists snags. The UPF 50+ rated fabric provides the highest rated sun protection possible, blocking 98% of the sun's harmful rays. The swimsuit also has crossover no-slip straps and a fully lined bottom for a secure fit and maximum coverage. It is recommended to machine wash and line dry for best results.\",\n",
              "  'result': \"Some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece are:\\n- Bright colors, ruffles, and exclusive whimsical prints\\n- Four-way-stretch and chlorine-resistant fabric\\n- UPF 50+ rated fabric for high sun protection\\n- Crossover no-slip straps and fully lined bottom for a secure fit and coverage\\n- Machine washable and line dry for best results\"},\n",
              " {'query': 'What is the fabric composition of the Refresh Swimwear, V-Neck Tankini Contrasts top?',\n",
              "  'answer': 'The body of the swimtop is made of 82% recycled nylon and 18% Lycra® spandex, while the lining is made of 90% recycled nylon and 10% Lycra® spandex.',\n",
              "  'result': 'The Refresh Swimwear, V-Neck Tankini Contrasts top is made of 82% recycled nylon with 18% Lycra® spandex for the body, and 90% recycled nylon with 10% Lycra® spandex for the lining.'},\n",
              " {'query': 'What is the fabric composition of the EcoFlex 3L Storm Pants?',\n",
              "  'answer': 'The EcoFlex 3L Storm Pants are made of 100% nylon, exclusive of trim.',\n",
              "  'result': 'The fabric composition of the EcoFlex 3L Storm Pants is 100% nylon, exclusive of trim.'},\n",
              " {'query': \"What is the approximate weight of the Women's Campside Oxfords per pair?\",\n",
              "  'answer': \"The approximate weight of the Women's Campside Oxfords is 1 lb. 1 oz. per pair.\",\n",
              "  'result': \"The approximate weight of the Women's Campside Oxfords per pair is 1 lb. 1 oz.\"},\n",
              " {'query': 'What are the dimensions of the small and medium sizes for the Recycled Waterhog Dog Mat, Chevron Weave?',\n",
              "  'answer': 'The small size has dimensions of 18\" x 28\", while the medium size has dimensions of 22.5\" x 34.5\".',\n",
              "  'result': 'The dimensions of the small size for the Recycled Waterhog Dog Mat, Chevron Weave are 18\" x 28\", and the dimensions of the medium size are 22.5\" x 34.5\".'},\n",
              " {'query': \"What are some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece as described in the document?\",\n",
              "  'answer': \"Some key features of the swimsuit include bright colors, ruffles, and exclusive whimsical prints. The four-way-stretch and chlorine-resistant fabric keeps its shape and resists snags. The UPF 50+ rated fabric provides the highest rated sun protection possible, blocking 98% of the sun's harmful rays. The swimsuit also has crossover no-slip straps and a fully lined bottom for a secure fit and maximum coverage. It is recommended to machine wash and line dry for best results.\",\n",
              "  'result': \"Some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece are:\\n- Bright colors, ruffles, and exclusive whimsical prints\\n- Four-way-stretch and chlorine-resistant fabric\\n- UPF 50+ rated fabric for high sun protection\\n- Crossover no-slip straps and fully lined bottom for a secure fit and coverage\\n- Machine washable and line dry for best results\"},\n",
              " {'query': 'What is the fabric composition of the Refresh Swimwear, V-Neck Tankini Contrasts top?',\n",
              "  'answer': 'The body of the swimtop is made of 82% recycled nylon and 18% Lycra® spandex, while the lining is made of 90% recycled nylon and 10% Lycra® spandex.',\n",
              "  'result': 'The fabric composition of the Refresh Swimwear, V-Neck Tankini Contrasts top is as follows:\\n- Body: 82% recycled nylon, 18% Lycra® spandex\\n- Lining: 90% recycled nylon, 10% Lycra® spandex'},\n",
              " {'query': 'What is the fabric composition of the EcoFlex 3L Storm Pants?',\n",
              "  'answer': 'The EcoFlex 3L Storm Pants are made of 100% nylon, exclusive of trim.',\n",
              "  'result': 'The fabric composition of the EcoFlex 3L Storm Pants is 100% nylon, exclusive of trim.'}]"
            ]
          },
          "metadata": {},
          "execution_count": 79
        }
      ],
      "source": [
        "predictions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 80,
      "id": "6012a3e0",
      "metadata": {
        "height": 30,
        "tags": [],
        "id": "6012a3e0"
      },
      "outputs": [],
      "source": [
        "from langchain.evaluation.qa import QAEvalChain"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 81,
      "id": "724b1c0b",
      "metadata": {
        "height": 47,
        "tags": [],
        "id": "724b1c0b"
      },
      "outputs": [],
      "source": [
        "llm = ChatOpenAI(temperature=0, openai_api_key=OPENAI_API_KEY)\n",
        "eval_chain = QAEvalChain.from_llm(llm)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 82,
      "id": "8b46ae55",
      "metadata": {
        "height": 47,
        "tags": [],
        "id": "8b46ae55"
      },
      "outputs": [],
      "source": [
        "graded_outputs = eval_chain.evaluate(examples, predictions)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 83,
      "id": "dc42eb35-c2d7-4581-8004-d315ade63eef",
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dc42eb35-c2d7-4581-8004-d315ade63eef",
        "outputId": "7690094a-731a-448c-806a-eadb7989c61d"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'results': 'CORRECT'},\n",
              " {'results': 'CORRECT'},\n",
              " {'results': 'CORRECT'},\n",
              " {'results': 'CORRECT'},\n",
              " {'results': 'CORRECT'},\n",
              " {'results': 'CORRECT'},\n",
              " {'results': 'CORRECT'},\n",
              " {'results': 'CORRECT'},\n",
              " {'results': 'CORRECT'},\n",
              " {'results': 'CORRECT'},\n",
              " {'results': 'CORRECT'},\n",
              " {'results': 'CORRECT'}]"
            ]
          },
          "metadata": {},
          "execution_count": 83
        }
      ],
      "source": [
        "graded_outputs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 84,
      "id": "3437cfbe",
      "metadata": {
        "height": 149,
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3437cfbe",
        "outputId": "9636bb84-7ee3-4798-a721-41b3188052cf"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Example 0:\n",
            "Question: Do the Cozy Comfort Pullover Set        have side pockets?\n",
            "Real Answer: Yes\n",
            "Predicted Answer: Yes, the Cozy Comfort Pullover Set has side seam pockets and a back zip pocket, as well as two elastic mesh water bottle pockets.\n",
            "\n",
            "Example 1:\n",
            "Question: What collection is the Ultra-Lofty         850 Stretch Down Hooded Jacket from?\n",
            "Real Answer: The DownTek collection\n",
            "Predicted Answer: The Ultra-Lofty 850 Stretch Down Hooded Jacket is from the DownTek collection.\n",
            "\n",
            "Example 2:\n",
            "Question: What is the approximate weight of the Women's Campside Oxfords per pair?\n",
            "Real Answer: The approximate weight of the Women's Campside Oxfords is 1 lb. 1 oz. per pair.\n",
            "Predicted Answer: The approximate weight of the Women's Campside Oxfords per pair is 1 lb. 1 oz.\n",
            "\n",
            "Example 3:\n",
            "Question: What are the dimensions of the small and medium sizes for the Recycled Waterhog Dog Mat, Chevron Weave?\n",
            "Real Answer: The small size has dimensions of 18\" x 28\", while the medium size has dimensions of 22.5\" x 34.5\".\n",
            "Predicted Answer: The dimensions for the small size of the Recycled Waterhog Dog Mat, Chevron Weave are 18\" x 28\", and for the medium size, they are 22.5\" x 34.5\".\n",
            "\n",
            "Example 4:\n",
            "Question: What are some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece as described in the document?\n",
            "Real Answer: Some key features of the swimsuit include bright colors, ruffles, and exclusive whimsical prints. The four-way-stretch and chlorine-resistant fabric keeps its shape and resists snags. The UPF 50+ rated fabric provides the highest rated sun protection possible, blocking 98% of the sun's harmful rays. The swimsuit also has crossover no-slip straps and a fully lined bottom for a secure fit and maximum coverage. It is recommended to machine wash and line dry for best results.\n",
            "Predicted Answer: Some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece are:\n",
            "- Bright colors, ruffles, and exclusive whimsical prints\n",
            "- Four-way-stretch and chlorine-resistant fabric\n",
            "- UPF 50+ rated fabric for high sun protection\n",
            "- Crossover no-slip straps and fully lined bottom for a secure fit and coverage\n",
            "- Machine washable and line dry for best results\n",
            "\n",
            "Example 5:\n",
            "Question: What is the fabric composition of the Refresh Swimwear, V-Neck Tankini Contrasts top?\n",
            "Real Answer: The body of the swimtop is made of 82% recycled nylon and 18% Lycra® spandex, while the lining is made of 90% recycled nylon and 10% Lycra® spandex.\n",
            "Predicted Answer: The Refresh Swimwear, V-Neck Tankini Contrasts top is made of 82% recycled nylon with 18% Lycra® spandex for the body, and 90% recycled nylon with 10% Lycra® spandex for the lining.\n",
            "\n",
            "Example 6:\n",
            "Question: What is the fabric composition of the EcoFlex 3L Storm Pants?\n",
            "Real Answer: The EcoFlex 3L Storm Pants are made of 100% nylon, exclusive of trim.\n",
            "Predicted Answer: The fabric composition of the EcoFlex 3L Storm Pants is 100% nylon, exclusive of trim.\n",
            "\n",
            "Example 7:\n",
            "Question: What is the approximate weight of the Women's Campside Oxfords per pair?\n",
            "Real Answer: The approximate weight of the Women's Campside Oxfords is 1 lb. 1 oz. per pair.\n",
            "Predicted Answer: The approximate weight of the Women's Campside Oxfords per pair is 1 lb. 1 oz.\n",
            "\n",
            "Example 8:\n",
            "Question: What are the dimensions of the small and medium sizes for the Recycled Waterhog Dog Mat, Chevron Weave?\n",
            "Real Answer: The small size has dimensions of 18\" x 28\", while the medium size has dimensions of 22.5\" x 34.5\".\n",
            "Predicted Answer: The dimensions of the small size for the Recycled Waterhog Dog Mat, Chevron Weave are 18\" x 28\", and the dimensions of the medium size are 22.5\" x 34.5\".\n",
            "\n",
            "Example 9:\n",
            "Question: What are some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece as described in the document?\n",
            "Real Answer: Some key features of the swimsuit include bright colors, ruffles, and exclusive whimsical prints. The four-way-stretch and chlorine-resistant fabric keeps its shape and resists snags. The UPF 50+ rated fabric provides the highest rated sun protection possible, blocking 98% of the sun's harmful rays. The swimsuit also has crossover no-slip straps and a fully lined bottom for a secure fit and maximum coverage. It is recommended to machine wash and line dry for best results.\n",
            "Predicted Answer: Some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece are:\n",
            "- Bright colors, ruffles, and exclusive whimsical prints\n",
            "- Four-way-stretch and chlorine-resistant fabric\n",
            "- UPF 50+ rated fabric for high sun protection\n",
            "- Crossover no-slip straps and fully lined bottom for a secure fit and coverage\n",
            "- Machine washable and line dry for best results\n",
            "\n",
            "Example 10:\n",
            "Question: What is the fabric composition of the Refresh Swimwear, V-Neck Tankini Contrasts top?\n",
            "Real Answer: The body of the swimtop is made of 82% recycled nylon and 18% Lycra® spandex, while the lining is made of 90% recycled nylon and 10% Lycra® spandex.\n",
            "Predicted Answer: The fabric composition of the Refresh Swimwear, V-Neck Tankini Contrasts top is as follows:\n",
            "- Body: 82% recycled nylon, 18% Lycra® spandex\n",
            "- Lining: 90% recycled nylon, 10% Lycra® spandex\n",
            "\n",
            "Example 11:\n",
            "Question: What is the fabric composition of the EcoFlex 3L Storm Pants?\n",
            "Real Answer: The EcoFlex 3L Storm Pants are made of 100% nylon, exclusive of trim.\n",
            "Predicted Answer: The fabric composition of the EcoFlex 3L Storm Pants is 100% nylon, exclusive of trim.\n",
            "\n"
          ]
        }
      ],
      "source": [
        "for i, eg in enumerate(examples):\n",
        "    print(f\"Example {i}:\")\n",
        "    print(\"Question: \" + predictions[i]['query'])\n",
        "    print(\"Real Answer: \" + predictions[i]['answer'])\n",
        "    print(\"Predicted Answer: \" + predictions[i]['result'])\n",
        "    # print(\"Predicted Grade: \" + graded_outputs[i]['text'])\n",
        "    print()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "721d127a-a9e3-465d-a8ae-0e2c4b4a2659",
      "metadata": {
        "id": "721d127a-a9e3-465d-a8ae-0e2c4b4a2659"
      },
      "source": [
        "### Example 2\n",
        "One can also easily evaluate your QA chains with the metrics offered in ragas"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install torch torchvision torchaudio"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7brjwPO5kHPq",
        "outputId": "de26da1d-ef3d-418b-e77b-b5ea8f05227a"
      },
      "id": "7brjwPO5kHPq",
      "execution_count": 85,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (2.3.0+cu121)\n",
            "Requirement already satisfied: torchvision in /usr/local/lib/python3.10/dist-packages (0.18.0+cu121)\n",
            "Requirement already satisfied: torchaudio in /usr/local/lib/python3.10/dist-packages (2.3.0+cu121)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch) (3.15.4)\n",
            "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch) (4.12.2)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch) (1.12.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch) (3.3)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch) (3.1.4)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch) (2023.6.0)\n",
            "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch) (8.9.2.26)\n",
            "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch) (12.1.3.1)\n",
            "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch) (11.0.2.54)\n",
            "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch) (10.3.2.106)\n",
            "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch) (11.4.5.107)\n",
            "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch) (12.1.0.106)\n",
            "Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /usr/local/lib/python3.10/dist-packages (from torch) (2.20.5)\n",
            "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch) (12.1.105)\n",
            "Requirement already satisfied: triton==2.3.0 in /usr/local/lib/python3.10/dist-packages (from torch) (2.3.0)\n",
            "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch) (12.5.82)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from torchvision) (1.25.2)\n",
            "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.10/dist-packages (from torchvision) (9.4.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch) (2.1.5)\n",
            "Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch) (1.3.0)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 86,
      "id": "a5ef0493-34ff-4801-b405-69c76ce86c38",
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a5ef0493-34ff-4801-b405-69c76ce86c38",
        "outputId": "b672ad35-7b10-4368-98af-fefa010c1986"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.10/dist-packages/langchain/indexes/vectorstore.py:129: UserWarning: Using InMemoryVectorStore as the default vectorstore.This memory store won't persist data. You should explicitlyspecify a vectorstore when using VectorstoreIndexCreator\n",
            "  warnings.warn(\n"
          ]
        }
      ],
      "source": [
        "from langchain_huggingface import HuggingFaceEmbeddings\n",
        "loader = TextLoader(\"/content/nyc_text.txt\")\n",
        "index = VectorstoreIndexCreator(embedding=HuggingFaceEmbeddings(model_name=\"all-MiniLM-L6-v2\", model_kwargs = {'device': 'cpu'})).from_loaders([loader])\n",
        "\n",
        "\n",
        "llm = ChatOpenAI(temperature= 0, openai_api_key=OPENAI_API_KEY)\n",
        "qa_chain = RetrievalQA.from_chain_type(\n",
        "    llm,\n",
        "    retriever=index.vectorstore.as_retriever(),\n",
        "    return_source_documents=True,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 87,
      "id": "f0449cae-de25-4ef6-ae64-78ccf5e06a5a",
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        },
        "id": "f0449cae-de25-4ef6-ae64-78ccf5e06a5a",
        "outputId": "398290ef-dcc9-4d28-8030-a01406d817a0"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'New York City was originally named New Amsterdam by Dutch colonists in 1626. When the city came under British control in 1664, it was renamed New York after King Charles II of England granted the lands to his brother, the Duke of York. The city has been continuously named New York since November 1674.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 87
        }
      ],
      "source": [
        "# testing it out\n",
        "\n",
        "question = \"How did New York City get its name?\"\n",
        "result = qa_chain.invoke({\"query\": question})\n",
        "result[\"result\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 88,
      "id": "9e846b3d-f79f-46eb-8075-c816268c0500",
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9e846b3d-f79f-46eb-8075-c816268c0500",
        "outputId": "4bd37c3d-4fcd-4a10-a633-9636d19fb430"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'query': 'How did New York City get its name?',\n",
              " 'result': 'New York City was originally named New Amsterdam by Dutch colonists in 1626. When the city came under British control in 1664, it was renamed New York after King Charles II of England granted the lands to his brother, the Duke of York. The city has been continuously named New York since November 1674.',\n",
              " 'source_documents': [Document(id='f1703fa1-92ac-4168-a513-d65c5f4423c4', metadata={'source': '/content/nyc_text.txt'}, page_content='The city and its metropolitan area constitute the premier gateway for legal immigration to the United States. As many as 800 languages are spoken in New York, making it the most linguistically diverse city in the world. New York City is home to more than 3.2 million residents born outside the U.S., the largest foreign-born population of any city in the world as of 2016.New York City traces its origins to a trading post founded on the southern tip of Manhattan Island by Dutch colonists in approximately 1624. The settlement was named New Amsterdam (Dutch: Nieuw Amsterdam) in 1626 and was chartered as a city in 1653. The city came under British control in 1664 and was renamed New York after King Charles II of England granted the lands to his brother, the Duke of York. The city was regained by the Dutch in July 1673 and was renamed New Orange for one year and three months; the city has been continuously named New York since November 1674. New York City was the capital of the United States'),\n",
              "  Document(id='f496931a-6baf-4187-afc6-a9ca4bb0b620', metadata={'source': '/content/nyc_text.txt'}, page_content='New York City has been a metropolitan municipality with a Strong mayor–council form of government since its consolidation in 1898. In New York City, the city government is responsible for public education, correctional institutions, public safety, recreational facilities, sanitation, water supply, and welfare services.'),\n",
              "  Document(id='396e53fa-2934-4020-96d0-ef30721088d7', metadata={'source': '/content/nyc_text.txt'}, page_content=\"Despite New York's heavy reliance on its vast public transit system, streets are a defining feature of the city. The Commissioners' Plan of 1811 greatly influenced the city's physical development. Several of the city's streets and avenues, including Broadway, Wall Street, Madison Avenue, and Seventh Avenue are also used as metonyms for national industries there: the theater, finance, advertising, and fashion organizations, respectively.\"),\n",
              "  Document(id='3c9ec71a-e088-46c2-9568-de89880113b6', metadata={'source': '/content/nyc_text.txt'}, page_content=\"In the pre-Columbian era, the area of present-day New York City was inhabited by Algonquian Native Americans, including the Lenape. Their homeland, known as Lenapehoking, included the present-day areas of Staten Island, Manhattan, the Bronx, the western portion of Long Island (including the areas that would later become the boroughs of Brooklyn and Queens), and the Lower Hudson Valley.The first documented visit into New York Harbor by a European was in 1524 by Italian Giovanni da Verrazzano, an explorer from Florence in the service of the French crown. He claimed the area for France and named it Nouvelle Angoulême (New Angoulême). A Spanish expedition, led by the Portuguese captain Estêvão Gomes sailing for Emperor Charles V, arrived in New York Harbor in January 1525 and charted the mouth of the Hudson River, which he named Río de San Antonio ('Saint Anthony's River'). The Padrón Real of 1527, the first scientific map to show the East Coast of North America continuously, was informed\")]}"
            ]
          },
          "metadata": {},
          "execution_count": 88
        }
      ],
      "source": [
        "result"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "069d9da8-a593-4fc6-9d4b-fea2af6bdfd0",
      "metadata": {
        "id": "069d9da8-a593-4fc6-9d4b-fea2af6bdfd0"
      },
      "source": [
        "Now in order to evaluate the qa system we generated a few relevant questions. We've generated a few question for you but feel free to add any you want."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 89,
      "id": "a2e2cade-0005-41c1-b775-c6a7175bcf3b",
      "metadata": {
        "tags": [],
        "id": "a2e2cade-0005-41c1-b775-c6a7175bcf3b"
      },
      "outputs": [],
      "source": [
        "eval_questions = [\n",
        "    \"What is the population of New York City as of 2020?\",\n",
        "    \"Which borough of New York City has the highest population?\",\n",
        "    \"What is the economic significance of New York City?\",\n",
        "    \"How did New York City get its name?\",\n",
        "    \"What is the significance of the Statue of Liberty in New York City?\",\n",
        "]\n",
        "\n",
        "eval_answers = [\n",
        "    \"8,804,190\",\n",
        "    \"Brooklyn\",\n",
        "    \"New York City's economic significance is vast, as it serves as the global financial capital, housing Wall Street and major financial institutions. Its diverse economy spans technology, media, healthcare, education, and more, making it resilient to economic fluctuations. NYC is a hub for international business, attracting global companies, and boasts a large, skilled labor force. Its real estate market, tourism, cultural industries, and educational institutions further fuel its economic prowess. The city's transportation network and global influence amplify its impact on the world stage, solidifying its status as a vital economic player and cultural epicenter.\",\n",
        "    \"New York City got its name when it came under British control in 1664. King Charles II of England granted the lands to his brother, the Duke of York, who named the city New York in his own honor.\",\n",
        "    \"The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants who arrived in the U.S. by ship in the late 19th and early 20th centuries, representing hope and freedom for those seeking a better life. It has since become an iconic landmark and a global symbol of cultural diversity and freedom.\",\n",
        "]\n",
        "\n",
        "examples = [\n",
        "    {\"query\": q, \"ground_truths\": [eval_answers[i]]}\n",
        "    for i, q in enumerate(eval_questions)\n",
        "]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 90,
      "id": "aac9358e-f8bc-4992-aea3-c83160ff0ab0",
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aac9358e-f8bc-4992-aea3-c83160ff0ab0",
        "outputId": "91203c4c-bacb-45e9-ea4e-607baaf75c7b"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'query': 'What is the population of New York City as of 2020?',\n",
              "  'ground_truths': ['8,804,190']},\n",
              " {'query': 'Which borough of New York City has the highest population?',\n",
              "  'ground_truths': ['Brooklyn']},\n",
              " {'query': 'What is the economic significance of New York City?',\n",
              "  'ground_truths': [\"New York City's economic significance is vast, as it serves as the global financial capital, housing Wall Street and major financial institutions. Its diverse economy spans technology, media, healthcare, education, and more, making it resilient to economic fluctuations. NYC is a hub for international business, attracting global companies, and boasts a large, skilled labor force. Its real estate market, tourism, cultural industries, and educational institutions further fuel its economic prowess. The city's transportation network and global influence amplify its impact on the world stage, solidifying its status as a vital economic player and cultural epicenter.\"]},\n",
              " {'query': 'How did New York City get its name?',\n",
              "  'ground_truths': ['New York City got its name when it came under British control in 1664. King Charles II of England granted the lands to his brother, the Duke of York, who named the city New York in his own honor.']},\n",
              " {'query': 'What is the significance of the Statue of Liberty in New York City?',\n",
              "  'ground_truths': ['The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants who arrived in the U.S. by ship in the late 19th and early 20th centuries, representing hope and freedom for those seeking a better life. It has since become an iconic landmark and a global symbol of cultural diversity and freedom.']}]"
            ]
          },
          "metadata": {},
          "execution_count": 90
        }
      ],
      "source": [
        "examples"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6a21efe8-7c30-449a-9b8e-5b79778e305b",
      "metadata": {
        "id": "6a21efe8-7c30-449a-9b8e-5b79778e305b"
      },
      "source": [
        "#### Introducing RagasEvaluatorChain"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "139c2214-a6eb-4d4f-9403-7e1574b97a36",
      "metadata": {
        "id": "139c2214-a6eb-4d4f-9403-7e1574b97a36"
      },
      "source": [
        "`RagasEvaluatorChain` creates a wrapper around the metrics ragas provides (documented [here](https://github.com/explodinggradients/ragas/blob/main/docs/metrics.md)), making it easier to run these evaluation with langchain and langsmith.\n",
        "\n",
        "The evaluator chain has the following APIs\n",
        "\n",
        "- `__call__()`: call the `RagasEvaluatorChain` directly on the result of a QA chain.\n",
        "- `evaluate()`: evaluate on a list of examples (with the input queries) and predictions (outputs from the QA chain).\n",
        "- `evaluate_run()`: method implemented that is called by langsmith evaluators to evaluate langsmith datasets.\n",
        "\n",
        "lets see each of them in action to learn more."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 91,
      "id": "022c8aae-fe5f-4274-b638-9209151b9491",
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "022c8aae-fe5f-4274-b638-9209151b9491",
        "outputId": "6a0fda04-7db2-4575-c107-38efefb63b9f"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Manhattan (New York County) has the highest population density of any borough in New York City.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 91
        }
      ],
      "source": [
        "result = qa_chain.invoke({\"query\": eval_questions[1]})\n",
        "result[\"result\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 92,
      "id": "eae31c80-42c9-4b1f-95b3-c05ceadb103f",
      "metadata": {
        "tags": [],
        "id": "eae31c80-42c9-4b1f-95b3-c05ceadb103f"
      },
      "outputs": [],
      "source": [
        "key_mapping = {\n",
        "    \"query\": \"question\",\n",
        "    \"result\": \"answer\",\n",
        "    \"source_documents\": \"contexts\"\n",
        "}\n",
        "\n",
        "result_updated = {}\n",
        "for old_key, new_key in key_mapping.items():\n",
        "    if old_key in result:\n",
        "        result_updated[new_key] = result[old_key]\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 93,
      "id": "ecd4dd9f-16d7-43d4-ac8e-6c5aa5e3f7b0",
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ecd4dd9f-16d7-43d4-ac8e-6c5aa5e3f7b0",
        "outputId": "d712fd0f-6b45-4ac4-f1ef-a75f91a269c2"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'question': 'Which borough of New York City has the highest population?',\n",
              " 'answer': 'Manhattan (New York County) has the highest population density of any borough in New York City.',\n",
              " 'contexts': [Document(id='9e00e060-c4df-4f47-bb7f-271ebe2e79f1', metadata={'source': '/content/nyc_text.txt'}, page_content=\"New York City is the most populous city in the United States, with 8,804,190 residents incorporating more immigration into the city than outmigration since the 2010 United States census. More than twice as many people live in New York City as compared to Los Angeles, the second-most populous U.S. city; and New York has more than three times the population of Chicago, the third-most populous U.S. city. New York City gained more residents between 2010 and 2020 (629,000) than any other U.S. city, and a greater amount than the total sum of the gains over the same decade of the next four largest U.S. cities, Los Angeles, Chicago, Houston, and Phoenix, Arizona combined. New York City's population is about 44% of New York State's population, and about 39% of the population of the New York metropolitan area. The majority of New York City residents in 2020 (5,141,538, or 58.4%) were living on Long Island, in Brooklyn, or in Queens. The New York City metropolitan statistical area, has the\"),\n",
              "  Document(id='e555e482-fc16-4b49-a835-0e7fed4836d9', metadata={'source': '/content/nyc_text.txt'}, page_content=\"New York, often called New York City or NYC, is the most populous city in the United States. With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km2), New York City is the most densely populated major city in the United States and more than twice as populous as Los Angeles, the nation's second-largest city. New York City is located at the southern tip of New York State. It constitutes the geographical and demographic center of both the Northeast megalopolis and the New York metropolitan area, the largest metropolitan area in the U.S. by both population and urban area. With over 20.1 million people in its metropolitan statistical area and 23.5 million in its combined statistical area as of 2020, New York is one of the world's most populous megacities, and over 58 million people live within 250 mi (400 km) of the city. New York City is a global cultural, financial, entertainment, and media center with a significant influence on commerce, health care and life\"),\n",
              "  Document(id='5017fdc7-b376-437c-a8a6-56103b8769ca', metadata={'source': '/content/nyc_text.txt'}, page_content=\"Manhattan (New York County) is the geographically smallest and most densely populated borough. It is home to Central Park and most of the city's skyscrapers, and is sometimes locally known as The City. Manhattan's population density of 72,033 people per square mile (27,812/km2) in 2015 makes it the highest of any county in the United States and higher than the density of any individual American city.Manhattan is the cultural, administrative, and financial center of New York City and contains the headquarters of many major multinational corporations, the United Nations headquarters, Wall Street, and a number of important universities. The borough of Manhattan is often described as the financial and cultural center of the world.Most of the borough is situated on Manhattan Island, at the mouth of the Hudson River and the East River, and its southern tip, at the confluence of the two rivers, represents the birthplace of New York City itself. Several small islands also compose part of the\"),\n",
              "  Document(id='850a71e6-db11-40fa-b28d-e5da2f71060c', metadata={'source': '/content/nyc_text.txt'}, page_content=\"=== Population density ===\\n\\nIn 2020, the city had an estimated population density of 29,302.37 inhabitants per square mile (11,313.71/km2), rendering it the nation's most densely populated of all larger municipalities (those with more than 100,000 residents), with several small cities (of fewer than 100,000) in adjacent Hudson County, New Jersey having greater density, as per the 2010 census. Geographically co-extensive with New York County, the borough of Manhattan's 2017 population density of 72,918 inhabitants per square mile (28,154/km2) makes it the highest of any county in the United States and higher than the density of any individual American city. The next three densest counties in the United States, placing second through fourth, are also New York boroughs: Brooklyn, the Bronx, and Queens respectively.\\n\\n\\n=== Race and ethnicity ===\")]}"
            ]
          },
          "metadata": {},
          "execution_count": 93
        }
      ],
      "source": [
        "result_updated"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "result_updated['contexts']"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VSR_9bQ2uS7r",
        "outputId": "f7e71fb5-b234-46a2-ccb9-86ad8a6ee9f7"
      },
      "id": "VSR_9bQ2uS7r",
      "execution_count": 94,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Document(id='9e00e060-c4df-4f47-bb7f-271ebe2e79f1', metadata={'source': '/content/nyc_text.txt'}, page_content=\"New York City is the most populous city in the United States, with 8,804,190 residents incorporating more immigration into the city than outmigration since the 2010 United States census. More than twice as many people live in New York City as compared to Los Angeles, the second-most populous U.S. city; and New York has more than three times the population of Chicago, the third-most populous U.S. city. New York City gained more residents between 2010 and 2020 (629,000) than any other U.S. city, and a greater amount than the total sum of the gains over the same decade of the next four largest U.S. cities, Los Angeles, Chicago, Houston, and Phoenix, Arizona combined. New York City's population is about 44% of New York State's population, and about 39% of the population of the New York metropolitan area. The majority of New York City residents in 2020 (5,141,538, or 58.4%) were living on Long Island, in Brooklyn, or in Queens. The New York City metropolitan statistical area, has the\"),\n",
              " Document(id='e555e482-fc16-4b49-a835-0e7fed4836d9', metadata={'source': '/content/nyc_text.txt'}, page_content=\"New York, often called New York City or NYC, is the most populous city in the United States. With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km2), New York City is the most densely populated major city in the United States and more than twice as populous as Los Angeles, the nation's second-largest city. New York City is located at the southern tip of New York State. It constitutes the geographical and demographic center of both the Northeast megalopolis and the New York metropolitan area, the largest metropolitan area in the U.S. by both population and urban area. With over 20.1 million people in its metropolitan statistical area and 23.5 million in its combined statistical area as of 2020, New York is one of the world's most populous megacities, and over 58 million people live within 250 mi (400 km) of the city. New York City is a global cultural, financial, entertainment, and media center with a significant influence on commerce, health care and life\"),\n",
              " Document(id='5017fdc7-b376-437c-a8a6-56103b8769ca', metadata={'source': '/content/nyc_text.txt'}, page_content=\"Manhattan (New York County) is the geographically smallest and most densely populated borough. It is home to Central Park and most of the city's skyscrapers, and is sometimes locally known as The City. Manhattan's population density of 72,033 people per square mile (27,812/km2) in 2015 makes it the highest of any county in the United States and higher than the density of any individual American city.Manhattan is the cultural, administrative, and financial center of New York City and contains the headquarters of many major multinational corporations, the United Nations headquarters, Wall Street, and a number of important universities. The borough of Manhattan is often described as the financial and cultural center of the world.Most of the borough is situated on Manhattan Island, at the mouth of the Hudson River and the East River, and its southern tip, at the confluence of the two rivers, represents the birthplace of New York City itself. Several small islands also compose part of the\"),\n",
              " Document(id='850a71e6-db11-40fa-b28d-e5da2f71060c', metadata={'source': '/content/nyc_text.txt'}, page_content=\"=== Population density ===\\n\\nIn 2020, the city had an estimated population density of 29,302.37 inhabitants per square mile (11,313.71/km2), rendering it the nation's most densely populated of all larger municipalities (those with more than 100,000 residents), with several small cities (of fewer than 100,000) in adjacent Hudson County, New Jersey having greater density, as per the 2010 census. Geographically co-extensive with New York County, the borough of Manhattan's 2017 population density of 72,918 inhabitants per square mile (28,154/km2) makes it the highest of any county in the United States and higher than the density of any individual American city. The next three densest counties in the United States, placing second through fourth, are also New York boroughs: Brooklyn, the Bronx, and Queens respectively.\\n\\n\\n=== Race and ethnicity ===\")]"
            ]
          },
          "metadata": {},
          "execution_count": 94
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "contexts_str_list = []\n",
        "\n",
        "# Iterate through each Document in result_updated['contexts']\n",
        "for document in result_updated['contexts']:\n",
        "    # Append the page_content of each Document to contexts_str_list\n",
        "    contexts_str_list.append(document.page_content)\n",
        "\n",
        "# Join the list of strings with newline character\n",
        "contexts_str = \"\\n\".join(contexts_str_list)\n",
        "\n",
        "# Assign the concatenated string back to result_updated['contexts']\n",
        "result_updated['contexts'] = contexts_str"
      ],
      "metadata": {
        "id": "wpsnOUiluVJs"
      },
      "id": "wpsnOUiluVJs",
      "execution_count": 95,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "result_updated"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GQuldD6DuXXi",
        "outputId": "7a8b8758-b907-41a1-8dbd-5130c81f3fb3"
      },
      "id": "GQuldD6DuXXi",
      "execution_count": 96,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'question': 'Which borough of New York City has the highest population?',\n",
              " 'answer': 'Manhattan (New York County) has the highest population density of any borough in New York City.',\n",
              " 'contexts': \"New York City is the most populous city in the United States, with 8,804,190 residents incorporating more immigration into the city than outmigration since the 2010 United States census. More than twice as many people live in New York City as compared to Los Angeles, the second-most populous U.S. city; and New York has more than three times the population of Chicago, the third-most populous U.S. city. New York City gained more residents between 2010 and 2020 (629,000) than any other U.S. city, and a greater amount than the total sum of the gains over the same decade of the next four largest U.S. cities, Los Angeles, Chicago, Houston, and Phoenix, Arizona combined. New York City's population is about 44% of New York State's population, and about 39% of the population of the New York metropolitan area. The majority of New York City residents in 2020 (5,141,538, or 58.4%) were living on Long Island, in Brooklyn, or in Queens. The New York City metropolitan statistical area, has the\\nNew York, often called New York City or NYC, is the most populous city in the United States. With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km2), New York City is the most densely populated major city in the United States and more than twice as populous as Los Angeles, the nation's second-largest city. New York City is located at the southern tip of New York State. It constitutes the geographical and demographic center of both the Northeast megalopolis and the New York metropolitan area, the largest metropolitan area in the U.S. by both population and urban area. With over 20.1 million people in its metropolitan statistical area and 23.5 million in its combined statistical area as of 2020, New York is one of the world's most populous megacities, and over 58 million people live within 250 mi (400 km) of the city. New York City is a global cultural, financial, entertainment, and media center with a significant influence on commerce, health care and life\\nManhattan (New York County) is the geographically smallest and most densely populated borough. It is home to Central Park and most of the city's skyscrapers, and is sometimes locally known as The City. Manhattan's population density of 72,033 people per square mile (27,812/km2) in 2015 makes it the highest of any county in the United States and higher than the density of any individual American city.Manhattan is the cultural, administrative, and financial center of New York City and contains the headquarters of many major multinational corporations, the United Nations headquarters, Wall Street, and a number of important universities. The borough of Manhattan is often described as the financial and cultural center of the world.Most of the borough is situated on Manhattan Island, at the mouth of the Hudson River and the East River, and its southern tip, at the confluence of the two rivers, represents the birthplace of New York City itself. Several small islands also compose part of the\\n=== Population density ===\\n\\nIn 2020, the city had an estimated population density of 29,302.37 inhabitants per square mile (11,313.71/km2), rendering it the nation's most densely populated of all larger municipalities (those with more than 100,000 residents), with several small cities (of fewer than 100,000) in adjacent Hudson County, New Jersey having greater density, as per the 2010 census. Geographically co-extensive with New York County, the borough of Manhattan's 2017 population density of 72,918 inhabitants per square mile (28,154/km2) makes it the highest of any county in the United States and higher than the density of any individual American city. The next three densest counties in the United States, placing second through fourth, are also New York boroughs: Brooklyn, the Bronx, and Queens respectively.\\n\\n\\n=== Race and ethnicity ===\"}"
            ]
          },
          "metadata": {},
          "execution_count": 96
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 97,
      "id": "00d5cc30-d25e-41bd-8695-9246b73938bc",
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "00d5cc30-d25e-41bd-8695-9246b73938bc",
        "outputId": "210975ee-d97f-4e80-a85a-1e71489cd0f6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: recordclass in /usr/local/lib/python3.10/dist-packages (0.22.0.2)\n"
          ]
        }
      ],
      "source": [
        "!pip install --no-cache-dir recordclass"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 98,
      "id": "775d55d9-437d-40c4-bb5f-87c7b65fa567",
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "775d55d9-437d-40c4-bb5f-87c7b65fa567",
        "outputId": "52e522ca-eec2-44d5-9051-6789b8c6eb9f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: ragas==0.1.9 in /usr/local/lib/python3.10/dist-packages (0.1.9)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from ragas==0.1.9) (1.25.2)\n",
            "Requirement already satisfied: datasets in /usr/local/lib/python3.10/dist-packages (from ragas==0.1.9) (2.20.0)\n",
            "Requirement already satisfied: tiktoken in /usr/local/lib/python3.10/dist-packages (from ragas==0.1.9) (0.7.0)\n",
            "Requirement already satisfied: langchain in /usr/local/lib/python3.10/dist-packages (from ragas==0.1.9) (0.2.7)\n",
            "Requirement already satisfied: langchain-core in /usr/local/lib/python3.10/dist-packages (from ragas==0.1.9) (0.2.12)\n",
            "Requirement already satisfied: langchain-community in /usr/local/lib/python3.10/dist-packages (from ragas==0.1.9) (0.2.7)\n",
            "Requirement already satisfied: langchain-openai in /usr/local/lib/python3.10/dist-packages (from ragas==0.1.9) (0.1.14)\n",
            "Requirement already satisfied: openai>1 in /usr/local/lib/python3.10/dist-packages (from ragas==0.1.9) (1.35.13)\n",
            "Requirement already satisfied: pysbd>=0.3.4 in /usr/local/lib/python3.10/dist-packages (from ragas==0.1.9) (0.3.4)\n",
            "Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.10/dist-packages (from ragas==0.1.9) (1.6.0)\n",
            "Requirement already satisfied: appdirs in /usr/local/lib/python3.10/dist-packages (from ragas==0.1.9) (1.4.4)\n",
            "Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from openai>1->ragas==0.1.9) (3.7.1)\n",
            "Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai>1->ragas==0.1.9) (1.7.0)\n",
            "Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from openai>1->ragas==0.1.9) (0.27.0)\n",
            "Requirement already satisfied: pydantic<3,>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from openai>1->ragas==0.1.9) (2.8.0)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from openai>1->ragas==0.1.9) (1.3.1)\n",
            "Requirement already satisfied: tqdm>4 in /usr/local/lib/python3.10/dist-packages (from openai>1->ragas==0.1.9) (4.66.4)\n",
            "Requirement already satisfied: typing-extensions<5,>=4.7 in /usr/local/lib/python3.10/dist-packages (from openai>1->ragas==0.1.9) (4.12.2)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from datasets->ragas==0.1.9) (3.15.4)\n",
            "Requirement already satisfied: pyarrow>=15.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets->ragas==0.1.9) (16.1.0)\n",
            "Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets->ragas==0.1.9) (0.6)\n",
            "Requirement already satisfied: dill<0.3.9,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from datasets->ragas==0.1.9) (0.3.8)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets->ragas==0.1.9) (2.0.3)\n",
            "Requirement already satisfied: requests>=2.32.2 in /usr/local/lib/python3.10/dist-packages (from datasets->ragas==0.1.9) (2.32.3)\n",
            "Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets->ragas==0.1.9) (3.4.1)\n",
            "Requirement already satisfied: multiprocess in /usr/local/lib/python3.10/dist-packages (from datasets->ragas==0.1.9) (0.70.16)\n",
            "Requirement already satisfied: fsspec[http]<=2024.5.0,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from datasets->ragas==0.1.9) (2023.6.0)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets->ragas==0.1.9) (3.9.5)\n",
            "Requirement already satisfied: huggingface-hub>=0.21.2 in /usr/local/lib/python3.10/dist-packages (from datasets->ragas==0.1.9) (0.23.4)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from datasets->ragas==0.1.9) (24.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from datasets->ragas==0.1.9) (6.0.1)\n",
            "Requirement already satisfied: SQLAlchemy<3,>=1.4 in /usr/local/lib/python3.10/dist-packages (from langchain->ragas==0.1.9) (2.0.31)\n",
            "Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from langchain->ragas==0.1.9) (4.0.3)\n",
            "Requirement already satisfied: langchain-text-splitters<0.3.0,>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from langchain->ragas==0.1.9) (0.2.2)\n",
            "Requirement already satisfied: langsmith<0.2.0,>=0.1.17 in /usr/local/lib/python3.10/dist-packages (from langchain->ragas==0.1.9) (0.1.84)\n",
            "Requirement already satisfied: tenacity!=8.4.0,<9.0.0,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain->ragas==0.1.9) (8.4.2)\n",
            "Requirement already satisfied: jsonpatch<2.0,>=1.33 in /usr/local/lib/python3.10/dist-packages (from langchain-core->ragas==0.1.9) (1.33)\n",
            "Requirement already satisfied: dataclasses-json<0.7,>=0.5.7 in /usr/local/lib/python3.10/dist-packages (from langchain-community->ragas==0.1.9) (0.6.7)\n",
            "Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken->ragas==0.1.9) (2024.5.15)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->ragas==0.1.9) (1.3.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->ragas==0.1.9) (23.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->ragas==0.1.9) (1.4.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->ragas==0.1.9) (6.0.5)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->ragas==0.1.9) (1.9.4)\n",
            "Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai>1->ragas==0.1.9) (3.7)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai>1->ragas==0.1.9) (1.2.1)\n",
            "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /usr/local/lib/python3.10/dist-packages (from dataclasses-json<0.7,>=0.5.7->langchain-community->ragas==0.1.9) (3.21.3)\n",
            "Requirement already satisfied: typing-inspect<1,>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from dataclasses-json<0.7,>=0.5.7->langchain-community->ragas==0.1.9) (0.9.0)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai>1->ragas==0.1.9) (2024.6.2)\n",
            "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai>1->ragas==0.1.9) (1.0.5)\n",
            "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai>1->ragas==0.1.9) (0.14.0)\n",
            "Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.10/dist-packages (from jsonpatch<2.0,>=1.33->langchain-core->ragas==0.1.9) (3.0.0)\n",
            "Requirement already satisfied: orjson<4.0.0,>=3.9.14 in /usr/local/lib/python3.10/dist-packages (from langsmith<0.2.0,>=0.1.17->langchain->ragas==0.1.9) (3.10.6)\n",
            "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1.9.0->openai>1->ragas==0.1.9) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.20.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1.9.0->openai>1->ragas==0.1.9) (2.20.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets->ragas==0.1.9) (3.3.2)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets->ragas==0.1.9) (2.0.7)\n",
            "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy<3,>=1.4->langchain->ragas==0.1.9) (3.0.3)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->ragas==0.1.9) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->ragas==0.1.9) (2023.4)\n",
            "Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->ragas==0.1.9) (2024.1)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->datasets->ragas==0.1.9) (1.16.0)\n",
            "Requirement already satisfied: mypy-extensions>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain-community->ragas==0.1.9) (1.0.0)\n"
          ]
        }
      ],
      "source": [
        "!pip install ragas==0.1.9"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install nest_asyncio"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "v_D3PCGnomGH",
        "outputId": "9d2ac345-49c1-4a86-8683-5f621fff8ddb"
      },
      "id": "v_D3PCGnomGH",
      "execution_count": 99,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: nest_asyncio in /usr/local/lib/python3.10/dist-packages (1.6.0)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import nest_asyncio\n",
        "import asyncio\n",
        "\n",
        "nest_asyncio.apply()"
      ],
      "metadata": {
        "id": "gdbFUqLYugyy"
      },
      "id": "gdbFUqLYugyy",
      "execution_count": 100,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 101,
      "id": "29b14c07-bf6c-4e86-ad1a-2a6c4d1a509d",
      "metadata": {
        "tags": [],
        "id": "29b14c07-bf6c-4e86-ad1a-2a6c4d1a509d"
      },
      "outputs": [],
      "source": [
        "from ragas.integrations.langchain import EvaluatorChain\n",
        "# from ragas import evaluate\n",
        "from ragas.metrics import (\n",
        "    faithfulness,\n",
        "    answer_relevancy,\n",
        "    context_relevancy,\n",
        "    context_recall,\n",
        ")\n",
        "import os\n",
        "\n",
        "os.environ[\"OPENAI_API_KEY\"] = OPENAI_API_KEY\n",
        "# create evaluation chains\n",
        "faithfulness_chain   = EvaluatorChain(metric=faithfulness)\n",
        "answer_rel_chain     = EvaluatorChain(metric=answer_relevancy)\n",
        "context_rel_chain    = EvaluatorChain(metric=context_relevancy)\n",
        "context_recall_chain = EvaluatorChain(metric=context_recall)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "41a8b636-d738-41bd-ac68-21cce2c4b720",
      "metadata": {
        "id": "41a8b636-d738-41bd-ac68-21cce2c4b720"
      },
      "source": [
        "1. `__call__()`\n",
        "\n",
        "Directly run the evaluation chain with the results from the QA chain. Do note that metrics like context_relevancy and faithfulness require the `source_documents` to be present."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 102,
      "id": "7815671c-1fc8-46ba-8356-4a0bd5558530",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7815671c-1fc8-46ba-8356-4a0bd5558530",
        "outputId": "9842d338-ccb8-46da-8087-730ee9edf0b7"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'query': 'Which borough of New York City has the highest population?',\n",
              " 'result': 'Manhattan (New York County) has the highest population density of any borough in New York City.',\n",
              " 'source_documents': [Document(id='9e00e060-c4df-4f47-bb7f-271ebe2e79f1', metadata={'source': '/content/nyc_text.txt'}, page_content=\"New York City is the most populous city in the United States, with 8,804,190 residents incorporating more immigration into the city than outmigration since the 2010 United States census. More than twice as many people live in New York City as compared to Los Angeles, the second-most populous U.S. city; and New York has more than three times the population of Chicago, the third-most populous U.S. city. New York City gained more residents between 2010 and 2020 (629,000) than any other U.S. city, and a greater amount than the total sum of the gains over the same decade of the next four largest U.S. cities, Los Angeles, Chicago, Houston, and Phoenix, Arizona combined. New York City's population is about 44% of New York State's population, and about 39% of the population of the New York metropolitan area. The majority of New York City residents in 2020 (5,141,538, or 58.4%) were living on Long Island, in Brooklyn, or in Queens. The New York City metropolitan statistical area, has the\"),\n",
              "  Document(id='e555e482-fc16-4b49-a835-0e7fed4836d9', metadata={'source': '/content/nyc_text.txt'}, page_content=\"New York, often called New York City or NYC, is the most populous city in the United States. With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km2), New York City is the most densely populated major city in the United States and more than twice as populous as Los Angeles, the nation's second-largest city. New York City is located at the southern tip of New York State. It constitutes the geographical and demographic center of both the Northeast megalopolis and the New York metropolitan area, the largest metropolitan area in the U.S. by both population and urban area. With over 20.1 million people in its metropolitan statistical area and 23.5 million in its combined statistical area as of 2020, New York is one of the world's most populous megacities, and over 58 million people live within 250 mi (400 km) of the city. New York City is a global cultural, financial, entertainment, and media center with a significant influence on commerce, health care and life\"),\n",
              "  Document(id='5017fdc7-b376-437c-a8a6-56103b8769ca', metadata={'source': '/content/nyc_text.txt'}, page_content=\"Manhattan (New York County) is the geographically smallest and most densely populated borough. It is home to Central Park and most of the city's skyscrapers, and is sometimes locally known as The City. Manhattan's population density of 72,033 people per square mile (27,812/km2) in 2015 makes it the highest of any county in the United States and higher than the density of any individual American city.Manhattan is the cultural, administrative, and financial center of New York City and contains the headquarters of many major multinational corporations, the United Nations headquarters, Wall Street, and a number of important universities. The borough of Manhattan is often described as the financial and cultural center of the world.Most of the borough is situated on Manhattan Island, at the mouth of the Hudson River and the East River, and its southern tip, at the confluence of the two rivers, represents the birthplace of New York City itself. Several small islands also compose part of the\"),\n",
              "  Document(id='850a71e6-db11-40fa-b28d-e5da2f71060c', metadata={'source': '/content/nyc_text.txt'}, page_content=\"=== Population density ===\\n\\nIn 2020, the city had an estimated population density of 29,302.37 inhabitants per square mile (11,313.71/km2), rendering it the nation's most densely populated of all larger municipalities (those with more than 100,000 residents), with several small cities (of fewer than 100,000) in adjacent Hudson County, New Jersey having greater density, as per the 2010 census. Geographically co-extensive with New York County, the borough of Manhattan's 2017 population density of 72,918 inhabitants per square mile (28,154/km2) makes it the highest of any county in the United States and higher than the density of any individual American city. The next three densest counties in the United States, placing second through fourth, are also New York boroughs: Brooklyn, the Bronx, and Queens respectively.\\n\\n\\n=== Race and ethnicity ===\")]}"
            ]
          },
          "metadata": {},
          "execution_count": 102
        }
      ],
      "source": [
        "# Recheck the result that we are going to validate.\n",
        "result"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8cfdd07c-956c-458f-8844-e056f29e3c83",
      "metadata": {
        "id": "8cfdd07c-956c-458f-8844-e056f29e3c83"
      },
      "source": [
        "**Faithfulness**"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "result_updated"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Sv1tsWjerSNk",
        "outputId": "431f7c7d-66c2-4f04-bdbc-5bd0cf9baadc"
      },
      "id": "Sv1tsWjerSNk",
      "execution_count": 103,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'question': 'Which borough of New York City has the highest population?',\n",
              " 'answer': 'Manhattan (New York County) has the highest population density of any borough in New York City.',\n",
              " 'contexts': \"New York City is the most populous city in the United States, with 8,804,190 residents incorporating more immigration into the city than outmigration since the 2010 United States census. More than twice as many people live in New York City as compared to Los Angeles, the second-most populous U.S. city; and New York has more than three times the population of Chicago, the third-most populous U.S. city. New York City gained more residents between 2010 and 2020 (629,000) than any other U.S. city, and a greater amount than the total sum of the gains over the same decade of the next four largest U.S. cities, Los Angeles, Chicago, Houston, and Phoenix, Arizona combined. New York City's population is about 44% of New York State's population, and about 39% of the population of the New York metropolitan area. The majority of New York City residents in 2020 (5,141,538, or 58.4%) were living on Long Island, in Brooklyn, or in Queens. The New York City metropolitan statistical area, has the\\nNew York, often called New York City or NYC, is the most populous city in the United States. With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km2), New York City is the most densely populated major city in the United States and more than twice as populous as Los Angeles, the nation's second-largest city. New York City is located at the southern tip of New York State. It constitutes the geographical and demographic center of both the Northeast megalopolis and the New York metropolitan area, the largest metropolitan area in the U.S. by both population and urban area. With over 20.1 million people in its metropolitan statistical area and 23.5 million in its combined statistical area as of 2020, New York is one of the world's most populous megacities, and over 58 million people live within 250 mi (400 km) of the city. New York City is a global cultural, financial, entertainment, and media center with a significant influence on commerce, health care and life\\nManhattan (New York County) is the geographically smallest and most densely populated borough. It is home to Central Park and most of the city's skyscrapers, and is sometimes locally known as The City. Manhattan's population density of 72,033 people per square mile (27,812/km2) in 2015 makes it the highest of any county in the United States and higher than the density of any individual American city.Manhattan is the cultural, administrative, and financial center of New York City and contains the headquarters of many major multinational corporations, the United Nations headquarters, Wall Street, and a number of important universities. The borough of Manhattan is often described as the financial and cultural center of the world.Most of the borough is situated on Manhattan Island, at the mouth of the Hudson River and the East River, and its southern tip, at the confluence of the two rivers, represents the birthplace of New York City itself. Several small islands also compose part of the\\n=== Population density ===\\n\\nIn 2020, the city had an estimated population density of 29,302.37 inhabitants per square mile (11,313.71/km2), rendering it the nation's most densely populated of all larger municipalities (those with more than 100,000 residents), with several small cities (of fewer than 100,000) in adjacent Hudson County, New Jersey having greater density, as per the 2010 census. Geographically co-extensive with New York County, the borough of Manhattan's 2017 population density of 72,918 inhabitants per square mile (28,154/km2) makes it the highest of any county in the United States and higher than the density of any individual American city. The next three densest counties in the United States, placing second through fourth, are also New York boroughs: Brooklyn, the Bronx, and Queens respectively.\\n\\n\\n=== Race and ethnicity ===\"}"
            ]
          },
          "metadata": {},
          "execution_count": 103
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 104,
      "id": "304f5f0f-a237-4584-becb-a35607caf26b",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "304f5f0f-a237-4584-becb-a35607caf26b",
        "outputId": "bdd16fd6-835d-48ac-ad2d-f6683cb5f50e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/langchain_core/_api/deprecation.py:139: LangChainDeprecationWarning: The method `Chain.__call__` was deprecated in langchain 0.1.0 and will be removed in 0.3.0. Use invoke instead.\n",
            "  warn_deprecated(\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.5"
            ]
          },
          "metadata": {},
          "execution_count": 104
        }
      ],
      "source": [
        "eval_result = faithfulness_chain(result_updated)\n",
        "eval_result['faithfulness']"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8fbd78a9-d7c5-42a0-8705-4c544ec6408e",
      "metadata": {
        "id": "8fbd78a9-d7c5-42a0-8705-4c544ec6408e"
      },
      "source": [
        "High faithfulness_score means that there are exact consistency between the source documents and the answer.\n",
        "\n",
        "You can check lower faithfulness scores by changing the result (answer from LLM) or source_documents to something else."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 105,
      "id": "15f42b97-a84f-4015-9da5-fa3b0b703c24",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "15f42b97-a84f-4015-9da5-fa3b0b703c24",
        "outputId": "bb8790a3-f12b-4530-d8f9-a83d4f81d266"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'question': 'Which borough of New York City has the highest population?',\n",
              " 'answer': 'we are the champions',\n",
              " 'contexts': 'Queen'}"
            ]
          },
          "metadata": {},
          "execution_count": 105
        }
      ],
      "source": [
        "fake_result = result_updated.copy()\n",
        "fake_result[\"answer\"] = \"we are the champions\"\n",
        "fake_result[\"contexts\"] = \"Queen\"\n",
        "\n",
        "fake_result"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "eval_result = faithfulness_chain(fake_result)\n",
        "eval_result[\"faithfulness\"]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-I50x_fJvM6f",
        "outputId": "9635c05b-d710-455c-95ef-a920aeaa5ac0"
      },
      "id": "-I50x_fJvM6f",
      "execution_count": 106,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:ragas.metrics._faithfulness:No statements were generated from the answer.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "nan"
            ]
          },
          "metadata": {},
          "execution_count": 106
        }
      ]
    },
    {
      "cell_type": "markdown",
      "id": "02348380-159a-4578-a95e-493cdcfcebe7",
      "metadata": {
        "id": "02348380-159a-4578-a95e-493cdcfcebe7"
      },
      "source": [
        "**Context Relevancy**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 107,
      "id": "0fd2fb9e-19d1-4cf6-9773-897546c2d6bb",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0fd2fb9e-19d1-4cf6-9773-897546c2d6bb",
        "outputId": "5a5629d6-58a9-45c0-bc1a-18c30725b09f"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'question': 'Which borough of New York City has the highest population?',\n",
              " 'answer': 'Manhattan (New York County) has the highest population density of any borough in New York City.',\n",
              " 'contexts': \"New York City is the most populous city in the United States, with 8,804,190 residents incorporating more immigration into the city than outmigration since the 2010 United States census. More than twice as many people live in New York City as compared to Los Angeles, the second-most populous U.S. city; and New York has more than three times the population of Chicago, the third-most populous U.S. city. New York City gained more residents between 2010 and 2020 (629,000) than any other U.S. city, and a greater amount than the total sum of the gains over the same decade of the next four largest U.S. cities, Los Angeles, Chicago, Houston, and Phoenix, Arizona combined. New York City's population is about 44% of New York State's population, and about 39% of the population of the New York metropolitan area. The majority of New York City residents in 2020 (5,141,538, or 58.4%) were living on Long Island, in Brooklyn, or in Queens. The New York City metropolitan statistical area, has the\\nNew York, often called New York City or NYC, is the most populous city in the United States. With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km2), New York City is the most densely populated major city in the United States and more than twice as populous as Los Angeles, the nation's second-largest city. New York City is located at the southern tip of New York State. It constitutes the geographical and demographic center of both the Northeast megalopolis and the New York metropolitan area, the largest metropolitan area in the U.S. by both population and urban area. With over 20.1 million people in its metropolitan statistical area and 23.5 million in its combined statistical area as of 2020, New York is one of the world's most populous megacities, and over 58 million people live within 250 mi (400 km) of the city. New York City is a global cultural, financial, entertainment, and media center with a significant influence on commerce, health care and life\\nManhattan (New York County) is the geographically smallest and most densely populated borough. It is home to Central Park and most of the city's skyscrapers, and is sometimes locally known as The City. Manhattan's population density of 72,033 people per square mile (27,812/km2) in 2015 makes it the highest of any county in the United States and higher than the density of any individual American city.Manhattan is the cultural, administrative, and financial center of New York City and contains the headquarters of many major multinational corporations, the United Nations headquarters, Wall Street, and a number of important universities. The borough of Manhattan is often described as the financial and cultural center of the world.Most of the borough is situated on Manhattan Island, at the mouth of the Hudson River and the East River, and its southern tip, at the confluence of the two rivers, represents the birthplace of New York City itself. Several small islands also compose part of the\\n=== Population density ===\\n\\nIn 2020, the city had an estimated population density of 29,302.37 inhabitants per square mile (11,313.71/km2), rendering it the nation's most densely populated of all larger municipalities (those with more than 100,000 residents), with several small cities (of fewer than 100,000) in adjacent Hudson County, New Jersey having greater density, as per the 2010 census. Geographically co-extensive with New York County, the borough of Manhattan's 2017 population density of 72,918 inhabitants per square mile (28,154/km2) makes it the highest of any county in the United States and higher than the density of any individual American city. The next three densest counties in the United States, placing second through fourth, are also New York boroughs: Brooklyn, the Bronx, and Queens respectively.\\n\\n\\n=== Race and ethnicity ===\"}"
            ]
          },
          "metadata": {},
          "execution_count": 107
        }
      ],
      "source": [
        "result_updated"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Find matching ground truths\n",
        "for item in examples:\n",
        "    if item['query'] == result_updated['question']:\n",
        "        result_updated['ground_truth'] = item['ground_truths']\n",
        "        break\n",
        "\n",
        "if 'ground_truths' in result_updated:\n",
        "    del result_updated['ground_truths']\n",
        "\n",
        "result_updated"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V2Zfrp8ZwfS1",
        "outputId": "c8826fc6-35bc-4359-9200-6158a7f1538d"
      },
      "id": "V2Zfrp8ZwfS1",
      "execution_count": 108,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'question': 'Which borough of New York City has the highest population?',\n",
              " 'answer': 'Manhattan (New York County) has the highest population density of any borough in New York City.',\n",
              " 'contexts': \"New York City is the most populous city in the United States, with 8,804,190 residents incorporating more immigration into the city than outmigration since the 2010 United States census. More than twice as many people live in New York City as compared to Los Angeles, the second-most populous U.S. city; and New York has more than three times the population of Chicago, the third-most populous U.S. city. New York City gained more residents between 2010 and 2020 (629,000) than any other U.S. city, and a greater amount than the total sum of the gains over the same decade of the next four largest U.S. cities, Los Angeles, Chicago, Houston, and Phoenix, Arizona combined. New York City's population is about 44% of New York State's population, and about 39% of the population of the New York metropolitan area. The majority of New York City residents in 2020 (5,141,538, or 58.4%) were living on Long Island, in Brooklyn, or in Queens. The New York City metropolitan statistical area, has the\\nNew York, often called New York City or NYC, is the most populous city in the United States. With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km2), New York City is the most densely populated major city in the United States and more than twice as populous as Los Angeles, the nation's second-largest city. New York City is located at the southern tip of New York State. It constitutes the geographical and demographic center of both the Northeast megalopolis and the New York metropolitan area, the largest metropolitan area in the U.S. by both population and urban area. With over 20.1 million people in its metropolitan statistical area and 23.5 million in its combined statistical area as of 2020, New York is one of the world's most populous megacities, and over 58 million people live within 250 mi (400 km) of the city. New York City is a global cultural, financial, entertainment, and media center with a significant influence on commerce, health care and life\\nManhattan (New York County) is the geographically smallest and most densely populated borough. It is home to Central Park and most of the city's skyscrapers, and is sometimes locally known as The City. Manhattan's population density of 72,033 people per square mile (27,812/km2) in 2015 makes it the highest of any county in the United States and higher than the density of any individual American city.Manhattan is the cultural, administrative, and financial center of New York City and contains the headquarters of many major multinational corporations, the United Nations headquarters, Wall Street, and a number of important universities. The borough of Manhattan is often described as the financial and cultural center of the world.Most of the borough is situated on Manhattan Island, at the mouth of the Hudson River and the East River, and its southern tip, at the confluence of the two rivers, represents the birthplace of New York City itself. Several small islands also compose part of the\\n=== Population density ===\\n\\nIn 2020, the city had an estimated population density of 29,302.37 inhabitants per square mile (11,313.71/km2), rendering it the nation's most densely populated of all larger municipalities (those with more than 100,000 residents), with several small cities (of fewer than 100,000) in adjacent Hudson County, New Jersey having greater density, as per the 2010 census. Geographically co-extensive with New York County, the borough of Manhattan's 2017 population density of 72,918 inhabitants per square mile (28,154/km2) makes it the highest of any county in the United States and higher than the density of any individual American city. The next three densest counties in the United States, placing second through fourth, are also New York boroughs: Brooklyn, the Bronx, and Queens respectively.\\n\\n\\n=== Race and ethnicity ===\",\n",
              " 'ground_truth': ['Brooklyn']}"
            ]
          },
          "metadata": {},
          "execution_count": 108
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "eval_result = context_recall_chain(result_updated)\n",
        "eval_result[\"context_recall\"]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WR4llBhkwiDR",
        "outputId": "95f6ec96-182f-4331-cba9-3305217e3b48"
      },
      "id": "WR4llBhkwiDR",
      "execution_count": 109,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.3333333333333333"
            ]
          },
          "metadata": {},
          "execution_count": 109
        }
      ]
    },
    {
      "cell_type": "markdown",
      "id": "118cba80-c2a3-408e-9d8d-857521cbe723",
      "metadata": {
        "id": "118cba80-c2a3-408e-9d8d-857521cbe723"
      },
      "source": [
        "High context_recall_score means that the ground truth is present in the source documents.\n",
        "\n",
        "You can check lower context recall scores by changing the source_documents to something else."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 110,
      "id": "f6610c5f-5f8e-406c-885c-093012a5dc44",
      "metadata": {
        "id": "f6610c5f-5f8e-406c-885c-093012a5dc44"
      },
      "outputs": [],
      "source": [
        "from langchain.schema import Document\n",
        "\n",
        "# Create the fake result with a Document\n",
        "fake_result = result_updated.copy()\n",
        "fake_result[\"contexts\"] = [Document(page_content=\"Tesla cars\")]\n",
        "\n",
        "# Extract page_content from each Document in the list\n",
        "contexts = [doc.page_content for doc in fake_result['contexts']]\n",
        "\n",
        "# Update the fake_result with the extracted contexts\n",
        "fake_result['contexts'] = contexts"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "fake_result"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hdbSYEAAwzuC",
        "outputId": "d18f05b2-45fb-48b8-e02d-6d8b82e0faf3"
      },
      "id": "hdbSYEAAwzuC",
      "execution_count": 111,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'question': 'Which borough of New York City has the highest population?',\n",
              " 'answer': 'Manhattan (New York County) has the highest population density of any borough in New York City.',\n",
              " 'contexts': ['Tesla cars'],\n",
              " 'ground_truth': ['Brooklyn']}"
            ]
          },
          "metadata": {},
          "execution_count": 111
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "eval_result = context_recall_chain(fake_result)\n",
        "eval_result[\"context_recall\"]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oCHU_WNew1_m",
        "outputId": "d48f63fe-2aca-48f0-a54d-a74b32021e4d"
      },
      "id": "oCHU_WNew1_m",
      "execution_count": 112,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.0"
            ]
          },
          "metadata": {},
          "execution_count": 112
        }
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3dc06403-c514-47fd-ac82-d95ece8d2f06",
      "metadata": {
        "id": "3dc06403-c514-47fd-ac82-d95ece8d2f06"
      },
      "source": [
        "2. `evaluate()`\n",
        "\n",
        "Evaluate a list of inputs/queries and the outputs/predictions from the QA chain."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 113,
      "id": "f6b23708-b94d-4649-acd9-0e268f72f94a",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "f6b23708-b94d-4649-acd9-0e268f72f94a",
        "outputId": "daef1650-5633-4468-99a0-c34a3806b3e6"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'query': 'What is the population of New York City as of 2020?',\n",
              "  'answer': ['8,804,190']},\n",
              " {'query': 'Which borough of New York City has the highest population?',\n",
              "  'answer': ['Brooklyn']},\n",
              " {'query': 'What is the economic significance of New York City?',\n",
              "  'answer': [\"New York City's economic significance is vast, as it serves as the global financial capital, housing Wall Street and major financial institutions. Its diverse economy spans technology, media, healthcare, education, and more, making it resilient to economic fluctuations. NYC is a hub for international business, attracting global companies, and boasts a large, skilled labor force. Its real estate market, tourism, cultural industries, and educational institutions further fuel its economic prowess. The city's transportation network and global influence amplify its impact on the world stage, solidifying its status as a vital economic player and cultural epicenter.\"]},\n",
              " {'query': 'How did New York City get its name?',\n",
              "  'answer': ['New York City got its name when it came under British control in 1664. King Charles II of England granted the lands to his brother, the Duke of York, who named the city New York in his own honor.']},\n",
              " {'query': 'What is the significance of the Statue of Liberty in New York City?',\n",
              "  'answer': ['The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants who arrived in the U.S. by ship in the late 19th and early 20th centuries, representing hope and freedom for those seeking a better life. It has since become an iconic landmark and a global symbol of cultural diversity and freedom.']}]"
            ]
          },
          "metadata": {},
          "execution_count": 113
        }
      ],
      "source": [
        "# run the queries as a batch for efficiency\n",
        "predictions = qa_chain.batch(examples)\n",
        "\n",
        "# evaluate\n",
        "from langchain.evaluation.qa import QAEvalChain\n",
        "\n",
        "for example in examples:\n",
        "    if 'ground_truths' in example:\n",
        "        example['answer'] = example.pop('ground_truths')\n",
        "\n",
        "examples"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "predictions"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9s8qY8l8xQev",
        "outputId": "72cdc685-0641-4f63-8757-0961b8b6385f"
      },
      "id": "9s8qY8l8xQev",
      "execution_count": 114,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'query': 'What is the population of New York City as of 2020?',\n",
              "  'ground_truths': ['8,804,190'],\n",
              "  'result': 'The population of New York City as of 2020 is 8,804,190 residents.',\n",
              "  'source_documents': [Document(id='9e00e060-c4df-4f47-bb7f-271ebe2e79f1', metadata={'source': '/content/nyc_text.txt'}, page_content=\"New York City is the most populous city in the United States, with 8,804,190 residents incorporating more immigration into the city than outmigration since the 2010 United States census. More than twice as many people live in New York City as compared to Los Angeles, the second-most populous U.S. city; and New York has more than three times the population of Chicago, the third-most populous U.S. city. New York City gained more residents between 2010 and 2020 (629,000) than any other U.S. city, and a greater amount than the total sum of the gains over the same decade of the next four largest U.S. cities, Los Angeles, Chicago, Houston, and Phoenix, Arizona combined. New York City's population is about 44% of New York State's population, and about 39% of the population of the New York metropolitan area. The majority of New York City residents in 2020 (5,141,538, or 58.4%) were living on Long Island, in Brooklyn, or in Queens. The New York City metropolitan statistical area, has the\"),\n",
              "   Document(id='e555e482-fc16-4b49-a835-0e7fed4836d9', metadata={'source': '/content/nyc_text.txt'}, page_content=\"New York, often called New York City or NYC, is the most populous city in the United States. With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km2), New York City is the most densely populated major city in the United States and more than twice as populous as Los Angeles, the nation's second-largest city. New York City is located at the southern tip of New York State. It constitutes the geographical and demographic center of both the Northeast megalopolis and the New York metropolitan area, the largest metropolitan area in the U.S. by both population and urban area. With over 20.1 million people in its metropolitan statistical area and 23.5 million in its combined statistical area as of 2020, New York is one of the world's most populous megacities, and over 58 million people live within 250 mi (400 km) of the city. New York City is a global cultural, financial, entertainment, and media center with a significant influence on commerce, health care and life\"),\n",
              "   Document(id='850a71e6-db11-40fa-b28d-e5da2f71060c', metadata={'source': '/content/nyc_text.txt'}, page_content=\"=== Population density ===\\n\\nIn 2020, the city had an estimated population density of 29,302.37 inhabitants per square mile (11,313.71/km2), rendering it the nation's most densely populated of all larger municipalities (those with more than 100,000 residents), with several small cities (of fewer than 100,000) in adjacent Hudson County, New Jersey having greater density, as per the 2010 census. Geographically co-extensive with New York County, the borough of Manhattan's 2017 population density of 72,918 inhabitants per square mile (28,154/km2) makes it the highest of any county in the United States and higher than the density of any individual American city. The next three densest counties in the United States, placing second through fourth, are also New York boroughs: Brooklyn, the Bronx, and Queens respectively.\\n\\n\\n=== Race and ethnicity ===\"),\n",
              "   Document(id='83dd8b7b-0e0f-4d56-9017-c28d1eb20bfa', metadata={'source': '/content/nyc_text.txt'}, page_content=\"New York's population reached all-time highs in the 2000 census and then again in the 2010 census.\")]},\n",
              " {'query': 'Which borough of New York City has the highest population?',\n",
              "  'ground_truths': ['Brooklyn'],\n",
              "  'result': 'Manhattan (New York County) has the highest population density of any borough in New York City.',\n",
              "  'source_documents': [Document(id='9e00e060-c4df-4f47-bb7f-271ebe2e79f1', metadata={'source': '/content/nyc_text.txt'}, page_content=\"New York City is the most populous city in the United States, with 8,804,190 residents incorporating more immigration into the city than outmigration since the 2010 United States census. More than twice as many people live in New York City as compared to Los Angeles, the second-most populous U.S. city; and New York has more than three times the population of Chicago, the third-most populous U.S. city. New York City gained more residents between 2010 and 2020 (629,000) than any other U.S. city, and a greater amount than the total sum of the gains over the same decade of the next four largest U.S. cities, Los Angeles, Chicago, Houston, and Phoenix, Arizona combined. New York City's population is about 44% of New York State's population, and about 39% of the population of the New York metropolitan area. The majority of New York City residents in 2020 (5,141,538, or 58.4%) were living on Long Island, in Brooklyn, or in Queens. The New York City metropolitan statistical area, has the\"),\n",
              "   Document(id='e555e482-fc16-4b49-a835-0e7fed4836d9', metadata={'source': '/content/nyc_text.txt'}, page_content=\"New York, often called New York City or NYC, is the most populous city in the United States. With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km2), New York City is the most densely populated major city in the United States and more than twice as populous as Los Angeles, the nation's second-largest city. New York City is located at the southern tip of New York State. It constitutes the geographical and demographic center of both the Northeast megalopolis and the New York metropolitan area, the largest metropolitan area in the U.S. by both population and urban area. With over 20.1 million people in its metropolitan statistical area and 23.5 million in its combined statistical area as of 2020, New York is one of the world's most populous megacities, and over 58 million people live within 250 mi (400 km) of the city. New York City is a global cultural, financial, entertainment, and media center with a significant influence on commerce, health care and life\"),\n",
              "   Document(id='5017fdc7-b376-437c-a8a6-56103b8769ca', metadata={'source': '/content/nyc_text.txt'}, page_content=\"Manhattan (New York County) is the geographically smallest and most densely populated borough. It is home to Central Park and most of the city's skyscrapers, and is sometimes locally known as The City. Manhattan's population density of 72,033 people per square mile (27,812/km2) in 2015 makes it the highest of any county in the United States and higher than the density of any individual American city.Manhattan is the cultural, administrative, and financial center of New York City and contains the headquarters of many major multinational corporations, the United Nations headquarters, Wall Street, and a number of important universities. The borough of Manhattan is often described as the financial and cultural center of the world.Most of the borough is situated on Manhattan Island, at the mouth of the Hudson River and the East River, and its southern tip, at the confluence of the two rivers, represents the birthplace of New York City itself. Several small islands also compose part of the\"),\n",
              "   Document(id='850a71e6-db11-40fa-b28d-e5da2f71060c', metadata={'source': '/content/nyc_text.txt'}, page_content=\"=== Population density ===\\n\\nIn 2020, the city had an estimated population density of 29,302.37 inhabitants per square mile (11,313.71/km2), rendering it the nation's most densely populated of all larger municipalities (those with more than 100,000 residents), with several small cities (of fewer than 100,000) in adjacent Hudson County, New Jersey having greater density, as per the 2010 census. Geographically co-extensive with New York County, the borough of Manhattan's 2017 population density of 72,918 inhabitants per square mile (28,154/km2) makes it the highest of any county in the United States and higher than the density of any individual American city. The next three densest counties in the United States, placing second through fourth, are also New York boroughs: Brooklyn, the Bronx, and Queens respectively.\\n\\n\\n=== Race and ethnicity ===\")]},\n",
              " {'query': 'What is the economic significance of New York City?',\n",
              "  'ground_truths': [\"New York City's economic significance is vast, as it serves as the global financial capital, housing Wall Street and major financial institutions. Its diverse economy spans technology, media, healthcare, education, and more, making it resilient to economic fluctuations. NYC is a hub for international business, attracting global companies, and boasts a large, skilled labor force. Its real estate market, tourism, cultural industries, and educational institutions further fuel its economic prowess. The city's transportation network and global influence amplify its impact on the world stage, solidifying its status as a vital economic player and cultural epicenter.\"],\n",
              "  'result': \"New York City is a global hub of business and commerce, with a diverse economy that includes industries such as banking and finance, health care, life sciences, technology, retail, transportation, tourism, real estate, media, advertising, legal services, and more. It is a major center for worldwide banking and finance, with Wall Street being a symbol of the financial industry. The city's economy is one of the largest in the world, with a gross metropolitan product of over $2.4 trillion as of 2021. Additionally, New York City is home to the headquarters of the United Nations and is considered an important center for international diplomacy.\",\n",
              "  'source_documents': [Document(id='396e53fa-2934-4020-96d0-ef30721088d7', metadata={'source': '/content/nyc_text.txt'}, page_content=\"Despite New York's heavy reliance on its vast public transit system, streets are a defining feature of the city. The Commissioners' Plan of 1811 greatly influenced the city's physical development. Several of the city's streets and avenues, including Broadway, Wall Street, Madison Avenue, and Seventh Avenue are also used as metonyms for national industries there: the theater, finance, advertising, and fashion organizations, respectively.\"),\n",
              "   Document(id='51e38f7b-029e-42f5-8a01-a7a921f47a64', metadata={'source': '/content/nyc_text.txt'}, page_content=\"sciences, research, technology, education, politics, tourism, dining, art, fashion, and sports. Home to the headquarters of the United Nations, New York is an important center for international diplomacy, and is sometimes described as the capital of the world.Situated on one of the world's largest natural harbors and extending into the Atlantic Ocean, New York City comprises five boroughs, each of which is coextensive with a respective county of the state of New York. The five boroughs, which were created in 1898 when local governments were consolidated into a single municipal entity, are: Brooklyn (in Kings County), Queens (in Queens County), Manhattan (in New York County), The Bronx (in Bronx County), and Staten Island (in Richmond County).As of 2021, the New York metropolitan area is the largest metropolitan economy in the world with a gross metropolitan product of over $2.4 trillion. If the New York metropolitan area were a sovereign state, it would have the eighth-largest economy\"),\n",
              "   Document(id='15488f9a-d6ae-4f0b-bc66-f71121409373', metadata={'source': '/content/nyc_text.txt'}, page_content=\"New York City is a global hub of business and commerce and an established safe haven for global investors, and is sometimes described as the capital of the world. The term global city was popularized by sociologist Saskia Sassen in her 1991 work, The Global City: New York, London, Tokyo. New York is a center for worldwide banking and finance, health care and life sciences, medical technology and research, retailing, world trade, transportation, tourism, real estate, new media, traditional media, advertising, legal services, accountancy, insurance, both musical and prose theater, fashion, and the arts in the United States; while Silicon Alley, metonymous for New York's broad-spectrum high technology sphere, continues to expand. The Port of New York and New Jersey is a major economic engine, handling a maritime cargo volume in the ten months through October 2022 of over 8.2 million TEUs, benefitting post-Panamax from the expansion of the Panama Canal, and accelerating ahead of\"),\n",
              "   Document(id='f496931a-6baf-4187-afc6-a9ca4bb0b620', metadata={'source': '/content/nyc_text.txt'}, page_content='New York City has been a metropolitan municipality with a Strong mayor–council form of government since its consolidation in 1898. In New York City, the city government is responsible for public education, correctional institutions, public safety, recreational facilities, sanitation, water supply, and welfare services.')]},\n",
              " {'query': 'How did New York City get its name?',\n",
              "  'ground_truths': ['New York City got its name when it came under British control in 1664. King Charles II of England granted the lands to his brother, the Duke of York, who named the city New York in his own honor.'],\n",
              "  'result': 'New York City was originally named New Amsterdam by Dutch colonists in 1626. When the city came under British control in 1664, it was renamed New York after King Charles II of England granted the lands to his brother, the Duke of York. The city has been continuously named New York since November 1674.',\n",
              "  'source_documents': [Document(id='f1703fa1-92ac-4168-a513-d65c5f4423c4', metadata={'source': '/content/nyc_text.txt'}, page_content='The city and its metropolitan area constitute the premier gateway for legal immigration to the United States. As many as 800 languages are spoken in New York, making it the most linguistically diverse city in the world. New York City is home to more than 3.2 million residents born outside the U.S., the largest foreign-born population of any city in the world as of 2016.New York City traces its origins to a trading post founded on the southern tip of Manhattan Island by Dutch colonists in approximately 1624. The settlement was named New Amsterdam (Dutch: Nieuw Amsterdam) in 1626 and was chartered as a city in 1653. The city came under British control in 1664 and was renamed New York after King Charles II of England granted the lands to his brother, the Duke of York. The city was regained by the Dutch in July 1673 and was renamed New Orange for one year and three months; the city has been continuously named New York since November 1674. New York City was the capital of the United States'),\n",
              "   Document(id='f496931a-6baf-4187-afc6-a9ca4bb0b620', metadata={'source': '/content/nyc_text.txt'}, page_content='New York City has been a metropolitan municipality with a Strong mayor–council form of government since its consolidation in 1898. In New York City, the city government is responsible for public education, correctional institutions, public safety, recreational facilities, sanitation, water supply, and welfare services.'),\n",
              "   Document(id='396e53fa-2934-4020-96d0-ef30721088d7', metadata={'source': '/content/nyc_text.txt'}, page_content=\"Despite New York's heavy reliance on its vast public transit system, streets are a defining feature of the city. The Commissioners' Plan of 1811 greatly influenced the city's physical development. Several of the city's streets and avenues, including Broadway, Wall Street, Madison Avenue, and Seventh Avenue are also used as metonyms for national industries there: the theater, finance, advertising, and fashion organizations, respectively.\"),\n",
              "   Document(id='3c9ec71a-e088-46c2-9568-de89880113b6', metadata={'source': '/content/nyc_text.txt'}, page_content=\"In the pre-Columbian era, the area of present-day New York City was inhabited by Algonquian Native Americans, including the Lenape. Their homeland, known as Lenapehoking, included the present-day areas of Staten Island, Manhattan, the Bronx, the western portion of Long Island (including the areas that would later become the boroughs of Brooklyn and Queens), and the Lower Hudson Valley.The first documented visit into New York Harbor by a European was in 1524 by Italian Giovanni da Verrazzano, an explorer from Florence in the service of the French crown. He claimed the area for France and named it Nouvelle Angoulême (New Angoulême). A Spanish expedition, led by the Portuguese captain Estêvão Gomes sailing for Emperor Charles V, arrived in New York Harbor in January 1525 and charted the mouth of the Hudson River, which he named Río de San Antonio ('Saint Anthony's River'). The Padrón Real of 1527, the first scientific map to show the East Coast of North America continuously, was informed\")]},\n",
              " {'query': 'What is the significance of the Statue of Liberty in New York City?',\n",
              "  'ground_truths': ['The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants who arrived in the U.S. by ship in the late 19th and early 20th centuries, representing hope and freedom for those seeking a better life. It has since become an iconic landmark and a global symbol of cultural diversity and freedom.'],\n",
              "  'result': 'The Statue of Liberty in New York City is a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants as they arrived in the U.S. in the late 19th and early 20th centuries. It represents freedom, cultural diversity, and is an iconic landmark that symbolizes hope and opportunity for many people.',\n",
              "  'source_documents': [Document(id='819922fa-2450-4ab2-81cc-a25524b310af', metadata={'source': '/content/nyc_text.txt'}, page_content='from 1785 until 1790, and has been the largest U.S. city since 1790. The Statue of Liberty greeted millions of immigrants as they came to the U.S. by ship in the late 19th and early 20th centuries, and is a symbol of the U.S. and its ideals of liberty and peace. In the 21st century, New York City has emerged as a global node of creativity, entrepreneurship, and as a symbol of freedom and cultural diversity. The New York Times has won the most Pulitzer Prizes for journalism and remains the U.S. media\\'s \"newspaper of record\". In 2019, New York City was voted the greatest city in the world in a survey of over 30,000 people from 48 cities worldwide, citing its cultural diversity.Many districts and monuments in New York City are major landmarks, including three of the world\\'s ten most visited tourist attractions in 2013. A record 66.6 million tourists visited New York City in 2019. Times Square is the brightly illuminated hub of the Broadway Theater District, one of the world\\'s busiest'),\n",
              "   Document(id='31d9b4bc-0469-4b45-9d7d-1ae6e15401a8', metadata={'source': '/content/nyc_text.txt'}, page_content=\"The Statue of Liberty National Monument and Ellis Island Immigration Museum are managed by the National Park Service and are in both New York and New Jersey. They are joined in the harbor by Governors Island National Monument. Historic sites under federal management on Manhattan Island include Stonewall National Monument; Castle Clinton National Monument; Federal Hall National Memorial; Theodore Roosevelt Birthplace National Historic Site; General Grant National Memorial (Grant's Tomb); African Burial Ground National Monument; and Hamilton Grange National Memorial. Hundreds of properties are listed on the National Register of Historic Places or as a National Historic Landmark.\"),\n",
              "   Document(id='867211eb-ee41-45a9-8f3d-56f4c4ffbf1f', metadata={'source': '/content/nyc_text.txt'}, page_content=\"Tourism is a vital industry for New York City, and NYC & Company represents the city's official bureau of tourism. New York has witnessed a growing combined volume of international and domestic tourists, reflecting over 60 million visitors to the city per year, the world's busiest tourist destination. Approximately 12 million visitors to New York City have been from outside the United States, with the highest numbers from the United Kingdom, Canada, Brazil, and China. Multiple sources have called New York the most photographed city in the world.I Love New York (stylized I ❤ NY) is both a logo and a song that are the basis of an advertising campaign and have been used since 1977 to promote tourism in New York City, and later to promote New York State as well. The trademarked logo, owned by New York State Empire State Development, appears in souvenir shops and brochures throughout the city and state, some licensed, many not. The song is the state song of New York.\"),\n",
              "   Document(id='2e368f21-73c3-4c8f-b692-5a2c94a86306', metadata={'source': '/content/nyc_text.txt'}, page_content=\"The majority of the most high-profile tourist destinations to the city are situated in Manhattan. These include Times Square; Broadway theater productions; the Empire State Building; the Statue of Liberty; Ellis Island; the United Nations headquarters; the World Trade Center (including the National September 11 Memorial & Museum and One World Trade Center); the art museums along Museum Mile; green spaces such as Central Park, Washington Square Park, the High Line, and the medieval gardens of The Cloisters; the Stonewall Inn; Rockefeller Center; ethnic enclaves including the Manhattan Chinatown, Koreatown, Curry Hill, Harlem, Spanish Harlem, Little Italy, and Little Australia; luxury shopping along Fifth and Madison Avenues; and events such as the Halloween Parade in Greenwich Village; the Brooklyn Bridge (shared with Brooklyn); the Macy's Thanksgiving Day Parade; the lighting of the Rockefeller Center Christmas Tree; the St. Patrick's Day Parade; seasonal activities such as ice\")]}]"
            ]
          },
          "metadata": {},
          "execution_count": 114
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 119,
      "id": "d661f6b9-04c7-40b7-874b-25f53cfab9d9",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 211
        },
        "id": "d661f6b9-04c7-40b7-874b-25f53cfab9d9",
        "outputId": "7a991dbb-c5a1-4eed-a0a5-926342baadfa"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "evaluating...\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "AttributeError",
          "evalue": "'EvaluatorChain' object has no attribute 'evaluate'",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-119-9027d758f18a>\u001b[0m in \u001b[0;36m<cell line: 3>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# evaluate context recall\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"evaluating...\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfaithfulness_chain\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mevaluate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexamples\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpredictions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0mr\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mAttributeError\u001b[0m: 'EvaluatorChain' object has no attribute 'evaluate'"
          ]
        }
      ],
      "source": [
        "# evaluate context recall\n",
        "print(\"evaluating...\")\n",
        "r = faithfulness_chain.evaluate(examples, predictions)\n",
        "r"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "04KBV6dVxaMq"
      },
      "id": "04KBV6dVxaMq",
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.8"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}