lab-langchain-evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1b00dae2-c271-4a69-a579-742e084a9058",
   "metadata": {},
   "source": [
    "# Lab | Langchain Evaluation\n",
    "\n",
    "## Intro\n",
    "\n",
    "Pick different sets of data and re-run this notebook. The point is for you to understand all steps involve and the many different ways one can and should evaluate LLM applications.\n",
    "\n",
    "What did you learn? - Let's discuss that in class"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52824b89-532a-4e54-87e9-1410813cd39e",
   "metadata": {},
   "source": [
    "## LangChain: Evaluation\n",
    "\n",
    "### Outline:\n",
    "\n",
    "* Example generation\n",
    "* Manual evaluation (and debuging)\n",
    "* LLM-assisted evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b7ed03ed-1322-49e3-b2a2-33e94fb592ef",
   "metadata": {
    "height": 98,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from dotenv import load_dotenv, find_dotenv\n",
    "import os\n",
    "_ = load_dotenv(find_dotenv())\n",
    "\n",
    "OPENAI_API_KEY  = os.getenv('OPENAI_API_KEY') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce74685f-50e9-48cf-8e2c-fac05e8e3ffd",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e7e29d2c-ba67-4cba-8ded-375fe040b9ba",
   "metadata": {},
   "source": [
    "### Example 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28008949",
   "metadata": {},
   "source": [
    "#### Create our QandA application"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "974acf8e-8f88-42de-88f8-40a82cb58e8b",
   "metadata": {
    "height": 115,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.chains import RetrievalQA\n",
    "from langchain_openai import ChatOpenAI\n",
    "from langchain.llms import OpenAI\n",
    "from langchain_huggingface import HuggingFaceEmbeddings\n",
    "from langchain.document_loaders import CSVLoader, TextLoader\n",
    "from langchain.indexes import VectorstoreIndexCreator\n",
    "from langchain.vectorstores import DocArrayInMemorySearch\n",
    "from langchain.chains import LLMChain\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9ec1106d",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "file = '../data/OutdoorClothingCatalog_1000.csv'\n",
    "loader = CSVLoader(file_path=file)\n",
    "data = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "550eb642-c223-4d78-8f92-0f265ef78b86",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# !pip install --upgrade --force-reinstall sentence-transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b31c218f",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.11/site-packages/langchain_core/_api/deprecation.py:139: LangChainDeprecationWarning: The class `HuggingFaceEmbeddings` was deprecated in LangChain 0.2.2 and will be removed in 0.3.0. An updated version of the class exists in the langchain-huggingface package and should be used instead. To use it run `pip install -U langchain-huggingface` and import as `from langchain_huggingface import HuggingFaceEmbeddings`.\n",
      "  warn_deprecated(\n",
      "/opt/anaconda3/lib/python3.11/site-packages/sentence_transformers/cross_encoder/CrossEncoder.py:11: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n",
      "  from tqdm.autonotebook import tqdm, trange\n"
     ]
    }
   ],
   "source": [
    "index = VectorstoreIndexCreator(\n",
    "    vectorstore_cls=DocArrayInMemorySearch,\n",
    "    embedding=HuggingFaceEmbeddings(model_name=\"all-MiniLM-L6-v2\", model_kwargs = {'device': 'cpu'})\n",
    ").from_loaders([loader])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a2006054",
   "metadata": {
    "height": 183,
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.11/site-packages/langchain_core/_api/deprecation.py:139: LangChainDeprecationWarning: The class `ChatOpenAI` was deprecated in LangChain 0.0.10 and will be removed in 0.3.0. An updated version of the class exists in the langchain-openai package and should be used instead. To use it run `pip install -U langchain-openai` and import as `from langchain_openai import ChatOpenAI`.\n",
      "  warn_deprecated(\n"
     ]
    }
   ],
   "source": [
    "llm = ChatOpenAI(temperature = 0.0)\n",
    "qa = RetrievalQA.from_chain_type(\n",
    "    llm=llm, \n",
    "    chain_type=\"stuff\", \n",
    "    retriever=index.vectorstore.as_retriever(), \n",
    "    verbose=True,\n",
    "    chain_type_kwargs = {\n",
    "        \"document_separator\": \"<<<<>>>>>\"\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "791ebd73",
   "metadata": {},
   "source": [
    "#### Coming up with test datapoints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "fb04a0f9",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(page_content=\": 10\\nname: Cozy Comfort Pullover Set, Stripe\\ndescription: Perfect for lounging, this striped knit set lives up to its name. We used ultrasoft fabric and an easy design that's as comfortable at bedtime as it is when we have to make a quick run out.\\n\\nSize & Fit\\n- Pants are Favorite Fit: Sits lower on the waist.\\n- Relaxed Fit: Our most generous fit sits farthest from the body.\\n\\nFabric & Care\\n- In the softest blend of 63% polyester, 35% rayon and 2% spandex.\\n\\nAdditional Features\\n- Relaxed fit top with raglan sleeves and rounded hem.\\n- Pull-on pants have a wide elastic waistband and drawstring, side pockets and a modern slim leg.\\n\\nImported.\", metadata={'source': '../data/OutdoorClothingCatalog_1000.csv', 'row': 10})"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "fe4a88c2",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(page_content=': 11\\nname: Ultra-Lofty 850 Stretch Down Hooded Jacket\\ndescription: This technical stretch down jacket from our DownTek collection is sure to keep you warm and comfortable with its full-stretch construction providing exceptional range of motion. With a slightly fitted style that falls at the hip and best with a midweight layer, this jacket is suitable for light activity up to 20° and moderate activity up to -30°. The soft and durable 100% polyester shell offers complete windproof protection and is insulated with warm, lofty goose down. Other features include welded baffles for a no-stitch construction and excellent stretch, an adjustable hood, an interior media port and mesh stash pocket and a hem drawcord. Machine wash and dry. Imported.', metadata={'source': '../data/OutdoorClothingCatalog_1000.csv', 'row': 11})"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[11]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d548aef",
   "metadata": {},
   "source": [
    "#### Hard-coded examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "106fbd91-f7bc-4d6b-b090-54b6a485ce39",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.prompts import PromptTemplate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "0c5d5af6-36db-4421-b635-46384e677847",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "answer='Yes, the Cozy Comfort Pullover Set is available in black, grey, and navy blue.'\n"
     ]
    }
   ],
   "source": [
    "from langchain.prompts import PromptTemplate\n",
    "from langchain.schema import BaseOutputParser\n",
    "from pydantic import BaseModel, Field\n",
    "\n",
    "examples = [\n",
    "    {\n",
    "        \"query\": \"Do the Cozy Comfort Pullover Set\\\n",
    "        have side pockets?\",\n",
    "        \"answer\": \"Yes\"\n",
    "    },\n",
    "    {\n",
    "        \"query\": \"What collection is the Ultra-Lofty \\\n",
    "        850 Stretch Down Hooded Jacket from?\",\n",
    "        \"answer\": \"The DownTek collection\"\n",
    "    }\n",
    "]\n",
    "\n",
    "# Define the prompt template\n",
    "prompt_template = PromptTemplate(\n",
    "    input_variables=[\"query\"],\n",
    "    template=\"Examples:\\n\"\n",
    "             \"1. Query: Do the Cozy Comfort Pullover Set have side pockets?\\n\"\n",
    "             \"   Answer: Yes\\n\"\n",
    "             \"2. Query: What collection is the Ultra-Lofty 850 Stretch Down Hooded Jacket from?\\n\"\n",
    "             \"   Answer: The DownTek collection\\n\"\n",
    "             \"Query: {query}\\n\"\n",
    "             \"Answer:\"\n",
    ")\n",
    "\n",
    "# Define the output model\n",
    "class Answer(BaseModel):\n",
    "    answer: str = Field(description=\"The answer to the query\")\n",
    "\n",
    "# Create the output parser\n",
    "class AnswerOutputParser(BaseOutputParser):\n",
    "    def parse(self, text: str) -> Answer:\n",
    "        # Split the response to get the answer\n",
    "        answer = text.strip().split(\"Answer:\")[-1].strip()\n",
    "        return Answer(answer=answer)\n",
    "\n",
    "# Initialize the LLM\n",
    "# llm = OpenAI()\n",
    "llm = ChatOpenAI()\n",
    "\n",
    "# Create the LLMChain\n",
    "llm_chain = LLMChain(\n",
    "    llm=llm,\n",
    "    prompt=prompt_template,\n",
    "    output_parser=AnswerOutputParser()\n",
    ")\n",
    "\n",
    "# Example query\n",
    "query = \"Is the Cozy Comfort Pullover Set available in different colors?\"\n",
    "\n",
    "# Run the chain\n",
    "result = llm_chain.run({\"query\": query})\n",
    "\n",
    "# Print the result\n",
    "print(result)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7ce3e4f",
   "metadata": {},
   "source": [
    "#### LLM-Generated examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "d44f8376",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.evaluation.qa import QAGenerateChain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "34e87816",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "example_gen_chain = QAGenerateChain.from_llm(ChatOpenAI())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "acb34772-368f-4b5e-b4bd-da9b637cc7e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "llm_chain = LLMChain(llm=llm, prompt=prompt_template)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "62abae09",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.11/site-packages/langchain/chains/llm.py:368: UserWarning: The apply_and_parse method is deprecated, instead pass an output parser directly to LLMChain.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "new_examples = example_gen_chain.apply_and_parse(\n",
    "    [{\"doc\": t} for t in data[:5]]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "97ab28b5",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'qa_pairs': {'query': \"According to the document, what is the approximate weight of the Women's Campside Oxfords per pair?\",\n",
       "  'answer': \"The approximate weight of the Women's Campside Oxfords per pair is 1 lb. 1 oz.\"}}"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_examples[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "0ebe4228",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(page_content=\": 0\\nname: Women's Campside Oxfords\\ndescription: This ultracomfortable lace-to-toe Oxford boasts a super-soft canvas, thick cushioning, and quality construction for a broken-in feel from the first time you put them on. \\n\\nSize & Fit: Order regular shoe size. For half sizes not offered, order up to next whole size. \\n\\nSpecs: Approx. weight: 1 lb.1 oz. per pair. \\n\\nConstruction: Soft canvas material for a broken-in feel and look. Comfortable EVA innersole with Cleansport NXT® antimicrobial odor control. Vintage hunt, fish and camping motif on innersole. Moderate arch contour of innersole. EVA foam midsole for cushioning and support. Chain-tread-inspired molded rubber outsole with modified chain-tread pattern. Imported. \\n\\nQuestions? Please contact us for any inquiries.\", metadata={'source': '../data/OutdoorClothingCatalog_1000.csv', 'row': 0})"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "7693fe86-feeb-4d73-b400-e66e79315274",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'query': \"According to the document, what is the approximate weight of the Women's Campside Oxfords per pair?\",\n",
       "  'answer': \"The approximate weight of the Women's Campside Oxfords per pair is 1 lb. 1 oz.\"},\n",
       " {'query': 'What are the dimensions of the small and medium sizes of the Recycled Waterhog Dog Mat, Chevron Weave?',\n",
       "  'answer': 'The small size of the Recycled Waterhog Dog Mat, Chevron Weave has dimensions of 18\" x 28\", while the medium size has dimensions of 22.5\" x 34.5\".'},\n",
       " {'query': \"What are some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece as described in the document?\",\n",
       "  'answer': \"Some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece include bright colors, ruffles, exclusive whimsical prints, four-way-stretch and chlorine-resistant fabric, UPF 50+ rated fabric for sun protection, crossover no-slip straps, fully lined bottom for secure fit and maximum coverage. It is recommended to machine wash and line dry for best results.\"},\n",
       " {'query': 'What is the fabric composition of the Refresh Swimwear V-Neck Tankini Contrasts top?',\n",
       "  'answer': 'The body of the tankini top is made of 82% recycled nylon and 18% Lycra® spandex, while the lining is made of 90% recycled nylon and 10% Lycra® spandex.'},\n",
       " {'query': 'What technology does the EcoFlex 3L Storm Pants feature that makes them more breathable than previous versions?',\n",
       "  'answer': 'The EcoFlex 3L Storm Pants feature TEK O2 technology, which offers the most breathability ever tested by the company.'}]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d_flattened = [data['qa_pairs'] for data in new_examples]\n",
    "d_flattened"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "faf25f2f",
   "metadata": {},
   "source": [
    "#### Combine examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "ada2a3fc",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# examples += new_example\n",
    "examples += d_flattened"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "2184b9d7-22ab-43a5-9ba5-b27fef024874",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'query': 'Do the Cozy Comfort Pullover Set        have side pockets?',\n",
       " 'answer': 'Yes'}"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "examples[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "9cdf5cf5",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'query': 'Do the Cozy Comfort Pullover Set        have side pockets?',\n",
       " 'result': 'Yes, the Cozy Comfort Pullover Set has side pockets.'}"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qa.invoke(examples[0][\"query\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63f3cb08",
   "metadata": {},
   "source": [
    "### Manual Evaluation - Fun part"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "fcaf622e",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import langchain\n",
    "langchain.debug = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "8a142638",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32;1m\u001b[1;3m[chain/start]\u001b[0m \u001b[1m[chain:RetrievalQA] Entering Chain run with input:\n",
      "\u001b[0m{\n",
      "  \"query\": \"Do the Cozy Comfort Pullover Set        have side pockets?\"\n",
      "}\n",
      "\u001b[32;1m\u001b[1;3m[chain/start]\u001b[0m \u001b[1m[chain:RetrievalQA > chain:StuffDocumentsChain] Entering Chain run with input:\n",
      "\u001b[0m[inputs]\n",
      "\u001b[32;1m\u001b[1;3m[chain/start]\u001b[0m \u001b[1m[chain:RetrievalQA > chain:StuffDocumentsChain > chain:LLMChain] Entering Chain run with input:\n",
      "\u001b[0m{\n",
      "  \"question\": \"Do the Cozy Comfort Pullover Set        have side pockets?\",\n",
      "  \"context\": \": 73\\nname: Cozy Cuddles Knit Pullover Set\\ndescription: Perfect for lounging, this knit set lives up to its name. We used ultrasoft fabric and an easy design that's as comfortable at bedtime as it is when we have to make a quick run out. \\n\\nSize & Fit \\nPants are Favorite Fit: Sits lower on the waist. \\nRelaxed Fit: Our most generous fit sits farthest from the body. \\n\\nFabric & Care \\nIn the softest blend of 63% polyester, 35% rayon and 2% spandex.\\n\\nAdditional Features \\nRelaxed fit top with raglan sleeves and rounded hem. \\nPull-on pants have a wide elastic waistband and drawstring, side pockets and a modern slim leg. \\nImported.<<<<>>>>>Berber fleece-lined pockets keep hands warm. One pocket has a hidden security pocket tucked inside. Imported.<<<<>>>>>: 10\\nname: Cozy Comfort Pullover Set, Stripe\\ndescription: Perfect for lounging, this striped knit set lives up to its name. We used ultrasoft fabric and an easy design that's as comfortable at bedtime as it is when we have to make a quick run out.\\n\\nSize & Fit\\n- Pants are Favorite Fit: Sits lower on the waist.\\n- Relaxed Fit: Our most generous fit sits farthest from the body.\\n\\nFabric & Care\\n- In the softest blend of 63% polyester, 35% rayon and 2% spandex.\\n\\nAdditional Features\\n- Relaxed fit top with raglan sleeves and rounded hem.\\n- Pull-on pants have a wide elastic waistband and drawstring, side pockets and a modern slim leg.\\n\\nImported.<<<<>>>>>Additional Features: Bonded construction insulates for extra warmth and won't stretch out of shape. Classic shirt jac styling, with collar and modified hem for extra coverage. Full snap placket for easy on/off. Chest pockets with snap closures. Pencil pocket on chest. Underarm gussets enhance mobility. Berber fleece-lined pockets keep hands warm. One pocket has a hidden security pocket tucked inside. Imported\"\n",
      "}\n",
      "\u001b[32;1m\u001b[1;3m[llm/start]\u001b[0m \u001b[1m[chain:RetrievalQA > chain:StuffDocumentsChain > chain:LLMChain > llm:ChatOpenAI] Entering LLM run with input:\n",
      "\u001b[0m{\n",
      "  \"prompts\": [\n",
      "    \"System: Use the following pieces of context to answer the user's question. \\nIf you don't know the answer, just say that you don't know, don't try to make up an answer.\\n----------------\\n: 73\\nname: Cozy Cuddles Knit Pullover Set\\ndescription: Perfect for lounging, this knit set lives up to its name. We used ultrasoft fabric and an easy design that's as comfortable at bedtime as it is when we have to make a quick run out. \\n\\nSize & Fit \\nPants are Favorite Fit: Sits lower on the waist. \\nRelaxed Fit: Our most generous fit sits farthest from the body. \\n\\nFabric & Care \\nIn the softest blend of 63% polyester, 35% rayon and 2% spandex.\\n\\nAdditional Features \\nRelaxed fit top with raglan sleeves and rounded hem. \\nPull-on pants have a wide elastic waistband and drawstring, side pockets and a modern slim leg. \\nImported.<<<<>>>>>Berber fleece-lined pockets keep hands warm. One pocket has a hidden security pocket tucked inside. Imported.<<<<>>>>>: 10\\nname: Cozy Comfort Pullover Set, Stripe\\ndescription: Perfect for lounging, this striped knit set lives up to its name. We used ultrasoft fabric and an easy design that's as comfortable at bedtime as it is when we have to make a quick run out.\\n\\nSize & Fit\\n- Pants are Favorite Fit: Sits lower on the waist.\\n- Relaxed Fit: Our most generous fit sits farthest from the body.\\n\\nFabric & Care\\n- In the softest blend of 63% polyester, 35% rayon and 2% spandex.\\n\\nAdditional Features\\n- Relaxed fit top with raglan sleeves and rounded hem.\\n- Pull-on pants have a wide elastic waistband and drawstring, side pockets and a modern slim leg.\\n\\nImported.<<<<>>>>>Additional Features: Bonded construction insulates for extra warmth and won't stretch out of shape. Classic shirt jac styling, with collar and modified hem for extra coverage. Full snap placket for easy on/off. Chest pockets with snap closures. Pencil pocket on chest. Underarm gussets enhance mobility. Berber fleece-lined pockets keep hands warm. One pocket has a hidden security pocket tucked inside. Imported\\nHuman: Do the Cozy Comfort Pullover Set        have side pockets?\"\n",
      "  ]\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[llm/end]\u001b[0m \u001b[1m[chain:RetrievalQA > chain:StuffDocumentsChain > chain:LLMChain > llm:ChatOpenAI] [1.27s] Exiting LLM run with output:\n",
      "\u001b[0m{\n",
      "  \"generations\": [\n",
      "    [\n",
      "      {\n",
      "        \"text\": \"Yes, the Cozy Comfort Pullover Set has side pockets.\",\n",
      "        \"generation_info\": {\n",
      "          \"finish_reason\": \"stop\",\n",
      "          \"logprobs\": null\n",
      "        },\n",
      "        \"type\": \"ChatGeneration\",\n",
      "        \"message\": {\n",
      "          \"lc\": 1,\n",
      "          \"type\": \"constructor\",\n",
      "          \"id\": [\n",
      "            \"langchain\",\n",
      "            \"schema\",\n",
      "            \"messages\",\n",
      "            \"AIMessage\"\n",
      "          ],\n",
      "          \"kwargs\": {\n",
      "            \"content\": \"Yes, the Cozy Comfort Pullover Set has side pockets.\",\n",
      "            \"response_metadata\": {\n",
      "              \"token_usage\": {\n",
      "                \"completion_tokens\": 13,\n",
      "                \"prompt_tokens\": 504,\n",
      "                \"total_tokens\": 517\n",
      "              },\n",
      "              \"model_name\": \"gpt-3.5-turbo\",\n",
      "              \"system_fingerprint\": null,\n",
      "              \"finish_reason\": \"stop\",\n",
      "              \"logprobs\": null\n",
      "            },\n",
      "            \"type\": \"ai\",\n",
      "            \"id\": \"run-a161a758-1627-4ee4-a205-3a12f334bd82-0\",\n",
      "            \"tool_calls\": [],\n",
      "            \"invalid_tool_calls\": []\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    ]\n",
      "  ],\n",
      "  \"llm_output\": {\n",
      "    \"token_usage\": {\n",
      "      \"completion_tokens\": 13,\n",
      "      \"prompt_tokens\": 504,\n",
      "      \"total_tokens\": 517\n",
      "    },\n",
      "    \"model_name\": \"gpt-3.5-turbo\",\n",
      "    \"system_fingerprint\": null\n",
      "  },\n",
      "  \"run\": null\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[chain/end]\u001b[0m \u001b[1m[chain:RetrievalQA > chain:StuffDocumentsChain > chain:LLMChain] [1.27s] Exiting Chain run with output:\n",
      "\u001b[0m{\n",
      "  \"text\": \"Yes, the Cozy Comfort Pullover Set has side pockets.\"\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[chain/end]\u001b[0m \u001b[1m[chain:RetrievalQA > chain:StuffDocumentsChain] [1.27s] Exiting Chain run with output:\n",
      "\u001b[0m{\n",
      "  \"output_text\": \"Yes, the Cozy Comfort Pullover Set has side pockets.\"\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[chain/end]\u001b[0m \u001b[1m[chain:RetrievalQA] [1.30s] Exiting Chain run with output:\n",
      "\u001b[0m{\n",
      "  \"result\": \"Yes, the Cozy Comfort Pullover Set has side pockets.\"\n",
      "}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'query': 'Do the Cozy Comfort Pullover Set        have side pockets?',\n",
       " 'result': 'Yes, the Cozy Comfort Pullover Set has side pockets.'}"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qa.invoke(examples[0][\"query\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "b3d6bef0",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Turn off the debug mode\n",
    "langchain.debug = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5bdbdce",
   "metadata": {},
   "source": [
    "### LLM assisted evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "id": "a54769b0-3daf-4cac-b259-89a10dd9b5a2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "examples += d_flattened"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "8ea95385-1b4c-440a-9fea-8500b4cc2154",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'query': 'Do the Cozy Comfort Pullover Set        have side pockets?',\n",
       "  'answer': 'Yes'},\n",
       " {'query': 'What collection is the Ultra-Lofty         850 Stretch Down Hooded Jacket from?',\n",
       "  'answer': 'The DownTek collection'},\n",
       " {'query': \"According to the document, what is the approximate weight of the Women's Campside Oxfords per pair?\",\n",
       "  'answer': \"The approximate weight of the Women's Campside Oxfords per pair is 1 lb. 1 oz.\"},\n",
       " {'query': 'What are the dimensions of the small and medium sizes of the Recycled Waterhog Dog Mat, Chevron Weave?',\n",
       "  'answer': 'The small size of the Recycled Waterhog Dog Mat, Chevron Weave has dimensions of 18\" x 28\", while the medium size has dimensions of 22.5\" x 34.5\".'},\n",
       " {'query': \"What are some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece as described in the document?\",\n",
       "  'answer': \"Some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece include bright colors, ruffles, exclusive whimsical prints, four-way-stretch and chlorine-resistant fabric, UPF 50+ rated fabric for sun protection, crossover no-slip straps, fully lined bottom for secure fit and maximum coverage. It is recommended to machine wash and line dry for best results.\"},\n",
       " {'query': 'What is the fabric composition of the Refresh Swimwear V-Neck Tankini Contrasts top?',\n",
       "  'answer': 'The body of the tankini top is made of 82% recycled nylon and 18% Lycra® spandex, while the lining is made of 90% recycled nylon and 10% Lycra® spandex.'},\n",
       " {'query': 'What technology does the EcoFlex 3L Storm Pants feature that makes them more breathable than previous versions?',\n",
       "  'answer': 'The EcoFlex 3L Storm Pants feature TEK O2 technology, which offers the most breathability ever tested by the company.'}]"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "a4dca05a",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "predictions = qa.batch(examples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "ae7e8b4e-4468-4048-8544-c9936704ea93",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'query': 'Do the Cozy Comfort Pullover Set        have side pockets?',\n",
       "  'answer': 'Yes',\n",
       "  'result': 'Yes, the Cozy Comfort Pullover Set does have side pockets.'},\n",
       " {'query': 'What collection is the Ultra-Lofty         850 Stretch Down Hooded Jacket from?',\n",
       "  'answer': 'The DownTek collection',\n",
       "  'result': 'The Ultra-Lofty 850 Stretch Down Hooded Jacket is from the DownTek collection.'},\n",
       " {'query': \"According to the document, what is the approximate weight of the Women's Campside Oxfords per pair?\",\n",
       "  'answer': \"The approximate weight of the Women's Campside Oxfords per pair is 1 lb. 1 oz.\",\n",
       "  'result': \"The approximate weight of the Women's Campside Oxfords per pair is 1 lb. 1 oz.\"},\n",
       " {'query': 'What are the dimensions of the small and medium sizes of the Recycled Waterhog Dog Mat, Chevron Weave?',\n",
       "  'answer': 'The small size of the Recycled Waterhog Dog Mat, Chevron Weave has dimensions of 18\" x 28\", while the medium size has dimensions of 22.5\" x 34.5\".',\n",
       "  'result': 'The dimensions of the small size of the Recycled Waterhog Dog Mat, Chevron Weave are 18\" x 28\", and the dimensions of the medium size are 22.5\" x 34.5\".'},\n",
       " {'query': \"What are some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece as described in the document?\",\n",
       "  'answer': \"Some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece include bright colors, ruffles, exclusive whimsical prints, four-way-stretch and chlorine-resistant fabric, UPF 50+ rated fabric for sun protection, crossover no-slip straps, fully lined bottom for secure fit and maximum coverage. It is recommended to machine wash and line dry for best results.\",\n",
       "  'result': \"Some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece are:\\n- Bright colors, ruffles, and exclusive whimsical prints\\n- Four-way-stretch and chlorine-resistant fabric\\n- UPF 50+ rated fabric for high sun protection\\n- Crossover no-slip straps and fully lined bottom for a secure fit and coverage\\n- Machine washable and line dry for best results\"},\n",
       " {'query': 'What is the fabric composition of the Refresh Swimwear V-Neck Tankini Contrasts top?',\n",
       "  'answer': 'The body of the tankini top is made of 82% recycled nylon and 18% Lycra® spandex, while the lining is made of 90% recycled nylon and 10% Lycra® spandex.',\n",
       "  'result': 'The fabric composition of the Refresh Swimwear V-Neck Tankini Contrasts top is as follows:\\n- Body: 82% recycled nylon, 18% Lycra® spandex\\n- Lining: 90% recycled nylon, 10% Lycra® spandex'},\n",
       " {'query': 'What technology does the EcoFlex 3L Storm Pants feature that makes them more breathable than previous versions?',\n",
       "  'answer': 'The EcoFlex 3L Storm Pants feature TEK O2 technology, which offers the most breathability ever tested by the company.',\n",
       "  'result': 'The EcoFlex 3L Storm Pants feature TEK O2 technology, which offers the most breathability tested by the brand.'}]"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "6012a3e0",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.evaluation.qa import QAEvalChain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "724b1c0b",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "llm = ChatOpenAI(temperature=0)\n",
    "eval_chain = QAEvalChain.from_llm(llm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "8b46ae55",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "graded_outputs = eval_chain.evaluate(examples, predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "dc42eb35-c2d7-4581-8004-d315ade63eef",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'results': 'CORRECT'},\n",
       " {'results': 'CORRECT'},\n",
       " {'results': 'CORRECT'},\n",
       " {'results': 'CORRECT'},\n",
       " {'results': 'CORRECT'},\n",
       " {'results': 'CORRECT'},\n",
       " {'results': 'CORRECT'}]"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "graded_outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "3437cfbe",
   "metadata": {
    "height": 149,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example 0:\n",
      "Question: Do the Cozy Comfort Pullover Set        have side pockets?\n",
      "Real Answer: Yes\n",
      "Predicted Answer: Yes, the Cozy Comfort Pullover Set does have side pockets.\n",
      "\n",
      "Example 1:\n",
      "Question: What collection is the Ultra-Lofty         850 Stretch Down Hooded Jacket from?\n",
      "Real Answer: The DownTek collection\n",
      "Predicted Answer: The Ultra-Lofty 850 Stretch Down Hooded Jacket is from the DownTek collection.\n",
      "\n",
      "Example 2:\n",
      "Question: According to the document, what is the approximate weight of the Women's Campside Oxfords per pair?\n",
      "Real Answer: The approximate weight of the Women's Campside Oxfords per pair is 1 lb. 1 oz.\n",
      "Predicted Answer: The approximate weight of the Women's Campside Oxfords per pair is 1 lb. 1 oz.\n",
      "\n",
      "Example 3:\n",
      "Question: What are the dimensions of the small and medium sizes of the Recycled Waterhog Dog Mat, Chevron Weave?\n",
      "Real Answer: The small size of the Recycled Waterhog Dog Mat, Chevron Weave has dimensions of 18\" x 28\", while the medium size has dimensions of 22.5\" x 34.5\".\n",
      "Predicted Answer: The dimensions of the small size of the Recycled Waterhog Dog Mat, Chevron Weave are 18\" x 28\", and the dimensions of the medium size are 22.5\" x 34.5\".\n",
      "\n",
      "Example 4:\n",
      "Question: What are some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece as described in the document?\n",
      "Real Answer: Some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece include bright colors, ruffles, exclusive whimsical prints, four-way-stretch and chlorine-resistant fabric, UPF 50+ rated fabric for sun protection, crossover no-slip straps, fully lined bottom for secure fit and maximum coverage. It is recommended to machine wash and line dry for best results.\n",
      "Predicted Answer: Some key features of the Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece are:\n",
      "- Bright colors, ruffles, and exclusive whimsical prints\n",
      "- Four-way-stretch and chlorine-resistant fabric\n",
      "- UPF 50+ rated fabric for high sun protection\n",
      "- Crossover no-slip straps and fully lined bottom for a secure fit and coverage\n",
      "- Machine washable and line dry for best results\n",
      "\n",
      "Example 5:\n",
      "Question: What is the fabric composition of the Refresh Swimwear V-Neck Tankini Contrasts top?\n",
      "Real Answer: The body of the tankini top is made of 82% recycled nylon and 18% Lycra® spandex, while the lining is made of 90% recycled nylon and 10% Lycra® spandex.\n",
      "Predicted Answer: The fabric composition of the Refresh Swimwear V-Neck Tankini Contrasts top is as follows:\n",
      "- Body: 82% recycled nylon, 18% Lycra® spandex\n",
      "- Lining: 90% recycled nylon, 10% Lycra® spandex\n",
      "\n",
      "Example 6:\n",
      "Question: What technology does the EcoFlex 3L Storm Pants feature that makes them more breathable than previous versions?\n",
      "Real Answer: The EcoFlex 3L Storm Pants feature TEK O2 technology, which offers the most breathability ever tested by the company.\n",
      "Predicted Answer: The EcoFlex 3L Storm Pants feature TEK O2 technology, which offers the most breathability tested by the brand.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i, eg in enumerate(examples):\n",
    "    print(f\"Example {i}:\")\n",
    "    print(\"Question: \" + predictions[i]['query'])\n",
    "    print(\"Real Answer: \" + predictions[i]['answer'])\n",
    "    print(\"Predicted Answer: \" + predictions[i]['result'])\n",
    "    # print(\"Predicted Grade: \" + graded_outputs[i]['text'])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "721d127a-a9e3-465d-a8ae-0e2c4b4a2659",
   "metadata": {},
   "source": [
    "### Example 2\n",
    "One can also easily evaluate your QA chains with the metrics offered in ragas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "a5ef0493-34ff-4801-b405-69c76ce86c38",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.11/site-packages/langchain/indexes/vectorstore.py:129: UserWarning: Using InMemoryVectorStore as the default vectorstore.This memory store won't persist data. You should explicitlyspecify a vectorstore when using VectorstoreIndexCreator\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "from langchain_huggingface import HuggingFaceEmbeddings\n",
    "loader = TextLoader(\"../data/nyc_text.txt\")\n",
    "index = VectorstoreIndexCreator(embedding=HuggingFaceEmbeddings(model_name=\"all-MiniLM-L6-v2\", model_kwargs = {'device': 'mps'})).from_loaders([loader])\n",
    "\n",
    "\n",
    "llm = ChatOpenAI(temperature= 0)\n",
    "qa_chain = RetrievalQA.from_chain_type(\n",
    "    llm,\n",
    "    retriever=index.vectorstore.as_retriever(),\n",
    "    return_source_documents=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "f0449cae-de25-4ef6-ae64-78ccf5e06a5a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'New York City was originally named New Amsterdam by Dutch colonists in 1626. When the city came under British control in 1664, it was renamed New York after King Charles II of England granted the lands to his brother, the Duke of York. The city has been continuously named New York since November 1674.'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# testing it out\n",
    "\n",
    "question = \"How did New York City get its name?\"\n",
    "result = qa_chain.invoke({\"query\": question})\n",
    "result[\"result\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "9e846b3d-f79f-46eb-8075-c816268c0500",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'query': 'How did New York City get its name?',\n",
       " 'result': 'New York City was originally named New Amsterdam by Dutch colonists in 1626. When the city came under British control in 1664, it was renamed New York after King Charles II of England granted the lands to his brother, the Duke of York. The city has been continuously named New York since November 1674.',\n",
       " 'source_documents': [Document(page_content='The city and its metropolitan area constitute the premier gateway for legal immigration to the United States. As many as 800 languages are spoken in New York, making it the most linguistically diverse city in the world. New York City is home to more than 3.2 million residents born outside the U.S., the largest foreign-born population of any city in the world as of 2016.New York City traces its origins to a trading post founded on the southern tip of Manhattan Island by Dutch colonists in approximately 1624. The settlement was named New Amsterdam (Dutch: Nieuw Amsterdam) in 1626 and was chartered as a city in 1653. The city came under British control in 1664 and was renamed New York after King Charles II of England granted the lands to his brother, the Duke of York. The city was regained by the Dutch in July 1673 and was renamed New Orange for one year and three months; the city has been continuously named New York since November 1674. New York City was the capital of the United States', metadata={'source': '../data/nyc_text.txt'}),\n",
       "  Document(page_content='New York City has been a metropolitan municipality with a Strong mayor–council form of government since its consolidation in 1898. In New York City, the city government is responsible for public education, correctional institutions, public safety, recreational facilities, sanitation, water supply, and welfare services.', metadata={'source': '../data/nyc_text.txt'}),\n",
       "  Document(page_content=\"Despite New York's heavy reliance on its vast public transit system, streets are a defining feature of the city. The Commissioners' Plan of 1811 greatly influenced the city's physical development. Several of the city's streets and avenues, including Broadway, Wall Street, Madison Avenue, and Seventh Avenue are also used as metonyms for national industries there: the theater, finance, advertising, and fashion organizations, respectively.\", metadata={'source': '../data/nyc_text.txt'}),\n",
       "  Document(page_content=\"In the pre-Columbian era, the area of present-day New York City was inhabited by Algonquian Native Americans, including the Lenape. Their homeland, known as Lenapehoking, included the present-day areas of Staten Island, Manhattan, the Bronx, the western portion of Long Island (including the areas that would later become the boroughs of Brooklyn and Queens), and the Lower Hudson Valley.The first documented visit into New York Harbor by a European was in 1524 by Italian Giovanni da Verrazzano, an explorer from Florence in the service of the French crown. He claimed the area for France and named it Nouvelle Angoulême (New Angoulême). A Spanish expedition, led by the Portuguese captain Estêvão Gomes sailing for Emperor Charles V, arrived in New York Harbor in January 1525 and charted the mouth of the Hudson River, which he named Río de San Antonio ('Saint Anthony's River'). The Padrón Real of 1527, the first scientific map to show the East Coast of North America continuously, was informed\", metadata={'source': '../data/nyc_text.txt'})]}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "069d9da8-a593-4fc6-9d4b-fea2af6bdfd0",
   "metadata": {},
   "source": [
    "Now in order to evaluate the qa system we generated a few relevant questions. We've generated a few question for you but feel free to add any you want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "a2e2cade-0005-41c1-b775-c6a7175bcf3b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "eval_questions = [\n",
    "    \"What is the population of New York City as of 2020?\",\n",
    "    \"Which borough of New York City has the highest population?\",\n",
    "    \"What is the economic significance of New York City?\",\n",
    "    \"How did New York City get its name?\",\n",
    "    \"What is the significance of the Statue of Liberty in New York City?\",\n",
    "]\n",
    "\n",
    "eval_answers = [\n",
    "    \"8,804,190\",\n",
    "    \"Brooklyn\",\n",
    "    \"New York City's economic significance is vast, as it serves as the global financial capital, housing Wall Street and major financial institutions. Its diverse economy spans technology, media, healthcare, education, and more, making it resilient to economic fluctuations. NYC is a hub for international business, attracting global companies, and boasts a large, skilled labor force. Its real estate market, tourism, cultural industries, and educational institutions further fuel its economic prowess. The city's transportation network and global influence amplify its impact on the world stage, solidifying its status as a vital economic player and cultural epicenter.\",\n",
    "    \"New York City got its name when it came under British control in 1664. King Charles II of England granted the lands to his brother, the Duke of York, who named the city New York in his own honor.\",\n",
    "    \"The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants who arrived in the U.S. by ship in the late 19th and early 20th centuries, representing hope and freedom for those seeking a better life. It has since become an iconic landmark and a global symbol of cultural diversity and freedom.\",\n",
    "]\n",
    "\n",
    "examples = [\n",
    "    {\"query\": q, \"ground_truths\": [eval_answers[i]]}\n",
    "    for i, q in enumerate(eval_questions)\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "aac9358e-f8bc-4992-aea3-c83160ff0ab0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'query': 'What is the population of New York City as of 2020?',\n",
       "  'ground_truths': ['8,804,190']},\n",
       " {'query': 'Which borough of New York City has the highest population?',\n",
       "  'ground_truths': ['Brooklyn']},\n",
       " {'query': 'What is the economic significance of New York City?',\n",
       "  'ground_truths': [\"New York City's economic significance is vast, as it serves as the global financial capital, housing Wall Street and major financial institutions. Its diverse economy spans technology, media, healthcare, education, and more, making it resilient to economic fluctuations. NYC is a hub for international business, attracting global companies, and boasts a large, skilled labor force. Its real estate market, tourism, cultural industries, and educational institutions further fuel its economic prowess. The city's transportation network and global influence amplify its impact on the world stage, solidifying its status as a vital economic player and cultural epicenter.\"]},\n",
       " {'query': 'How did New York City get its name?',\n",
       "  'ground_truths': ['New York City got its name when it came under British control in 1664. King Charles II of England granted the lands to his brother, the Duke of York, who named the city New York in his own honor.']},\n",
       " {'query': 'What is the significance of the Statue of Liberty in New York City?',\n",
       "  'ground_truths': ['The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants who arrived in the U.S. by ship in the late 19th and early 20th centuries, representing hope and freedom for those seeking a better life. It has since become an iconic landmark and a global symbol of cultural diversity and freedom.']}]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "examples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a21efe8-7c30-449a-9b8e-5b79778e305b",
   "metadata": {},
   "source": [
    "#### Introducing RagasEvaluatorChain"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "139c2214-a6eb-4d4f-9403-7e1574b97a36",
   "metadata": {},
   "source": [
    "`RagasEvaluatorChain` creates a wrapper around the metrics ragas provides (documented [here](https://github.com/explodinggradients/ragas/blob/main/docs/metrics.md)), making it easier to run these evaluation with langchain and langsmith.\n",
    "\n",
    "The evaluator chain has the following APIs\n",
    "\n",
    "- `__call__()`: call the `RagasEvaluatorChain` directly on the result of a QA chain.\n",
    "- `evaluate()`: evaluate on a list of examples (with the input queries) and predictions (outputs from the QA chain). \n",
    "- `evaluate_run()`: method implemented that is called by langsmith evaluators to evaluate langsmith datasets.\n",
    "\n",
    "lets see each of them in action to learn more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "022c8aae-fe5f-4274-b638-9209151b9491",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Manhattan (New York County) has the highest population density of any borough in New York City.'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = qa_chain.invoke({\"query\": eval_questions[1]})\n",
    "result[\"result\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "eae31c80-42c9-4b1f-95b3-c05ceadb103f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "key_mapping = {\n",
    "    \"query\": \"question\",\n",
    "    \"result\": \"answer\",\n",
    "    \"source_documents\": \"contexts\"\n",
    "}\n",
    "\n",
    "result_updated = {}\n",
    "for old_key, new_key in key_mapping.items():\n",
    "    if old_key in result:\n",
    "        result_updated[new_key] = result[old_key]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "ecd4dd9f-16d7-43d4-ac8e-6c5aa5e3f7b0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'question': 'Which borough of New York City has the highest population?',\n",
       " 'answer': 'Manhattan (New York County) has the highest population density of any borough in New York City.',\n",
       " 'contexts': [Document(page_content=\"New York City is the most populous city in the United States, with 8,804,190 residents incorporating more immigration into the city than outmigration since the 2010 United States census. More than twice as many people live in New York City as compared to Los Angeles, the second-most populous U.S. city; and New York has more than three times the population of Chicago, the third-most populous U.S. city. New York City gained more residents between 2010 and 2020 (629,000) than any other U.S. city, and a greater amount than the total sum of the gains over the same decade of the next four largest U.S. cities, Los Angeles, Chicago, Houston, and Phoenix, Arizona combined. New York City's population is about 44% of New York State's population, and about 39% of the population of the New York metropolitan area. The majority of New York City residents in 2020 (5,141,538, or 58.4%) were living on Long Island, in Brooklyn, or in Queens. The New York City metropolitan statistical area, has the\", metadata={'source': '../data/nyc_text.txt'}),\n",
       "  Document(page_content=\"New York, often called New York City or NYC, is the most populous city in the United States. With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km2), New York City is the most densely populated major city in the United States and more than twice as populous as Los Angeles, the nation's second-largest city. New York City is located at the southern tip of New York State. It constitutes the geographical and demographic center of both the Northeast megalopolis and the New York metropolitan area, the largest metropolitan area in the U.S. by both population and urban area. With over 20.1 million people in its metropolitan statistical area and 23.5 million in its combined statistical area as of 2020, New York is one of the world's most populous megacities, and over 58 million people live within 250 mi (400 km) of the city. New York City is a global cultural, financial, entertainment, and media center with a significant influence on commerce, health care and life\", metadata={'source': '../data/nyc_text.txt'}),\n",
       "  Document(page_content=\"Manhattan (New York County) is the geographically smallest and most densely populated borough. It is home to Central Park and most of the city's skyscrapers, and is sometimes locally known as The City. Manhattan's population density of 72,033 people per square mile (27,812/km2) in 2015 makes it the highest of any county in the United States and higher than the density of any individual American city.Manhattan is the cultural, administrative, and financial center of New York City and contains the headquarters of many major multinational corporations, the United Nations headquarters, Wall Street, and a number of important universities. The borough of Manhattan is often described as the financial and cultural center of the world.Most of the borough is situated on Manhattan Island, at the mouth of the Hudson River and the East River, and its southern tip, at the confluence of the two rivers, represents the birthplace of New York City itself. Several small islands also compose part of the\", metadata={'source': '../data/nyc_text.txt'}),\n",
       "  Document(page_content=\"=== Population density ===\\n\\nIn 2020, the city had an estimated population density of 29,302.37 inhabitants per square mile (11,313.71/km2), rendering it the nation's most densely populated of all larger municipalities (those with more than 100,000 residents), with several small cities (of fewer than 100,000) in adjacent Hudson County, New Jersey having greater density, as per the 2010 census. Geographically co-extensive with New York County, the borough of Manhattan's 2017 population density of 72,918 inhabitants per square mile (28,154/km2) makes it the highest of any county in the United States and higher than the density of any individual American city. The next three densest counties in the United States, placing second through fourth, are also New York boroughs: Brooklyn, the Bronx, and Queens respectively.\\n\\n\\n=== Race and ethnicity ===\", metadata={'source': '../data/nyc_text.txt'})]}"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_updated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "00d5cc30-d25e-41bd-8695-9246b73938bc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# !pip install --no-cache-dir recordclass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "id": "775d55d9-437d-40c4-bb5f-87c7b65fa567",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# !pip install ragas==0.1.9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "29b14c07-bf6c-4e86-ad1a-2a6c4d1a509d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from ragas.integrations.langchain import EvaluatorChain \n",
    "# from ragas import evaluate\n",
    "from ragas.metrics import (\n",
    "    faithfulness,\n",
    "    answer_relevancy,\n",
    "    context_relevancy,\n",
    "    context_recall,\n",
    ")\n",
    "\n",
    "# create evaluation chains\n",
    "faithfulness_chain   = EvaluatorChain(metric=faithfulness)\n",
    "answer_rel_chain     = EvaluatorChain(metric=answer_relevancy)\n",
    "context_rel_chain    = EvaluatorChain(metric=context_relevancy)\n",
    "context_recall_chain = EvaluatorChain(metric=context_recall)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41a8b636-d738-41bd-ac68-21cce2c4b720",
   "metadata": {},
   "source": [
    "1. `__call__()`\n",
    "\n",
    "Directly run the evaluation chain with the results from the QA chain. Do note that metrics like context_relevancy and faithfulness require the `source_documents` to be present."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7815671c-1fc8-46ba-8356-4a0bd5558530",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Recheck the result that we are going to validate.\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cfdd07c-956c-458f-8844-e056f29e3c83",
   "metadata": {},
   "source": [
    "**Faithfulness**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "304f5f0f-a237-4584-becb-a35607caf26b",
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_result = faithfulness_chain(result_updated)\n",
    "eval_result[\"faithfulness_score\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fbd78a9-d7c5-42a0-8705-4c544ec6408e",
   "metadata": {},
   "source": [
    "High faithfulness_score means that there are exact consistency between the source documents and the answer.\n",
    "\n",
    "You can check lower faithfulness scores by changing the result (answer from LLM) or source_documents to something else."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15f42b97-a84f-4015-9da5-fa3b0b703c24",
   "metadata": {},
   "outputs": [],
   "source": [
    "fake_result = result.copy()\n",
    "fake_result[\"result\"] = \"we are the champions\"\n",
    "eval_result = faithfulness_chain(fake_result)\n",
    "eval_result[\"faithfulness_score\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02348380-159a-4578-a95e-493cdcfcebe7",
   "metadata": {},
   "source": [
    "**Context Relevancy**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fd2fb9e-19d1-4cf6-9773-897546c2d6bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_result = context_recall_chain(result)\n",
    "eval_result[\"context_recall_score\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "118cba80-c2a3-408e-9d8d-857521cbe723",
   "metadata": {},
   "source": [
    "High context_recall_score means that the ground truth is present in the source documents.\n",
    "\n",
    "You can check lower context recall scores by changing the source_documents to something else."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6610c5f-5f8e-406c-885c-093012a5dc44",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.schema import Document\n",
    "fake_result = result.copy()\n",
    "fake_result[\"source_documents\"] = [Document(page_content=\"I love christmas\")]\n",
    "eval_result = context_recall_chain(fake_result)\n",
    "eval_result[\"context_recall_score\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dc06403-c514-47fd-ac82-d95ece8d2f06",
   "metadata": {},
   "source": [
    "2. `evaluate()`\n",
    "\n",
    "Evaluate a list of inputs/queries and the outputs/predictions from the QA chain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6b23708-b94d-4649-acd9-0e268f72f94a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# run the queries as a batch for efficiency\n",
    "predictions = qa_chain.batch(examples)\n",
    "\n",
    "# evaluate\n",
    "print(\"evaluating...\")\n",
    "r = faithfulness_chain.evaluate(examples, predictions)\n",
    "r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d661f6b9-04c7-40b7-874b-25f53cfab9d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# evaluate context recall\n",
    "print(\"evaluating...\")\n",
    "r = context_recall_chain.evaluate(examples, predictions)\n",
    "r"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}