DermaMNIST/DermaMNIST_Analysis/DermaMNIST-Corrected_HAM10000-duplicates-removed.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1ff67729",
   "metadata": {},
   "source": [
    "## Goal\n",
    "\n",
    "In this notebook, we take the DermaMNIST-C partitions (that were created by moving all the 'leaking' `lesion ID`s from validation and testing partitions to the training partition) and further improve them by fixing the leakage arising out of undetected duplicates in HAM10000.\n",
    "\n",
    "Therefore, we load the DermaMNIST-C metadata, and for each new duplicate pair detected in HAM10000, we check if the two images in the pair belong to different partitions. We observed that of the 18 pairs, 5 pairs had 1 image in training and 1 in testing, 2 pairs had 1 image in training and 1 in validation, and the remaining 11 pairs had both images in training.\n",
    "\n",
    "Therefore, to fix this, for the 7 'leaking' pairs, we move both images of the pair to the training partition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4d87fe3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set()\n",
    "\n",
    "from pathlib import Path\n",
    "from typing import Union"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93436262",
   "metadata": {},
   "source": [
    "## Read metadata files:\n",
    "- HAM10000 original metadata file.\n",
    "- DermaMNIST-C metadata file.\n",
    "- List of confirmed duplicate pairs in HAM10000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e363f7b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "ham10k_metadata_file: Union[str, Path] = Path(\"HAM10000_metadata.csv\")\n",
    "dermamnist_c_metadata_file: Union[str, Path] = Path(\"CSV_files/combined_metadata_corrected.csv\")\n",
    "ham10k_duplicates_file: Union[str, Path] = Path(\n",
    "    \"../HAM10000_DuplicateConfirmation/AnalysesOutputFiles/newly_discovered_duplicates.csv\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "900b92b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "ham10k_metadata: pd.DataFrame = pd.read_csv(ham10k_metadata_file, header=\"infer\")\n",
    "dermamnist_c_metadata: pd.DataFrame = pd.read_csv(dermamnist_c_metadata_file, header=\"infer\")\n",
    "ham10k_duplicates: pd.DataFrame = pd.read_csv(ham10k_duplicates_file, header=\"infer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "923b5436",
   "metadata": {},
   "source": [
    "### Assert whether the dataframes have the same number of rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a5bbe8c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert ham10k_metadata.shape[0] == dermamnist_c_metadata.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0586778",
   "metadata": {},
   "source": [
    "### Create a deep copy of the DermaMNIST-C metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "043b6454",
   "metadata": {},
   "outputs": [],
   "source": [
    "dermamnist_c_metadata_new = dermamnist_c_metadata.copy(deep=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57f364de",
   "metadata": {},
   "source": [
    "### Fix the leakage due to HAM10000 duplicates, and save the new metadata to file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5fede2a9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Duplicate pair moved to training.\n",
      "Duplicate pair moved to training.\n",
      "Duplicate pair moved to training.\n",
      "Duplicate pair moved to training.\n",
      "Duplicate pair moved to training.\n",
      "Duplicate pair moved to training.\n",
      "Duplicate pair moved to training.\n"
     ]
    }
   ],
   "source": [
    "for _, row in ham10k_duplicates.iterrows():\n",
    "    img_id1, img_id2 = (\n",
    "        row.from_img.split(\".\")[0], row.to_img.split(\".\")[0]\n",
    "    )\n",
    "    \n",
    "    split1 = dermamnist_c_metadata_new[\n",
    "        dermamnist_c_metadata_new[\"image_id\"] == img_id1\n",
    "    ].split.values[0]\n",
    "    split2 = dermamnist_c_metadata_new[\n",
    "        dermamnist_c_metadata_new[\"image_id\"] == img_id2\n",
    "    ].split.values[0]\n",
    "    \n",
    "    if split1 != split2:\n",
    "        dermamnist_c_metadata_new.loc[\n",
    "            dermamnist_c_metadata_new[\"image_id\"] == img_id1, \"split\"\n",
    "        ] = \"train\"\n",
    "        dermamnist_c_metadata_new.loc[\n",
    "            dermamnist_c_metadata_new[\"image_id\"] == img_id2, \"split\"\n",
    "        ] = \"train\"\n",
    "        print(\"Duplicate pair moved to training.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d79c014f",
   "metadata": {},
   "outputs": [],
   "source": [
    "dermamnist_c_metadata_new.to_csv(\n",
    "    Path(\"CSV_files/combined_metadata_corrected-HAM10000_corrected.csv\"), \n",
    "    index=False\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}