noisy-repositories.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "121fa5f3-ee8e-450d-8b21-2332e10eceae",
   "metadata": {},
   "source": [
    "# "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f891707c",
   "metadata": {},
   "source": [
    "# Dealing with Noise\n",
    "\n",
    "While there are strong data models for repository structures across the different repository providers, what people put into their repositories, and how those get reported out is a completely different beast. Directly using one of the repository APIs can generally provide us with \"clean\" repository identifers, however, when we begin to search for repository citations elsewhere (publications in particular), the results become quite a bit messier.\n",
    "\n",
    "The \"noise\" in these names comes from a number of different sources. Even simply identifying a repository can be problematic. The same repository could look very different depending on where, how and who is reporting the repository:\n",
    "\n",
    "* SimonGoring/repository-name\n",
    "* https://github.com/SimonGoring/repository-name.git\n",
    "* SimonGoring/repositoryname\n",
    "* https://github.com/SimonGoring/repository--name\n",
    "* repository-name\n",
    "* SimonGoring\n",
    "\n",
    "If we are relying on secondary sources to discover these repositories, we may get any number of these. Assuming `SimonGoring/repository-name` is a valid resource we also want to find and index the repository so we can learn a little bit about it. A survey user may respond with just `repository-name` if they've already identified theur user name in a survey. The name `SimonGoring/repository--name` may appear in a full-text document if we're not careful how punctuation marks are rendered. We may also encounter `https://github.com/SimonGoring/repository-name.git`, the canonical address of the actual `git` service.\n",
    "\n",
    "So we can think of several sources of \"noise\" in these repositories, they include noise in discovering names, noise in identifying the languages used, noise in understanding aggregate patterns and noise in interpreting user behaviours.\n",
    "\n",
    "# Noise in Repository Names (Discovery)\n",
    "\n",
    "In our tools for scraping we try a number of different ways of figuring out whether a repository name is valid. The first pass was to [use regular expressions](https://github.com/UW-Madison-DSI/OSPO_Data_Management/blob/main/xddsource/gddospo/gdd_tools.py#L9). When searching through plaintext journal articles we use the following regular expression to look for specific open, online code repositories:\n",
    "\n",
    "```regex\n",
    "((github)|(gitlab)|(bitbucket)).com\\/((\\s{0,1})[\\w,\\-,\\_]+\\/*){1,2}\n",
    "```\n",
    "\n",
    "This regular expression looks for text strings that begin with either `github`, `gitlab` or `bitbucket` followed by a `.com/`. The regex accepts either no or one space following this, but the subsequent text must contain valid characters for a repository owner, or repository name: any letter or number, and dashes or underscores.  \n",
    "\n",
    "By placing this regex into a function (see `repotest()` below), we can examine how it behaves across several different repository naming conventions, and attempt to extract the key User/Repository structure that is used throughout most code repository APIs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "83534229",
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'repo': None, 'highlight': 'SimonGoring/repository-name'},\n",
       " {'repo': 'github.com/SimonGoring/repository-name',\n",
       "  'highlight': 'https://github.com/SimonGoring/repository-name.git'},\n",
       " {'repo': None, 'highlight': 'SimonGoring/repositoryname'},\n",
       " {'repo': 'github.com/SimonGoring/repository--name',\n",
       "  'highlight': 'https://github.com/SimonGoring/repository--name'},\n",
       " {'repo': None, 'highlight': 'repository-name'},\n",
       " {'repo': None, 'highlight': 'SimonGoring'}]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "def repotest(string:str) -> dict:\n",
    "    \"\"\"Check to see if a repository is referenced in a text string.\n",
    "\n",
    "    string A character string returned from geodeepdive highlights.\n",
    "    returns dict With properties 'repo' for the repostory (if found) and 'highlight', for consistency with xDeepDive.\n",
    "    \"\"\"\n",
    "    test = re.search(r'((github)|(gitlab)|(bitbucket)).com\\/((\\s{0,1})[\\w,\\-,\\_]+\\/*){1,2}', string)\n",
    "    if test is None:\n",
    "        output = {'repo': None, 'highlight': string}\n",
    "    else:\n",
    "        test_no_space = re.sub(r'\\s', '', test[0])\n",
    "        test_no_punct = re.sub(r'[^\\w\\s]$', '', test_no_space)\n",
    "        output = {'repo': test_no_punct, 'highlight': string}\n",
    "    return output\n",
    "\n",
    "repo_list = ['SimonGoring/repository-name', 'https://github.com/SimonGoring/repository-name.git',\n",
    "             'SimonGoring/repositoryname', \n",
    "             'https://github.com/SimonGoring/repository--name',\n",
    "             'repository-name', 'SimonGoring']\n",
    "\n",
    "[repotest(i) for i in repo_list]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1ea778b-b357-42ca-aca3-64bb21906355",
   "metadata": {},
   "source": [
    "We can see in the block above that we are returning the canonical \"User/Repository\" structure from all of these strings. That helps us, but it's not enough. We need to ensure that we know the repository host (GitHub, BitBucket, etc.) and we need to ensure that the repository itself exists.\n",
    "\n",
    "## Discovery with xDeepDive\n",
    "\n",
    "If we put this into a pipeline, we might want to directly call the [xDeepDive API](https://xdd.wisc.edu/api) and test to see if the highlight returned actually contains a repository name, so we can associate that repository with the article (and article DOI):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "acdd2d85",
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'success': {'v': 1,\n",
       "  'data': [{'pubname': 'Cold Spring Harbor Laboratory',\n",
       "    'publisher': 'bioRxiv',\n",
       "    '_gddid': '601b1f3eea8bd37226c43771',\n",
       "    'title': 'CovRadar: Continuously tracking and filtering SARS-CoV-2 mutations for molecular surveillance',\n",
       "    'doi': '10.1101/2021.02.03.429146',\n",
       "    'coverDate': '2021-02-03',\n",
       "    'URL': 'https://www.biorxiv.org/content/10.1101/2021.02.03.429146v1',\n",
       "    'authors': 'Alice Wittig; Fábio Miranda; Ming Tang; Martin Hölzer; Bernhard Y Renard; Stephan Fuchs',\n",
       "    'hits': 2,\n",
       "    'highlight': ['://covradar.net, its open-source code is available at https://gitlab.com/dacs-hpi/covradar. Contact',\n",
       "     '. com/pallets/flask) and Dash (https://github.com/ plotly/dash) and connected to a MySQL database']}],\n",
       "  'license': 'https://creativecommons.org/licenses/by-nc/2.0/'}}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from requests import get\n",
    "from json import loads\n",
    "\n",
    "gddurl = (\"https://xdd.wisc.edu/api/v1/snippets?\"\n",
    "          + \"term=gitlab.com,bitbucket.com,github.com\"\n",
    "          + \"&clean&article_limit=1\")\n",
    "loads(get(gddurl).content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5fd22b7-542f-4e6b-9674-ed948e31da6c",
   "metadata": {},
   "source": [
    "In the above snippets we can see, first that we are getting a result, but also that we have two different results, one is a clearly formed github repository (`https://gitlab.com/dacs-hpi/covradar`) and the other, a repository that is visible to a reader, but that would otherwise fail our regex test (`https://github.com/ plotly/dash`). A challenge for us is managing spaces within a test string. In addition to these typographic errors we may also encounter misspellings, incomplete URLs, poorly contructed references to repositories (see the initial list, where the repository owner and name are mentioned, but the repository management system (e.g., GitHub) is not mentioned.\n",
    "\n",
    "You can see how the xDD pipeline works in the [OSPO GitHub repository](https://github.com/).\n",
    "\n",
    "Most of our work to date has focued on development around the GitHub API. This is, in large part because our results from repositories cited in publications indicate that GitHub makes up, by far, the majority of repositories cited:\n",
    "\n",
    "| Repository Org | Count |\n",
    "| ------- | ------ |\n",
    "| Github | 16,773 |\n",
    "| Gitlab | 160 |\n",
    "| Bitbucket | 0 |\n",
    "\n",
    "We currently add all repositories, but manage secondary workflow pathways through the individual repository organizers APIs. Once we've begun populating the OSPO Database we can begin to support other pathways for data entry and validation, including direct user submission and, potentially, human labelling from the snippets we have encountered, but before we undertake this we should have a well maintained data structure and robust computational platform.\n",
    "\n",
    "## Validating Repositories\n",
    "\n",
    "Once we get a User/Repository combination, we need to validate that it is a repository. We can do this in two steps. The first step would be to simply see if we get a server response at the requested URL. If we are intending to run this application at scale we want to ensure that this call both sends and returns a small payload:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "cd3245bf-a986-4695-b389-5630bc4e3f28",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The HEAD call uses 33 bytes and the GET call uses 383257 bytes\n"
     ]
    }
   ],
   "source": [
    "from requests import head\n",
    "from sys import getsizeof\n",
    "\n",
    "head_call = head('https://github.com/NeotomaDB/neotoma2').content\n",
    "get_call = get('https://github.com/NeotomaDB/neotoma2').content\n",
    "\n",
    "print(f\"The HEAD call uses {getsizeof(head_call)} bytes and the GET call uses {getsizeof(get_call)} bytes\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34e15c14-b355-4de2-ab55-3764ae1adc97",
   "metadata": {},
   "source": [
    "In 2023 GitHub estimated that there were at least 23 million public repositories and approximately 420 million total repositories. reducing our overall footprint is therefore good practice.\n",
    "\n",
    "If we do get a valid repository, we then wish to recover some basic information about the repository owner, the date the repository was created and normalize repository metadata across our data model to improve discovery and reduce the chances of introducing artifacts arising from data duplication or other issues. If we don't get a valid repository we have some choices.\n",
    "\n",
    "### Options\n",
    "\n",
    "1. Ignore the snippet and move on.\n",
    "2. Save the snippet to file and move on.\n",
    "3. Submit the incorrectly formatted URL to the database and indicate it is invalid.\n",
    "4. Scrape the owner's other repositories and look for a near match.\n",
    "\n",
    "### Solutions\n",
    "\n",
    "1. We may decide to simply ignore the repository for now. During further development we may come across a better method for obtaining GitHub repositories that supercedes the current implementation, so it may be better to save time now and focus on other solutions, however, such solutions may require more computational power on our part, because we would have to re-scrape _all_ repositories through the xDeepDive API, including ones we've already checked.\n",
    "2. We could ignore improperly formatted repository hits and save the `dict` objects to file. This would allow us to move forward with our work, and revisit these scripts at a later time. However, we need to ensure that storage is durable, and we don't know exactly how big the file may get.\n",
    "3. Submitting the invalid URL to the database would allow us to scrape the CrossRef metadata of the citing publication, it would allow us to use some of the data infrastructure we've built with our data model and any downstream applications, but it would mean extra complexity within the data model.\n",
    "4. This is something that could happen in addition to either point 2 or 3. In this case, we would check if the owner was a valid owner as well, and then perform some form of search within that owner's repositories.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5bebfbf-470c-48c3-9f60-2bc30f11d30c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}