Load files from a GitHub repo

[Nneji123]

Is it possible to use the document loader and load all files in a GitHub repo? Like for example the Flet(Flutter for python) repo I'd like to clone it and then feed everything in that repo to the document loader

[sergerdn]

Hi, @Nneji123

I believe that many people intend to gather information about the necessary libraries and create a customized version of ChatGPT with up-to-date documentation, instead of relying on the outdated default version.
However, I believe that this task is very complex.

[Nneji123]

Yes even when I tried it on my end I couldn't read all the files in repo. Hopefully there'll be new updates on Loading repos directly

[sergerdn]

What did you do and what happened? Could you share your code?

git clone "some repo" some_repo
tree some_repo

[Nneji123]

I don't have the code anymore(I was using a colab notebook) and I tried using the document loader to read all filetypes in a directory. I wish I could share the code and the errors I got but unfortunately I can't

[sergerdn]

I hope I can assist you once you post your proof of concept code.

NOT tested:

import os
from git import Repo

def clone_repository(repo_url, local_path):
    if not os.path.exists(local_path):
        os.makedirs(local_path)
    Repo.clone_from(repo_url, local_path)

repo_url = "https://github.com/yourusername/yourrepo.git"
local_path = "local/path/to/clone/yourrepo"

# Clone the repository
clone_repository(repo_url, local_path)

# Load files from the cloned repository
for root, _, files in os.walk(local_path):
    for file in files:
        file_path = os.path.join(root, file)
        with open(file_path, "r") as f:
            content = f.read()
            # Process the content of the file as needed, detect type of file, use properly loaders, add post process file, etc

https://github.com/gitpython-developers/GitPython