Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add try except for github readers #174

Merged
merged 3 commits into from
Jun 6, 2024
Merged

Add try except for github readers #174

merged 3 commits into from
Jun 6, 2024

Conversation

rwood-97
Copy link
Contributor

@rwood-97 rwood-97 commented Apr 8, 2024

This PR adds try/excepts for the llama index github readers in order to fix the error we are getting loading our data in Azure.

Fixes #157

@rwood-97
Copy link
Contributor Author

rwood-97 commented Apr 8, 2024

This works with running the container!

Annoyingly how it is working is that if ANY request fails then it just skips that part of loading data. So whats currently running has skipped The Turing Way and Hut23 repo (we should see if this is always the two causing issues) but has all the info from RDS courses, wikis, etc etc.

Ideally we should add a boolean flag in llama-index to be something like fail_on_http_error and then we can force it to continue even if it fails with one file rather than raise exception. I will make a ticket on llama index about this and mb try address it next friday in OS hack session.

@rwood-97 rwood-97 requested a review from rchan26 April 8, 2024 10:49
Copy link
Collaborator

@rchan26 rchan26 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks a lot @rwood-97 for your work on this and in the llama-index repo! looks good to me

@rchan26 rchan26 merged commit 318d2cf into main Jun 6, 2024
3 of 4 checks passed
@rchan26 rchan26 deleted the pulumi-try-except branch June 6, 2024 08:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add try/except with max number of retries for building index
2 participants