Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Backfill S3 Events Data Lake #98

Open
bshien opened this issue Nov 5, 2024 · 0 comments
Open

[FEATURE] Backfill S3 Events Data Lake #98

bshien opened this issue Nov 5, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@bshien
Copy link
Collaborator

bshien commented Nov 5, 2024

Is your feature request related to a problem?

Coming from #76

If the automation app docker that listens to and stores GitHub Events to an S3 Bucket is down, then we need to backfill the events that were not stored.

What solution would you like?

A lambda that uses this API to get the missing events and store them into S3. The API has a limitation that it can only get the past 3 days of events.

There are two API endpoints we can use to backfill the s3 events.

List Deliveries
Get Deliveries

We can first use List Deliveries to list the delivery ids of the last 3 days worth of events. Then, using these delivery ids query Get Deliveries for the payload, taking care that the structure of the json uploaded to s3 is compatible.

Do you have any additional context?

Part of #57

@bshien bshien added enhancement New feature or request untriaged Issues that have not yet been triaged labels Nov 5, 2024
@bshien bshien self-assigned this Nov 5, 2024
@bshien bshien changed the title [FEATURE] Backfill S3 Events [FEATURE] Backfill S3 Events Data Lake Nov 5, 2024
@bshien bshien removed the untriaged Issues that have not yet been triaged label Nov 5, 2024
@bshien bshien removed their assignment Nov 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: 🆕 New
Development

No branches or pull requests

1 participant