Container logs sync enhancement #35
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yhabteab
force-pushed
the
container-logs
branch
2 times, most recently
from
7914b81 to
96e0535
Compare
lippserd
reviewed
Jul 3, 2023
| @@ -645,7 +645,7 @@ func (db *Database) query(ctx context.Context, query string, scope ...interface{ | |||
| if len(scope) == 1 && IsStruct(scope[0]) { | |||
| rows, err = db.NamedQueryContext(ctx, query, scope[0]) | |||
| } else { | |||
| rows, err = db.QueryxContext(ctx, query, scope...) | |||
| rows, err = db.QueryxContext(ctx, db.Rebind(query), scope...) | |||
There was a problem hiding this comment.
Can you explain that change please? And why it's only needed in the else branch?
There was a problem hiding this comment.
Can you explain that change please?
I just noticed that by accident. Right now there's no problem with it because the ? placeholders for mysql don't need a rebind, db.Rebind() is also a no-op for mysql driver. This is only relevant for PostgreSQL.
And why it's only needed in the else branch?
Named queries are rebound by the library itself.
lippserd
force-pushed
the
container-logs
branch
from
96e0535 to
11107f5
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces quite significant changes in how
Pod -> Container -> Container_{*}are synced. Namely, all container related, likeContainerDevices,ContainerMountsetc. are now no longer pod event driven being synced, but rather the respective containers. Specifically, this means that the containers themselves are no longer allowed to be automatically deleted cascading. Instead, the container events are now processed separately. Towards this end, this PR introduces two new sync channelsupsertPodsanddeletePods. These operate roughly like the following:main()invokes theschemav1#SyncContainers()function and passes through the two new chans, which are then polled simultaneously.For all uploaded
v1.Pods, this function goes over each of their containers and schedules/cancels the container log synchronisation depending on the container state. These jobs are currently rescheduled every 5 minutes for each individual container and are allowed to run a maximum of60jobs at a time. These numbers are randomly chosen by me and will certainly change or become configurable in the future.However, for all deleted
v1.Pods, the flow control looks a bit different. Since we are just passing through the entity checksums for all the deletion streams, we first need to load all the containers for each streamed pod ID from the database. This process is done concurrently for each pod ID, and thereafter the containers are passed to the next stage where the deletion stream is triggered for them. This deletion process first removes all container references such asContainer_{device, mounts, logs}from the database before deleting the container itself.