Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: [MR-353] Asynchronously mark checkpoint readonly and sync #3682

Merged
merged 18 commits into from
Feb 7, 2025

Conversation

ShuoWangNSL
Copy link
Contributor

@ShuoWangNSL ShuoWangNSL commented Jan 30, 2025

Currently, during checkpointing, we block until we sync the entire checkpoint and have it moved to the checkpoints folder.
This step can be done asynchronously.
Moreover, due to the asynchronously writing of pbuf files, this step should be moved after that and right before removing the marker for unverified checkpoint.

@ShuoWangNSL ShuoWangNSL force-pushed the shuo/async_ReadOnly_syncfs branch from 28432d3 to 756144d Compare January 30, 2025 09:22
rs/state_layout/src/state_layout.rs Outdated Show resolved Hide resolved
rs/state_layout/src/state_layout.rs Outdated Show resolved Hide resolved
@ShuoWangNSL ShuoWangNSL force-pushed the shuo/async_ReadOnly_syncfs branch from 756144d to c6bec38 Compare February 3, 2025 19:31
@ShuoWangNSL ShuoWangNSL changed the title Async MarkCheckpointReadOnlyAndSync feat: [MR-353] Asynchronously mark checkpoint readonly and sync Feb 5, 2025
@github-actions github-actions bot added the feat label Feb 5, 2025
@ShuoWangNSL ShuoWangNSL force-pushed the shuo/async_ReadOnly_syncfs branch from 2eaf361 to f014ac4 Compare February 6, 2025 00:27
@ShuoWangNSL ShuoWangNSL marked this pull request as ready for review February 6, 2025 01:06
@ShuoWangNSL ShuoWangNSL requested a review from a team as a code owner February 6, 2025 01:06
Copy link
Member

@schneiderstefan schneiderstefan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea to bundle it with validation.

rs/state_layout/src/state_layout.rs Outdated Show resolved Hide resolved
@ShuoWangNSL ShuoWangNSL enabled auto-merge February 7, 2025 14:28
@ShuoWangNSL ShuoWangNSL added this pull request to the merge queue Feb 7, 2025
Merged via the queue into master with commit 6c5d194 Feb 7, 2025
24 checks passed
@ShuoWangNSL ShuoWangNSL deleted the shuo/async_ReadOnly_syncfs branch February 7, 2025 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants