Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve ChecksummedBytes::extend and clarify data integrity guarantee #575

Merged
merged 4 commits into from
Oct 26, 2023

Conversation

passaro
Copy link
Contributor

@passaro passaro commented Oct 25, 2023

Description of change

ChecksummedBytes maintains a data buffer and a checksum and guarantees that only validated data can be accessed. Transformations such as split_off, extend, or slice (introduced in this change), may trigger a validation (and return an IntegrityError on failure), or propagate existing checksum(s) if possible, allowing for later validation.

This change clarifies the data integrity guarantee in the docs and optimizes the extend method to avoid re-validation when the checksums for both slices can be combined. It also avoid a redundant buffer allocation.

Also moves ChecksummedBytes and related types into a new root module.

Does this change impact existing behavior?

No.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).

ChecksummedBytes maintains a data buffer and a checksum and guarantees that only validated data can be accessed. Transformations such as `split_off`, `extend`, or `slice` (introduced in this change), may trigger a validation (and return an IntegrityError on failure), or propagate existing checksum(s) if possible, allowing for later validation.

This change clarifies the data integrity guarantee in the docs and optimizes the extend method to avoid re-validation when the checksums for both slices can be combined. It also avoid a redundant buffer allocation.

Signed-off-by: Alessandro Passaro <[email protected]>
@passaro passaro temporarily deployed to PR integration tests October 25, 2023 15:23 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests October 25, 2023 15:23 — with GitHub Actions Inactive
@passaro passaro had a problem deploying to PR integration tests October 25, 2023 15:23 — with GitHub Actions Failure
@passaro passaro temporarily deployed to PR integration tests October 25, 2023 15:23 — with GitHub Actions Inactive
@passaro
Copy link
Contributor Author

passaro commented Oct 25, 2023

For a clearer diff of the changes in ChecksummedBytes see just commit 711d37d.

@passaro passaro requested a review from jamesbornholt October 26, 2023 05:27
@passaro passaro temporarily deployed to PR integration tests October 26, 2023 10:05 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests October 26, 2023 10:05 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests October 26, 2023 10:05 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests October 26, 2023 10:05 — with GitHub Actions Inactive
dannycjones
dannycjones previously approved these changes Oct 26, 2023
Copy link
Contributor

@dannycjones dannycjones left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, one small comment on split_off tests.

Let's get a second review on extend and shrink_to_fit.

mountpoint-s3/src/checksums.rs Outdated Show resolved Hide resolved
Signed-off-by: Alessandro Passaro <[email protected]>
@passaro passaro temporarily deployed to PR integration tests October 26, 2023 17:51 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests October 26, 2023 17:51 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests October 26, 2023 17:51 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests October 26, 2023 17:51 — with GitHub Actions Inactive
@jamesbornholt jamesbornholt added this pull request to the merge queue Oct 26, 2023
Merged via the queue into awslabs:main with commit 524eac6 Oct 26, 2023
18 checks passed
@passaro passaro deleted the checksummed_bytes branch October 27, 2023 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants