Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-7003. Tweak read-replicas tool to be compatible with EC replication type #7528

Merged
merged 12 commits into from
Feb 6, 2025

Conversation

len548
Copy link
Contributor

@len548 len548 commented Dec 4, 2024

What changes were proposed in this pull request?

The ozone debug read-replicas command creates a directory with downloaded replicas of a given key and a manifest file in it. Currently in case of an EC key, it shows an exception message in the manifest file.

The cause of this issue comes from the getKeysEveryReplica method in RPCClient class. In this method createInputStream method creates an input stream through BlockInputStreamFactory. BlockInputStreamFactory creates a different input stream based on replication type. If the replication type is EC, it creates ECBlockInputStream, otherwise BlockInputStream is created. The error occurs when ECBlockInputStream is read. To download EC replicas, alternatively we can use BlockInputStream instead of ECBlockInputStream. That is why this patch handles this issue by setting the replication type to RATIS ONE even if the key is EC replicated so that BlockInputStream is used instead of ECBlockInputStream.
Also the length of blocks of an EC key has to be calculated properly in getKeysEveryReplica method, this patch introduces internalBlockLength method to determine the length of each block.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-7003

How was this patch tested?

Added robot tests and ran them successfully.

@len548 len548 changed the title HDDS-7003. Provide ability to reconstruct EC files using read-replicas tool HDDS-7003. Tweak read-replicas tool to be compatible with EC replication type Dec 9, 2024
Copy link
Contributor

@dombizita dombizita left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this @len548, this is a nice improvement to the read-replicas tool! Please take a look at my suggestions.

@dombizita dombizita requested a review from fapifta December 11, 2024 10:18
@adoroszlai adoroszlai added the tools Tools that helps with debugging label Dec 12, 2024
@len548 len548 requested a review from dombizita December 16, 2024 12:27
@len548
Copy link
Contributor Author

len548 commented Jan 24, 2025

Thank you for reveiws, @weimingdiit and @dombizita. I hope my comments answer your questions.

@dombizita
Copy link
Contributor

@len548 Please resolve the conflicts, otherwise it looks good to me!

Copy link
Contributor

@fapifta fapifta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for working on this one @len548 it looks good, let's get this in if CI goes through green.

@adoroszlai adoroszlai merged commit 5bedec0 into apache:master Feb 6, 2025
43 checks passed
@adoroszlai
Copy link
Contributor

Thanks @len548 for the patch, @dombizita, @fapifta, @peterxcli, @weimingdiit for the review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tools Tools that helps with debugging
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants