-
Notifications
You must be signed in to change notification settings - Fork 518
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HDDS-7003. Tweak read-replicas tool to be compatible with EC replication type #7528
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for working on this @len548, this is a nice improvement to the read-replicas
tool! Please take a look at my suggestions.
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
Show resolved
Hide resolved
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/dist/src/main/smoketest/debug/ozone-debug-tests-ec3-2.robot
Show resolved
Hide resolved
Thank you for reveiws, @weimingdiit and @dombizita. I hope my comments answer your questions. |
@len548 Please resolve the conflicts, otherwise it looks good to me! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for working on this one @len548 it looks good, let's get this in if CI goes through green.
Thanks @len548 for the patch, @dombizita, @fapifta, @peterxcli, @weimingdiit for the review. |
What changes were proposed in this pull request?
The ozone debug read-replicas command creates a directory with downloaded replicas of a given key and a manifest file in it. Currently in case of an EC key, it shows an exception message in the manifest file.
The cause of this issue comes from the getKeysEveryReplica method in RPCClient class. In this method createInputStream method creates an input stream through BlockInputStreamFactory. BlockInputStreamFactory creates a different input stream based on replication type. If the replication type is EC, it creates ECBlockInputStream, otherwise BlockInputStream is created. The error occurs when ECBlockInputStream is read. To download EC replicas, alternatively we can use BlockInputStream instead of ECBlockInputStream. That is why this patch handles this issue by setting the replication type to RATIS ONE even if the key is EC replicated so that BlockInputStream is used instead of ECBlockInputStream.
Also the length of blocks of an EC key has to be calculated properly in getKeysEveryReplica method, this patch introduces internalBlockLength method to determine the length of each block.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7003
How was this patch tested?
Added robot tests and ran them successfully.