Avoid get block info when positioned read for first retry #18417
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@jiacheliu3 @dbw9580 Would you like to take a look at this PR? |
jiacheliu3
approved these changes
Nov 17, 2023
Comment on lines
305
to
306
| BlockInfo blockInfo = lastException == null | ||
| ? mStatus.getBlockInfo(blockId) : mBlockStore.getInfo(blockId); |
There was a problem hiding this comment.
@dbw9580 do you think it's possible that we can use mStatus.getBlockInfo(blockId) even if lastException != null?
maobaolong
commented
Nov 17, 2023
| @@ -302,11 +302,15 @@ private int positionedReadInternal(long pos, byte[] b, int off, int len) throws | |||
| try { | |||
| // Positioned read may be called multiple times for the same block. Caching the in-stream | |||
| // allows us to avoid the block store rpc to open a new stream for each call. | |||
| BlockInfo blockInfo = lastException == null | |||
There was a problem hiding this comment.
Add a threshold of outdate time
There was a problem hiding this comment.
updateStream check outdate time
|
This looks like flaky test. |
dbw9580
reviewed
Dec 8, 2023
Co-authored-by: Bowen Ding <[email protected]>
jiacheliu3
added
the
type-performance
|
good to merge after all checks are green |
|
the new UT doesn't work? |
|
@jiacheliu3 Glad to say the ci checks are all passed, Please talk a look whether this could be merged. |
jiacheliu3
approved these changes
Jan 8, 2024
| @@ -6307,6 +6307,11 @@ public String toString() { | |||
| .setConsistencyCheckLevel(ConsistencyCheckLevel.WARN) | |||
| .setScope(Scope.CLIENT) | |||
| .build(); | |||
| public static final PropertyKey USER_FILE_IN_STREAM_STATUS_EXPIRATION_TIME = | |||
| durationBuilder(Name.USER_FILE_IN_STREAM_STATUS_EXPIRATION_TIME) | |||
| .setDefaultValue("5min") | |||
There was a problem hiding this comment.
whenever a property key is added, remember to add description:
setDescription("Specifies how long the file metadata can be cached and reused during the FileInStream. Once the specified expiration time has elapsed, the file metadata will be reloaded from the Alluxio master. The cache reduces the number of metadata requests to the Master. The default is 5 minutes.")
|
alluxio-bot, merge this please |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Reduce talk to master to improve the performance for small file scenario.
I guess this is introduced by ed53d54
Why are the changes needed?
The block info can be get from file status, so we should not ask it from master every time.
Does this PR introduce any user facing changes?
No