Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support use default block size to replace the Hdfs ufs #14324

Open
wants to merge 4 commits into
base: master-2.x
Choose a base branch
from
Open

Support use default block size to replace the Hdfs ufs #14324

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e1e2060
Support use default block size to replace the HdfsUnderFileSystem blo…
maobaolong Oct 28, 2021
06c39f9
Move the customize block size logic to high level
maobaolong Dec 22, 2021
5584b61
Update core/common/src/main/java/alluxio/conf/PropertyKey.java
maobaolong Jun 12, 2024
5e8583d
Update core/common/src/main/java/alluxio/conf/PropertyKey.java
maobaolong Jun 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions core/common/src/main/java/alluxio/conf/PropertyKey.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3927,6 +3927,14 @@ public String toString() {
.setDescription("Preferred media type while storing file's blocks.")
.setScope(Scope.CLIENT)
.build();

public static final PropertyKey USER_BLOCK_SIZE_ENABLED =
maobaolong marked this conversation as resolved.
Show resolved Hide resolved
new Builder(Name.USER_BLOCK_SIZE_ENABLED)
.setDefaultValue(false)
.setDescription("Use Default block size for Alluxio files.")
maobaolong marked this conversation as resolved.
Show resolved Hide resolved
.setConsistencyCheckLevel(ConsistencyCheckLevel.WARN)
.setScope(Scope.ALL)
.build();
public static final PropertyKey USER_BLOCK_SIZE_BYTES_DEFAULT =
new Builder(Name.USER_BLOCK_SIZE_BYTES_DEFAULT)
.setDefaultValue("64MB")
Expand Down Expand Up @@ -6281,6 +6289,8 @@ public static final class Name {
"alluxio.user.block.read.metrics.enabled";
public static final String USER_BLOCK_REMOTE_READ_BUFFER_SIZE_BYTES =
"alluxio.user.block.remote.read.buffer.size.bytes";
public static final String USER_BLOCK_SIZE_ENABLED =
"alluxio.user.block.size.enabled";
public static final String USER_BLOCK_SIZE_BYTES_DEFAULT =
"alluxio.user.block.size.bytes.default";
public static final String USER_BLOCK_READ_RETRY_SLEEP_MIN =
Expand Down
6 changes: 5 additions & 1 deletion underfs/hdfs/src/main/java/alluxio/underfs/hdfs/HdfsUnderFileSystem.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -395,8 +395,12 @@ public UfsFileStatus getFileStatus(String path) throws IOException {
FileStatus fs = hdfs.getFileStatus(tPath);
String contentHash =
UnderFileSystemUtils.approximateContentHash(fs.getLen(), fs.getModificationTime());
long blockSize = fs.getBlockSize();
if (mUfsConf.getBoolean(PropertyKey.USER_BLOCK_SIZE_ENABLED)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this flag enabled, getFileStatus() will start returning wrong block size. So I don't think this the right place to patch up.

What's the exact reason you want to override UFS block-size with Alluxio block-size?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

I just reference the ObjectUnderFileSystem#getFileStatus method.

The exact reason for customize the block size is that in our alluxio cluster, cephfs is our underfilesystem, the blockSize from cephfs is 4MB, so a 1 GB big file will cost 1GB/4MB blocks, i want to customize the alluxio block size, for example 64MB, it would be better to reduce the block count.

blockSize = mUfsConf.getBytes(PropertyKey.USER_BLOCK_SIZE_BYTES_DEFAULT);
}
return new UfsFileStatus(path, contentHash, fs.getLen(), fs.getModificationTime(),
fs.getOwner(), fs.getGroup(), fs.getPermission().toShort(), fs.getBlockSize());
fs.getOwner(), fs.getGroup(), fs.getPermission().toShort(), blockSize);
}

@Override
Expand Down