Refactor syncInterval calculation mechanism #18410
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
### What changes are proposed in this pull request? Add metrics for the ages of the pages in page cache store ### Why are the changes needed? To make sure we're not keep the data in cache too long ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18035 change-id: cid-36d458b68e059ce8cb934db7459bf4af3f298b93
### What changes are proposed in this pull request?
The LsCommand displays DataInAlluxioPercentage, but it always shows 0% which is misleading.
This change changes the DataInAlluxioPercentage column to a fix string FILE in the Lscommand.
### Why are the changes needed?
Please clarify why the changes are needed. For instance,
As mentioned above, it fixes the bug that LsCommand always shows 0% for DataInAlluxio.
### Does this PR introduce any user facing changes?
Yes, it will change the display of LsCommand.
It will change the output of LsCommand for files but not directories. For files, it will be the follow format
```
./alluxio fs ls /file_path
FileSize PeristenceState Timestamp FILE FilePath
```
Please list the user-facing changes introduced by your change, including
It changes the percentage of data in Alluxio to a fix string FILE as shown above.
pr-link: Alluxio#18009
change-id: cid-8b50cec273f5c8edbe7fa14640c59bd41a431e42
### What changes are proposed in this pull request? - Change the serialize/deserialize interface for future use(Alluxio#17959) - Expose the TTL and timeout of the lease used in `ServiceDiscoveryRecipe` - Expose the path prefix of the `ServiceDiscoveryRecipe` for future use ### Why are the changes needed? etcd will be used in the near future for more areas than worker service discovery. so expose more information for future use. ### Does this PR introduce any user facing changes? nope pr-link: Alluxio#18010 change-id: cid-7cc0b9f5cfaee860e587941c37867797b7fe7bdb
Additional `init` commands to golang CLI as part of Alluxio#17522 `bin/alluxio-bash format` -> `bin/alluxio init format` `bin/alluxio-bash copyDir` -> `bin/alluxio init copyDir` pr-link: Alluxio#18040 change-id: cid-511177a75c48222ff7e0c4122cb2aa17a9f86576
### What changes are proposed in this pull request? 1. fix bugs in `alluxio.proxy.s3.S3ObjectTask.CompleteMultipartUploadTask#validateParts` 2. `CompleteMultipartUpload` executes the specified parts with the specified upload ID rather than all parts. 3. add `initiate / complete / abort MPU and upload part` unit tests. 4. remove `XAttr` 5. show all upload-parts to user while listing parts ### Why are the changes needed? there are problems in validateParts() of complete MPU 1. can’t recognize S3 Error: MalformedXML 3. throw unexpected error: InvalidPartOrder we need to fix validateParts() of complete MPU to keep s3 proxy behaviors as consistent with aws as possible. ### Does this PR introduce any user facing changes? - can't get the bucket name and object name of the part while executing ListMultipartUploads - now complete MPU's response is like |part list|response(before)|response(now)| |-|-|-| | [] | 500 |MalformedXML| |[1,3]| InvalidPartOrder | 200 | |[2,1]| InvalidPartOrder |InvalidPartOrder| |[3,3]| InvalidPartOrder |InvalidPartOrder| pr-link: Alluxio#18046 change-id: cid-2d5dbc5491e5d54ca48d20a230a3b0b8dbf4ee97
### What changes are proposed in this pull request? Fix multiple mount options specified by `-o` not recognized by Alluxio Fuse. The `-o` option can be specified multiple times, and each time it can take a comma separated list of `key=value` mount options. ### Why are the changes needed? Fuse mounting with `bin/alluxio-fuse mount hdfs://10.10.1.2:9000/ /work/alluxio_fuse -o kernel_cache -o attr_timeout=6000 -o entry_timeout=6000` errors with ``` Exception in thread "main" com.beust.jcommander.ParameterException: "-o": couldn't convert "kernel_cache,attr_timeout=6000,entry_timeout=6000" to a `key=value` pair because contains more than 1 `=` at alluxio.fuse.options.MountCliOptions$KvPairsConverter.convert(MountCliOptions.java:74) at alluxio.fuse.options.MountCliOptions$KvPairsConverter.convert(MountCliOptions.java:50) at com.beust.jcommander.JCommander.convertValue(JCommander.java:1333) at com.beust.jcommander.ParameterDescription.addValue(ParameterDescription.java:249) ``` The bash scripts concatenates the multiple occurrences of `-o` into a comma-separated list and passes it to the Java program. The PR does the other way around, preserving them and splitting the kv pair list in a `-o` option into multiple options. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18026 change-id: cid-d9b5dcf65ba3de1bd9324ed01f74944171d5ef30
### What changes are proposed in this pull request? Return relative path in listStatus results. ### Why are the changes needed? After Alluxio#17839 the worker only receives and sends absolute UFS paths. This change convert them back to Alluxio relative path according to client's perspective about the UFS. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18014 change-id: cid-c874c95cb9331c59de7be7e66d3cc9cc0e1a653f
- remove job subcommands that were referring to the old job service - update `load` subcommand and add necessary flags - remove checks in the java class that referred to the old load command pr-link: Alluxio#18045 change-id: cid-31fce5258be608e7d34648e07940857d30340c60
### What changes are proposed in this pull request? add S3 list-parts UTs. ### Why are the changes needed? test for s3 list parts ### Does this PR introduce any user facing changes? none pr-link: Alluxio#18048 change-id: cid-89e5c55284c4b0e7bf081ff6bb7d76c356149629
update Rename and exists tests for PagedDoraWorker. ### What changes are proposed in this pull request? I have added two unit test for PagedDoraWorker. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. One function for test Rename in PagedDoraWorker. 2.One function for test Exists in PagedDoraWorker. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18050 change-id: cid-06100f520a97ba73b5c4a30ff5fd574b4a578b75
### What changes are proposed in this pull request? Remove the code of web under filesystem. ### Why are the changes needed? We will remove some unused UFSes, now is WebUnderFileSystem. Some UFSes are currently in a pending state (such as Kodo). If users continue to require them in the future, we can consider ongoing support; otherwise, they will be removed in the future, too. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18051 change-id: cid-ad1ebd19a7c8f847a3607b6237523020a1b5d86b
### What changes are proposed in this pull request? Add a `RunWithWriters` interface apart from `Run` for future use. ### Why are the changes needed? In the future, we might need to parse the output of java in golang CLI. In this way we can reuse more code. ### Does this PR introduce any user facing changes? nope pr-link: Alluxio#18054 change-id: cid-9e13b73f1532ee8e3d95559b1a236035685149ec
remove `info master` command from cli, as underlying java command is already removed pr-link: Alluxio#18055 change-id: cid-c612ac3f0ae865dafb8ecbf73dc48417e0e1863e
### What changes are proposed in this pull request? 1. Make all buckets visible to any user. 2. add `listAllBuckets` UT. ### Why are the changes needed? in dora version, buckets' authentication depends on the ufs owner rather than the alluxio user , so we show all buckets to everyone. ### Does this PR introduce any user facing changes? all buckets are visible to everyone pr-link: Alluxio#18041 change-id: cid-0c723f85250ee5a39e84ac060d1334649a41e6d1
Original-author: humengyu <[email protected]> ### What changes are proposed in this pull request? Support hdfs trash in hdfs ufs. ### Why are the changes needed? Alluxio#17723 HDFS trash is a very useful feature, and we should support it. It allows users to recover deleted files from the trash after accidental deletion. ## Usage: add config in `alluxio-site.properties`: ``` alluxio.underfs.hdfs.trash.enable=true ``` add config in hdfs `core-site.xml`: ``` <property> <name>fs.trash.interval</name> <value>1440</value> </property> ``` ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs YES 2. addition or removal of property keys YES 3. webui NO pr-link: Alluxio#18036 change-id: cid-42a8bad756cad279ab8ea2c743c1e052f5b67356
### What changes are proposed in this pull request? fix the bug where fuse has the bug of "java.lang.NoClassDefFoundError: Could not initialize class io.vertx.core.dns.AddressResolverOptions" after jetcd dependency was introduced ### Why are the changes needed? classes from native side, if not explicitly loaded, it will spew "NoClassDefFoundError" at runtime when the relevant classes was invoked. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18039 change-id: cid-999fbcf89cba5d11fded608e0da3e395f6af20c8
### What changes are proposed in this pull request? Remove the code of web under filesystem. ### Why are the changes needed? We will remove some unused UFSes, now is WebUnderFileSystem. Some UFSes are currently in a pending state (such as Kodo). If users continue to require them in the future, we can consider ongoing support; otherwise, they will be removed in the future, too. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18058 change-id: cid-742ed90a086612dbd978102c9464a36be8837973
### What changes are proposed in this pull request? - Enables splitting HTTP requests into HTTPRequest and multiple HTTPContent segments to handle large object uploads. - Supports AWS v4 chunked content for object uploading. - Provides support for the POST bulk delete objects API. - Optimized asynchronous processing logic. ### Why are the changes needed? We need to accomplish large object uploads by handling multiple HttpContent messages. AWS v4 chunked upload support is required. The existing asynchronous processing logic needs to be adapted to accommodate the new way of handling HttpContent. ### Does this PR introduce any user facing changes? Adds POST bulk delete objects API in netty s3 api. pr-link: Alluxio#17894 change-id: cid-78036fc4484cc5fae5b6c911b186bf802e854c9f
### What changes are proposed in this pull request? Fix the copy handler's ufs manager creation and refactor some progress report. ### Why are the changes needed? Found a bug during e2e tests for GCS. ### Does this PR introduce any user facing changes? na pr-link: Alluxio#18061 change-id: cid-1ef059d582390ec33484692ff15833f46d6eabbf
### What changes are proposed in this pull request? Add a new function in golang CLI to support reformatting the json output to yaml and so on. ### Why are the changes needed? For `info` commands, it'll provide a easier way to change the format. ### Does this PR introduce any user facing changes? nope pr-link: Alluxio#18060 change-id: cid-d8b13ef66abc46bc0b2a8f575ccd412aca5edb20
### What changes are proposed in this pull request? Chooses a preferred worker from worker list to fetch files pr-link: Alluxio#18063 change-id: cid-3449212125c0c6b7d89c617670348b7251856687
### What changes are proposed in this pull request? Enable dora meta store WAL ### Why are the changes needed? Otherwise data will lose after worker restarts ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18062 change-id: cid-db0ed0738f3470c91062441537d2d4b4fcd033a3
### What changes are proposed in this pull request? This is a very small change which allows children of `DoraFileSystem` to have a ref to its `FileSystemContext`. pr-link: Alluxio#18064 change-id: cid-d6c74f4b633230da578296edc5a1b86a6f2c534b
### What changes are proposed in this pull request? Tracked in Alluxio#18065 . In Alluxio version 3.0, we plan to remove some rarely used UFS connectors. This PR is to remove Swift. Most storage systems using the Swift interface provide S3-compatible APIs, you can still try to use the S3 connector to integrate with these storage systems. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18066 change-id: cid-b2912c56efd7e26b7338d90f622184231abd52a2
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18069 change-id: cid-b044da28c68aae3c31fb110adec4146adf2faaf3
### What changes are proposed in this pull request? Tracked in Alluxio#18065 . In Alluxio version 3.0, we plan to remove some rarely used UFS connectors. This PR is to remove Kodo. Kodo also provides S3-compatible APIs, you can still try to use the S3 connector to integrate with kodo. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18067 change-id: cid-9206c917e136ca4fada3c9089e2933d9308a822c
### What changes are proposed in this pull request? fix the skip error handling ### Why are the changes needed? Previously we consider the skip as a failure, which is not correct. Consider skip only as skip. ### Does this PR introduce any user facing changes? na pr-link: Alluxio#18072 change-id: cid-8bc842be694b28a9a78c3b984eae57195ed10f8b
### What changes are proposed in this pull request? Reorder "Get Started" section of main nav menu ### Why are the changes needed? better user journey ### Does this PR introduce any user facing changes? n/a pr-link: Alluxio#18071 change-id: cid-4926667172302929aa2fbfb39bc7d19e4b9bbbf3
### What changes are proposed in this pull request? label ufs contributed/maintained by community and update github links ### Why are the changes needed? bring more awareness of os community ### Does this PR introduce any user facing changes? n/a pr-link: Alluxio#18078 change-id: cid-d33c619b5c737f78a73963d7689d144705d3bca5
### What changes are proposed in this pull request? - remove redundant info, add redirect link to section on local cache quick start doc - update external links to open in new tab ### Why are the changes needed? better user journey ### Does this PR introduce any user facing changes? n/a pr-link: Alluxio#18083 change-id: cid-062ece308861d9ccb90f85f10e8e43e858b87d5f
### What changes are proposed in this pull request? Improve UpdateChecker ### Why are the changes needed? 1. remove irrelevant master features 2. turn on fuse update check pr-link: Alluxio#18101 change-id: cid-8f6801fedda5f7710d111939e50cfd5f4372b7e5
This pull request fixes a bug on stress worker bench. 1. When using the remote worker policy, running `findNextWorker()`, multiple threads increment `mNextIndex`, leading to possible array out of bounds errors when getting worker info. 2. If use remote worker policy and the cluster size is 1, no workers can be found in `findNextWorker()`, resulting in errors. This fix turns `findNextWorker()` into a synchronized function, which can avoid data corruption. In addition, this policy can only be used when cluster size > 1. pr-link: Alluxio#18377 change-id: cid-2e391549b5d81ca3f1ff60f7a6c7eb59e99cc736
a previous PR Alluxio#18313 restored the integration tests but caused the unit tests to also run with the integration tests. the issue was correctly identified as an issue with the maven project list but the underlying problem was that the project name was not correctly set; `dora/tests` is not a module. it was previously a module until `dora/tests/testcontainers` was introduced, thus separating testcontainer tests vs integration tests. this fix updates the previous project list with the correct module name pr-link: Alluxio#18392 change-id: cid-5f899f224abfbea4b7608d72cc2d38fed6449836
…3.x [DOCFIX] Fix doc errors for Tencent Cloud EMR Please outline the changes and how this PR fixes the issue. Tencent Cloud EMR has updated some version information and needs to be synchronized to community documents Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#17747 change-id: cid-73d26a59de4f1023081ed0ffaf91cffcc2f8db0e
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#17785 change-id: cid-657e47ef7657714a572a1f9811bbcc524bb8cfcc
### What changes are proposed in this pull request? Fix a deadlock. ### Why are the changes needed? On the Alluxio client, there is a monitor lock of `FileSystemContext` to sync individual client-to-master operations. One application thread can successfully hold the monitor lock (`alluxio.client.file.FileSystemContext`) in order to get worker list, but blocked by another “lock” `FileSystemContextReinitializer` (waiting for latch on on-going RPCs down to zero) for acquiring the block master client to really connect to master process (waiting for other callers involving `FileSystemContextReinitializer` to finish). On the other hand, another heartbeat thread "config-hash-master-heartbeat-0" is awaking periodically to sync with the master process to fetch the latest configuration. This thread detected the conf update and thus entered `FileSystemContextReinitializer` (bumping latch) but was blocked by waiting for the monitor lock of `alluxio.client.file.FileSystemContext` in order to get the master address. This PR moves `getMasterAddress` outside `reinit` block to avoid holding the `Reinitializer` object and wait for the monitor object of `FileSystemContext`. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18109 change-id: cid-0fab540f7dbef3661acb44e7472425add163b019
…and resolve a trivial conflict Fix local cache identifier If not including `modification time`, it may have consistency issues. NO pr-link: Alluxio#17514 change-id: cid-c9c2e91e53d4a95d4f3a165918fc23c02f623891
### What changes are proposed in this pull request? Delegate `getFileBlockLocation` to external file system in `LocalCacheFileSystem`. ### Why are the changes needed? Otherwise, `LocalCacheFileSystem` inherits the default behavior of `org.apache.hadoop.fs.FileSystem` which returns `localhost` only. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#17672 change-id: cid-eb545dbd8ed42001d074fecfb9c8d6b118a559c1
### What changes are proposed in this pull request? Support read bytebuffer for non ByteBufferReadable input stream ### Why are the changes needed? Without this changes, a non ByteBufferReadable input stream can throw exception when the bytebuffer apis are called. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#17982 change-id: cid-1363dbbeca327cbb7b4d39cea1afbb8c51905483
### What changes are proposed in this pull request? The v1 and v2 versions of proxy use uniform judgment conditions. pr-link: Alluxio#18371 change-id: cid-1842a0b96b5233f9a456e1aee7dbd95e2bd2d5e0
### What changes are proposed in this pull request? Support creating directories and files with default permissions based on configuration propertykey in HDFS API. ### Why are the changes needed? Alluxio Hdfs api hasn't the corresponding implementation of Mkdir without permission parameter. If it does not carry permission, the umask property in the configuration item shall prevail. ### Does this PR introduce any user facing changes? pr-link: Alluxio#18253 change-id: cid-6ddd2243bac00ebbdbdff1e731036c0d3d6228c8
… existed objects in OSS ufs Using the user of the OSS bucket to represent the user of the object loaded from OSS. Add the mapping way from OSS username to the custom username. Add the default permission mode for existing files loaded from OSS. the object loaded from OSS ufs will be null user, null group, and 777 by default. pr-link: Alluxio#18262 change-id: cid-7e56861c9def695876cde32d1e0aa453b512470d
### What changes are proposed in this pull request? The old one is ignored, now bring it back. ### Why are the changes needed? Improve code quality. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18376 change-id: cid-2cdb7611d32e50fc16044192edf1122f4a3cd646
Also print file path in getStatus() exception on client side ### What changes are proposed in this pull request? Use generic Exception to catch all exceptions in getStatus(), just like that in all other handlers. File path will be included in log message for getStatus(). ### Why are the changes needed? Sometimes, exceptions are thrown in under file system and/or other third-party libraries. We don't know the exact type of the specific exceptions. If they are not caught, client will report unknown exceptions in logs. This is misleading. ### Does this PR introduce any user facing changes? Client will show logs with detailed exception information if there is any. For getStatus(), file path will also be shown in log. pr-link: Alluxio#18398 change-id: cid-b6acee3dd0c3e2e03e66bbf195cbda1e95f6bf64
### What changes are proposed in this pull request? Merge missing commits from master-2.x to main. The commits in 2023/07/01~2023/11/08 from Alluxio/alluxio@main...master-2.x will be included by this PR. We do this merge to catch missing fixes from `master-2.x` and catch the train before `main` cuts a release. Alluxio#17747 is not cherry picked because tencent cloud EMR doc is removed Alluxio#17755 is not cherry picked because DistLoadCliRunner has been removed in 3.x Alluxio#17758 is not cherry picked because MonoBlockStore has been removed in 3.x Alluxio#17641 is not cherry picked because the PR has already been in main Alluxio#17781 is not cherry picked because the PR has already been in main Alluxio#17722 is not cherry picked because the alluxio-fuse command has been changed a lot Alluxio#17489 is not cherry picked because audit log on master is no longer in 3.x Alluxio#17865 is not cherry picked because replication on job service is no longer in 3.x Alluxio#17858 is not cherry picked because it is already in main Alluxio#18090 is not cherry picked because generate-tarball has been rewritten in 3.x Alluxio#18091 is not cherry picked because the change is already in main Alluxio#17474 is not cherry picked because reconfiguration feature is not defined in 3.x Alluxio#17735 is not cherry picked because MonoBlockStore is no longer in 3.x Alluxio#18133 is not cherry picked because the issue is about master metadata and no longer relevant in 3.x Alluxio#17910 is not cherry picked because I prefer to do that manually Alluxio#17983 is not cherry picked because the web UI has been reworked Alluxio#17984 is not cherry picked because Mount/Unmount commands have been reworked in 3.x Alluxio#18103 is not cherry picked because worker cache metrics have been reworked in 3.x Alluxio#18185 is not cherry picked because the report command has been reworked in 3.x Alluxio#18222 is not cherry picked because Mount/Unmount operations have been reworked in 3.x Alluxio#18143 is not cherry picked because the change is already in main Alluxio#18303 is not cherry picked because the change is already in main Alluxio#18208 is not cherry picked because cache metrics have been reworked in 3.x Alluxio#17002 is not cherry picked because the owner has been notified separately Alluxio#18334 is not cherry picked because the bash scripts have been reworked in 3.x Alluxio#18326 is not cherry picked because the owner has been notified separately pr-link: Alluxio#18397 change-id: cid-dbf8cbb2d9e721a5a0a1e5028a3c9577438a2ac0
### What changes are proposed in this pull request? meta.getLastModified() could be null, and should avoid the NPE. the parameter is nullable, so give it a null in this case. ### Why are the changes needed? meta.getLastModified() could be null, and should avoid the NPE. the parameter is nullable, so give it a null in this case. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18403 change-id: cid-9d3c16f8c2b403dff668f52649432a5d7ca8f1a3
YichuanSun
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Refactor the syncInterval calculation in PagedDoraWorker. The old one is duplicated at least three times.
Why are the changes needed?
Improve code quality.
Does this PR introduce any user facing changes?
No.