Add configuration to set timeout for AsyncContext #18457
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
### What changes are proposed in this pull request? Add a `RunWithWriters` interface apart from `Run` for future use. ### Why are the changes needed? In the future, we might need to parse the output of java in golang CLI. In this way we can reuse more code. ### Does this PR introduce any user facing changes? nope pr-link: Alluxio#18054 change-id: cid-9e13b73f1532ee8e3d95559b1a236035685149ec
remove `info master` command from cli, as underlying java command is already removed pr-link: Alluxio#18055 change-id: cid-c612ac3f0ae865dafb8ecbf73dc48417e0e1863e
### What changes are proposed in this pull request? 1. Make all buckets visible to any user. 2. add `listAllBuckets` UT. ### Why are the changes needed? in dora version, buckets' authentication depends on the ufs owner rather than the alluxio user , so we show all buckets to everyone. ### Does this PR introduce any user facing changes? all buckets are visible to everyone pr-link: Alluxio#18041 change-id: cid-0c723f85250ee5a39e84ac060d1334649a41e6d1
Original-author: humengyu <[email protected]> ### What changes are proposed in this pull request? Support hdfs trash in hdfs ufs. ### Why are the changes needed? Alluxio#17723 HDFS trash is a very useful feature, and we should support it. It allows users to recover deleted files from the trash after accidental deletion. ## Usage: add config in `alluxio-site.properties`: ``` alluxio.underfs.hdfs.trash.enable=true ``` add config in hdfs `core-site.xml`: ``` <property> <name>fs.trash.interval</name> <value>1440</value> </property> ``` ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs YES 2. addition or removal of property keys YES 3. webui NO pr-link: Alluxio#18036 change-id: cid-42a8bad756cad279ab8ea2c743c1e052f5b67356
### What changes are proposed in this pull request? fix the bug where fuse has the bug of "java.lang.NoClassDefFoundError: Could not initialize class io.vertx.core.dns.AddressResolverOptions" after jetcd dependency was introduced ### Why are the changes needed? classes from native side, if not explicitly loaded, it will spew "NoClassDefFoundError" at runtime when the relevant classes was invoked. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18039 change-id: cid-999fbcf89cba5d11fded608e0da3e395f6af20c8
### What changes are proposed in this pull request? Remove the code of web under filesystem. ### Why are the changes needed? We will remove some unused UFSes, now is WebUnderFileSystem. Some UFSes are currently in a pending state (such as Kodo). If users continue to require them in the future, we can consider ongoing support; otherwise, they will be removed in the future, too. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18058 change-id: cid-742ed90a086612dbd978102c9464a36be8837973
### What changes are proposed in this pull request? - Enables splitting HTTP requests into HTTPRequest and multiple HTTPContent segments to handle large object uploads. - Supports AWS v4 chunked content for object uploading. - Provides support for the POST bulk delete objects API. - Optimized asynchronous processing logic. ### Why are the changes needed? We need to accomplish large object uploads by handling multiple HttpContent messages. AWS v4 chunked upload support is required. The existing asynchronous processing logic needs to be adapted to accommodate the new way of handling HttpContent. ### Does this PR introduce any user facing changes? Adds POST bulk delete objects API in netty s3 api. pr-link: Alluxio#17894 change-id: cid-78036fc4484cc5fae5b6c911b186bf802e854c9f
### What changes are proposed in this pull request? Fix the copy handler's ufs manager creation and refactor some progress report. ### Why are the changes needed? Found a bug during e2e tests for GCS. ### Does this PR introduce any user facing changes? na pr-link: Alluxio#18061 change-id: cid-1ef059d582390ec33484692ff15833f46d6eabbf
### What changes are proposed in this pull request? Add a new function in golang CLI to support reformatting the json output to yaml and so on. ### Why are the changes needed? For `info` commands, it'll provide a easier way to change the format. ### Does this PR introduce any user facing changes? nope pr-link: Alluxio#18060 change-id: cid-d8b13ef66abc46bc0b2a8f575ccd412aca5edb20
### What changes are proposed in this pull request? Chooses a preferred worker from worker list to fetch files pr-link: Alluxio#18063 change-id: cid-3449212125c0c6b7d89c617670348b7251856687
### What changes are proposed in this pull request? Enable dora meta store WAL ### Why are the changes needed? Otherwise data will lose after worker restarts ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18062 change-id: cid-db0ed0738f3470c91062441537d2d4b4fcd033a3
### What changes are proposed in this pull request? This is a very small change which allows children of `DoraFileSystem` to have a ref to its `FileSystemContext`. pr-link: Alluxio#18064 change-id: cid-d6c74f4b633230da578296edc5a1b86a6f2c534b
### What changes are proposed in this pull request? Tracked in Alluxio#18065 . In Alluxio version 3.0, we plan to remove some rarely used UFS connectors. This PR is to remove Swift. Most storage systems using the Swift interface provide S3-compatible APIs, you can still try to use the S3 connector to integrate with these storage systems. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18066 change-id: cid-b2912c56efd7e26b7338d90f622184231abd52a2
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18069 change-id: cid-b044da28c68aae3c31fb110adec4146adf2faaf3
### What changes are proposed in this pull request? Tracked in Alluxio#18065 . In Alluxio version 3.0, we plan to remove some rarely used UFS connectors. This PR is to remove Kodo. Kodo also provides S3-compatible APIs, you can still try to use the S3 connector to integrate with kodo. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18067 change-id: cid-9206c917e136ca4fada3c9089e2933d9308a822c
### What changes are proposed in this pull request? fix the skip error handling ### Why are the changes needed? Previously we consider the skip as a failure, which is not correct. Consider skip only as skip. ### Does this PR introduce any user facing changes? na pr-link: Alluxio#18072 change-id: cid-8bc842be694b28a9a78c3b984eae57195ed10f8b
### What changes are proposed in this pull request? Reorder "Get Started" section of main nav menu ### Why are the changes needed? better user journey ### Does this PR introduce any user facing changes? n/a pr-link: Alluxio#18071 change-id: cid-4926667172302929aa2fbfb39bc7d19e4b9bbbf3
### What changes are proposed in this pull request? label ufs contributed/maintained by community and update github links ### Why are the changes needed? bring more awareness of os community ### Does this PR introduce any user facing changes? n/a pr-link: Alluxio#18078 change-id: cid-d33c619b5c737f78a73963d7689d144705d3bca5
### What changes are proposed in this pull request? - remove redundant info, add redirect link to section on local cache quick start doc - update external links to open in new tab ### Why are the changes needed? better user journey ### Does this PR introduce any user facing changes? n/a pr-link: Alluxio#18083 change-id: cid-062ece308861d9ccb90f85f10e8e43e858b87d5f
Reduce CPU overhead by replacing while loop with await. The previous version waits for finishing sending data by using a **while loop**. In that way, cpu overhead would be pretty high. Thus, instead of waiting for finishing sending data with a **while loop**, we use a **BlockingQueue** to notice it when all data have been sent out successfully. pr-link: Alluxio#18068 change-id: cid-19a79070508e464299ff61645d711e9bc498654f
### What changes are proposed in this pull request? We will make copy existing directory fail the validation, which will skip copy the directory. But in the meanwhile, we shall pass the validation for move, since we want to delete any existing moved directory. ### Why are the changes needed? bugs from e2e tests. ### Does this PR introduce any user facing changes? na pr-link: Alluxio#18085 change-id: cid-7619ee0b6902dd95ae520a70631ddffc31b7670c
### What changes are proposed in this pull request? rearrange nav menu to move: - fuse sdk docs under reference/client apis/posix api - presto sdk under compute/presto - glossary & troubleshooting under reference ### Why are the changes needed? better doc flow ### Does this PR introduce any user facing changes? n/a pr-link: Alluxio#18084 change-id: cid-be60f2eb69f210c0d2044b1e2ecf9e2fa28f062a
New cli for collectInfo does not seem to work, fix the bug to keep the arguments in line with the Java code. pr-link: Alluxio#18080 change-id: cid-2a677899ad0964d79349e287b170afa0715c6cf7
### What changes are proposed in this pull request? Improve pom module name pr-link: Alluxio#18032 change-id: cid-c1ba348d25318ff0581e2bcbc5748bba1602631e
### What changes are proposed in this pull request? Worker checks UFS rename operation result and throws expcetion if failed ### Why are the changes needed? The rename operation result is omitted before. If operation fails, Alluxio client does not know the result. ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18089 change-id: cid-d6f93fe33adb42d2f9f5500a170cb2e15ab27aba
### What changes are proposed in this pull request? Remove tiered identity ### Why are the changes needed? This feature is no longer supported ### Does this PR introduce any user facing changes? N/A Entry points have been removed previously. This PR removes the src code pr-link: Alluxio#18033 change-id: cid-92a6d61ad51f12b708721e26ceab063359b5f2e5
### What changes are proposed in this pull request? Fix alluxio-fuse bug with `/tmp` symlink on macOS On macOS, the directory `/tmp/` is a symbolic link to `/private/tmp/`. This caused the `fuse_mounted` function to fail when attempting to mount a UFS to `/tmp/mnt`, as the `mount` command outputs the path as `/private/tmp/mnt`. This commit corrects this issue by resolving the symlink before checking the mount status. ### Why are the changes needed? Fix this bug which fails to detect if alluxio-fuse successfully mounts a remote UFS ### Does this PR introduce any user facing changes? NA pr-link: Alluxio#18093 change-id: cid-60a42e81b3c5f6ffe3f75b733817f4a58ec047cc
### What changes are proposed in this pull request? should use interface factory for membership mgr module for scheduler at time of master startup, otherwise setting STATIC membership mgr will not start master. ### Why are the changes needed? if alluxio.worker.membership.manager.type is set to STATIC, master won't be able to start as it needs etcd related info. ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18096 change-id: cid-c052114c1aef013049dcf9fccaded4663bf2dd8a
This PR fixed the shaded client version problem in dora example pom.
Current version in dora/example/pom.xml written as `<version>2.9.0</version>`. As a result, all updates on the shaded client after 2.9.0 won't work (e.g. removal of swift ufs). Now it is written as `<version>${project.version}</version>`
pr-link: Alluxio#18100
change-id: cid-0cb94bb5a3360b3cd347c2c09c7c741517ce8647
### What changes are proposed in this pull request? Delegate `getFileBlockLocation` to external file system in `LocalCacheFileSystem`. ### Why are the changes needed? Otherwise, `LocalCacheFileSystem` inherits the default behavior of `org.apache.hadoop.fs.FileSystem` which returns `localhost` only. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#17672 change-id: cid-eb545dbd8ed42001d074fecfb9c8d6b118a559c1
### What changes are proposed in this pull request? Support read bytebuffer for non ByteBufferReadable input stream ### Why are the changes needed? Without this changes, a non ByteBufferReadable input stream can throw exception when the bytebuffer apis are called. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#17982 change-id: cid-1363dbbeca327cbb7b4d39cea1afbb8c51905483
### What changes are proposed in this pull request? The v1 and v2 versions of proxy use uniform judgment conditions. pr-link: Alluxio#18371 change-id: cid-1842a0b96b5233f9a456e1aee7dbd95e2bd2d5e0
### What changes are proposed in this pull request? Support creating directories and files with default permissions based on configuration propertykey in HDFS API. ### Why are the changes needed? Alluxio Hdfs api hasn't the corresponding implementation of Mkdir without permission parameter. If it does not carry permission, the umask property in the configuration item shall prevail. ### Does this PR introduce any user facing changes? pr-link: Alluxio#18253 change-id: cid-6ddd2243bac00ebbdbdff1e731036c0d3d6228c8
… existed objects in OSS ufs Using the user of the OSS bucket to represent the user of the object loaded from OSS. Add the mapping way from OSS username to the custom username. Add the default permission mode for existing files loaded from OSS. the object loaded from OSS ufs will be null user, null group, and 777 by default. pr-link: Alluxio#18262 change-id: cid-7e56861c9def695876cde32d1e0aa453b512470d
### What changes are proposed in this pull request? The old one is ignored, now bring it back. ### Why are the changes needed? Improve code quality. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18376 change-id: cid-2cdb7611d32e50fc16044192edf1122f4a3cd646
Also print file path in getStatus() exception on client side ### What changes are proposed in this pull request? Use generic Exception to catch all exceptions in getStatus(), just like that in all other handlers. File path will be included in log message for getStatus(). ### Why are the changes needed? Sometimes, exceptions are thrown in under file system and/or other third-party libraries. We don't know the exact type of the specific exceptions. If they are not caught, client will report unknown exceptions in logs. This is misleading. ### Does this PR introduce any user facing changes? Client will show logs with detailed exception information if there is any. For getStatus(), file path will also be shown in log. pr-link: Alluxio#18398 change-id: cid-b6acee3dd0c3e2e03e66bbf195cbda1e95f6bf64
### What changes are proposed in this pull request? Merge missing commits from master-2.x to main. The commits in 2023/07/01~2023/11/08 from Alluxio/alluxio@main...master-2.x will be included by this PR. We do this merge to catch missing fixes from `master-2.x` and catch the train before `main` cuts a release. Alluxio#17747 is not cherry picked because tencent cloud EMR doc is removed Alluxio#17755 is not cherry picked because DistLoadCliRunner has been removed in 3.x Alluxio#17758 is not cherry picked because MonoBlockStore has been removed in 3.x Alluxio#17641 is not cherry picked because the PR has already been in main Alluxio#17781 is not cherry picked because the PR has already been in main Alluxio#17722 is not cherry picked because the alluxio-fuse command has been changed a lot Alluxio#17489 is not cherry picked because audit log on master is no longer in 3.x Alluxio#17865 is not cherry picked because replication on job service is no longer in 3.x Alluxio#17858 is not cherry picked because it is already in main Alluxio#18090 is not cherry picked because generate-tarball has been rewritten in 3.x Alluxio#18091 is not cherry picked because the change is already in main Alluxio#17474 is not cherry picked because reconfiguration feature is not defined in 3.x Alluxio#17735 is not cherry picked because MonoBlockStore is no longer in 3.x Alluxio#18133 is not cherry picked because the issue is about master metadata and no longer relevant in 3.x Alluxio#17910 is not cherry picked because I prefer to do that manually Alluxio#17983 is not cherry picked because the web UI has been reworked Alluxio#17984 is not cherry picked because Mount/Unmount commands have been reworked in 3.x Alluxio#18103 is not cherry picked because worker cache metrics have been reworked in 3.x Alluxio#18185 is not cherry picked because the report command has been reworked in 3.x Alluxio#18222 is not cherry picked because Mount/Unmount operations have been reworked in 3.x Alluxio#18143 is not cherry picked because the change is already in main Alluxio#18303 is not cherry picked because the change is already in main Alluxio#18208 is not cherry picked because cache metrics have been reworked in 3.x Alluxio#17002 is not cherry picked because the owner has been notified separately Alluxio#18334 is not cherry picked because the bash scripts have been reworked in 3.x Alluxio#18326 is not cherry picked because the owner has been notified separately pr-link: Alluxio#18397 change-id: cid-dbf8cbb2d9e721a5a0a1e5028a3c9577438a2ac0
### What changes are proposed in this pull request? meta.getLastModified() could be null, and should avoid the NPE. the parameter is nullable, so give it a null in this case. ### Why are the changes needed? meta.getLastModified() could be null, and should avoid the NPE. the parameter is nullable, so give it a null in this case. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18403 change-id: cid-9d3c16f8c2b403dff668f52649432a5d7ca8f1a3
### What changes are proposed in this pull request? Remove empty pages when they are accessed by the get page methods. Empty pages except for the vert first page of a file are most likely malformed and are a result of a failed put attempt. ### Why are the changes needed? Reading empty pages causes 0 bytes to be read, and in some cases causing an infinite loop in the caller. When used with `getDataFileChannel`, it results in the channel to be created multiple times for a given empty page, because the channels are always 0 sized. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#18401 change-id: cid-fff8f48af3814e3c71317377aa0b065599c06a57
### What changes are proposed in this pull request? Replace the FOLDER_SUFFIX from "_$folder$" to "/" in OSS UFS ### Why are the changes needed? We don't want to use '_$ Folder $" to represent the directory. We want to use a more general "/" instead to avoid some issues ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18408 change-id: cid-d16023b954bcf0ad9f37c9f629c5d0799443af8f
PR Alluxio#18377 removes setting property `alluxio.user.worker.selection.policy` by mistake. This will cause remote only policy not working. This PR fixes this problem. pr-link: Alluxio#18407 change-id: cid-a53dce950a07cdd661b3237731037a0e9efb052c
### What changes are proposed in this pull request? The `RuntimeException(FileAlreadyExistException)` changed to `AlreadyExistException()` in `PagedDoraWorker` ### Why are the changes needed? `RuntimeException(FileAlreadyExistException)` seems won't be caught in `DoraWorkerClientServiceHandler` when convertin it to `AlluxioRuntimeException`, which willl cause the loss status code ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18337 change-id: cid-9a520c49579847bae6da21302f484ba713eeb4d9
Introduction Chapter translation pr-link: Alluxio#18176 change-id: cid-c31d1ab3c4365f0bdb0d0b17418c92885a109276
### What changes are proposed in this pull request? Improve page loading performance by loading the page form UFS directly, instead of creating a reader. ### Why are the changes needed? To improve the data loading performance & reduce the memory consumption ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18389 change-id: cid-46e89dcfd22dcbbb4090df7f6c8861408ef8cb67
### What changes are proposed in this pull request? As the title states, this PR does a few things: 1. Allow rename() to overwrite an existing path, if specified in the option 2. The method `rename` in `AbstractFileSystem`, now will process the input path and rerun `rename` method when caught `AlluxioException` or `AlluxioRuntimeException`. Instead of log the exception and return end. 3. Also add some path checks during the rename 4. Reenable `FileSystemRenameIntegrationTest` by adding configs to `LocalAlluxioClusterResource` and reenable UT cases. pr-link: Alluxio#18263 change-id: cid-2870bf87fea8a3b2419e5b10a05423ff2dede6a2
### What changes are proposed in this pull request? Now WORKER_FUSE_ENABLED is invalid. ### Why are the changes needed? improve code quality. ### Does this PR introduce any user facing changes? no. pr-link: Alluxio#18349 change-id: cid-8064a8ef19179c31cdce01a833847ff530c1f776
### What changes are proposed in this pull request? Add call points of the new metrics ### Why are the changes needed? The new metrics are already defined in the previous pr, and need to be called in this pr. ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18351 change-id: cid-1b35a87f41a1f836005c368881378329f4c77b25
Fixed a single spelling error. ### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18310 change-id: cid-77c21de69cf13a6c94be418adf2be95e0f347d42
### What changes are proposed in this pull request? clean up unmount of fuse ### Why are the changes needed? alluxio-fuse may cause AlluxioFuse continues to hang, and also the current flow to unmount is scattered and all over the place. ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18241 change-id: cid-0246bd831466396697efe1977547d365ad63ba3d
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18412 change-id: cid-fa4b35605d8f97d79c159140613aacf27d3c1836
Fix the bug that `HdfsUfsStatusIterator` returns wrong path (the getName() method of `UfsStatus` should return a relative path, but not just the file name). pr-link: Alluxio#18424 change-id: cid-8b15fbd345f26562a06c5d9f52bdbc90a53b2039
Fix the bug that HDFS iterator doesn't work. pr-link: Alluxio#18429 change-id: cid-dbb5f1a246bce706f7b3b5d179ad158218dccf83
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Fix #18456.
Does this PR introduce any user facing changes?
Add a property
alluxio.proxy.s3.v2.async.context.timeout.