Fix tests in AlluxioJniFuseFileSystemTest #18269
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
### What changes are proposed in this pull request? Combined Tecent COS & COSN Docs ### Why are the changes needed? Redundant information can be consolidated via tabs  ### Does this PR introduce any user facing changes? Web UI pr-link: Alluxio#17821 change-id: cid-dc876db94f3192130a8ec20ccca14373514eefb7
Fix Fuse docs pr-link: Alluxio#17815 change-id: cid-1b51e79e4b9fab28fe7c624fab3c7a821f1e862b
### What changes are proposed in this pull request? Remove secondary master ### Why are the changes needed? Secondary Master is no longer used ### Does this PR introduce any user facing changes? - Property key `alluxio.secondary.master.metastore.dir` is removed pr-link: Alluxio#17823 change-id: cid-6eabcd4e53d29aba7cea403ea8172a6af4a1328d
### What changes are proposed in this pull request? Stop shading alluxio-test. ### Why are the changes needed? Shading a module is slow and error-prone. The shading of the `alluxio-test` module was previously introduced in Alluxio#7765 for journal-related tests. There is no longer a need to shade this module. As part of this PR, `listStatusRootTest` (introduced in Alluxio#15361) has been moved to the S3 specific tests. This move was made because the test failed after the shading was removed on my local MacOS (although it passes CI on Linux), likely due to an unusual setting of the root directory on MacOS. I am unsure why removing shading on `alluxio-test` also impacts `alluxio-integration-tools-validation`. However, based on the description of Alluxio#15361, it does seem to fit better in the S3 specific tests. ### Does this PR introduce any user facing changes? n/a pr-link: Alluxio#17798 change-id: cid-e7570b42d9dd096af93053f84a3be226b66f9120
### What changes are proposed in this pull request? Remove the experimental feature to fallback async-write when Alluxio space is full. ### Why are the changes needed? This feature has been experimental for a few years, not been able to graduate. Given it adds a good amount of complexity, we plan to remove it. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including - Property key `alluxio.user.file.ufs.tier.enabled` is removed pr-link: Alluxio#17796 change-id: cid-72acdb41ec1164f0ec7abdd5284a580271430855
### What changes are proposed in this pull request? Change for future extension ### Why are the changes needed? Change for future extension ### Does this PR introduce any user facing changes? NA pr-link: Alluxio#17812 change-id: cid-1efc20ddcd97fd4b0d38f507f0bee5f8e883fbf9
remove the accordion , so the `toc` navbar on right side can jump to metric content correctly pr-link: Alluxio#17822 change-id: cid-e0303a9f1a217a1b6a5fc6d5d90ec05d430bf1e5
- don't allow any var args in the entrypoing scripts; specify all configurations via env vars - specify which projects to build via ALLUXIO_DOCKER_MVN_PROJECT_LIST, add -am flag to build dependent projects - specify test prefix via ALLUXIO_DOCKER_MVN_TESTS - remove ALLUXIO_DOCKER_MVN_RUNTOEND - remove "libfuse version = 2" permutation from fuse integration tests - can't pass in an env var that contains a `=`, which was attempted in the format of `ADDITIONAL_ARGS=property=key` and caused parsing failure - combine several integration test permutations to reduce overall time spent, since the permutations are finishing much faster than the slowest builds = unit test and checkstyle builds pr-link: Alluxio#17811 change-id: cid-fca8ed001d33e49867ebb1732e6a180d6726f6b5
### What changes are proposed in this pull request? Add interfaces to allow using custom modules to inject in worker ### Why are the changes needed? The `LocalAlluxioCluster` can be reused outside the repo with custom worker implemenetation. ### Does this PR introduce any user facing changes? no pr-link: Alluxio#17829 change-id: cid-6b1d30a95e2eb583cb55d077e5bf88f80cbfae97
### What changes are proposed in this pull request? add data & metadata cache fuse docs to local cache overview ### Why are the changes needed? better user flow ### Does this PR introduce any user facing changes? Web UI pr-link: Alluxio#17828 change-id: cid-6a5682d1efcd5d006ae913c1016c239da4bc6b9e
Monitor was merged in when we were moving stuff from `master-2.x` into `main`. pr-link: Alluxio#17836 change-id: cid-4d1eaaa0174fce8280bb97f7c5e18223d2b457d9
Normalize the HTTP RESTful API: 1) the list files API: `curl -X GET "http://<worker_ip>:<port>/v1/files?path=<path>"`  Now the server will also return the lengths of the files. 2) the get page API: `curl GET "http://<worker_ip>:<port>/v1/file/<file_id>/page/<page_index>"`  pr-link: Alluxio#17833 change-id: cid-567e6e9baf6121b3260c0c0073bbd81d94e1c363
### What changes are proposed in this pull request? 1. Remove all creation and usage of `BaseFileSystem` 2. There are some job-related APIs in `BaseFileSystem`, which are still used in 3.0. Moved those to `UfsBaseFileSystem`. 3. Removed direct users of the `BaseFileSystem`, including `MountCommand/UnmountCommand/UpdateMountCommand`. The `mount()` API on `FileSystem` will be gone. ### Why are the changes needed? `BaseFileSystem` is the main entrance of the 2.x File/Block API on the client side. It links file I/O to AlluxioFileInStream and AlluxioFileOutStream. It also routes metadata RPCs to the master. In 3.0, we introduced `UfsBaseFileSystem` and `DoraCacheFileSystem` (together with some others). We have switched to the new code path in Alluxio#17612, we remove the entrance to the old code path and say goodbye. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 4. addition or removal of property keys 5. webui pr-link: Alluxio#17830 change-id: cid-db9be604dfd12d97173c088cd1914206139de55b
### What changes are proposed in this pull request? Added additional logic for building groups of artifacts via a single command. ### Why are the changes needed? Helpful for internal automation. Allows us to create custom setups for building artifacts ### Does this PR introduce any user facing changes? Adds `dev/scripts/build-artifact.sh` `presets` command pr-link: Alluxio#17834 change-id: cid-82ad6b6a314e1b091eaba9f690facb7d7ce4f167
pr-link: Alluxio#17840 change-id: cid-6fef2b5209574d06267d2b28dbc9e9d308578818
### What changes are proposed in this pull request? move information to better fitting pages update links that were plain text in config properties ### Why are the changes needed? remove redundant/irrelevant information links that are plain text are not user friendly ### Does this PR introduce any user facing changes? Web UI pr-link: Alluxio#17835 change-id: cid-2722135b7789ab72b7f992e02a338dd11fd4a297
this build is failing the compilation step, claiming it can't find dora/tests jar when building dora/microbench. strange that it wasn't happening when validating Alluxio#17811 pr-link: Alluxio#17842 change-id: cid-d339245bb1f66d4dae2ce63fe41c7a77470c3a6e
### What changes are proposed in this pull request? Cleanup top level POM dependencies - Move dependency version definition to <dependencyManagement> section - Remove unused properties - Sort properties and dependencies alphabetically - Remove unused profile ### Does this PR introduce any user facing changes? No pr-link: Alluxio#17826 change-id: cid-7646f3126f9b04e23f239fbd9fd8ccc8c6a4801b
### What changes are proposed in this pull request? Add HDFS UFS integration test using HDFS mini cluster ### Why are the changes needed? To test HDFS UFS related features better in alluxio ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#17832 change-id: cid-7f8df37f7ca6cf4c360b8d3f79edfade858f8e06
### What changes are proposed in this pull request? Rollback to use localcacheInStream for presto local cache ### Why are the changes needed? Fix the oom and out of bound issue in prestodb's local cache ### Does this PR introduce any user facing changes? no pr-link: Alluxio#17845 change-id: cid-42a3ac6c88a44a95e21879b757e984249deee0aa
### What changes are proposed in this pull request? Add metrics for num of request to external storage when cache miss ### Why are the changes needed? For cost saving ### Does this PR introduce any user facing changes? no pr-link: Alluxio#17846 change-id: cid-53374d78e218543bf77b458b04c5ed5b3f633457
### What changes are proposed in this pull request? Add metrics for num of client position read fallback ### Why are the changes needed? counting the number of fallback ### Does this PR introduce any user facing changes? no pr-link: Alluxio#17847 change-id: cid-520d266df7c019a24e43035eff08904149d56f73
### What changes are proposed in this pull request? This PR removes: 1. Load jobs and SetReplica jobs, together with their corresponding `SetReplication` and `DistributedLoad` commands 2. Replication handling logic at the Master 3. JobUtils.loadXXX methods ### Why are the changes needed? These features have no place in 3.x. Replication management will be implemented on the new architecture. And Load feature has been implemented by using the master-side scheduler instead of job service. Another root reason for removing these code is we want to remove `BlockLocationPolicy`. `JobUtils.loadXXX` methods are one of the main users which directly creates `BlockLocationPolicy` objects. This will be a prerequisite for Alluxio#17831 ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 5. webui pr-link: Alluxio#17843 change-id: cid-8cb6d7098fc79761bb4f37819ae3f1cf63d28aa7
### What changes are proposed in this pull request? if catch `InvalidArgumentRuntimeException`, convert to `IllegalArgumentException` in `AbstractFileSystem.java`. if catch other `AlluxioRuntimeException`, convert to `IOException` in `AbstractFileSystem.java`. ### Why are the changes needed? make users clear about the exception type. ### Does this PR introduce any user facing changes? none pr-link: Alluxio#17788 change-id: cid-4017fc9547c0fa80be1efbc5bf5ead45910a6ae3
### What changes are proposed in this pull request? Add metrics counter for the page invalidated by TTL ### Why are the changes needed? ### Does this PR introduce any user facing changes? no pr-link: Alluxio#17849 change-id: cid-c9b2087e31637c64885e95a8d87ca27d31d8754a
This reverts commit 479699b. ### What changes are proposed in this pull request? The PR breaks the tarball build. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#17851 change-id: cid-8cc6108924224cda31045da9bc364b548c426b3e
### What changes are proposed in this pull request? 1. A new file `S3ObjectTest.java` was created to run object related UTs for S3 proxy based on Dora version. 2. fix a bug in`alluxio.proxy.s3.S3ObjectTask.CopyObjectTask#continueTask` ### Why are the changes needed? The original s3 API unit tests were very messy and irregular. It is difficult for us to confirm which testing scenarios we have correctly covered. We want to set up s3mock as the UFS and reorganize the unit test for S3 proxy based on Dora version. ### Does this PR introduce any user facing changes? none pr-link: Alluxio#17777 change-id: cid-bfd4232d1c10f588c752f2b8f5a9a4d38e4e5f3d
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#17854 change-id: cid-be8f68a8545a41391f5681f865e98cb4d5bbc0f7
What changes are proposed in this pull request? I have created a test and create a liststatus test for its function. Why are the changes needed? Please clarify why the changes are needed. For instance, add a unit test for DoraWorkerClientServiceHandler. Does this PR introduce any user facing changes? No. pr-link: Alluxio#18059 change-id: cid-b82706a4419700f017584f3e5579d2ef3410aeb3
Make fuse max reader concurrency configurable. The default value was 64 and it was unchangeable. pr-link: Alluxio#18129 change-id: cid-9c55821622329bd1e608da2e7445e8ab591df38a
Fix typo from alluxio.max.fuse.reader.concurrency to alluxio.fuse.max.reader.concurrency pr-link: Alluxio#18134 change-id: cid-434086cf6ba9e9f8d173e3417fc8518963dfa102
update usages of bin/alluxio, bin/alluxio-start.sh and bin/alluxio-stop.sh to their new counterparts simplify section of CephFS.md and remove sections related to mounting. the ufs must be configured as the root mount via alluxio-site.properties. pr-link: Alluxio#18136 change-id: cid-fa7d0eec00c8fb136680ef6d5a2c7ee78571d123
### What changes are proposed in this pull request? Support accessing OSS through proxy by configuring alluxio properties or system properties. ### Why are the changes needed? When accessing OSS through a proxy, the OSS client cannot recognize the proxy configuration in system property and environment variables. So it has to proactively set proxy-related configurations in the configuration. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18139 change-id: cid-5e30dfd90747d4a1aafe9b2ff985331f05fefec6
### What changes are proposed in this pull request? If don't set oss.proxy.host, the default value should be NULL ### Why are the changes needed? If don't set oss.proxy.host, the default value should be NULL ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18142 change-id: cid-bce2790e583445c4ba6720d2f0a64551fb19de20
### What changes are proposed in this pull request? In this change the path conversion logic is extracted to static utility methods for code reuse (because other classes may use the same path resolution logic). The method names are slightly improved, to distinguish the member methods in `DoraCacheFileSystem` (may be inherited) from the static utility methods. pr-link: Alluxio#18140 change-id: cid-557fa148f6daa41f0b296132e5a2ecae6c5d6c22
pr-link: Alluxio#18138 change-id: cid-e9f862385bdfe6cc5e6938eb49907055449deb4a
### What changes are proposed in this pull request? 1. Move the path resolution logic from `DoraCacheFileSystem` to `PathUtils` where it makes more sense 2. Fix the alluxioPathToUfsPath resolution by handling the case where the ufs path may have no matching alluxio path, making the util method more generic pr-link: Alluxio#18146 change-id: cid-ebace1efcf58e385bbf71b599e4b5a15a2199f7e
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Better error handling when a stream is closed. Log more and don't swallow errors ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18145 change-id: cid-b1146421ea5bc51e9a5eea5bf11ce1a5d8466912
Use jackson JSON as a standard format for reports.
Output example for `bin/alluxio info report summary`:
```
{
"mSafeMode":false,
"mZookeeper":false,
"mRaftJournal":true,
"version":"304-SNAPSHOT",
"uptime":"0 day(s), 0 hour(s), 3 minute(s), and 7 second(s)",
"rpcPort":19998,
"webPort":19999,
"masterAddress":"Ec2Cluster-masters-0:19998",
"masterVersions":[
{
"version":"304-SNAPSHOT",
"state":"PRIMARY",
"host":"Ec2Cluster-masters-0",
"port":19998
}
],
"started":"08-24-2023 06:43:32:856",
"zookeeperAddress":[
],
"raftJournalAddress":[
"Ec2Cluster-masters-0:19200"
],
"liveWorkers":2,
"lostWorkers":0,
"freeCapacity":"2048.00MB",
"totalCapacityOnTiers":{
"MEM":"2048.00MB"
},
"usedCapacityOnTiers":{
"MEM":"0B"
}
}
```
pr-link: Alluxio#18047
change-id: cid-bf6d54f47390a4d2bd84e4baac2ea2863d4638e1
### What changes are proposed in this pull request? refactor the cli code ### Why are the changes needed? make the code more flexible and easy for adding more functions in a cleaner way ### Does this PR introduce any user facing changes? nope pr-link: Alluxio#18152 change-id: cid-d8f937075174d913daf32387d781161096f03345
|
Please remove the DoraFix annotation in the future. |
YichuanSun
commented
Oct 16, 2023
Comment on lines
121
to
123
| @DoraTestTodoItem(action = DoraTestTodoItem.Action.FIX, owner = "LuQQiu", | ||
| comment = "waiting on security metadata to be implemented in Dora") | ||
| @Ignore |
There was a problem hiding this comment.
I don't know whether "security metadata" is implemented, please check it.
There was a problem hiding this comment.
The same for the code follows.
apc999
force-pushed
the
main
branch
2 times, most recently
from
2fec0ec to
b597c61
Compare
|
Due to this PR, the test of statfs() cannot be fixed in this stage. |
LuQQiu
reviewed
Oct 27, 2023
dora/integration/fuse/src/test/java/alluxio/fuse/AlluxioJniFuseFileSystemTest.java
Outdated
Show resolved
Hide resolved
YichuanSun
commented
Oct 31, 2023
Comment on lines
+692
to
+701
| String fuseMountAlluxioPath = conf.getString(PropertyKey.FUSE_MOUNT_ALLUXIO_PATH); | ||
| // If we have set an FUSE_MOUNT_ALLUXIO_PATH, use it as the "MountedRootPath". | ||
| if (!fuseMountAlluxioPath.isEmpty() | ||
| && (!fuseMountAlluxioPath.equals(PropertyKey.FUSE_MOUNT_ALLUXIO_PATH.getDefaultValue()))) { | ||
| return fuseMountAlluxioPath; | ||
| } | ||
| if (options.isPresent()) { | ||
| return options.get().getUfsAddress(); | ||
| } | ||
| return Strings.EMPTY; |
There was a problem hiding this comment.
The old code is that if getUfsFileSystemOptions() is successful, then return the Ufs root address as the "MountedRootPath", while we have set the PropertyKey.FUSE_MOUNT_ALLUXIO_PATH where we expect alluxio to mount to. So the "MountedRootPath" is not the FUSE_MOUNT_ALLUXIO_PATH we expected, then asserts error.
There was a problem hiding this comment.
The values in Optional<UfsFileSystemOptions> options have higher priority in my mind
There was a problem hiding this comment.
You are right, but if i change the priority, all the ignored tests(except 1) in AlluxioJniFuseFileSystemTest.java are passed. I'm confused and not sure of the correctness of my fix.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Fix broken tests in AlluxioJniFuseFileSystemTest. Remain only one tests in AlluxioJniFuseFileSystemTest not fixed.
Why are the changes needed?
improve code quality.
Does this PR introduce any user facing changes?
no.