This repository has been archived by the owner on Dec 20, 2024. It is now read-only.
feature: handle client error and repair #582
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lowzj
force-pushed
the
handle-client-error
branch
from
7ab5140 to
938eeec
Compare
Codecov Report
@@ Coverage Diff @@
## master #582 +/- ##
==========================================
- Coverage 47.34% 47.33% -0.02%
==========================================
Files 97 97
Lines 5544 5544
==========================================
- Hits 2625 2624 -1
Misses 2702 2702
- Partials 217 218 +1
Continue to review full report at Codecov.
|
lowzj
force-pushed
the
handle-client-error
branch
from
938eeec to
d0eac20
Compare
lowzj
force-pushed
the
handle-client-error
branch
2 times, most recently
from
124a020 to
a128d65
Compare
chenchaobing
reviewed
Jun 5, 2019
...rc/main/java/com/dragonflyoss/dragonfly/supernode/service/repair/BaseClientErrorHandler.java
Show resolved
Hide resolved
chenchaobing
reviewed
Jun 5, 2019
...rc/main/java/com/dragonflyoss/dragonfly/supernode/service/repair/BaseClientErrorHandler.java
Show resolved
Hide resolved
chenchaobing
reviewed
Jun 5, 2019
| @Slf4j | ||
| public abstract class BaseClientErrorHandler implements ClientErrorHandler { | ||
| private static final ExecutorService executor = ExecutorsUtil | ||
| .newThreadPool("client-error", 20, 200); |
There was a problem hiding this comment.
i guess may not need so many threads.
chenchaobing
reviewed
Jun 5, 2019
| String localMd5 = getMd5(info.getTaskId(), info.getRange()); | ||
| if (StringUtils.isNotBlank(localMd5)) { | ||
| if (!StringUtils.equals(md5FromMeta, localMd5) // data in supernode is not right | ||
| && !StringUtils.equals(localMd5, info.getExpectedMd5())) { |
There was a problem hiding this comment.
i think if data in supernode is not right then we will remove the task from the super.
is condition !StringUtils.equals(localMd5, info.getExpectedMd5()) necessary?
There was a problem hiding this comment.
Before supernode completing downloading file from remote server, the meta data is not written into local disk, so the md5FromMeta may be null.
lowzj
force-pushed
the
handle-client-error
branch
2 times, most recently
from
e87d9a0 to
b0e4354
Compare
Signed-off-by: lowzj <[email protected]>
lowzj
force-pushed
the
handle-client-error
branch
from
b0e4354 to
ba6de46
Compare
|
LGTM. |
starnop
pushed a commit
to starnop/Dragonfly
that referenced
this pull request
Nov 27, 2019
feature: handle client error and repair
inoc603
pushed a commit
to inoc603/Dragonfly
that referenced
this pull request
Dec 23, 2019
feature: handle client error and repair
sungjunyoung
pushed a commit
to sungjunyoung/Dragonfly
that referenced
this pull request
May 8, 2022
* feat: create preheat_test.go Signed-off-by: zzy987 <[email protected]> * feat: add post request Signed-off-by: zzy987 <[email protected]> * fix: try to fix header name Signed-off-by: zzy987 <[email protected]> * fix: try fix page not found Signed-off-by: zzy987 <[email protected]> * fix: try to install kubectl in cdn pod Signed-off-by: zzy987 <[email protected]> * fix: remove sudo Signed-off-by: zzy987 <[email protected]> * chore: put curl in manager Signed-off-by: zzy987 <[email protected]> * fix: correct json content Signed-off-by: zzy987 <[email protected]> * fix: correct request url Signed-off-by: zzy987 <[email protected]> * chore: check curl result Signed-off-by: zzy987 <[email protected]> * fix: correct request url Signed-off-by: zzy987 <[email protected]> * chore: check the cmd string in github environment Signed-off-by: zzy987 <[email protected]> * fix: use service Signed-off-by: zzy987 <[email protected]> * fix: delete cluster-name Signed-off-by: zzy987 <[email protected]> * chore: use " in header Signed-off-by: zzy987 <[email protected]> * fix: curl command format Signed-off-by: zzy987 <[email protected]> * feat: use idgen.taskid to get the filename Signed-off-by: zzy987 <[email protected]> * chore: add an error message Signed-off-by: zzy987 <[email protected]> * feat: add concurrency test Signed-off-by: zzy987 <[email protected]> * fix: ab use proxy Signed-off-by: zzy987 <[email protected]> * feat: add curl pod Signed-off-by: zzy987 <[email protected]> * feat: add function CurlCommand Signed-off-by: zzy987 <[email protected]> * fix: variable usage Signed-off-by: zzy987 <[email protected]> * chore: remove curl in cdn pod Signed-off-by: zzy987 <[email protected]> * chore: modify code format Signed-off-by: zzy987 <[email protected]> * feat: use curl in file-server Signed-off-by: zzy987 <[email protected]> * fix: correct command in file-server Signed-off-by: zzy987 <[email protected]> * chore: add a e2e-test, check if it is the reason of no preheat Signed-off-by: zzy987 <[email protected]> * chore: can test preheat now Signed-off-by: zzy987 <[email protected]> * fix: check result after concurrency test Signed-off-by: zzy987 <[email protected]> * chore: add comments, add function check preheat result Signed-off-by: zzy987 <[email protected]> * chore: delete e2e_test in e2e/manager Signed-off-by: zzy987 <[email protected]> * chore: refactor curlCommand, add func getCDNs getFS Signed-off-by: zzy987 <[email protected]> * feat: add image preheat Signed-off-by: zzy987 <[email protected]> * chore: rename functions Signed-off-by: zzy987 <[email protected]> * fix: delete functions, use stable task id in test Signed-off-by: zzy987 <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: lowzj [email protected]
Ⅰ. Describe what this PR did
dfget reports downloading errors to supernode, supernode checks and repairs local data automatically, currently repair these client errors:
FILE_NOT_EXISTFILE_MD5_NOT_MATCHadd a new api into supernode to delete the task's data manually if any error occurred
client-side pr: #583
Ⅱ. Does this pull request fix one issue?
fixes #545
fixes #311
fixes #68
Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews