Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DAOS-16686 dfuse: check the end offset for concurrent fetch #15719

Merged
merged 17 commits into from
Jan 17, 2025
Merged

DAOS-16686 dfuse: check the end offset for concurrent fetch #15719

merged 17 commits into from
Jan 17, 2025

Conversation

wangdi1
Copy link
Contributor

@wangdi1 wangdi1 commented Jan 11, 2025

It should check the end offset of fetch during inflight fetch check.

Run-GHA: true
Skip-func-hw-test: true
Required-githooks: true

Before requesting gatekeeper:

  • Two review approvals and any prior change requests have been resolved.
  • Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
  • Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
  • Commit messages follows the guidelines outlined here.
  • Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

  • You are the appropriate gatekeeper to be landing the patch.
  • The PR has 2 reviews by people familiar with the code, including appropriate owners.
  • Githooks were used. If not, request that user install them and check copyright dates.
  • Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
  • All builds have passed. Check non-required builds for any new compiler warnings.
  • Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
  • If applicable, the PR has addressed any potential version compatibility issues.
  • Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
  • Extra checks if forced landing is requested
    • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
    • No new NLT or valgrind warnings. Check the classic view.
    • Quick-build or Quick-functional is not used.
  • Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

@wangdi1
DAOS-16686 dfuse: check the end offset for concurrent fetch
d1dbe4b 
It should check the end offset of fetch during inflight
fetch check.

Run-GHA: true
Skip-func-hw-test: true
Required-githooks: true

Signed-off-by: Di Wang <[email protected]>
@wangdi1 wangdi1 requested a review from jolivier23 January 11, 2025 03:51
@github-actions GitHub Actions
Copy link

Ticket title is 'Concurrent reads hit the network even when caching enabled in dfuse'
Status is 'In Progress'
Labels: 'google-cloud-daos'
https://daosio.atlassian.net/browse/DAOS-16686

jolivier23
jolivier23 previously approved these changes Jan 11, 2025
View reviewed changes
@jolivier23
Test flags
44550bc 
Skip-func-test-hw: false
Allow-unstable-test: true
Test-tag: dfuse

Required-githooks: true

Change-Id: I23f98548260f3bea3c9bf954bac0dbb937b0eb7a
Signed-off-by: Jeff Olivier <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15719/3/execution/node/1234/log

@wangdi1
DAOS-16686 dfuse: pick the right eqt for pre-read
b665230 
Pick the right eqt to do progress for pre read and
do not need pre-read if open by O_TRUNC.

Required-githooks: true
Skip-func-test-hw: false
Allow-unstable-test: true
Test-tag: dfuse

Signed-off-by: Di Wang <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15719/5/testReport/

jolivier23
jolivier23 reviewed Jan 12, 2025
View reviewed changes
src/client/dfuse/ops/read.c
sem_post(&eqt->de_sem);

/* Now ensure there are more descriptors for the next request */
d_slab_restock(eqt->de_read_slab);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you restock elsewhere?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

restock is actually not needed in this path, since no RPC is issued, so no buffer needed.

jolivier23
jolivier23 reviewed Jan 12, 2025
View reviewed changes
src/client/dfuse/ops/open.c Outdated
@@ -1,5 +1,5 @@
/**
* (C) Copyright 2016-2024 Intel Corporation.
* (C) Copyright 2016-2025 Intel Corporation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should also be 2024 and add a google one. If you pull in latest google/2.6, it will do this for you and stop changing intel one to 2025

jolivier23
jolivier23 reviewed Jan 12, 2025
View reviewed changes
src/client/dfuse/ops/read.c Outdated
@@ -1,5 +1,6 @@
/**
* (C) Copyright 2016-2024 Intel Corporation.
* (C) Copyright 2016-2025 Intel Corporation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be 2024

@wangdi1
DAOS-16686 tests: remvoe pre-read counter
72cb36a 
remove pre read counter check.

Required-githooks: true
Skip-func-test-hw: false
Allow-unstable-test: true
Test-tag: dfuse

Signed-off-by: Di Wang <[email protected]>
@wangdi1
DAOS-16686 dfuse: fix copyright
3b7ca05 
fix copyright

Required-githooks: true
Skip-func-test-hw: false
Allow-unstable-test: true
Test-tag: dfuse

Signed-off-by: Di Wang <[email protected]>
jolivier23
jolivier23 reviewed Jan 13, 2025
View reviewed changes
utils/node_local_test.py Outdated
@@ -1,7 +1,9 @@
#!/usr/bin/env python3
"""Node local test (NLT).

(C) Copyright 2020-2024 Intel Corporation.
(C) Copyright 2020-2025 Intel Corporation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
(C) Copyright 2020-2025 Intel Corporation.
(C) Copyright 2020-2024 Intel Corporation.

jolivier23
jolivier23 reviewed Jan 13, 2025
View reviewed changes
utils/node_local_test.py
@@ -1,7 +1,8 @@
#!/usr/bin/env python3
"""Node local test (NLT).

(C) Copyright 2020-2025 Intel Corporation.
(C) Copyright 2020-2024 Intel Corporation.
(C) Copyright 2025 Google LLC
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
(C) Copyright 2025 Google LLC
(C) Copyright 2025 Google LLC

@jolivier23
Update utils/node_local_test.py
d1125d9 
Signed-off-by: Jeff Olivier <[email protected]>
@jolivier23
Allow-unstable-test: true
b65217c 
Test-tag: dfuse
Skip-func-test-hw: false

Required-githooks: true

Change-Id: I34611aee171d830464fec39ccd0ddf901166b632
Signed-off-by: Jeff Olivier <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15719/11/execution/node/1235/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15719/11/execution/node/1504/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15719/11/execution/node/1525/log

@wangdi1
DAOS-16686 dfuse: revert page-size alignment inflight check.
973fc51 
Revert size adjust for inflight read check since IL
does not do page size alignment.

Allow-unstable-test: true
Test-tag: dfuse
Skip-func-test-hw: false

Required-githooks: true

Signed-off-by: Di Wang <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15719/12/testReport/

@wangdi1
DAOS-16686 tests: remove pre-read check
0b15886 
remove pre read check

Required-githooks: true
Allow-unstable-test: true
Test-tag: dfuse
Skip-func-test-hw: false

Signed-off-by: Di Wang <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15719/13/testReport/

@wangdi1
DAOS-16686 dfuse: disable the pre-read check for now
95a1dd6 
Disable the pre-read counter check for now, since we
changed the pre-read behavior a bit, and will revisit
these counter check later.

Required-githooks: true
Allow-unstable-test: true
Test-tag: dfuse
Skip-func-test-hw: false

Signed-off-by: Di Wang <[email protected]>
@wangdi1
DAOS-16686 dfuse: disable the pre-read check for now
172af4a 
Disable the pre-read counter check for now, since we
changed the pre-read behavior a bit, and will revisit
these counter check later.

Required-githooks: true
Allow-unstable-test: true
Test-tag: test_dfuse_pre_read DaosBuild
Skip-func-test-hw: false

Signed-off-by: Di Wang <[email protected]>
@wangdi1
Merge remote-tracking branch 'refs/remotes/origin/wangdi/google_dfuse…
6341cba 
…_fix' into wangdi/google_dfuse_fix

Required-githooks: true
Allow-unstable-test: true
Test-tag: test_dfuse_pre_read DaosBuild
Skip-func-test-hw: false
@wangdi1
Merge branch 'google_2.6' into wangdi/google_dfuse_fix
f500d31 
Required-githooks: true
Allow-unstable-test: true
Test-tag: test_dfuse_pre_read DaosBuild
Skip-func-test-hw: false
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15719/19/testReport/

@jolivier23
Update read.py
9615c15 
Just fix the pylint errors

Signed-off-by: Jeff Olivier <[email protected]>
@jolivier23
Merge branch 'google/2.6' into wangdi/google_dfuse_fix
f24bc18 
Signed-off-by: Jeff Olivier <[email protected]>
@jolivier23 jolivier23 merged commit 0405fff into google/2.6 Jan 17, 2025
40 of 49 checks passed
@jolivier23 jolivier23 deleted the wangdi/google_dfuse_fix branch January 17, 2025 17:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jolivier23 jolivier23 jolivier23 approved these changes

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@wangdi1 @daosbuild1 @jolivier23