Set CHPL_RT_MASTERIP in hopes that it will help with our recent gasnet timeouts #26714
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t timeouts Recently, GASNet timeouts have been suffering from an error: ``` *** GASNET WARNING: AMUDP_SPMDStartup_AMUDP_NDEBUG returning an error code: AM_ERR_RESOURCE (Problem with requested resource) from function AMUDP_SPMDStartup at /ptmp/jenkins/chapel-ci/workspace/chapcs-correctness-test-gasnet-everything/third-party/gasnet/gasnet-src/other/amudp/amudp_spmd.cpp:1026 reason: worker failed DNSLookup on master host name ``` Michael hypothesizes that setting CHPL_RT_MASTERIP may help with this failure mode, and I think it's worth a try, so this adds it to common-oversubscribed.bash to see what happens. --- Signed-off-by: Brad Chamberlain <[email protected]>
|
@mppf / @riftEmber / @e-kayrakli / @tzinsky : Can one or more of you give this a look and see what you think? I'm not confident it will help, but if this file is used whenever we oversubscribe GASNet it shouldn't hurt. |
Michael points out that "oversubscribed" here isn't particularly GASNet-specific, and generally applies to running multiple things on a node simultaneously (potentially multiple single-locale Chapel programs, potentially programs using other CHPL_COMM layers), and that common-gasnet.bash contains code that's more specific to local, oversubscribed GASNet launching, which is a great point — so moving the setting there instead to be more precise. --- Signed-off-by: Brad Chamberlain <[email protected]>
mppf
approved these changes
Feb 14, 2025
--- Signed-off-by: Brad Chamberlain <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Recently, GASNet test configurations have been suffering from a timeout error across multiple tests that manifests as:
Michael hypothesized that setting CHPL_RT_MASTERIP may help with this failure mode, and I agree it's worth a try, so this adds it to common-gasnet.bash (where we also set up local launching and the like) to see what happens.
This could help with resolving https://github.com/Cray/chapel-private/issues/7062, but I'm not confident.