[Stateful DL] Add out of order implementation #1423
Conversation
|
Hi @michael-diggin! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at [email protected]. Thanks! |
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks! |
facebook-github-bot
added
the
CLA Signed
|
@michael-diggin thanks for making this PR! Can add a warning on the first call to |
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/data/1423
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit 9a43220 with merge base d6232f1 ( FLAKY - The following job failed but was likely due to flakiness present on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
No problem, done. I've gone with having it log on every call, but happy to add some state to track if it's the first call and to check that/only log once. |
|
Hi @andrewkho, just bumping this - I’ve made the changes above. It would be great to get another review if you get a chance this week! |
|
sorry for the delay @michael-diggin , let's just roll it out since we're going to cut release branch soon. I'll start the CI and once it's green we can merge. Thank you again! |
Sounds good, thanks! |
|
I think there might be an issue with the worker that is running one of the tests - it hasn't made progress in about 90 mins |
FYI @andrewkho, I would think it just needs the CI to be rerun? |
|
@michael-diggin I think we're good to land, I'll go ahead and merge, thanks again for the contribution! |
Fixes #1414
Changes
The tests I've added were the exact same ones from the main PyTorch repo, so the functionality of the flag is the exact same. There was a few changes required to be compatible with the few slight differences of the
_StatefulMultiProcessingDataLoaderIter.I have also tested some simple cases of stopping/resuming both an index and iterable ds and it seems to work correctly (the same types that are used in the new test cases). I haven't added extra tests for that into this PR just to keep the size manageable, but would add them in a follow up PR as well as checking extra edge cases.