Adding CCL Async test cases to TG nightly and bug fix #16700
Conversation
…assignment + teardown
caixunshiren
requested review from
SeanNijjar,
jvegaTT,
tt-aho and
a team
as code owners
tests/ttnn/unit_tests/operations/ccl/test_all_gather_async_TG_nightly.py
Outdated
Show resolved
Hide resolved
tests/ttnn/unit_tests/operations/ccl/test_all_gather_async_TG_nightly.py
Outdated
Show resolved
Hide resolved
tests/ttnn/unit_tests/operations/ccl/test_reduce_scatter_async_TG_nightly.py
Outdated
Show resolved
Hide resolved
| if enable_persistent_fabric: | ||
| ttnn.synchronize_devices(mesh_device, sub_device_ids=sub_device_stall_group) |
There was a problem hiding this comment.
Was this added for debug or what's the purpose of this? Should be removed?
There was a problem hiding this comment.
This is needed for multi-iteration to prevent race condition, when persistent fabric is enabled. One device might get too fast and increment the semaphore on the slow device for the next call of ccl before the slow device reset the semaphore for previous iteration. There seems to be no good way to prevent it at the moment unless a sync is introduced or a new global semaphore is used every time (but that's essentially a sync).
There was a problem hiding this comment.
Is this something that should be documented for consumers of the API? Seems like an important detail
There was a problem hiding this comment.
This is a separate issue from #16634?
ttnn/cpp/ttnn/operations/experimental/ccl/all_gather_async/device/all_gather_async_program.cpp
Outdated
Show resolved
Hide resolved
Overview
Known Issues
Several issues are exposed in CCLs by these test cases. These failing ones are commented out for now:
Checklist