Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kineto collection of nccl:coalesced don't record metadata #178

Open
x41lakazam opened this issue Jan 7, 2025 · 0 comments
Open

Kineto collection of nccl:coalesced don't record metadata #178

x41lakazam opened this issue Jan 7, 2025 · 0 comments

Comments

@x41lakazam
Copy link

Chakra's kineto trace add metadata to the NCCL kernels, including message size, PG attributes (name, description, ranks, etc).
However in a trace I just recorded some kernels don't include any metadata.

The trace records gpt3/175b_fp8 training script from NeMo Launcher.

Here is the report of a normal all_reduce operation, we can see info about PG.
image

While here is the report of the nccl coalesced allgather, which displays no metadata:
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant