Does the new profiler api of profiler plugin save the dump file when app is completed?

[Xiaoaier-Z-L]

when i try the profile plugin with nccl's example, i found the dump file is saved when my app is completed. can i profile it when my app is running? for example, i use pytorch for distributed training, but it may need one month's time, how can i record the op times by profile plugin?

[gcongiu]

Yes, the example profiler plugin saves the traces collected when the communicator is finalized. You can extend the example to dump traces to a file at regular intervals while they are generated instead of doing it at comm finalize.

[Xiaoaier-Z-L]

Understood, thank you very much for your prompt response: this is extremely helpful to me!

[Xiaoaier-Z-L]

When using the example of the profile plugin to record traces, I noticed that it always saves exactly 65 lines of content, even after increasing the training time or iteration count. Since I'm not yet familiar with the principles of the profile plugin and am learning through the example, this behavior feels puzzling to me.
Is the trace output truncated by default?

[gcongiu]

The example plugin only stores a limited number of traces to keep memory low. You can increase the event pool sizes through env variable. Please find more info in the documentation https://github.com/NVIDIA/nccl/tree/master/ext-profiler/example#changing-the-profiler-memory-pool-sizes