resource tracker and train process kill

[Feng204825]

Describe the bug
During the fine-tuning process of the model, there is a resource leak issue that causes the training process to be terminated. Initial investigation shows that resources are sufficient. I would appreciate your help in determining the cause of this issue.

To Reproduce
Please provide a code snippet of a minimal reproducible example for the error.

python -m cli.train -cp conf/finetune run_name=example_run model=moirai_1.1_R_small data=etth1 val_data=etth1  

Expected behavior
During the fine-tuning process of the model, there is a resource leak issue that causes the training process to be terminated. Initial investigation shows that resources are sufficient. I would appreciate your help in determining the cause of this issue.

Environment

• Operating system: Ubuntu 20.04.6 LTS
• Python version: Python 3.10.16
• PyTorch version:torch 2.4.1
• uni2ts version:1.2.0

[liuxu77]

Hi @Feng204825, thanks for asking this question. I don't think this is a problem, I guess it's because you terminated the last command and started a new one immediately, causing some resources not to be fully released. To prevent this, after you stop the program, wait for all resources to be released before starting the next one. Hope this helps.