fluent freezes suddenly when running on HPC using pyFluent

[abbas-Aldor]

When I submit the job, everything starts as expected. A Fluent log file (".trn") is automatically generated, which contains console log information, including details about specifying processors, reading the mesh, and starting the simulation. By printing this file to the screen using "cat ...", I can observe the progress of the iterations. However, the simulation consistently freezes around iteration 350 (though the exact iteration varies slightly across attempts). At this point, no additional output is printed—neither warnings nor errors—leaving the last recorded line as the most recent iteration. It’s worth noting that the Slurm job time is not exceeded, and the job itself continues to run.

[ansjschuetz]

When it's hung like that, can you, at least on the machine running the node-0 process...

• check memory status (still sufficient free memory available, or is it maybe already using swap space??)
• maybe attach "strace -p " to the node-0 process, just to see whether it's actually hung in parallel communication or still working?
  If it's indeed hung in parallel communication, check all other processes the same way. If they're all healthy, yet hung in parallel communication, then you've probably run into a bug...

[abbas-Aldor]

• check memory status (still sufficient free memory available, or is it maybe already using swap space??)
• maybe attach "strace -p " to the node-0 process, just to see whether it's actually hung in parallel communication or still working?
  If it's indeed hung in parallel communication, check all other processes the same way. If they're all healthy, yet hung in parallel communication, then you've probably run into a bug...

Thank you @ansjschuetz for your reply.

• I checked the memory status, there is enough memory available
• node-0 is still working, but with low percentage (dropped to 20% of usage maybe it is for mpi communication with other nodes), and all other nodes are the same.

I saw in another blogs ([link] #1628, and [link] https://discuss.ansys.com/discussion/2680/pyfluent-for-running-jobs-on-hpc-using-pbs-scheduler) that maybe this was a bug and solved? I am not sure what to do to solve this problem

Regards,
Abbas

[seanpearsonuk]

@abbas-Aldor Please create a GitHub issue with all detailed information that could help to reproduce the issue.