Uncertainpy model run in hpc clusters

[anirban6908]

Hi,
The uncertainpy calculation quickly goes out of control with increasing no. of uncertain parameters mainly in terms of memory requirement for any personal machines. I could be wrong but don't think the multiprocessing module is suitable for distributed computing? I was thinking about taking a pass at run_model.py to configure the map function to accept parallelization frameworks such ipyparallel or mpi4py? Are there obvious issues in doing that?

Thanks

[simetenn]

Hi!

It sounds like a good idea. In itself, the parallelization of the model evaluations is trivial, as each model evaluation is independent of each other.

Uncertainpy uses multiprocess which is a fork of multiprocessing, developed as a part of pathos https://github.com/uqfoundation/multiprocess. I am not certain if multiprocess itself can be used for distributed computing. However, pathos itself is a framework for heterogeneous computing, so it is worth looking into to see if Pathos/multiprocess can be used.

One thing that can complicate things is that Uncertainpy passes objects around, and not only functions (this is something that multiprocess supports, while multiprocessing doesn't). This can perhaps complicate the parallelization depending on the parallelization framework used.

Do not hesitate to ask if you have questions.