Question: production use

[dineshbvadhia]

Hi. I have previously used ipyparallel successfully on a multicore laptop and then ran the same code on a 1024 node HPC. The next stage is to run the code on multicore server machines. In this case, the software would be packaged to be distributed to different organizations and run as an interactive production system. The question is whether ipyparallel is suitable for unmanaged production environments?

[minrk]

I'm not sure I can really answer that. It's open source so comes entirely without warrantee or support, etc. and is not a super active project, but it is used in production by a number of folks.

Depending on what you are using it for, dask distributed is often a more mature and sophisticated tool.to choose. These days, IPython Parallel is most appropriate for MPI-style tasks (especially actual literal MPI).

[dineshbvadhia]

@minrk Don't worry, wasn't looking for a warrantee :). Good to know some folks are using ipyparallel in production.

What I like about ipyparallel is the 'real' cluster computing with stay-alive workers which can be created on each node (core). In my case, each worker runs an http-server acting on incoming requests from the master. MPI is a nice to have but as long as an equivalent gather-communication facility is available then good. Is this possible with dask-distributed?

[minrk]

Yes, absolutely. Dask distributed can have long-running workers on many nodes. It also has a much more sophisticated understanding of workers and threads (e.g. you can have one worker on a machine with 8 threads and it will use all resources appropriately).

[dineshbvadhia]

@minrk Dask Distributed sounds promising and will look into it. Will be sad to leave ipyparallel though as it has served me incredibly well :(.

On a related subject: Are you aware of similar cluster computing functionality written in rust preferably available through python?

[minrk]

Don't get me wrong, you're welcome to use IPython Parallel if you're happy with it and it's been working well for you. These days, dask is just a more mature tool used very widely in production.

I'm not up on the rust ecosystem, so I'm not sure on that one.