enhancement: addresses #29 using vectorization instead of multiprocessing #132
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
From #29 :
TLDR:
multprocessing
might be overkill for this task. Vectorisation usingnumpy
arrays works. I've swapped the lists forndarray
where possible and it's worked. Run the test usingpytest
to show the results are the same and the newBiaxialBendingResults.get_results_list()
is faster.Why
multiprocessing
might not workI feel the need to explain this because my solution veers away from your original idea of parallelising the operation.
I've worked quite a bit with async in web servers so I have a good idea of how multiprocessing works, including in Python. Compared to the database operations and handling http requests, accessing the contents of a list are quite small. So small that spreading out the processes using a pool might make the operation run even slower than it already is.
Vectorisation with
numpy
arraysVector operations will give you that sense of "parallel computing" without the overhead of spreading the tasks over CPU cores. The simplest way to do this is with numpy arrays, i.e changing from this
to this
works well enough because:
Here's a github gist of the tests I ran with random results data between 1 result and 1,000,000. The benchmark is well into the 10,000x speed up for thousands of results.
There's a link to the Colab Notebook so you can run the test yourself.
I am in the process of doing something similar for
MomentInteractionResults.get_results_list()
. Now, I would have added the implementations ofmultiprocessing
that I tried out but I felt it unnecessary given the explanations above, but you're welcome to ask for them. I like this library very much btw.]
tests