Parse GAF files in parallel #200
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The parallelizable way seems to be just as fast as the line-by-line parser. So no need for the sequential one, I guess?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is truly of dubious utility, but it was sorta fun to experiment with. Presumably, a similar parallel treatment could be applied to the GFA parser someday?
Parsing the GAF in parallel scales pretty well in my experiments. On my M1 Max laptop (10 cores total: 8 performance, 2 efficiency), the parallel version goes from 5.131 seconds to 608.0 ms, so a speedup of 8.4×. Or in terms of lines per second, that's from about 9.6 million to 80.6 million. Nice!
I can't pretend this is actually all that insightful or surprising, but it might be useful.