Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Taskflow map #323

Merged
merged 73 commits into from
Feb 25, 2025
Merged

Taskflow map #323

merged 73 commits into from
Feb 25, 2025

Conversation

ekg
Copy link
Collaborator

@ekg ekg commented Feb 23, 2025

Changes the mapper to use taskflow for efficient multithreading. Still WIP.

ekg added 30 commits February 19, 2025 15:36
@ekg
add taskflow
30dc7d3
@ekg
Based on the provided implementation, here's a concise commit message…
3b2130e 
… that captures the essence of the changes:

feat: Introduce Taskflow-based parallel execution for mapping
@ekg
feat: Add mapQueryTaskflow method declaration to Map class
5ef58d6
@ekg
refactor: Implement batch processing for query mapping with improved …
af163f8 
…parallelization
@ekg
feat: Add query batch size configuration to ComputeMap
728d494
@ekg
refactor: Integrate fragment processing into Taskflow using subflows …
fdb7e80 
…and parallel processing
@ekg
refactor: Improve Taskflow fragment processing with explicit tasks an…
5d64ee0 
…d dependencies
@ekg
refactor: Enforce serial batch processing in Taskflow query mapping
4b4f9a3
@ekg
fix: Resolve compilation errors in computeMap.hpp lambda captures and…
107e067 
… constructor
@ekg
fix: Resolve lambda capture and sequence retrieval issues in computeM…
38b8d45 
…ap.hpp
@ekg
fix: Correct function parameter order in computeMap.hpp
880a9ca
@ekg
fix: Correct subflow variable names and references in computeMap.hpp
293a4c5
@ekg
fix: Capture Taskflow subflow reference in lambda functions
24ed151
@ekg
refactor: Fix memory management in fragment processing
f9235de
@ekg
fix: Dereference fragment pointer in processFragment call
7157327
@ekg
refactor: Improve task synchronization with join and finalize tasks i…
a27ff49 
…n fragment processing
@ekg
refactor: Improve task parallelism with join and finalize tasks
26cc059
@ekg
remove reference to atomic queues
37510eb
@ekg
feat: Add strategic debug logging for mapping pipeline diagnostics
568f84a
@ekg
feat: Add debug logging to processFragment method in computeMap.hpp
41b62ed
@ekg
feat: Add detailed debug logging for fragment processing and queueing
fda0f4e
@ekg
fix: Use fragment index in debug message to resolve undefined variable
eaa0cc9
@ekg
fix: Add sf.join() to ensure subflow tasks execute in computeMap
df742f4
@ekg
refactor: Add explicit task dependencies and sf.join() to prevent con…
b36cf1b 
…currency memory issues
@ekg
refactor: Fix Taskflow subflow memory safety and task lifecycle issues
9d6f923
@ekg
fix: Ensure fragment tasks run by adding sf.join() in subflow callbacks
94f8e78
@ekg
feat: Add taskflow graph dump to stdout and task.dot file
b64eb0c
@ekg
refactor: Optimize Taskflow-based parallel mapping with improved memo…
44ad0ae 
…ry management and task dependencies
@ekg
fix: Resolve compilation errors in computeMap.hpp
aedbda1
@ekg
fix: Capture sf in lambda to resolve compilation error
7397a46
ekg added 29 commits February 23, 2025 19:45
@ekg
refactor: Improve memory management and prevent segmentation faults i…
2423078 
…n taskflow fragment processing
@ekg
fix: Refactor FragmentData creation and memory management
439fb88
@ekg
fix: Resolve shared_ptr and atomic initialization issues in computeMa…
154218c 
…p.hpp
@ekg
refactor: Update fragment initialization with zero atomic counter
689034e
@ekg
fix: Resolve FragmentData constructor parameter type mismatches
1084a74
@ekg
refactor: Update FragmentData constructor with explicit type conversi…
37776e2 
…ons and shared_ptr wrapping
@ekg
chore: Comment out L1 and L2 mapping debug logging statements
cca0086
@ekg
refactor: Remove debug logging statements from fragment processing
8a22f82
@ekg
refactor: Comment out [DEBUG] logging statements in computeMap.hpp
7d1370c
@ekg
style: Comment out debug logging statement in computeMap.hpp
e523c12
@ekg
refactor: Simplify query processing with nested subflow parallelism
3f24625
@ekg
fix: Capture subset_flow in lambda to resolve compilation error
fb2c4a5
@ekg
fix: Capture subset_flow by reference in lambda to resolve compilat…
c54ee77 
…ion error
@ekg
fix: Resolve lambda capture issues in nested taskflow lambdas
43cd780
@ekg
refactor: Remove unnecessary subset_flow capture in lambda
8315f9b
@ekg
refactor: Update subflow task creation and variable naming in query p…
804b4de 
…rocessing
@ekg
correctly use subflows in taskflow mapper
d611426
@ekg
Merge branch 'main' of github.com:waveygang/wfmash into taskflow-map
895b92c
@ekg
refactor: Restore scaffold mapping expansion and debug code
0e2ae38
@ekg
refactor: Simplify fragment processing with thread-local storage
0dc8aef
@ekg
refactor: Add thread-local results storage for fragment processing
8ccc233
@ekg
fix: Resolve thread-local variable capture and processFragment argume…
3e9afeb 
…nt issues
@ekg
fix: Resolve FragmentData constructor and result aggregation issues
4f2e311
@ekg
refactor: Update fragment processing and result handling in computeMap
4cbcfb5
@ekg
feat: Add optional counter to FragmentData constructor
e1b2fe5
@ekg
fix: Improve thread-local result merging in mapQueryTaskflow
bd2b824
@ekg
refactor: Remove redundant lambda capture of results_mutex
00c6b21
@ekg
refactor: Remove deprecated atomic queue-based coordination system
d420fd2
@ekg
refactor: Remove unused query batch size parameter and constant
6473fc0
@ekg ekg merged commit afd6f0f into main Feb 25, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@ekg