Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Draft: Parsers returns multiple items & Reuse Items buffer instead of streams #2233

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

Draft: Parsers returns multiple items & Reuse Items buffer instead of streams #2233

wants to merge 1 commit into from

Conversation

AmmarAbouZor
Copy link
Member

  • This PR is a draft to demonstrate a proposal to change producer loop having one buffer for parsed items that will be cleared and refilled on each iteration to reuse memory.
  • This change is removing the stream API for the producer because the new logic will provide the buffer to fill instead of pulling the items from the producer.
  • DLT parser is changed to parse all possible bytes returning multiple items instead of one item only

@AmmarAbouZor AmmarAbouZor force-pushed the draft-remove-stream branch 7 times, most recently from 29f512b to 394b9cd Compare February 20, 2025 15:41
@AmmarAbouZor
Buffer in producer & DLT parse multiple items.
c433be2 
Producer:
* Use internal buffers for the parsed items to reuse the memory avoiding
  allocating new memory on each call.
* Removed stream interface to avoid getting into multiple mutable
  reference for producer and its buffer at the same time.
* Fix all unit tests ensuring all of them are passing without any change
  in their core functionality.
* Remove index counter because it's not used anywhere.

DLT Parser:
* Parse all provided bytes returning all available items until hitting
  the first error.

Chipmunk-CLI:
* Integrate changes in producer
* Ensure all external dependencies are used.

Indexer-CLI:
* Integrate changes in producer.

Benchmarks:
* Fix parser multiple time to return an iterator instead vector to avoid
  allocating memory.

TODOs:
* Add temp timer for benchmarking.
* Old DLT parse code still commented out for comparison.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@AmmarAbouZor