Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add a "size" option to limit CSV lines #4

Open
fbaligand opened this issue Oct 21, 2020 · 2 comments
Open

add a "size" option to limit CSV lines #4

fbaligand opened this issue Oct 21, 2020 · 2 comments
Labels
enhancement New feature or request

Comments

@fbaligand
Copy link

Add a "size" option to be able to not dump all the documents, but a limited size.
This is particularly useful when for tests, but also to be able to control max file size and max batch time.
@fabiopipitone fabiopipitone added the enhancement New feature or request label Oct 21, 2020
@fabiopipitone
Copy link
Owner

Working on it and I guess it'll be available in the next release. Had to find a workaround to use it with multiprocessing mode and I had to force the users to use the load_balanced suprocess splitting if they want to mix the multiprocessing with the query_size limitation.

@fbaligand
Copy link
Author

Nice!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@fbaligand @fabiopipitone