Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mark partition as busy when a new batch is sent to it #281

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

Mark partition as busy when a new batch is sent to it #281

wants to merge 10 commits into from

Commits on May 29, 2017

  1. Set partition as busy when a new batch is sent to it. 

    This prevent a spider from being overwhelmed by requests because it
takes too long to process a batch. The DB Worker will be able to send
a new batch once a crawler requests more requests and send the offset
message to update its state to the DB Worker.
    @isra17
    isra17 committed May 29, 2017
    Configuration menu
    Copy the full SHA
    174b162 View commit details
    Browse the repository at this point in the history

  2. Add unittest for busy partition on new_batch

    @isra17
    isra17 committed May 29, 2017
    Configuration menu
    Copy the full SHA
    2f7e3cd View commit details
    Browse the repository at this point in the history

  3. Remove debug log

    @isra17
    isra17 committed May 29, 2017
    Configuration menu
    Copy the full SHA
    d4aa8db View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2017

  1. Track lag by regularly checking consumer and producer lag

    @isra17
    isra17 committed Jul 4, 2017
    Configuration menu
    Copy the full SHA
    0b646e4 View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2017

  1. Fix partition test

    @isra17
    isra17 committed Jul 6, 2017
    Configuration menu
    Copy the full SHA
    dacd2b8 View commit details
    Browse the repository at this point in the history

  2. revert travis

    @isra17
    isra17 committed Jul 6, 2017
    Configuration menu
    Copy the full SHA
    f7d2cd0 View commit details
    Browse the repository at this point in the history

  3. Review code

    @isra17
    isra17 committed Jul 6, 2017
    Configuration menu
    Copy the full SHA
    64b61de View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2017

  1. Init partitions_offset dict

    @isra17
    isra17 committed Jul 11, 2017
    Configuration menu
    Copy the full SHA
    58ff5f4 View commit details
    Browse the repository at this point in the history

  2. Handle new partition

    @isra17
    isra17 committed Jul 11, 2017
    Configuration menu
    Copy the full SHA
    d2ef60e View commit details
    Browse the repository at this point in the history

Commits on Jul 21, 2017

  1. Fix bug where no spider is running

    @isra17
    isra17 committed Jul 21, 2017
    Configuration menu
    Copy the full SHA
    a848c73 View commit details
    Browse the repository at this point in the history