[Bifrost] Design improvements for find_tail #2593
Conversation
Test Results 7 files ±0 7 suites ±0 3m 32s ⏱️ -55s Results for commit 638c13b. ± Comparison against base commit 3b7fd88. This pull request removes 2 tests.♻️ This comment has been updated with latest results. |
AhmedSoliman
changed the title
This PR introduces a few changes resulting in significant improvements in failover time, and performance of common operation like find_tail(). The result is failover time that in the hundreds of milliseconds in the happy path and in the order of a couple of seconds in the unhappy path. `find_tail()` is also now significantly cheaper to run if the sequencer is running, this enables parallelization of `find_tail()` runs in logs controller (and more frequent as well). The latter comment will be reflected in a separate PR.
|
Five minute test with random partitions looks good! I will re-run it a few more times to see how it behaves. https://github.com/restatedev/jepsen/actions/runs/13080247655/job/36501963775
The gaps during the partitions (grey bars) indicate that no processing seems to be happening, and sometimes we don't recover for a couple of on/off cycles - so 15-25s since the first partition event. I added some clients-side timeouts to the test driver to ride out some of the short-term unavailability but I'd want to double check that I'm not starving out Jepsen's worker threads with these. I was under the impression that there is dedicated concurrency per Restate worker node. Some more analysis required but at first glance it looks like a big improvement over before with no long-term lockups. |
This PR introduces a few changes resulting in significant improvements in failover time, and performance of common operation like find_tail().
The result is failover time that in the hundreds of milliseconds in the happy path and in the order of a couple of seconds in the unhappy path.
find_tail()is also now significantly cheaper to run if the sequencer is running, this enables parallelization offind_tail()runs in logs controller (and more frequent as well). The latter comment will be reflected in a separate PR.