Improve invoker observability #2779
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
muhamadazmy
force-pushed
the
pr2779
branch
2 times, most recently
from
b6a361e to
25d2e6c
Compare
muhamadazmy
force-pushed
the
pr2779
branch
5 times, most recently
from
f203158 to
97e6cba
Compare
tillrohrmann
approved these changes
Mar 3, 2025
There was a problem hiding this comment.
Thanks for this improvement @muhamadazmy. LGTM. +1 for merging :-)
Comment on lines
+35
to
+37
| INVOKER_SEG_QUEUE_LEN, | ||
| Unit::Count, | ||
| "Number of invocations in the queue" |
There was a problem hiding this comment.
Maybe we call it pending invoker tasks to align it with the INVOKER_INVOCATION_TASK?
muhamadazmy
force-pushed
the
pr2779
branch
3 times, most recently
from
075fd92 to
04caf76
Compare
Summary: FindTailAttr allows find_tail users to adjust durability and accuracy. Currently, two modes are supported: Approximate: A fast check that usually returns immediately with the last known tail. Durable (default): Ensures a reliable tail find. Loglet implementations can choose to always run Durable find_tail.
This PR improves only on the PP <-> Bifrost observability. We already collect meterics for both PP and Bifrost but I think those extra metrics can be useful - Utilization of the PP requests queue. Once full the PP will start dropping requests as 'Busy' and I thought it's important to measure how utilized this queue is - Commit to read latency of records. Measures the latency between the moment a record is committed until it's read by the PP - LSN lag (applied to the log tail) This is also shown in `resatectl partition list` Fixes restatedev#2756
Add few extra metrics to measure number of queued commands, and number of in flight invocation tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Improve invoker observability
Add few extra metrics to measure number of queued commands,
and number of in flight invocation tasks
Stack created with Sapling. Best reviewed with ReviewStack.