Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[stats] Rewrite stat management to use single threaded event loop #8815

Open
wants to merge 118 commits into
base: main
Choose a base branch
from

Conversation

max-hoffman
Copy link
Contributor

@max-hoffman max-hoffman commented Feb 4, 2025

Reference statspro/doc.go for more detailed overview. Replaced stats management with an event loop and shared bucket cache. Rudimentary GC and branch syncing seems like a must for thousands of branches.

Branch add/delete seems to work as I'd expect under high concurrency. The GC and branch syncing concurrency is a bit sloppy right now. It doesn't deadlock, but thousands of concurrent adds a second while running GC/branch sync every <10ms doesn't true up the way I would want.

Early perf testing seems to have little/no impact on TPC-C when the job ticker is 100-500ms. Every few minutes during TPC-C stats throws spurious errors that appear to self-resolve. It doesn't exactly clutter output, but I think users would be happier if I figured how out how to reduce those errors.

fixes: #8844

@coffeegoddd
Copy link
Contributor

@max-hoffman DOLT

comparing_percentages
100.000000 to 100.000000
version result total
4e12e77 ok 5937457
version total_tests
4e12e77 5937457
correctness_percentage
100.0

@coffeegoddd
Copy link
Contributor

@max-hoffman DOLT

comparing_percentages
100.000000 to 100.000000
version result total
10f8bc3 ok 5937457
version total_tests
10f8bc3 5937457
correctness_percentage
100.0

@coffeegoddd
Copy link
Contributor

@max-hoffman DOLT

comparing_percentages
100.000000 to 100.000000
version result total
10e67a7 ok 5937457
version total_tests
10e67a7 5937457
correctness_percentage
100.0

@coffeegoddd
Copy link
Contributor

@max-hoffman DOLT

comparing_percentages
100.000000 to 100.000000
version result total
58879d6 ok 5937457
version total_tests
58879d6 5937457
correctness_percentage
100.0

@coffeegoddd
Copy link
Contributor

@max-hoffman DOLT

comparing_percentages
100.000000 to 100.000000
version result total
0e4c2e5 ok 5937457
version total_tests
0e4c2e5 5937457
correctness_percentage
100.0

@coffeegoddd
Copy link
Contributor

@max-hoffman DOLT

comparing_percentages
100.000000 to 100.000000
version result total
bb69370 ok 5937457
version total_tests
bb69370 5937457
correctness_percentage
100.0

@coffeegoddd
Copy link
Contributor

@coffeegoddd DOLT

comparing_percentages
100.000000 to 100.000000
version result total
687f497 ok 5937457
version total_tests
687f497 5937457
correctness_percentage
100.0

@coffeegoddd
Copy link
Contributor

@max-hoffman DOLT

comparing_percentages
100.000000 to 100.000000
version result total
7864cf1 ok 5937457
version total_tests
7864cf1 5937457
correctness_percentage
100.0

@coffeegoddd
Copy link
Contributor

@max-hoffman DOLT

comparing_percentages
100.000000 to 100.000000
version result total
6c8a0db ok 5937457
version total_tests
6c8a0db 5937457
correctness_percentage
100.0

@coffeegoddd
Copy link
Contributor

@max-hoffman DOLT

comparing_percentages
100.000000 to 100.000000
version result total
9e15822 ok 5937457
version total_tests
9e15822 5937457
correctness_percentage
100.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Panic from vector index (possibly from stats)
3 participants