Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance of query when seq is far behind log #206

Merged
merged 2 commits into from
Apr 4, 2022

Conversation

arj03
Copy link
Member

@arj03 arj03 commented Apr 2, 2022

I noticed this when doing a query: first jitdb updates seq and only after that, it updates what is needed for the query. There is an assumption that seq is in sync with the log after running a query, but instead of eagerly updated only seq index and then the rest. Why not try updating the indexes we need and only after that seq if needed. In most cases the seq update will then be a no op.

Running my normal 1GB query on empty db this is 15.7s compared to 17.6s before. I know this is a bit of a worst case example and in most real cases the difference won't be very big as seq is often very close to being in sync with the log because it is updated on any query.

package.json Show resolved Hide resolved
@arj03 arj03 marked this pull request as draft April 2, 2022 09:16
@arj03 arj03 closed this Apr 2, 2022
@arj03 arj03 mentioned this pull request Apr 2, 2022
@arj03 arj03 reopened this Apr 2, 2022
@github-actions
Copy link

github-actions bot commented Apr 2, 2022

Benchmark results

Part Speed Heap Change Samples
Count 1 big index (3rd run) 0.53ms ± 0.18ms 3.09 kB ± 13.97 kB 54
Create an index twice concurrently 583.53ms ± 5.31ms -3.67 kB ± 28.84 kB 94
Load core indexes 0.86ms ± 0.01ms 86.83 B ± 181.41 B 9134
Load two indexes concurrently 446.05ms ± 12.36ms 32.74 kB ± 120.95 kB 22
Paginate 10 results 22.78ms ± 1.3ms -588.48 B ± 18164.37 B 25
Paginate 20000 msgs with pageSize=5 6559.4ms ± 35.75ms 2.63 MB ± 2.36 MB 5
Paginate 20000 msgs with pageSize=500 591.57ms ± 8.12ms 114.45 kB ± 724.51 kB 18
Query 1 big index (1st run) 785.7ms ± 6.92ms -42.55 kB ± 60.34 kB 70
Query 1 big index (2nd run) 265.94ms ± 0.93ms -11 kB ± 15.2 kB 53
Query 3 indexes (1st run) 842.65ms ± 8.35ms 35.5 kB ± 108.34 kB 65
Query 3 indexes (2nd run) 224.56ms ± 3.19ms 42.3 kB ± 32.92 kB 55
Query a prefix map (1st run) 291.58ms ± 4.61ms 1.46 kB ± 428.3 kB 21
Query a prefix map (2nd run) 18.09ms ± 1.11ms 14.23 kB ± 56.67 kB 20

@arj03
Copy link
Member Author

arj03 commented Apr 2, 2022

This now also fixes #207

@arj03 arj03 marked this pull request as ready for review April 2, 2022 17:15
@ssbc ssbc deleted a comment from github-actions bot Apr 2, 2022
@arj03 arj03 mentioned this pull request Apr 3, 2022
Copy link
Member

@staltz staltz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems very correct! Thanks

@arj03 arj03 merged commit 6d62f3a into master Apr 4, 2022
@arj03 arj03 deleted the faster-empty-db-query branch April 4, 2022 07:24
@github-actions
Copy link

github-actions bot commented Apr 4, 2022

Benchmark results

Part Speed Heap Change Samples
Count 1 big index (3rd run) 0.35ms ± 0.03ms -1.54 kB ± 16.31 kB 53
Create an index twice concurrently 654.44ms ± 7.86ms 46.4 kB ± 45.5 kB 83
Load core indexes 1.4ms ± 0.02ms 99.68 B ± 238.91 B 6680
Load two indexes concurrently 534.72ms ± 11.52ms 88.9 kB ± 169.66 kB 19
Paginate 10 results 24.33ms ± 0.78ms -884.67 B ± 10334.01 B 24
Paginate 20000 msgs with pageSize=5 6754.43ms ± 100.09ms 2.88 MB ± 2.57 MB 5
Paginate 20000 msgs with pageSize=500 637.71ms ± 19.19ms -171.98 kB ± 437.17 kB 19
Query 1 big index (1st run) 736.19ms ± 9.13ms -44.16 kB ± 49.83 kB 75
Query 1 big index (2nd run) 305.08ms ± 3.26ms 6.77 kB ± 14.55 kB 42
Query 3 indexes (1st run) 783.27ms ± 8.84ms -40.11 kB ± 98.7 kB 70
Query 3 indexes (2nd run) 253.82ms ± 6.61ms 96.71 kB ± 43.74 kB 49
Query a prefix map (1st run) 286.2ms ± 6.32ms -214.29 kB ± 354.11 kB 21
Query a prefix map (2nd run) 16.5ms ± 0.91ms 4.24 kB ± 35.19 kB 21

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants