Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](Nereids) lock table in ascending order of table IDs (#45045) #45681

Merged
merged 1 commit into from
Dec 20, 2024

Conversation

morrySnow
Copy link
Contributor

pick from master #45045

Problem Summary:

Doris's table locks are fair read-write locks. If two threads acquire read locks on tables in different orders and simultaneously a third thread attempts to acquire a write lock on one of these tables, a deadlock can form between the two threads trying to acquire read locks. This PR changes the lock acquisition order for queries to follow the order of table IDs, ensuring that the lock acquisition order for tables is consistent among different threads.

Execute table locking operations in ascending order of table IDs

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

pick from master apache#45045

Problem Summary:

Doris's table locks are fair read-write locks. If two threads acquire
read locks on tables in different orders and simultaneously a third
thread attempts to acquire a write lock on one of these tables, a
deadlock can form between the two threads trying to acquire read locks.
This PR changes the lock acquisition order for queries to follow the
order of table IDs, ensuring that the lock acquisition order for tables
is consistent among different threads.

Execute table locking operations in ascending order of table IDs
@Thearas
Copy link
Contributor

Thearas commented Dec 19, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40259 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 188a85d649e97da2ce20c39acb94566279ef3b62, data reload: false

------ Round 1 ----------------------------------
q1	17575	7355	7284	7284
q2	2043	168	161	161
q3	10719	1078	1211	1078
q4	10546	711	793	711
q5	7758	2793	2794	2793
q6	238	148	145	145
q7	980	609	612	609
q8	9571	1962	1963	1962
q9	7838	6379	6400	6379
q10	7008	2250	2297	2250
q11	453	263	254	254
q12	394	211	206	206
q13	17779	2958	2944	2944
q14	241	216	209	209
q15	565	516	525	516
q16	670	592	636	592
q17	984	592	533	533
q18	7087	6463	6522	6463
q19	1378	1013	1040	1013
q20	464	202	193	193
q21	4036	3004	3208	3004
q22	1041	981	960	960
Total cold run time: 109368 ms
Total hot run time: 40259 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7344	7526	7181	7181
q2	333	227	232	227
q3	2892	2864	2835	2835
q4	1985	1748	1803	1748
q5	5672	5693	5697	5693
q6	218	140	138	138
q7	2240	1757	1772	1757
q8	3294	3517	3507	3507
q9	8702	8848	8789	8789
q10	3516	3505	3473	3473
q11	594	504	501	501
q12	791	582	613	582
q13	16309	3126	3130	3126
q14	294	265	284	265
q15	548	519	520	519
q16	726	670	664	664
q17	1854	1618	1629	1618
q18	8242	7713	7628	7628
q19	6214	1594	1637	1594
q20	2102	1862	1844	1844
q21	5291	5270	5250	5250
q22	1085	1016	1009	1009
Total cold run time: 80246 ms
Total hot run time: 59948 ms

@morrySnow morrySnow merged commit b6ca836 into apache:branch-3.0 Dec 20, 2024
22 of 24 checks passed
@morrySnow morrySnow deleted the 3.0_45045 branch December 20, 2024 01:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants