[fix](coordinator) Fix wrong bucket assignments by coordinator #45365

Gabriel39 · 2024-12-12T11:56:21Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Split scan ranges evenly into parallelExecInstanceNum instances.

For a fragment contains co-located join,

  scan (id = 0) -> join build (id = 2)
                      |
  scan (id = 1) -> join probe (id = 2)

If both of scan (id = 0) and scan (id = 1) are serial operators, we will plan local exchanger
after them:

  scan (id = 0) -> local exchange -> join build (id = 2)
                                           |
  scan (id = 1) -> local exchange -> join probe (id = 2)

And there is another more complicated scenario, for example, scan (id = 0) has 10 partitions and
3 buckets which means 3 * 10 tablets and scan (id = 1) has 3 buckets and no partition which means
3 tablets totally. If expected parallelism is 8, we will get a serial scan (id = 0) and a
non-serial scan (id = 1). For this case, we will plan another plan with local exchange:

  scan (id = 0)          ->         join build (id = 2)
                                           |
  scan (id = 1) -> local exchange -> join probe (id = 2)

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2024-12-12T11:56:26Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Gabriel39 · 2024-12-12T11:58:05Z

run buildall

github-actions · 2024-12-12T12:00:44Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-12T12:32:36Z

TeamCity be ut coverage result:
Function Coverage: 38.75% (10104/26074)
Line Coverage: 29.67% (84736/285550)
Region Coverage: 28.74% (43502/151345)
Branch Coverage: 25.30% (22097/87340)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1239840f396645b0831a74fcd8949f04568a6171_1239840f396645b0831a74fcd8949f04568a6171/report/index.html

doris-robot · 2024-12-12T12:37:22Z

TPC-H: Total hot run time: 40332 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1239840f396645b0831a74fcd8949f04568a6171, data reload: false

------ Round 1 ----------------------------------
q1	18538	7691	7333	7333
q2	2043	204	174	174
q3	12146	1063	1176	1063
q4	10532	778	775	775
q5	7599	2724	2709	2709
q6	238	146	143	143
q7	979	618	610	610
q8	9578	1846	1958	1846
q9	7520	6914	6703	6703
q10	7222	2354	2349	2349
q11	482	265	259	259
q12	480	234	239	234
q13	19350	3062	3037	3037
q14	248	217	224	217
q15	595	543	518	518
q16	1304	595	590	590
q17	1404	551	614	551
q18	7301	6725	6640	6640
q19	1378	984	983	983
q20	460	183	178	178
q21	3935	3259	3103	3103
q22	387	318	317	317
Total cold run time: 113719 ms
Total hot run time: 40332 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7232	7290	7229	7229
q2	338	239	232	232
q3	3069	2926	3066	2926
q4	2093	1842	1795	1795
q5	5602	5642	5662	5642
q6	229	143	142	142
q7	2243	1858	1863	1858
q8	3309	3492	3558	3492
q9	9296	9318	9394	9318
q10	3634	3571	3560	3560
q11	609	497	506	497
q12	868	655	656	655
q13	14496	3312	3231	3231
q14	310	283	273	273
q15	560	530	532	530
q16	698	663	650	650
q17	1833	1604	1620	1604
q18	8138	7666	7717	7666
q19	1816	1585	1633	1585
q20	2157	1849	1899	1849
q21	6082	5830	5561	5561
q22	619	557	540	540
Total cold run time: 75231 ms
Total hot run time: 60835 ms

doris-robot · 2024-12-12T12:49:43Z

TPC-DS: Total hot run time: 198696 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1239840f396645b0831a74fcd8949f04568a6171, data reload: false

query1	1193	789	764	764
query2	6212	2174	2066	2066
query3	12570	7040	7184	7040
query4	32649	23427	23349	23349
query5	3850	480	470	470
query6	277	200	179	179
query7	3917	298	320	298
query8	295	227	230	227
query9	9208	2687	2688	2687
query10	446	240	239	239
query11	17791	15072	15110	15072
query12	146	98	98	98
query13	1515	430	411	411
query14	9332	7181	7903	7181
query15	326	212	215	212
query16	8174	477	468	468
query17	1745	630	575	575
query18	2148	329	321	321
query19	361	166	161	161
query20	125	115	117	115
query21	207	112	115	112
query22	4800	4574	4607	4574
query23	34609	34096	33682	33682
query24	10441	2534	2588	2534
query25	603	388	383	383
query26	1365	152	151	151
query27	2664	291	295	291
query28	7721	2459	2456	2456
query29	866	419	417	417
query30	232	172	147	147
query31	1001	832	852	832
query32	93	55	60	55
query33	760	323	284	284
query34	971	526	516	516
query35	886	773	759	759
query36	1137	969	958	958
query37	135	73	79	73
query38	4620	4379	4381	4379
query39	1507	1492	1482	1482
query40	293	101	98	98
query41	43	42	43	42
query42	110	98	103	98
query43	540	525	510	510
query44	1352	838	827	827
query45	189	179	166	166
query46	1171	732	735	732
query47	2064	1908	1904	1904
query48	433	341	325	325
query49	920	385	396	385
query50	849	392	396	392
query51	7431	7293	7175	7175
query52	106	83	87	83
query53	252	172	172	172
query54	1015	395	396	395
query55	82	77	80	77
query56	245	235	225	225
query57	1263	1104	1099	1099
query58	217	200	217	200
query59	3261	3017	3278	3017
query60	271	245	245	245
query61	110	136	102	102
query62	869	674	670	670
query63	218	180	183	180
query64	3999	664	627	627
query65	3219	3211	3175	3175
query66	928	288	293	288
query67	15978	15551	15568	15551
query68	4884	575	560	560
query69	443	253	250	250
query70	1273	1142	1157	1142
query71	323	247	243	243
query72	6329	4032	4112	4032
query73	763	359	355	355
query74	10319	8923	9029	8923
query75	3435	2661	2679	2661
query76	2730	1095	1214	1095
query77	468	267	267	267
query78	10333	9409	9378	9378
query79	1839	591	604	591
query80	1160	416	447	416
query81	548	239	238	238
query82	947	117	120	117
query83	247	149	147	147
query84	243	68	85	68
query85	1440	312	300	300
query86	440	300	295	295
query87	4771	4585	4496	4496
query88	3516	2218	2189	2189
query89	419	283	300	283
query90	2136	187	187	187
query91	173	104	101	101
query92	65	51	50	50
query93	2564	553	552	552
query94	1086	297	297	297
query95	356	252	258	252
query96	625	275	281	275
query97	2811	2684	2672	2672
query98	218	198	195	195
query99	1540	1319	1291	1291
Total cold run time: 302770 ms
Total hot run time: 198696 ms

doris-robot · 2024-12-12T12:55:10Z

ClickBench: Total hot run time: 31.68 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1239840f396645b0831a74fcd8949f04568a6171, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.04
query3	0.23	0.07	0.07
query4	1.64	0.11	0.10
query5	0.42	0.41	0.41
query6	1.17	0.65	0.64
query7	0.01	0.02	0.02
query8	0.04	0.03	0.03
query9	0.60	0.50	0.50
query10	0.54	0.60	0.55
query11	0.14	0.10	0.11
query12	0.14	0.10	0.11
query13	0.61	0.61	0.59
query14	2.83	2.89	2.82
query15	0.90	0.82	0.82
query16	0.38	0.40	0.39
query17	1.05	1.05	1.07
query18	0.22	0.21	0.21
query19	1.86	1.77	1.98
query20	0.01	0.01	0.01
query21	15.36	0.59	0.58
query22	2.83	1.94	1.75
query23	16.95	0.97	0.80
query24	3.25	1.44	0.22
query25	0.17	0.20	0.07
query26	0.52	0.14	0.13
query27	0.04	0.05	0.04
query28	10.95	1.08	1.06
query29	12.55	3.23	3.21
query30	0.25	0.07	0.06
query31	2.86	0.38	0.37
query32	3.28	0.47	0.45
query33	3.00	3.00	3.08
query34	16.68	4.47	4.46
query35	4.50	4.52	4.45
query36	0.68	0.48	0.49
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.26 s
Total hot run time: 31.68 s

Gabriel39 · 2024-12-13T02:24:22Z

run buildall

doris-robot · 2024-12-13T03:00:42Z

TPC-H: Total hot run time: 39652 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8dd03c59b3a7dbd8d048c663a8ba7750bb6c3a86, data reload: false

------ Round 1 ----------------------------------
q1	17597	7428	7293	7293
q2	2051	175	171	171
q3	10547	1077	1178	1077
q4	10230	714	674	674
q5	7578	2665	2616	2616
q6	242	149	147	147
q7	985	640	595	595
q8	9243	1795	1920	1795
q9	6613	6448	6401	6401
q10	7020	2296	2304	2296
q11	469	251	243	243
q12	423	230	233	230
q13	17793	3010	2984	2984
q14	250	206	215	206
q15	569	528	534	528
q16	657	577	581	577
q17	982	603	524	524
q18	7141	6719	6661	6661
q19	1352	978	982	978
q20	488	181	178	178
q21	4345	3169	3233	3169
q22	379	314	309	309
Total cold run time: 106954 ms
Total hot run time: 39652 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7298	7217	7186	7186
q2	324	231	229	229
q3	2857	2771	2891	2771
q4	2055	1759	1743	1743
q5	5643	5610	5608	5608
q6	230	143	151	143
q7	2211	1830	1810	1810
q8	3346	3551	3486	3486
q9	9068	8946	8977	8946
q10	3627	3526	3525	3525
q11	598	494	509	494
q12	856	639	603	603
q13	11489	3265	3206	3206
q14	305	277	267	267
q15	571	515	518	515
q16	687	664	652	652
q17	1856	1610	1616	1610
q18	8348	7940	7583	7583
q19	1797	1453	1539	1453
q20	2113	1872	1849	1849
q21	5609	5481	5483	5481
q22	603	562	552	552
Total cold run time: 71491 ms
Total hot run time: 59712 ms

doris-robot · 2024-12-13T03:13:17Z

TPC-DS: Total hot run time: 195620 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8dd03c59b3a7dbd8d048c663a8ba7750bb6c3a86, data reload: false

query1	1298	995	964	964
query2	6236	2085	2067	2067
query3	10926	4601	4372	4372
query4	66178	28703	23212	23212
query5	4920	479	486	479
query6	453	173	174	173
query7	5662	303	284	284
query8	327	222	232	222
query9	9477	2692	2686	2686
query10	460	244	247	244
query11	17312	14988	15720	14988
query12	151	102	101	101
query13	1571	423	414	414
query14	10833	6959	7359	6959
query15	216	180	196	180
query16	7054	453	470	453
query17	1091	582	587	582
query18	1817	318	319	318
query19	229	159	162	159
query20	122	121	117	117
query21	215	107	106	106
query22	4588	4781	4483	4483
query23	34582	34030	33962	33962
query24	5516	2391	2455	2391
query25	487	381	388	381
query26	632	155	148	148
query27	1798	283	283	283
query28	4429	2465	2464	2464
query29	665	434	410	410
query30	212	150	145	145
query31	1014	867	867	867
query32	61	60	52	52
query33	428	300	303	300
query34	946	529	519	519
query35	852	758	774	758
query36	1082	988	997	988
query37	128	74	74	74
query38	4418	4409	4315	4315
query39	1490	1452	1463	1452
query40	203	99	98	98
query41	46	41	42	41
query42	106	99	94	94
query43	517	495	504	495
query44	1209	881	835	835
query45	196	167	176	167
query46	1170	722	730	722
query47	2014	1966	1924	1924
query48	443	326	332	326
query49	742	408	395	395
query50	854	397	385	385
query51	7461	7252	7081	7081
query52	95	89	93	89
query53	253	180	174	174
query54	507	401	387	387
query55	76	76	73	73
query56	248	232	233	232
query57	1221	1114	1142	1114
query58	229	202	227	202
query59	3257	3032	2972	2972
query60	263	243	238	238
query61	111	116	105	105
query62	770	684	695	684
query63	217	185	184	184
query64	1360	681	656	656
query65	3253	3226	3219	3219
query66	758	299	287	287
query67	15790	15734	15604	15604
query68	4043	589	574	574
query69	430	251	251	251
query70	1200	1144	1169	1144
query71	367	255	243	243
query72	6314	4032	4074	4032
query73	757	351	355	351
query74	9292	8883	8903	8883
query75	3379	2638	2643	2638
query76	1880	1057	1096	1057
query77	482	262	262	262
query78	10432	9732	9283	9283
query79	1671	614	591	591
query80	1155	422	484	422
query81	505	231	225	225
query82	977	115	116	115
query83	175	149	142	142
query84	274	70	74	70
query85	1007	310	301	301
query86	420	305	292	292
query87	4710	4541	4714	4541
query88	3552	2229	2196	2196
query89	408	298	296	296
query90	1939	182	189	182
query91	141	102	106	102
query92	66	49	50	49
query93	2087	557	551	551
query94	823	291	288	288
query95	362	250	253	250
query96	622	276	283	276
query97	2808	2666	2654	2654
query98	214	197	190	190
query99	1598	1314	1319	1314
Total cold run time: 317335 ms
Total hot run time: 195620 ms

doris-robot · 2024-12-13T03:18:47Z

ClickBench: Total hot run time: 33.29 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8dd03c59b3a7dbd8d048c663a8ba7750bb6c3a86, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.08
query4	1.62	0.11	0.10
query5	0.44	0.40	0.42
query6	1.16	0.66	0.66
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.57	0.49	0.50
query10	0.57	0.61	0.57
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.60	0.60	0.59
query14	2.83	2.72	2.82
query15	0.90	0.82	0.81
query16	0.38	0.38	0.39
query17	1.06	1.07	1.08
query18	0.22	0.21	0.20
query19	1.93	1.75	1.93
query20	0.01	0.01	0.01
query21	15.37	0.61	0.60
query22	2.52	2.61	2.39
query23	16.87	0.94	0.88
query24	3.21	1.27	2.02
query25	0.21	0.07	0.12
query26	0.50	0.12	0.13
query27	0.04	0.07	0.04
query28	9.59	1.10	1.06
query29	12.53	3.29	3.20
query30	0.25	0.06	0.06
query31	2.86	0.38	0.37
query32	3.29	0.47	0.46
query33	2.99	3.00	3.08
query34	17.01	4.46	4.42
query35	4.59	4.43	4.51
query36	0.66	0.48	0.50
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.95 s
Total hot run time: 33.29 s

github-actions · 2024-12-13T06:23:54Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-12-13T06:23:56Z

PR approved by anyone and no changes requested.

Split scan ranges evenly into `parallelExecInstanceNum` instances. For a fragment contains co-located join, scan (id = 0) -> join build (id = 2) | scan (id = 1) -> join probe (id = 2) If both of `scan (id = 0)` and `scan (id = 1)` are serial operators, we will plan local exchanger after them: scan (id = 0) -> local exchange -> join build (id = 2) | scan (id = 1) -> local exchange -> join probe (id = 2) And there is another more complicated scenario, for example, `scan (id = 0)` has 10 partitions and 3 buckets which means 3 * 10 tablets and `scan (id = 1)` has 3 buckets and no partition which means 3 tablets totally. If expected parallelism is 8, we will get a serial scan (id = 0) and a non-serial scan (id = 1). For this case, we will plan another plan with local exchange: scan (id = 0) -> join build (id = 2) | scan (id = 1) -> local exchange -> join probe (id = 2)

…nator #45365 (#45401) Cherry-picked from #45365 Co-authored-by: Gabriel <[email protected]>

[fix](coordinator) Fix wrong bucket assignments by coordinator

1239840

Gabriel39 added the dev/3.0.x label Dec 12, 2024

Gabriel39 added 2 commits December 13, 2024 09:58

update

1aff5f2

update

8dd03c5

924060929 approved these changes Dec 13, 2024

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Dec 13, 2024

Mryange approved these changes Dec 13, 2024

View reviewed changes

Gabriel39 merged commit 1d64cb0 into apache:master Dec 13, 2024
28 of 30 checks passed

github-actions bot mentioned this pull request Dec 13, 2024

branch-3.0: [fix](coordinator) Fix wrong bucket assignments by coordinator #45365 #45401

Merged

Gabriel39 added a commit that referenced this pull request Dec 13, 2024

branch-3.0: [fix](coordinator) Fix wrong bucket assignments by coordi…

f53bb5d

…nator #45365 (#45401) Cherry-picked from #45365 Co-authored-by: Gabriel <[email protected]>

Gabriel39 added dev/3.0.4-merged and removed dev/3.0.x labels Dec 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](coordinator) Fix wrong bucket assignments by coordinator #45365

[fix](coordinator) Fix wrong bucket assignments by coordinator #45365

Gabriel39 commented Dec 12, 2024 •

edited

Loading

Thearas commented Dec 12, 2024

Gabriel39 commented Dec 12, 2024

github-actions bot commented Dec 12, 2024

doris-robot commented Dec 12, 2024

doris-robot commented Dec 12, 2024

doris-robot commented Dec 12, 2024

doris-robot commented Dec 12, 2024

Gabriel39 commented Dec 13, 2024

doris-robot commented Dec 13, 2024

doris-robot commented Dec 13, 2024

doris-robot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

[fix](coordinator) Fix wrong bucket assignments by coordinator #45365

[fix](coordinator) Fix wrong bucket assignments by coordinator #45365

Conversation

Gabriel39 commented Dec 12, 2024 • edited Loading

What problem does this PR solve?

Check List (For Author)

Check List (For Reviewer who merge this PR)

Thearas commented Dec 12, 2024

Gabriel39 commented Dec 12, 2024

github-actions bot commented Dec 12, 2024

doris-robot commented Dec 12, 2024

doris-robot commented Dec 12, 2024

doris-robot commented Dec 12, 2024

doris-robot commented Dec 12, 2024

Gabriel39 commented Dec 13, 2024

doris-robot commented Dec 13, 2024

doris-robot commented Dec 13, 2024

doris-robot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

Gabriel39 commented Dec 12, 2024 •

edited

Loading