Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

branch-3.0: [fix](report) report queue size based on the number of Bes and config #46190 #47197

Open
wants to merge 1 commit into
base: branch-3.0
Choose a base branch
from

Conversation

deardeng
Copy link
Contributor

cherry pick from #46190

…apache#46190)


A large cluster has many BE nodes, and the report occasionally
encounters a "queue full" error. The report queue size will be modified
based on the number of BE instances.
@deardeng deardeng requested a review from dataroaring as a code owner January 19, 2025 02:44
@deardeng
Copy link
Contributor Author

run buildall

@Thearas
Copy link
Contributor

Thearas commented Jan 19, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@doris-robot
Copy link

TPC-H: Total hot run time: 40762 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dc5ab28cdde9a4975b1a947a5fb1cb78166ad0fb, data reload: false

------ Round 1 ----------------------------------
q1	17573	8094	7199	7199
q2	2047	171	183	171
q3	10601	1083	1168	1083
q4	10534	734	714	714
q5	7735	2808	2800	2800
q6	240	147	144	144
q7	983	622	615	615
q8	9342	1949	2073	1949
q9	6522	6390	6388	6388
q10	7014	2264	2264	2264
q11	481	269	263	263
q12	435	217	213	213
q13	17825	3009	2994	2994
q14	244	212	206	206
q15	567	534	522	522
q16	703	628	612	612
q17	961	538	545	538
q18	7187	6601	6610	6601
q19	1402	1076	1096	1076
q20	474	203	198	198
q21	4123	3182	3305	3182
q22	1115	1030	1035	1030
Total cold run time: 108108 ms
Total hot run time: 40762 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7268	7146	7188	7146
q2	330	239	243	239
q3	2981	3029	3039	3029
q4	2090	1922	1863	1863
q5	5812	5820	5833	5820
q6	228	150	146	146
q7	2245	1878	1841	1841
q8	3371	3628	3499	3499
q9	8809	8907	8805	8805
q10	3536	3559	3553	3553
q11	592	503	506	503
q12	784	626	632	626
q13	9680	3143	3158	3143
q14	309	293	266	266
q15	578	521	532	521
q16	737	670	691	670
q17	1834	1605	1599	1599
q18	8299	7687	7564	7564
q19	1630	1592	1613	1592
q20	2140	1873	1890	1873
q21	5522	5224	5438	5224
q22	1155	1064	1031	1031
Total cold run time: 69930 ms
Total hot run time: 60553 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197718 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dc5ab28cdde9a4975b1a947a5fb1cb78166ad0fb, data reload: false

query1	1305	916	947	916
query2	6253	2117	2064	2064
query3	10783	4196	4163	4163
query4	66260	28854	23680	23680
query5	4971	455	456	455
query6	406	202	194	194
query7	5643	315	309	309
query8	308	230	226	226
query9	8944	2702	2708	2702
query10	464	267	259	259
query11	17320	15186	15764	15186
query12	157	102	109	102
query13	1525	465	440	440
query14	9947	7873	7333	7333
query15	208	191	189	189
query16	7210	500	475	475
query17	1080	598	599	598
query18	1861	326	329	326
query19	241	161	178	161
query20	123	110	107	107
query21	211	112	105	105
query22	4604	4614	4469	4469
query23	34565	34159	34381	34159
query24	6125	2958	2933	2933
query25	557	431	420	420
query26	676	173	179	173
query27	1848	356	372	356
query28	4405	2487	2482	2482
query29	732	476	471	471
query30	242	169	168	168
query31	979	864	821	821
query32	72	55	57	55
query33	424	299	282	282
query34	914	509	538	509
query35	834	726	725	725
query36	1070	948	967	948
query37	123	76	78	76
query38	4112	4153	3968	3968
query39	1500	1446	1478	1446
query40	211	112	107	107
query41	47	47	46	46
query42	115	104	108	104
query43	552	491	480	480
query44	1175	819	815	815
query45	185	166	172	166
query46	1172	743	745	743
query47	2030	1903	1942	1903
query48	489	399	402	399
query49	729	381	387	381
query50	881	446	428	428
query51	7471	7098	7108	7098
query52	107	90	88	88
query53	259	183	188	183
query54	558	446	460	446
query55	76	75	74	74
query56	263	234	247	234
query57	1267	1116	1121	1116
query58	214	210	215	210
query59	3196	2954	2759	2759
query60	273	245	255	245
query61	107	106	111	106
query62	762	671	652	652
query63	219	189	195	189
query64	1361	669	627	627
query65	3270	3182	3260	3182
query66	702	315	309	309
query67	15927	15666	15604	15604
query68	4002	585	588	585
query69	435	296	267	267
query70	1211	1120	1073	1073
query71	369	263	268	263
query72	6393	4142	3981	3981
query73	778	343	355	343
query74	10065	9032	8952	8952
query75	3380	2602	2646	2602
query76	1831	1072	1090	1072
query77	538	292	273	273
query78	10553	9702	9665	9665
query79	1762	592	591	591
query80	1393	431	440	431
query81	530	240	244	240
query82	1270	114	114	114
query83	163	143	143	143
query84	284	87	77	77
query85	979	314	283	283
query86	392	295	300	295
query87	4439	4323	4273	4273
query88	3687	2409	2359	2359
query89	414	285	289	285
query90	1976	187	194	187
query91	181	148	151	148
query92	64	50	52	50
query93	1998	552	547	547
query94	843	299	291	291
query95	358	256	262	256
query96	625	287	277	277
query97	3349	3199	3171	3171
query98	218	202	204	202
query99	1585	1277	1302	1277
Total cold run time: 318685 ms
Total hot run time: 197718 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants