Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](rowset meta) truncate segments key bounds if too large to avoid RowsetMetaCloudPB exceeds fdb's 100KB limits #45287

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

[opt](rowset meta) truncate segments key bounds if too large to avoid RowsetMetaCloudPB exceeds fdb's 100KB limits #45287

wants to merge 6 commits into from

Conversation

bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Dec 11, 2024 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@bobhan1 bobhan1 force-pushed the truncate-segmnet-key-bounds branch from f848d0f to 6754c7f Compare December 11, 2024 09:35
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@bobhan1 bobhan1 force-pushed the truncate-segmnet-key-bounds branch from 6754c7f to c6e01f4 Compare December 11, 2024 10:36
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@bobhan1 bobhan1 force-pushed the truncate-segmnet-key-bounds branch from c6e01f4 to a870868 Compare December 11, 2024 12:08
github-actions[bot]
github-actions bot reviewed Dec 11, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/test/olap/segments_key_bounds_truncation_test.cpp Show resolved Hide resolved
@bobhan1 bobhan1 force-pushed the truncate-segmnet-key-bounds branch from b29e183 to 4de3fe4 Compare December 12, 2024 03:42
zhannngchen
zhannngchen reviewed Dec 12, 2024
View reviewed changes
be/src/olap/base_tablet.h Outdated Show resolved Hide resolved
be/src/util/slice.cpp Outdated Show resolved Hide resolved
@bobhan1 bobhan1 force-pushed the truncate-segmnet-key-bounds branch 7 times, most recently from 4b70373 to c2c2892 Compare December 17, 2024 07:59
github-actions[bot]
github-actions bot reviewed Dec 17, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/test/olap/segments_key_bounds_truncation_test.cpp Show resolved Hide resolved
github-actions[bot]
github-actions bot reviewed Dec 18, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/test/olap/segments_key_bounds_truncation_test.cpp Show resolved Hide resolved
@bobhan1 bobhan1 force-pushed the truncate-segmnet-key-bounds branch from 286cb7a to 3c33012 Compare December 18, 2024 04:05
github-actions[bot]
github-actions bot reviewed Dec 18, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/test/olap/segments_key_bounds_truncation_test.cpp Show resolved Hide resolved
github-actions[bot]
github-actions bot reviewed Dec 18, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/test/olap/segments_key_bounds_truncation_test.cpp Show resolved Hide resolved
be/test/olap/segments_key_bounds_truncation_test.cpp Show resolved Hide resolved
@bobhan1 bobhan1 marked this pull request as ready for review December 18, 2024 06:39
@bobhan1 bobhan1 force-pushed the truncate-segmnet-key-bounds branch from 59f196e to 59d64f4 Compare December 18, 2024 08:00
@bobhan1
Copy link
Contributor Author

bobhan1 commented Dec 18, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40571 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 59d64f429a274bf1f8dfcff61f71bab681cfeade, data reload: false

------ Round 1 ----------------------------------
q1	17599	7532	7261	7261
q2	2037	191	181	181
q3	10528	1194	1211	1194
q4	10241	760	717	717
q5	7628	2748	2755	2748
q6	237	142	144	142
q7	1022	633	615	615
q8	9240	1895	1937	1895
q9	6673	6445	6549	6445
q10	6990	2341	2346	2341
q11	460	267	262	262
q12	404	233	228	228
q13	17785	2996	2901	2901
q14	238	216	215	215
q15	575	499	483	483
q16	668	620	580	580
q17	995	640	673	640
q18	7460	6840	6734	6734
q19	1336	1064	1071	1064
q20	491	188	191	188
q21	4139	3441	3419	3419
q22	391	318	320	318
Total cold run time: 107137 ms
Total hot run time: 40571 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7331	7236	7208	7208
q2	333	231	236	231
q3	2909	2903	3020	2903
q4	2069	1903	1778	1778
q5	5607	5730	5615	5615
q6	223	138	141	138
q7	2171	1769	1771	1769
q8	3407	3549	3494	3494
q9	8945	8876	8931	8876
q10	3566	3578	3582	3578
q11	616	501	505	501
q12	806	652	609	609
q13	13999	3039	3074	3039
q14	295	270	260	260
q15	552	528	503	503
q16	694	650	644	644
q17	1810	1596	1590	1590
q18	7948	7537	7393	7393
q19	1733	1634	1636	1634
q20	2114	1847	1820	1820
q21	5521	5306	5324	5306
q22	639	574	554	554
Total cold run time: 73288 ms
Total hot run time: 59443 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190947 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 59d64f429a274bf1f8dfcff61f71bab681cfeade, data reload: false

query1	983	399	379	379
query2	6528	2540	2395	2395
query3	6717	226	221	221
query4	33837	23805	23313	23313
query5	4331	518	492	492
query6	304	207	191	191
query7	4633	302	305	302
query8	311	247	249	247
query9	9694	2738	2773	2738
query10	482	291	272	272
query11	17917	15118	15188	15118
query12	159	112	115	112
query13	1679	424	426	424
query14	10419	6682	6731	6682
query15	286	180	200	180
query16	8315	469	468	468
query17	1641	610	559	559
query18	2130	300	303	300
query19	393	190	162	162
query20	127	118	119	118
query21	205	104	111	104
query22	4631	4398	4190	4190
query23	34607	33641	33668	33641
query24	10429	2350	2399	2350
query25	655	408	402	402
query26	1197	155	155	155
query27	2899	326	329	326
query28	8032	2413	2408	2408
query29	850	413	427	413
query30	303	153	149	149
query31	1025	824	863	824
query32	102	75	74	74
query33	781	316	311	311
query34	951	499	513	499
query35	879	751	741	741
query36	1089	928	961	928
query37	146	93	92	92
query38	4307	4139	4080	4080
query39	1484	1464	1431	1431
query40	203	110	108	108
query41	51	44	52	44
query42	125	110	115	110
query43	545	501	500	500
query44	1278	813	824	813
query45	186	171	179	171
query46	1164	706	706	706
query47	1984	1832	1866	1832
query48	407	335	323	323
query49	1151	442	396	396
query50	831	392	412	392
query51	7148	7089	7200	7089
query52	109	102	100	100
query53	262	185	189	185
query54	1060	422	458	422
query55	94	92	90	90
query56	275	256	265	256
query57	1254	1129	1163	1129
query58	274	255	264	255
query59	3302	3155	3272	3155
query60	301	270	280	270
query61	108	106	106	106
query62	887	698	687	687
query63	215	195	198	195
query64	4079	670	645	645
query65	3288	3206	3257	3206
query66	854	373	324	324
query67	15833	15474	15558	15474
query68	4944	531	533	531
query69	459	288	281	281
query70	1203	1162	1139	1139
query71	445	270	274	270
query72	6554	4093	4102	4093
query73	763	357	358	357
query74	10493	8859	8931	8859
query75	3531	2648	2673	2648
query76	2996	1085	1027	1027
query77	527	295	286	286
query78	10270	10102	9418	9418
query79	2177	606	608	606
query80	900	444	434	434
query81	548	244	233	233
query82	563	130	133	130
query83	251	150	153	150
query84	234	74	75	74
query85	1305	319	317	317
query86	505	298	286	286
query87	4563	4511	4496	4496
query88	4019	2237	2218	2218
query89	419	301	313	301
query90	2057	199	206	199
query91	139	109	111	109
query92	75	71	73	71
query93	1582	547	542	542
query94	910	297	291	291
query95	398	280	273	273
query96	629	289	283	283
query97	2852	2746	2738	2738
query98	237	209	198	198
query99	1591	1344	1315	1315
Total cold run time: 300865 ms
Total hot run time: 190947 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.74 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 59d64f429a274bf1f8dfcff61f71bab681cfeade, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.05	0.03
query3	0.24	0.07	0.07
query4	1.63	0.10	0.11
query5	0.43	0.39	0.41
query6	1.15	0.65	0.64
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.52	0.50
query10	0.56	0.58	0.56
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.71	2.76	2.87
query15	0.91	0.83	0.83
query16	0.39	0.39	0.38
query17	1.04	1.08	1.06
query18	0.24	0.22	0.22
query19	1.93	1.99	1.86
query20	0.01	0.01	0.02
query21	15.36	0.59	0.57
query22	2.66	2.38	2.20
query23	16.98	1.01	0.87
query24	3.03	1.65	1.72
query25	0.27	0.13	0.06
query26	0.60	0.14	0.13
query27	0.04	0.04	0.04
query28	9.52	1.09	1.08
query29	12.55	3.23	3.23
query30	0.25	0.06	0.07
query31	2.86	0.39	0.40
query32	3.28	0.48	0.47
query33	3.17	3.02	3.08
query34	16.93	4.41	4.43
query35	4.46	4.47	4.47
query36	0.67	0.48	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.01 s
Total hot run time: 33.74 s

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@bobhan1 bobhan1 force-pushed the truncate-segmnet-key-bounds branch from a0503e7 to 2d33fa3 Compare December 26, 2024 08:13
@bobhan1
Copy link
Contributor Author

bobhan1 commented Dec 26, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 26, 2024
@bobhan1
Copy link
Contributor Author

bobhan1 commented Dec 26, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.91% (10126/26026)
Line Coverage: 29.91% (85570/286117)
Region Coverage: 29.05% (43727/150540)
Branch Coverage: 25.59% (22313/87198)
Coverage Report: http://coverage.selectdb-in.cc/coverage/90f50958ad30556bf527cb86647b33d157d1bf9a_90f50958ad30556bf527cb86647b33d157d1bf9a/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 32650 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 90f50958ad30556bf527cb86647b33d157d1bf9a, data reload: false

------ Round 1 ----------------------------------
q1	17586	6196	6054	6054
q2	2062	305	172	172
q3	10721	1233	721	721
q4	10197	873	426	426
q5	7486	2193	1970	1970
q6	209	177	150	150
q7	895	797	607	607
q8	9228	1421	1217	1217
q9	5230	4926	5005	4926
q10	6724	2304	1841	1841
q11	476	273	266	266
q12	358	353	219	219
q13	17760	3645	3047	3047
q14	234	233	217	217
q15	562	492	502	492
q16	640	638	602	602
q17	587	854	322	322
q18	7184	6431	6520	6431
q19	3058	967	538	538
q20	302	313	182	182
q21	2809	2162	1943	1943
q22	353	335	307	307
Total cold run time: 104661 ms
Total hot run time: 32650 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6333	6251	6246	6246
q2	236	327	233	233
q3	2248	2681	2321	2321
q4	1399	1778	1348	1348
q5	4373	4788	4874	4788
q6	186	178	140	140
q7	2101	1956	1818	1818
q8	2637	2807	2693	2693
q9	7348	7305	7406	7305
q10	3069	3379	2724	2724
q11	575	517	493	493
q12	672	758	605	605
q13	3425	3776	3134	3134
q14	280	296	282	282
q15	583	513	499	499
q16	664	696	668	668
q17	1225	1754	1259	1259
q18	7677	7528	7416	7416
q19	869	1149	1099	1099
q20	1995	2032	1916	1916
q21	5635	5314	4865	4865
q22	639	643	586	586
Total cold run time: 54169 ms
Total hot run time: 52438 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196711 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 90f50958ad30556bf527cb86647b33d157d1bf9a, data reload: false

query1	1296	983	926	926
query2	6513	2339	2376	2339
query3	10955	4769	4745	4745
query4	33025	23804	23453	23453
query5	4177	606	455	455
query6	269	186	179	179
query7	4002	499	303	303
query8	314	254	239	239
query9	9284	2723	2720	2720
query10	470	305	244	244
query11	17971	15665	15214	15214
query12	170	108	106	106
query13	1590	549	427	427
query14	9878	7364	7219	7219
query15	247	208	185	185
query16	8175	650	460	460
query17	1553	782	596	596
query18	2094	431	326	326
query19	216	194	164	164
query20	123	125	123	123
query21	269	133	107	107
query22	4534	4739	4387	4387
query23	34488	33707	33338	33338
query24	6611	2403	2320	2320
query25	500	448	412	412
query26	775	286	155	155
query27	2076	470	337	337
query28	5285	2511	2460	2460
query29	639	567	447	447
query30	204	182	150	150
query31	951	885	845	845
query32	82	55	58	55
query33	477	352	309	309
query34	774	866	524	524
query35	831	854	780	780
query36	1022	1044	954	954
query37	111	92	78	78
query38	4239	4208	4157	4157
query39	1555	1633	1487	1487
query40	225	116	102	102
query41	48	48	43	43
query42	120	106	104	104
query43	537	504	515	504
query44	1359	847	829	829
query45	193	176	173	173
query46	906	1062	660	660
query47	2027	1995	1905	1905
query48	393	413	323	323
query49	728	466	376	376
query50	650	673	411	411
query51	7285	7170	7268	7170
query52	102	99	95	95
query53	232	251	185	185
query54	494	510	416	416
query55	84	82	76	76
query56	252	265	243	243
query57	1292	1244	1164	1164
query58	231	215	235	215
query59	3124	3142	3073	3073
query60	266	276	248	248
query61	110	108	104	104
query62	867	813	758	758
query63	220	220	193	193
query64	3518	1005	642	642
query65	3320	3273	3237	3237
query66	781	417	317	317
query67	16691	15816	15562	15562
query68	9787	758	515	515
query69	509	286	251	251
query70	1209	1123	1125	1123
query71	439	300	251	251
query72	6285	4093	4006	4006
query73	793	755	359	359
query74	10116	9170	8789	8789
query75	4672	3148	2651	2651
query76	5577	1183	800	800
query77	1011	361	290	290
query78	10094	10212	9784	9784
query79	4695	871	600	600
query80	674	515	433	433
query81	498	264	233	233
query82	210	142	118	118
query83	191	161	142	142
query84	282	92	74	74
query85	742	346	297	297
query86	345	291	307	291
query87	4427	4432	4323	4323
query88	4196	2222	2207	2207
query89	412	340	322	322
query90	2210	187	188	187
query91	137	137	103	103
query92	67	54	59	54
query93	2772	898	528	528
query94	683	396	290	290
query95	321	265	251	251
query96	481	613	283	283
query97	2737	2804	2700	2700
query98	233	197	209	197
query99	1692	1559	1406	1406
Total cold run time: 302297 ms
Total hot run time: 196711 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.46 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 90f50958ad30556bf527cb86647b33d157d1bf9a, data reload: false

query1	0.04	0.03	0.02
query2	0.10	0.05	0.05
query3	0.23	0.06	0.06
query4	1.64	0.08	0.09
query5	0.42	0.41	0.40
query6	1.17	0.66	0.66
query7	0.02	0.02	0.02
query8	0.06	0.05	0.05
query9	0.55	0.49	0.50
query10	0.58	0.57	0.56
query11	0.17	0.12	0.12
query12	0.16	0.13	0.13
query13	0.62	0.59	0.59
query14	2.84	2.74	2.85
query15	0.91	0.84	0.83
query16	0.37	0.39	0.37
query17	0.98	1.07	0.98
query18	0.18	0.18	0.19
query19	1.88	1.80	2.03
query20	0.02	0.01	0.01
query21	15.38	0.96	0.66
query22	0.77	0.80	0.71
query23	14.98	1.48	0.68
query24	2.10	0.38	0.22
query25	0.15	0.09	0.09
query26	0.28	0.19	0.18
query27	0.08	0.07	0.09
query28	13.44	1.71	1.13
query29	12.69	4.11	3.40
query30	0.24	0.08	0.06
query31	2.84	0.58	0.40
query32	3.23	0.59	0.48
query33	3.24	3.14	3.08
query34	16.43	5.32	4.51
query35	4.53	4.55	4.54
query36	0.62	0.51	0.49
query37	0.19	0.16	0.15
query38	0.16	0.16	0.15
query39	0.06	0.05	0.04
query40	0.17	0.13	0.13
query41	0.10	0.05	0.05
query42	0.07	0.05	0.05
query43	0.05	0.05	0.05
Total cold run time: 104.74 s
Total hot run time: 31.46 s

@bobhan1 bobhan1 force-pushed the truncate-segmnet-key-bounds branch from 90f5095 to 25f1cf7 Compare January 16, 2025 08:24
@bobhan1 bobhan1 requested a review from yiguolei as a code owner January 16, 2025 08:24
@bobhan1
Copy link
Contributor Author

bobhan1 commented Jan 16, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32735 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 25f1cf7c04237b013d9e6c0858c515a6c4b7622a, data reload: false

------ Round 1 ----------------------------------
q1	17862	5568	5448	5448
q2	2051	309	173	173
q3	10532	1295	745	745
q4	10229	1003	550	550
q5	7646	2482	2235	2235
q6	200	166	133	133
q7	906	765	629	629
q8	9246	1437	1208	1208
q9	5329	5006	4945	4945
q10	6834	2352	1887	1887
q11	478	287	259	259
q12	350	364	218	218
q13	17763	3791	3201	3201
q14	241	230	209	209
q15	528	474	463	463
q16	625	621	608	608
q17	590	882	333	333
q18	7399	6612	6456	6456
q19	1788	954	538	538
q20	325	329	189	189
q21	3012	2297	1999	1999
q22	379	327	309	309
Total cold run time: 104313 ms
Total hot run time: 32735 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5591	5541	5540	5540
q2	242	339	233	233
q3	2244	2683	2363	2363
q4	1454	1819	1332	1332
q5	4403	4876	4841	4841
q6	173	169	135	135
q7	2122	1976	1861	1861
q8	2639	2876	2718	2718
q9	7269	7163	7227	7163
q10	3002	3292	2817	2817
q11	622	545	498	498
q12	668	760	594	594
q13	3611	4064	3354	3354
q14	279	305	272	272
q15	520	485	475	475
q16	661	712	652	652
q17	1256	1748	1236	1236
q18	7750	7423	7295	7295
q19	857	1174	1065	1065
q20	2024	2044	1899	1899
q21	5841	5523	4992	4992
q22	614	602	582	582
Total cold run time: 53842 ms
Total hot run time: 51917 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187183 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 25f1cf7c04237b013d9e6c0858c515a6c4b7622a, data reload: false

query1	990	390	386	386
query2	6529	2121	2017	2017
query3	6794	211	211	211
query4	33670	23310	23067	23067
query5	4332	608	456	456
query6	307	196	171	171
query7	4618	506	308	308
query8	284	226	221	221
query9	9506	2676	2660	2660
query10	467	312	257	257
query11	17999	15373	15298	15298
query12	170	110	105	105
query13	1686	524	398	398
query14	9680	6480	7082	6480
query15	253	202	201	201
query16	7806	647	496	496
query17	1644	757	562	562
query18	2072	408	305	305
query19	216	189	165	165
query20	121	112	111	111
query21	214	125	104	104
query22	4160	4228	4069	4069
query23	34176	32845	32811	32811
query24	6484	2325	2255	2255
query25	506	445	378	378
query26	1200	274	152	152
query27	2010	464	327	327
query28	5212	2421	2412	2412
query29	700	544	415	415
query30	234	187	162	162
query31	962	864	787	787
query32	114	66	60	60
query33	514	352	300	300
query34	742	860	501	501
query35	831	814	760	760
query36	1014	1068	956	956
query37	138	102	78	78
query38	4214	4121	4205	4121
query39	1457	1416	1379	1379
query40	202	111	101	101
query41	53	56	51	51
query42	117	99	104	99
query43	528	540	505	505
query44	1339	803	784	784
query45	181	173	162	162
query46	849	1048	634	634
query47	1804	1848	1751	1751
query48	391	395	316	316
query49	787	485	398	398
query50	622	669	395	395
query51	6824	6782	6684	6684
query52	102	101	93	93
query53	228	246	181	181
query54	486	492	422	422
query55	83	81	80	80
query56	265	253	239	239
query57	1146	1176	1082	1082
query58	257	229	238	229
query59	2950	3240	2944	2944
query60	277	268	252	252
query61	114	117	128	117
query62	794	712	657	657
query63	225	200	188	188
query64	4560	1018	659	659
query65	3249	3136	3190	3136
query66	1139	437	305	305
query67	16120	15493	15423	15423
query68	6419	806	528	528
query69	466	286	268	268
query70	1210	1132	1163	1132
query71	384	294	264	264
query72	5943	3799	3885	3799
query73	661	753	360	360
query74	10105	9182	8868	8868
query75	3180	3162	2678	2678
query76	3180	1185	764	764
query77	482	347	285	285
query78	9967	9995	9354	9354
query79	3277	813	593	593
query80	580	531	436	436
query81	489	292	239	239
query82	638	156	123	123
query83	169	172	149	149
query84	238	91	75	75
query85	784	421	295	295
query86	353	322	300	300
query87	4474	4642	4486	4486
query88	4596	2193	2153	2153
query89	395	339	285	285
query90	1893	189	199	189
query91	138	138	106	106
query92	67	58	131	58
query93	1346	870	530	530
query94	721	401	294	294
query95	340	265	249	249
query96	494	609	286	286
query97	2773	2871	2779	2779
query98	236	199	208	199
query99	1483	1375	1270	1270
Total cold run time: 287199 ms
Total hot run time: 187183 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.52 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 25f1cf7c04237b013d9e6c0858c515a6c4b7622a, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.24	0.07	0.07
query4	1.61	0.10	0.11
query5	0.42	0.42	0.41
query6	1.14	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.60	0.51	0.50
query10	0.57	0.57	0.56
query11	0.16	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.59	0.60
query14	2.85	2.76	2.73
query15	0.91	0.84	0.82
query16	0.38	0.38	0.39
query17	1.09	1.07	1.07
query18	0.22	0.22	0.21
query19	1.89	1.85	2.08
query20	0.01	0.01	0.01
query21	15.37	0.92	0.58
query22	0.76	0.80	0.72
query23	15.21	1.41	0.53
query24	2.67	1.44	2.08
query25	0.14	0.20	0.16
query26	0.26	0.13	0.13
query27	0.06	0.06	0.05
query28	14.29	1.02	0.45
query29	12.59	3.99	3.30
query30	0.24	0.10	0.07
query31	2.82	0.61	0.38
query32	3.23	0.56	0.45
query33	3.05	3.11	3.10
query34	16.78	5.17	4.53
query35	4.57	4.65	4.55
query36	0.65	0.50	0.49
query37	0.10	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.17 s
Total hot run time: 31.52 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 41.54% (10827/26064)
Line Coverage: 31.93% (91487/286501)
Region Coverage: 31.08% (46875/150802)
Branch Coverage: 27.18% (23749/87362)
Coverage Report: http://coverage.selectdb-in.cc/coverage/25f1cf7c04237b013d9e6c0858c515a6c4b7622a_25f1cf7c04237b013d9e6c0858c515a6c4b7622a/report/index.html

@bobhan1 bobhan1 force-pushed the truncate-segmnet-key-bounds branch from 25f1cf7 to d5a53e6 Compare February 5, 2025 02:45
@bobhan1
Copy link
Contributor Author

bobhan1 commented Feb 5, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32363 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d5a53e611ae339d7155adb7ab4a222b6f215caf8, data reload: false

------ Round 1 ----------------------------------
q1	17609	5497	5408	5408
q2	2061	308	183	183
q3	10494	1234	744	744
q4	10241	973	545	545
q5	7640	2391	2174	2174
q6	191	166	131	131
q7	913	760	598	598
q8	9227	1403	1215	1215
q9	5257	4888	4935	4888
q10	6834	2344	1888	1888
q11	492	274	244	244
q12	341	362	212	212
q13	17782	3685	3114	3114
q14	226	226	211	211
q15	533	479	473	473
q16	624	609	580	580
q17	595	912	336	336
q18	7335	6403	6493	6403
q19	2367	956	548	548
q20	303	322	193	193
q21	2811	2180	1965	1965
q22	374	345	310	310
Total cold run time: 104250 ms
Total hot run time: 32363 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5620	5453	5478	5453
q2	235	327	233	233
q3	2298	2687	2351	2351
q4	1454	1776	1402	1402
q5	4360	4758	4722	4722
q6	167	166	126	126
q7	2047	1968	1851	1851
q8	2643	2848	2719	2719
q9	7337	7242	7310	7242
q10	3013	3291	2761	2761
q11	560	513	474	474
q12	659	730	611	611
q13	3554	3975	3259	3259
q14	298	299	271	271
q15	537	479	463	463
q16	644	681	650	650
q17	1252	1748	1266	1266
q18	7776	7468	7361	7361
q19	814	1213	1042	1042
q20	1999	2073	1926	1926
q21	5732	5215	4983	4983
q22	647	625	556	556
Total cold run time: 53646 ms
Total hot run time: 51722 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.08% (10998/26139)
Line Coverage: 32.36% (92886/287041)
Region Coverage: 31.50% (47609/151133)
Branch Coverage: 27.54% (24107/87532)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d5a53e611ae339d7155adb7ab4a222b6f215caf8_d5a53e611ae339d7155adb7ab4a222b6f215caf8/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 190192 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d5a53e611ae339d7155adb7ab4a222b6f215caf8, data reload: false

query1	1347	939	926	926
query2	6139	2069	2081	2069
query3	11006	4426	4408	4408
query4	61185	29147	23089	23089
query5	5517	615	441	441
query6	428	199	176	176
query7	5553	532	302	302
query8	341	250	226	226
query9	8380	2642	2605	2605
query10	461	315	242	242
query11	17556	15352	15407	15352
query12	155	109	107	107
query13	1451	556	412	412
query14	11455	6401	7327	6401
query15	208	200	195	195
query16	7287	621	458	458
query17	1106	712	560	560
query18	1856	421	305	305
query19	191	175	155	155
query20	129	114	111	111
query21	205	119	105	105
query22	4444	4777	4482	4482
query23	34228	33531	33373	33373
query24	5585	2352	2420	2352
query25	495	492	420	420
query26	649	280	157	157
query27	1584	476	336	336
query28	4278	2530	2476	2476
query29	528	565	418	418
query30	214	181	162	162
query31	916	890	817	817
query32	71	59	62	59
query33	446	346	298	298
query34	748	844	505	505
query35	809	844	745	745
query36	1026	1063	975	975
query37	131	104	80	80
query38	4322	4323	4264	4264
query39	1505	1452	1438	1438
query40	206	115	105	105
query41	50	50	49	49
query42	115	102	98	98
query43	512	540	500	500
query44	1354	821	814	814
query45	183	177	179	177
query46	874	1056	658	658
query47	1962	1910	1869	1869
query48	388	404	328	328
query49	702	495	403	403
query50	645	685	403	403
query51	4340	4343	4272	4272
query52	106	101	89	89
query53	233	260	191	191
query54	482	500	454	454
query55	86	81	84	81
query56	257	277	241	241
query57	1199	1178	1141	1141
query58	240	244	238	238
query59	3248	3270	3176	3176
query60	297	274	248	248
query61	115	112	121	112
query62	766	722	678	678
query63	221	192	194	192
query64	1384	1025	672	672
query65	3281	3144	3174	3144
query66	720	380	290	290
query67	16222	15553	15358	15358
query68	3430	838	536	536
query69	468	303	262	262
query70	1205	1159	1153	1153
query71	387	288	243	243
query72	6018	2701	3961	2701
query73	668	739	348	348
query74	9753	9173	8868	8868
query75	3200	3155	2688	2688
query76	3185	1201	768	768
query77	479	354	268	268
query78	10122	9890	9353	9353
query79	2740	802	597	597
query80	1657	523	448	448
query81	556	276	236	236
query82	363	154	116	116
query83	270	174	148	148
query84	289	99	78	78
query85	769	358	302	302
query86	429	313	300	300
query87	4542	4533	4498	4498
query88	3662	2130	2093	2093
query89	383	318	291	291
query90	1491	181	185	181
query91	132	136	107	107
query92	66	57	53	53
query93	1913	868	575	575
query94	712	411	304	304
query95	328	258	248	248
query96	486	618	276	276
query97	2833	2913	2740	2740
query98	211	202	199	199
query99	1290	1392	1231	1231
Total cold run time: 308160 ms
Total hot run time: 190192 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.89 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d5a53e611ae339d7155adb7ab4a222b6f215caf8, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.05	0.05
query3	0.24	0.05	0.05
query4	1.64	0.08	0.08
query5	0.42	0.40	0.41
query6	1.16	0.67	0.66
query7	0.02	0.02	0.02
query8	0.05	0.05	0.05
query9	0.55	0.49	0.51
query10	0.56	0.56	0.56
query11	0.16	0.13	0.12
query12	0.16	0.13	0.12
query13	0.61	0.59	0.60
query14	2.73	2.77	2.78
query15	0.92	0.83	0.83
query16	0.38	0.38	0.38
query17	1.06	1.06	1.06
query18	0.18	0.18	0.19
query19	1.92	1.72	1.99
query20	0.02	0.02	0.02
query21	15.36	0.96	0.65
query22	0.78	0.76	0.69
query23	15.00	1.55	0.67
query24	1.93	0.40	0.24
query25	0.14	0.09	0.09
query26	0.29	0.20	0.18
query27	0.09	0.09	0.08
query28	13.37	1.17	0.55
query29	12.64	4.15	3.40
query30	0.25	0.08	0.05
query31	2.84	0.60	0.40
query32	3.23	0.58	0.48
query33	2.99	3.02	3.07
query34	16.16	5.12	4.58
query35	4.55	4.57	4.55
query36	0.62	0.49	0.47
query37	0.20	0.16	0.15
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.10	0.05	0.05
query42	0.07	0.06	0.05
query43	0.05	0.05	0.04
Total cold run time: 103.93 s
Total hot run time: 30.89 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gavinchou gavinchou gavinchou left review comments

@github-actions github-actions[bot] github-actions[bot] left review comments

@dataroaring dataroaring dataroaring left review comments

@zhannngchen zhannngchen zhannngchen left review comments

@yiguolei yiguolei Awaiting requested review from yiguolei yiguolei is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@bobhan1 @doris-robot @gavinchou @zhannngchen @dataroaring