Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](Nereids) insert lock all target tables #47033

Merged
merged 1 commit into from
Jan 16, 2025
Merged

[fix](Nereids) insert lock all target tables #47033

merged 1 commit into from
Jan 16, 2025

Conversation

morrySnow
Copy link
Contributor

What problem does this PR solve?

Related PR: #45045

Problem Summary:

we get target table twice, lock them all

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@morrySnow
[fix](Nereids) insert lock all target tables
7309a10 
we get target table twice, lock them all
@Thearas
Copy link
Contributor

Thearas commented Jan 15, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32191 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7309a1044f17bf7bd182d5bd29f0533fd7ef1586, data reload: false

------ Round 1 ----------------------------------
q1	17611	5530	5339	5339
q2	2051	307	171	171
q3	10407	1301	728	728
q4	10215	980	542	542
q5	7505	2428	2195	2195
q6	192	165	135	135
q7	893	768	610	610
q8	9248	1377	1176	1176
q9	5230	4857	4978	4857
q10	6864	2320	1900	1900
q11	499	277	253	253
q12	342	359	219	219
q13	17783	3737	3057	3057
q14	228	229	199	199
q15	514	482	470	470
q16	617	606	586	586
q17	561	874	331	331
q18	7083	6383	6472	6383
q19	1233	961	539	539
q20	326	325	195	195
q21	2856	2229	1989	1989
q22	363	335	317	317
Total cold run time: 102621 ms
Total hot run time: 32191 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5475	5441	5466	5441
q2	236	341	243	243
q3	2268	2674	2295	2295
q4	1476	1850	1373	1373
q5	4403	4786	4648	4648
q6	161	154	124	124
q7	2127	1992	1840	1840
q8	2677	2791	2702	2702
q9	7273	7216	7143	7143
q10	2972	3262	2789	2789
q11	611	506	521	506
q12	690	766	627	627
q13	3515	4024	3260	3260
q14	280	304	271	271
q15	519	470	466	466
q16	655	700	637	637
q17	1217	1718	1256	1256
q18	7721	7359	7335	7335
q19	790	1157	1106	1106
q20	2015	2113	1887	1887
q21	5617	5338	4965	4965
q22	607	592	553	553
Total cold run time: 53305 ms
Total hot run time: 51467 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187954 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7309a1044f17bf7bd182d5bd29f0533fd7ef1586, data reload: false

query1	966	386	375	375
query2	6523	2098	2084	2084
query3	6796	214	211	211
query4	33540	24021	23083	23083
query5	4294	594	448	448
query6	288	197	188	188
query7	4603	479	315	315
query8	289	262	221	221
query9	9551	2707	2672	2672
query10	442	313	246	246
query11	18313	15409	14967	14967
query12	158	104	103	103
query13	1639	540	405	405
query14	10158	6437	6965	6437
query15	245	197	188	188
query16	7764	617	479	479
query17	1615	749	584	584
query18	1998	413	315	315
query19	212	182	161	161
query20	127	114	111	111
query21	212	124	104	104
query22	4214	4202	4288	4202
query23	34021	33248	33509	33248
query24	6612	2245	2298	2245
query25	520	492	423	423
query26	1193	276	159	159
query27	1979	474	332	332
query28	5263	2461	2427	2427
query29	651	550	418	418
query30	235	188	158	158
query31	953	886	771	771
query32	79	64	61	61
query33	515	363	307	307
query34	763	854	550	550
query35	812	820	759	759
query36	1055	1039	955	955
query37	133	98	82	82
query38	4192	4241	4225	4225
query39	1449	1385	1394	1385
query40	212	115	102	102
query41	51	50	51	50
query42	120	108	104	104
query43	516	549	498	498
query44	1298	812	794	794
query45	194	175	168	168
query46	848	1027	641	641
query47	1844	1848	1769	1769
query48	378	396	325	325
query49	788	485	393	393
query50	632	650	400	400
query51	7032	6885	6766	6766
query52	99	106	94	94
query53	235	250	203	203
query54	478	497	426	426
query55	84	77	82	77
query56	266	286	250	250
query57	1149	1151	1096	1096
query58	243	231	261	231
query59	2910	3032	2891	2891
query60	267	271	244	244
query61	134	113	112	112
query62	794	716	683	683
query63	228	192	192	192
query64	4396	1020	665	665
query65	3337	3220	3157	3157
query66	1054	407	309	309
query67	15925	15614	15283	15283
query68	5141	821	537	537
query69	463	292	261	261
query70	1196	1165	1108	1108
query71	374	297	251	251
query72	5742	4056	3901	3901
query73	649	756	365	365
query74	9816	9164	9112	9112
query75	3211	3182	2661	2661
query76	3169	1185	867	867
query77	462	384	302	302
query78	10233	10065	9255	9255
query79	2354	795	615	615
query80	732	537	436	436
query81	526	289	241	241
query82	244	151	123	123
query83	174	174	151	151
query84	238	96	71	71
query85	807	373	303	303
query86	386	315	287	287
query87	4515	4472	4304	4304
query88	4244	2222	2196	2196
query89	388	334	296	296
query90	1970	192	192	192
query91	130	135	108	108
query92	68	58	56	56
query93	2599	852	544	544
query94	664	412	285	285
query95	339	265	264	264
query96	506	615	286	286
query97	2780	2827	2787	2787
query98	226	197	200	197
query99	1255	1367	1265	1265
Total cold run time: 285485 ms
Total hot run time: 187954 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.05 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7309a1044f17bf7bd182d5bd29f0533fd7ef1586, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.03
query3	0.24	0.07	0.07
query4	1.61	0.10	0.11
query5	0.43	0.42	0.41
query6	1.14	0.66	0.65
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.60	0.51	0.52
query10	0.56	0.55	0.56
query11	0.15	0.11	0.10
query12	0.13	0.12	0.11
query13	0.61	0.59	0.59
query14	2.73	2.76	2.76
query15	0.90	0.83	0.82
query16	0.39	0.41	0.38
query17	0.96	0.96	1.07
query18	0.22	0.22	0.21
query19	1.85	1.78	1.96
query20	0.01	0.01	0.01
query21	15.36	0.93	0.59
query22	0.77	0.79	0.57
query23	15.37	1.35	0.50
query24	3.14	0.72	1.62
query25	0.17	0.10	0.04
query26	0.40	0.17	0.14
query27	0.06	0.06	0.06
query28	13.26	1.03	0.42
query29	12.57	3.88	3.25
query30	0.25	0.08	0.06
query31	2.83	0.59	0.38
query32	3.23	0.56	0.46
query33	3.07	3.02	3.01
query34	16.77	5.17	4.50
query35	4.51	4.46	4.49
query36	0.81	0.48	0.49
query37	0.10	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.02
query40	0.16	0.12	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.76 s
Total hot run time: 30.05 s

@morrySnow
Copy link
Contributor Author

run p0

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jan 16, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit f3d8bce into apache:master Jan 16, 2025
26 of 27 checks passed
@morrySnow morrySnow deleted the try_fix_insert_lock branch January 16, 2025 04:08
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Jan 17, 2025
@morrySnow
branch-2.1: [fix](Nereids) insert lock all target tables apache#47033
4f51fc5 
pick from master apache#47033

Related PR: apache#45045

Problem Summary:

we get target table twice, lock them all
@morrySnow morrySnow mentioned this pull request Jan 17, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Jan 17, 2025
@morrySnow
[fix](Nereids) insert lock all target tables (#47033)
bcba791 
### What problem does this PR solve?

Related PR: #45045

Problem Summary:

we get target table twice, lock them all
@github-actions github-actions bot mentioned this pull request Jan 17, 2025
Closed
morrySnow added a commit that referenced this pull request Jan 17, 2025
@morrySnow
branch-2.1: [fix](Nereids) insert lock all target tables #47033 (#47172)
257e07a 
pick from master #47033
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Jan 20, 2025
@morrySnow
branch-3.0: [fix](Nereids) insert lock all target tables apache#47033
05fc58c 
pick from master apache#47033
@morrySnow morrySnow mentioned this pull request Jan 20, 2025
Merged
dataroaring pushed a commit that referenced this pull request Jan 21, 2025
@morrySnow
branch-3.0: [fix](Nereids) insert lock all target tables #47033 (#47243)
7e6fbd6 
pick from master #47033
@morrySnow morrySnow mentioned this pull request Jan 23, 2025
Merged
16 tasks
morrySnow added a commit that referenced this pull request Jan 24, 2025
@morrySnow
[fix](Nereids) Use the schema saved during planning as the schema of …
4aaa657 
…the original target table (#47337)

### What problem does this PR solve?

Related PR: #47033 #45045

Problem Summary:

because schema change does not involve recreating the table object, but
rather rebuilding the full schema. So, we should use the schema saved
during planning as the schema of the original target table.
github-actions bot pushed a commit that referenced this pull request Jan 24, 2025
@morrySnow
[fix](Nereids) Use the schema saved during planning as the schema of …
bbeece8 
…the original target table (#47337)

### What problem does this PR solve?

Related PR: #47033 #45045

Problem Summary:

because schema change does not involve recreating the table object, but
rather rebuilding the full schema. So, we should use the schema saved
during planning as the schema of the original target table.
morrySnow added a commit that referenced this pull request Jan 24, 2025
@morrySnow
[fix](Nereids) Use the schema saved during planning as the schema of …
0c9de65 
…the original target table (#47337)

Related PR: #47033 #45045

Problem Summary:

because schema change does not involve recreating the table object, but
rather rebuilding the full schema. So, we should use the schema saved
during planning as the schema of the original target table.
morrySnow added a commit that referenced this pull request Jan 24, 2025
@morrySnow
[fix](Nereids) Use the schema saved during planning as the schema of …
baa63dc 
…the original target table (#47337)

Related PR: #47033 #45045

Problem Summary:

because schema change does not involve recreating the table object, but
rather rebuilding the full schema. So, we should use the schema saved
during planning as the schema of the original target table.
morrySnow added a commit that referenced this pull request Feb 11, 2025
@morrySnow
branch-2.1: [fix](Nereids) Use the schema saved during planning as th…
91f8d4e 
…e schema of the original target table #47337

pick from master #47337

Related PR: #47033 #45045

Problem Summary:

because schema change does not involve recreating the table object, but
rather rebuilding the full schema. So, we should use the schema saved
during planning as the schema of the original target table.
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@morrySnow @lzyy2024
[fix](Nereids) insert lock all target tables (apache#47033)
a862d28 
### What problem does this PR solve?

Related PR: apache#45045

Problem Summary:

we get target table twice, lock them all
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@morrySnow @lzyy2024
[fix](Nereids) Use the schema saved during planning as the schema of …
e22d40c 
…the original target table (apache#47337)

### What problem does this PR solve?

Related PR: apache#47033 apache#45045

Problem Summary:

because schema change does not involve recreating the table object, but
rather rebuilding the full schema. So, we should use the schema saved
during planning as the schema of the original target table.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@starocean999 starocean999 starocean999 approved these changes

@Jibing-Li Jibing-Li Jibing-Li approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@morrySnow @Thearas @doris-robot @starocean999 @Jibing-Li @dataroaring