[BugFix] Fix dynamic partition table unexpectly stop scheduling #45235

nshangyiming · 2024-05-07T12:25:21Z

Periodically re-collect all the schedulable tables and put them into scheduling.
Also refactor some code and add more logging to make it easier to debug problems.

Fixes SR-27131

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

Signed-off-by: Dejun Xia <[email protected]>

fe/fe-core/src/main/java/com/starrocks/clone/DynamicPartitionScheduler.java

fe/fe-core/src/main/java/com/starrocks/common/util/DynamicPartitionUtil.java

fe/fe-core/src/main/java/com/starrocks/catalog/DynamicPartitionProperty.java

sonarqubecloud · 2024-05-07T12:33:09Z

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-05-08T04:54:11Z

[FE Incremental Coverage Report]

✅ pass : 77 / 91 (84.62%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/common/proc/CurrentQueryBackendInstanceProcDir.java	0	1	00.00%	[47]
🔵	com/starrocks/common/util/DynamicPartitionUtil.java	28	34	82.35%	[437, 447, 449, 458, 459, 461]
🔵	com/starrocks/catalog/DynamicPartitionProperty.java	6	7	85.71%	[205]
🔵	com/starrocks/clone/DynamicPartitionScheduler.java	36	42	85.71%	[528, 529, 692, 693, 698, 699]
🔵	com/starrocks/catalog/Database.java	1	1	100.00%	[]
🔵	com/starrocks/catalog/OlapTable.java	2	2	100.00%	[]
🔵	com/starrocks/sql/ast/ShowDynamicPartitionStmt.java	1	1	100.00%	[]
🔵	com/starrocks/qe/ShowExecutor.java	3	3	100.00%	[]

github-actions · 2024-05-08T04:54:16Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-05-08T06:37:48Z

@Mergifyio backport branch-3.3

github-actions · 2024-05-08T06:37:50Z

@Mergifyio backport branch-3.2

github-actions · 2024-05-08T06:37:51Z

@Mergifyio backport branch-3.1

github-actions · 2024-05-08T06:37:53Z

@Mergifyio backport branch-3.0

mergify · 2024-05-08T06:38:04Z

backport branch-2.5

✅ Backports have been created

#45271 [BugFix] Fix dynamic partition table unexpectly stop scheduling (backport #45235) has been created for branch branch-2.5 but encountered conflicts

Signed-off-by: Dejun Xia <[email protected]> (cherry picked from commit d97d273) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/clone/DynamicPartitionScheduler.java

Signed-off-by: Dejun Xia <[email protected]> (cherry picked from commit d97d273) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/catalog/DynamicPartitionProperty.java # fe/fe-core/src/main/java/com/starrocks/clone/DynamicPartitionScheduler.java # fe/fe-core/src/main/java/com/starrocks/qe/ShowExecutor.java

Signed-off-by: Dejun Xia <[email protected]> (cherry picked from commit d97d273) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/catalog/DynamicPartitionProperty.java # fe/fe-core/src/main/java/com/starrocks/clone/DynamicPartitionScheduler.java # fe/fe-core/src/main/java/com/starrocks/common/util/DynamicPartitionUtil.java # fe/fe-core/src/main/java/com/starrocks/qe/ShowExecutor.java

Signed-off-by: Dejun Xia <[email protected]> (cherry picked from commit d97d273) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/catalog/DynamicPartitionProperty.java # fe/fe-core/src/main/java/com/starrocks/catalog/OlapTable.java # fe/fe-core/src/main/java/com/starrocks/clone/DynamicPartitionScheduler.java # fe/fe-core/src/main/java/com/starrocks/common/util/DynamicPartitionUtil.java # fe/fe-core/src/main/java/com/starrocks/qe/ShowExecutor.java

…port #45235) (#45269) Signed-off-by: yiming <[email protected]> Co-authored-by: yiming <[email protected]>

…port #45235) (#45268) Signed-off-by: yiming <[email protected]> Co-authored-by: yiming <[email protected]>

…port #45235) (#45269) Signed-off-by: yiming <[email protected]> Co-authored-by: yiming <[email protected]> (cherry picked from commit f2020df) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/catalog/OlapTable.java # fe/fe-core/src/main/java/com/starrocks/clone/DynamicPartitionScheduler.java

…port #45235) (#45313) Signed-off-by: yiming <[email protected]> Signed-off-by: Dejun Xia <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: yiming <[email protected]>

…port #45235) (#45313) Signed-off-by: yiming <[email protected]> Signed-off-by: Dejun Xia <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: yiming <[email protected]> (cherry picked from commit 5ef58ac) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/clone/DynamicPartitionScheduler.java # fe/fe-core/src/main/java/com/starrocks/common/util/DynamicPartitionUtil.java # fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java

…Rocks#45235) Signed-off-by: Dejun Xia <[email protected]>

…port #45235) (#45350) Signed-off-by: Dejun Xia <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Dejun Xia <[email protected]>

…port #45235) (#45267) Signed-off-by: yiming <[email protected]> Co-authored-by: yiming <[email protected]>

nshangyiming added 2 commits May 6, 2024 16:22

fix

cb210b3

Signed-off-by: Dejun Xia <[email protected]>

[BugFix] Fix dynamic partition table unexpectly stop scheduling

4d4b598

Signed-off-by: Dejun Xia <[email protected]>

nshangyiming requested review from a team as code owners May 7, 2024 12:25

starrocks-cr bot reviewed May 7, 2024

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/clone/DynamicPartitionScheduler.java Show resolved Hide resolved

starrocks-cr bot reviewed May 7, 2024

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/common/util/DynamicPartitionUtil.java Show resolved Hide resolved

starrocks-cr bot reviewed May 7, 2024

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/catalog/DynamicPartitionProperty.java Show resolved Hide resolved

github-actions bot added 3.3 3.2 3.1 3.0 2.5 labels May 7, 2024

nshangyiming changed the title ~~[BugFix] Fix dynamic partition table unexpectly stop schedulin~~ [BugFix] Fix dynamic partition table unexpectly stop scheduling May 7, 2024

mergify bot assigned nshangyiming May 7, 2024

Astralidea approved these changes May 8, 2024

View reviewed changes

nshangyiming enabled auto-merge (squash) May 8, 2024 06:18

HangyuanLiu approved these changes May 8, 2024

View reviewed changes

meegoo approved these changes May 8, 2024

View reviewed changes

nshangyiming merged commit d97d273 into StarRocks:main May 8, 2024
63 checks passed

github-actions bot removed the 3.3 label May 8, 2024

github-actions bot removed 3.2 3.1 labels May 8, 2024

mergify bot mentioned this pull request May 8, 2024

[BugFix] Fix dynamic partition table unexpectly stop scheduling (backport #45235) #45267

Merged

42 tasks

mergify bot mentioned this pull request May 8, 2024

[BugFix] Fix dynamic partition table unexpectly stop scheduling (backport #45235) #45268

Merged

42 tasks

mergify bot mentioned this pull request May 8, 2024

[BugFix] Fix dynamic partition table unexpectly stop scheduling (backport #45235) #45269

Merged

42 tasks

mergify bot mentioned this pull request May 8, 2024

[BugFix] Fix dynamic partition table unexpectly stop scheduling (backport #45235) #45270

Closed

42 tasks

mergify bot mentioned this pull request May 8, 2024

[BugFix] Fix dynamic partition table unexpectly stop scheduling (backport #45235) #45271

Closed

42 tasks

github-actions bot added 3.3 3.2 3.1 2.5 labels May 8, 2024

wanpengfei-git pushed a commit that referenced this pull request May 8, 2024

[BugFix] Fix dynamic partition table unexpectly stop scheduling (back…

f2020df

…port #45235) (#45269) Signed-off-by: yiming <[email protected]> Co-authored-by: yiming <[email protected]>

github-actions bot added the 3.1-merged label May 8, 2024

wanpengfei-git pushed a commit that referenced this pull request May 8, 2024

[BugFix] Fix dynamic partition table unexpectly stop scheduling (back…

315a01f

…port #45235) (#45268) Signed-off-by: yiming <[email protected]> Co-authored-by: yiming <[email protected]>

github-actions bot added the 3.2-merged label May 8, 2024

mergify bot mentioned this pull request May 8, 2024

[BugFix] Fix dynamic partition table unexpectly stop scheduling (backport #45235) #45313

Merged

42 tasks

github-actions bot added the 3.0-merged label May 9, 2024

mergify bot mentioned this pull request May 9, 2024

[BugFix] Fix dynamic partition table unexpectly stop scheduling (backport #45235) #45350

Merged

42 tasks

node pushed a commit to vivo/starrocks that referenced this pull request May 9, 2024

[BugFix] Fix dynamic partition table unexpectly stop scheduling (Star…

6ba7913

…Rocks#45235) Signed-off-by: Dejun Xia <[email protected]>

github-actions bot added the 2.5-merged label May 9, 2024

nshangyiming added a commit that referenced this pull request May 10, 2024

[BugFix] Fix dynamic partition table unexpectly stop scheduling (back…

227a55f

…port #45235) (#45267) Signed-off-by: yiming <[email protected]> Co-authored-by: yiming <[email protected]>

github-actions bot added the 3.3-merged label May 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BugFix] Fix dynamic partition table unexpectly stop scheduling #45235

[BugFix] Fix dynamic partition table unexpectly stop scheduling #45235

nshangyiming commented May 7, 2024 •

edited

Loading

sonarqubecloud bot commented May 7, 2024

github-actions bot commented May 8, 2024

github-actions bot commented May 8, 2024

github-actions bot commented May 8, 2024

github-actions bot commented May 8, 2024

github-actions bot commented May 8, 2024

github-actions bot commented May 8, 2024

mergify bot commented May 8, 2024 •

edited

Loading

[BugFix] Fix dynamic partition table unexpectly stop scheduling #45235

[BugFix] Fix dynamic partition table unexpectly stop scheduling #45235

Conversation

nshangyiming commented May 7, 2024 • edited Loading

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

sonarqubecloud bot commented May 7, 2024

Quality Gate passed

github-actions bot commented May 8, 2024

[FE Incremental Coverage Report]

file detail

github-actions bot commented May 8, 2024

[BE Incremental Coverage Report]

github-actions bot commented May 8, 2024

github-actions bot commented May 8, 2024

github-actions bot commented May 8, 2024

github-actions bot commented May 8, 2024

mergify bot commented May 8, 2024 • edited Loading

✅ Backports have been created

nshangyiming commented May 7, 2024 •

edited

Loading

mergify bot commented May 8, 2024 •

edited

Loading