Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: scale-out new replicas through data actions #8425

Open
wants to merge 17 commits into
base: main
Choose a base branch
from

Conversation

leon-inf
Copy link
Contributor

@leon-inf leon-inf commented Nov 7, 2024

No description provided.

@github-actions github-actions bot added the size/XXL Denotes a PR that changes 1000+ lines. label Nov 7, 2024
@leon-inf leon-inf added this to the Release 1.0.0 milestone Nov 7, 2024
@leon-inf leon-inf force-pushed the support/hscale-out-by-data-action branch 2 times, most recently from a680c98 to 8649377 Compare November 8, 2024 02:29
@leon-inf leon-inf force-pushed the support/hscale-out-by-data-action branch from 8649377 to 0055daf Compare November 11, 2024 07:13
@leon-inf leon-inf requested a review from Ursasi November 11, 2024 09:49
pkg/controller/component/replicas.go Outdated Show resolved Hide resolved
pkg/kbagent/server/streaming_server.go Outdated Show resolved Hide resolved
pkg/kbagent/service/streaming.go Outdated Show resolved Hide resolved
pkg/kbagent/service/task.go Outdated Show resolved Hide resolved
pkg/kbagent/service/task_new_replica.go Show resolved Hide resolved
pkg/kbagent/setup.go Show resolved Hide resolved
controllers/apps/transformer_component_status.go Outdated Show resolved Hide resolved
@@ -41,7 +41,8 @@ import (
)

const (
defaultPort = 3501
defaultHTTPPort = 3501
defaultStreamingPort = 3502
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you consider using a single port to serve both kinds of actions? we can use the websocket protocol for data streaming.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will make the data streaming protocol more constrained and complicated if sharing a same port.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it won't be constrained because http is just a transport layer. but the advantage is that it only uses one port, it's useful when the cluster is deployed under the host network. I'm fine with both, just raise an option for consideration.

@leon-inf leon-inf force-pushed the support/hscale-out-by-data-action branch 4 times, most recently from 74b814d to 0cd4a4a Compare November 13, 2024 01:47
@leon-inf leon-inf force-pushed the support/hscale-out-by-data-action branch from 0cd4a4a to 87cf581 Compare November 13, 2024 02:23

anyPod := func() []*corev1.Pod {
i := rand.Int() % len(pods)
return []*corev1.Pod{pods[i]}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the target pod is random, will it affect multiple reconciles?

and it is better to use an available pod rather than a failed pod

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The controller does not check the availability of target Pods, and leaving it to the Addon and Action to choose, because the controller has no way of knowing which pods meet the availability requirement for a specific action.

rolePods = append(rolePods, pods[i])
}
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and if the target role pods are not available(such as seconaday), can we select a fallback pod?

@leon-inf leon-inf force-pushed the support/hscale-out-by-data-action branch from 7253759 to a13e5c2 Compare November 13, 2024 07:42
@leon-inf leon-inf force-pushed the support/hscale-out-by-data-action branch 3 times, most recently from c09b36d to 871a813 Compare November 14, 2024 02:43
@leon-inf leon-inf force-pushed the support/hscale-out-by-data-action branch from 871a813 to 6e7035d Compare November 14, 2024 03:09
@leon-inf leon-inf marked this pull request as ready for review November 14, 2024 03:11
@leon-inf leon-inf requested review from Y-Rookie and a team as code owners November 14, 2024 03:11
Copy link

codecov bot commented Nov 14, 2024

Codecov Report

Attention: Patch coverage is 31.97368% with 517 lines in your changes missing coverage. Please review.

Project coverage is 60.67%. Comparing base (7ec2c5c) to head (faedfaa).

Files with missing lines Patch % Lines
pkg/controller/component/replicas.go 0.00% 188 Missing ⚠️
pkg/kbagent/service/task.go 0.00% 93 Missing ⚠️
pkg/controller/component/kbagent.go 51.72% 53 Missing and 3 partials ⚠️
pkg/kbagent/service/task_new_replica.go 0.00% 51 Missing ⚠️
pkg/controller/component/kbagent_task_event.go 0.00% 33 Missing ⚠️
pkg/kbagent/service/streaming.go 31.25% 33 Missing ⚠️
controllers/apps/transformer_component_workload.go 65.16% 20 Missing and 11 partials ⚠️
pkg/controller/component/lifecycle/kbagent.go 75.00% 9 Missing and 1 partial ⚠️
pkg/kbagent/service/service.go 46.15% 7 Missing ⚠️
controllers/apps/transformer_component_status.go 66.66% 4 Missing and 2 partials ⚠️
... and 3 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8425      +/-   ##
==========================================
- Coverage   61.27%   60.67%   -0.61%     
==========================================
  Files         351      354       +3     
  Lines       41752    41821      +69     
==========================================
- Hits        25584    25375     -209     
- Misses      13870    14187     +317     
+ Partials     2298     2259      -39     
Flag Coverage Δ
unittests 60.67% <31.97%> (-0.61%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size/XXL Denotes a PR that changes 1000+ lines.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants