Sovereign Cloud/NS2: Configurable HAP mode per plan

[ebensom]

Description

Currently it is hardcoded in KEB provisioning process whether to use "shared" or "dedicated" hyperscaler account pool (HAP) via account provider. For trial and sap-converged-cloud plans, shared pool, while for others, the "dedicated" account pool is used.

The pool mode (i.e. "shared" or "dedicated") shall be made configurable per plan in KEB , or otherwise it shall be made configurable which plans should use shared HAP. The provision process shall be adjusted to consider the pool mode configuration for the given plan of the Kyma runtime when interacting with the account provider.

Reasons

Kyma runtime Sovereign Cloud / NS2 would work using shared pools for all supported plans.

Attachments

See how HAP works now: https://github.com/kyma-project/kyma-environment-broker/blob/main/docs/contributor/03-10-hyperscaler-account-pool.md

AC

• 1) Create a proposal how to handle such configuration per plan, and per BTP region
  • Hap Configuration Rules Doc #1643
  • New Hap Rule Doc #1675
  • Expression Parsing with Antlr and Simple String Splitting #1703
• 2) Implementation
  • 2.1) implementation with validating rules should be possible to be reused locally with some script. That we can check rules and see search labels per rule without a need to run KEB.
    • Rule Entry Parser #1698
  • 2.2) use above library in the KEB code/flow for HAP
    • Hap Rule Configs Wiring #1644
  • 2.3) when KEB starts we logs if validation failed, what faild. If pass without validation errors print all outcome search label for secret binding.
  • 2.4) final implementation in KEB done
  • 2.5) document this tool, that our users could use when designing rules
• 3) Configuration for every landscape
• 4) Docs
• 5) RNs

Progress:

• no release yet
• working on hap rules tool. Specification is finished.

[PK85]

1 ) Result of my analysis:

• plans trial (hyperscaler-type: trial) and sap-converged-cloud (hyperscaler-type: openstack_eu-de-1) use shared subscriptions that are marked with shared flag. NOTE (eu-de-1 is a hyperscaler region)
• eu_access cf-eu11 (hyperscaler-type: aws ) and cf-ch20 (hyperscaler-type: azure) use dedicated subscription that are marked with eu_access flag
• assured workloads for gcp cf-sa30 (hyperscaler-type: gcp_cf-sa30) use dedicated subscription. NOTE (cf-sa30 is a platform region)

Above rules are hardcoded in the KEB code.

2 ) Lets check if proposal from Wojciech would work, implementation proposal: https://github.com/kyma-project/kyma-environment-broker/pull/1609/files which keeps hardcoded rules + adds configuration option that is resolved at the end.

Configuration from the above PR:

hap: 
  sharedSecretPlans: "trial:*;sap-converged-cloud:*"

3 ) Example configurations and results

Example 1:

hap: 
  sharedSecretPlans: ""

Result 1:
Behaves exactly the same like configuration from point 2). All hardcoded rules are still in the code and have higher priority than sharedSecretPlans. Problem: I cannot say that trial or sap-converged-cloud must use dedicated subscription.

---

Example 2:

hap: 
  sharedSecretPlans: "aws:*"

Result 2:
Problem cf-eu11 is the aws region and it is hardcoded in KEB code that this is euAccess subscription required. That means subscription prepared for AWS provider will not work for this region, or some of them must be marked with eu_access flag. Same story for azure and cf-ch20 platform region. That means admins cannot have one shared pool for aws and one shared pool for azure

---

Example 3:

hap: 
  sharedSecretPlans: "gcp:*"

Result 3:
Problem that gcp cf-sa30 search for hyperscaler-type: gcp_cf-sa30. That means admins cannot have one shared pool for gcp

---

Example 4: Let say we want to remove hardcoded rules
Result 4: TODO what type of configuration we need to translate all hardcoded rules into it

[ralikio]

The example no 2 from #1520 (comment) has been dismissed because shared pools do not use euAccess functionality.

To address the above comments and cover all cases (especially about removing hardcoded configuration) I have analyzed the existing pools thinking how we can implement it starting from "zero".

Let's imaging that HAP pool functionality does not exist yet and we are doing selection of pools based only on simple HAP type (without any regions). The environment is populated with SecretBindings with label 'hyperscaler-type' (a new label, the same name as existing hyperscaler-type just for convenience, name can be tailored later) forming a pool of dedicated bindings per hyperscaler type that the plan is provisioned on:

In this case KEB selects a binding based on region that is a physical attribute known during the provisioning process. This might be simple label construction process without any complicated implementation.

Now adding shared attribute to the equation we end up with new pools created out of bindings with `shared: true' attribute. In this case we can say the the attribute is logical because it does not correspond to any physical feature directly. With addition of a new attribute we need a way to know when to choose the correct attribute but configure that selection rule (instead of currently hardcoded logic) we can reuse rules from the #1609 POC ending up with the following pools:

KEB can simplify its logic and use the rules only the deciding if shared: true label should be used when selecting a secret binding.

To make the selection replicate the current implementation we need to add euAccess pools selection. All of them are currently based on region selection, however, we do use another label euAccess: true to select correct binding. Because this feature we can handle that in two ways: we can select pools based on regions and if non found fallback to regular search or we can introduce a logical euAccess label with another set of configuration to compute if the label should be used for given SKR.

Because region selection is not only need for euAccess I recommend using the first approach, below shows the final configuration for all of our pools by adding KSA and open stack regional pools:

When selecting second or hybrid approach we could implement dynamic pool labels evaluation (aka. knowing what labels to add to the secret bindings query) based on helm values but I not recommend such approach since it is harder to maintain.

The selection algorithm might look like this:

Additional options include adding third configuration value for mapping of hyperscaler-type and leaving its suffix for as it is now.

[ralikio]

🎨 NEW DESIGN CHANGES 🎨

Initial functionality reorganization in POC PR together with new configuration examples: https://github.com/kyma-project/kyma-environment-broker/pull/1609/files#diff-6927a77757e4fe1d3078c5b81ce28e6d78d9623a161eab6f806f906d8494c0ecR245.

[ralikio]

Documentation refactored and can seen here: #1643
Extracted Configs in #1644

TODO:

• rules validation,
• rules processing

[jaroslaw-pieszka]

If there is any sorting involved when calculating priority we should be absolutely sure that only one item has the highest priority and the selection is unambiguous.

To avoid the case where we have two top priority items and select one of them kind of randomly.
Example (priority - rule name) :
4 - rule16
4 - rule1
3 - rule0
2 - rule3
1 - rule7

Two top priority rules (rule1 and rule 16)

Moreover we should consider checking this not only while parsing configuration but also when we try to find secret binding (having the actual values of plan, provider, hyperscaler region and platform region).