Skip to content

Commit

Permalink
Second Read Corrections
Browse files Browse the repository at this point in the history
  • Loading branch information
@ralikio
ralikio committed Jan 29, 2025
1 parent b020638 commit c949335
Showing 1 changed file with 50 additions and 57 deletions.
107 changes: 50 additions & 57 deletions docs/contributor/03-11-hap-rules.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,22 +3,23 @@
> [!NOTE]
> This feature is still being developed and will be available soon.
The Hyperscaler Account Pool (HAP) Rule Configuration is a functionality that allows to control pool selection from a configurable helm property.
The HAP Pool is a set of Gardener's SecretBindings K8S resources that share the same labels.
The HAP Pool Selection is a process that for every SKR creation request decides on which secret binding is the most suitable to use.
The process generates a list of [search labels](#search-labels) based on SKR's attributes.
The [search labels](#search-labels) are used to query Gardener's Secret Bindings.
[search labels](#search-labels) are generated by evaluating a Rule consisting of Rule Entries and if an entry applies to the request then the rule outputs its specific [search labels](#search-labels) to query the correct HAP Pool.
More about HAP process can be found in [Hyperscaler Account Pool](03-10-hyperscaler-account-pool.md) document.
The Hyperscaler Account Pool (HAP) Rule Configuration is a functionality that allows to control pool selection from a configurable helm property.
A pool is a set of Gardener's SecretBinding resources that share the same search labels.
Pool selection is a process that decides on which SecretBinding pool is the most suitable to use for every SKR creation request.
The process generates a list of [search labels](#search-labels) based on SKR's attributes.

The [search labels](#search-labels) are used to query Gardener's secret bindings.
[search labels](#search-labels) are generated by evaluating a rule consisting of rule entries and if an entry applies to the request then the rule outputs its specific [search labels](#search-labels) to query the correct pool.
More about HAP process can be found in [Hyperscaler Account Pool](03-10-hyperscaler-account-pool.md) document.

![Rules Evaluation Process](../assets/hap-rules-general.drawio.svg)

## Rule Configuration

The Rule is configured in values.yaml in `hap.rule` property.
It is a list of strings, each of which is a single Rule Entry that corresponds to a single HAP Pool.
Each Rule Entry is evaluated during a cluster provisioning process.
If more than one rule applies the best matching rule is selected based on the rule [priority](#priority). The following examples presents the Rule Configuration in values.yaml:
The rule is configured in values.yaml in `hap.rule` property.
It is a list of strings, each of which is a single rule entry that corresponds to a single pool.
Each rule entry is evaluated during cluster provisioning process.
If more than one rule applies the best matching rule is selected based on the [priority](#priority). The following examples presents example configuration in values.yaml:

```
hap:
Expand All @@ -29,7 +30,7 @@ hap:
- rule_n
```

Every HAP Pool that the Rule Entry represents must be preconfigured in a Gardener Cluster that KEB connects to.
Every pool that a rule entry represents must be preconfigured in a Gardener cluster that KEB connects to.

## Rule Format

Expand All @@ -39,8 +40,8 @@ The following example shows the Rule Entry format:
PLAN(INPUT_ATTR_1=VAL_1, INPUT_ATTR_2=VAL_2, ..., INPUT_ATTR_N=VAL_N) -> OUTPUT_ATTR_1, OUTPUT_ATTR_2, ..., OUTPUT_ATTR_M
```

Every rule entry consists of input and output attributes separated by the arrow symbol - `->`.
Input attributes are used to match the rule entry with the SKR request and to modify [search labels](#search-labels). Output attributes are only used to modify the [search labels](#search-labels).
Every rule entry consists of input and output attributes separated by the arrow symbol - `->`.
Input attributes are used to match rule entry with the SKR request and to modify [search labels](#search-labels). Output attributes are only used to modify the [search labels](#search-labels).
In its minimal form each rule consists of a `PLAN` that the rule applies to.
In its extended form a rule entry contains lists of input attributes and its values passed in the form of `ATTR=VAL` pairs in parantheses.

Expand All @@ -51,7 +52,7 @@ In its extended form a rule entry contains lists of input attributes and its val
Output attributes does not support values.
Description of when the rule is triggered can be found in the [Rule Evaluation](#evaluation) section.
Rule Attributes are described in [Rule Attributes](#rule-attributes) section.
Rule attributes are described in [Rule Attributes](#rule-attributes) section.

Currently, possible OUTPUT_ATTR_x attributes values are `S` or `EU`.

Expand All @@ -61,29 +62,22 @@ hap:
- azure(INPUT_ATTR_1=VAL_1,INPUT_ATTR_2=VAL_2,...,INPUT_ATTR_N) -> OUTPUT_ATTR_1=VAL_1,OUTPUT_ATTR_2=VAL_2,...,OUTPUT_ATTR_M
```

The input attribute types include **platformRegion** (`PR`), **hyperscalerRegion** (`HR`). The output attribute types include **shared** (`S`) and **euAccess** (`EU`).
The input attribute types include **platformRegion** (`PR`) and **hyperscalerRegion** (`HR`). The output attribute types include **shared** (`S`) and **euAccess** (`EU`).
Each attribute can be used at most once in a single rule entry.
**shared** and **euAccess** attributes are used only to apply their labels if rest of matched attributes are equal to SKR's values.

## Rule Evaluation

HAP evaluates a set of rules to determine what labels to use when querying secret bindings.
Rule is evaluated during cluster provisioning.
Rule entries are analyzed one by one.
Input attributes from every entry are compared with values from SKR input, if they are the same the rule entry is considered matched.
Rule is evaluated during cluster provisioning. Rule entries are analyzed one by one.
Input attributes from every entry are compared with values from SKR input. If they are the same then the rule entry is considered matched.
If there is more than one triggered rule then only one is selected in accordance to priority rules described in the [Priority](#priority) section.
If there are no rule entries trigered then the error is returned. No fallback behaviour is defined in such case.

## Uniqnuess & Priority

Only one rule can be triggered.

If more than one rule entry match the request then only one is selected and applied.

The process of selecting the best matching rule is based on a rule uniqueness and priority.

Only one rule can be triggered. If more than one rule entry match the request then only one is selected and applied. The process of selecting the best matching rule is based on a rule uniqueness and priority.
Rule entry uniqueness is determined by its plan and input parameters with its values (identification attributes) not including input parameters with `*`.

Output parameters are not taken into account when establishing rule entry uniqueness. For example the following rule will fail on startup because every SKR can be matched by both rules:

```
Expand All @@ -97,15 +91,14 @@ hap:
```

Rule configuration must contain only unique entries in the scope of that list.
If rule configuration contains duplicated rule entries an error failing KEB startup is returned.

Rule entry priority is selected by sorting all the rules entries that apply to the request by the number of identification attributes they contain.
For example, a rule with no attributes and only plan has lower priority then a rule with the same plan but a platform region attribute (`gcp` < `gcp(PR=cf-sa30)`).
If rule configuration contains duplicated rule entries an error that fails KEB startup is returned.

Rule entry priority is selected by sorting all rule entries that apply to the request by the number of identification attributes they contain.
For example, a rule with no attributes and only a plan has lower priority then a rule with the same plan but a platform region attribute (`gcp` < `gcp(PR=cf-sa30)`).
After sorting the entry that specifies most attributes (hence is the most specific) is selected.
Input attributes with value `*` is not taken into account when calculating priority.
Input attributes with value `*` are not taken into account when calculating priority.

The following example shows the priority of the rules from the general rule to more specific ones:
The following example shows priority of rules starting from the lowest priority:

```
aws -> S # search labels: hyperscalerType: aws, shared: true
Expand All @@ -115,19 +108,11 @@ aws(PR=cf-eu11, HR=westeu) -> EU, S # search labels: hyperscalerType: aws_cf-eu1

## Search Labels

The HAP stores credentials to the hyperscaler accounts in Kubernetes Secrets that the SecretBindings points to. KEB searches for those SecretBindings using thier labels **hyperscalerType**, **shared** and **euAccess**.
HAP stores credentials to hyperscaler accounts in Kubernetes Secrets that SecretBindings points to. KEB searches for SecretBindings using labels **hyperscalerType**, **shared** and **euAccess**.

The **hyperscaler-type** contains hyperscaler name and region information in the format `hyperscaler_type: <HYPERSCALER_NAME>[_<PLATFORM_REGION>][_<HYPERSCALER_REGION>]`, where both `_<PLATFORM_REGION>` and `_<HYPERSCALER_REGION>` are optional. The **hypercaler-type** label is mandatory. It is value is hardcoded and computed based on a plan in [hyperscaler_type.go](https://github.com/kyma-project/kyma-environment-broker/blob/main/common/hyperscaler/hyperscaler_type.go). This means that not all plans share their name with hyperscaler types, e.g. sap-converged-plan has `openstack` hyperscalerType and `trial` plan can have either `azure` or `aws` depending on the configured provider type. The **euAccess** and **shared** labels contain boolean values and they used to divide existing pools to secrets used by EU restricted regions and secrets shared by multiple Global Accounts. The **euAccess** and **shared** labels are optional.
The **hyperscaler-type** contains hyperscaler name and region information in the format of `hyperscaler_type: <HYPERSCALER_NAME>[_<PLATFORM_REGION>][_<HYPERSCALER_REGION>]`, where both `_<PLATFORM_REGION>` and `_<HYPERSCALER_REGION>` are optional. The **hypercaler-type** label is mandatory. Its value is computed based on a plan and regions and mapped in [hyperscaler_type.go](https://github.com/kyma-project/kyma-environment-broker/blob/main/common/hyperscaler/hyperscaler_type.go). Not all plans share their name with hyperscaler types, e.g. sap-converged-plan has `openstack` hyperscalerType and `trial` plan can have either `azure` or `aws` depending on the configured provider type.

Every rule must contain at least a plan and apply `hyperscaler_type: <HYPERSCALER_NAME>` label. For example, the following example shows a simple rule entry configuration and a SecretBinding pool that this configuration corresponds to:

```
hap:
rule:
- gcp # pool: hyperscalerType: gcp
```

The following table shows the mapping of plans to hyperscaler types:
The following table shows a mapping of plans to hyperscaler types:

| Plan | Hyperscaler Type |
|--------------------- |------------------ |
Expand All @@ -139,17 +124,27 @@ The following table shows the mapping of plans to hyperscaler types:
| sap-converged-cloud | openstack_<HR> |
| trial | aws |

Rest of the examples in the document follow the same structure of presenting an example configuration and corresponding HAP Pools as above snippet with corresponding pool in the rule's comment.
The **euAccess** and **shared** labels contain boolean values and they are used to divide existing pools to secrets used by EU restricted regions and secrets shared by multiple Global Accounts. The **euAccess** and **shared** labels are optional.

Every rule must contain at least a plan and apply `hyperscaler_type: <HYPERSCALER_NAME>` label. The following example shows a simple rule entry configuration and a SecretBinding pool that this configuration corresponds to:

```
hap:
rule:
- gcp # pool: hyperscalerType: gcp
```

Rest of examples in the document follow the same structure of presenting a configuration and its corresponding pools in the comment as the snippet above.

## Rule Attributes

Existence of an attribute in the rule means a set of clusters it applies to is constrained not only by a plan but by the attribute itself. The following section describes attributes that can be used in the rule entry.
Existence of an attribute in a rule means a set of clusters it applies to is constrained not only by a plan but by the attribute itself. The following section describes attributes that can be used in rule entries.

### Platform Region Attribute

It is possible to extend the rule entry with specified platform region by adding a `PR` rule attribute. If an actual value of SKR's platform region attribute matches PR rule attribute then `hyperscalerType: <HYPERSCALER_NAME>[_<PLATFORM_REGION>]` label is used.
It is possible to extend a rule entry with specified platform region by adding a `PR` rule attribute. If an actual value of SKR's platform region attribute matches the PR rule attribute then `hyperscalerType: <HYPERSCALER_NAME>[_<PLATFORM_REGION>]` label is used.

Below configuration means that if a gcp cluster is provisioned in cf-sa30 platform region then KEB would search for secret bindings with `hyperscalerType: gcp_cf-sa30` label.
Below configuration means that if a gcp cluster is provisioned in cf-sa30 platform region, then KEB would search for secret bindings with `hyperscalerType: gcp_cf-sa30` label.

```
hap:
Expand All @@ -159,7 +154,7 @@ hap:

### Hyperscaler Region Attribute

A region that an SKR is provisioned in can be matched with an attribute named `HR`. Below configuration specifies azure plan as one for which to use hyperscaler region based search. In this case, cluster provisioned in with the azure plan would require a secret binding with `hyperscalerType: azure_<HYPERSCALER_REGION>` label.
A region that an SKR is provisioned in can be matched with an attribute named `HR`. Below configuration specifies the azure plan as one for which to use hyperscaler region based search. In this case, cluster provisioned with the azure plan would require a secret binding with `hyperscalerType: azure_<HYPERSCALER_REGION>` label.

```
hap:
Expand All @@ -169,7 +164,7 @@ hap:

### Shared & EU Access attribute

The `shared` and `euAccess` attributes does not correspond to any SKR's property. Those atributes only used to add its search labels. If the rule entry contains `shared` or `euAccess` attributes then when triggered a `shared: true` or `euAccess: true` labels are added to the [search labels](#search-labels) depending on which one of those two attributes is listed in the rule entry. Shared label on a SecretBindings marked it as assignable to more than one SKR and euAccess is a non-functional label not used by KEB (more on that can be found in [HAP Pool document](03-10-hyperscaler-account-pool.md)). Below configuration specifies that all gcp clusters will use the same pool of shared secret bindings marked with labels `hyperscalerType: gcp`, `shared: true` and azure clusters in the region cf-ch20 will use the pool of secret bindings marked with labels `hyperscalerType: azure`, `euAccess: true`.
The `shared` and `euAccess` attributes does not correspond to any SKR's property. Those atributes are only used to add search labels. If the rule entry contains `shared` or `euAccess` attributes then when triggered, a `shared: true` or `euAccess: true` labels are added to the [search labels](#search-labels). Shared label on a SecretBindings marks it as assignable to more than one SKR and euAccess is used to mark EU regions (more on that can be found in [HAP Pool document](03-10-hyperscaler-account-pool.md)). Below configuration specifies that all gcp clusters will use the same pool of shared secret bindings marked with labels `hyperscalerType: gcp`, `shared: true` and azure clusters in the region cf-ch20 will use a pool of secret bindings marked with labels `hyperscalerType: azure`, `euAccess: true`.

```
hap:
Expand All @@ -180,7 +175,7 @@ hap:

### Attribute with "*"

Input attributes values can be replaced with `*` which means that all SKR values of that attribute are matched and used when creating [search labels](#search-labels). For example, the following configuration makes KEB search SecretBindings with `hyperscalerType: gcp_<PR>`, where `PR` corresponds to an SKR value, for all `gcp` clusters:
Input attributes' values can be replaced with `*`, which means that all SKR's values of that attribute are matched and used when creating [search labels](#search-labels). For example, the following configuration makes KEB search for SecretBindings with `hyperscalerType: gcp_<PR>`, where `PR` corresponds to an SKR value, for all `gcp` clusters:

```
hap:
Expand All @@ -192,7 +187,7 @@ hap:
The attributes that support `*` are: `PR`, `HR`.

> ![NOTE]
> The above configuration example effectively disables usage of SecretBindings with only `hyperscalerType: gcp` label.
> The above configuration example effectively disables usage of SecretBindings labeled only with `hyperscalerType: gcp`.
### Atributes Summary

Expand All @@ -218,16 +213,14 @@ KEB validates HAP Rules during startup. If the configuration is invalid, KEB wil

The constraints used for validation during KEB startup include:
* Rules format check - all the rules must comply with the format specified above.
* Every rule Entry must refer to an existing plan - if you define a rule with non existing pools then an error is returned during KEB startup that fails the process.
* Every supported plan needs at least one rule entry, if no rule entries for plan defined then error is returned for SKR.
* Uniqueness validation check - KEB checks that all rule entries are unique in the scope of the rule.
Specifying more than one rule with the same number of identification attributes is prohibited.
If more than one such attribute is specified then the error is returned failing KEB startup. For more details see [Uniqnuess & Priority](#uniqnuess--priority) section.
* Every rule entry must refer to an existing plan - if you define a rule with non existing plan then an error is returned during KEB startup that fails the process.
* Every supported plan needs at least one rule entry, if no rule entry is defined for a plan then an error is returned during KEB startup.
* Uniqueness validation check - KEB checks if all rule entries are unique in the scope of the rule. Specifying more than one entry with the same number of identification attributes is prohibited. If more than one such attribute is specified then the error is returned failing KEB startup. For more details see [Uniqnuess & Priority](#uniqnuess--priority) section.


## Initial Configuration

The last example shows initial configuration create to mimic the current bahaviour of KEB at the time of writing the document. The configuration enforces that:
The last example shows initial configuration created to mimic the current bahaviour of KEB at the time of writing the document. The configuration enforces that:
* azure, aws, gcp have their own pools of dedicated bindings.
* gcp clusters in the region cf-sa30 use the pool of secret bindings marked with labels: `hyperscalerType: gcp_cf-sa30`,
* sap-converged-cloud clusters use the pool of secret bindings marked with labels: `hyperscalerType: openstack_<HYPERSCALER_REGION>` and all these pools are shared.
Expand Down

0 comments on commit c949335

Please sign in to comment.