Extend Kyma Provisioning Parameters to allow multiple Worker Node Pools definitions

[PK85]

See:

kyma-project/kyma#18709 (comment)

Plan:

1. We keep our main worker pool nodes as they are. It will be for Kyma workloads
2. We need to support additionalWorkerNodes for customers (we decided that they are by default min 3 and three zones)

AC

• 0) extend schema with _load_current_config parameter (CIS released staging: DONE, canary: nope, prod: nope)
  Enable current configuration load on Preview #1654
• 1) design KEB schema API, validations, defaults. Present to the team.
• 2) use our preview plan where new schema is visible, and we update RuntimeCR
  Additional worker node pools schema on Preview #1659, Introduce HA zones #1668,
  Disable changing HA zones setting #1671, Additional workers in RuntimeCR #1691
• 3) Check how frogs wants to implement that in KIM, see: Multiple worker groups [KIM/feature] infrastructure-manager#46 (comment)
• 4) Implementation for all plans. DO NOT expose additionalWorkerPool for trial and free. Azure_lite max node same as main worker pool and HA not available in the schema, always one zone.
  Additional worker node pools feature flag #1695, Remove HA zones for azure lite plan #1697
• 5) new e2e SKR tests
• 6) docs Additional worker node pools docs #1617, Update additional worker node pools docs #1678
  • 6.1 Help Portal documentation:
    - [ ] Available Plans in the Kyma Environment
    - [ ] Provisioning and Updating Parameters in the Kyma Environment
    - [ ] [NEW] Additional Worker Node Pools (in the Administration and Operations in the Kyma Environment section)
    - [ ] RN
• 7) deploy (featureFlag introduced, when deployed on prod we can remove FF)
  • 7.1) dev
  • 7.2) stage
  • 7.3) prod

[KsaweryZietara]

23.12.2024

1. We've decided to extend machine type list with new types (for example GPUs) in next task.
2. We've decided to extend the schema with _load_current_config: true parameter, which loads current instance configuration into the update instance schema. In the next step we will test additionalWorkerNodePools on preview plan. If everything works, we will release additionalWorkerNodePools on every plan.

[KsaweryZietara]

Additional Worker Node Pools will be available on:

• azure, aws, gcp, sap-converged-cloud, preview:
  • autoScalerMin: 3 (default - 3)
  • autoScalerMax: 300 (default - 20)
• azure_lite:
  • autoScalerMin: 2 (default - 2)
  • autoScalerMax: 40 (default - 10)

[PK85]

After Kyma provisionin with additional worker pool, the KCP cli do not dispay new parameters, see:

"data": [
    {
      "instanceID": "6E28D25F-5AEA-4364-B2BE-C01A34A9BB94",
      "runtimeID": "3c99a607-59ea-4e07-bdc9-500299b93ec1",
      "globalAccountID": "8cd57dc2-edb2-45e0-af8b-7d881006e516",
      "subscriptionGlobalAccountID": "",
      "subAccountID": "6c6280f7-d1bc-47f2-b485-25e5f03ded4d",
      "region": "eu-central-1",
      "subAccountRegion": "cf-us10-staging",
      "shootName": "c-79b33ab",
      "serviceClassID": "47c9dcbf-ff30-448e-ab36-d3bad66ba281",
      "serviceClassName": "kymaruntime",
      "servicePlanID": "361c511f-f939-4621-b228-d0fb79a1fe15",
      "servicePlanName": "aws",
      "provider": "AWS",
      "parameters": {
        "autoScalerMin": 3,
        "autoScalerMax": 20,
        "name": "pk-provisioning-api-l16k6h7a",
        "machineType": "m6i.large",
        "region": "eu-central-1",
        "oidc": {
          "clientID": "",
          "groupsClaim": "",
          "issuerURL": "",
          "signingAlgs": [],
          "usernameClaim": "",
          "usernamePrefix": ""
        },
        "networking": {
          "nodes": "10.250.0.0/16",
          "pods": "10.96.0.0/13",
          "services": "10.104.0.0/13"
        },
        "modules": {
          "default": true,
          "list": null
        }
      },

[jaroslaw-pieszka]

Support for new parameters will be addressed in separate task (in progress already).