This project has adopted the Contributor Covenant in version 2.1 as our code of conduct. Please see the details in our CODE_OF_CONDUCT.md. All contributors must abide by the code of conduct.
By participating in this project, you agree to abide by its Code of Conduct at all times.
This project follows the REUSE standard for software licensing. Each file contains copyright and license information, and license texts can be found in the ./LICENSES folder. For more information visit https://reuse.software/.
This Gateway requires a PostgreSQL database that will be preconfigured by the Gateway's init container.
You can select a platform (e.g. caas) to use predefined settings (e.g. securityContext) specifically dedicated to the platform.
Note that you can overwrite platform specific values in the values.yaml.
To add a new platform specific values.yaml, add the required values as platforName.yaml to the platforms folder.
Note: Setting platform specific values for the sub-chart by the platform specific platformName.yaml of your main-chart will not work, as the sub-chart platforms have precedence.
No detailed configuration is necessary. PostgreSQL will be deployed together with the Gateway. You should change the default passwords!
If you want to add routes, services, etc. you can set specific curl command to deploy you preferc configuration.
The Gateway can be accessed via created Ingress/Route. See the Parameters section for details.
By default, URLs will have the format <Release.Name>[-<Suffix>]-<.Release.Namespace>.<.Values.global.domain>.
Setting dedicated hostnames for an ingress will overwrite the created URL. The value set in hostname needs to be fully qualified, as .Values.global.domain will not be added automatically!
Be aware that exposing the Admin-API for Community Edition can be dangerous, as the API is not protected by any RBAC. Thus it can be accessed by anyone having access to the API url.
Therefore the Admin-API-Ingress is disabled. For Mor details see External access.
By default, we protect the Admin API via a dedicated service and route together with the jwt-keycloak. You need to add the used issuer.
If you enable SSL verification the Gateway will try to verify all traffic against a bundle of trusted CA certificates which needs to be specified explicitely.
You can enable this by setting sslVerify to true in the values.yaml. If you do so, you must provide your own truststore by setting the trustedCaCertificates field with the content of your CA certificates in PEM format otherwise Kong won't start.
Example values.yaml:
trustedCaCertificates: |
-----BEGIN CERTIFICATE-----
<CA certificate 01 in PEM format here>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<CA certificate 02 in PEM format here>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<CA certificate 03 in PEM format here>
-----END CERTIFICATE-----Of course Helm let's you reference multiple values files when installing a deployment so you could also outsource trustedCaCertificates wo its own values file, for example my-trustes-ca-certificates.yaml.
Only TLS versions TLSv1.2 and TLSv.1.3 are allowed. TLSv1.1 is NOT supported.
If "https" is used but no SNI is configured, the API gateway provides a default server certificate issued for "https://localhost". You can replace the default certificate by a custom server-certificate by specyfing the secret name in the variable defaultTlsSecret.
Example values.yaml:
defaultTlsSecret: my-https-secretHere are some examples how to create a corresponding secret from PEM files. For more details s. Kubernetes documentation.
kubectl create secret tls my-https-secret --key=key.pem --cert=cert.pem
oc create secret generic my-https-secret-2 --from-file=tls.key=key.pem --from-file=tls.crt=cert.pem
Setup and some upgrades require specific migration steps to be run before and after changing the Kong version via a newer image or starting it for the first time. There the chart provides specialised jobs for each of those steps.
Bootstrapping is required when Kong starts for the first time and needs to setup its database. This task is handled by the job job-kong-bootstrap.yml.
It will be run if "migrations: bootstrap" is set in the values.yaml. This can be uncommented if no further execution is wished, but this is also prohibited by keeping the job itself.
Running the job again will do no harm in any way, as the executed bootstrap recognises the database as already initialised.
If you deploy a new instance of this Gateway, make sure migrations is set to bootstrap.
Upgrading to a newer version may require running migration steps (e.g. database changes). To run those jobs set "migrations: upgrade" in the values.yaml.
As a result job-kong-pre-upgrade-migrations.yml will run and job-kong-post-upgrade-migrations.yml will be run after successfull deployments to complete the upgrade.
Warning: Uncomment "migrations: upgrade" if you deploy again after a successfull deployment or set it to "migrations: bootstrap". Otherwise migrations will be executed again.
Note: Those jobs are only ment to be used for upgrading.
The following section contains special advice for dedicated updates and maybe necessary steps to be taken if updating from a certain version to another. Although updates in minor versions, whilst keeping the same major verison, do not contain breaking changes, implications may occour.
This version introduces Kong 2.8.1 and requires migrations to be run.
It also requires to adapt to the changed securityContext settings of the plugins in the ````values.yaml```.
Version 1.23.0 introduces a new issuer service version. If in use, this requires to set values for the new secret secret-issuer-service.yml.
Replace jsonWebKey: changeme and publicKey: changeme.
With introduction of Kong CE, a dedicated Admin-API handling has been introduced to proted the Admin-API. This required changes to the ingress of the Admin-API.
Those changes are only reflected in the ingress-admin.yml and not in the route-admin.yml. Using Kong CE will work, but deploying
the Admin-API-Route will provide unsecured access to the Admin-API.
The bundled Zipkin-plugin has been replaced by the ENI-Zipkin pluging. Behaviour and configuration differ slightly to the used one. To avoid complications, we strongly recommend removing the existing Zipkin-Plugin before upgrading. This can be done via a DELETE call on the Admin-API (Token required).
Lookup all plugins and find the Zipkin-Plugin-ID:
via GET on https://admin-api-url.me/plugins
Deleting the existing plugin:
via DELETE on https://admin-api-url.me/plugins/<zipkinPluginId>
We changed the integration of the ENI-plugins. Therefore names of the plugins changed and and eni-prefixed plugins have been removed from the image. Therefore the configuration of Kong itself, precisely the database, needs to be updated. You can do this by activating the jobs migration. This will delete the "old" ENI-plugins to allow the configuration of the new ones.
migrations: jobs
The migration from 2.x.x to 4.x.x is not possible. Please upgrade first from 2.x.x to 3.x.x as described above and afterwards without any migrations configuration from 3.x.x to 4.x.x
Starting from version 5 and above the htpasswd needs to be generated and set manually.
This is necessary as double encoded base64 secrets are not supported by Vault.
See chapter htpasswd.
You can create the htpasswd for admin user with Apache htpasswd tool.
Prerequisit: existing gatewayAdminApiKey for the deployment.
- Execute the following statement:
htpasswd -cb htpasswd admin gatewayAdminApiKey - Look up the htpasswd file you've just created and copy its content into the desired secret.
Make sure no spaces or line breaks are copied. - Optional but recommended: check if htpasswd is valid:
htpasswd -vb htpasswd admin gatewayAdminApiKey - Deploy and check if setup jobs can access the Kong admin-api and also if amin-api is accessible via the admin-api-route.
During the operation of the Gateway we discovered some issues that need some more advanced Kong or Kubernetes settings. The following paragraph explains which helm-chart settings are responsible, how to use them and what effects they have.
In some environments, especially in AWS "prod", we use the autoscaler to update workload ressources.
The autoscaling ia documented here.
(Since chart version 5.4.0 we use kubernetes API autoscaling/v2)
Following helm-chart variables controls the autoscaler properties for the Gateway:
| Helm-Chart variable | Kubernetes property (HorizontalPodAutoscaler) | default value | documentation link |
|---|---|---|---|
autoscaling.enabled |
false | ||
autoscaling.minReplicas |
spec.minReplicas |
3 | k8s_hpe_spec |
autoscaling.maxReplicas |
spec.maxReplicas |
10 | k8s_hpe_spec |
autoscaling.cpuUtilizationPercentage |
spec.metrics.resource.target.averageUtilization |
80 | k8s_hpe_spec |
In Kubernetes it is recommended to distribute the pods over several nodes. If a kubernetes node gets into problems, there are enough pods on other nodes to take on the load.
For this reason we provide the topologyKey flag in our helm-chart.
| Helm-Chart variable | Kubernetes property (Deployment,Pod) | default value | documentation link |
|---|---|---|---|
topologyKey |
spec.affinity.podAntiAffinity.topologyKey |
kubernetes.io/hostname AWS | topologyKey |
topologyKey |
spec.affinity.podAntiAffinity.topologyKey |
topology.kubernetes.io/zone CaaS | topologyKey |
Note: The default topologyKey for CaaS is different than for AWS and is specified in
platform/caas.yaml(s. next paragraph)
CaaS platform has certain requirements regarding securityContext in deployments.
Some fields like privileged: false must be set, even though they correspond to the default values.
This applies to both pods.securityContext and container[].securityContext and the absence of some values is difficult to detect, because CaaS refuses the deployment with a 503 http-code.
For this reason, there is one single flags global.platform: caas, which imports values from file platform/caas.yaml and thus applies all required to the deployment.
Individual values can be overwritten as usual.
The same approach can be used to extend the helm-chart for other platforms.
This Gateway is fully operational only when both components Kong and Jumper are operational. This is especially important when deploying as "Rolling Update" in customer environments.
For this reason, each container deployed in a gateway pod has its own settings for readinessProbe, livenessProbe and configurable initial delays.
The Probe-URLs are configured as follows:
http://localhost:8100/statusas readiness probe for Konghttp://localhost:8100/statusas liveness probe for Konghttp://localhost:808x/actuator/health/readinessas readiness probe for each Jumper pod ("jumper", "jegacyJumper")http://localhost:808x/actuator/health/livenessas liveness probe for each Jumper pod ("jumper", "jegacyJumper")
The details for readiness and liveness probes cen be tuned by following values:
| Helm-Chart variable | Container | Kubernetes setting | default value |
|---|---|---|---|
| startup.livenessProbe.initialDelay | kong | livenessProbe.initialDelaySeconds | 5 |
| startup.readinessProbe.initialDelay | kong | readinessProbe.initialDelaySeconds | 5 |
| jumper.startup.readinessProbe.initialDelay | jumper | livenessProbe.initialDelaySeconds | 25 |
| jumper.startup.readinessProbe.initialDelay | jumper | readinessProbe.initialDelaySeconds | 25 |
| jumper.startup.readinessProbe.initialDelay | legacyJumper | livenessProbe.initialDelaySeconds | 25 |
| jumper.startup.readinessProbe.initialDelay | legacyJumper | readinessProbe.initialDelaySeconds | 25 |
| not configurable yet | all cont. | ...Probe.timeoutSeconds | 5 |
| not configurable yet | all cont. | ...Probe.periodSeconds | 10 |
| not configurable yet | all cont. | ...Probe.successThreshold | 1 |
| not configurable yet | kong | ...Probe.failureThreshold | 6 |
| not configurable yet | all jumper | ...Probe.failureThreshold | 3 |
With the default setting, Kong has the following problem: while Rover is doing larger updates via the Admin-API (keyword "Reconciller"),unacceptable latencies arise in the Gateway runtime.
The problem is similar to the following already reported but still open issue #7543 in Github
The solution to the problem seems to be in asynchronous refresh or routes and tuning with the following Kong variables:
| Helm-Chart variable | Kong property | default value | documentation link |
|---|---|---|---|
| nginxWorkerProcesses | KONG_NGINX_WORKER_PROCESSES | 4 | |
| workerConsistency | KONG_WORKER_CONSISTENCY | eventual | worker_consistency |
| workerStateUpdateFrequency | KONG_DB_UPDATE_FREQUENCY | 10 | db_update_frequency |
| dbUpdatePropagation | KONG_DB_UPDATE_PROPAGATION | 0 | db_update_propagation |
| dbUpdateFrequency | KONG_WORKER_STATE_UPDATE_FREQUENCY | 10 | worker_state_update_frequency |
This is a short overlook about important parameters in the values.yaml.
| Parameter | Description | Default |
|---|---|---|
global |
Common values for all DHEI-Helm-Charts | |
global.platform |
Determines where the chart will be deployed | kubernetes |
global.storageClassName |
Overwrites the setting determined by the platform | gp2 |
global.domain |
URL for cluster external access set in Ingress/Route | nil |
global.labels |
Define global labels | tif.telekom.de/group |
global.ingress.annotations |
Set annotations for all ingress, can be extended by ingress specific ones | nil |
global.ingress.ingressClassName |
Set ingressClassName for all ingress in the chart. Can overwritten by specific ingress settings. Tdi default is triton-ingress. | nil |
global.image.repository |
Set default repository for all images | mtr.devops.telekom.de |
global.image.organisation |
Set default organisation for all images | tif-public |
global.image.force |
Replace repository/organisation also if image is set as custom "image:" value | false |
global.database.location |
Specifiy if you want to deploy a PostgreSQL or use an external database | local |
global.database.port |
Port of the database | 5432 |
global.database.database |
Name of the database | kong |
global.database.schema |
Name of the schema | public |
global.database.username |
Username for accessing the database | kong |
global.database.password |
The users password | changeme |
global.tracing |
Set generic tracing settings, will be used if not specified explicitly | Trace settings |
global.tracing.collectorUrl |
URL of the Zipkin-Collector (e.g. Jaeger-Collector), http(s) mandatory | http://guardians-drax-collector.skoll:9411/api/v2/spans |
global.tracing.defaultServiceName |
Name of the service shown in e.g. Jaeger | stargate |
global.tracing.sampleRatio |
How often to sample requests that do not contain trace ids. Set to 0 to turn sampling off, or to 1 to sample all requests. | 1 |
migrations |
Determine the migrations behaviuor for a new instance or upgrade | none |
adminApi.enabled |
Create service for accessing Kong Admin API | true |
adminApi.tls.enabled |
Access Admin API via https instead of http | false |
adminApi.ingress.enabled |
Create ingress for Admin API. Default depends on Edition | CE: falseEE: true |
adminApi.ingress.hostname |
Set dedicated hostname for Admin API ingress (or route), overwrites global URL | nil |
adminApi.ingress.annotations |
Merges specific into global ingress annotations | nil |
adminApi.ingress.ingressClassName |
Set ingressClassName for the Admin API ingress | nil |
adminApi.access_log |
Set the log target for access log | /dev/stdout |
adminApi.ingress.annotations |
Set the log target for error log | /dev/stderr |
proxy.ingress.enabled |
Create ingress for proxy | true |
proxy.ingress.hostname |
Set dedicated hostname for proxy ingress (or route), overwrites global URL | nil |
proxy.ingress.ingressClassName |
Set ingressClassName for the proxy ingress | nil |
proxy.ingress.annotations |
Merges specific into global ingress annotations | ssl-passthrough |
proxy.access_log |
Set the log target for access log | /dev/stdout |
proxy.ingress.annotations |
Set the log target for error log | /dev/stderr |
configuration |
Set a script to run after deployment for configuration of the Gateway | default admin-api conf |
templateChangeTriggers |
List of (template) yaml files fo which a checksum annotation will be created | [] |
sslVerify |
Controls whether to check forward proxy traffic against CA certificates | true |
sslVerifyDepth |
SSL Verification depth | 1 |
setupJobs.backoffLimit |
How often should be retried to run the job successfully | 20 |
setupJobs.activeDeadlineSeconds |
How long should be retried to run the job successfully | 300 |
zipkin.enabled |
Enable tracing via ENI-Zipkin-Plugin | false |
zipkin.collectorUrl |
URL of the Zipkin-Collector (e.g. Jaeger-Collector), http(s) mandatory | https://collector-zipkin-http-drax-guardians.live.dhei.telekom.de:443/api/v2/spans |
zipkin.sampleRatio |
How often to sample requests that do not contain trace ids. Set to 0 to turn sampling off, or to 1 to sample all requests. | 1 |
zipkin.includeCredential |
Should the credential of the currently authenticated consumer be included in metadata sent to the Zipkin server? | true |
zipkin.defaultServiceName |
Name of the service shown in e.g. Jaeger | stargate |
zipkin.luaSslTrustedCertificate |
CA certificate for the Zipkin-Collector-URL | nil |
trustedCaCertificates |
CA certificates in PEM format (string) | nil |
defaultTlsSecret |
Name of the secret containing the default server certificates | nil |
plugins. |
This section contains all Kong plugins settings | See the following plugins |
prometheus.enabled |
Controls whether to annotate pods with prometheus scraping information or not | true |
prometheus.port |
Sets the port at which metrics can be accessed | 9542 |
prometheus.path |
Sets the endpoint at which at which metrics can be accessed | /metrics |
prometheus.podMonitor.enabled |
Enables a podmonitor which can be used by the prometheus operator to collect metrics | false |
prometheus.podMonitor.scheme |
HTTP scheme to use for scraping | http |
prometheus.podMonitor.interval |
Interval at which metrics should be scraped | 15s |
prometheus.podMonitor.scrapeTimeout |
Timeout after which the scrape of prometheus is ended | 3s |
prometheus.podMonitor.honorLabels |
HonorLabels chooses the metric’s labels on collisions with target labels | true |
prometheus.serviceMonitor.enabled |
Enables a servicemonitor which can be used by the prometheus operator to collect metrics | true |
prometheus.serviceMonitor.scheme |
HTTP scheme to use for scraping | http |
prometheus.serviceMonitor.interval |
Interval at which metrics should be scraped | 15s |
prometheus.serviceMonitor.scrapeTimeout |
Timeout after which the scrape of prometheus is ended | 3s |
prometheus.serviceMonitor.honorLabels |
HonorLabels chooses the metric’s labels on collisions with target labels | true |
jwtKeycloak.enabled |
Activate or deactivate the jwt-keycloak plugin | true |
jwtKeycloak.setupJob |
Set required values for the provieded configuration. Can be ignored for costum config | |
jwtKeycloak.setupJob.pluginId |
If you want to alter the already configured plugin, set the pluginId | 24f1d5a5-4d31-4abc-b539-bed6d3cd7f0a |
jwtKeycloak.setupJob.allowedIss |
Set the Iris URL you want the Gateway to use for Admin API athentication | https://changeme/auth/realms/default |
jumper |
Configure the Jumper (by Hyperion) | 1.5.5 |
issuerService |
Confgiure the Issuer-Service (by Hyperion) | 1.0.0 |
postgresql.image |
Specifiy the PostgreSQL image | postgres-12.3-debian |
postgresql.securityContext |
Specifiy the security context | nil |
postgresql.resources |
Assign ressources, e.g. limits, for Postgres | Memory limits |
externalDatabase.host |
If an external database is used, this is the url of the database instance | nil |
externalDatabase.ssl |
Use ssl for connection | false |
externalDatabase.sslVerify |
Use the provided certificate | false |
externalDatabase.luaSslTrustedCertificate |
Provide certificate | nil |
replicas |
Set the number of Gateway replicas | 1 |
autoscaling.enabled |
Enables Pod Autoscaling with Target CPU usage | false |
autoscaling.minReplicas |
Minimum number of replicas if autoscaling is enabled | $replicas |
autoscaling.maxReplicas |
Maximum number of replicas if autoscaling is enabled | 10 |
autoscaling.cpuUtilizationPercentage |
Number of target CPU Utilization | 80 |
logFormat |
Selects the mginx log format default, json or plain |
default |
If the Gateway deployment fails to come up, please have a look at the logs of the container.
Log message:
Error: /usr/local/share/lua/5.1/opt/kong/cmd/start.lua:37: nginx configuration is invalid (exit code 1):
nginx: [emerg] SSL_CTX_load_verify_locations("/usr/local/opt/kong/tif/trusted-ca-certificates.pem") failed (SSL: error:0B084088:x509 certificate routines:X509_load_cert_crl_file:no certificate or crl found)
nginx: configuration file /opt/kong/nginx.conf test failed
Solution:
This error happens if sslVerify is set to true but no valid certificates could be found.
Please make sue that trustedCaCertificates is set probably or set sslVerify to false if you don't wish to use ssl verification.
| Environment | Compatible |
|---|---|
| OTC | Yes |
| AppAgile | Unverified |
| AWS EKS | Yes |
| CaaS | Yes |
This Helm Chart is also compatible with Sapling, DHEI's universal solution for deploying Helm Charts to multiple Telekom cloud platforms.
