diff --git a/README.md b/README.md new file mode 100644 index 0000000..5d822a2 --- /dev/null +++ b/README.md @@ -0,0 +1,190 @@ +# Istio with GD.CN + +GoodData CN is almost ready for running on clusters with Istio Gateway (in sidecar mode), so you don't need to +use Nginx Ingress controller. + +Recent helm charts to not require any modifications, but a few things need to be taken into account when using Istio: + +* port names: gooddata-cn services follow Istio recommendation on naming ports. However, etcd and pulsar charts do not. + Fortunately it is possible to override default port names. Refer to [gdcn-values.yaml](gdcn-values.yaml) and + [pulsar-values.yaml](pulsar-values.yaml) and see how to add `tcp-` prefix to ports and make Istio protocol discovery happy. +* gooddata-cn defines `nginx` as default `ingressClassName`. I assume this ingress class is NOT available on your istio-enabled + cluster. This is expected until we update our apps to support Istio's custom resources natively. So when you create a new Organization, + a new Ingress will be created but will not be used (because of unregistered ingress class name). You will need to create VirtualService + for your organization manually. + + +## Requirements for local test + +* Running Docker daemon +* curl +* [KinD binary](https://kind.sigs.k8s.io/docs/user/quick-start/) +* [cloud-provider-kind](https://kind.sigs.k8s.io/docs/user/loadbalancer/) +* Valid GoodData CN License key, stored in `GDCN_LICENSE` env variable +* [istioctl binary](https://github.com/istio/istio/releases) +* [kubectl](https://kubernetes.io/docs/tasks/tools/) +* Optionally [helm binary](https://helm.sh), if you want Kiali UI for Istio + +## Setup + +1. Create KIND cluster + +```bash +kind create cluster --name kind +# get it from https://github.com/kubernetes-sigs/cloud-provider-kind/releases +cloud-provider-kind -v 0 & +``` + +1. Install Istio and related stuff +I tested with "native sidecar" mode, so it requires k8s 1.29+. It resolves strange issues with jobs not being terminated. + +```bash +istioctl install --set values.pilot.env.ENABLE_NATIVE_SIDECARS=true -y --set meshConfig.accessLogFile=/dev/stdout +# These are optional but recommended for better visiblity +kubectl apply -f https://raw.githubusercontent.com/istio/istio/1.24.2/samples/addons/prometheus.yaml +kubectl apply -f https://raw.githubusercontent.com/istio/istio/1.24.2/samples/addons/grafana.yaml +helm upgrade --install -n istio-system kiali-server --repo https://kiali.org/helm-charts kiali-server --set auth.strategy=anonymous +``` + +1. Install Apache Pulsar and GoodData CN + +```bash +kubectl apply -f namespaces.yaml + +kind load docker-image apachepulsar/pulsar:3.3.3 +helm -n pulsar upgrade --install \ + --repo https://pulsar.apache.org/charts pulsar pulsar --version 3.5.0 \ + --values pulsar-values.yaml + +# GD CN License key +kubectl -n gooddata create secret generic gdcn-license --from-literal=license=$GDCN_LICENSE + +# provision certificate and key for *.example.com, keep in files _.example.com.crt and _.example.com.key +# I'm using my own local CA, so cacert should be passed to k8s secret as well. + +# secrets must be stored in istio-system so SDS can find them (?) +kubectl -n istio-system create secret generic star.example.com \ + --from-file=cert=_.example.com.crt \ + --from-file=key=_.example.com.key \ + --from-file=cacert=ca.crt + + +# Install official gooddata chart +helm -n gooddata upgrade --install \ + --repo https://charts.gooddata.com/ \ + gooddata-cn gooddata-cn --version 3.25.0 \ + --values gdcn-values.yaml + +# OR from local chart files +helm -n gooddata upgrade --install \ + gooddata-cn ./folder-with-extracted-gooddata-cn-chart \ + --values gdcn-values.yaml --set image.defaultTag=3.25.0 + +``` + +1. Create Istio Ingress GW in "gooddata" namespace. + +```bash +kubectl apply -f gateway.yaml +``` + +1. Create VirtualService for Dex + +```bash +kubectl apply -f istio-virtual-service-dex.yaml +``` + +1. Create delegate VS (shared VS without hosts or attached gateway) + +```bash +kubectl apply -f istio-virtual-service.yaml +``` + +1. Update `/etc/hosts` with example hostnames. +This guide uses hostnames in example domain (RFC-6761), so we update local dns resolver. + +```bash +LB_IP=$(kubectl get svc -n istio-system -l istio=ingressgateway -o jsonpath='{.items[0]..status.loadBalancer.ingress[0].ip}') +echo "$LB_IP auth.example.com org1.example.com org2.example.com" | sudo tee -a /etc/hosts +``` + +1. Create org1 and org2 Organziations + +```bash +kubectl apply -f organizations.yaml +``` + +1. Create VS for these two organizations +Note these VirtualServices are very simple, they just hold the hostname and gateway reference. Routes are stored in delegated VS created earilier. +It allows us to keep configuration clean and DRY. + +```bash +kubectl apply -f orgs-vs.yaml +``` + +1. Create user + +* In dex, create one user (dex is shared by both orgs, so any orgnanization hostname will work): + + ```bash + curl -X POST -H 'Content-type: application/json' \ + -d '{"email": "john.doe@example.com","password": "mypassword","displayName": "John Doe"}' \ + -H 'Authorization: Bearer YWRtaW46Ym9vdHN0cmFwOkdkY05hczEyMw' -k https://org1.example.com/api/v1/auth/users + ``` + +Note the authentication Id returned by the API. Use it in the following 2 requests: + +* Map dex user to `admin` user in both organizations: + + ```bash + curl -X PATCH -k https://org1.example.com/api/v1/entities/users/admin -H "Authorization: Bearer YWRtaW46Ym9vdHN0cmFwOkdkY05hczEyMw" \ + -H "Content-Type: application/vnd.gooddata.api+json" -d '{ + "data": { + "id": "admin", + "type": "user", + "attributes": { + "authenticationId": "<>", + "email": "john.doe@example.com", + "firstname": "John", + "lastname": "Doe" + } + } + }' + curl -X PATCH -k https://org2.example.com/api/v1/entities/users/admin -H "Authorization: Bearer YWRtaW46Ym9vdHN0cmFwOkdkY05hczEyMw" \ + -H "Content-Type: application/vnd.gooddata.api+json" -d '{ + "data": { + "id": "admin", + "type": "user", + "attributes": { + "authenticationId": "<>", + "email": "john.doe@example.com", + "firstname": "John", + "lastname": "Doe" + } + } + }' + + ``` + +1. Login to UI with username `john.doe@example.com` and password `mypassword` to https://org1.example.com/ or https://org2.example.com/. + +## Issues to solve + +* ~~job pods keep running, the main container is "Completed" but "istio-proxy" container remains running.~~ +RESOLVED by using native sidecars. + +* Readiness probe of calcique and afm-exec-api fail for a long time, becuase they are unable to resolve headless service to metadata-api or calcique, respectively. + They will recover enventually, but it takes a few minutes. + +* mTLS setup +* ~~TLS setup on Gateway~~ (DONE) +* cert-manager integration +* How to handle organization hostnames that do not match `*.exmaple.com` wildcard? + +## Useful links + +* [Istio docs](https://istio.io/latest/docs/concepts/traffic-management/) +* [Envoy response flags](https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#config-access-log-format-response-flags) +* [GoodData Docs](https://www.gooddata.com/docs/cloud-native/3.17/install/install-locally/prepare/) +* [Istio by Example (cool!)](https://istiobyexample.dev/) +* [Istio config validator](https://github.com/getyourguide/istio-config-validator) diff --git a/gateway.yaml b/gateway.yaml new file mode 100644 index 0000000..93865d5 --- /dev/null +++ b/gateway.yaml @@ -0,0 +1,26 @@ +apiVersion: networking.istio.io/v1 +kind: Gateway +metadata: + name: gooddata-cn-gw + namespace: gooddata +spec: + selector: + istio: ingressgateway + servers: + - hosts: + - "*.example.com" + port: + name: http + number: 80 + protocol: HTTP + tls: + httpsRedirect: true + - hosts: + - "*.example.com" + port: + name: https + number: 443 + protocol: HTTPS + tls: + mode: SIMPLE + credentialName: star.example.com diff --git a/gdcn-values.yaml b/gdcn-values.yaml new file mode 100644 index 0000000..6663c2c --- /dev/null +++ b/gdcn-values.yaml @@ -0,0 +1,36 @@ +deployVisualExporter: false +license: + existingSecret: gdcn-license + +postgresql-ha: + pgpool: + replicaCount: 1 + postgresql: + replicaCount: 1 + service: + portName: tcp-postgresql + +etcd: + service: + clientPortNameOverride: tcp-client + peerPortNameOverride: tcp-peer + metricsPortNameOverride: http-metrics + +redis-ha: + hardAntiAffinity: false + +image: + pullPolicy: IfNotPresent + +# ingress: +# lbProtocol: https + +replicaCount: 1 + +dex: + ingress: + authHost: auth.example.com + +metadataApi: + encryptor: + enabled: false diff --git a/istio-virtual-service-dex.yaml b/istio-virtual-service-dex.yaml new file mode 100644 index 0000000..2027728 --- /dev/null +++ b/istio-virtual-service-dex.yaml @@ -0,0 +1,19 @@ +apiVersion: networking.istio.io/v1 +kind: VirtualService +metadata: + name: gooddata-cn-dex + namespace: gooddata +spec: + hosts: + - auth.example.com + gateways: + - gooddata-cn-gw + http: + - match: + - uri: + prefix: /dex + route: + - destination: + host: gooddata-cn-dex.gooddata.svc.cluster.local + port: + number: 32000 diff --git a/istio-virtual-service.yaml b/istio-virtual-service.yaml new file mode 100644 index 0000000..e4849e9 --- /dev/null +++ b/istio-virtual-service.yaml @@ -0,0 +1,142 @@ +apiVersion: networking.istio.io/v1beta1 +kind: VirtualService +metadata: + name: gooddata-cn + namespace: gooddata +spec: + http: + - match: + # Matches for gooddata-cn-afm-exec-api service + - uri: + regex: /api/v\d+/actions/workspaces/[^/]+/execution/.* + - uri: + regex: /api/v\d+/actions/workspaces/[^/]+/ai/.* + - uri: + regex: /api/v\d+/schemas/afm + route: + - destination: + host: gooddata-cn-afm-exec-api.gooddata.svc.cluster.local + port: + number: 9000 + + - match: + # Matches for gooddata-cn-auth-service service + - uri: + regex: /api/v\d+/auth/.* + - uri: + regex: /api/v\d+/schemas/auth + - uri: + prefix: /login + - uri: + prefix: /oauth2 + - uri: + prefix: /logout + - uri: + prefix: /appLogin + - uri: + regex: /api/v\d+/actions/invite + - uri: + regex: /api/v\d+/profile + route: + - destination: + host: gooddata-cn-auth-service.gooddata.svc.cluster.local + port: + number: 9050 + + - match: + # Matches for gooddata-cn-automation service + - uri: + regex: /api/v\d+/actions/notificationChannels(/.*)? + - uri: + regex: /api/v\d+/actions/notifications(/.*)? + - uri: + regex: /api/v\d+/schemas/automation + route: + - destination: + host: gooddata-cn-automation.gooddata.svc.cluster.local + port: + number: 9097 + + - match: +# # Matches for gooddata-cn-scan-model service + - uri: + regex: /api/v\d+/actions/dataSources/[^/]+/scan.* + - uri: + regex: /api/v\d+/actions/dataSources/[^/]+/test + - uri: + regex: /api/v\d+/actions/dataSource/test + - uri: + regex: /api/v\d+/actions/dataSources/[^/]+/computeColumnStatistics + - uri: + regex: /api/v\d+/schemas/scan + route: + - destination: + host: gooddata-cn-scan-model.gooddata.svc.cluster.local + port: + number: 9060 + + - match: + # Matches for gooddata-cn-apidocs service + - uri: + prefix: /apidocs + route: + - destination: + host: gooddata-cn-apidocs.gooddata.svc.cluster.local + port: + number: 9999 + + - match: + # Matches for gooddata-cn-export-controller service + - uri: + regex: /api/v\d+/actions/workspaces/[^/]+/export/.* + - uri: + regex: /api/v\d+/schemas/export + route: + - destination: + host: gooddata-cn-export-controller.gooddata.svc.cluster.local + port: + number: 6580 + + - match: + # Matches for gooddata-cn-result-cache service + - uri: + regex: /api/v\d+/actions/collectCacheUsage + - uri: + regex: /api/v\d+/schemas/result + - uri: + regex: /api/v\d+/actions/fileStorage/.* + route: + - destination: + host: gooddata-cn-result-cache.gooddata.svc.cluster.local + port: + number: 9040 + + - match: + # Matches for gooddata-cn-metadata-api service + - uri: + prefix: /api + route: + - destination: + host: gooddata-cn-metadata-api.gooddata.svc.cluster.local + port: + number: 9007 + + - match: + # Matches for gooddata-cn-api-gateway service + - uri: + prefix: /analyze + - uri: + prefix: /components + - uri: + prefix: /dashboards + - uri: + prefix: /modeler + - uri: + prefix: /metrics + - uri: + prefix: / + route: + - destination: + host: gooddata-cn-api-gateway.gooddata.svc.cluster.local + port: + number: 9092 diff --git a/namespaces.yaml b/namespaces.yaml new file mode 100644 index 0000000..5919782 --- /dev/null +++ b/namespaces.yaml @@ -0,0 +1,18 @@ +--- +apiVersion: v1 +kind: Namespace +metadata: + labels: + istio-injection: enabled + kubernetes.io/metadata.name: pulsar + name: pulsar + name: pulsar +--- +apiVersion: v1 +kind: Namespace +metadata: + labels: + istio-injection: enabled + kubernetes.io/metadata.name: gooddata + name: gooddata + name: gooddata diff --git a/organizations.yaml b/organizations.yaml new file mode 100644 index 0000000..70b32ac --- /dev/null +++ b/organizations.yaml @@ -0,0 +1,39 @@ +--- +apiVersion: controllers.gooddata.com/v1 +kind: Organization +metadata: + name: org1 + namespace: gooddata +spec: + adminGroup: adminGroup + adminUser: admin + # Token is YWRtaW46Ym9vdHN0cmFwOkdkY05hczEyMw== + adminUserToken: $5$1234567890123456$pA9PBFxCwLbVOB.fImbUCUjoyQblla4EdSkHhVmy9A7 + hostname: org1.example.com + id: org1 + name: Organization One + entitlements: + - expiry: "2026-04-26T01:23:45Z" + name: Contract + - name: Unlimited_Workspaces + - name: Unlimited_Users + +--- +apiVersion: controllers.gooddata.com/v1 +kind: Organization +metadata: + name: org2 + namespace: gooddata +spec: + adminGroup: adminGroup + adminUser: admin + # Token is YWRtaW46Ym9vdHN0cmFwOkdkY05hczEyMw== + adminUserToken: $5$1234567890123456$pA9PBFxCwLbVOB.fImbUCUjoyQblla4EdSkHhVmy9A7 + hostname: org2.example.com + id: org2 + name: Organization Two + entitlements: + - expiry: "2026-04-26T01:23:45Z" + name: Contract + - name: Unlimited_Workspaces + - name: Unlimited_Users diff --git a/orgs-vs.yaml b/orgs-vs.yaml new file mode 100644 index 0000000..1de3c1b --- /dev/null +++ b/orgs-vs.yaml @@ -0,0 +1,28 @@ +--- +apiVersion: networking.istio.io/v1 +kind: VirtualService +metadata: + name: org1 + namespace: gooddata +spec: + hosts: + - "org1.example.com" + gateways: + - gooddata-cn-gw + http: + - delegate: + name: gooddata-cn +--- +apiVersion: networking.istio.io/v1 +kind: VirtualService +metadata: + name: org2 + namespace: gooddata +spec: + hosts: + - "org2.example.com" + gateways: + - gooddata-cn-gw + http: + - delegate: + name: gooddata-cn diff --git a/pulsar-values.yaml b/pulsar-values.yaml new file mode 100644 index 0000000..a8eb75d --- /dev/null +++ b/pulsar-values.yaml @@ -0,0 +1,28 @@ +# Totally minimal configuration, no HA +defaultPulsarImageTag: 3.3.3 +defaultPulsarImageRepository: apachepulsar/pulsar +components: + proxy: false + toolset: false + +zookeeper: + replicaCount: 1 + podMonitor: + enabled: false +bookkeeper: + replicaCount: 1 + podMonitor: + enabled: false +broker: + replicaCount: 1 + podMonitor: + enabled: false +autorecovery: + podMonitor: + enabled: false +kube-prometheus-stack: + enabled: false +proxy: + podMonitor: + enabled: false +tcpPrefix: tcp-