-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
0 parents
commit ae54c08
Showing
9 changed files
with
526 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,190 @@ | ||
# Istio with GD.CN | ||
|
||
GoodData CN is almost ready for running on clusters with Istio Gateway (in sidecar mode), so you don't need to | ||
use Nginx Ingress controller. | ||
|
||
Recent helm charts to not require any modifications, but a few things need to be taken into account when using Istio: | ||
|
||
* port names: gooddata-cn services follow Istio recommendation on naming ports. However, etcd and pulsar charts do not. | ||
Fortunately it is possible to override default port names. Refer to [gdcn-values.yaml](gdcn-values.yaml) and | ||
[pulsar-values.yaml](pulsar-values.yaml) and see how to add `tcp-` prefix to ports and make Istio protocol discovery happy. | ||
* gooddata-cn defines `nginx` as default `ingressClassName`. I assume this ingress class is NOT available on your istio-enabled | ||
cluster. This is expected until we update our apps to support Istio's custom resources natively. So when you create a new Organization, | ||
a new Ingress will be created but will not be used (because of unregistered ingress class name). You will need to create VirtualService | ||
for your organization manually. | ||
|
||
|
||
## Requirements for local test | ||
|
||
* Running Docker daemon | ||
* curl | ||
* [KinD binary](https://kind.sigs.k8s.io/docs/user/quick-start/) | ||
* [cloud-provider-kind](https://kind.sigs.k8s.io/docs/user/loadbalancer/) | ||
* Valid GoodData CN License key, stored in `GDCN_LICENSE` env variable | ||
* [istioctl binary](https://github.com/istio/istio/releases) | ||
* [kubectl](https://kubernetes.io/docs/tasks/tools/) | ||
* Optionally [helm binary](https://helm.sh), if you want Kiali UI for Istio | ||
|
||
## Setup | ||
|
||
1. Create KIND cluster | ||
|
||
```bash | ||
kind create cluster --name kind | ||
# get it from https://github.com/kubernetes-sigs/cloud-provider-kind/releases | ||
cloud-provider-kind -v 0 & | ||
``` | ||
|
||
1. Install Istio and related stuff | ||
I tested with "native sidecar" mode, so it requires k8s 1.29+. It resolves strange issues with jobs not being terminated. | ||
|
||
```bash | ||
istioctl install --set values.pilot.env.ENABLE_NATIVE_SIDECARS=true -y --set meshConfig.accessLogFile=/dev/stdout | ||
# These are optional but recommended for better visiblity | ||
kubectl apply -f https://raw.githubusercontent.com/istio/istio/1.24.2/samples/addons/prometheus.yaml | ||
kubectl apply -f https://raw.githubusercontent.com/istio/istio/1.24.2/samples/addons/grafana.yaml | ||
helm upgrade --install -n istio-system kiali-server --repo https://kiali.org/helm-charts kiali-server --set auth.strategy=anonymous | ||
``` | ||
|
||
1. Install Apache Pulsar and GoodData CN | ||
|
||
```bash | ||
kubectl apply -f namespaces.yaml | ||
|
||
kind load docker-image apachepulsar/pulsar:3.3.3 | ||
helm -n pulsar upgrade --install \ | ||
--repo https://pulsar.apache.org/charts pulsar pulsar --version 3.5.0 \ | ||
--values pulsar-values.yaml | ||
|
||
# GD CN License key | ||
kubectl -n gooddata create secret generic gdcn-license --from-literal=license=$GDCN_LICENSE | ||
|
||
# provision certificate and key for *.example.com, keep in files _.example.com.crt and _.example.com.key | ||
# I'm using my own local CA, so cacert should be passed to k8s secret as well. | ||
|
||
# secrets must be stored in istio-system so SDS can find them (?) | ||
kubectl -n istio-system create secret generic star.example.com \ | ||
--from-file=cert=_.example.com.crt \ | ||
--from-file=key=_.example.com.key \ | ||
--from-file=cacert=ca.crt | ||
|
||
|
||
# Install official gooddata chart | ||
helm -n gooddata upgrade --install \ | ||
--repo https://charts.gooddata.com/ \ | ||
gooddata-cn gooddata-cn --version 3.25.0 \ | ||
--values gdcn-values.yaml | ||
|
||
# OR from local chart files | ||
helm -n gooddata upgrade --install \ | ||
gooddata-cn ./folder-with-extracted-gooddata-cn-chart \ | ||
--values gdcn-values.yaml --set image.defaultTag=3.25.0 | ||
|
||
``` | ||
|
||
1. Create Istio Ingress GW in "gooddata" namespace. | ||
|
||
```bash | ||
kubectl apply -f gateway.yaml | ||
``` | ||
|
||
1. Create VirtualService for Dex | ||
|
||
```bash | ||
kubectl apply -f istio-virtual-service-dex.yaml | ||
``` | ||
|
||
1. Create delegate VS (shared VS without hosts or attached gateway) | ||
|
||
```bash | ||
kubectl apply -f istio-virtual-service.yaml | ||
``` | ||
|
||
1. Update `/etc/hosts` with example hostnames. | ||
This guide uses hostnames in example domain (RFC-6761), so we update local dns resolver. | ||
|
||
```bash | ||
LB_IP=$(kubectl get svc -n istio-system -l istio=ingressgateway -o jsonpath='{.items[0]..status.loadBalancer.ingress[0].ip}') | ||
echo "$LB_IP auth.example.com org1.example.com org2.example.com" | sudo tee -a /etc/hosts | ||
``` | ||
|
||
1. Create org1 and org2 Organziations | ||
|
||
```bash | ||
kubectl apply -f organizations.yaml | ||
``` | ||
|
||
1. Create VS for these two organizations | ||
Note these VirtualServices are very simple, they just hold the hostname and gateway reference. Routes are stored in delegated VS created earilier. | ||
It allows us to keep configuration clean and DRY. | ||
|
||
```bash | ||
kubectl apply -f orgs-vs.yaml | ||
``` | ||
|
||
1. Create user | ||
|
||
* In dex, create one user (dex is shared by both orgs, so any orgnanization hostname will work): | ||
|
||
```bash | ||
curl -X POST -H 'Content-type: application/json' \ | ||
-d '{"email": "[email protected]","password": "mypassword","displayName": "John Doe"}' \ | ||
-H 'Authorization: Bearer YWRtaW46Ym9vdHN0cmFwOkdkY05hczEyMw' -k https://org1.example.com/api/v1/auth/users | ||
``` | ||
|
||
Note the authentication Id returned by the API. Use it in the following 2 requests: | ||
|
||
* Map dex user to `admin` user in both organizations: | ||
|
||
```bash | ||
curl -X PATCH -k https://org1.example.com/api/v1/entities/users/admin -H "Authorization: Bearer YWRtaW46Ym9vdHN0cmFwOkdkY05hczEyMw" \ | ||
-H "Content-Type: application/vnd.gooddata.api+json" -d '{ | ||
"data": { | ||
"id": "admin", | ||
"type": "user", | ||
"attributes": { | ||
"authenticationId": "<<authenticationId-returned-above>>", | ||
"email": "[email protected]", | ||
"firstname": "John", | ||
"lastname": "Doe" | ||
} | ||
} | ||
}' | ||
curl -X PATCH -k https://org2.example.com/api/v1/entities/users/admin -H "Authorization: Bearer YWRtaW46Ym9vdHN0cmFwOkdkY05hczEyMw" \ | ||
-H "Content-Type: application/vnd.gooddata.api+json" -d '{ | ||
"data": { | ||
"id": "admin", | ||
"type": "user", | ||
"attributes": { | ||
"authenticationId": "<<authenticationId-returned-above>>", | ||
"email": "[email protected]", | ||
"firstname": "John", | ||
"lastname": "Doe" | ||
} | ||
} | ||
}' | ||
|
||
``` | ||
|
||
1. Login to UI with username `[email protected]` and password `mypassword` to https://org1.example.com/ or https://org2.example.com/. | ||
|
||
## Issues to solve | ||
|
||
* ~~job pods keep running, the main container is "Completed" but "istio-proxy" container remains running.~~ | ||
RESOLVED by using native sidecars. | ||
|
||
* Readiness probe of calcique and afm-exec-api fail for a long time, becuase they are unable to resolve headless service to metadata-api or calcique, respectively. | ||
They will recover enventually, but it takes a few minutes. | ||
|
||
* mTLS setup | ||
* ~~TLS setup on Gateway~~ (DONE) | ||
* cert-manager integration | ||
* How to handle organization hostnames that do not match `*.exmaple.com` wildcard? | ||
|
||
## Useful links | ||
|
||
* [Istio docs](https://istio.io/latest/docs/concepts/traffic-management/) | ||
* [Envoy response flags](https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#config-access-log-format-response-flags) | ||
* [GoodData Docs](https://www.gooddata.com/docs/cloud-native/3.17/install/install-locally/prepare/) | ||
* [Istio by Example (cool!)](https://istiobyexample.dev/) | ||
* [Istio config validator](https://github.com/getyourguide/istio-config-validator) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
apiVersion: networking.istio.io/v1 | ||
kind: Gateway | ||
metadata: | ||
name: gooddata-cn-gw | ||
namespace: gooddata | ||
spec: | ||
selector: | ||
istio: ingressgateway | ||
servers: | ||
- hosts: | ||
- "*.example.com" | ||
port: | ||
name: http | ||
number: 80 | ||
protocol: HTTP | ||
tls: | ||
httpsRedirect: true | ||
- hosts: | ||
- "*.example.com" | ||
port: | ||
name: https | ||
number: 443 | ||
protocol: HTTPS | ||
tls: | ||
mode: SIMPLE | ||
credentialName: star.example.com |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
deployVisualExporter: false | ||
license: | ||
existingSecret: gdcn-license | ||
|
||
postgresql-ha: | ||
pgpool: | ||
replicaCount: 1 | ||
postgresql: | ||
replicaCount: 1 | ||
service: | ||
portName: tcp-postgresql | ||
|
||
etcd: | ||
service: | ||
clientPortNameOverride: tcp-client | ||
peerPortNameOverride: tcp-peer | ||
metricsPortNameOverride: http-metrics | ||
|
||
redis-ha: | ||
hardAntiAffinity: false | ||
|
||
image: | ||
pullPolicy: IfNotPresent | ||
|
||
# ingress: | ||
# lbProtocol: https | ||
|
||
replicaCount: 1 | ||
|
||
dex: | ||
ingress: | ||
authHost: auth.example.com | ||
|
||
metadataApi: | ||
encryptor: | ||
enabled: false |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
apiVersion: networking.istio.io/v1 | ||
kind: VirtualService | ||
metadata: | ||
name: gooddata-cn-dex | ||
namespace: gooddata | ||
spec: | ||
hosts: | ||
- auth.example.com | ||
gateways: | ||
- gooddata-cn-gw | ||
http: | ||
- match: | ||
- uri: | ||
prefix: /dex | ||
route: | ||
- destination: | ||
host: gooddata-cn-dex.gooddata.svc.cluster.local | ||
port: | ||
number: 32000 |
Oops, something went wrong.