Journey into Theiacloud on Darwin

[paulreuter]

Here are my experiences with testing the Theiacloud demo on my local dev machine, which is a M1 mac, with some open questions left.

Out-of-the-box, the architecture behaves a little bit different (and more troublesome) than some of the guides expect, so for future mac users, here is what I found:

VirtualBox is unavailable for this architecuture, so I used Docker (for Desktop).

I modified the minikube_getting_started.tf to use driver="docker" in the "cluster" module parameters, then preloaded minikube with 'minikube start --driver=docker --download-only'

It took some time to figure out why neither terraform nor helm were properly setting up the environment - on MacOs the Ingresses are not automatically reachable from the host machine. This caused the keycloak setup to fail.

What finally worked was opening up tunnels in a different terminal with 'sudo minikube tunnel'. This should be done after the initial cluster setup.

With this, the environment finally spun up. I could not access the landing page until I added 127.0.0.1.nip.io to my /etc/hosts file. With that, the landing page is reachable, but receives an internal server error when trying to launch Theia from the service container.

I brought up a debugging instance of the theiacloud-service and redirected the landing page. This showed me that the queries to the CRDs came up empty, perhaps because no namespace was used. I overrode the default namespace to use "theiacloud", and now the workspace CR is properly created.

I am not sure if this is an issue due to launching the service in the dev environment - perhaps the deployed container is being properly configured by terraform. The server error may also have been caused by the next issue:

The session CR was not created, due to a mismatch of APIs. The service was calling with v4beta, but the CRD did not have a service definition for that (I suspect the helm chart is not up-to-date). Updating the CRD definition gave me a proper session custom resource, and instance pods are created.

Unfortunately, the oauth2-proxy do not configure properly.
It seems that ports on the host are not reachable from within the pods. This is probably the root cause for all the remaining woes.

Overriding the OIDC issuer url to "http://keycloak.keycloak.svc.cluster.local/keycloak/realms/TheiaCloud" helped the oauth2-proxy to go online, but that caused later issues with redirecting, as that url is not resolvable to the browser from the outside.

Similarly, the operator was not able to detect the running instance pod to update the url, as it could not reach the instance due to the access via the host url, and after manually setting the url in the session CR, the callback to the oauth2proxy was similarly blocked.

Clearly, I am still missing a piece of the puzzle. If anyone has experience in this area, I could use some advice.

[jfaltermeier]

Hi,

sorry to hear about the troubles on Mac. Our regular developers are on Linux.
We started to use terraform so that users may more easily contribute fixes for their OS, if something does not work out of the box. So if we find a setup that works for you, we would happily accept a contribution.

Regarding the default Minikube driver used in our terraform charts.
We tried to pick a default driver that is available on all platforms, however if there is no "one fits all", we may also change this.
According to https://minikube.sigs.k8s.io/docs/drivers/ Virtualbox should be available for macOS as well. I now checked with the Virtualbox page and it seems that only Intel Hosts are fully supported: https://www.virtualbox.org/wiki/Downloads However there seems to be a Developer preview for macOS / Arm64 (M1/M2) hosts. So this might be something to try.
QEMU is marked as experimental on Windows, so we did not chose this as the default.
Docker mentions that "The ingress, and ingress-dns addons are currently only supported on Linux. ", so we did not chose it as a default.

So I think your ingress problems might originate from the Docker driver. Most issues sound related to networking. We currently require that the hostnames set via the ingresses are reachable from the host and within the cluster.
Maybe testing with the QEMU driver or with the VirtualBox Developer Preview might help here.

Regarding launching from the dev environment in general. Our released helm charts map to tags: https://github.com/eclipsesource/theia-cloud/tags
So starting your development environment from, as of writing, tag 0.8.0.MS9 should avoid those CRD version mismatches.

[paulreuter]

Hi,

given that it was my first foray into Kubernetes, some trouble was expected. I think the team did a great job in making things accessible, and I appreciate the effort into putting together getting started configurations for beginners like me. I apologize if I came across as venting. Starting on darwin may not have been the best choice :-)

I think you are right with the problems originating from docker. I will ditch it and report back on that issue.

About the versioning issues. Running

$ help repo update 
$ helm pull theia-cloud-remote/theia-cloud

downloads 'https://github.com/eclipsesource/theia-cloud-helm/releases/download/theia-cloud-0.7.8/theia-cloud-0.7.8.tgz'
The CRD definitions in that archive also match those installed by terraform.

$ helm pull theia-cloud-remote/theia-cloud --untar --version 0.8.0.MS9
Error: chart "theia-cloud" matching 0.8.0.MS9 not found in theia-cloud-remote index. (try 'helm repo update'): improper constraint: 0.8.0.MS9

Is there another way to get the appropriate chart?

[jfaltermeier]

Below should be the required commands, currently shown for the base chart.
Please also not that helm can't update CRDs at the moment: https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#some-caveats-and-explanations

So if the CRDs were already installed, you have to manually remove them, e.g.

# check installed crds
 kubectl get crds

# delete theia cloud related
 kubectl delete crd appdefinitions.theia.cloud sessions.theia.cloud workspaces.theia.cloud

Steps to see contents from helm repo:

Update Helm Repo

~$ helm repo update theia-cloud-remote
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "theia-cloud-remote" chart repository
Update Complete. ⎈Happy Helming!⎈

Check available version (--devel is required because the 0.8.0 versions currently have a pre-release tag. This is also why helm was pulling in the 0.7.8 version)

~$ helm search repo theia-cloud-remote/theia-cloud-base --versions --devel
NAME                               	CHART VERSION	APP VERSION	DESCRIPTION           
theia-cloud-remote/theia-cloud-base	0.8.0-MS9v1  	0.8.0-MS9  	Theia-cloud base chart
theia-cloud-remote/theia-cloud-base	0.8.0-MS8v2  	0.8.0-MS8  	Theia-cloud base chart
theia-cloud-remote/theia-cloud-base	0.8.0-MS8v1  	0.8.0-MS8  	Theia-cloud base chart
theia-cloud-remote/theia-cloud-base	0.7.8        	0.7.8      	Theia-cloud base chart
theia-cloud-remote/theia-cloud-base	0.7.7        	0.7.7      	Theia-cloud base chart
theia-cloud-remote/theia-cloud-base	0.7.6        	0.7.6      	Theia-cloud base chart
theia-cloud-remote/theia-cloud-base	0.7.5        	0.7.5      	Theia-cloud base chart
theia-cloud-remote/theia-cloud-base	0.7.4        	0.7.4      	Theia-cloud base chart
theia-cloud-remote/theia-cloud-base	0.7.3        	0.7.3      	Theia-cloud base chart
theia-cloud-remote/theia-cloud-base	0.7.2        	0.7.2      	Theia-cloud base chart

Specify version to install

~$ helm install theia-cloud-base theia-cloud-remote/theia-cloud-base --version 0.8.0-MS9v1

[paulreuter]

Many thanks, I abandoned minikube entirely and got it running on GKE. With the updated charts this was a smooth process, I only had to add a PersistentVolume definition "default" for the instances to spin up. Thank you for the great work on this project, I will continue to use it.

[paulreuter]

I only now saw that the helm error message points to an URL on github, where I can browse the tags. helm pull with version 0.8.0.MS9v1 also does not work, but I can download it locally.

I am using the helm repo
https://github.eclipsesource.com/theia-cloud-helm
according to docs/Install.md
Perhaps there is some stale data over there?

[jfaltermeier]

I think the version should be 0.8.0-MS9v1. I've added the command how to list versions in a helm repo above.
I will open a ticket to adjust the git tags to match the helm versions to avoid this kind of confusion.

helm pull theia-cloud-remote/theia-cloud-base --untar --version 0.8.0-MS9v1

[paulreuter]

While updating and experimenting a bit, I noticed some things that don't quite line up.
The initial cluster configuration was done with terraform using gke_getting_started.

I then made some changes in the value file terraform/modules/helm/theia-cloud.yaml (replaced demo image with one of mine)
Unfortunately, terraform didn't pick up changes in this file - perhaps only the tf files are considered?

I went on to update the cluster using the helm chart instead and found some discrepancies.
The chart installed by terraform has put the operator, service, and landing-page in the "theiacloud" namespace.
The chart installed by helm install instead puts them into the default or no namespace.

Also, 'helm list' only shows charts directly installed with the helm command, not those from terraform. (and correspondingly, can only uninstall it's own charts).

It seems that one should not mix these approaches. Do you have a recommendation which one to use?

[jfaltermeier]

When using helm directly you can specify the namespace as well:

# list installation in default namespace
# you can use  kubectl config view  to look at the kubectl contexts
~$ helm ls

# list installations in specific namespace
~$ helm ls -n theiacloud

# list installations all namespaces
~$ helm ls --all-namespaces

-n <namespace> can also used with helm install, uninstall

Regarding running terraform again. I think this should work, because as far as I know the terraform helm provider will also just perform a helm upgrade. But I haven't tested this a lot yet.

You can check if the app definition was updated via:

# get namespaces and names of appdefinitions
~$ kubectl get appdefinitions --all-namespaces

# check the appdefinition available
~$ kubectl get appdefinitions.v5beta.theia.cloud theia-cloud-demo -n theiacloud -o yaml

Whether to use terraform or helm depends on how much you want to control with terraform, I guess.
If you want to manage your whole infrastructure in a common way, I would suggest terraform.
If you don't want to (or can't, e.g. because of some company policy) manage the Cluster with terraform you can also use helm directly.

The important thing is that when you use terraform, you should do all changes via terraform, because the tool maintains a state. If this state does not reflect the actual truth anymore, terrform may fail.