DATASOURCES_DEFAULT_HOST can't be empty when mariadbDependencyEnabled and postgresqlDependencyEnabled are disabled

[vaaleyard]

I'm trying to use an external database as the passbolt db, and I want to pass its variables to fetch from a secret I have in the my kubernetes cluster.

I have a secret in kubernetes with four variables, which has the connection settings for the database:

DATASOURCES_DEFAULT_DATABASE:  8 bytes
DATASOURCES_DEFAULT_HOST:      58 bytes
DATASOURCES_DEFAULT_PASSWORD:  28 bytes
DATASOURCES_DEFAULT_USERNAME:  5 bytes

My values file is something like this:

redisDependencyEnabled: false
mariadbDependencyEnabled: false

passboltEnv:
  plain:
    EMAIL_TRANSPORT_DEFAULT_TLS: false
    PASSBOLT_SSL_FORCE: false
    APP_FULL_BASE_URL: "https://passbolt.tld"

  # with extraenvfrom it also doesn't work
  # extraEnvFrom:
  #   - secretRef:
  #       name: passbolt-config-secret
  
  extraEnv:
  - name: DATASOURCES_DEFAULT_HOST
    valueFrom:
      secretKeyRef:
        name: passbolt-config-secret
        key: DATASOURCES_DEFAULT_HOST
  - name: DATASOURCES_DEFAULT_DATABASE
    valueFrom:
      secretKeyRef:
        name: passbolt-config-secret
        key: DATASOURCES_DEFAULT_DATABASE
  - name: DATASOURCES_DEFAULT_USERNAME
    valueFrom:
      secretKeyRef:
        name: passbolt-config-secret
        key: DATASOURCES_DEFAULT_USERNAME
  - name: DATASOURCES_DEFAULT_DATABASE
    valueFrom:
      secretKeyRef:
        name: passbolt-config-secret
        key: DATASOURCES_DEFAULT_DATABASE

Troubleshooting the error message:

Error: UPGRADE FAILED: execution error at (passbolt/templates/deployment.yaml:122:24): DATASOURCES_DEFAULT_HOST can't be empty when mariadbDependencyEnabled and postgresqlDependencyEnabled are disabled

and going to _helpers.tpl#L67 it looks like I have to obligatory pass the HOST variable as plain... Wouldn't it be better to also allow it to pass as a secret variable? Because it doesn't make sense to also leave this var in plain text.

[dlen]

Hey @vaaleyard,

Yes this is intentional as we considered that for the database hostname it would be enough to store it as a configmap.
There are no plans in the short term to change that. I'm leaving this issue open in case your suggestion shows community traction in order to implement it.

[vaaleyard]

And is it possible to pass the other DATASOURCES_DEFAULT_* variables as a secret? Like in the example above
Because in secret-env.yaml#L22 it uses the passboltEnv.secret directly...

[dlen]

I think you would have problems with this function https://github.com/passbolt/charts-passbolt/blob/main/templates/_helpers.tpl#L67 that doesn't consider that the host could be stored in a secret.

[emouawad]

+1 to using existing database and load necessary envs from secret

[vaaleyard]

The only "important" vars I could set this way was these ones:

  extraEnv:
  - name: DATASOURCES_DEFAULT_PASSWORD
    valueFrom:
      secretKeyRef:
        name: &secret passbolt-config-db-secret
        key: password
  - name: DATASOURCES_DEFAULT_USERNAME
    valueFrom:
      secretKeyRef:
        name: *secret
        key: username
  - name: DATASOURCES_DEFAULT_DATABASE
    valueFrom:
      secretKeyRef:
        name: *secret
        key: username
  - name: CACHE_CAKE_DEFAULT_PASSWORD
    valueFrom:
      secretKeyRef:
        name: &secret passbolt-config-secret
        key: CACHE_CAKE_DEFAULT_PASSWORD

At least these ones works.

[Tecnobutrul]

Hello.

Thank you for your contribution. We are working on a fix for this issue and it will be included in the next release, which will be available in a few days.

[timoRumlaut]

Hey,
stumbled over this one and facing the same issues.
I am provisioning an AWS postgres database and writing all important infos to a secret. Because the hostname has some kind of id and is not static.
In the helm-chart I am also trying to extraEnv this variables, but HOST and PORT is not possible this way.

At least I am getting:

✗ Rendering Helm Charts
✗ 1 error occurred:
        * rendering helm chart passbolt for release passbolt has failed: execution error at (passbolt/templates/job-create-jwt.yaml:58:24): DATASOURCES_DEFAULT_HOST can't be empty when mariadbDependencyEnabled and postgresqlDependencyEnabled are disabled

Isnt this issue solved yet?

Thanks and have a nice one

[vaaleyard]

This is not solved.

You have to pass the variable in the values directly, but I leave the variable empty in the git repository.

[dlen]

As a workaround: If your db hostname id is not static you could retrieve it by any means and deploy it using a --set parameter when invoking the helm upgrade in your pipelines

[timoRumlaut]

Tried it this way as well:

passboltEnv:
  plain:
    DATASOURCES_DEFAULT_HOST: "${DATABASE_HOST_FROM_SECRET}"
  extraEnv:
    - name: DATABASE_HOST_FROM_SECRET
      valueFrom:
        secretKeyRef:
          name: rds-passbolt-user-app-rw
          key: host

without luck ;-). Used a similar workaround for another helm deployment.

Unfortunately, DATASOURCES_DEFAULT_HOST is not allowed to be empty when disabling intergrated postgres or mariadb, but defining the variable in plain: and extraEnv: block is either not allowed.
Also tried to change the helm-chart itself, but am not so firm with helm these days.

Easiest way for my understanding would be, if the helm-chart respects if DATASOURCES_DEFAULT_HOST and PORT are defined via extraEnv:, it dont needs to be defined via plain:.

[timoRumlaut]

Ok, more than that. I tried to set the variables hard coded for testing and maybe find a better solution afterwards.
Services starting so far, but access is not possible. Showing, that there is another variable inside the passbolt container, which will not be change in favor of DATASOURCE_DEFAULT_USERNAME and PASSWORD, but is properly needed during runtime.

DATASOURCES_DEFAULT_URL=postgres://CHANGEME:CHANGEME@<my-aws-postgres-database>.rds.amazonaws.com:5432/passbolt?schema=passbolt

You see, host, port, database seems to be changed in that DEFAULT_URL, but USERNAME and PASSWORD are not respected from variables.

edit: this seems to happen during first user creation command inside php

[Tecnobutrul]

Hello! One of the main concerns that we had when we created this helm charts was it had to be batteries included solution. So you can install and test passbolt on a kubernetes cluster in one shot, without any configuration. To do so the charts calculates some of the needed configurations (database service url, for instance) if it is possible from other values. That's why we introduced the error message that you are facing to help users to understand what is missing when they introduced an incompatible configuration.

For this particular issue, we need to know the database host at render time in order to create the DATASOURCES_DEFAULT_URL environment variable for postgresql. If that value is already set on a secret in the k8s cluster, it is imposible to calculate the url so it returns an error and stop the installation.

Our proposal solution is, we keep checking for incompatible configurations as long the user doesn't set any .Values.passboltEnv.extraEnvFrom. In the case that the user added an entry on .Values.passboltEnv.extraEnvFrom we are going to print a warning if we don't find some of the needed values but we let the process run.

Proposal solution

Create a secret with the environment values:

apiVersion: v1
kind: Secret
metadata:
  name: test-secret
data:
  DATASOURCES_DEFAULT_URL: CHANGEME
  DATASOURCES_DEFAULT_DATABASE: CHANGEME
  DATASOURCES_DEFAULT_PASSWORD: CHANGEME
  DATASOURCES_DEFAULT_HOST: CHANGEME

Add the following configuration on your values.yaml

passboltEnv:
  extraEnvFrom:
    - secretRef:
        name: test-secret

Then, after the helm install/upgrade you will see a warning (we could add a flag to silent this message if needed):

Looks like you are configuring passbolt with postgresql using external secrets. Be sure that your secret contains
 the following environment values otherwise you will run into errors.
 - DATASOURCES_DEFAULT_URL
 - DATASOURCES_DEFAULT_DATABASE
 - DATASOURCES_DEFAULT_PASSWORD
 - DATASOURCES_DEFAULT_HOST

We would love to hear your feedback on this proposal, and feel free to share a different one in case this one doesn't fit your needs.