High availability for Redis, Solr and Postgres

[gislab-augsburg]

Proposed changes:

• Make number of replicas configurable for Solr statefulset
• Make number of replicas configurable for Redis statefulset

Background: We recently had an OpenShift update which produced some Solr Errors. Not a big thing, but as solr needs circa 1 minute for startup, the possibility of scaling up the number of pods/ containers would be good for high availability.

I added Redis for completeness. I see no reason why Redis replica count is only used for deployment and not for statefulset, but please do not hesitate to correct me here.

@BWibo I addressed the branch release/3.0.0 because we use it at the moment. No problem to change this to the main branch if it's more convenient for you.

[gislab-augsburg]

Oh, I did not test my changes carefully enough. Seems like 2 solr pods/containers cannot connect simultanously to ckan.

The first solr pod is working, the second produces an error :

2024-03-21 15:08:05.287 ERROR (qtp1426435610-18) [] o.a.s.s.HttpSolrCall org.apache.solr.core.SolrCoreInitializationException: SolrCore 'ckan' is not available due to init failure: Index dir '/var/solr/data/ckan/data/index/' of core 'ckan' is already locked. The most likely cause is another Solr server (or another solr core in this server) also configured to use this directory; other possible causes may be specific to lockType: native => org.apache.solr.core.SolrCoreInitializationException: SolrCore 'ckan' is not available due to init failure: Index dir '/var/solr/data/ckan/data/index/' of core 'ckan' is already locked. The most likely cause is another Solr server (or another solr core in this server) also configured to use this directory; other possible causes may be specific to lockType: native
at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:2217)
org.apache.solr.core.SolrCoreInitializationException: SolrCore 'ckan' is not available due to init failure: Index dir '/var/solr/data/ckan/data/index/' of core 'ckan' is already locked. The most likely cause is another Solr server (or another solr core in this server) also configured to use this directory; other possible causes may be specific to lockType: native

Also the second ckan pod is crashing because of no connection to solr:

/srv/app/start_ckan.sh: Ignoring /srv/app/docker-entrypoint.d/* (not an sh or py file)
Not all environment variables are set. Generating sessions...
[prerun] Start check_db_connection...
[prerun] Start check_solr_connection...
[prerun] Unable to connect to solr...try again in a while.
[prerun] Start check_solr_connection...
[prerun] Unable to connect to solr...try again in a while.
[prerun] Start check_solr_connection...
[prerun] Unable to connect to solr...try again in a while.
[prerun] Start check_solr_connection...
[prerun] Unable to connect to solr...try again in a while.
[prerun] Start check_solr_connection...
[prerun] Unable to connect to solr...try again in a while.
[prerun] Start check_solr_connection...
[prerun] Giving up after 5 tries...
[CKAN prerun] FAILED. Exiting...

[BWibo]

Hehe, yes. High availability is significantly more complex than just raising the number of replicas - at least for stateful applications.

Let's have 3.0.0 released without this. We'll address this issue in another release.

There are HA charts available, that we can use. We'll have to check how much effort it is to integrate them and if they work with our app.

• Redis: https://artifacthub.io/packages/helm/bitnami/redis
  • Here is a good chance, that this will be easy.
• PostgreSQL: https://artifacthub.io/packages/helm/bitnami/postgresql-ha
  • Database initialization could be tricky. Would probably also solve Decouple database initialization #10.
• Solr: https://artifacthub.io/packages/helm/apache-solr/solr, https://artifacthub.io/packages/helm/bitnami/solr
  • Requires more reseach.