v4.10.1 wazuh-production-ready.yml playbook fails with FQDNs in hosts file (OpenSearch Security not initialized)

[czuk]

I tried several times to create a cluster using v4.10.1 wazuh-production-ready.yml with FQDNs in the ansible hosts file,e.g.

wi1 ansible_host=wazind01.domain.local private_ip=wazind01.domain.local indexer_node_name=node-1
wi2 ansible_host=wazind02.domain.local private_ip=wazind02.domain.local indexer_node_name=node-2
wi3 ansible_host=wazind03.domain.local private_ip=wazind03.domain.local indexer_node_name=node-3
dashboard  ansible_host=waznode01.domain.local private_ip=waznode01.domain.local
manager ansible_host=waznode02.domain.local private_ip=waznode02.domain.local
worker  ansible_host=waznode03.domain.local private_ip=waznode03.domain.local

[wi_cluster]
wi1
wi2
wi3

[all:vars]
ansible_ssh_user=ubuntu
ansible_ssh_private_key_file=/home/ubuntu/.ssh/id_ed25519
ansible_ssh_extra_args='-o StrictHostKeyChecking=no

Each time it failed to bring up the cluster. I was getting "OpenSearch Security not initialized" errors in the logs and the accepted answer seems to be to run this command:

/usr/share/wazuh-indexer/plugins/opensearch-security/tools/securityadmin.sh -cd /usr/etc/wazuh-indexer/opensearch-security/ -nhnv -icl -cacert /etc/wazuh-indexer/certs/root-ca.pem -cert /etc/wazuh-indexer/certs/admin.pem -key /etc/wazuh-indexer/certs/admin-key.pem -h wazind01.domain.local

Which looped with timeouts:

Will connect to wazind01.domain.local:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.16.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]. This is not an error, will keep on trying ...
  Root cause: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE] (java.net.SocketTimeoutException/java.net.SocketTimeoutException)
   * Try running securityadmin.sh with -icl (but no -cn) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

Eventually I decided to take DNS resolution out of the equation and changed the ansible hosts file to:

wi1 ansible_host=10.x.x.62 private_ip=10.1x.x.62 indexer_node_name=node-1
wi2 ansible_host=10.x.x.63 private_ip=10.x.x.63 indexer_node_name=node-2
wi3 ansible_host=10.x.x.64 private_ip=10.x.x.64 indexer_node_name=node-3
dashboard  ansible_host=10.x.x.65 private_ip=10.x.x.65
manager ansible_host=10.x.x.66 private_ip=10.x.x.66
worker  ansible_host=10.x.x.67 private_ip=10.x.x.67

[wi_cluster]
wi1
wi2
wi3

[all:vars]
ansible_ssh_user=ubuntu
ansible_ssh_private_key_file=/home/ubuntu/.ssh/id_ed25519
ansible_ssh_extra_args='-o StrictHostKeyChecking=no'

And the playbook works, and I now have a working cluster. DNS resolution has been tested from all nodes and is working correctly.