Merge branch 'tls' of https://github.com/deviantony/docker-elk

README.md

@@ -17,7 +17,7 @@ This repo provides a Dockerized ELK stack into which usage information is dumped
 # Elastic stack (ELK) on Docker
 
 [![Elastic Stack version](https://img.shields.io/badge/Elastic%20Stack-8.6.2-00bfb3?style=flat&logo=elastic-stack)](https://www.elastic.co/blog/category/releases)
-[![Build Status](https://github.com/deviantony/docker-elk/workflows/CI/badge.svg?branch=main)](https://github.com/deviantony/docker-elk/actions?query=workflow%3ACI+branch%3Amain)
+[![Build Status](https://github.com/deviantony/docker-elk/workflows/CI/badge.svg?branch=tls)](https://github.com/deviantony/docker-elk/actions?query=workflow%3ACI+branch%3Atls)
 [![Join the chat](https://badges.gitter.im/Join%20Chat.svg)](https://app.gitter.im/#/room/#deviantony_docker-elk:gitter.im)
 
 Run the latest version of the [Elastic stack][elk-stack] with Docker and Docker Compose.
@@ -79,6 +79,7 @@ own_. [sherifabdlnaby/elastdocker][elastdocker] is one example among others of p
    * [How to configure Logstash](#how-to-configure-logstash)
    * [How to disable paid features](#how-to-disable-paid-features)
    * [How to scale out the Elasticsearch cluster](#how-to-scale-out-the-elasticsearch-cluster)
+   * [How to re-generate TLS certificates](#how-to-re-generate-tls-certificates)
    * [How to re-execute the setup](#how-to-re-execute-the-setup)
    * [How to reset a password programmatically](#how-to-reset-a-password-programmatically)
 1. [Extensibility](#extensibility)
@@ -145,10 +146,22 @@ instructions from the [documentation][mac-filesharing] to add more locations.
 Clone this repository onto the Docker host that will run the stack with the command below:
 
 ```sh
-git clone https://github.com/deviantony/docker-elk.git
+git clone --branch tls https://github.com/deviantony/docker-elk.git
 ```
 
-Then, start the stack components locally with Docker Compose:
+Then, generate X.509 certificates and private keys to enable secure communications over TLS between components:
+
+```sh
+docker-compose up tls
+```
+
+> **Note**  
+> All Elastic components — including [extensions](#how-to-enable-the-provided-extensions) — are pre-configured to use
+> the certificates generated by this command. To change the DNS names and IP addresses to include in the certificates,
+> or re-generate them at a later time, refer to [How to re-generate TLS
+> certificates](#how-to-re-generate-tls-certificates).
+
+After TLS certificates have been generated, start the stack components locally with Docker Compose:
 
 ```sh
 docker-compose up
@@ -189,15 +202,15 @@ reset the passwords of all aforementioned Elasticsearch users to random secrets.
     of them.
 
     ```sh
-    docker-compose exec elasticsearch bin/elasticsearch-reset-password --batch --user elastic
+    docker-compose exec elasticsearch bin/elasticsearch-reset-password --batch --user elastic --url https://localhost:9200
     ```
 
     ```sh
-    docker-compose exec elasticsearch bin/elasticsearch-reset-password --batch --user logstash_internal
+    docker-compose exec elasticsearch bin/elasticsearch-reset-password --batch --user logstash_internal --url https://localhost:9200
     ```
 
     ```sh
-    docker-compose exec elasticsearch bin/elasticsearch-reset-password --batch --user kibana_system
+    docker-compose exec elasticsearch bin/elasticsearch-reset-password --batch --user kibana_system --url https://localhost:9200
     ```
 
     If the need for it arises (e.g. if you want to [collect monitoring information][ls-monitoring] through Beats and
@@ -352,6 +365,35 @@ You can also cancel an ongoing trial before its expiry date — and thus revert
 
 Follow the instructions from the Wiki: [Scaling out Elasticsearch](https://github.com/deviantony/docker-elk/wiki/Elasticsearch-cluster)
 
+### How to re-generate TLS certificates
+
+To re-generate TLS certificates and private keys, first ensure that the [tls/instances.yml](./tls/instances.yml) file
+contains a list of certificates with suitable domains and IP addresses for your environment (defaults are suitable for
+local installations without access from other hosts on the network).
+
+Then, remove existing TLS certificates and private keys using the command below:
+
+```console
+$ find tls/certs -name ca -prune -or -type d -mindepth 1 -exec rm -rfv {} +
+tls/certs/kibana/kibana.key
+tls/certs/kibana/kibana.crt
+tls/certs/kibana
+tls/certs/apm-server/apm-server.crt
+tls/certs/apm-server/apm-server.key
+tls/certs/apm-server
+tls/certs/fleet-server/fleet-server.key
+tls/certs/fleet-server/fleet-server.crt
+tls/certs/fleet-server
+tls/certs/elasticsearch/elasticsearch.key
+tls/certs/elasticsearch/elasticsearch.crt
+tls/certs/elasticsearch
+```
+
+and run the `tls` service again.
+
+Alternatively, you can refer to the documentation page [Manually configure security][es-tls] from the Elastic
+documentation to generate certificates and private keys manually.
+
 ### How to re-execute the setup
 
 To run the setup container again and re-initialize all users for which a password was defined inside the `.env` file,
@@ -389,7 +431,8 @@ users][builtin-users]), you can use the Elasticsearch API instead and achieve th
 In the example below, we reset the password of the `elastic` user (notice "/user/elastic" in the URL):
 
 ```sh
-curl -XPOST -D- 'http://localhost:9200/_security/user/elastic/_password' \
+curl -XPOST -D- 'https://localhost:9200/_security/user/elastic/_password' \
+    --cacert tls/certs/ca/ca.crt \
     -H 'Content-Type: application/json' \
     -u elastic:<your current elastic password> \
     -d '{"password" : "<your new password>"}'
@@ -491,6 +534,7 @@ See the following Wiki pages:
 [mac-filesharing]: https://docs.docker.com/desktop/settings/mac/#file-sharing
 
 [builtin-users]: https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-users.html
+[es-tls]: https://www.elastic.co/guide/en/elasticsearch/reference/current/manually-configure-security.html
 [ls-monitoring]: https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
 [sec-cluster]: https://www.elastic.co/guide/en/elasticsearch/reference/current/secure-cluster.html
 

docker-compose.yml

@@ -1,6 +1,29 @@
 version: '3.7'
 
 services:
+  # The 'tls' service runs a one-off script which initializes TLS certificates and
+  # private keys for all components of the stack inside the local tls/ directory.
+  #
+  # This task only needs to be performed once, *before* the first stack startup.
+  #
+  # By default, it is excluded from the services started by 'docker compose up'
+  # due to the non-default profile it belongs to. To run it, either provide the
+  # '--profile=setup' CLI flag to Compose commands, or "up" the service by name
+  # such as 'docker compose up tls'.
+  tls:
+    profiles:
+      - setup
+    build:
+      context: tls/
+      args:
+        ELASTIC_VERSION: ${ELASTIC_VERSION}
+    user: root  # ensures we can write to the local tls/ directory.
+    init: true
+    volumes:
+      - ./tls/entrypoint.sh:/entrypoint.sh:ro,Z
+      - ./tls/instances.yml:/usr/share/elasticsearch/tls/instances.yml:ro,Z
+      # - ./tls/certs:/usr/share/elasticsearch/tls/certs:z
+      - ${METRICS_CRTS_DIR}:/usr/share/elasticsearch/tls/certs:z
 
   # The 'setup' service runs a one-off script which initializes users inside
   # Elasticsearch — such as 'logstash_internal' and 'kibana_system' — with the
@@ -20,6 +43,8 @@ services:
       - ./setup/lib.sh:/lib.sh:ro,Z
       - ./setup/roles:/roles:ro,Z
       - setup:/state:Z
+      # (!) CA certificate. Generate using the 'tls' service.
+      - ./tls/certs/ca/ca.crt:/ca.crt:ro,z
     environment:
       ELASTIC_PASSWORD: ${ELASTIC_PASSWORD:-}
       LOGSTASH_INTERNAL_PASSWORD: ${LOGSTASH_INTERNAL_PASSWORD:-}
@@ -42,6 +67,10 @@ services:
     volumes:
       - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro,Z
       - elasticsearch:/usr/share/elasticsearch/data:Z
+      # (!) TLS certificates. Generate using the 'tls' service.
+      - ./tls/certs/ca/ca.crt:/usr/share/elasticsearch/config/ca.crt:ro,z
+      - ./tls/certs/elasticsearch/elasticsearch.crt:/usr/share/elasticsearch/config/elasticsearch.crt:ro,z
+      - ./tls/certs/elasticsearch/elasticsearch.key:/usr/share/elasticsearch/config/elasticsearch.key:ro,z
     # ports:
     #   - 9200:9200
     #   - 9300:9300
@@ -69,6 +98,8 @@ services:
       - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro,z
       - ./logstash/pipeline:/usr/share/logstash/pipeline:ro,z
       - ./logstash/config/hs-template.json:/etc/logstash/hs-template.json:ro,Z
+      # (!) CA certificate. Generate using the 'tls' service.
+      - ./tls/certs/ca/ca.crt:/usr/share/logstash/config/ca.crt:ro,z
       # Easy viewing of logs on host for debug
       # - ./logs/logstash:${METRICS_LOGS_DIR}/logstash/
     # ports:
@@ -96,6 +127,10 @@ services:
         ELASTIC_VERSION: ${ELASTIC_VERSION}
     volumes:
       - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:ro,Z
+      # (!) TLS certificates. Generate using the 'tls' service.
+      - ./tls/certs/ca/ca.crt:/usr/share/kibana/config/ca.crt:ro,z
+      - ./tls/certs/kibana/kibana.crt:/usr/share/kibana/config/kibana.crt:ro,Z
+      - ./tls/certs/kibana/kibana.key:/usr/share/kibana/config/kibana.key:ro,Z
     # ports:
     #   - 5601:5601
     environment:
@@ -131,19 +166,6 @@ services:
       ELASTIC_PROXY_PORT: ${ELASTIC_PROXY_PORT}
       KIBANA_PROXY_PORT: ${KIBANA_PROXY_PORT}
       LOGSTASH_PROXY_PORT: ${LOGSTASH_PROXY_PORT}
-
-  self-cert:
-    build:
-      context: self-cert/
-    volumes: 
-      - ${METRICS_CRTS_DIR}:/etc/certs
-    # On first run, or to update certs, run `docker compose --profile cert-update up`
-    profiles:
-      - cert-update
-    command: >
-      openssl req -new -newkey rsa:2048 -x509 -nodes -days 365
-      -subj  "/C=US/ST=MA/L=Arlington/O=CUAHSI/OU=IT/CN=${METRICS_HOSTNAME}"
-      -keyout /etc/certs/${METRICS_HOSTNAME}.key -out /etc/certs/${METRICS_HOSTNAME}.crt;
 
   caddy:
     build:

elasticsearch/config/elasticsearch.yml

@@ -10,11 +10,30 @@ network.host: 0.0.0.0
 #
 xpack.license.self_generated.type: basic
 xpack.security.enabled: true
-# The API key service is required by Fleet, but disabled by default when the
-# Elasticsearch API is not served over TLS.
-xpack.security.authc.api_key.enabled: true
+
+##
+## TLS configuration
+## See instructions from README to enable.
+##
+
+## Communications between nodes in a cluster
+## see https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls.html#tls-transport
+#
+xpack.security.transport.ssl.enabled: true
+xpack.security.transport.ssl.verification_mode: certificate
+xpack.security.transport.ssl.certificate_authorities: [ ca.crt ]
+xpack.security.transport.ssl.certificate: elasticsearch.crt
+xpack.security.transport.ssl.key: elasticsearch.key
+
+## HTTP client communications
+## see https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls.html#tls-http
+#
+xpack.security.http.ssl.enabled: true
+xpack.security.http.ssl.certificate_authorities: [ ca.crt ]
+xpack.security.http.ssl.certificate: elasticsearch.crt
+xpack.security.http.ssl.key: elasticsearch.key
 
 # increase threadpool queue size to avoid Courier Fetch errors
 # # $ watch curl localhost:9200/_cat/thread_pool
 # # https://qbox.io/blog/thread-pools-elasticsearch-search-request-errors
-thread_pool.search.queue_size: 3000
+thread_pool.search.queue_size: 3000

extensions/curator/config/curator.yml

@@ -3,7 +3,8 @@
 
 elasticsearch:
   client:
-    hosts: [ http://elasticsearch:9200 ]
+    hosts: [ https://elasticsearch:9200 ]
+    ca_certs: /.curator/ca.crt
   other_settings:
     username: elastic
     password: ${ELASTIC_PASSWORD}

extensions/curator/curator-compose.yml

@@ -8,6 +8,8 @@ services:
     volumes:
       - ./extensions/curator/config/curator.yml:/.curator/curator.yml:ro,Z
       - ./extensions/curator/config/delete_log_files_curator.yml:/.curator/delete_log_files_curator.yml:ro,Z
+      # (!) CA certificate. Generate using the 'tls' service.
+      - ./tls/certs/ca/ca.crt:/.curator/ca.crt:ro,z
     environment:
       ELASTIC_PASSWORD: ${ELASTIC_PASSWORD:-}
     networks:

extensions/enterprise-search/config/enterprise-search.yml

@@ -20,9 +20,11 @@ ent_search.external_url: http://localhost:3002
 kibana.host: http://localhost:5601
 
 # Elasticsearch URL and credentials
-elasticsearch.host: http://elasticsearch:9200
+elasticsearch.host: https://elasticsearch:9200
 elasticsearch.username: elastic
 elasticsearch.password: ${ELASTIC_PASSWORD}
+elasticsearch.ssl.enabled: true
+elasticsearch.ssl.certificate_authority: /usr/share/enterprise-search/config/ca.crt
 
 # Allow Enterprise Search to modify Elasticsearch settings. Used to enable auto-creation of Elasticsearch indexes.
 allow_es_settings_modification: true

extensions/enterprise-search/enterprise-search-compose.yml

@@ -8,6 +8,8 @@ services:
         ELASTIC_VERSION: ${ELASTIC_VERSION}
     volumes:
       - ./extensions/enterprise-search/config/enterprise-search.yml:/usr/share/enterprise-search/config/enterprise-search.yml:ro,Z
+      # (!) CA certificate. Generate using the 'tls' service.
+      - ./tls/certs/ca/ca.crt:/usr/share/enterprise-search/config/ca.crt:ro,z
     environment:
       JAVA_OPTS: -Xms2g -Xmx2g
       ENT_SEARCH_DEFAULT_PASSWORD: 'changeme'

extensions/filebeat/config/filebeat.yml

@@ -26,9 +26,13 @@ monitoring:
     password: ${BEATS_SYSTEM_PASSWORD}
 
 output.elasticsearch:
-  hosts: [ http://elasticsearch:9200 ]
+  hosts: [ https://elasticsearch:9200 ]
   username: filebeat_internal
   password: ${FILEBEAT_INTERNAL_PASSWORD}
+  ssl:
+    # Using ${path.home} here while monitoring is enabled causes Beats to exit with:
+    #   missing field accessing 'ssl.certificate_authorities.0'
+    certificate_authorities: [ /usr/share/filebeat/ca.crt ]
 
 ## HTTP endpoint for health checking
 ## https://www.elastic.co/guide/en/beats/filebeat/current/http-endpoint.html

extensions/filebeat/filebeat-compose.yml

@@ -26,6 +26,8 @@ services:
         source: /var/run/docker.sock
         target: /var/run/docker.sock
         read_only: true
+      # (!) CA certificate. Generate using the 'tls' service.
+      - ./tls/certs/ca/ca.crt:/usr/share/filebeat/ca.crt:ro,z
     environment:
       FILEBEAT_INTERNAL_PASSWORD: ${FILEBEAT_INTERNAL_PASSWORD:-}
       BEATS_SYSTEM_PASSWORD: ${BEATS_SYSTEM_PASSWORD:-}

extensions/fleet/README.md

@@ -17,6 +17,25 @@ The Fleet Server exposes the TCP port `8220` for Agent to Server communications.
 
 ## Usage
 
+### CA Certificate Fingerprint
+
+Before starting Fleet Server, take note of the CA certificate's SHA256 fingerprint printed by the `docker-compose up
+tls` command (it is safe to run it multiple times), and use it as the value of the commented `ca_trusted_fingerprint`
+setting inside the [`kibana/config/kibana.yml`][config-kbn] file.
+
+The fingerprint appears on a line similar to the one below, in the output of the aforementioned command:
+
+```none
+⠿ SHA256 fingerprint: 846637d1bb82209640d31b79869a370c8e47c2dc15c7eafd4f3d615e51e3d503
+```
+
+This fingerprint is required for Fleet Server (and other Elastic Agents) to be able to verify the authenticity of the CA
+certificate presented by Elasticsearch during TLS handshakes.
+
+Restart Kibana with `docker-compose restart kibana` if it is already running.
+
+### Startup
+
 To include Fleet Server in the stack, run Docker Compose from the root of the repository with an additional command line
 argument referencing the `fleet-compose.yml` file:
 

extensions/fleet/agent-apmserver-compose.yml

@@ -18,11 +18,16 @@ services:
         ELASTIC_VERSION: ${ELASTIC_VERSION}
     volumes:
       - apm-server:/usr/share/elastic-agent/state:Z
+      # (!) TLS certificates. Generate using the 'tls' service.
+      - ./tls/certs/ca/ca.crt:/usr/share/elastic-agent/ca.crt:ro,z
+      - ./tls/certs/apm-server/apm-server.crt:/usr/share/elastic-agent/apm-server.crt:ro,Z
+      - ./tls/certs/apm-server/apm-server.key:/usr/share/elastic-agent/apm-server.key:ro,Z
     environment:
       FLEET_ENROLL: '1'
       FLEET_TOKEN_POLICY_NAME: Agent Policy APM Server
-      FLEET_INSECURE: '1'
-      FLEET_URL: http://fleet-server:8220
+      FLEET_URL: https://fleet-server:8220
+      ELASTICSEARCH_HOST: https://elasticsearch:9200
+      ELASTICSEARCH_CA: /usr/share/elastic-agent/ca.crt
       # Enrollment.
       # (a) Auto-enroll using basic authentication
       ELASTICSEARCH_USERNAME: elastic

extensions/fleet/fleet-compose.yml

@@ -8,11 +8,19 @@ services:
         ELASTIC_VERSION: ${ELASTIC_VERSION}
     volumes:
       - fleet-server:/usr/share/elastic-agent/state:Z
+      # (!) TLS certificates. Generate using the 'tls' service.
+      - ./tls/certs/ca/ca.crt:/usr/share/elastic-agent/ca.crt:ro,z
+      - ./tls/certs/fleet-server/fleet-server.crt:/usr/share/elastic-agent/fleet-server.crt:ro,Z
+      - ./tls/certs/fleet-server/fleet-server.key:/usr/share/elastic-agent/fleet-server.key:ro,Z
     environment:
       FLEET_SERVER_ENABLE: '1'
-      FLEET_SERVER_INSECURE_HTTP: '1'
       FLEET_SERVER_HOST: 0.0.0.0
       FLEET_SERVER_POLICY_ID: fleet-server-policy
+      FLEET_URL: https://fleet-server:8220
+      FLEET_SERVER_CERT: /usr/share/elastic-agent/fleet-server.crt
+      FLEET_SERVER_CERT_KEY: /usr/share/elastic-agent/fleet-server.key
+      ELASTICSEARCH_HOST: https://elasticsearch:9200
+      ELASTICSEARCH_CA: /usr/share/elastic-agent/ca.crt
       # Fleet plugin in Kibana
       KIBANA_FLEET_SETUP: '1'
       # Enrollment.

extensions/heartbeat/config/heartbeat.yml

@@ -8,9 +8,11 @@ heartbeat.monitors:
 - type: http
   schedule: '@every 5s'
   urls:
-    - http://elasticsearch:9200
+    - https://elasticsearch:9200
   username: heartbeat_internal
   password: ${HEARTBEAT_INTERNAL_PASSWORD}
+  ssl:
+    certificate_authorities: [ '${path.home}/ca.crt' ]
 
 - type: icmp
   schedule: '@every 5s'
@@ -27,9 +29,13 @@ monitoring:
     password: ${BEATS_SYSTEM_PASSWORD}
 
 output.elasticsearch:
-  hosts: [ http://elasticsearch:9200 ]
+  hosts: [ https://elasticsearch:9200 ]
   username: heartbeat_internal
   password: ${HEARTBEAT_INTERNAL_PASSWORD}
+  ssl:
+    # Using ${path.home} here while monitoring is enabled causes Beats to exit with:
+    #   missing field accessing 'ssl.certificate_authorities.0'
+    certificate_authorities: [ /usr/share/heartbeat/ca.crt ]
 
 ## HTTP endpoint for health checking
 ## https://www.elastic.co/guide/en/beats/heartbeat/current/http-endpoint.html