[BUG] Docker stack not running correctly when mounting config folder

[manodupont]

I am trying to launch a stack with docker :
This is the docker-compose.yml

version: "3"

services:
  os01:
    image: opensearchproject/opensearch:1.1.0
    environment:
      OPENSEARCH_JAVA_OPTS: "-Xms1024m -Xmx1024m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      node.name: os01
      plugins.security.ssl.transport.pemkey_filepath: certificates/os01/os01.key # relative path
      plugins.security.ssl.transport.pemcert_filepath: certificates/os01/os01.pem
      plugins.security.ssl.http.pemkey_filepath: certificates/os01/os01.key
      plugins.security.ssl.http.pemcert_filepath: certificates/os01/os01.pem
      DISABLE_INSTALL_DEMO_CONFIG: "true"
      JAVA_HOME: /usr/share/opensearch/jdk
      bootstrap.memory_lock: "true" # along with the memlock settings below, disables swapping
      network.host: "0.0.0.0"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./opensearch.yml:/usr/share/opensearch/config/opensearch.yml
      - ./config.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml
      - ./certs:/usr/share/opensearch/config/certificates
      - os-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200

  dashboards:
    image: opensearchproject/opensearch-dashboards:1.1.0
    ports:
      - 5601:5601
    volumes:
      - ./certs:/usr/share/opensearch-dashboards/config/certificates:rw
      - ./opensearch-dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml
    environment:
      DISABLE_INSTALL_DEMO_CONFIG: "true"

volumes:
  os-data1:

As soon as you launch that, you get this error

opensearch1-os01-1  | Disabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
opensearch1-os01-1  | Enabling OpenSearch Security Plugin
opensearch1-os01-1  | sed: cannot rename /usr/share/opensearch/config/sed6FI3Gg: Device or resource busy
opensearch1-os01-1  | Exception in thread "main" org.opensearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /usr/share/opensearch/config/certificates/ca
opensearch1-os01-1  | Likely root cause: java.nio.file.AccessDeniedException: /usr/share/opensearch/config/certificates/ca
opensearch1-os01-1  | 	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
opensearch1-os01-1  | 	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
opensearch1-os01-1  | 	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
opensearch1-os01-1  | 	at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:431)
opensearch1-os01-1  | 	at java.base/java.nio.file.Files.newDirectoryStream(Files.java:476)
opensearch1-os01-1  | 	at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:300)
opensearch1-os01-1  | 	at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
opensearch1-os01-1  | 	at java.base/java.nio.file.Files.walkFileTree(Files.java:2840)
opensearch1-os01-1  | 	at org.opensearch.common.logging.LogConfigurator.configure(LogConfigurator.java:233)
opensearch1-os01-1  | 	at org.opensearch.common.logging.LogConfigurator.configure(LogConfigurator.java:142)
opensearch1-os01-1  | 	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:367)
opensearch1-os01-1  | 	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:182)
opensearch1-os01-1  | 	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:173)
opensearch1-os01-1  | 	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:99)
opensearch1-os01-1  | 	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:140)
opensearch1-os01-1  | 	at org.opensearch.cli.Command.main(Command.java:103)
opensearch1-os01-1  | 	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:139)
opensearch1-os01-1  | 	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:105)
opensearch1-os01-1  | For complete error details, refer to the log at /usr/share/opensearch/logs/os-cluster.log
opensearch1-os01-1  | Killing performance analyzer process 15
opensearch1-os01-1  | OpenSearch exited with code 1
opensearch1-os01-1  | Performance analyzer exited with code 143

Any idea would be very helpful.

If you want to reproduce you will need to have the certs installed etc ...., just follow the steps in there : https://github.com/flavienbwk/opensearch-docker-compose

which is using the exact same docker-compose as I posted here.

[scratchings]

This causes the same issue for swarm mode in Docker where opensearch.yml is mounted as a 'config' and thus is read-only. #1579 is the same problem.
The sed command attempts to modify the stock opensearch.yml to reflect features requested by environment variables. This is not appropriate (or in these cases is not possible) when someone has specified a custom opensearch.yml file.

[peterzhuamazon]

Fixed in opensearch-project/opensearch-build#1130