[Bug]: opensearch.keystore.tmp: Read-only file system

[jbnjohnathan]

Describe the bug

When running opensearch 2.9.0 in a docker container in read only mode the following error is thrown

Exception in thread "main" org.opensearch.bootstrap.BootstrapException: java.nio.file.FileSystemException: /usr/share/opensearch/config/opensearch.keystore.tmp: Read-only file system

To reproduce

Use the opensearch 2.9.0 docker image in a kubernetes cluster where the securityContext sets the pod to read only filesystem.
mount /usr/share/opensearch/logs to emptyDir to allow it to add log files here
mount /tmp to emptyDir to allow it to write temp files

Expected behavior

Expected opensearch to run without issues

Screenshots

If applicable, add screenshots to help explain your problem.

Host / Environment

OpenShift kubernetes cluster

Additional context

Issue is very similar to opensearch-project/helm-charts#187 which was fixed in 2022

Relevant log output

WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.9.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
Exception in thread "main" org.opensearch.bootstrap.BootstrapException: java.nio.file.FileSystemException: /usr/share/opensearch/config/opensearch.keystore.tmp: Read-only file system
Likely root cause: java.nio.file.FileSystemException: /usr/share/opensearch/config/opensearch.keystore.tmp: Read-only file system
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
        at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:484)
        at java.base/java.nio.file.Files.newOutputStream(Files.java:228)
        at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:394)
        at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:387)
        at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:220)
        at org.apache.lucene.backward_codecs.store.EndiannessReverserUtil.createOutput(EndiannessReverserUtil.java:54)
        at org.opensearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:528)
        at org.opensearch.bootstrap.Bootstrap.loadSecureSettings(Bootstrap.java:276)
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:368)
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
        at org.opensearch.cli.Command.main(Command.java:101)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)

[gaiksaya]

Hi @jbnjohnathan

Does the solution mentioned by you here opensearch-project/helm-charts#369 (comment) solve this issue?

[jbnjohnathan]

opensearch-project/helm-charts#369 (comment)

No, as you see from my comment in that thread I get to the same error message. But I will reference this issue in that comment as well

[peterzhuamazon]

Adding @prudhvigodithi for more researching on this.
We need to write certain logs and changes to config for OpenSearch.
So not sure if a full-ledge readonly setup is going to run on docker for this.