Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

XNAT service failing to start after kube host reboot #103

Open
markendr opened this issue Aug 22, 2022 · 1 comment
Open

XNAT service failing to start after kube host reboot #103

markendr opened this issue Aug 22, 2022 · 1 comment
Assignees

Comments

@markendr
Copy link
Contributor

markendr commented Aug 22, 2022

Describe the bug

If the kubernetes host is rebooted the XNAT service fails to start up. The tomcat log on the xnat-web pod shows database connection errors, indicating that XNAT/tomcat has attempted to start before the postgresql service is available.

Steps to reproduce

  1. reboot the kube host, eg. systemctl reboot
  2. wait around 2 minutes for the host and kubernetes to start
  3. kubectl get po shows xnat-web running but not ready
  4. confirm tomcat log on xnat-web show database connection errors
  5. kill the tomcat process on the xnat-web pod to relaunch
  6. wait around 90 seconds for xnat-web pod to successfully restart

Step 2 can be delayed 20 mins or more but xnat-web will not recover

Expected behavior

if database is not available, xnat-pod can recover once database becomes available

XNAT Chart Version

1.1.4

Version of Helm

3.8

Version of Kubernetes

v1.24.3

Additional context

No response

@fxusyd fxusyd self-assigned this Aug 23, 2022
@markendr
Copy link
Contributor Author

markendr commented Sep 6, 2022

Tried liveness and startup probe with various failureThreshold settings and all working as expected. Can the issue#103 branch be merged into main?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants