Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix OOM issue on broker wait-zookeeper-ready initContainer #568

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

got-root
Copy link

@got-root got-root commented Feb 3, 2025

Motivation

Getting OOM issues on broker startup in the wait-zookeeper-ready initContainer:

pulsar cluster stellar-pulsar isn't initialized yet ... check in 3 seconds ...                                                                                                                                                         
[1.007s][error][gc] Failed to commit memory (Not enough space)                                                                                                                                                                         
[1.694s][error][gc] Failed to commit memory (Not enough space)                                                                                                                                                                         
[2.282s][error][gc] Failed to commit memory (Not enough space)                                                                                                                                                                         
[2.785s][error][gc] Failed to commit memory (Not enough space)                                                                                                                                                                         
[3.086s][error][gc] Failed to commit memory (Not enough space)                                                                                                                                                                         
[3.086s][error][gc] Forced to lower max Java heap size from 2048M(100%) to 1508M(74%)                                                                                                                                                  
[3.086s][error][gc] Failed to allocate initial Java heap (2048M)                                                                                                                                                                       
Error: Could not create the Java Virtual Machine.                                                                                                                                                                                      
Error: A fatal exception has occurred. Program will exit.                                                                                                                                                                              
pulsar cluster stellar-pulsar isn't initialized yet ... check in 3 seconds ...                                                                                                                                                         
[1.114s][error][gc] Failed to commit memory (Not enough space)                                                                                                                                                                         
[1.738s][error][gc] Failed to commit memory (Not enough space)                                                                                                                                                                         
[2.111s][error][gc] Failed to commit memory (Not enough space)                                                                                                                                                                         
[2.421s][error][gc] Failed to commit memory (Not enough space)                                                                                                                                                                         
[2.762s][error][gc] Failed to commit memory (Not enough space)                                                                                                                                                                         
[2.762s][error][gc] Forced to lower max Java heap size from 2048M(100%) to 1508M(74%)                                                                                                                                                  
[2.762s][error][gc] Failed to allocate initial Java heap (2048M)                                                                                                                                                                       
Error: Could not create the Java Virtual Machine.                                                                                                                                                                                      
Error: A fatal exception has occurred. Program will exit.                                                                                                                                                                              
pulsar cluster stellar-pulsar isn't initialized yet ... check in 3 seconds ...                                                                                                                                                         
[1.082s][error][gc] Failed to commit memory (Not enough space)                                                                                                                                                                         
[1.707s][error][gc] Failed to commit memory (Not enough space)                                                                                                                                                                         
[2.067s][error][gc] Failed to commit memory (Not enough space)                                                                                                                                                                         
[2.354s][error][gc] Failed to commit memory (Not enough space)                                                                                                                                                                         
[2.644s][error][gc] Failed to commit memory (Not enough space)                                                                                                                                                                         
[2.644s][error][gc] Forced to lower max Java heap size from 2048M(100%) to 1508M(74%)                                                                                                                                                  
[2.644s][error][gc] Failed to allocate initial Java heap (2048M)                                                                                                                                                                       
Error: Could not create the Java Virtual Machine.                                                                                                                                                                                      
Error: A fatal exception has occurred. Program will exit. 
Killed

Modifications

Looks like the check to see if zookeeper is running mistakenly has set the BOOKIE_MEM env var when it should be setting PULSAR_MEM instead.

Verifying this change

  • Make sure that the change passes the CI checks.
  • Tested on live env

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant