Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug(pg): Demonstrating (or trying to) bug with many crons. #102

Closed
wants to merge 4 commits into from

Conversation

elliotcourant
Copy link
Collaborator

@elliotcourant elliotcourant commented Nov 8, 2023

This demonstrates an issue where it is easily possible to have more jobs or job handlers than the connection pool will allow. In tests the max conns defaults to 2 but otherwise pgxconn defaults to runtime.NumCPU.

I'm running into an issue where I'm running several small containers with sometimes less than a single CPU core each (since I don't need much). But because of how the connection pool works this ends up causing deadlocks since the acquire function calls themselves do not have timeouts (since ctx is just cascaded throughout).

More details can be seen here about my specific use case: monetr/monetr#1608


I guess overall, does it make sense to have a connection pool that has an arbitrary hard limit on connections? Even if someone were to just configure a higher limit (or maybe a significantly higher limit); most of the connections will be consumed not by jobs themselves but by the listeners associated with those jobs? Which means that you must have at least N max conns for N jobs?

The other thing would be how ctx is cascaded throughout, if I call Start or StartCron I cannot pass a WithTimeout context because that will cause actual parts of the handler or listener to timeout (like listening for notifications which is async anyway) where I might want to specify "the setup of this job should only take 10 seconds"?
I think the approach with how timeouts should happen with async work happening in the background vs work that is synchronous initiated by a caller should be thought about as part of this as well.

@acaloiaro
Copy link
Owner

Opened #103 for this.

@acaloiaro
Copy link
Owner

Fixed by #103

@acaloiaro acaloiaro closed this Jan 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants