Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable JSON encoding for interprocess communication #301

Merged
merged 16 commits into from
Apr 1, 2024

Conversation

dmgav
Copy link
Contributor

@dmgav dmgav commented Mar 31, 2024

Queue Server is using JSON RPC protocol for communication between the manager process and other processes. In this PR the communication code is modified so that the messages are not encoded as JSON string by the process sending the message and the decoded by the receiving process. Conversion data (e.g. a large array) to and from a JSON string is slow and completely unnecessary for interprocess communication within one application. Instead, the JSON RPC message is sent as a dictionary via multiprocessing.Pipe, which automatically pickles the outgoing message and then unpickles it at the receiving end. The change does not affect user-facing API, but is expected to make Queue Server more stable due to shorter interprocess communication delays.

Other changes: json-rpc package is replaced with custom implementation of JSON RPC handler, which supports the option to enable/disable JSON encoding (json-rpc is no longer a dependency); new parameter use_json in the constructors of PipeJsonRpcReceive and PipeJsonRpcSendAsync classes; better processing of SIGTERM by the watchdog process, which is now reliably closes the manager and the worker processes (e.g. if kill <pid> is called on the watchdog process, it does not work with kill -9 <pid>)

@dmgav dmgav merged commit 838487f into bluesky:main Apr 1, 2024
35 checks passed
@dmgav dmgav deleted the my-rpc branch April 1, 2024 02:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant