Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes #1723: For inter-router connectors, force set the dataConnectio… #1724

Closed
wants to merge 1 commit into from

Conversation

ganeshmurthy
Copy link
Contributor

…nCount to 1 if it is set to 0

… dataConnectionCount to 1 if it is set to 0
@@ -618,7 +618,7 @@
"required": false
},
"dataConnectionCount": {
"description": "The number of parallel data connections to carry streaming data between routers. Applies only to interior routers",
"description": "The number of parallel data connections to carry streaming data between routers. Applies only to interior routers. Defaults to 'auto', don't set to 0 (zero), if set to zero, will be forced to 1",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not something you changed but maybe this description should be more specific WRT the connection role. Like "Applies only to connectors with a role of "inter-router"??? Or something like that...

@kgiusti
Copy link
Contributor

kgiusti commented Jan 29, 2025

Stepping back a bit - I don't like how the data connection count is stored as a string then constantly re-computed every time a new connector is created. I'm not being critical about your patch - reviewing your patch leads me to believe the original implementation of the data connection count leaves room for improvement.

For example, the data connection count is stored in the router configuration record. It's value is initially read in dispatch.c::qd_dispatch_configure_router. That code simply stores it in a string. But then the actual computation of the data connection count integer is buried in the connection manager. That computation must be repeated every time a new connector is created.

This seems complicated and the logic is needlessly spread across files.

Wouldn't it be simpler to:

  1. Compute the actual value of data connection count in the qd_dispatch_configure_router routine where the configuration is read?
    1a) By "compute" I mean translate the string "auto" to the actual integer value based on the number of worker threads. The number of worker threads is also read from the same configuration record as the data connection count so it's immediately accessible.
  2. store the data connection count as an integer, not a string. We need an integer, not a string.

Then you can remove all the extra logic in connection_manager.c::qd_dispatch_configure_connector() for computing the integer value of data connection count and simply copy it from the qd parameter.

Then those access functions that go through the core (via the management thread - shudder) can be removed. Specifically connection_manager.c::auto_calc_connection_count and also qdr_core_get_worker_thread_count which is totally unnecessary since the thread count is available in the qd parameter.

I think that would greatly simplify the implementation of the data_connection_count configuration - what do you think?

@ganeshmurthy
Copy link
Contributor Author

what do you think?

I will embark on this mission and will keep you posted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants