DM-48885: Add support for configuring number of messages to be consumed and timeout in salobj Remote/SalInfo.

.github/workflows/enforce_rebase.yaml

@@ -0,0 +1,9 @@
+name: enforce_rebase
+
+on: push
+
+jobs:
+  call-workflow:
+    uses: lsst-ts/tssw_workflows/.github/workflows/enforce_rebase.yaml@main
+
+

.github/workflows/news_creation.yaml

@@ -0,0 +1,8 @@
+name: news_creation
+
+on: push
+
+jobs:
+  call-workflow:
+    uses: lsst-ts/tssw_workflows/.github/workflows/news_creation.yaml@main
+

.ts_pre_commit_config.yaml

@@ -1,6 +1,10 @@
-check-yaml: true
-check-xml: true
 black: true
+check-xml: true
+check-yaml: true
+clang-format: false
 flake8: true
+format-xmllint: false
 isort: true
 mypy: true
+ruff: false
+towncrier: true

doc/index.rst

@@ -54,6 +54,20 @@ Examples:
   In particular, it does not output telemetry at regular intervals.
 * `TestScript` is an example of a trivial :ref:`Python SAL Script<lsst.ts.salobj_python_sal_scripts>`.
 
+.. _lsst.ts.salobj-fine-tune-read-loop:
+
+Fine Tunning the Read Loop
+--------------------------
+
+It is possible to fine tune the read loop by a combination of two arguments in the `Remote` class; `num_messages` and `consume_messages_timeout`.
+By default these parameters are set to work for the majority of use cases, e.g.; normal CSC and SAL Script operations.
+However, in some applications, it might be necessary to tune the read loop to improve handling of high throughput topics.
+
+If dealing with a high volume of data one might want to increase `num_messages` and reduce `consume_messages_timeout`.
+This allows the read loop to batch process a higher number of messages at each loop.
+However, when increasing the number of messages processed at every loop, this might increase the read latency as the loop only processes the messages if it received the number of messages requested or `consume_messages_timeout` has passed.
+When trying to optimize the read loop it is import to test a combination of values for these parameters.
+
 .. _lsst.ts.salobj-cleanup:
 
 Cleanup

doc/news/DM-48885.perf.1.rst

@@ -0,0 +1 @@
+Updated ``Remote`` to allow users to specify ``num_messages`` and ``consume_messages_timeout``, which is passed on to ``SalInfo`` and used to set how the read loop behaves.

doc/news/DM-48885.perf.rst

@@ -0,0 +1,2 @@
+Updated ``SalInfo`` to replace the use of ``Consumer.pool`` with ``Consumer.consume`` and allow users to specify the number of images to consume and the timeout.
+By default, ``num_messages=1`` and ``timeout=0.1``, which falls back to the original behavior with ``Consumer.pool``.

doc/news/README.rst

@@ -0,0 +1,40 @@
+Recording Changes
+=================
+
+This directory contains "news fragments" which are small, structured text files that contain information about changes or updates that will be included in the release notes.
+These fragments are used to automatically generate changelogs or release notes.
+They can be written restructured text format or plain text.
+
+Each file should be named like ``<JIRA TICKET>.<TYPE>.<EXT>`` with a file extension defining the markup format (``rst|md``).
+The ``<TYPE>`` should be one of:
+
+* ``feature``: A new feature
+* ``bugfix``: A bug fix.
+* ``perf``: A performance enhancement.
+* ``doc``: A documentation improvement.
+* ``removal``: A deprecation or removal of API.
+* ``misc``: Other minor changes and/or additions
+
+An example file name would therefore look like ``DM-40534.doc.rst``.
+
+Each developer now has to create the news fragments for the changes they have made on their own branches,
+instead of adding them to the release notes directly.
+The news fragments are then automatically integrated into the release notes by the ``towncrier`` tool.
+
+You can test how the content will be integrated into the release notes by running ``towncrier build --draft --version=v<X.XX.X>``.
+Note that you have to run it from the root repository directory (i.e. the ``ts_salobj``).
+
+In order to update the release notes file for real, the person responsible for the releasing the notes should run:
+
+.. code-block:: bash
+
+   $ towncrier build --version=v<X.XX.X>
+
+
+.. note::
+
+   When running towncrier to build the changelog, you may be prompted to confirm the deletion of fragments.
+   If you would like to retain the fragments in the doc/news directory do not confirm the deletion.
+
+Note also that ``towncrier`` can be installed from PyPI or conda-forge.
+

doc/version_history.rst

@@ -6,6 +6,8 @@
 Version History
 ###############
 
+.. towncrier release notes start
+
 v8.0.0
 ------
 

python/lsst/ts/salobj/remote.py

@@ -74,6 +74,10 @@ class Remote:
         and call `start` manually after you have added all topics.
         Warning: if `False` then `self.start_task` will not exist
         and the remote cannot be used as an async context manager.
+    num_messages : `int`
+        Number of messages to consume in the read loop.
+    consume_messages_timeout : `float`
+        Timeout to wait for new messages to arrive in the read loop.
 
     Raises
     ------
@@ -144,6 +148,8 @@ def __init__(
         exclude: Iterable[str] | None = None,
         evt_max_history: int = 1,
         start: bool = True,
+        num_messages: int = 1,
+        consume_messages_timeout: float = 0.1,
     ) -> None:
         if include is not None and exclude is not None:
             raise ValueError("Cannot specify both include and exclude")
@@ -153,7 +159,13 @@ def __init__(
         if not isinstance(domain, Domain):
             raise TypeError(f"domain {domain!r} must be an lsst.ts.salobj.Domain")
 
-        self.salinfo = SalInfo(domain=domain, name=name, index=index)
+        self.salinfo = SalInfo(
+            domain=domain,
+            name=name,
+            index=index,
+            num_messages=num_messages,
+            consume_messages_timeout=consume_messages_timeout,
+        )
         try:
             if not readonly:
                 for cmd_name in self.salinfo.command_names: