Hotfix for the subprocess flushing issues

CHANGELOG.md

@@ -49,6 +49,29 @@ This is the current release cycle, so stay tuned for future releases!
 - **Add management buttons to pipes' cards.**  
   For your convenience, you may now sync, verify, clear, drop, and delete pipes directly from cards.
 
+- **Designate your packages as plugins with the `meerschaum.plugins` entry point.**  
+  You may now specify your existing packages as Meerschaum plugins by adding the `meerschaum.plugins` [Entrypoint](https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/#using-package-metadata) to your package metadata:
+
+  ```python
+  from setuptools import setup
+
+  setup(
+      ...,
+      entry_points = {
+          'meerschaum.plugins': [
+              'foo = foo',
+          ],
+      },
+  )
+  ```
+
+  or if you are using `pyproject.toml`:
+
+  ```toml
+  [project.entry-points."meerschaum.plugins"]
+  foo = "foo"
+  ```
+
 - **Pre- and post-sync hooks are printed separately.**  
   The results of sync hooks are now printed right after execution rather than after the sync.
 

docs/mkdocs/news/changelog.md

@@ -49,6 +49,29 @@ This is the current release cycle, so stay tuned for future releases!
 - **Add management buttons to pipes' cards.**  
   For your convenience, you may now sync, verify, clear, drop, and delete pipes directly from cards.
 
+- **Designate your packages as plugins with the `meerschaum.plugins` entry point.**  
+  You may now specify your existing packages as Meerschaum plugins by adding the `meerschaum.plugins` [Entrypoint](https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/#using-package-metadata) to your package metadata:
+
+  ```python
+  from setuptools import setup
+
+  setup(
+      ...,
+      entry_points = {
+          'meerschaum.plugins': [
+              'foo = foo',
+          ],
+      },
+  )
+  ```
+
+  or if you are using `pyproject.toml`:
+
+  ```toml
+  [project.entry-points."meerschaum.plugins"]
+  foo = "foo"
+  ```
+
 - **Pre- and post-sync hooks are printed separately.**  
   The results of sync hooks are now printed right after execution rather than after the sync.
 

docs/mkdocs/reference/plugins/writing-plugins.md

@@ -4,20 +4,9 @@
 
 Meerschaum's plugin system is designed to be simple so you can get your plugins working quickly. Plugins are Python packages defined in the Meerschaum configuration `plugins` directory and are imported at startup under the global namespace `plugins`.
 
-!!! warning "Performance Warning"
-    To get the best performance and user experience, try to keep module-level code to a minimum ― especially heavy imports. Plugins are loaded at startup, and those tiny delays add up!
-    ```python hl_lines="2"
-    ### BAD - DON'T DO THIS
-    import pandas as pd
-    def fetch(pipe, **kw):
-    	  return pd.read_csv('data.csv')
-    ```
-    ``` python hl_lines="3"
-    ### GOOD - DO THIS INSTEAD
-    def fetch(pipe, **kw):
-    	  import pandas as pd
-    	  return pd.read_csv('data.csv')
-    ```
+!!! info inline end ""
+    Looking to make your existing package a plugin? See [Package Management](#package-management) below.
+
 
 To create your plugin, follow these steps:
 
@@ -73,6 +62,22 @@ To create your plugin, follow these steps:
 
         You may find the supplied arguments useful for your implementation (e.g. `begin` and `end` `#!python datetime.datetime` objects). Run `mrsm -h` or `mrsm show help` to see the available keyword arguments.
 
+!!! info "Imports impact performance"
+    For optimal performance, keep module-level code to a minimum ― especially heavy imports.
+    ```python hl_lines="2"
+    ### BAD - DON'T DO THIS
+    import pandas as pd
+    def fetch(pipe, **kw):
+        return pd.read_csv('data.csv')
+    ```
+    ```python hl_lines="3"
+    ### GOOD - DO THIS INSTEAD
+    def fetch(pipe, **kw):
+        import pandas as pd
+        return pd.read_csv('data.csv')
+    ```
+
+
 ## Functions
 
 Plugins are just modules with functions. This section explains the roles of the following special functions:
@@ -607,6 +612,34 @@ Plugins are just Python modules, so you can write custom code and share it among
 
 At run time, plugins are imported under the global `plugins` namespace, but you'll probably be testing plugins directly when the `plugins` namespace isn't created. That's where [`Plugin` objects](https://docs.meerschaum.io/#meerschaum.Plugin) come in handy: they contain a number of convenience functions so you can cross-pollinate between plugins.
 
+### Package Management
+
+Meerschaum plugins only need to be modules ― no package metadata required. But if you plan on managing your plugins as proper packages (e.g. to publish to repositories like [PyPI](https://pypi.org/)), simply add the [entry point](https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/#using-package-metadata) `meerschaum.plugins` to your package metadata:
+
+??? note "Creating Plugins from a Package Entry Point"
+
+    === "`setup.py`"
+    
+        ```python
+        from setuptools import setup
+        
+        setup(
+            ...,
+            entry_points = {
+                'meerschaum.plugins': [
+                    'foo = foo',
+                ],
+            },
+        )
+        ```
+
+    === "`pyproject.toml`"
+    
+        ```toml
+        [project.entry-points."meerschaum.plugins"]
+        foo = "foo"
+        ```
+
 ### Import Another Plugin
 
 Accessing the member `module` of the `Plugin` object will import its module:

meerschaum/api/__init__.py

@@ -30,21 +30,26 @@
 CHECK_UPDATE = os.environ.get(STATIC_CONFIG['environment']['runtime'], None) != 'docker'
 
 endpoints = STATIC_CONFIG['api']['endpoints']
-aiofiles = attempt_import('aiofiles', lazy=False, check_update=CHECK_UPDATE)
+
+(
+    fastapi,
+    aiofiles,
+    starlette_responses,
+    multipart,
+    packaging_version,
+) = attempt_import(
+    'fastapi',
+    'aiofiles',
+    'starlette.responses',
+    'multipart',
+    'packaging.version',
+    lazy = False,
+    check_update = CHECK_UPDATE,
+)
 typing_extensions = attempt_import(
     'typing_extensions', lazy=False, check_update=CHECK_UPDATE,
     venv = None,
 )
-pydantic_dataclasses = attempt_import(
-    'pydantic.dataclasses', lazy=False, check_update=CHECK_UPDATE,
-)
-fastapi = attempt_import('fastapi', lazy=False, check_update=CHECK_UPDATE)
-starlette_reponses = attempt_import(
-    'starlette.responses', warn=False, lazy=False,
-    check_update=CHECK_UPDATE,
-)
-python_multipart = attempt_import('multipart', lazy=False, check_update=CHECK_UPDATE)
-packaging_version = attempt_import('packaging.version', check_update=CHECK_UPDATE)
 from meerschaum.api._chain import check_allow_chaining, DISALLOW_CHAINING_MESSAGE
 uvicorn_config_path = API_UVICORN_RESOURCES_PATH / SERVER_ID / 'config.json'
 

meerschaum/config/_jobs.py

@@ -18,7 +18,7 @@
         'max_file_size': 100_000,
         'lines_to_show': 30,
         'refresh_files_seconds': 5,
-        'min_buffer_len': 10,
+        'min_buffer_len': 5,
         'timestamp_format': '%Y-%m-%d %H:%M',
         'follow_timestamp_format': '%H:%M',
         'colors': [

meerschaum/config/_version.py

@@ -2,4 +2,4 @@
 Specify the Meerschaum release version.
 """
 
-__version__ = "2.2.0rc3"
+__version__ = "2.2.0"

meerschaum/config/static/__init__.py

@@ -60,6 +60,7 @@
         'gid': 'MRSM_GID',
         'noask': 'MRSM_NOASK',
         'id': 'MRSM_SERVER_ID',
+        'daemon_id': 'MRSM_DAEMON_ID',
         'uri_regex': r'MRSM_([a-zA-Z0-9]*)_(\d*[a-zA-Z][a-zA-Z0-9-_+]*$)',
         'prefix': 'MRSM_',
     },

meerschaum/utils/daemon/Daemon.py

@@ -19,6 +19,7 @@
 from datetime import datetime, timezone
 from meerschaum.utils.typing import Optional, Dict, Any, SuccessTuple, Callable, List, Union
 from meerschaum.config import get_config
+from meerschaum.config.static import STATIC_CONFIG
 from meerschaum.config._paths import DAEMON_RESOURCES_PATH, LOGS_RESOURCES_PATH
 from meerschaum.config._patch import apply_patch_to_config
 from meerschaum.utils.warnings import warn, error
@@ -170,9 +171,11 @@ def _run_exit(
             log_refresh_seconds,
             partial(self.rotating_log.refresh_files, start_interception=True),
         )
+
         try:
             os.environ['LINES'], os.environ['COLUMNS'] = str(int(lines)), str(int(columns))
             with self._daemon_context:
+                os.environ[STATIC_CONFIG['environment']['daemon_id']] = self.daemon_id
                 self.rotating_log.refresh_files(start_interception=True)
                 try:
                     with open(self.pid_path, 'w+', encoding='utf-8') as f:
@@ -501,7 +504,7 @@ def _handle_sigterm(self, signal_number: int, stack_frame: 'frame') -> None:
             daemon_context.close()
 
         _close_pools()
-        raise SystemExit(1)
+        raise SystemExit(0)
 
 
     def _send_signal(

meerschaum/utils/packages/__init__.py

@@ -829,8 +829,11 @@ def pip_install(
                     check_wheel = False, debug = debug,
                 ):
                     warn(
-                        f"Failed to install `setuptools` and `wheel` for virtual environment '{venv}'.",
-                        color=False,
+                        (
+                            "Failed to install `setuptools` and `wheel` for virtual "
+                            + f"environment '{venv}'."
+                        ),
+                        color = False,
                     )
 
         if requirements_file_path is not None:
@@ -893,13 +896,16 @@ def pip_install(
                         f"Failed to clean up package '{_install_no_version}'.",
                     )
 
-        success = run_python_package(
+        rc = run_python_package(
             'pip',
             _args + _packages,
             venv = venv,
             env = _get_pip_os_env(),
             debug = debug,
-        ) == 0
+        )
+        if debug:
+            print(f"{rc=}")
+        success = rc == 0
 
     msg = (
         "Successfully " + ('un' if _uninstall else '') + "installed packages." if success 

meerschaum/utils/packages/_packages.py

@@ -142,14 +142,13 @@
     'tornado'                        : 'tornado>=6.1.0',
 }
 packages['api'] = {
-    'uvicorn'                        : 'uvicorn[standard]>=0.22.0',
-    'gunicorn'                       : 'gunicorn>=20.1.0',
+    'uvicorn'                        : 'uvicorn[standard]>=0.29.0',
+    'gunicorn'                       : 'gunicorn>=22.0.0',
     'dotenv'                         : 'python-dotenv>=0.20.0',
     'websockets'                     : 'websockets>=11.0.3',
-    'fastapi'                        : 'fastapi>=0.100.0',
-    'passlib'                        : 'passlib>=1.7.4',
+    'fastapi'                        : 'fastapi>=0.111.0',
     'fastapi_login'                  : 'fastapi-login>=1.7.2',
-    'multipart'                      : 'python-multipart>=0.0.5',
+    'multipart'                      : 'python-multipart>=0.0.9',
     'httpx'                          : 'httpx>=0.24.1',
     'websockets'                     : 'websockets>=11.0.3',
 }

meerschaum/utils/process.py

@@ -11,6 +11,7 @@
 from __future__ import annotations
 import os, signal, subprocess, sys, platform
 from meerschaum.utils.typing import Union, Optional, Any, Callable, Dict, Tuple
+from meerschaum.config.static import STATIC_CONFIG
 
 def run_process(
         *args,
@@ -68,9 +69,18 @@ def run_process(
     if platform.system() == 'Windows':
         foreground = False
 
-    if line_callback is not None:
+    def print_line(line):
+        sys.stdout.write(line.decode('utf-8'))
+        sys.stdout.flush()
+
+    if capture_output or line_callback is not None:
+        kw['stdout'] = subprocess.PIPE
+        kw['stderr'] = subprocess.STDOUT
+    elif os.environ.get(STATIC_CONFIG['environment']['daemon_id']):
         kw['stdout'] = subprocess.PIPE
         kw['stderr'] = subprocess.STDOUT
+        if line_callback is None:
+            line_callback = print_line
 
     if 'env' not in kw:
         kw['env'] = os.environ
@@ -112,15 +122,6 @@ def new_pgid():
         kw['preexec_fn'] = new_pgid
 
     try:
-        # fork the child
-        #  stdout, stderr = (
-            #  (sys.stdout, sys.stderr) if not capture_output
-            #  else (subprocess.PIPE, subprocess.PIPE)
-        #  )
-        if capture_output:
-            kw['stdout'] = subprocess.PIPE
-            kw['stderr'] = subprocess.PIPE
-
         child = subprocess.Popen(*args, **kw)
 
         # we can't set the process group id from the parent since the child
@@ -197,6 +198,8 @@ def timeout_handler():
     while proc.poll() is None:
         line = proc.stdout.readline()
         line_callback(line)
+
     if timeout_seconds is not None:
         watchdog_thread.cancel()
+
     return proc.poll()

requirements/api.txt

@@ -1,11 +1,10 @@
-uvicorn[standard]>=0.22.0
-gunicorn>=20.1.0
+uvicorn[standard]>=0.29.0
+gunicorn>=22.0.0
 python-dotenv>=0.20.0
 websockets>=11.0.3
-fastapi>=0.100.0
-passlib>=1.7.4
+fastapi>=0.111.0
 fastapi-login>=1.7.2
-python-multipart>=0.0.5
+python-multipart>=0.0.9
 httpx>=0.24.1
 numpy>=1.18.5
 pandas[parquet]>=2.0.1

requirements/full.txt

@@ -56,12 +56,11 @@ dash-extensions>=1.0.4
 dash-daq>=0.5.0
 terminado>=0.12.1
 tornado>=6.1.0
-uvicorn[standard]>=0.22.0
-gunicorn>=20.1.0
+uvicorn[standard]>=0.29.0
+gunicorn>=22.0.0
 python-dotenv>=0.20.0
 websockets>=11.0.3
-fastapi>=0.100.0
-passlib>=1.7.4
+fastapi>=0.111.0
 fastapi-login>=1.7.2
-python-multipart>=0.0.5
+python-multipart>=0.0.9
 httpx>=0.24.1