⚡️ v2.1.0 Upgrade webterm to xterm.js.

ACKNOWLEDGEMENTS.md

@@ -333,10 +333,12 @@ I've included a list of the dynamic dependencies below to credit the open source
 
 ### pydantic
 
-*Data validation using Python type hints*
-
-  - [Samuel Colvin <[email protected]>, Eric Jolibois <[email protected]>, Hasan Ramezani <[email protected]>, Adrian Garcia Badaracco <[email protected]>, Terrence Dorsey <[email protected]>, David Montague <[email protected]>](mailto:Samuel Colvin <[email protected]>, Eric Jolibois <[email protected]>, Hasan Ramezani <[email protected]>, Adrian Garcia Badaracco <[email protected]>, Terrence Dorsey <[email protected]>, David Montague <[email protected]>)
+*Data validation and settings management using python type hints*
 
+  - Samuel Colvin
+  - [[email protected]](mailto:[email protected])
+  - [https://github.com/pydantic/pydantic](https://github.com/pydantic/pydantic)
+  - License: MIT
 
 ### Pygments
 
@@ -505,13 +507,6 @@ I've included a list of the dynamic dependencies below to credit the open source
   - [http://www.tornadoweb.org/](http://www.tornadoweb.org/)
   - License: Apache-2.0
 
-### tornado_xstatic
-
-*Utilities for using XStatic in Tornado applications*
-
-  - [Thomas Kluyver <[email protected]>](mailto:Thomas Kluyver <[email protected]>)
-
-
 ### typing_extensions
 
 *Backported and Experimental Type Hints for Python 3.8+*
@@ -563,19 +558,3 @@ I've included a list of the dynamic dependencies below to credit the open source
 
   - [Daniel Holth <[email protected]>](mailto:Daniel Holth <[email protected]>)
 
-
-### XStatic
-
-*XStatic base package with minimal support code*
-
-  - Thomas Waldmann
-  - [[email protected]](mailto:[email protected])
-  - [https://github.com/xstatic-py/xstatic](https://github.com/xstatic-py/xstatic)
-  - License: MIT license
-
-### XStatic-term.js
-
-*term.js 0.0.7 (XStatic packaging standard)*
-
-  - [https://github.com/chjj/term.js](https://github.com/chjj/term.js)
-  - License: (same as term.js)

CHANGELOG.md

@@ -1,10 +1,101 @@
 # 🪵 Changelog
 
-## 2.0.x Releases
+## 2.1.x Releases
 
 This is the current release cycle, so stay tuned for future releases!
 
-### v2.0.8
+### v2.1.0
+
+- **Replace `term.js` with `xterm.js`.**  
+  This has been a long time coming. The webterm has been migrated to `xterm.js` which has continuous support from `term.js` which was last updated almost 10 years ago.
+
+- **Deprecate the legacy web pseudo-terminal.**  
+  Clicking the "Execute" button on the web console will now execute the command directly in the webterm. Additionally, changing the instance select will now automatically switch the webterm's context to the desired instance.
+
+- **Fix an issue when starting existing jobs.**  
+  A bug has been fixed which prevented jobs from restarting specifically by name.
+
+- **Add `MRSM_VENVS_DIR`.**  
+  Like `MRSM_PLUGINS_DIR`, you can now designate a virtual environments directory separate from the root directory. This is particularly useful for production deployments, and `MRSM_VENVS_DIR` has been set to `/home/meerschaum/venvs` in the official Docker images to allow for mounting `/meerschaum` to persistent volumes.
+
+- **Allow syncing `NULL` values into indices.**  
+  Syncing `None` within an index will now be coalesced into a magic value when applying updates.
+
+  ```python
+  import meerschaum as mrsm
+
+  pipe = mrsm.Pipe(
+      'allow', 'null', 'indices',
+      instance = 'sql:local',
+      columns = ['a', 'b'],
+  )
+  pipe.sync([{'a': 1, 'b': 1}])
+  pipe.sync([{'b': 1}])
+  pipe.sync([{'a': 1}])
+  pipe.sync([{'c': 1}])
+
+  print(pipe.get_data())
+  #       a     b     c
+  # 0  <NA>  <NA>     1
+  # 1  <NA>     1  <NA>
+  # 2     1  <NA>  <NA>
+  # 3     1     1  <NA>
+  ```
+
+- **Syncing `Decimal` objects will now enforce `numeric` dtypes.**  
+  For example, syncing a `Decimal` onto a integer column will update the dtype to `numeric`, like when syncing a float after an integer.
+
+  ```python
+  import meerschaum as mrsm
+  from decimal import Decimal
+
+  pipe = mrsm.Pipe(
+      'demo', 'decimal', 'coersion',
+      instance = 'sql:local',
+      columns = ['id'],
+  )
+  pipe.sync([{'id': 1, 'foo': 10}])
+  pipe.sync([{'id': 1, 'foo': Decimal('20')}])
+  print(pipe.dtypes)
+  # {'id': 'int64[pyarrow]', 'foo': 'numeric'}
+
+  df = pipe.get_data()
+  print(f"{df['foo'][0]=}")
+  # df['foo'][0]=Decimal('20') 
+  ```
+
+- **Improve `IS NULL` and `IS NOT NULL` checks for `params`.**  
+  Mixing null-like values (e.g. `NaN`, `<NA>`, `None`) in `params` will now separate out nulls.
+
+  ```python
+  from meerschaum.utils.sql import build_where
+  print(build_where({'a': ['_<NA>', '_1', '_2']}))
+  # WHERE
+  #   ("a" NOT IN ('1', '2')
+  #   AND "a" IS NOT NULL)
+  print(build_where({'a': ['NaN', '1', '2']}))
+  # WHERE
+  #   ("a" IN ('1', '2')
+  #   OR "a" IS NULL)
+  ```
+
+- **Add colors to `mrsm show columns`.**
+
+- **Fix a unicode decoding error when showing logs.**
+
+- **Remove `xstatic` dependencies.**  
+  The `xterm.js` files are now bundled as static assets, so the `term.js` files are no longer needed. Hurray for removing dependencies!
+
+- **Other bugfixes.**  
+  A handful of minor bugfixes have been included in this release:
+    - Removed non-connector environment variables like `MRSM_WORK_DIR` from the `mrsm show connectors` output.
+    - Improving symlinks handling for multi-processed situations (`mrsm start api`).
+
+## 2.0.x Releases
+
+At long last, 2.0 has arrived! The 2.0 releases brought incredible change, from standardizing chunking to adding `Pipe.verify()` and `Pipe.deduplicate()` to introducing first-class `numeric` support. See the full release notes below for the complete picture.
+
+### v2.0.8 – v2.0.9
 
 - **Cast `None` to `Decimal('NaN')` for `numeric` columns.**  
   To allow for all-null numeric columns, `None` (and other null-like types) are coerced to `Decimal('NaN')`.

Dockerfile

@@ -9,6 +9,7 @@ ENV MRSM_USER=$mrsm_user \
     MRSM_USER=meerschaum \
     MRSM_ROOT_DIR=$mrsm_root_dir \
     MRSM_WORK_DIR=$mrsm_root_dir \
+    MRSM_VENVS_DIR=/home/meerschaum/venvs \
     MRSM_RUNTIME=docker \
     MRSM_HOME=/home/meerschaum \
     MRSM_SRC=/home/meerschaum/src \

docs/mkdocs/get-started/start-api.md

@@ -28,12 +28,12 @@ You can use Meerschaum from its web interface. From within a `mrsm` shell, type
 
 3. On the left side of the dashboard are the available commands, like in the `mrsm` shell.
 
-    <img src="/assets/screenshots/dash.png"/>
+    <img src="/assets/screenshots/dash-bootstrap.png"/>
 
     !!! tip "Adding custom commands"
         You can add your own actions with [Meerschaum plugins](/reference/plugins/writing-plugins/#action-plugins).
 
-    <img src="/assets/screenshots/dash-statistics.png"/>
+    <img src="/assets/screenshots/dash-pipe-statistics.png"/>
 
     Also like the `mrsm` shell, the web console lets you control pipes from several instances (note the instances drop-down on the top right).
 

docs/mkdocs/index.md

@@ -228,8 +228,8 @@ h1 {
               url = f"https://api.weather.gov/stations/{station}/observations"
               response = self.session.get(url, params=params)
               yield [
-                feature['properties']
-                for feature in response.json()['features']
+                  feature['properties']
+                  for feature in response.json()['features']
               ]
 
       @property

docs/mkdocs/news/acknowledgements.md

@@ -333,10 +333,12 @@ I've included a list of the dynamic dependencies below to credit the open source
 
 ### pydantic
 
-*Data validation using Python type hints*
-
-  - [Samuel Colvin <[email protected]>, Eric Jolibois <[email protected]>, Hasan Ramezani <[email protected]>, Adrian Garcia Badaracco <[email protected]>, Terrence Dorsey <[email protected]>, David Montague <[email protected]>](mailto:Samuel Colvin <[email protected]>, Eric Jolibois <[email protected]>, Hasan Ramezani <[email protected]>, Adrian Garcia Badaracco <[email protected]>, Terrence Dorsey <[email protected]>, David Montague <[email protected]>)
+*Data validation and settings management using python type hints*
 
+  - Samuel Colvin
+  - [[email protected]](mailto:[email protected])
+  - [https://github.com/pydantic/pydantic](https://github.com/pydantic/pydantic)
+  - License: MIT
 
 ### Pygments
 
@@ -505,13 +507,6 @@ I've included a list of the dynamic dependencies below to credit the open source
   - [http://www.tornadoweb.org/](http://www.tornadoweb.org/)
   - License: Apache-2.0
 
-### tornado_xstatic
-
-*Utilities for using XStatic in Tornado applications*
-
-  - [Thomas Kluyver <[email protected]>](mailto:Thomas Kluyver <[email protected]>)
-
-
 ### typing_extensions
 
 *Backported and Experimental Type Hints for Python 3.8+*
@@ -563,19 +558,3 @@ I've included a list of the dynamic dependencies below to credit the open source
 
   - [Daniel Holth <[email protected]>](mailto:Daniel Holth <[email protected]>)
 
-
-### XStatic
-
-*XStatic base package with minimal support code*
-
-  - Thomas Waldmann
-  - [[email protected]](mailto:[email protected])
-  - [https://github.com/xstatic-py/xstatic](https://github.com/xstatic-py/xstatic)
-  - License: MIT license
-
-### XStatic-term.js
-
-*term.js 0.0.7 (XStatic packaging standard)*
-
-  - [https://github.com/chjj/term.js](https://github.com/chjj/term.js)
-  - License: (same as term.js)

docs/mkdocs/news/changelog.md

@@ -1,10 +1,101 @@
 # 🪵 Changelog
 
-## 2.0.x Releases
+## 2.1.x Releases
 
 This is the current release cycle, so stay tuned for future releases!
 
-### v2.0.8
+### v2.1.0
+
+- **Replace `term.js` with `xterm.js`.**  
+  This has been a long time coming. The webterm has been migrated to `xterm.js` which has continuous support from `term.js` which was last updated almost 10 years ago.
+
+- **Deprecate the legacy web pseudo-terminal.**  
+  Clicking the "Execute" button on the web console will now execute the command directly in the webterm. Additionally, changing the instance select will now automatically switch the webterm's context to the desired instance.
+
+- **Fix an issue when starting existing jobs.**  
+  A bug has been fixed which prevented jobs from restarting specifically by name.
+
+- **Add `MRSM_VENVS_DIR`.**  
+  Like `MRSM_PLUGINS_DIR`, you can now designate a virtual environments directory separate from the root directory. This is particularly useful for production deployments, and `MRSM_VENVS_DIR` has been set to `/home/meerschaum/venvs` in the official Docker images to allow for mounting `/meerschaum` to persistent volumes.
+
+- **Allow syncing `NULL` values into indices.**  
+  Syncing `None` within an index will now be coalesced into a magic value when applying updates.
+
+  ```python
+  import meerschaum as mrsm
+
+  pipe = mrsm.Pipe(
+      'allow', 'null', 'indices',
+      instance = 'sql:local',
+      columns = ['a', 'b'],
+  )
+  pipe.sync([{'a': 1, 'b': 1}])
+  pipe.sync([{'b': 1}])
+  pipe.sync([{'a': 1}])
+  pipe.sync([{'c': 1}])
+
+  print(pipe.get_data())
+  #       a     b     c
+  # 0  <NA>  <NA>     1
+  # 1  <NA>     1  <NA>
+  # 2     1  <NA>  <NA>
+  # 3     1     1  <NA>
+  ```
+
+- **Syncing `Decimal` objects will now enforce `numeric` dtypes.**  
+  For example, syncing a `Decimal` onto a integer column will update the dtype to `numeric`, like when syncing a float after an integer.
+
+  ```python
+  import meerschaum as mrsm
+  from decimal import Decimal
+
+  pipe = mrsm.Pipe(
+      'demo', 'decimal', 'coersion',
+      instance = 'sql:local',
+      columns = ['id'],
+  )
+  pipe.sync([{'id': 1, 'foo': 10}])
+  pipe.sync([{'id': 1, 'foo': Decimal('20')}])
+  print(pipe.dtypes)
+  # {'id': 'int64[pyarrow]', 'foo': 'numeric'}
+
+  df = pipe.get_data()
+  print(f"{df['foo'][0]=}")
+  # df['foo'][0]=Decimal('20') 
+  ```
+
+- **Improve `IS NULL` and `IS NOT NULL` checks for `params`.**  
+  Mixing null-like values (e.g. `NaN`, `<NA>`, `None`) in `params` will now separate out nulls.
+
+  ```python
+  from meerschaum.utils.sql import build_where
+  print(build_where({'a': ['_<NA>', '_1', '_2']}))
+  # WHERE
+  #   ("a" NOT IN ('1', '2')
+  #   AND "a" IS NOT NULL)
+  print(build_where({'a': ['NaN', '1', '2']}))
+  # WHERE
+  #   ("a" IN ('1', '2')
+  #   OR "a" IS NULL)
+  ```
+
+- **Add colors to `mrsm show columns`.**
+
+- **Fix a unicode decoding error when showing logs.**
+
+- **Remove `xstatic` dependencies.**  
+  The `xterm.js` files are now bundled as static assets, so the `term.js` files are no longer needed. Hurray for removing dependencies!
+
+- **Other bugfixes.**  
+  A handful of minor bugfixes have been included in this release:
+    - Removed non-connector environment variables like `MRSM_WORK_DIR` from the `mrsm show connectors` output.
+    - Improving symlinks handling for multi-processed situations (`mrsm start api`).
+
+## 2.0.x Releases
+
+At long last, 2.0 has arrived! The 2.0 releases brought incredible change, from standardizing chunking to adding `Pipe.verify()` and `Pipe.deduplicate()` to introducing first-class `numeric` support. See the full release notes below for the complete picture.
+
+### v2.0.8 – v2.0.9
 
 - **Cast `None` to `Decimal('NaN')` for `numeric` columns.**  
   To allow for all-null numeric columns, `None` (and other null-like types) are coerced to `Decimal('NaN')`.

docs/mkdocs/reference/environment.md

@@ -50,7 +50,7 @@ export MRSM_PLUGINS_DIR='[
 You can temporarily register new connectors in a variable in the form `MRSM_<TYPE>_<LABEL>`, where `<TYPE>` is either `SQL` or `API`, and `<LABEL>` is the label for the connector (converted to lower case). Check here for more information about [environment connectors](/reference/connectors/#-environment-connectors), but in a nutshell, set the variable to the [URI](https://en.wikipedia.org/wiki/Uniform_Resource_Identifier) of your connector.
 
 ```bash
-MRSM_SQL_FOO=sqlite:////tmp/temp.db\
+MRSM_SQL_FOO=sqlite:////tmp/temp.db \
 MRSM_API_BAR=http://user:pass@localhost:8000 \
   mrsm show connectors
 ```
@@ -78,4 +78,12 @@ The variable `MRSM_PATCH` behaves the same way as `MRSM_CONFIG`. The difference
 MRSM_CONFIG='{"foo": "bar": 123}' \
 MRSM_PATCH='baz:MRSM{foo:bar}' \
   mrsm show config baz
+```
+
+## **`MRSM_VENVS_DIR`**  
+
+Like `MRSM_PLUGINS_DIR`, you can designate a separate directory outside of the Meerschaum root to contain virtual environments. Generally you should not need to set this, but this is useful for sharing virtual environments between deployments as well as separating package data from user data (e.g. Kubernetes deployments).
+
+```bash
+MRSM_VENVS_DIR='venvs/' mrsm show plugins
 ```

meerschaum/_internal/entry.py

@@ -92,21 +92,24 @@ def entry_with_args(
     kw['action'] = remove_leading_action(kw['action'], _actions=_actions)
 
     with Venv(plugin, debug=kw.get('debug', False)):
+        action_name = ' '.join(action_function.__name__.split('_') + kw.get('action', []))
         try:
             result = action_function(**kw)
         except Exception as e:
             if kw.get('debug', False):
                 import traceback
                 traceback.print_exception(type(e), e, e.__traceback__)
             result = False, (
-                f"Failed to execute '{' '.join([action_function.__name__] + kw['action'])}' "
+                f"Failed to execute `{action_name}` "
                 + "with exception:\n\n" +
                 f"{e}."
                 + (
                     "\n\nRun again with '--debug' to see a full stacktrace."
                     if not kw.get('debug', False) else ''
                 )
             )
+        except KeyboardInterrupt:
+            result = False, f"Cancelled action `{action_name}`."
 
     ### Clean up stray virtual environments.
     for venv in [venv for venv in active_venvs]: