v1.0.1

v1.0.1

• Added citus as an official database flavor.
  Citus is a distributed database built on PostgreSQL. When an id column is provided, Meerschaum will call create_distributed_table() on the pipe's ID index. Citus has also been added to the official test suite.
• Changed end datetimes to be exclusive.
  The end parameter now generates < instead of <=. This shouldn't be a major breaking change but is important to be aware of.
• Bumped rich to v12.4.4.

v1.0.0

v1.0.0: Mutable at Last

What's New

• Inserts and Updates
  An additional layer of processing separates new rows from updated rows. Meerschaum uses your datetime and id columns (if you specified an id column) to determine which rows have changed. Therefore a primary key is not required, as long as the datetime column is unique or the datetime and id columns together emulate a composite primary key.

  Meerschaum will insert new rows as before as well as creating a temporary table (same name as the pipe's target but with a leading underscore). The syncing engine then issues the appropriate MERGE or UPDATE query to update all of the rows in a batch.

  For example, the following lines of code will result in a table with only 1 row:

  >>> import meerschaum as mrsm
  >>> pipe = mrsm.Pipe('foo', 'bar', columns={'datetime': 'dt', 'id': 'id'})
  >>>
  >>> ### Insert the first row.
  >>> pipe.sync([{'dt': '2022-06-26', 'id': 1, 'value': 10}])
  >>>
  >>> ### Duplicate row, no change.
  >>> pipe.sync([{'dt': '2022-06-26', 'id': 1, 'value': 10}])
  >>>
  >>> ### Update the value columns of the first row.
  >>> pipe.sync([{'dt': '2022-06-26', 'id': 1, 'value': 100}])

• Data Type Enforcement.
  Incoming DataFrames will be cast to the pipe's existing data types, and if you want total control, you can manually specify the Pandas data types for as many columns as you like under the dtypes key of Pipe.parameters, e.g.:

  columns:
    datetime: timestamp_utc
    id: station_id
  dtypes:
    timestamp_utc: 'datetime64[ns]'
    station_id: 'Int64'
    station_name: 'object'

• Allow for NULL in INT columns.
  Before pandas v1.3.0, including a null value in an int column would cast it to a float. Now pd.NA has been added and is leveraged in Meerschaum's data type inference system.

• Plugins respect .gitignore
  When publishing a plugin that is contained in a git repository, Meerschaum will parse your .gitignore file to determine which files to omit.

• Private API Mode.
  Require authentication on all API endpoints with start api --private.

• No Authentication API Mode.
  Adding --no-auth will disable all authentication on all API endpoints, including the web console.

Bugfixes

• Plugin packaging
  A breaking bug in the process of packaging and publishing Meerschaum plugins has been patched.

• Correct object names in Oracle SQL.
  Oracle has finally been brought up to speed with other flavors.

• Multi-module plugins fix.
  A small but important fix for multi-module plugins has been applied for their virtual environments.

• Improved virtual environment handling for Debian systems.
  If venv is not available, Meerschaum now better handles falling back to virtualenv.

• Allow for syncing lists of dicts.
  In addition to syncing a dict of lists, Pipe.sync() now supports a list of dicts.

• Allow for begin to equal None for Pipe.fetch().
  The behavior of determining begin from Pipe.get_sync_time() only takes place when begin is omitted, not when it is None. Now begin=None will not add a lower bound to the query.

v0.6.5

v0.6.3 – v0.6.5: Durable Venvs

• Improved durability of the virtual environments.
  The function meerschaum.utils.packages.manually_import_module() behaves as expected, allowing you to import different versions of modules. More work needs to be done to see if reintroducing import hooks would be beneficial.
• Activate plugin virtual environments for API plugins.
  If a plugin uses the @api_plugin decorator, its virtual environment will be activated before starting the API server. This could potentially cause problems if you have many API plugins with conflicting dependencies, but this could be mitigated by isolating environments with MRSM_ROOT_DIR.
• Changed name to import_name for determine_version().
  The first argument in meerschaum.utils.packages.determine_version() has been renamed from name to the less-ambiguous import_name.
• Shortened the IDs for API environments.
  Rather than a long UUID, each instance of the API server will have a randomly generated ID of six letters. Keep in mind it is randomly generated, so please excuse any randomly generated words.
• Removed version enforcement for uvicorn and gunicorn.
  Uvicorn has a lot of hidden imports, and using our home-brewed import system breaks things. Instead, we now use the default attempt_import behavior of check_update=False.
• Reintroduced verify packages to setup.py.
  Upgrading Meerschaum will check if the virtual environment packages satisfy their required versions.
• Moved pkg_resources patch from the module-level.
  In v0.6.3, the monkey-patching for flask-compress happened at the module level, but this was quickly moved to a lazy patch in v0.6.4.
• Bugfixes
  Versions 0.6.3 and 0.6.4 were yanked due to some unforeseen broken features.
• Bumped several dependencies.

v0.6.2

v0.6.0 – v0.6.2: Robust Plugins and Beautiful Pipes

Potentially Breaking Changes

• Renamed meerschaum.connectors.sql.tools to meerschaum.utils.sql.
  A dummy module was created at the old import path, but this will be removed in future releases.
• Migrated to meerschaum.core.
  Important class definitions (e.g. User) have been migrated from meerschaum._internal to meerschaum.core. You can still import meerschaum.core.Pipe as mrsm.Pipe, however.
• Moved meerschaum.actions.shell to meerschaum._internal.shell.
  Finally marked it off the to-do list!
• Pipe.__str__() and Pipe.__repr__() now return stylized strings.
  This should make reading logs significantly more pleasant. You can add syntax highlighting back to strings containing Pipe() with meerschaum.utils.formatting.highlight_pipes().

New Features

• Plugins
  Exposed the meerschaum.Plugin class, which will make cross-pollinating between plugins simpler.
• Uninstall procedure
  Plugins now ship with a proper uninstall() method.
• Sharing dependencies
  Plugins may now import dependencies from a required plugin's virtual environment. E.g. if plugin foo requires plugin:bar, and bar requires pandas, then foo will be able to import pandas.
• Allow other repos for required plugins.
  You can now specify the keys of a required plugin's repo following @, e.g. foo may require plugin:bar@api:main.
• Isolate package cache.
  Each virtual environment now uses an isolated cache folder.
• Handle multiple versions of packages in determine_version()
  When verifying packages, if multiple dist-info directories are found in a virtual environment, import the package in a subprocess to determine its __version__.
• Specify a target table.
  Pipe.target (Pipe.parameters['target']) now governs the name of the underlying SQL table.

Bugfixes

• Circular dependency resolver
  Multiple plugins may now depend on each other without entering a recursive loop.
• Held back dash_extensions due to breaking API changes.
  Future releases will migrate to dash_extensions>1.0.0.
• Fixed meerschaum.plugins.add_plugin_argument().
  Refactoring broke something awhile back; more plugins-focused tests are needed.
• Fixed an issue with fontawesome and mkdocs-material.
• Fixed pickling issue with mrsm.Pipe.

Documentation

• pdoc changes.
  Added __pdoc__ and __all__ to public modules to simplify the package docs.
• Lots of cleanup.
  Almost all of the docstrings have been edited.

v0.5.14

• Added tags.
  Pipes may be grouped together pipes. Check the docs for more information.
• Tags may be negated.
  Like the key negation added in v0.5.13, you can choose to ignore tags by prefacing them with _.
• Bugfixes for DuckDB
• Updated documentation.

v0.5.13

• Key negation when selecting pipes.
  Prefix connector, metric, or location with _ to select pipes that do NOT have that key.
• Added the setup plugins command.
  Run the command setup plugins followed by a list of plugins to execute their setup() functions.
• Renamed pipes' keys methods function.
  The function meerschaum.utils.get_pipes.methods() is renamed to meerschaum.utils.get_pipes.fetch_pipes_keys().
• Improved stability for DuckDB.
• Bumped depdenencies.
  DuckDB, FastAPI, and Uvicorn have been updated to their latest stable versions.

v0.5.12

• Improved Oracle support.
  Oracle SQL has been fully integrated into the testing suite, and critical bugs have been addressed.
• Added the install required command.
  When developing plugins, run the command install required to install the packages in the plugin's required list into its virtual environment.
• Migrated docstrings.
  To improve legibility, many docstrings have been rewritten from reST- to numpy-style. This will make browsing docs.meerschaum.io easier.

v0.5.10

• Added the clear pipes command.
  Users may now delete specific rows within a pipe using pipe.clear(). This new method includes support for the --begin, --end, and --params flags.
• Changed the default behavior of --begin.
  The --begin flag is now only included when the user specifies and no longer defaults to the sync time.
• Added --timeout-seconds.
  The flags --timeout-seconds and --timeout make the syncing engine sync each pipe in a separate subprocess and will kill the process if the sync exceeds the number of provided seconds.
• Fixed shell argparse bug.
  When editing command line arguments within the shell, edge cases no longer cause the shell to exit.

v0.5.9

v0.5.9

• Stable interim release.
  This release includes minor fixes and is intended to be a stable release while development continues on new features.

v0.5.6

v0.5.6

• Added support for gunicorn.
  Gunicorn may be used to manage API processes with the --production or --gunicorn flags. The --production flag is not default in the Docker image of the API server.
• Updated bootstrap pipes flow.
  The interactive bootstrapping wizard now makes use of the new register() plugins API as well as asking for the value column.
• Fixed edge cases in Pipe.filter_existing().
  Better enforcement of NaT as well as --begin and --end now reduces edge-case bugs and unexpected behavior.
• Re-introduced the full Docker image.
  Inclusion of the start gui command led to the full version of the Docker image requiring GTK and dependencies. Now you can forward the GUI with docker run -e DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix bmeares/meerschaum:full start gui
• Fixed plugin installation bug (again).
• Allow for plugins with hyphens in the name.
• Lots of refactoring and tiny bugfixes.