v1.4.4

v1.4.0 – v1.4.4

• Added in-place syncing for SQL pipes.
  This feature is big (enough to warrant a new point release). When pipes with the same instance connector and data source connector are synced, the method sync_pipe_inplace() is invoked. For SQL pipes, this means the entire syncing process will now happen entirely in SQL, which can lead to massive performance improvements.

  import meerschaum as mrsm
  import pandas as pd
  
  conn = mrsm.get_connector('sql', 'local')
  conn.to_sql(pd.DataFrame([{'a': 1}]), 'foo')
  
  pipe = mrsm.Pipe(
      "sql:local", "foo",
      instance = "sql:local",
      parameters = {
          "query": "SELECT * FROM foo"
      },
  )
  ### This will no longer load table data into memory.
  pipe.sync()

  This applies even when the source table's schema changes, just like the dynamic columns feature added in v1.3.0.

  To disable this behavior, run the command edit config system and set the value under the keys experimental:inplace_sync to false.

• Added negation to --params.
  The build_where() function now allows you to negate certain values when prefixed with an underscore (_):

  ### Show recent data, excluding where `id` equals `2`.
  mrsm show data --params id:_2

• Added --params to SQL pipes' queries.
  Specifying parameters when syncing SQL pipes will add those constraints to the fetch stage.

• Skip invalid parameters in --params.
  If a column does not exist in a pipe's table, the value will be ignored in --params.

• Fixed environment issue when starting the Web API with gunicorn.

• Added an emoji to the SQL Query option of the web console.

• Fixed an edge case with data type enforcement.

• Other bugfixes

v1.3.13

v1.3.13

• Fixed an issue when displaying backtrack data on the Web Console.
  Certain values like pd.NA would break the Recent Data view on the Web Console. Now the values are cast to strings before building the table.

• Added YAML and JSON support to editing parameters.
  YAML is now the default, and toggle buttons have been added to switch the encoding. Line numbers have also been added to the editors.

• Removed the index column from the CSV downloader.
  When the download button is clicked, the dataframe's index column will be omitted from the CSV file.

• Changed the download filename to <target>.csv.
  The download process will now name the CSV file after the table rather than the pipe.

• Web Console improvements.
  The items in the actions builder now are presented with a monospace font. Actions and subactions will have underscores represented as spaces.

• Activating the virtual environment None will not override your current working directory.
  This is especially useful when testing the API. Activating the virtual environment None will insert behind your current working directory or '' in sys.path.

• Added WebSocket Secure Support.
  This has been coming a long time, so I'm proud to announce that the web console can now detect whether the client is connecting via HTTPS and (assuming the server has the appropriate proxy configuration) will connect via WSS.

v1.3.12

v1.3.10 – v1.3.12

• Fixed virtual environment issues when syncing.
  This one's a doozy. Before this patch, there would frequently be warnings and sometimes exceptions thrown when syncing a lot of pipes with a lot of threads. This kind of race condition can be hard to pin down, so this patch reworks the virtual environment resolution system by keeping track of which threads have activated the environments and refusing to deactivate if other threads still depend on the environment. To enforce this behavior, most manual ivocations of activate_venv() were replaced with the Venv context manager. Finally, the last stage of each action is to clean up any stray virtual environments. Note: You may still run into the wrong version of a package being imported into your plugin if you're syncing a lot of plugins concurrently.

• Allow custom instance connectors to be selected on the web console.
  Provided all of the appropriate interface methods are implemented, selecting a custom instance connector from the instance dropdown should no longer throw an error.

• Bugfixes and improvements to the virtual environment system.
  This patch should resolve your virtual environment woes but at a somewhat significant performance penalty. Oh well, 'tis the price we must pay for correct and determinstic code!

• Fixed custom flags added by add_plugin_argument().
  Refactoring work to decouple plugins from the argument parser had the unintended side effect of skipping over custom flags until after sysargs had already been parsed. This patch ensures all plugins with add_plugin_argument in their root module will be loaded before parsing.

• Upgraded dash-extensions, dash, and dash-bootstrap-components.
  At long last, dash-extensions will no longer need to be held back.

• Added additional websockets endpoints.
  The endpoints /ws, /dash/ws/, and /dashws resolve to the same handler. This is to allow compatability with different versions of dash-extensions.

• Allow for custom arguments to be added from outside plugins.
  The function add_plugin_argument() will now accept arguments when executed from outside a plugin.

• Fixed verify packages.

v1.3.9

v1.3.6 – v1.3.9

• Allow for syncing multiple data types per column.
  The highlight of this release is support for syncing multiple data types per column. When different data types are encountered, the underlying column will be converted to TEXT:

  import meerschaum as mrsm
  pipe = mrsm.Pipe('a', 'b', instance='sql:memory')
  
  pipe.sync([{'a': 1}])
  print(pipe.get_data())
  #    a
  # 0  1
  
  pipe.sync([{'a': 'foo'}])
  #      a
  # 0    1
  # 1  foo

• Cleaned up the Web Console.
  The web console's navbar is now more mobile-friendly, and a "sign out" button has been added.

• Removed plugins import when checking for environment connectors.
  This should make some commands feel more snappy by lazy loading custom connectors.

• Fixed an issue with an updated versin of uvicorn.

• Fixed an issue with docker-compose.

• Fixed an issue with FastAPI on Python 3.7.

• Added support for Python 3.11.

• Renamed meerschaum.actions.arguments to meerschaum._internal.arguments.

v1.3.5

v1.3.4 – v1.3.5

• Define environment connectors with JSON or URIs.
  Connectors defined as environment variables may now have their attributes set as JSON in addition to a URI string.

• Custom Connectors may now be defined as environment variables.
  You may now set environment variables for custom connectors defined via @make_connector, e.g.:

  MRSM_FOO_BAR='{"foo": "bar"}' mrsm show connectors

• Allow for custom connectors to be instance connectors.
  Add the property IS_INSTANCE = True your custom connector to add it to the official list of instance types:

  from meerschaum.connectors import make_connector, Connector
  
  @make_connector
  class FooConnector(Connector):
      IS_INSTANCE = True
  
      def __init__(label: str, **kw):
          super().__init__('foo', label, **kw)

• Install packages for all plugins with mrsm install required.
  The default behavior for mrsm install required with no plugins named is now to install dependencies for all plugins.

• Syncing bugfixes.

v1.3.3

v1.3.2 – 1.3.3

• Fixed a bug with begin and end bounds in Pipe.get_data().
  A safety measure was incorrectly checking if the quoted version of a column was in pipe.get_columns_types(), not the unquoted version. This patch restores functionality for pipe.get_data().
• Fixed an issue with an upgraded version of SQLAlchemy.
• Added a parameters editor the Web UI.
  You may now edit your pipes' parameters in the browser through the Web UI!
• Added a SQL query editor to the Web UI.
  Like the parameters editor, you can edit your pipes' SQL queries in the browser.
• Added a Sync Documents option to the Web UI.
  You can directly sync documents into pipes on the Web UI.
• Added the arguments order, limit, begin_add_minutes, and end_add_minutes to Pipe.get_data().
  These new arguments will give you finer control over the data selection behavior.
• Enforce consistent ordering of indices in Pipe.get_data().
• Allow syncing JSON-encoded strings.
  This patch allows pipes to sync JSON strings without first needing them to be deserialized.
• Fixed an environment error with Ubuntu 18.04.
• Bumped duckdb and duckdb-engine.
• Added a basic CLI for duckdb.
  This will probably be replaced later down the line.

v1.3.1

v1.3.1

• Fixed data type enforcement issues.
  A serious bug in data type enforcement has been patched.
• Allow Pipe.dtypes to be edited.
  You can now set keys in Pipe.dtypes and persist them with Pipe.edit().
• Added Pipe.update().
  Pipe.update() is an alias to Pipe.edit(interactive=False).
• Pipe.delete() no longer deletes local attributes.
  It still removes Pipe.id, but local attributes will now remain intact.
• Fixed dynamic columns on DuckDB.
  DuckDB does not allow for altering tables when indices are created, so this patch will drop and rebuild indices when tables are altered.
• Replaced CLOB with NVARCHAR(2000) on Oracle SQL.
  This may require migrating existing pipes to use the new data type.
• Enforce integers are of type INTEGER on Oracle SQL.
  Lots of data type enforcement has been added for Oracle SQL.
• Removed datetime warnings when syncing pipes without a datetime column.
• Removed grabbing the current time for the sync time if a sync time cannot be determined.

v1.3.0: Dynamic Columns

v1.3.0: Dynamic Columns

Improvements

• Syncing now handles dynamic columns.
  Syncing a pipe with new columns will trigger an ALTER TABLE query to append the columns to your table:

  import meerschaum as mrsm
  pipe = mrsm.Pipe('foo', 'bar', instance='sql:memory')
  
  pipe.sync([{'a': 1}])
  print(pipe.get_data())
  #    a
  # 0  1
  
  pipe.sync([{'b': 1}])
  print(pipe.get_data())
  #       a     b
  # 0     1  <NA>
  # 1  <NA>     1

  If you've specified index columns, you can use this feature to fill in NULL values in your table:

  import meerschaum as mrsm
  pipe = mrsm.Pipe(
      'foo', 'bar',
      columns = {'id': 'id_col'},
      instance = 'sql:memory',
  )
  
  pipe.sync([{'id_col': 1, 'a': 10.0}])
  pipe.sync([{'id_col': 1, 'b': 20.0}])
  
  print(pipe.get_data())
  #    id_col     a     b
  # 0       1  10.0  20.0

• Add as many indices as you like.
  In addition to the special index column labels datetime, id, and value, the values of all keys within the Pipe.columns dictionary will be treated as indices when creating and updating tables:

  import meerschaum as mrsm
  indices = {'micro': 'station', 'macro': 'country'}
  pipe = mrsm.Pipe('demo', 'weather', columns=indices, instance='sql:memory')
  
  docs = [{'station': 1, 'country': 'USA', 'temp_f': 80.6}]
  pipe.sync(docs)
  
  docs = [{'station': 1, 'country': 'USA', 'temp_c': 27.0}]
  pipe.sync(docs)
  
  print(pipe.get_data())
  #    station  country  temp_f  temp_c
  # 0        1      USA    80.6    27.0

• Added a default 60-second timeout for pipe attributes.
  All parameter properties (e.g. Pipe.columns, Pipe.target, Pipe.dtypes, etc.) will sync with the instance every 60 seconds. The in-memory attributes will be patched on top of the database values, so your unsaved state won't be lost (persist your state with Pipe.edit()). You can change the timeout duration with mrsm edit config pipes under the keys attributes:local_cache_timeout_seconds. To disable this caching behavior, set the value to null.

• Added custom indices and Pandas data types to the Web UI.

v1.2.9

v1.2.9

This is a relatively minor release:

• Fixed virtual environment verification to work with Windows junctions.

This release patches breaking changes on Windows installs.

v1.2.8

v1.2.8

• Custom connectors may now have register(pipe) methods.
  Just like the module-level register(pipe) plugin function, custom connectors may also provide this function as a class member.
• Print a traceback if fetch(pipe) breaks.
  A more verbose traceback is printed if a plugin breaks during the syncing process.
• Cleaned up sync pipes output.
  This patch cleans up the syncing process's pretty output.
• Respect --nopretty in sync pipes.
  This flag will only print JSON-encoded dictionaries for sync pipes. Tracebacks may still interfere without standard output, however.