v1.2.9

v1.2.9

This is a relatively minor release:

• Fixed virtual environment verification to work with Windows junctions.

This release patches breaking changes on Windows installs.

v1.2.8

v1.2.8

• Custom connectors may now have register(pipe) methods.
  Just like the module-level register(pipe) plugin function, custom connectors may also provide this function as a class member.
• Print a traceback if fetch(pipe) breaks.
  A more verbose traceback is printed if a plugin breaks during the syncing process.
• Cleaned up sync pipes output.
  This patch cleans up the syncing process's pretty output.
• Respect --nopretty in sync pipes.
  This flag will only print JSON-encoded dictionaries for sync pipes. Tracebacks may still interfere without standard output, however.

v1.2.7

v1.2.5 – v1.2.7

• Venv context managers do not deactivate previously activated venvs.
  You can safely use Venv without worrying about deactivating your previously activated environments.
• Better handling of nested plugin dependencies.
• Plugin.get_dependencies() will not trigger an import.
  If you want certainty about a plugin's required list, trigger an import manually. Otherwise, it will use ast.literal_eval() to determine the required list from the source itself. This only works for statically set required lists.
• Provide rich traceback for broken plugins.
  If a plugin fails to import, a nice traceback is printed out in addition to a warning.
• Only cache Pipe.dtypes if the pipe exists.
• Pass current environment to subprocesses.
  This should retain any custom configuration you've set in the main process.
• Hard-code port 5432 as the target DB container port in the stack.
  Changing the host port now will not change the target port in the container.
• Fixed a bug with background jobs and to_sql().
  The --name flag was conflicting with to_sql().
• Reimplemented apply_patch_to_config().
  This patch removes cascadict as a vendored dependency and replaces it with a simpler implementation.
• Removed network request for shell connectivity status.
  The shell now simply checks for the existence of the connector. This may occasionally print an inaccurate connection status, but the speed benefit is worth it.
• Moved dill and other required dependencies into the sql dependency group.
• Replaced redengine with rocketry.
• Patched Literal into typing for Python 3.7.
• Fixed shell commands.
  This includes falling back to ' '.join instead of shlex.join for Python 3.7.

v1.2.2

v1.2.1 – v1.2.2

• Added the @make_connector decorator.
  Plugins may now extend the base Connector class to provide custom connectors. For most cases, the built-in plugin connector should work fine. This addition opens up the internal connector system so that plugin authors may now add new types. See below for a minimal example of a new connector class:

  # ~/.config/meerschaum/plugins/example.py
  
  from meerschaum.connectors import make_connector, Connector
  
  REQUIRED_ATTRIBUTES = {'username', 'password'}
  
  @make_connector
  class FooConnector(Connector):
      
      def __init__(self, label: str, **kw):
          """
          Instantiate the base class and verify the required attributes are present.
          """
          kw['label'] = label
          super().__init__('foo', **kw)
          self.verify_attributes(REQUIRED_ATTRIBUTES)
  
      
      def fetch(self, pipe, **kw):
          """
          Return a dataframe (or list of dicts / dict of lists).
          The same as you would in a regular fetch plugin, except that now
          we can store configuration within the connector itself.
          """
          return [{self.username: '2020-01-01'}]

• Allow for omitting datetime column index.
  The datetime column name is still highly recommended, but recent changes have allowed pipes to be synced without a dedicated datetime axis. Plenty of warnings will be thrown if you sync your pipes without specifying a datetime index. If a datetime column can be found, it will be used as the index.

  import meerschaum as mrsm
  ### This will work but throw warnings.
  mrsm.Pipe('foo', 'bar').sync([{'a': 1}])

v1.2.0

v1.2.0

Improvements

• Added the action start connectors.
  This command allows you to wait until all of the specified connectors are available and accepting connections. This feature is very handy when paired with the new MRSM_SQL_X URI environment variables.

• Added MRSM_PLUGINS_DIR.
  This one's been on my to-do list for quite a bit! You can now place your plugins in a dedicated, version-controlled directory outside of your root directory.

  Like MRSM_ROOT_DIR, specify a path with the environment variable MRSM_PLUGINS_DIR:

  MRSM_PLUGINS_DIR=plugins \
    mrsm show plugins

• Allow for symlinking in URI environment variables.
  You may now reference configuration keys within URI variables:

  MRSM_SQL_FOO=postgresql://user:MRSM{meerschaum:connectors:sql:main:password}@localhost:5432/db

• Increased token expiration window to 12 hours.
  This should reduce the number of login requests needed.

• Improved virtual environment verification.
  More edge cases have been addressed.

• Symlink Meerschaum into Plugin virtual environments.
  If plugins do not specify Meerschaum in the required list, Meerschaum will be symlinked to the currently running package.

Breaking changes

• API endpoints for registering and editing users changed.
  To comply with OAuth2 convention, the API endpoint for registering a user is now a url-encoded form submission to /users/register (/user/edit for editing).

  You must upgrade both the server and client to v1.2.0+ to login to your API instances.

• Replaced meerschaum.utils.sql.update_query() with meerschaum.utils.sql.get_update_queries().
  The new function returns a list of query strings rather than a single query. These queries are executed within a single transaction.

Bugfixes

• Removed version enforcement in pip_install().
  This changed behavior allows for custom version constraints to be specified in Meerschaum plugins.

• Backported UPDATE FROM query for older versions of SQLite.
  The current mutable data logic uses an UPDATE FROM query, but this syntax is only present in versions of SQLite greater than 3.33.0 (released 2020-08-14). This releases splits the same logic into DELETE and INSERT queries for older versions of SQLite.

• Fixed missing suggestions for shell-only commands.
  Completions for commands like instance are now suggested.

• Fixed an issue with killing background jobs.
  The signals were not being sent correctly, so this release includes better job process management.

• Install from requirements.txt if required is not present.

• Fixed issue with custom actions with underscores on the Web UI.

v1.1.10

v1.1.9 – v1.1.10

• Fixed plugins virtual environments.
  A typo in v1.1.8 temporarily broke plugins, and this patch fixes that change.
• Fixed Meerschaum on Windows.
  A change in a previous release allowed for dist-packages for the root user (not advised but supported). The check for root (os.geteuid()) does not exist on Windows, so this patch accounts for that behavior.
• Tweaked screen clearing on Windows.
  Meerschaum now always uses clear or cls on Windows instead of ANSI escape sequences.

v1.1.8

v1.1.5 – v1.1.8

• Fixed MRSM_PATCH behavior.
  In the docker image, MRSM_PATCH is used to overwrite host for sql:main. This patch restores that behavior (with a performance boost).
• Fixed virtual environment verification.
  This patch prevents circular symlinks.
• Fixed manually_import_module().
  Previous refactoring efforts had broken manually_import_module().
• Refactoring
  While trying to implement multi-thread configuration patching (discarded for the time being), much of the configuration system was cleaned up.

v1.1.4

v1.1.1 – v1.1.4

Bugfixes

The first four versions following the initial v1.1.0 release addressed breaking changes and edge cases. Below are some notable issues resolved:

• Fixed broken Docker images.
  Changes to the environment and package systems broke functionality of the Docker images. For example, v1.1.0 switched to a stricter package management policy, but this new policy broke the mechanism behind the Docker images (user-level vs venv-level packages).
• Verify virtual environments for multiple Python versions.
  When a virtual environment is first activated, Meerschaum now verifies that the python symlinks point to the correct versions. This is necessary due to a quirk in venv and virtualenv where activating an existing environment with a different Python version overwrites the existing python symlink. It also ensures that the symlinks specify the correct version number, e.g. python3.10. This bevavior is now automatic but may be invoked with mrsm verify venvs.
• Fixed inconsistent environment behavior with gunicorn.
  This one was tricky to troubleshoot. Due to the migration to the user-level Docker image, a subtle bug surfaced where the environment variables for gunicorn were incorrectly serialized.
• Fixed slow-performing edge cases in determine_version().
  Inconsistencies in naming conventions in some packages like pygments led to failures to quickly determine the version.
• Fixed Web API actions.
  In v1.1.0, the default virtual environment was pinned to mrsm, and this broke a function which relied on the old inferred default value of None. Always remember: explicit is better than implicit.
• Fixed start job for existing jobs.
  The same naming change broke daemon_action(). Explcit code is important, folks!

v1.1.0: URIs, environment connectors, custom actions with spaces, + more

v1.1.0

What's New

• Underscores in actions may now be parsed as spaces.
  This took way more work than expected, but anyway, custom actions with underscores in the function names are now treated as spaces! Consider the following:

  @make_action
  def foo_bar(**kw):
      return True, "Huzzah!"

  The above action may now be executed as foo bar or foo_bar:

  mrsm foo bar

• Create a SQLConnector or APIConnector directly from a URI.
  If you already have a connection string, you can skip providing credentials and build a connector directly from the URI. If you omit a label, then the lowercase form of '<username>@<host>/<database>' is used:

  from meerschaum.connectors import SQLConnector
  uri = 'postgresql://user:pass@localhost:5432/db'
  
  conn = SQLConnector(uri=uri)
  print(conn)
  # sql:user@localhost/db
  
  conn = SQLConnector('foo', uri=uri)
  print(conn)
  # sql:foo
  
  conn = SQLConnector.from_uri(uri)
  print(conn)
  # sql:user@localhost/db

  The APIConnector may also be built from a URI:

  from meerschaum.connectors import APIConnector
  uri = 'http://user:pass@localhost:8000'
  
  conn = APIConnector(uri=uri)
  print(conn)
  # api:username@localhost
  
  conn = APIConnector('bar', uri=uri)
  print(conn)
  # api:bar
  
  conn = APIConnector.from_uri(uri)
  print(conn)
  # api:user@localhost

• Define temporary connectors from URIs in environment variables.
  If you set environment variables with the format MRSM_SQL_<LABEL> to valid URIs, new connectors will be available under the keys sql:<label>, where <label> is the lowercase form of <LABEL>:

  export MRSM_SQL_FOO=sqlite://///path/to/sqlite.db
  mrsm show connectors sql:foo

  You can set as many connectors as you like, and they're treated the same as connectors registered in your permanent configuration.

  ### The following environment variable was already exported:
  ### MRSM_SQL_FOO=sqlite://///path/to/sqlite.db
  
  import meerschaum as mrsm
  conn = mrsm.get_connector('sql', 'foo')
  print(conn.database)
  # /path/to/sqlite.db

Bugfixes

• Resolved issues with conflicting virtual and base environments.

Potentially Breaking Changes

• The database file path for SQLite and DuckDB is now required.
  When creating a SQLConnector with the flavors sqlite or duckdb, the attribute database (a file path or :memory:) is now required.
• Removed --config and --root-dir.
  These flags were added very early on but have always caused issues. Instead, please use the environment variables MRSM_CONFIG or MRSM_PATCH for modifying the runtime configuration, and use MRSM_ROOT_DIR to specify a file path to the root Meerschaum directory.
• Virtual environments must be None for standard library packages.
  When importing a built-in module with attempt_import(), specify venv=None to avoid attempting installation.

v1.0.6

v1.0.6

• Plugins may now have requirements.txt.
  If a plugin contains a file named requirements.txt, the file will be parsed alongside the packages specified in the required list.

• Added the module meerschaum.utils.venv.
  Functions related to virtual environment management () have been migrated from meerschaum.utils.packages to meerschaum.utils.venv.

• Added the Venv class.
  You can now manage your virtual environments with the Venv context manager:

  from meerschaum.utils.venv import Venv
  
  with Venv():
      import pandas
  
  with Venv('foo'):
      import bar

  You can also activate the environments for a Plugin:

  from meerschaum.utils.venv import Venv
  from meerschaum.plugins import Plugin
  
  with Venv(Plugin('noaa')):
      import requests

• Removed --isolated from pip_install.
  Virtual environments will now respect environment variables and your global pip configuration (~/.pip/pip.conf).

• Fixed issues for Python 3.7