Skip to content

⚡️ v2.6.14 Speed up dtype enforcement, fix Oracle auto-incrementing IDs.

Compare
Choose a tag to compare
@bmeares bmeares released this 13 Nov 05:47
· 21 commits to main since this release
33a3828

v2.6.10 – v2.6.14

  • Improve datetime timezone-awareness enforcement performance.
    Datetime columns are only parsed for timezone awareness if the desired awareness differs. This drastically speeds up sync times.

  • Switch to tz_localize() when stripping timezone information.
    The previous method of using a lambda to replace individual tzinfo attributes did not scale well. Using tz_localize() can be vectorized and greatly speeds up syncs, especially with large chunks.

  • Add enforce_dtypes to Pipe.filter_existing().
    You may optionally enforce dtype information during filter_existing(). This may be useful when implementing custom syncs for instance connectors. Note this may impact memory and compute performance.

    import meerschaum as mrsm
    import pandas as pd
    
    pipe = mrsm.Pipe('a', 'b', instance='sql:local')
    pipe.sync([{'a': 1}])
    
    df = pd.DataFrame([{'a': '2'}])
    
    ### `enforce_dtypes=True` will suppress the differing dtypes warning.
    unseen, update, delta = pipe.filter_existing(df, enforce_dtypes=True)
    print(delta)
  • Fix query_df() for null parameters.
    This is useful for when you may use query_df() with only select_columns or omit_columns.

  • Fix autoincrementing IDs for Oracle SQL.

  • Enforce security settings for creating jobs.
    Jobs and remote actions will only be accessible to admin users when running with --secure (system:permissions:actions:non_admin in config).