v2.3.1

What's Changed

• Added support for Changelog topics

  • Changelog topics provide fault tolerance capabilities to state stores.
    Each state store now has a corresponding changelog topic to keep track of the state updates in Kafka.
  • Changelog topics are enabled by default and can be disabled.
  • See more about changelog topics in the docs

• Application.run() class now verifies that topics exist before starting the application.
  If topics don't exist, the Application instance will try to create them automatically if auto_create_topics is set to True (default).
  The topic parameters can also be specified, see more in the docs

• This is the first non-alpha release of Quix Streams v2. It can now be installed from pip without the --pre flag.

Breaking changes

• The partition assignment strategy is now always set to cooperative-sticky and cannot be configured anymore because the consumer relies on the incremental_assign() API for recovery.
  Previously, the assignment strategy was set to range by default, and range is a non-cooperative strategy.
  Since cooperative and non-cooperative (eager) strategies must not be mixed, all consumers in the group must first leave the group, and then join it again after upgrading the application to this version.

Full Changelog: v2.2.1a...v2.3.1

v2.2.1a

What's Changed

Bugfixes

• Updates to work with Redpanda brokers via Quix platform by @daniil-quix in #283
• Fix RocksDB does not retry opening if it's locked by another process by @harisbotic in #272
• Add name property to window s by @harisbotic in #277

Docs

• Add docs about windows by @daniil-quix in #274
• Update docs config by @daniil-quix in #276
• Fix windows docs by @daniil-quix in #278
• Update windows docs by @stereosky in #279
• Update readme by @daniil-quix in #284
• add a line to the readme about sliding windows by @tim-quix in #285

New Contributors

• @stereosky made their first contribution in #279

Full Changelog: v2.2.0a...v2.2.1a

v2.2.0a

What's Changed

Added Windowed Aggregations

• Implemented two types of time-based windows:

  • Tumbling windows slice time into even non-overlapping periods.
    Example: [0, 10), [10, 20), [20,30)

  • Hopping windows slice time into even overlapping periods with a fixed step.
    Example: [0,10), [1, 11), [2, 12)

• Support for various aggregation functions: sum, count, mean, min, max, reduce
• Two modes of emitting aggregated results:
  • final - to emit results when the window is closed
  • current - to emit results for each incoming message as it's processed
• Support for out-of-order processing

Find more about Windowed Aggregations in the docs

Full Changelog: v2.1.4a...v2.2.0a

v2.1.4a

What's Changed

• Add support for Column families by @daniil-quix in #268
• Update the mapping of broker settings for Quix apps by @daniil-quix in #270
• Fix bug in RocksDB transaction on deleting cached key by @harisbotic in #269
• Add API to create a pre-configured Producer and Consumer on the Application level by @harisbotic in #261

Full Changelog: v2.1.3a...v2.1.4a

v2.1.3a

What's Changed

• Implement better handling for all broker down error by @peter-quix in #264

## Repo cleanup

• Make URLs absolute in the docs by @daniil-quix in #256
• Update repo banner by @SteveRosam in #262
• Update README.md by @SteveRosam in #263

Full Changelog: v2.1.2a...v2.1.3a

v2.1.2a

What's Changed

• Fix ssl timeout errors for Quix applications (#254)
• Various docs fixes & updates #252 #253

Full Changelog: v2.1.1a...v2.1.2a

v2.1.1a

What's Changed

• [New] Implemented StreamingDataFrame.apply(expand=True) to produce multiple individual values based on the collection. [#246]
  You may use .apply(expand=True) to produce multiple values to the output topics for a single input message
  Docs

• [Fix] Make "and" and "or" comparisons lazy in StreamingSeries [#248]
  In previous versions, code like sdf['a'] | sdf['b'] and sdf['a'] & sdf['b'] was always evaluating both sides of the expression, while usual and and or can evaluate the right side lazily.

• Various docs updates: #247 #250 #251

Full Changelog: v2.1alpha1...v2.1.1a

v2.1a1

Release v2.1a1

This release brings new features and breaking API changes.
Read on for more details.

What's changed

Many updates to StreamingDataFrame and StreamingSeries (ex-Column) [#238]

We updated how streaming data is processed by StreamingDataFrame and StreamingSeries classes.
They now use the same engine to apply functions to the message values.

It introduces both new features and breaking changes:

1. [breaking] Changes to StreamingDataFrame and new methods .update() and .filter()

• Functions passed to StreamingDataFrame.apply() should always return a new value. The result of the function will be propagated downstream.
  Previously, .apply() was used to both mutate values and generate new ones.
  Now it's discouraged, and the .update() method should be used instead to mutate data and perform side effects (e.g. to print to the console).
  Docs

• New method StreamingDataFrame.update() to mutate values in place.
  Docs

• New method StreamingDataFrame.filter() to filter values.
  Docs

• New syntax to filter values - dataframe[dataframe.apply(<func>)]
StreamingDataFrame can now filter data using a similar API as pandas.DataFrame.
  Docs

• New syntax to update values - dataframe['field'] = dataframe.apply(<func>)
StreamingDataFrame can now assign new keys to the values using functions.
  Docs`

• StreamingDataFrame doesn't require values to be dictionaries anymore.

• StreamingDataFrame methods now return a new StreamingDataFrame.
  Previously, many methods like .apply() and .to_topic() were mutating the state of the StreamingDataFrame instance.
  Now all of them return a new instance, and the current instance remains intact.

• Functions passed to .apply() now get only one argument - message value.
  Previously, functions passed to .apply() were provided with at least 2 positional arguments: message value and MessageContext with Kafka message metadata.
  Now they don't need to accept MessageContext to access a current message key, and it is stored globally for each incoming message.
  To get a current message key and or full message metadata use quixstreams.message_key() and quixstreams.message_context() in
  your custom functions.
  Docs

New method to check if the key exists in the message value - StreamingDataFrame.contains() [#235]

New method to clear the local state - Application.clear_state()[#239]

Use Application.clear_state() to clear the local state of the application.
Quix Streams keeps state data per consumer group, so the state will be cleared only for the
particular consumer group.

[breaking] Changes to logical operators |, & and ~ in StreamingSeries [#238]

The logical operators |, &, and ~ now do logical comparisons.
Previously it worked this way only with booleans, when for numbers it would do a bitwise operation.

Support for Python 3.12 [#244]

Quix Streams now works with Python 3.12

Full Changelog: v2.0alpha2...v2.1alpha1

v2.0a2

This is the first alpha release of the new Quix Streams 2.0.
It's a complete rewrite, providing a different API to work with streaming data in Python.

What's Changed

Introduced Streaming DataFrames - a primary API to define the processing pipeline

1. It provides a pandas.DataFrame-like API to structure the message transformations.
   For the full description of StreamingDataFrame API please see StreamingDataFrame: Detailed Overview

2. It supports stateful operations backed by RocksDB state storage. The details regarding work with state are outlined here - Stateful Processing

3. It supports different message serialization formats like strings, integers, doubles, JSON and Quix formats.
   Detailed overview of serialization in 2.0 can be found here - Serialization

Removed a dependency on .NET

The 2.0 version doesn't depend on .NET anymore and is pure Python.
It makes the library code more stable, safe, and easier to maintain and update.

Compatibility with Quix Streams 0.5.7

Although 2.0 introduces a totally new API, both 0.5.7 and 2.0 are mostly compatible on the data level - both 0.5.7 and 2.0a consumers and producers may exchange messages in the same format.

There are some limitations to the types of messages supported in 2.0.
Please see Upgrading from Legacy Quixstreams for more details

Full Changelog: v0.5.7...v2.0alpha2

v0.5.7

New

• Create leading edge buffer where tag is not part of the key by @peter-quix in #180

Bugfix

• Python app should subscribe by default similarly to C# by @peter-quix in #190

Full Changelog: v0.5.6...v0.5.7