v0.1.4

Daft 0.1.4 Release Notes

The Daft 0.1.4 release features our Image type columns!

New Features

Image Types

Our first Daft Image types have landed!

You can now construct an Image column with .image.decode() on a Binary column.

See PRs:

1. [Images] [1/N] Logical type for variable-shaped and fixed-shaped images. #955
2. [Images] [3/N] Add image decoding for uint8 images. #981
3. Image Resize for ImageType #967
4. [Images] [2/N] Add scaffolding for image decoding and other ops. #965

Documentation

1. Fix modin typo; add partial scale numbers; highlight highlights #986
2. Add Page on benchmarking #980
3. Fix link for broken link checker #972
4. Clean up DataType userguide/API docs #966
5. Add more complex datatypes to docs #961
6. Datatype docs #894

Bug Fixes

1. [Scheduler] Fix join performance bug #985
2. Table Slice and IntoPartitions Fix #962
3. size_bytes fix: Guard against calculating variance of one item #957

Build Changes

1. Bump hypothesis from 6.75.8 to 6.75.9 #979
2. Bump orjson from 3.8.14 to 3.9.0 #978
3. Bump hypothesis from 6.75.6 to 6.75.8 #976
4. Add s3fs to dev requirements #975
5. Bump python version to 3.9 for profiling and pin Dask version for python 3.8 #973
6. Bump log from 0.4.17 to 0.4.18 #971
7. Bump dask from 2023.5.0 to 2023.5.1 #970
8. Bump pandas from 2.0.1 to 2.0.2 #969
9. Bump hypothesis from 6.75.5 to 6.75.6 #968
10. Bump orjson from 3.8.13 to 3.8.14 #964
11. Bump hypothesis from 6.75.3 to 6.75.5 #963
12. finer feature flags for arrow2 for faster compile #960
13. Bump pytest-cov from 4.0.0 to 4.1.0 #959
14. Bump orjson from 3.8.12 to 3.8.13 #958

v0.1.3

Daft 0.1.3 Release Notes

The Daft 0.1.3 release features fixes for a few performance regressions.

Enhancements

1. Very basic s3 parquet microbenchmark #954

Bug Fixes

1. [I/O] Change back to random access read for Parquet. #953
2. [CI] Fix flaky Ray Datasets integration test. #952
3. [Ray Runner] Unfixing batch size for task awaiting #951
4. Testing object related performance fixes #949

Build Changes

1. [ci] [daft publish] pin urllib to < 2 for conda #950

v0.1.2

Daft 0.1.2 Release Notes

The Daft 0.1.2 release features performance improvements, bugfixes and some of our first Daft logical types!

New Features

Extension Types for Ray Runner and Embedding Logical Type

Adds our first “Logical Type”: Embeddings!

An Embedding is a “Logical Type” that encompasses a Fixed Size List. It is common in applications for Machine Learning and AI.

See: #929

Enhancements

1. Use PyArrow filesystem for tabular file reads #939
2. [I/O] Port to pyarrow filesystems by default. #942
3. Memoize ray.get for batch metadata lookup #937
4. [I/O] Expose user-provided fsspec filesystem arg in read APIs. #931
5. Introduce Logical Arrays and SeriesLike Trait #920
6. [Extension Types] Add support for cross-lang extension types. #899

Bug Fixes

1. fix concats for extension array for old versions of pyarrow #944

Build Changes

1. [ci] enable pyrunner for 310 #946
2. Add Pyarrow 6.0 in matrix for CI testing #945
3. Update requirement of tabulate to >=0.9.0 #940
4. unpin numpy for 3.7 to get dependabot to stop complaining #938
5. Bump slackapi/slack-github-action from 1.23.0 to 1.24.0 #936
6. Bump hypothesis from 6.75.2 to 6.75.3 #928
7. Bump dask from 2023.4.1 to 2023.5.0 #927
8. Bump serde from 1.0.162 to 1.0.163 #921

Documentation

1. Add comment to explain future annotations isort rule in dataframe.py #947
2. [Embedding tutorial] Suggest running on GPU cluster #932
3. Embeddings tutorial #930

v0.1.1

Daft 0.1.1 Release Notes

The Daft 0.1.1 release provides bugfixes and stabilization fixes.

Enhancements

1. Enable and test writing temporal types #897
2. Fix universal expressions on temporals #895
3. [Arrow Types] Add automatic Python object fallback for unsupported Arrow types. #886

Bug Fixes

1. Fix fsspec multithreading clobbering issue #898
2. Fix temporal unit tests for older versions of pyarrow #919
3. Fix colon URL downloads and default to strict mode for .url.download() #896
4. [CI] Fix flaky Datasets integration test. #917
5. Import daft in local benchmarking unit tests #887
6. Fix imports in microbenchmarks #885

Build Changes

1. enable python 3.10 unit tests #915
2. Update pyo3 to 0.18.3 #914
3. Bump serde from 1.0.160 to 1.0.162 #912
4. Bump arrow2 from 0.17.0 to 0.17.1 #910
5. Bump actions/upload-artifact from 2 to 3 #909
6. Bump actions/download-artifact from 2 to 3 #907
7. Bump actions/setup-python from 3 to 4 #906
8. Enable dependabot for pip, cargo and github-actions #904
9. pinned deps for requirements-dev.txt #903

Documentation

1. Fix README.rst quickstart #925
2. Fix typo: CSV -> Parquet #893
3. Add initial technical architecture docs #890
4. Fix 10-min tutorial link to colab #884

v0.1.0

Daft 0.1.0 Release Notes

Welcome to the first “minor” version release of Daft!

We hope everyone has had a great time using our 0.0.* releases, but buckle up and grab a drink while you read these release notes, because we built so much over the past month and this new release is BIG!

A big shoutout to the contributors who made this all possible - with 21,716 added and 16,090 deleted lines of code!

@xcharleslin @clarkzinzow @jeevb @samster25 @jaychia @FelixKleineBoesing

Main Highlights

1. We rebuilt all of our core execution code in Rust - giving us a 2x speedup across the board for many of our benchmarks!
2. Our type system just levelled up! We have a much more sophisticated type system written in Rust that can handle nested types, parametrized types and type promotion semantics.
3. The UDF API is much cleaner now with the introduction of the Daft Series object!
4. Python object columns are now much more featureful with support for casting and magic methods

The full list of changes is much too long to present in this release notes, but here it is anyways.

Enhancements

Rust Execution Backend

Our execution code was refactored into Rust!

Previously Daft relied on a mix of NumPy, Polars, Pandas and PyArrow for executing logic. This was problematic for a few reasons:

1. Difficult to perform performance optimizations
2. Difficult to understand and manage memory allocation
3. Dependency hell!
4. Flaky null-handling and broadcasting semantics depending on the library we used

As of 0.1.0, Daft is now statically linked to the wonderful Arrow2 Rust library which we use for executing all our kernels.

This has several implications:

1. Daft now has a Python-binded Rust execution layer, mainly comprising of the Table, Series and Expression abstractions.
2. Daft is much faster (up to 2x in many cases, especially for our default multithreaded Python runner!) as GIL contention is no longer a bottleneck
3. Daft’s Python dependencies have been greatly reduced, and is much more lightweight!

On the user-facing API, most of these changes are completely transparent - i.e. you just got a massive speedup for free!

New Features

Enhanced Type System

Our type system just levelled up!

1. Nested types were added #802
2. Types are now much more granular (e.g. int64 vs uint32 vs int8…)
3. Types are automatically promoted when necessary during certain operations (e.g. adding a Null array and a Int64 array results in Int64!)

As a result, we have much cleaner support for Null array handling since the Null type can be correctly type-promoted with our new supertype semantics.

Deprecations

As our first minor release, several APIs have changed substantially that you should be aware of. Moving forward, Daft APIs will maintain much stricter backward compatibility semantics.

UDFs

UDFs are much cleaner in 0.1.0!

UDFs now no longer require up-front declaration of which arguments have to be Expressions, and what input types they are passed in as (list, numpy, arrow etc). Instead:

1. Inputs are always passed in as daft.series.Series objects and users can now easily convert this to the format they care about using Series.to_pylist(), Series.to_numpy() etc.
2. Which inputs are going to be daft.series.Series vs Python objects is inferred at runtime by checking which arguments a user passes in are Expressions.

For more information, consult: UDF User Guide

Typing

Our old typing APIs have changed - the definitive typing API is now found at daft.DataType.

If you are declaring types (for instance as return types for UDFs), you should now use the DataType.* constructor methods!

Input/Output APIs

Creation of DataFrames has been promoted to module-level functions!

Before:

from daft import DataFrame

df = DataFrame.read_csv(...)

After:

import daft

df = daft.read_csv(...)

This is a big improvement in useability (moving forward, Daft will try to make it as easy as possible to use us by just importing the top-level daft module).

For more information, please see: API Documentation for Input/Output.