[ENH] Polars adapter enhancements

[julian-fong]

adds index support as part of #440 and is used to sync up polars conversion utilities between skpro and sktime.

Correponding sktime pr for polars conversion utilities is sktime/sktime#6455.

In this pr:

If a pandas Dataframe is a from_type and polars frame is a to_type then during the conversion, we will save the index (assumed never to be in multi-index format) and insert it as an individual column with column name __index__. Then the resulting pandas dataframe will be converted to a polars dataframe.

In the inverse function, if we are converting from polars dataframe to pandas dataframe, if the column __index__ exists in the pandas dataframe post-conversion, then we will map that column to the index before returning the pandas Dataframe

After this is merged, #447 will be implemented as a polars only estimator. tests will also be written to check polars input end to end and pandas input and output through the polars estimator (i.e pandas input into polars estimator -> polars predictions -> pandas output)

[pranavvp16]

Left some questions about how is index name being handled in the conversion utils. The code will get the job done currently, but suppose I convert a pandas DataFrame having a index with a name to polars, and then convert it back to pandas the index name would be lost.

[pranavvp16]

How about we retain the index name too. This code won't work for data frames that already have a name for their index. Does skpro assumes that data frame will always have a default RangeIndex without any name @fkiraly @julian-fong ??, there may be scenarios where a index having name is passed and the conversion util will fail in this case.

How sktime handles this is by appending the name infront of the __index__ convention. So suppose if the DataFrame has a index named 1 the util will convert it to __index__1 in polars frame.

[julian-fong]

thats a good point, i'll make that change then

[fkiraly]

Yes, I thought that's what is happening in the dask converter as well, is it?

[julian-fong]

Tests are failing in jobs because the __index__ is being returned and there is no easy way to get rid of it. i.e

┌───────────┬──────┐
│ __index__ ┆ a    │
│ ---       ┆ ---  │
│ i64       ┆ f64  │
╞═══════════╪══════╡
│ 0         ┆ 1.0  │
│ 1         ┆ 4.0  │
│ 2         ┆ 0.5  │
│ 3         ┆ -3.0 │
└───────────┴──────┘

is returned when it expects

┌──────┐
│ a    │
│ ---  │
│ f64  │
╞══════╡
│ 1.0  │
│ 4.0  │
│ 0.5  │
│ -3.0 │
└──────┘

@fkiraly any preferred method on how to solve this?

[pranavvp16]

@julian-fong I know why the tests are failing here, you need to change the polars fixture example to match this convention, in skpro the examples are located at skpra/datatypes/_table/_examples.py. What you can do here is import the conversion util and convert the pandas DataFrame that is being used

[fkiraly]

Tests are failing in jobs because the __index__ is being returned and there is no easy way to get rid of it. i.e

┌───────────┬──────┐
│ __index__ ┆ a    │
│ ---       ┆ ---  │
│ i64       ┆ f64  │
╞═══════════╪══════╡
│ 0         ┆ 1.0  │
│ 1         ┆ 4.0  │
│ 2         ┆ 0.5  │
│ 3         ┆ -3.0 │
└───────────┴──────┘

is returned when it expects

┌──────┐
│ a    │
│ ---  │
│ f64  │
╞══════╡
│ 1.0  │
│ 4.0  │
│ 0.5  │
│ -3.0 │
└──────┘

@fkiraly any preferred method on how to solve this?

My answer would be to not include an __index__ if it is a RangeIndex, that is:

If you convert from pd_DataFrame_Table to the polars type:

1. check if index is RangeIndex
2. if yes, include no __index__ column; otherwise, include index as __index__[indexname] column.

[fkiraly]

Looks good - @pranavvp16, can you kindly review this, too?

[julian-fong]

Tests are failing in jobs because the __index__ is being returned and there is no easy way to get rid of it. i.e

┌───────────┬──────┐
│ __index__ ┆ a    │
│ ---       ┆ ---  │
│ i64       ┆ f64  │
╞═══════════╪══════╡
│ 0         ┆ 1.0  │
│ 1         ┆ 4.0  │
│ 2         ┆ 0.5  │
│ 3         ┆ -3.0 │
└───────────┴──────┘

is returned when it expects

┌──────┐
│ a    │
│ ---  │
│ f64  │
╞══════╡
│ 1.0  │
│ 4.0  │
│ 0.5  │
│ -3.0 │
└──────┘

@fkiraly any preferred method on how to solve this?

My answer would be to not include an __index__ if it is a RangeIndex, that is:

If you convert from pd_DataFrame_Table to the polars type:

1. check if index is RangeIndex
2. if yes, include no __index__ column; otherwise, include index as __index__[indexname] column.

Forgot to include async discussion but it is summarized as follows.

In the code: we will only ignore the addition of __index__ if the index of the pandas DataFrame is the trivial index i.e RangeIndex(0,n_rows)

Otherwise, we will build a new column called __index__* using the reset_index() function built in pandas.

[pranavvp16]

Looks good - @pranavvp16, can you kindly review this, too?

LGTM!

[julian-fong]

Thank you for the reviews!