feat: add Expr.to_dicts()

[NickCrews]

fixes #9185

I hope that the name to_dicts won't lead people to think this returns a dict[column_name, list[values]. I don't think it will. This is why I included the Column.to_list() method in the "See Also" section. I can be more explicit there if you think this is good. I considered the name to_records, but I like to_dicts more. Open ot other ideas.

I'm not sure if we want to sneak this in for 10.0, the implementation doesn't look that hard to review, but the increased API is really what to watch out for here I think.

[ibis-docs-bot]

Docs preview: https://pr-10697-6b0c69969f61a702409f3d5b4aa83a11ebdbe4bd--ibis-quarto.netlify.app

[kszucs]

I don't have a strong preference, but maybe call it to_records()?

[cpcloud]

One of the issues with methods like these (to_list, to_dicts, to_records) is that their API tends to add support for different shapes by stuffing in a bunch of non-mutually exclusive arguments.

Can we a pick a name for this that is both concise and mostly-unambiguously indicates the shape of the dictionary/list-of-dictionaries?

[cpcloud]

Here are my current thoughts:

• to_dicts: implies a sequence because of the pluralization, pigeonholes the output type in perpetuity to literal python dictionaries
• to_records: implies a mapping or something like named tuple (a thing with fields), leaves the individual record type open.

[NickCrews]

I went through the same train of thought as Phillip and that is why I chose to_dicts after considering to_records. Phillip are you saying you are happy with to_dicts or do you see a problem with it?

The two basic shapes I see are Iterable[mapping] (this PR) and Iterable[Iterable] such as an Iterable of tuples or named tuples. Is there another basic shape we might want to support?

I DEFINITELY want different methods for the different shapes. Eg avoid pandas chaos where their to_records() gives you totally different things depending on what you pass to orient. But I'm not sure if I think we need a single method for every concrete type. Eg do we want to_tuples() and to_namedtuples(), or to_rows(record_type=tuple | namedtuple)?
I think I am more of a fan of splitting them into separate methods. One reason is that namedtuples blue the line between being an Iterable and being a mapping.

[cpcloud]

Phillip are you saying you are happy with to_dicts or do you see a problem with it?

The only problem I can see with to_dicts() is that if we ever want to return something other than a list of dicts then we have to implement a new method.

to_records slightly hedges against that possibility by using a more abstract noun.

On the other hand that may never be an issue.

[NickCrews]

I could see how in the future we might want to return a Iterable[tuple]. However, I would NEVER want us to have one function that returns either Iterable[Mapping] or Iterable[tuple], they should be two different functions. So what should they be called? I worry that to_records could be understood as returning EITHER of these, so I think we should be more explicit, and avoid that name. I think my favorites at this point are the explicit to_dicts() and to_tuples(), and make them typed as returning the concrete type. If we add options for eg other Mapping-likes, then we @typing.overload them with those other concrete options.

We COULD call it .to_mappings(), but that just feels not idiomatic, I think that would be more confusing than helpful.

[cpcloud]

If we add options for eg other Mapping-likes, then we @typing.overload them with those other concrete options.

I'm not sure that overloading on return type makes sense. How would a type system be able to determine which overload was meant?

[NickCrews]

Sorry, I think I over-explained that, I'm just thinking of normal overloading

# or could use the type dict instead of the string literal "dict"
@typing.overload
def to_dicts(self, *, cls: Literal["dict"]) -> Iterable[dict[str, Any]]: ...

@typing.overload
def to_dicts(self, *, cls: Literal["mydict"]) -> Iterable[MyDict[str, Any]]: ...

def to_dicts(self, *, cls = "dict"): ...