Basic Support for Grouping/Aggregation

[bsowell]

We need some basic support for grouping/aggregation. One use case that has come up recently: It is sometimes easier to process documents by page. For example, you might want to to split the document by page and convert each page to an image for use with an image model. In Sycamore you might accomplish this by partitioning by page and then running explode. After processing each page, you might want to "regroup" the results by original document. You may want to be able to combine the text/binary representation of the documents, the elements, and/or the properties.

[mashah]

This sound more like map-reduce rather than group-by-agg. The regrouping is really a mapping, and the reduce may something as simple as adding to a set.

Other uses of this is for clustering, where similar documents are clustered to then pass to AI for some amount of generation work.

[bsowell]

That's fair. I'm not sure that Ray exposes a public reduce function, so it was easier to wrap their group-by aggregate, but once I finish a POC I'll take a step back and see if there is a way to generalize. Internally I believe Ray has effectively implemented MapReduce, but I just need to understand what abstractions we can use and what we want to expose.