How does Comet compare to Gluten? Are there any plans to integrate with Gluten?

[andygrove]

Comet and Gluten have similar architectures:

• They both replace Spark's physical plan with a custom plan
• They both use an Intermediate Representation to encode the plan. Gluten uses substrait, and Comet uses a simpler protobuf representation that is tightly coupled to Spark.
• Gluten typically delegates to Velox when used to accelerate Spark, and Comet delegates to DataFusion

The Comet project started before Gluten was an Apache project.

There are no plans to integrate Comet with Gluten as part of the Comet project.

However, there is no reason why Gluten could not add Comet/DataFusion as a backend query engine (DataFusion already supports Substrait, and Comet provides the Spark-compatible operators and expressions).

[xiaoxuandev]

Given that Gluten uses Substrait as an Intermediate Representation (IR) while Comet uses a protobuf, does Comet also need to support Substrait at some point? Since DataFusion already supports Substrait, would there be any benefit in aligning Comet with that approach for broader compatibility? Is there any plan?

[parthchandra]

There is no current plan for Substrait. There is however no reason why we could not move to Substrait in future. There are advantages in doing so, for example in supporting engines other than Spark, but that is a little further down the road.