Switch RDD bindings to using inline-java under the hood. #103
Conversation
Using inline-java slows down the compilation of sparkle, but is safer because we can thus get the benefit of *both* type checkers (Java and Haskell). In fact the extra safety isn't just theoretical: this patch also includes a fix to the binding for `treeAggregate`, which was supplying arguments in the wrong order. This is preliminary work ahead of implementing #57, which we can do serenely from the moment that the type checkers have our back. This patch only switches over RDD for now. The rest can come later.
|
This is not easy to have working on multinode setups unless inline-java is updated. For one of our projects we addressed this by not embedding generated bytecode in executables. Will submit the PR for this as soon as I can. |
Could you clarify what the issue is? Embedding the bytecode simplifies packaging/deployment quite a bit, and helps with modularity, so I'd be loath to abandon it. Note that the bindings don't require anonymous classes, even if user code might. |
|
When a executor receives a serialized object, it needs to load the class before decoding the object is possible. As only the driver executes |
|
On second check, none of the code introduced in this PR seems to be sending to the executors objects of classes defined with inline-java. So the problem I was discussing does not affect usage in this case, as you well noted. Excuse me for the noise. |
|
Right. I'd still be keen to see a ticket describing the issue you mention about inline-java in the general case though. |
|
@alpmestan Heh, surely not. ;) You're the author of most of the original bindings in this module, so Github rightly suggested to give you a shout. |
Using inline-java slows down the compilation of sparkle, but is safer
because we can thus get the benefit of both type checkers (Java and
Haskell). In fact the extra safety isn't just theoretical: this patch
also includes a fix to the binding for
treeAggregate, which wassupplying arguments in the wrong order.
This is preliminary work ahead of implementing #57, which we can do
serenely from the moment that the type checkers have our back.
This patch only switches over RDD for now. The rest can come later.