Using YDF in databricks with spark

[marcoreuter]

Hi!

I really love using your package, but I am using databricks with spark. Is there any way to make the distributed training of the package work in this environment currently? I've been trying, but with no luck. Are there plans for a future integration?

Thank you!

[rstz]

Hi,

we haven't tried yet, but we're actively working on improving distributed training support. I'll keep this open and report any news / experiments we can share.

[DawnweiSong]

In case anyone else like me encountering the same problem, here is my solution in DataBricks DCS 13.3:

!mkdir -p /tmp/xsong
!wget -O /tmp/xsong/ydf-0.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl https://files.pythonhosted.org/packages/ea/00/f36d107ce13552008b5f30d1b2348528cb97944c2f1247dfd913cc3f91b1/ydf-0.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
%pip install /tmp/xsong/ydf-0.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
%pip install --upgrade protobuf
dbutils.library.restartPython()