bug: PySpark write streams failing

[christophediprima]

What happened?

I am trying to use the spark backend to read from and write to kafka but with no success.

The to_kafka method fails when calling sql with the expression.

ibis/ibis/backends/pyspark/__init__.py

Line 1108 in 715d41e

df = self._session.sql(self.compile(expr, params=params, limit=limit))

pyspark.errors.exceptions.connect.AnalysisException: Queries with streaming sources must be executed with writeStream.start();

I am not sure you can createOrReplaceTempView out of a stream dataframe and query it with spark.sql.

What version of ibis are you using?

10.0.0

What backend(s) are you using, if any?

PySpark with connect

Relevant log output

python3.11/site-packages/pyspark/sql/connect/client/core.py", line 1539, in _handle_rpc_error
    raise convert_exception(info, status.message) from None
pyspark.errors.exceptions.connect.AnalysisException: Queries with streaming sources must be executed with writeStream.start();
kafka

Code of Conduct

• I agree to follow this project's Code of Conduct