[MINOR] pass the avro exception for better information

[prabodh1194]

Change Logs

In a few cases, Avro exception is pretty good. Highlighting the error message instead of suppressing it is more informational.

e.g.: In this spark connect example, there is a duplicate column error. But the error is visible on the driver. Not on the python client. This PR will bubble the error message proactively and improve the UX.

from pyspark.sql.session import SparkSession

sp = (
    SparkSession.builder.appName("Hudi").remote("sc://0.0.0.0").getOrCreate()
)

df_1 = sp.createDataFrame([
    (1, "foo"), (2, "bar"), (3, "baz")
], ["id", "name"])

df_2 = sp.createDataFrame([
    (1, "foo"), (2, "bar"), (3, "baz")
], ["id", "name"])

df = df_1.join(df_2, on="id")

df.write.format("hudi").option(
    "hoodie.table.name", "hudi_table"
).mode("overwrite").save("/tmp/hudi")

Error with this patch [more readable]

pyspark.errors.exceptions.connect.SparkConnectGrpcException: 
(org.apache.hudi.internal.schema.HoodieSchemaException) 
Duplicate field name in record hoodie.hudi_table.hudi_table_record: name type:UNION pos:2 and name type:UNION pos:1.

Error without this patch

pyspark.errors.exceptions.connect.SparkConnectGrpcException: 
(org.apache.hudi.internal.schema.HoodieSchemaException) 
Failed to convert struct type to avro schema: StructType(StructField(id,LongType,true),StructField(name,StringType,true),StructField(name,StringType,true))

Impact

NA

Risk level (write none, low medium or high below)

none

Documentation Update

na

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[prabodh1194]

i don't use hudi-1.0.0 yet, so raised PR against 0.15 only.

[danny0405]

Thanks for the contribution, can we make this change against master and then cherry pick to 0.x-branch if necessary. The 0.x-branch is the branch for new releases of 0.x Hudi.

[prabodh1194]

sure

[prabodh1194]

@danny0405 I have addressed your comments now. thank you.

[danny0405]

Thanks, I have re-trigger the Azure CI tests against the failures.

[prabodh1194]

@danny0405 do I need to do something to get this CI to pass?

[hudi-bot]

CI report:

• cba5757 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[prabodh1194]

@danny0405 all checks are green now 🎉

[prabodh1194]

@wombatu-kun can you please merge this to https://github.com/apache/hudi/tree/release-0.15.0 as well.

#11981