[HUDI-8817] fix backward compatibility issue #12615

Davis-Zhang-Onehouse · 2025-01-10T18:01:10Z

Change Logs

There are 3 configs at interplay:

table config hoodie.table.version - can be 6 or 8
writer config hoodie.write.table.version - can be 6 or 8
writer config hoodie.write.auto.upgrade - boolean flag to control transition during upgrade, avoiding unintended upgrades.
hoodie.table.version reflects the actual state of the table.
hoodie.write.table.version defines the intended state of data written by the writer.

During upgrade, the initial phase for writers should be configured with hoodie.write.auto.upgrade=false and hoodie.write.table.version=6 until when users are ready to start writing in the new format (i.e. after all readers have been upgraded).

For creating new tables using 1.0, by default i.e. when none of the writer configs are set, hoodie.table.version should be 8. hoodie.write.table.version is automatically inferred to be 8.

With my change that explicitly set the write version, when user does not override the config, the version would be 8 automatically.

For creating new tables using 1.0, if there is a need to create one with older format, then writers should be configured with hoodie.write.auto.upgrade=false and hoodie.write.table.version=6. In this case, hoodie.table.version should also be 6.

It means in hudi 1.0, if a user wants to create version 6 table, they would set the config above and the system should honor that configuration in metaClient.initTable. This is why this change enforces when metaClient is created, we set the table version to write version. This is what is missing without the change

when we specify write table version as 6

CREATE TABLE hudi_14_table_sql_005 (
    event_id INT,
    event_date STRING,
    event_name STRING,
    event_ts STRING,
    event_type STRING
) USING hudi
 OPTIONS(
    type = 'cow', -- or 'mor'
    primaryKey = 'event_id,event_date',
    preCombileField = 'event_ts',
    hoodie.write.table.version = 6
)
PARTITIONED BY (event_type)
LOCATION 's3://<some_bucket>/hudi_14_table_sql_005';

the hoodie.properties said the version is 8.

It is because when building the meta client table version is not set according to the write table version. Found similar misses else where, fixed as well with test coverage.

Impact

hoodie.write.table.version will also contribute to hoodie.table.version.

Risk level (write none, low medium or high below)

none

Documentation Update

none

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

yihua · 2025-01-11T00:42:59Z

hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java

@@ -1271,10 +1272,24 @@ public TableBuilder fromProperties(Properties properties) {
        setTableName(hoodieConfig.getString(HoodieTableConfig.NAME));
      }

-      if (hoodieConfig.contains(VERSION)) {
+      if (hoodieConfig.contains(VERSION) && hoodieConfig.contains(WRITE_TABLE_VERSION)) {


The logic in meta client should not consider write table version. Otherwise, there is no point checking whether hoodie.write.table.version and hoodie.table.version match.

CommonClientUtils#validateTableVersion has the validation. Does Spark SQL writer escape that?

yes, SQL writer does not call this function at all for create table stmt

hudi-bot · 2025-01-18T01:37:32Z

CI report:

640ffe5 UNKNOWN
a64c849 UNKNOWN
c887fde UNKNOWN
9bd1a1d Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

nsivabalan

@danny0405 : can you ensure we are covering all meta client instantiation flows in flink

nsivabalan

from what I know, we can;t let users directly directly dictate the table version.
based on our migration guide, they can only configure "hoodie.write.auto.upgrade"
ref: https://hudi.apache.org/docs/deployment#upgrading-to-100

@yihua @codope : can you clarify this.

nsivabalan

blocking on clarification

Davis-Zhang-Onehouse · 2025-01-24T04:21:44Z

blocking on clarification

we discussed offline, let me know any AIs I should follow or please resume the review. Thanks

nsivabalan · 2025-01-24T18:27:35Z

unblocking for now.
goal for this patch (open to splitting it up into multiple):

for new table creation, we should default to table version 8.
for new table creation, if user overrides "hoodie.write.table.version", we will honor the overridden value.

We need to validate and may be fix all writers (spark-ds, spark-sql, deltastreamer and spark structured streaming ingestion)

nsivabalan · 2025-01-24T18:31:23Z

Deltastreamer :

hudi/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java

Line 421 in ef5b9ca

    
           HoodieTableMetaClient initializeEmptyTable(HoodieTableMetaClient.TableBuilder tableBuilder, String partitionColumns,

hudi/hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java

Line 428 in ef5b9ca

public void testTableCreation() throws Exception {

spark-sql and spark-ds: guess this patch already have tests.

spark-streaming:

hudi/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestStructuredStreaming.scala

Line 208 in ef5b9ca

def testStructuredStreaming(tableType: HoodieTableType): Unit = {

danny0405 · 2025-01-25T03:14:15Z

...spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala

@@ -302,6 +303,7 @@ class HoodieSparkSqlWriterInternal {
          else KeyGeneratorType.getKeyGeneratorClassName(hoodieConfig)
        HoodieTableMetaClient.newTableBuilder()
          .setTableType(tableType)
+          .setTableVersion(Integer.valueOf(getStringWithAltKeys(parameters, HoodieWriteConfig.WRITE_TABLE_VERSION)))


Isn't table version and write table version two different config options, not sure why setting up the table version always from the write table version, the write table version is auxiliary for migration, I kind of think we should not set it up explicitly every time we initiate the table. can we detect the table version automically from the table config
and validate against the write table version then?

danny0405

We need to clarify the coordination of table version of write table version, confusing to me.

Davis-Zhang-Onehouse · 2025-01-25T05:37:56Z

Deltastreamer :

hudi/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java

Line 421 in ef5b9ca

HoodieTableMetaClient initializeEmptyTable(HoodieTableMetaClient.TableBuilder tableBuilder, String partitionColumns,

hudi/hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java

Line 428 in ef5b9ca

public void testTableCreation() throws Exception {

spark-sql and spark-ds: guess this patch already have tests.

spark-streaming:

hudi/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestStructuredStreaming.scala

Line 208 in ef5b9ca

def testStructuredStreaming(tableType: HoodieTableType): Unit = {

created jira 1.0.2 https://issues.apache.org/jira/browse/HUDI-8914

codope · 2025-01-25T11:32:21Z

There are 3 configs at interplay:

table config hoodie.table.version - can be 6 or 8
writer config hoodie.write.table.version - can be 6 or 8
writer config hoodie.write.auto.upgrade - boolean flag to control transition during upgrade, avoiding unintended upgrades.

hoodie.table.version reflects the actual state of the table.
hoodie.write.table.version defines the intended state of data written by the writer.

During upgrade, the initial phase for writers should be configured with hoodie.write.auto.upgrade=false and hoodie.write.table.version=6 until when users are ready to start writing in the new format (i.e. after all readers have been upgraded).

For creating new tables using 1.0, by default i.e. when none of the writer configs are set, hoodie.table.version should be 8. hoodie.write.table.version is automatically inferred to be 8.

For creating new tables using 1.0, if there is a need to create one with older format, then writers should be configured with hoodie.write.auto.upgrade=false and hoodie.write.table.version=6. In this case, hoodie.table.version should also be 6.

danny0405 · 2025-01-25T11:52:43Z

For creating new tables using 1.0, by default i.e. when none of the writer configs are set, hoodie.table.version should be 8. hoodie.write.table.version is automatically inferred to be 8.

For creating new tables using 1.0, if there is a need to create one with older format, then writers should be configured with hoodie.write.auto.upgrade=false and hoodie.write.table.version=6. In this case, hoodie.table.version should also be 6.

So it looks like there is even no need to explicitly set up the table version when creating new table with 1.0 code, just hard code it as 8 should be fine.

codope · 2025-01-25T14:53:42Z

So it looks like there is even no need to explicitly set up the table version when creating new table with 1.0 code, just hard code it as 8 should be fine.

By default, we don't need to. Only when users want to write in older format with 1.0, they need to explicitly set the writer table version and auto upgrade

Davis-Zhang-Onehouse · 2025-01-27T16:44:43Z

@danny0405

For creating new tables using 1.0, if there is a need to create one with older format, then writers should be configured with hoodie.write.auto.upgrade=false and hoodie.write.table.version=6. In this case, hoodie.table.version should also be 6.

It means in hudi 1.0, if a user wants to create version 6 table, they would set the config above and the system should honor that configuration in metaClient.initTable. This is why this change enforces when metaClient is created, we set the table version to write version. This is what is missing without the change

For creating new tables using 1.0, by default i.e. when none of the writer configs are set, hoodie.table.version should be 8. hoodie.write.table.version is automatically inferred to be 8.

With my change that explicitly set the write version, when user does not override the config, the version would be 8 automatically.
For sake of unblocking the change soon by 101 cut off which is today
@nsivabalan / @codope / @yihua does it make sense that we merge the PR first. If later Danny has any follow ups we can discuss tonight. Based on all the discussion so far I only see minor communication gap and we are heading in the right direction.

nsivabalan

since we are aligned here, I am going ahead for 1.0.1. Config clean up for (write table version and tbl version) and doc updates we can take it up as a follow up.

@danny0405 @yihua : if you have any follow ups that needs to go into 1.0.1, do speak up :)

Going ahead with the changes. We will have follow ups for any ask from danny

github-actions bot added the size:M PR with lines of changes in (100, 300] label Jan 10, 2025

Davis-Zhang-Onehouse force-pushed the HUDI-8817 branch from 256064d to 640ffe5 Compare January 10, 2025 18:34

github-actions bot added size:L PR with lines of changes in (300, 1000] and removed size:M PR with lines of changes in (100, 300] labels Jan 10, 2025

Davis-Zhang-Onehouse force-pushed the HUDI-8817 branch 2 times, most recently from a64c849 to 944075b Compare January 10, 2025 19:02

github-actions bot added size:M PR with lines of changes in (100, 300] and removed size:L PR with lines of changes in (300, 1000] labels Jan 10, 2025

yihua previously requested changes Jan 11, 2025

View reviewed changes

Davis-Zhang-Onehouse marked this pull request as draft January 12, 2025 18:39

Davis-Zhang-Onehouse force-pushed the HUDI-8817 branch from 944075b to c887fde Compare January 17, 2025 23:36

Make sure table version is set when creating metaclient

9bd1a1d

Davis-Zhang-Onehouse force-pushed the HUDI-8817 branch from c887fde to 9bd1a1d Compare January 17, 2025 23:40

Davis-Zhang-Onehouse marked this pull request as ready for review January 17, 2025 23:40

nsivabalan reviewed Jan 22, 2025

View reviewed changes

nsivabalan requested changes Jan 22, 2025

View reviewed changes

nsivabalan approved these changes Jan 24, 2025

View reviewed changes

danny0405 reviewed Jan 25, 2025

View reviewed changes

danny0405 previously requested changes Jan 25, 2025

View reviewed changes

nsivabalan approved these changes Jan 27, 2025

View reviewed changes

nsivabalan merged commit bc5566d into apache:master Jan 27, 2025
46 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HUDI-8817] fix backward compatibility issue #12615

[HUDI-8817] fix backward compatibility issue #12615

Davis-Zhang-Onehouse commented Jan 10, 2025 •

edited

Loading

yihua Jan 11, 2025

Davis-Zhang-Onehouse Jan 11, 2025

hudi-bot commented Jan 18, 2025

nsivabalan left a comment

nsivabalan left a comment

nsivabalan left a comment

Davis-Zhang-Onehouse commented Jan 24, 2025

nsivabalan commented Jan 24, 2025

nsivabalan commented Jan 24, 2025

danny0405 Jan 25, 2025

danny0405 left a comment

Davis-Zhang-Onehouse commented Jan 25, 2025

codope commented Jan 25, 2025 •

edited

Loading

danny0405 commented Jan 25, 2025

codope commented Jan 25, 2025

Davis-Zhang-Onehouse commented Jan 27, 2025

nsivabalan left a comment

[HUDI-8817] fix backward compatibility issue #12615

[HUDI-8817] fix backward compatibility issue #12615

Conversation

Davis-Zhang-Onehouse commented Jan 10, 2025 • edited Loading

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

yihua Jan 11, 2025

Choose a reason for hiding this comment

Davis-Zhang-Onehouse Jan 11, 2025

Choose a reason for hiding this comment

hudi-bot commented Jan 18, 2025

CI report:

nsivabalan left a comment

Choose a reason for hiding this comment

nsivabalan left a comment

Choose a reason for hiding this comment

nsivabalan left a comment

Choose a reason for hiding this comment

Davis-Zhang-Onehouse commented Jan 24, 2025

nsivabalan commented Jan 24, 2025

nsivabalan commented Jan 24, 2025

danny0405 Jan 25, 2025

Choose a reason for hiding this comment

danny0405 left a comment

Choose a reason for hiding this comment

Davis-Zhang-Onehouse commented Jan 25, 2025

codope commented Jan 25, 2025 • edited Loading

danny0405 commented Jan 25, 2025

codope commented Jan 25, 2025

Davis-Zhang-Onehouse commented Jan 27, 2025

nsivabalan left a comment

Choose a reason for hiding this comment

Davis-Zhang-Onehouse commented Jan 10, 2025 •

edited

Loading

codope commented Jan 25, 2025 •

edited

Loading