ddl_puller.go(ticdc): fix DDLs are ignored when schema versions are out of order (#11733)

[ti-chi-bot]

This is an automated cherry-pick of #11733

What problem does this PR solve?

Issue Number: close #11714

What is changed and how it works?

Description:
This pull request addresses a bug in TiCDC where a DDL job could be inadvertently dropped if a subsequent DDL job with a higher SchemaVersion but lower CommitTs is processed first. This behavior occurs because the ddlJobPuller processes DDL jobs based on their CommitTs in ascending order and uses a check against SchemaVersion that can prevent older DDL jobs from being processed, leading to data inconsistencies.

Background:
The issue manifests when two DDL jobs are processed:

• Job 60: ALTER TABLE a ADD COLUMN x, y, z with CommitTs 400
• Job 62: ALTER TABLE b ADD COLUMN y with CommitTs 300

In this scenario, Job 62 is processed first due to its lower CommitTs. The ddlJobPuller then updates its SchemaVersion to that of Job 62. When Job 60 is subsequently processed, the current logic discards it because its SchemaVersion is deemed older, even though its CommitTs is higher.

This issue occurs because the SchemaVersion increment and job metadata write to TiKV are separate transactions. During a TiDB owner change, different instances might write these transactions without synchronization, leading to potential out-of-order CommitTs relative to SchemaVersion.

Solution:
To resolve this, the check on SchemaVersion is removed, ensuring that only the CommitTs is verified:

Updated Code:

if job.BinlogInfo.FinishedTS <= p.getResolvedTs() {
    log.Info("ddl job finishedTs less than puller resolvedTs," +
        "discard the ddl job",
        zap.String("namespace", p.changefeedID.Namespace),
        zap.String("changefeed", p.changefeedID.ID),
        zap.String("schema", job.SchemaName),
        zap.String("table", job.TableName),
        zap.Uint64("startTs", job.StartTS),
        zap.Uint64("finishedTs", job.BinlogInfo.FinishedTS),
        zap.String("query", job.Query),
        zap.Uint64("pullerResolvedTs", p.getResolvedTs()))
    return true, nil
}

Reasoning:
The ResolvedTs check ensures that only DDL jobs with FinishedTS greater than the current ResolvedTs are processed. Since the ddlJobPuller receives DDLs sorted by CommitTs, any new DDL received with a FinishedTS greater than ResolvedTs must be handled, making the SchemaVersion check redundant.

Check List

Tests

• Unit test
• Integration test

Questions

Will it cause performance regression or break compatibility?

No.

Do you need to update user documentation, design documentation or monitoring documentation?

No.

Release note

Fix the issue that DDL jobs could be incorrectly discarded in TiCDC when their schema versions were not in a strictly linear order with commit timestamps due to TiDB owner changes.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign wk989898 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

@ti-chi-bot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
jenkins-ticdc/verify	383d30e	link	true	/test verify
pull-cdc-integration-kafka-test	383d30e	link	true	/test cdc-integration-kafka-test
pull-cdc-integration-mysql-test	383d30e	link	true	/test cdc-integration-mysql-test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.