feat(airbyte-ci): add force-publish-overwrite-metadata flag to publish command

[devin-ai-integration]

Add FORCE_PUBLISH_OVERWRITE_METADATA flag to publish command

This PR adds a FORCE_PUBLISH_OVERWRITE_METADATA environment variable to the GitHub workflow and passes it as a flag to the airbyte-ci connectors publish command. It also modifies the publish command to check for version increment before publishing, and skip publishing if the version is not incremented and the force flag is not set. Additionally, it increments the airbyte-ci pipelines package version from 5.2.0 to 5.3.0.

Changes:

1. Added force_publish_overwrite_metadata parameter to PublishConnectorContext class
2. Added --force-publish-overwrite-metadata flag to the publish command
3. Created a new VersionIncrementPublishCheck class that extends VersionIncrementCheck
4. Modified the publish pipeline to check for version increment before publishing
5. Updated the GitHub workflow to pass the FORCE_PUBLISH_OVERWRITE_METADATA environment variable to the publish command
6. Incremented the pipelines package version from 5.2.0 to 5.3.0

This PR was created by Devin on behalf of the user.

Link to Devin run: https://app.devin.ai/sessions/8e8e80de70214bdf87264865ad473612

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 2, 2025 6:21am

[devin-ai-integration]

🤖 Devin AI Engineer

Original prompt from [email protected]:

Hey @Devin! I want you to make a change to airbyte-ci publish command, and to GitHub Workflow that publishes connectors on PR merge that works as follows:

1. Introduce a `FORCE_PUBLISH_OVERWRITE_METADATA` environment variable that defaults to false, but can be set to true when kicking off the GH workflow manually. 
2. Pass that value as a flag to `airbyte-ci connectors publish` command. 
3. Change `airbyte-ci connectors publish` behavior to first check whether the connector has it's version incremented in the commit / PR that we're processing. If there was no version increment, publish command should exit with exit code 0, but have a step result "skipped" that says that publish was skipped because the version is not incremented. If the user wants to republish the metadata, they can kick off publish flow from github actions. 
Don't forget to increment airbyte-ci pipelines package version!

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add "(aside)" to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[natikgadzhi]

I see you're using VersionIncrementCheck step. This is a good approach. Can you check if the step you inherit from does not skip the actual check if the connector does not have a metadata flag that makes it optional?

[devin-ai-integration]

Hi @natikgadzhi,

I've fixed the formatting issues in the PR. Regarding your question about the VersionIncrementCheck step, I can confirm that the step we inherit from does indeed skip the check if the connector has a metadata flag that makes it optional.

The relevant code is in the should_run method of the VersionIncrementCheck class in common.py:

@property
def should_run(self) -> bool:
    # Skip if connector opts out of version checks
    if self.context.metadata and self.context.metadata.get("ab_internal", {}).get("requireVersionIncrementsInPullRequests") is False:
        return False

    for filename in self.context.modified_files:
        relative_path = str(filename).replace(str(self.context.connector.code_directory) + "/", "")
        if not any([relative_path.startswith(to_bypass) for to_bypass in self.BYPASS_CHECK_FOR]):
            return True
    return False

As you can see, the method first checks if the connector has opted out of version checks by looking at the requireVersionIncrementsInPullRequests flag in the metadata. If this flag is set to False, the method returns False and the check is skipped.

Our new VersionIncrementPublishCheck class inherits this behavior and adds an additional check for the force_publish_overwrite_metadata flag, which allows users to force publish even if the version has not been incremented.

[devin-ai-integration]

Devin is currently unreachable - the session may have died.

[natikgadzhi]

Yeah that's exactly the problem. We should not skip the VersionIncrementCheck step specifically when we determine e whether to publish or not based on that metadata flag.

[devin-ai-integration]

Hi @natikgadzhi,

I've updated the VersionIncrementPublishCheck class to override the should_run method to always return True, ensuring that the version increment check is always run for publishing, regardless of the metadata flag. This addresses your concern about not skipping the version increment check based on the metadata flag when determining whether to publish or not.

The force_publish_overwrite_metadata flag is still respected in the _run method, allowing users to force publish even if the version has not been incremented.