[Feature] Extend EXECUTION_MODE.AIRFLOW_ASYNC for dbt-databricks adapter

[pankajkoti]

Description

Continuing the work introduced in Cosmos 1.7 to support executing dbt project resources in a deferrable fashion using Airflow provider operators, this feature aims to extend EXECUTION_MODE.AIRFLOW_ASYNC to the dbt-databricks adapter.

Initially, this approach was implemented for BigQuery and refined through PRs #1474 and #1483 to ensure robustness. The next step is to explore how we can extend this functionality to Databricks, leveraging its capabilities to execute dbt models asynchronously while integrating seamlessly with Airflow’s deferrable operators.

Explore how we can utilize existing deferrable operators from the Apache Airflow Databricks provider or astronomer-providers. If suitable operators are unavailable, consider adding the necessary components to the Cosmos codebase to enable deferrable execution efficiently and accelerate development.

Use case/motivation

This enhancement will improve efficiency, reduce resource consumption, and provide users with a more scalable and optimized way to run dbt projects on Databricks within Airflow.

Related issues

No response

Are you willing to submit a PR?

• Yes, I am willing to submit a PR!