Databricks Lakehouse Pipeline

Not necessarily; Databricks Workflows covers scheduling and orchestration natively. Airflow is worth adding when the pipeline also needs to coordinate steps outside Databricks that Workflows doesn't reach.

Mainly when the workload genuinely needs Spark-scale distributed processing or unifies large-scale ML training with the data pipeline in one platform. A pure BI/reporting workload without heavy ML or big-data processing is usually cheaper on a warehouse-plus-dbt stack instead.

Usually because scheduled work runs on the wrong kind of compute. All-purpose clusters, the ones used interactively from notebooks, bill at a much higher DBU rate than job compute, so a nightly pipeline pointed at a shared all-purpose cluster costs several times what the same job costs on its own job cluster or on serverless jobs compute. Two more common leaks: interactive clusters left running because auto-termination was set long or turned off, and SQL warehouses sized for peak load that never auto-stop. Run every scheduled Spark job and dbt run on job or serverless compute, set short auto-termination on notebook clusters, and review the billing breakdown per compute type monthly.

A common split follows the lakehouse layers. Spark jobs in Python handle ingestion and the heavy lifting: reading raw files or streams, parsing semi-structured data, deduplicating large volumes, and anything that needs Python libraries. dbt takes over once data sits in clean Delta tables, building the business models analysts query as SQL with tests and documentation, running on a Databricks SQL warehouse through the dbt-databricks adapter. The orchestrator then runs them in order: Spark ingestion first, dbt after. Keeping business logic in dbt means analysts can read and change it without touching Spark code.

More than with most platforms, as long as you know where the line is. The data sits in Delta tables in your own cloud storage account, and Delta is an open format that Spark elsewhere, Trino, DuckDB, and other engines can read. Plain PySpark code and dbt models also move with modest changes. What doesn't move is the Databricks-specific layer: Unity Catalog permissions and lineage, scheduled Databricks jobs, declarative pipeline definitions, notebook-only code, and the performance of Databricks' own query engine. Keeping transformations in dbt and orchestration in Airflow, as this stack does, keeps most of the pipeline outside that layer.