Databricks

Databricks was founded in 2013 by the original creators of Apache Spark and later Delta Lake and MLflow. It introduced the lakehouse architecture — combining the low-cost open storage of a data lake with the performance, governance, and ACID transaction support of a data warehouse, without forcing data duplication across separate systems.

Delta Lake is the open-source storage layer at the core of the Databricks lakehouse. It adds ACID transactions, schema enforcement, time travel, and change data feed to data stored in Parquet format on cloud object stores (S3, ADLS, GCS). All Databricks tables default to Delta format, and Delta Lake is fully open-source and usable outside Databricks.

Databricks Workflows provides a native orchestration layer for scheduling and monitoring data pipelines, notebooks, Spark jobs, dbt runs, and ML training jobs with dependency management, retry logic, and observability.

Delta Live Tables (DLT) is a declarative ETL framework where engineers define transformation logic and Databricks automatically manages pipeline execution, dependency ordering, incremental processing, and data quality expectations (quarantine bad records).

Databricks SQL is a serverless SQL analytics layer optimised for BI queries on lakehouse tables. It supports standard SQL syntax, materialised views, streaming tables, and integrates with Tableau, Power BI, Looker, and other BI tools via JDBC/ODBC.

MLflow (open source, originated at Databricks) is the de facto standard for ML experiment tracking, model registry, and deployment. Databricks provides a managed MLflow service integrated with the platform.

Mosaic AI (rebranding of Databricks AI) covers model serving (deploy any registered MLflow model as a REST endpoint), AI gateway, agent frameworks, and fine-tuning. Databricks offers Foundation Model APIs — hosted access to open-source LLMs (Llama, Mistral, DBRX) without self-managing GPU infrastructure.

Unity Catalog is Databricks' unified governance layer: a single metastore for tables, files, ML models, and dashboards across all workspaces, cloud providers, and regions, with column-level lineage, attribute-based access control, and automated data classification.

DBU (Databricks Unit) is the billing dimension. Every workload (Jobs, SQL, ML, Streaming) consumes DBUs at a rate depending on the cluster type and tier. Standard tier: $0.07–$0.22/DBU; Premium tier: $0.13–$0.40/DBU; compute costs (cloud VM or serverless) are charged separately.

Databricks runs on the customer's cloud account (AWS, Azure, GCP) — the compute and storage infrastructure stays in the customer's VPC/subscription, while Databricks manages the control plane.