Airflow Self-Hosted
IntermediateData PipelineSelf-hosted Apache Airflow: the standard data-pipeline scheduler on your infrastructure.
Published 27 September 2026
About Airflow Self-Hosted
Airflow Self-Hosted runs Apache Airflow on your own server from the project's official Docker Compose file. It starts the full Airflow 3 platform: the API server behind the UI, the scheduler that triggers every DAG run, a separate DAG processor that parses your pipeline files, a Celery worker and a triggerer for execution, Redis as the queue between them, and PostgreSQL as the metadata database for DAGs, runs, and task history.
Airflow is the de facto standard orchestrator in data engineering. Pipelines are plain Python, with retries, backfills, and dependency ordering handled by the scheduler instead of a cron table, and hundreds of provider packages cover AWS, GCP, dbt, Spark, and nearly every warehouse a pipeline touches.
Self-hosting keeps pipeline code and run history private and off the managed tiers: Astronomer, Amazon MWAA, and Google Cloud Composer all price by workers and environment size. Pipelines on your own server can also reach internal databases over a private network without a cloud egress hop.
The trade is that Airflow is a scheduler you operate. Airflow's docs present the Compose file as a quick-start without production security guarantees, so a real deployment changes the default login and secret keys, puts TLS in front of port 8080, and plans for upgrades, worker capacity, and log retention. Teams that want plain Python with a lighter footprint should compare Prefect Self-Hosted.
Key Features
- ✓Official Docker Compose deployment: API server, scheduler, DAG processor, Celery worker, and triggerer
- ✓PostgreSQL metadata database for DAGs, runs, and task history
- ✓Redis as the message queue between scheduler and workers
- ✓Pipelines as plain Python with retries, backfills, and dependency ordering built in
- ✓REST API for triggering and watching runs from other systems
- ✓Hundreds of provider packages for AWS, GCP, dbt, Spark, and more
When to Use Airflow Self-Hosted
- →Data teams scheduling ELT and warehouse transformations as Python DAGs
- →Pipelines that must reach internal databases on a private network
- →Organizations replacing a managed Airflow bill with a fixed server
- →Machine-learning teams orchestrating training and batch-scoring jobs
- →Engineering teams consolidating cron jobs into one observable scheduler
Pros
- The most battle-tested open-source orchestrator in data engineering
- Python-native: no YAML DSL, and every integration is importable code
- A provider ecosystem no other orchestrator matches
Cons
- Operationally heavy: a multi-service deployment you harden, operate, and upgrade
- The official Compose file is a quick-start, not a production-hardened setup
- The UI is engineer-facing, not a tool for stakeholders
- Executor and worker sizing decisions arrive early and are hard to undo
Database Options for Airflow Self-Hosted
Airflow Self-Hosted with PostgreSQL
Airflow's recommended metadata database and the Compose default. It holds every DAG run, task instance, XCom value, and connection, and it grows with run history, so schedule backups and prune old records with airflow db clean.
Airflow Self-Hosted with MySQL
Airflow supports MySQL 8 for the metadata database. Choose it where your platform standard already runs MySQL, and accept that the Compose file, the Helm chart, and most deployment guides assume PostgreSQL, so you'll adapt the configuration yourself.
These are highlighted picks. To see all the tools, check the Databases category.
Hosting Options for Airflow Self-Hosted
Deploy Airflow Self-Hosted on Hetzner
The default: an 8 GB instance, the size Airflow's docs call ideal for the full Compose bundle, costs around €8-10/mo on a flat-rate root VPS, so pipeline count never touches the bill. DAGs that hold large data in memory are what push it to the next size up.
Deploy Airflow Self-Hosted on DigitalOcean
A root Droplet with snapshots and a cloud firewall in the panel, about $48/mo for 8 GB. Managed PostgreSQL (from about $15/mo) can hold the metadata database with automated backups, which keeps run history safe through Airflow upgrades.
Deploy Airflow Self-Hosted on Amazon Web Services
An EC2 instance at roughly $60-70/mo for 8 GB, placed where DAGs call S3, Redshift, or Glue so worker traffic stays inside the VPC. RDS can host the metadata database; compare the bill against MWAA, which runs the same scheduler as a managed environment.
Deploy Airflow Self-Hosted on Google Cloud Platform
A Compute Engine instance at roughly $50-70/mo for 8 GB. Airflow is the engine behind Cloud Composer, so GCS and BigQuery pipelines move over unchanged, and Cloud SQL can hold the metadata database while traffic stays inside the project.
These are highlighted picks. To see all the tools, check the Hosting & Cloud category.
Reverse Proxy Options for Airflow Self-Hosted
Airflow Self-Hosted with Traefik
Fronts the API server on port 8080 with TLS and label-based routing, keeping the other services internal. Start the API server with --proxy-headers so Airflow trusts the forwarded scheme and host, and set base_url to the public HTTPS address.
Airflow Self-Hosted with Caddy
Automatic certificates from one config block, a good fit for an internal but TLS-protected Airflow UI. Caddy forwards the scheme and host headers and passes WebSocket upgrades by default; Airflow still needs --proxy-headers and base_url set to match.
Airflow Self-Hosted with NGINX
The proxy in Airflow's own run-behind-proxy guide, which gives the full server block: forwarded headers, WebSocket upgrade headers, and HTTP/1.1. Set FORWARDED_ALLOW_IPS if NGINX runs on another host, or Airflow ignores the forwarded headers.
Frequently Asked Questions about Airflow Self-Hosted
Why self-host Airflow instead of Astronomer or a cloud's managed Airflow?
The managed Airflow offerings (Astronomer, Amazon MWAA, and Google Cloud Composer) price by workers and environment size and run around the clock, so the bill tracks your growth curve directly. Self-hosting flattens that into a fixed server bill and puts the scheduler on a private network next to the sources it reads. The trade is owning hardening, upgrades, worker capacity, and log retention.
What does the server need to run?
Airflow's Docker guide asks for at least 4 GB of memory and calls 8 GB ideal, and 8 GB is the honest size for the full Compose bundle: API server, scheduler, DAG processor, worker, triggerer, Redis, and PostgreSQL together. 2 vCPUs and 40 GB of SSD cover a small pipeline estate. DAGs that load large datasets into memory size the box more than DAG count does, and run history grows the database over time.
Which executor do I need, and do I need Redis?
LocalExecutor runs tasks in the scheduler's own processes and fits a single machine; the Celery executor hands queued tasks to workers over Redis, which is why the Compose file includes both. Stay on LocalExecutor while everything fits one server, and move to Celery when tasks start queueing behind each other. On LocalExecutor, Redis sits mostly idle; keeping it costs one small container and avoids rewriting the Compose file.
What changes before production, and how do upgrades work?
Airflow's docs present the Compose file as a quick-start without production security guarantees, so change the default airflow/airflow login, set your own secret and Fernet keys, and keep port 8080 behind TLS. PostgreSQL holds all state worth backing up; DAG files live in git. Upgrades are a new image tag, with the init service running the database migration first. Airflow 3 changed how tasks access the metadata database and removed SubDAGs, so a 2.x estate checks its DAGs before moving.
Airflow Self-Hosted or Prefect Self-Hosted?
Airflow is the established standard: DAGs, operators, and a provider ecosystem deep enough that most data teams already know it. Prefect is Python-first and lighter, with decorated functions, one server container, and a gentler learning curve. Choose Airflow when the ecosystem and the hiring pool matter; Prefect when the team wants plain Python and a small operation.
Stacks Related to Airflow Self-Hosted
GitLab Self-Hosted
Infrastructure
Self-hosted GitLab with Docker: your code, your CI, your infrastructure.
Sentry Self-Hosted
Infrastructure
Self-hosted Sentry error tracking on your own infrastructure, with PostgreSQL, Redis, ClickHouse, and Kafka under the hood.
Superset Self-Hosted
Infrastructure
Self-hosted Apache Superset: BI dashboards that never leave your server.
PostHog Self-Hosted
InfrastructureSelf-hosted PostHog: product analytics, session replay, and feature flags with every event on your own server.
Scores
Popularity5/5
Airflow is the de facto standard orchestrator in data engineering and a common line on data-engineering job descriptions.
Learning Curve4/5
Writing DAGs is plain Python, but operating Airflow means several services, executors, hardening the Compose setup, and upgrades that touch the metadata database.
Flexibility4/5
Anything Python can do can be a task, and hundreds of provider packages connect DAGs to nearly every cloud service and warehouse.
Performance4/5
One server runs a sizable pipeline estate, and the Celery executor spreads tasks across more workers when runs start queueing.
Portability3/5
DAGs are plain Python that run on any Airflow, self-hosted or managed, though they are written against Airflow's own operators and APIs.
Tools in the Airflow Self-Hosted Stack
Databases
DevOps & CI/CD
Data Engineering & ETL
Airflow Self-Hosted Pricing
Everything in the stack is free and open source; the bill is the server, an 8 GB instance for the full Compose bundle at $10-70/mo depending on the provider. The managed tiers this replaces, Astronomer, MWAA, and Cloud Composer, price by workers and environment size and run continuously.
Airflow's docs set 4 GB as the minimum and 8 GB as ideal for the full Compose bundle; data-heavy DAGs need more.
Apache-licensed, all of it.
Traefik, Caddy, or NGINX, free with free Let's Encrypt certificates.
Airflow Self-Hosted System Requirements
- CPU
- 2 vCPU
- RAM
- 4 GB minimum (the official recommendation), 8 GB for the full Compose bundle
- Disk
- 40 GB SSD; run history accumulates in PostgreSQL
- OS
- Any Linux with Docker
No official requirements published — tekyous guidance based on the bundle's services.
The official docs set a memory floor only (4 GB minimum, ideally 8 in the Docker guide); CPU and disk are estimates from real single-server deployments, one widely cited at 4 GB / 2 vCPU / 40 GB with 32 parallel workers. DAGs that hold large data in memory size the box.