Apache Airflow
Apache Airflow is the industry-standard platform for programmatically authoring, scheduling, and monitoring workflows. Originally created at Airbnb in 2014 and now a top-level Apache Software Foundation project, Airflow lets you define data pipelines as code using Python, making complex ETL, ELT, and ML pipelines versionable, testable, and collaborative.
Key Features
- DAGs as code — Workflows are defined as Python Directed Acyclic Graphs, so pipelines live in version control and benefit from code review and CI/CD.
- Powerful scheduler — Cron-like scheduling with backfill, catchup, and retry semantics for handling missed or failed runs.
- Rich web UI — Browser-based interface for inspecting DAG runs, task logs, and dependency trees in real time.
- Extensive operator ecosystem — 1,000+ community-maintained providers integrate with AWS, GCP, Azure, Kubernetes, databases, and countless SaaS APIs.
- Dynamic task mapping — Generate tasks at runtime from data, enabling large-scale parallel fan-out pipelines.
- Airflow 2.x stable API — The REST API and stable command-line interface make automation and CI integration straightforward.
- Active community — 46,000+ GitHub stars, monthly releases, and a huge ecosystem of plugins and documentation.
Why Use Apache Airflow?
If you run data pipelines at any scale, Airflow gives you a battle-tested orchestrator with a massive ecosystem. The “workflows as code” approach means your pipelines are reproducible, auditable, and maintainable — the same engineering practices that apply to application code apply to your data workflows.
Use Cases
- ETL/ELT data ingestion and transformation pipelines
- MLOps workflows — model training, evaluation, and deployment orchestration
- Scheduled reporting and data warehouse refreshes
- Infrastructure job orchestration with retries and alerting
Platform
Linux · macOS · Windows (via WSL/Docker) · Docker · Kubernetes
Licence
Apache License 2.0