Running Apache Airflow on Docker Using Docker Compose

Prerequisites

Before proceeding, ensure you have the following installed:

• Docker
• Docker Compose

Step 1: Clone the Official Apache Airflow Docker Repository

Apache Airflow provides an official Docker Compose setup. You can clone the repository to get the necessary configuration files:

git clone https://github.com/apache/airflow.git
cd airflow

Step 2: Define Environment Variables

Before starting Airflow, create an .env file to set environment variables:

echo -e "AIRFLOW_UID=$(id -u)" > .env

This ensures proper file permissions when running Airflow in a containerized environment.

Step 3: Set Up Docker Compose File

Inside the airflow directory, create a docker-compose.yaml file with the following content:

version: '3'
services:
  airflow:
    image: apache/airflow:latest
    container_name: airflow
    restart: always
    environment:
      - LOAD_EXAMPLES=False
    volumes:
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
      - ./plugins:/opt/airflow/plugins
    ports:
      - "8080:8080"
    command: webserver
  
  postgres:
    image: postgres:13
    container_name: airflow_postgres
    restart: always
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    ports:
      - "5432:5432"

  redis:
    image: redis:latest
    container_name: airflow_redis
    restart: always
    ports:
      - "6379:6379"

Step 4: Initialize the Airflow Database

Before launching Airflow, initialize the metadata database:

docker-compose up airflow-init

Step 5: Start Apache Airflow

Once initialization is complete, bring up the Airflow services:

docker-compose up -d

You can now access the Airflow web UI by navigating to:

http://localhost:8080

Step 6: Managing Airflow Services

To check running containers, use:

docker ps

To stop the services:

docker-compose down

To restart Airflow:

docker-compose restart

Conclusion

Running Apache Airflow using Docker Compose provides a streamlined, scalable, and reproducible environment for orchestrating workflows. With this setup, you can easily develop, test, and deploy Airflow workflows without worrying about complex installations. If you have any questions, feel free to leave a comment below!