I have been trying to migrate the local dockerized airflow from Windows 11 to MacOS (M1 Silicon) for the last two days.
The problem I have is that although containers are running, I cannot log in to Airflow with localhost:8080. In Windows 11 everything works like a charm.
While digging I read that M1 does not work very effectively while dockerized although I hope I can me things work.
Below are the containers’ statuses:
537543b6ebc9 dlh.azurecr.io/airflow-worker:v1 "/opt/bitnami/script…" 50 minutes ago Up 12 seconds 8793/tcp yaml-local-airflow-worker-1-1
cd2f0804a125 dlh.azurecr.io/airflow-worker:v1 "/opt/bitnami/script…" 50 minutes ago Up 13 seconds 8793/tcp yaml-local-airflow-worker-2-1
e1d79f6e1755 dlh.azurecr.io/airflow-scheduler:v1 "/opt/bitnami/script…" 50 minutes ago Up 12 seconds yaml-local-airflow-scheduler-1
1bc5f768195a dlh.azurecr.io/airflow:v1 "/opt/bitnami/script…" 50 minutes ago Up 13 seconds 0.0.0.0:8080->8080/tcp yaml-local-airflow-1
2cbb96a232c7 bitnami/postgresql:latest "/opt/bitnami/script…" 50 minutes ago Up 19 seconds (healthy) 5432/tcp yaml-local-postgresql-1
a157072edc13 bitnami/redis:latest "/opt/bitnami/script…" 50 minutes ago Up 19 seconds (healthy) 6379/tcp yaml-local-redis-1
and the yaml-file:
version: '2'
x-airflow-common:
&airflow-common
user: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}"
volumes:
- ../../airflow/dags/:/opt/bitnami/airflow/dags
- ../../airflow/plugins/:/opt/bitnami/airflow/plugins
- ../../airflow/config/:/opt/bitnami/airflow/config
- ../../airflow/scripts/:/mnt/airflowdata/scripts
- ../../airflow/variables/:/mnt/airflow/variables
- ../../airflow/certs/:/mnt/airflowdata/certs
- ../../airflow/python/requirements.txt:/bitnami/python/requirements.txt
environment:
AIRFLOW_EXECUTOR: CeleryExecutor
AIRFLOW_WEBSERVER_HOST: airflow
AIRFLOW_WEBSERVER_PORT_NUMBER: 8080
AIRFLOW_LOAD_EXAMPLES: no
AIRFLOW_SECRETS_BACKEND: "${AIRFLOW__SECRETS__BACKEND}"
env_file:
- .env
services:
postgresql:
image: 'bitnami/postgresql:latest'
environment:
- POSTGRESQL_DATABASE=airflow
- POSTGRESQL_USERNAME=airflowadmin
- POSTGRESQL_PASSWORD=bitnami123@$
- ALLOW_EMPTY_PASSWORD=yes
- BITNAMI_DEBUG=true
volumes:
- ../../../airflowdata/postgres/:/bitnami/postgresql
healthcheck:
test: ["CMD", "pg_isready", "-U", "airflowadmin"]
interval: 5s
retries: 5
redis:
image: 'bitnami/redis:latest'
environment:
- ALLOW_EMPTY_PASSWORD=yes
volumes:
- 'redis_data:/bitnami'
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 30s
retries: 50
airflow-worker-1:
<<: *airflow-common
image: datalakehousepacr.azurecr.io/airflow-worker:v1
depends_on:
redis:
condition: service_healthy
postgresql:
condition: service_healthy
airflow:
condition: "service_started"
airflow-worker-2:
<<: *airflow-common
image: datalakehousepacr.azurecr.io/airflow-worker:v1
depends_on:
redis:
condition: service_healthy
postgresql:
condition: service_healthy
airflow:
condition: "service_started"
airflow-scheduler:
<<: *airflow-common
image: datalakehousepacr.azurecr.io/airflow-scheduler:v1
depends_on:
redis:
condition: service_healthy
postgresql:
condition: service_healthy
airflow:
condition: "service_started"
airflow:
<<: *airflow-common
image: datalakehousepacr.azurecr.io/airflow:v1
ports:
- 8080:8080
depends_on:
redis:
condition: service_healthy
postgresql:
condition: service_healthy
volumes:
redis_data:
driver: local
FYI - In W11 I am able to set everything within 10-15 minutes so perhaps it’s something with the MacOS environment. Yet, I believe it’s all related to Bitnami’s image because I did some testing and was able to establish a testing Airflow environment, hence still do not know where to problem is.
