Airflow) Ec2에 docker-compose 로 올리기

airflow 스터디 환경 구현

 

최소 4g ram 필요해서 8g t2.large로 ec2 생성

 

docker, docker-compose 설치

docker install

    4  sudo yum -y upgrade
    5  sudo yum -y install docker
    7   sudo service docker start
    9  sudo usermod -aG docker ec2-user
   22  exit

docker-compose install
   26  sudo curl -L https://github.com/docker/compose/releases/download/1.25.0-rc2/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
   27  sudo chmod +x /usr/local/bin/docker-compose

 

memory  확인

[ec2-user@ip-172-31-39-60 ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.8G        194M        6.7G        440K        899M        7.4G
Swap:            0B          0B          0B

공식 문서 docker-compose 예시 실행

https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html 

Running Airflow in Docker — Airflow Documentation

 

에러남 -> docker-compose  버전 업그레이드

   56  sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
   58  sudo chmod +x /usr/local/bin/docker-compose
   59  docker-compose version
   60  docker-compose --version

 

 

docker ps
CONTAINER ID   IMAGE          COMMAND                  CREATED         STATUS                   PORTS      NAMES
3eda287e2e66   postgres:13    "docker-entrypoint.s…"   6 minutes ago   Up 6 minutes (healthy)   5432/tcp   airflow_postgres_1
394d6c88ec76   redis:latest   "docker-entrypoint.s…"   6 minutes ago   Up 6 minutes (healthy)   6379/tcp   airflow_redis_1

 

docker-compose up

[ec2-user@ip-172-31-39-60 ~]$ docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED              STATUS                        PORTS                                                 NAMES
1e7f270f175f   apache/airflow:2.2.5   "/usr/bin/dumb-init …"   About a minute ago   Up About a minute (healthy)   0.0.0.0:5555->5555/tcp, :::5555->5555/tcp, 8080/tcp   airflow_flower_1
ff9c776de148   apache/airflow:2.2.5   "/usr/bin/dumb-init …"   About a minute ago   Up About a minute (healthy)   8080/tcp                                              airflow_airflow-scheduler_1
2b4959b24a43   apache/airflow:2.2.5   "/usr/bin/dumb-init …"   About a minute ago   Up About a minute (healthy)   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp             airflow_airflow-webserver_1
666366536b98   apache/airflow:2.2.5   "/usr/bin/dumb-init …"   About a minute ago   Up About a minute (healthy)   8080/tcp                                              airflow_airflow-triggerer_1
063591d1275d   apache/airflow:2.2.5   "/usr/bin/dumb-init …"   About a minute ago   Up About a minute (healthy)   8080/tcp                                              airflow_airflow-worker_1
3eda287e2e66   postgres:13            "docker-entrypoint.s…"   9 minutes ago        Up 9 minutes (healthy)        5432/tcp                                              airflow_postgres_1
394d6c88ec76   redis:latest           "docker-entrypoint.s…"   9 minutes ago        Up 9 minutes (healthy)        6379/tcp                                              airflow_redis_1

 

방화벽 열기

ec2:8080

 

ec2:5555

+ postgres

-expose 안되서 밖에서 못봄

설치 끝