Airflow) Kubernetes에 올리기(yaml)

kubernetes에 airflow를 올렸어요

K8s로 airflow 올리기

native에서 테스트할 때는 아래처럼 2개로 테스트했는데

vm1 :

• scheduler,webserver
• celery worker1, celery flower 

vm2 

• celery worker2

k8s에서 세팅하니까, 이렇게 각각 pod로 생성되더라고요

NAME                                     READY   STATUS    RESTARTS   AGE pod/airflow-flower-6c6b7f5d68-lx4j8      1/1     Running   1          2d16h pod/airflow-postgresql-0                 1/1     Running   2          2d16h pod/airflow-redis-0                      1/1     Running   0          2d1h pod/airflow-scheduler-884ddd5f6-g2vxt    2/2     Running   4          2d16h pod/airflow-statsd-84f4f9898-257tr       1/1     Running   2          2d16h pod/airflow-webserver-5f9c4f59d9-sjf5d   1/1     Running   1          2d16h pod/airflow-worker-0                     2/2     Running   20         2d16h

 

git으로 dag 파일 연결하는 도중에 프록시 문제 때문에 exec 명령어로 dags 파일을 만들었어요

kubectl exec -it pod/airflow-scheduler-884ddd5f6-g2vxt  -n airflow  -- bash kubectl exec -it pod/airflow-webserver-5f9c4f59d9-sjf5d  -n airflow  -- bash kubectl exec -it pod/airflow-worker-0 -c worker -n airflow  -- bash

 

변경내역을 알아야하니까 scheduler에도 넣고

화면에서 보여줘야하니까 webserver에도 넣고

실제로 실행해야하니까 worker에도 넣으니까 

성공적으로 실행이 가능했어요

 

+ 2021.11.10,  git-sync이용하면 편해요

https://mightytedkim.tistory.com/40

Airflow) kuberentesPodOperator

K8S Operator

지금 올리고 있는데 에러가 나네요. 폐쇄망이라 그런지 어렵네요

 

$ k get all -n airflow-k8spodoperator NAME                              READY   STATUS                  RESTARTS   AGE pod/airflow-55f5b8fcd9-5nckc      1/2     CrashLoopBackOff        552        46h pod/airflow-596cfd5bf5-snchw      0/2     Init:ImagePullBackOff   0          46h pod/airflow-db-7df475d74c-mk49j   1/1     Running                 1          6d15h NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE service/airflow-db    ClusterIP      None            <none>        5432/TCP       6d15h service/airflow-svc   LoadBalancer   10.***.15.***   <pending>     80:31724/TCP   6d15h NAME                         READY   UP-TO-DATE   AVAILABLE   AGE deployment.apps/airflow      0/1     1            0           6d15h deployment.apps/airflow-db   1/1     1            1           6d15h NAME                                    DESIRED   CURRENT   READY   AGE replicaset.apps/airflow-55f5b8fcd9      1         1         0       6d15h replicaset.apps/airflow-596cfd5bf5      1         1         0       6d15h replicaset.apps/airflow-db-7df475d74c   1         1         1       6d15h

 

참고

https://airflow.apache.org/docs/apache-airflow/stable/kubernetes.html

https://humbledude.github.io/blog/2019/07/12/airflow-on-k8s/

https://airflow.apache.org/docs/apache-airflow/stable/kubernetes.html

https://engineering.linecorp.com/ko/blog/data-engineering-with-airflow-k8s-2/