Jupyterhub) Helm 설치_hub-db-dir 에러 해결

한줄 요약:  helm으로 따라했는데 버그였음. 그런데 다시해보니 재현이 안됨

요약

1. k8s에 jupyterhub를 설치함
2. hub 컨테이너 pvc 매핑 문제 발생

설명

1. k8s에 jupyterhub를 설치함

 

공식문서에 나온대로 따라하면 됨

https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/installation.html

helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/ helm repo update helm upgrade --cleanup-on-fail \   --install jupyterhub private-reop/jupyterhub \   --namespace jhub \   --create-namespace \   --version=1.20 \   --values config.yaml

hub, user-scheduler 실행이안됨

$ k get all -n jhub
NAME                                  READY   STATUS             RESTARTS       AGE
pod/continuous-image-puller-lphvt     1/1     Running            0              7m42s
pod/continuous-image-puller-r5kwg     1/1     Running            0              7m42s
pod/continuous-image-puller-rtnrq     1/1     Running            0              7m42s
pod/continuous-image-puller-tknhv     1/1     Running            0              7m42s
pod/continuous-image-puller-x79pb     1/1     Running            0              7m42s
pod/hub-5986b96879-c5fqj              0/1     Pending            0              7m42s
pod/proxy-65bfcc8fdf-hc9wp            1/1     Running            0              7m42s
pod/user-scheduler-67db665797-pmbsf   0/1     CrashLoopBackOff   6 (115s ago)   7m42s
pod/user-scheduler-67db665797-zvkbt   0/1     CrashLoopBackOff   6 (112s ago)   7m42s

NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)          AGE
service/hub            NodePort       10.100.171.226   <none>          8081:30011/TCP   7m42s
service/proxy-api      ClusterIP      10.108.232.250   <none>          8001/TCP         7m42s
service/proxy-public   LoadBalancer   10.98.248.160    172.17.172.22   80:31795/TCP     7m42s

NAME                                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/continuous-image-puller   5         5         5       5            5           <none>          7m42s

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hub              0/1     1            0           7m42s
deployment.apps/proxy            1/1     1            1           7m42s
deployment.apps/user-scheduler   0/2     2            0           7m42s

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/hub-5986b96879              1         1         0       7m42s
replicaset.apps/proxy-65bfcc8fdf            1         1         1       7m42s
replicaset.apps/user-scheduler-67db665797   2         2         0       7m42s

NAME                                READY   AGE
statefulset.apps/user-placeholder   0/0     7m42s

 

k get event로 보니까

$ k get event -A NAMESPACE   LAST SEEN   TYPE      REASON             OBJECT   jhub        2m21s       Warning   FailedScheduling   pod/hub-5986b96879-c5fqj                                   0/6 nodes are available: 6 pod has unbound immediate PersistentVolumeClaims. jhub        2m25s       Normal    FailedBinding      persistentvolumeclaim/hub-db-dir                           no persistent volumes available for this claim and no storage class is set jhub        2m22s       Warning   BackOff            pod/user-scheduler-67db665797-pmbsf                        Back-off restarting failed container jhub        2m23s       Warning   BackOff            pod/user-scheduler-67db665797-zvkbt                        Back-off restarting failed container

뭔가 pvc에 문제가 있다는 것을 발견

$ k describe pvc hub-db-dir -n jhub Name:          hub-db-dir Namespace:     jhub StorageClass:  #비어있음 Status:        Pending Annotations:   meta.helm.sh/release-name: jupyterhub                meta.helm.sh/release-namespace: jhub Finalizers:    [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode:    Filesystem Used By:       hub-5986b96879-c5fqj Events:   Type    Reason         Age                   From                         Message   ----    ------         ----                  ----                         -------   Normal  FailedBinding  60s (x422 over 106m)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

 

2. hub 컨테이너 pvc 매핑 문제 발생

pvc pending인 것을 확인하고 다시 선언. 

https://kienmn97.medium.com/persistent-storage-in-jupyterhub-on-kubernetes-cluster-running-on-minikube-4b469bdb1b86

pvc가 pending

 k get pvc -n jhub
NAMESPACE       NAME                                STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
jhub            hub-db-dir                          Pending                                                                                          12m

 

pvc 설정을 보니까 storageClassName이 비어있어서 채워줬음

cat hub-dir.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    meta.helm.sh/release-name: jupyterhub
    meta.helm.sh/release-namespace: jhub
  labels:
    app: jupyterhub
    app.kubernetes.io/managed-by: Helm
    chart: jupyterhub-1.2.0
    component: hub
    heritage: Helm
    release: jupyterhub
  name: hub-db-dir
  namespace: jhub
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: rook-ceph-block #추가
  volumeMode: Filesystem

올리니까 hub statefulset이 잘 올라감

pvc pending이 일어나는지 찾느라 시간이 좀 오래 걸렸음