k8s) worker 재시작했을 때 not ready 인 경우

어쩔때는 잘 올라오는데 어쩔때는 not ready고 쿠버는 참 재미진 것 같아요..

매번 믿지만, 가끔 뒤통수 맞아서 비슷한 사례가 있을 때마다 여기에 추가하려고요

(대단한건 없고 메모용입니다)

 

server 의 경우

서버에서는 대부분의 문제가 kubelet, docker(containerd)가 실행되지 않아서 생기더라고요

그래서 아래 명령어들을 확인합니다.

systemctl status kubelet
systemctl status docker
#systemctl status containerd

예시

[root@k8sworker3 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; disabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: inactive (dead)

# 살리기
[root@k8sworker3 ~]# systemctl start kubelet
[root@k8sworker3 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; disabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Thu 2023-03-09 13:23:24 KST; 1s ago
     Docs: 
https://kubernetes.io/docs/

 Main PID: 3435 (kubelet)
    Tasks: 11
   Memory: 104.3M
   CGroup: /system.slice/kubelet.service
           └─3435 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --

 

VM의 경우

1. readonly로 디스크가 변경된 경우

VM이 아니면 삼가 고인의 명복을 빕니다. 또 발생할 가능성이 있으니, 빨리 디스크를 바꿔주세요

저같은 경우, reboot 후에 /dev/xvda3 밑에 있는 /dev/mapper/ubuntu--vg-ubuntu--lv 가 ro(read only)로 되어있었어요

$ mount | grep mapper
/dev/mapper/ubuntu--vg-ubuntu--lv on /ytpe ext(ro,relatime)

docker,kubelet이 실행되지 않는 상황을 발견할거에요

 

$ mount -n -o rw,remount /

#만약 안되면
$ fsck /dev/mapper/ubuntu--vg-ubuntu--lv

$ mount | grep mapper
/dev/mapper/ubuntu--vg-ubuntu--lv on /ytpe ext(rw,relatime)

reboot 
$ systemctl start docker
$ systemctl start kubelet

2. logical volume, volume group 관련

lvdisplay Not Available이 뜰 때가 있어요

logical volume이 풀려서 그런건데

 lvdisplay

  --- Logical volume ---
  LV Path                /dev/ceph-8d868f14-e7d6-487c-ad1e-53329347f36f/osd-block-4219042a-35e8-421f-b829-507fc9be4577
  LV Name                osd-block-4219042a-35e8-421f-b829-507fc9be4577
  VG Name                ceph-8d868f14-e7d6-487c-ad1e-53329347f36f
  LV UUID                a7c3St-5yC9-qnP0-Uefy-1msp-pC2L-kSXrbf
  LV Write Access        read/write
  LV Creation host, time rook-ceph-osd-prepare-k8s02-cwsbx, 2022-02-22 19:17:33 +0900
  LV Status              NOT available
  LV Size                <128.00 GiB
  Current LE             32767
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

이럴 경우

vgchange -ay

  1 logical volume(s) in volume group "ceph-8d868f14-e7d6-487c-ad1e-53329347f36f" now active
  1 logical volume(s) in volume group "ceph-d2a805b7-1b67-46a0-b527-299da655bd47" now active
  1 logical volume(s) in volume group "ceph-291853e7-088b-4a27-a771-78aa6142acbe" now active

LV status가 변하게 됩니다.

 lvdisplay

  --- Logical volume ---
  LV Path                /dev/ceph-8d868f14-e7d6-487c-ad1e-53329347f36f/osd-block-4219042a-35e8-421f-b829-507fc9be4577
  LV Name                osd-block-4219042a-35e8-421f-b829-507fc9be4577
  VG Name                ceph-8d868f14-e7d6-487c-ad1e-53329347f36f
  LV UUID                a7c3St-5yC9-qnP0-Uefy-1msp-pC2L-kSXrbf
  LV Write Access        read/write
  LV Creation host, time rook-ceph-osd-prepare-k8s02-cwsbx, 2022-02-22 19:17:33 +0900
  LV Status              available
  # open                 0
  LV Size                <128.00 GiB
  Current LE             32767
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           251:0