기타/K8S

kubernetes) Kubesphere 제거시 남는 것들_helm,ulimit

MightyTedKim 2022. 4. 9. 11:51
728x90
반응형

 

한줄 요약: kubesphere 삭제할 때 정말 잘 지워졌는지 확인하자

요약

  1. 문제발생
    1. linux max user process 부족으로 master에 오류가 생김
  2. 원인 파악
    1. 설정된 max user process 자체가 너무 작음
    2. sendmail, postdrop, etcd-backup.sh 관련된 프로세스가 생김
  3. 해결 방안
    1. ulimit 변경 후 reboot 요청
    2. Kubesphere 관련 설정 마저 삭제 

설명

1. 문제 발생

> linux max user process 부족으로 master에 오류가 생김

아래는 예시

$k get pod -n jupyter
runtime: failed to create new OS thread (have 16 already; errno=11)
runtime: may need to increase max user processes (ulimit -u)
fatal error: newosproc

runtime stack:
runtime.throw(0x1e6d470, 0x9)
        /usr/local/go/src/runtime/panic.go:1117 +0x72
runtime.newosproc(0xc000501000)
        /usr/local/go/src/runtime/os_linux.go:160 +0x1be
runtime.newm1(0xc000501000)
        /usr/local/go/src/runtime/proc.go:2144 +0xdd
runtime.newm(0x20e8ad0, 0xc000073000, 0xf)
        /usr/local/go/src/runtime/proc.go:2123 +0xa6
runtime.startm(0x0, 0xc000c8ff01)
        /usr/local/go/src/runtime/proc.go:2378 +0xe5
runtime.wakep()
        /usr/local/go/src/runtime/proc.go:2477 +0x66
runtime.resetspinning()
        /usr/local/go/src/runtime/proc.go:3020 +0x59
runtime.schedule()
        /usr/local/go/src/runtime/proc.go:3176 +0x2b9
runtime.mstart1()
        /usr/local/go/src/runtime/proc.go:1313 +0x93
runtime.mstart()
        /usr/local/go/src/runtime/proc.go:1272 +0x6e

crontab에서 주석을 해도 다음날이 되면 계속 생김

#*/30 * * * * sh /usr/local/bin/kube-scripts/etcd-backup.sh

2. 원인 파악

  1. 설정된 max user process 자체가 너무 작음
    1. user process가 4906으로 설정되어있었음
    2. 적용하고 로그아웃하면 되지만, 해당 세션이 실행되는 프로세스도 로그아웃해야해서 재시작이 필요해보임
  2. sendmail, postdrop, etcd-backup.sh 관련된 프로세스가 생김
    1. 원인은 파악했고, 반복적인 프로세스가 없는지 찾아봄
    2. kubesphere를 설치하다가 삭제했는데, 제대로 삭제되지 않은 것으로 확인

구글이하다보니, kubesphere-delete.sh을 했는데 etcd-backup.sh 이 crontab에서 사라지지 않았다는 이슈를 발견했다.

- 참고: https://github.com/kubesphere/ks-installer/issues/1868

 

3. 해결 방안

  1. ulimit 변경 후 reboot 요청
  2. Kubesphere 관련 설정 마저 삭제 

리붓은 요청하면 되는거고, kubesphere 관련 설정은 삭제하면됨

 

kubesphere-delete.sh을 들여다보면, snapshot controller를 삭제한다.

# 가끔 helm 명령어 생략되는데, 나는 이게 실행이 안된거같음
# github 소스 
40행 helm uninstall -n kube-system snapshot-controller 2>/dev/null

- 참고: https://github.com/kubesphere/ks-installer/blob/master/scripts/kubesphere-delete.sh

 

역시나 helm으로 확인하니 아직 삭제되지 않은 kubesphere가 있다.

uninstall 다시 해주고, ns에 남아있는게 없는지 한번 더 확인해준다.

 

다음날 확인해보니, crontab에 추가되지 않은 것을 확인

root      2785 93988  0 Apr05 ?        00:00:00 /usr/sbin/postdrop -r
root     93988 17829  0 Apr05 ?        00:00:00 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t -f root
root     34216 47520  0 Apr05 ?        00:01:32 /usr/local/bin/etcdctl --endpoints=https://172.**.0.1:2379 snapshot save /var/backups/kube_etcd/etcd-2022-04-05-14-00-01/snapshot.db --cacert=/etc/ssl/etcd/ssl/ca.pem --cert=/etc/ssl/etcd/ssl/admin-hgkim01.pem --key=/etc/ssl/etcd/ssl/a
root     17823  3211  0 Apr05 ?        00:00:00 /usr/sbin/CROND -n
root     45798 17823  0 Apr05 ?        00:00:00 sh /usr/local/bin/kube-scripts/etcd-backup.sh
root     13649 17823  0 Apr05 ?        00:00:00 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t -f root
root     80035 13649  0 Apr05 ?        00:00:00 /usr/sbin/postdrop -r
root     19196 45798  0 Apr05 ?        00:01:39 /usr/local/bin/etcdctl --endpoints=https://172.**.0.1:2379 snapshot save /var/backups/kube_etcd/etcd-2022-04-05-15-00-01/snapshot.db --cacert=/etc/ssl/etcd/ssl/ca.pem --cert=/etc/ssl/etcd/ssl/admin-hgkim01.pem --key=/etc/ssl/etcd/ssl/a
root     76299     2  0 Apr05 ?        00:00:00 [kworker/61:2-cg]
root     80046     2  0 Apr05 ?        00:00:02 [kworker/50:2-mm]
root     76301     2  0 11:56 ?        00:00:00 [kworker/85:2-cg]
root     83111     2  0 11:59 ?        00:00:00 [kworker/86:2-mm]
root     86137  3211  0 12:00 ?        00:00:00 /usr/sbin/CROND -n
root      2638 86137  0 12:00 ?        00:00:00 sh /usr/local/bin/kube-scripts/etcd-backup.sh
root     41382 86137  0 12:00 ?        00:00:00 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t -f root
root     60446 41382  0 12:00 ?        00:00:00 /usr/sbin/postdrop -r
root     82753     2  0 12:00 ?        00:00:00 [kworker/62:2]
root     41734  2638  0 12:00 ?        00:00:03 /usr/local/bin/etcdctl --endpoints=https://172.**.0.1:2379 snapshot save /var/backups/kube_etcd/etcd-2022-04-07-12-00-01/snapshot.db --cacert=/etc/ssl/etcd/ssl/ca.pem --cert=/etc/ssl/etcd/ssl/admin-hgkim01.pem --key=/etc/ssl/etcd/ssl/a
root     85075     2  0 12:00 ?        00:00:00 [kworker/34:2-mm]
root     34139     2  0 12:09 ?        00:00:00 [kworker/35:0-ev]
(생랶)
root     42748     2  0 13:50 ?        00:00:00 [kworker/52:2-cg]
root     60279     2  0 13:51 ?        00:00:00 [kworker/11:1-mm]
root     84250     2  0 13:51 ?        00:00:00 [kworker/u194:1-]
root      2134     2  0 13:51 ?        00:00:00 [kworker/10:0-cg]
root     67134     2  0 13:52 ?        00:00:00 [kworker/59:0-mm]
root     22331     2  0 13:52 ?        00:00:00 [kworker/6:2-cgr]
root     83493     2  0 13:52 ?        00:00:00 [kworker/17:2-ev]

root     86276     2  0 14:05 ?        00:00:00 [kworker/13:1-mm]
root     34140     2  0 14:05 ?        00:00:00 [kworker/52:0-mm]
root     12940     2  0 14:05 ?        00:00:00 [kworker/66:0-mm]
728x90
반응형
저작자표시