기타/K8S
(Helm) helm3 upgrade failed 이슈
MightyTedKim
2022. 8. 1. 09:43
728x90
반응형
helm 사용하다가 helm upgrade가 안되는 문제가 발생했어요
Upgrade "airflow" failed: cannot patch "airflow" with kind PersistentVolumeClaim: PersistentVolumeClaim "grafana" is invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims
검색해보니, helm3는 deployment 내역을 secret으로 관리한데요
Since Helm 3, Helm supports three-way merge patch strategy²,
which makes it safer to bypass the error for Helm 3.
Similarly, you can mark a release revision in deployment history as deployed to bypass the error.
As Helm 3 stores deployment history as k8s Secrets
helm이 아니 k8s의 문제라고 해요
Yeah unfortunately this doesn't actually seem to be a helm problem.
Something failed about your release and it's in a bad state in kubernetes.
airflow helm 내역은 없어져서, jhub로 대체하면 아래처럼 STATUS failed로 나오더라고요
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
jupyterhub jhub 5 2022-07-11 09:50:55.374100197 +0900 KST failed jupyterhub-1.2.0 1.5.0
그래서 cli로 특정 ns에 있는 secret을 모두 deployed로 변경해줫어요
kubectl patch \
secret sh.helm.release.v1.airflow.v5 \
-n airflow \
--type=merge \
-p '{"metadata":{"labels":{"status":"deployed"}}}'
어떤 걸 patch하는지 보면, 아래 secret들이에요
$ k get secret -n airflow | grep sh.helm
sh.helm.release.v1.airflow.v1 helm.sh/release.v1 1 2
sh.helm.release.v1.airflow.v2 helm.sh/release.v1 1 9
sh.helm.release.v1.airflow.v3 helm.sh/release.v1 1 8
sh.helm.release.v1.airflow.v4 helm.sh/release.v1 1 8
sh.helm.release.v1.airflow.v5 helm.sh/release.v1 1 6
해결되었는데, 다른 방식들도 있다고 하더라고요
kubectl -n $namespace delete secret -lstatus=pending-upgrade
#!/usr/bin/env bash
RELEASES=$(helm list --namespace $NS_NAME --pending --output json | jq -r '.[] | select(.status=="pending-install")|.name')
if [[ ! -z "$RELEASES" ]]; then
helm delete --namespace $NS_NAME $RELEASES
fi
참고:
https://github.com/helm/helm/issues/5595#issuecomment-580449247
728x90
반응형