(Helm) helm3 upgrade failed 이슈

helm 사용하다가 helm upgrade가 안되는 문제가 발생했어요

Upgrade "airflow" failed: cannot patch "airflow" with kind PersistentVolumeClaim: PersistentVolumeClaim "grafana" is invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims

 

검색해보니, helm3는 deployment 내역을 secret으로 관리한데요

Since Helm 3, Helm supports three-way merge patch strategy²,
which makes it safer to bypass the error for Helm 3.
Similarly, you can mark a release revision in deployment history as deployed to bypass the error.
As Helm 3 stores deployment history as k8s Secrets

helm이 아니 k8s의 문제라고 해요

Yeah unfortunately this doesn't actually seem to be a helm problem.
Something failed about your release and it's in a bad state in kubernetes.

airflow helm 내역은 없어져서, jhub로 대체하면 아래처럼 STATUS failed로 나오더라고요

NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
jupyterhub              jhub            5               2022-07-11 09:50:55.374100197 +0900 KST failed          jupyterhub-1.2.0        1.5.0

그래서 cli로 특정 ns에 있는 secret을 모두 deployed로 변경해줫어요

kubectl patch \
   secret sh.helm.release.v1.airflow.v5  \
   -n airflow \
  --type=merge \
  -p '{"metadata":{"labels":{"status":"deployed"}}}'

어떤 걸 patch하는지 보면, 아래 secret들이에요

$ k get secret -n airflow | grep sh.helm
sh.helm.release.v1.airflow.v1              helm.sh/release.v1                    1      2
sh.helm.release.v1.airflow.v2              helm.sh/release.v1                    1      9
sh.helm.release.v1.airflow.v3              helm.sh/release.v1                    1      8
sh.helm.release.v1.airflow.v4              helm.sh/release.v1                    1      8
sh.helm.release.v1.airflow.v5              helm.sh/release.v1                    1      6

해결되었는데, 다른 방식들도 있다고 하더라고요

kubectl -n $namespace delete secret -lstatus=pending-upgrade

#!/usr/bin/env bash
RELEASES=$(helm list --namespace $NS_NAME --pending --output json | jq -r '.[] | select(.status=="pending-install")|.name')
if [[ ! -z "$RELEASES" ]]; then
  helm delete --namespace $NS_NAME $RELEASES
fi

 

참고: 

https://jacky-jiang.medium.com/how-to-fix-helm-upgrade-error-has-no-deployed-releases-mystery-3dd67b2eb126

https://github.com/helm/helm/issues/5595#issuecomment-580449247