mariadb) pvc 부족으로 인한 fail -> binary 파일이 원인

구글링할 때 이런 에러로 검색하니 나옴, pvc 할당을 늘려줌
-> binary 파일을 삭제하지 않아서 그런거였음

+ 추가
갑자기 pvc가 늘어난 원인을 찾았음
실시간 대용량 log 데이터를 바로 mariadb로 insert하고 있었음
그것도 1 connection, 1 insert
kafka, redis 같은 큐 사용 왜하지 않았냐고 물으니까.
이렇게 많을 줄 몰랐다고.. 말인지 방구인지
문제 오픈했는데, 일단 이렇게 진행하자는 대답이 들어옴...
믿을 수 없음. 책임만 있고 권한은 없는 관리자가 나임

요약

1. 문제발생
2. 원인분석
3. 조치

설명

1. 문제 발생

HQ mariadb 접속 안되는 문제 발생

2. 원인 분석

kubernetes pod CrashLoopBackOff, 재시작이 안되는 문제

-> pvc를 의심

$k get all -n mariadb NAME                    READY   STATUS             RESTARTS       AGE pod/mariadb-primary-0   0/1     CrashLoopBackOff   31 (98s ago)   50d   $k logs pod/mariadb-primary-0 -n mariadb mariadb 06:09:57.33 mariadb 06:09:57.33 Welcome to the Bitnami mariadb container mariadb 06:09:57.33 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb mariadb 06:09:57.33 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb/issues mariadb 06:09:57.33 mariadb 06:09:57.33 INFO  ==> ** Starting MariaDBsetup ** mariadb 06:09:57.35 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars mariadb 06:09:57.35 INFO  ==> Initializing mariadb database mariadb 06:09:57.36 WARN  ==> The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file. mariadb 06:09:57.36 INFO  ==> Using persisted data mariadb 06:09:57.38 INFO  ==> Running mysql_upgrade mariadb 06:09:57.39 INFO  ==> Starting mariadb in background mariadb 06:10:31.47 INFO  ==> Stopping mariadb mariadb 06:10:32.49 INFO  ==> ** MariaDB setup finished! ** mariadb 06:10:32.52 INFO  ==> ** Starting MariaDB ** ## 이후 로그 없이 crashboack 발생

pvc 상태 확인 

프로메테우스로 확인해보니 12mb 여유만 남아있음

->  kubelet_volume_stats_available_bytes{namespace='mariadb'}

 

3. 해결

pvc edit으로 16 -> 32g로 늘려줌

$k logs pod/mariadb-primary-0 -n mariadb mariadb 06:09:57.33 mariadb 06:09:57.33 Welcome to the Bitnami mariadb container mariadb 06:09:57.33 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb mariadb 06:09:57.33 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb/issues mariadb 06:09:57.33 mariadb 06:09:57.33 INFO  ==> ** Starting MariaDB setup ** mariadb 06:09:57.35 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars mariadb 06:09:57.35 INFO  ==> Initializing mariadb database mariadb 06:09:57.36 WARN  ==> The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file. mariadb 06:09:57.36 INFO  ==> Using persisted data mariadb 06:09:57.38 INFO  ==> Running mysql_upgrade mariadb 06:09:57.39 INFO  ==> Starting mariadb in background mariadb 06:10:31.47 INFO  ==> Stopping mariadb mariadb 06:10:32.49 INFO  ==> ** MariaDB setup finished! **   mariadb 06:10:32.52 INFO  ==> ** Starting MariaDB ** 2022-04-14  6:10:32 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (server 10.6.5-MariaDB-log) starting as process 1 ... 2022-04-14  6:10:32 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2022-04-14  6:10:32 0 [Note] InnoDB: Number of pools: 1 2022-04-14  6:10:32 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions 2022-04-14  6:10:32 0 [Note] mysqld: O_TMPFILE is not supported on /opt/bitnami/mariadb/tmp (disabling future attempts) 2022-04-14  6:10:32 0 [Note] InnoDB: Using Linux native AIO 2022-04-14  6:10:32 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728 2022-04-14  6:10:32 0 [Note] InnoDB: Completed initialization of buffer pool 2022-04-14  6:10:33 0 [Note] InnoDB: 128 rollback segments are active. 2022-04-14  6:10:33 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2022-04-14  6:10:33 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2022-04-14  6:10:33 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2022-04-14  6:10:33 0 [Note] InnoDB: 10.6.5 started; log sequence number 7957490508; transaction id 23213823 2022-04-14  6:10:33 0 [Note] Plugin 'FEEDBACK' is disabled. 2022-04-14  6:10:33 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool 2022-04-14  6:10:33 0 [Note] Server socket created on IP: '0.0.0.0'. 2022-04-14  6:10:33 0 [Warning] 'proxies_priv' entry '@% root@mariadb-primary-0' ignored in --skip-name-resolve mode. 2022-04-14  6:10:33 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections. Version: '10.6.5-MariaDB-log'  socket: '/opt/bitnami/mariadb/tmp/mysql.sock'  port: 3306  Source distribution 2022-04-14  6:10:33 0 [Note] InnoDB: Buffer pool(s) load completed at 220414  6:10:33

컨테이너 들어가서

데이터 확인해보니 따로 로그는 없고, 역시나 mariadb data로 pvc 대부분을 차지했음

$ du -sh ./data/ 16G     ./data/

원인을 다시 찾아보니 binary 때문이었음

binary 파일 정리함

https://mightytedkim.tistory.com/108

Mariadb) binary 파일 정리하기_k8s에서 pvc 넘침