kafka) kafka_exporter 설정 (prometheus,helm)

 

kafka_exporter를 이용하면 prometheus에서 metric을 조회할 수 있어요

예상 독자는 아래와 같아요

1. kafka cluster를 모니터링하고 싶은 분
2. prometheus helm을 사용하시는 분
3. 미래의 나

 

요약

1. kafka exporter 실행
2. prometheus 설정
3. grafana 대시보드

 

설명

1. kafka_exporter 실행

다운로드

wget https://github.com/danielqsj/kafka_exporter/releases https://github.com/danielqsj/kafka_exporter/releases/download/v1.2.0/kafka_exporter-1.2.0.linux-amd64.tar.gz

실행

tar -zxvf kafka_exporter-1.2.0.linux-amd64.tar.gz
cd kafka_exporter-1.2.0.linux-amd64

vi ./start.sh
#/bin/bash
nohup ./kafka_exporter \
--kafka.server=172.17.172.185:9092 \
--kafka.server=172.17.172.170:9092 \
--kafka.server=172.17.172.182:9092 \
--web.listen-address=":9102" > /dev/null 2>&1 &

확인

$ curl localhost:9102/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 4.1404e-05
go_gc_duration_seconds{quantile="0.25"} 8.2797e-05
go_gc_duration_seconds{quantile="0.5"} 0.000123694
go_gc_duration_seconds{quantile="0.75"} 0.000205605
go_gc_duration_seconds{quantile="1"} 0.000273476
go_gc_duration_seconds_sum 0.002096983
go_gc_duration_seconds_count 16
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 10
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.

kafka broker 가 있는 서버에 모두 설정해주면 끗

2. prometheus 설정

jmx exporter를 이용하는 방법도 있는데, 프로메테우스에 target 주면 끝이에요
https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml

GitHub - prometheus-community/helm-charts: Prometheus community Helm charts

1469행 쪽 보면 HGKIM_KAFKA_STATUS 라는 target 이름으로 추가할 수 있어요

  prometheus.yml:
    rule_files:
      - /etc/config/recording_rules.yml
      - /etc/config/alerting_rules.yml
    ## Below two files are DEPRECATED will be removed from this default values file
      - /etc/config/rules
      - /etc/config/alerts
    scrape_configs:
      - job_name: prometheus
        static_configs:
          - targets:
            - localhost:9090
      - job_name: 'HGKIM_KAFKA_STATUS' #kafka_exporter
        static_configs:
        - targets: ['172.17.172.185:9102', '172.17.172.170:9102', '172.17.172.171:9102']

k8s configmap에 저장이되요

$ k describe configmap -n monitoring monitoring-prometheus-server | grep KAFKA -a5
scrape_configs:
- job_name: prometheus
static_configs:
- targets:
- localhost:9090
- job_name: HGKIM_KAFKA_STATUS
static_configs:
- targets:
- 172.17.172.185:9102
- 172.17.172.170:9102
- 172.17.172.171:9102

확인
- promehteus web ui에 들어가면 아래처럼 나올거에요
ex) http://172.17.172.11:30043/targets

HGKIMP_KAFKA_STATUS (3/3 up)

EndpointStateLabelsLast ScrapeScrape DurationError

http://172.17.172.185:9102/metrics	UP	instance="172.17.172.185:9102"job="HGKIM_KAFKA_STATUS"	46.405s ago	30.875ms
http://172.17.172.170:9102/metrics	UP	instance="172.17.172.170:9102"job="HGKIM_KAFKA_STATUS"	46.360s ago	28.982ms
http://172.17.172.171:9102/metrics	UP	instance="172.17.172.171:9102"job="HGKIM_KAFKA_STATUS"	21.656s ago	30.093ms

prometheus에 입력해도되요

입력) kafka_brokers

결과

kafka_brokers{instance="172.17.172.170:9102", job="HGKIM_KAFKA_STATUS"}	3
kafka_brokers{instance="172.17.172.171:9102", job="HGKIM_KAFKA_STATUS"}	3
kafka_brokers{instance="172.17.172.185:9102", job="HGKIM_KAFKA_STATUS"}	3

이제 잘 수집되는지 확인했으니 대시보드를 만들어볼게요

3. granfana 대시보드
대시보드는 가장 유명한 거를 사용합니다
https://grafana.com/grafana/dashboards/7589

Kafka Exporter Overview dashboard for Grafana | Grafana Labs

유의사항은 jmx_exporter를 사용하지 않기 때문에 variable을 수정해줘야해요

JOB

• label_values(kafka_brokers, job)

INSTANCE

• label_values(kafka_brokers{job=~"$job"}, instance)
• all_values ".*"

TOPIC

• label_values(kafka_topic_partition_current_offset{instance='$instance',topic!='__consumer_offsets',topic!='--kafka'}, topic)

테스트해보려면 kafka client에서 dummy message를 많이 쏴보면되요

• [1m]으로 rate가 설정되어있기 때문에 바로 안보일 수 있어요. 기다리거나 [5m]으로 바꿔주세요

./bin/kafka-verifiable-producer.sh --topic hgkim --bootstrap-server cluster0:9092 --max-message 100000

참고
- https://github.com/prometheus-community/helm-charts/

GitHub - prometheus-community/helm-charts: Prometheus community Helm charts

- https://knowjea.github.io/2021/03/01/.Kafka,-Prometheus,-Grafana.html

Know Jea

- https://www.confluent.io/blog/monitor-kafka-clusters-with-prometheus-grafana-and-confluent/

Monitor Apache Kafka Clusters with Prometheus, Grafana, and Confluent