Observability

Monitoring and logging

Stack used: Grafana + Prometheus + Loki + Promtail

Install prometheus-kube-stack

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring

Get credentials:

kubectl --namespace monitoring get secrets prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo

Similarly with ArgoCD - prometheus-grafana service can be patched to access it by IP and port, but better to use subdomain.

Create secret:

kubectl create secret tls tls-secret-grafana \                                       
  --cert=/etc/letsencrypt/live/domain.name/fullchain.pem \
  --key=/etc/letsencrypt/live/domain.name/privkey.pem \
  --namespace=monitoring

Apply ingress:

kubectl apply -f monitoring/ingress.yaml

Access it at grafana.domain.name

Use login admin and obtained password earlier to sign in.

Alerting

Alerts are configured to be sent to Telegram. Change config at monitoring/alertmanager-config.yaml, rules at monitoring/rules.yaml.

If config changed:

Update secret in alertmanager

kubectl -n monitoring create secret generic alertmanager-prometheus-kube-prometheus-alertmanager --from-file=alertmanager.yaml=monitoring/alertmanager-config.yaml --dry-run=client -o yaml | kubectl apply -f -

Rollout statefulset to apply

kubectl -n monitoring rollout restart statefulset alertmanager-prometheus-kube-prometheus-alertmanager

If rules changed:

helm upgrade prometheus  -f monitoring/rules.yaml prometheus-community/kube-prometheus-stack -n monitoring

To see rules and alerts - patch service with prometheus (where port 9090) to LoadBalancer or NodePort.