Prometheus is confusing. It’s such a great project and there is all kinds of information out there, but it’s taken me a bit of legwork to understand it.
The first issue is: How are you going to install it? Since I’m running this on Kubernetes it makes sense to use whatever most people are doing. And commonly, the answer is Helm. I’m not a big fan of Helm because of all the added features like releases and heaviness in setting it up. I would instead prefer a light weight variable substitution system with jija2 and a bunch of manifest files but fine, helm is what we’ll use.
Now you have to find the right repo. So we do the following:
helm repo add prometheus-community https://prometheus-community.github.io/helm-chartsYou wanted prometheus so it’s in there right?
helm search repo prometheus-community
NAME CHART VERSION APP VERSION DESCRIPTION
prometheus-community/alertmanager 0.5.0 v0.21.0 The Alertmanager handles alerts sent by client ...
prometheus-community/kube-prometheus-stack 13.5.0 0.45.0 kube-prometheus-stack collects Kubernetes manif...
prometheus-community/prometheus 13.2.1 2.24.0 Prometheus is a monitoring system and time seri...
prometheus-community/prometheus-adapter 2.11.1 v0.8.3 A Helm chart for k8s prometheus adapter
prometheus-community/prometheus-blackbox-exporter 4.10.2 0.18.0 Prometheus Blackbox Exporter
prometheus-community/prometheus-cloudwatch-expo... 0.13.0 0.8.0 A Helm chart for prometheus cloudwatch-exporter
prometheus-community/prometheus-consul-exporter 0.4.0 0.4.0 A Helm chart for the Prometheus Consul Exporter
prometheus-community/prometheus-couchdb-exporter 0.2.0 1.0 A Helm chart to export the metrics from couchdb...
prometheus-community/prometheus-druid-exporter 0.9.0 v0.8.0 Druid exporter to monitor druid metrics with Pr...
prometheus-community/prometheus-elasticsearch-e... 4.1.0 1.1.0 Elasticsearch stats exporter for Prometheus
prometheus-community/prometheus-kafka-exporter 1.0.0 v1.2.0 A Helm chart to export the metrics from Kafka i...
prometheus-community/prometheus-mongodb-exporter 2.8.1 v0.10.0 A Prometheus exporter for MongoDB metrics
prometheus-community/prometheus-mysql-exporter 1.0.1 v0.12.1 A Helm chart for prometheus mysql exporter with...
prometheus-community/prometheus-nats-exporter 2.5.1 0.6.2 A Helm chart for prometheus-nats-exporter
prometheus-community/prometheus-node-exporter 1.14.2 1.0.1 A Helm chart for prometheus node-exporter
prometheus-community/prometheus-operator 9.3.2 0.38.1 DEPRECATED - This chart will be renamed. See ht...
prometheus-community/prometheus-pingdom-exporter 2.3.2 20190610-1 A Helm chart for Prometheus Pingdom Exporter
prometheus-community/prometheus-postgres-exporter 1.9.0 0.8.0 A Helm chart for prometheus postgres-exporter
prometheus-community/prometheus-pushgateway 1.7.0 1.3.0 A Helm chart for prometheus pushgateway
prometheus-community/prometheus-rabbitmq-exporter 0.6.0 v0.29.0 Rabbitmq metrics exporter for prometheus
prometheus-community/prometheus-redis-exporter 4.0.0 1.11.1 Prometheus exporter for Redis metrics
prometheus-community/prometheus-snmp-exporter 0.1.1 0.19.0 Prometheus SNMP Exporter
prometheus-community/prometheus-stackdriver-exp... 1.8.0 0.11.0 Stackdriver exporter for Prometheus
prometheus-community/prometheus-statsd-exporter 0.2.0 0.18.0 A Helm chart for prometheus stats-exporter
prometheus-community/prometheus-to-sd 0.4.0 0.5.2 Scrape metrics stored in prometheus format and ...So yeah, there’s a lot to chose from here. I decided to just go with the simple all in one helm chart because it had grafana, alertmanager and all the things I think I’ll need at some point. Maybe it’s a lot of bloat, but we had to start somewhere.
Next thing you’ll start thinking is: Hey, I need to customize this. So I started with grafana.
Grafana Config
There are so many things to customize I didn’t even scratch the surface. My biggest change here was adding sidecars to add my own custom notifiers (slack), dashboards (one’s we created), and data sources (Elasticsearch). (It grabs prometheus automatically).
...
grafana:
sidecar:
enabled: true
dashboards:
label: grafana_dashboard
labelValue: 1
searchNamespace: monitoring
datasources:
label: grafana_datasource
labelValue: 1
searchNamespace: monitoring
notifiers:
enabled: true
label: grafana_notifier
labelValue: 1
searchNamespace: monitoringMy dashboards were then just configmaps. You can use secrets as well to be more secure but I had no real secret configs in there, so it looked something like this:
apiVersion: v1
kind: ConfigMap
metadata:
name: super-dash
namespace: monitoring
labels:
grafana_dashboard: "1"
data:
super-dash.json: |-
{ <some json> }Notice the label value and name has to match the respected label in the config.
The other tricky thing is you need to deploy these dashboards, datasources, and notifiers before grafana loads so it can make use of it. In my circle ci make file I added the command:
kubectl rollout restart -n monitoring deployment kube-prom-grafanaThat way if the dashboards are updated we’ll automatically roll into the change.
After adding all the other configurations I set my eyes on Prometheus.
Prometheus Config
In the same values.yaml file I used with helm I didn’t really change much. I found that I could create Service Monitors. The idea of service monitors is that you watch the Kubernetes cluster for changes and if something new comes along you can start monitoring it. For example, I created a service monitor that looked like this:
# monitor all apps with metrics port in the wg namespace.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: app
labels:
# Make sure this matches your helm release
release: kube-prom
namespace: app
spec:
selector:
matchLabels:
name: myapp
endpoints:
- port: metricsThis was pretty frustrating to get working. So let me clarify what issues you’ll have doing this and how to fix it.
Getting the Service Monitor Picked up
I tried this first a dozen times and never saw anything in my Prometheus dashboard saying it was getting anything. It didn’t even update my config. I then stumbled across this post. I then realized the best way for me (in my case) to have the operator pick up my service monitor was to include the label release: kube-prom The reason for this was that I installed my helm chart with the release name being kube-prom. So that was the first ‘ah hah!’ moment. Then I could finally see it in the app.
Namespace
My apps are all running in the app kubernetes namespace, so I had to deploy it to the app namespace for it to find anything. I was before deploying to the monitoring namespace with the rest of my helm chart. Once I set it in the namespace where the service was running I could find it. Now this may sound really obvious to you but I thought for some reason it would just find it in all name spaces. Guess not.
On to Victory
Finally bringing it up I saw it automatically discover any app that had the metrics endpoint. Super rad. Previous to this we had been hard coding all the values in the config. Now we pick up new services automatically and get everything for free.
I’m really impressed with all the capabilities in these Prometheus operators and glad I finally got to really kick the tires on it this week. Now with a better understanding I’m hoping I can help our application team do a ton more with understanding the system.
