mirror of https://github.com/k3s-io/k3s
72 lines
3.3 KiB
Markdown
72 lines
3.3 KiB
Markdown
|
# Prometheus in Kubernetes
|
||
|
|
|
||
|
|
This is an experimental [Prometheus](http://prometheus.io/) setup for monitoring
|
||
|
|
Kubernetes services that expose prometheus-friendly metrics through address
|
||
|
|
http://service_address:service_port/metrics.
|
||
|
|
|
||
|
|
# Purpose
|
||
|
|
The purpose of the setup is to gather performance-related metrics during load
|
||
|
|
tests and analyze them to find and fix bottlenecks.
|
||
|
|
|
||
|
|
# Quick start
|
||
|
|
|
||
|
|
## Promdash/Prometheus
|
||
|
|
|
||
|
|
1. Pick a local directory for promdash. It can be any directory, preferably one which is stable and which you don't mind keeping around. Then (in our case, we use */mnt/promdash*, just run this docker command `docker run -v /mnt/promdash:/mnt/promdash -e DATABASE_URL=sqlite3:/mnt/promdash/file.sqlite3 prom/promdash ./bin/rake db:migrate`. In the future, we might use mysql as the promdash database, however, in any case, this 1 time db setup step is required.
|
||
|
|
|
||
|
|
Now quickly confirm that /mnt/promdash/file.sqlite3 exists, and has a non-zero size, and make sure its permissions are open so that containers can read from it. For example:
|
||
|
|
```
|
||
|
|
[jay@rhbd kubernetes]$ ls -altrh /mnt/promdash/
|
||
|
|
total 20K
|
||
|
|
drwxr-xr-x. 6 root root 4.0K May 6 23:12 ..
|
||
|
|
-rwxrwxrwx 1 root root 12K May 6 23:33 file.sqlite3
|
||
|
|
```
|
||
|
|
Looks open enough :).
|
||
|
|
|
||
|
|
1. Now, you can start this pod, like so `kubectl create -f cluster/add-ons/prometheus/prometheusB3.yaml`. This pod will start both prometheus, the server, as well as promdash, the visualization tool. You can then configure promdash, and next time you restart the pod - you're configuration will be remain (since the promdash directory was mounted as a local docker volume).
|
||
|
|
|
||
|
|
1. Finally, you can simply access localhost:3000, which will have promdash running. Then, add the prometheus server (locahost:9090)to as a promdash server, and create a dashboard according to the promdash directions.
|
||
|
|
|
||
|
|
## Prometheus
|
||
|
|
|
||
|
|
You can launch prometheus easily, by simply running.
|
||
|
|
|
||
|
|
`kubectl create -f cluster/addons/prometheus/prometheus.yaml`
|
||
|
|
|
||
|
|
This will bind to port 9090 locally. You can see the prometheus database at that URL.
|
||
|
|
|
||
|
|
# How it works
|
||
|
|
|
||
|
|
This is a v1beta1 based, containerized prometheus pod, which scrapes endpoints which are readable on the KUBERNETES_RO service (the internal kubernetes service running in the default namespace, which is visible to all pods).
|
||
|
|
|
||
|
|
1. The KUBERNETES_RO service is already running : providing read access to the API metrics.
|
||
|
|
|
||
|
|
1. The list of services to be monitored is passed as a command line aguments in
|
||
|
|
the yaml file.
|
||
|
|
|
||
|
|
1. The startup scripts assumes that each service T will have
|
||
|
|
2 environment variables set ```T_SERVICE_HOST``` and ```T_SERVICE_PORT```
|
||
|
|
|
||
|
|
1. Each can be configured manually in yaml file if you want to monitor something
|
||
|
|
that is not a regular Kubernetes service. For example, you can add comma delimted
|
||
|
|
endpoints which can be scraped like so...
|
||
|
|
```
|
||
|
|
- -t
|
||
|
|
- KUBERNETES_RO,MY_OTHER_METRIC_SERVICE
|
||
|
|
```
|
||
|
|
|
||
|
|
# Other notes
|
||
|
|
|
||
|
|
For regular Kubernetes services the env variables are set up automatically and injected at runtime.
|
||
|
|
|
||
|
|
By default the metrics are written to a temporary location (that can be changed
|
||
|
|
in the the volumes section of the yaml file). Prometheus' UI is available
|
||
|
|
at port 9090.
|
||
|
|
|
||
|
|
# TODO
|
||
|
|
|
||
|
|
- We should publish this image into the kube/ namespace.
|
||
|
|
- Possibly use postgre or mysql as a promdash database.
|
||
|
|
- push gateway (https://github.com/prometheus/pushgateway) setup.
|
||
|
|
- Setup high availability via NFS
|