# Prometheus in Kubernetes

This is an experimental [Prometheus](http://prometheus.io/) setup for monitoring
Kubernetes services that expose prometheus-friendly metrics through address
http://service_address:service_port/metrics.

# Purpose
The purpose of the setup is to gather performance-related metrics during load
tests and analyze them to find and fix bottlenecks.

# Quick start

## Promdash/Prometheus

1. Pick a local directory for promdash.  It can be any directory, preferably one which is stable and which you don't mind keeping around.  Then (in our case, we use */mnt/promdash*, just run this docker command `docker run -v /mnt/promdash:/mnt/promdash -e DATABASE_URL=sqlite3:/mnt/promdash/file.sqlite3 prom/promdash ./bin/rake db:migrate`.  In the future, we might use mysql as the promdash database, however, in any case, this 1 time db setup step is required.

Now quickly confirm that /mnt/promdash/file.sqlite3 exists, and has a non-zero size, and make sure its permissions are open so that containers can read from it.  For example:
```
    [jay@rhbd kubernetes]$ ls -altrh /mnt/promdash/
    total 20K
    drwxr-xr-x. 6 root root 4.0K May  6 23:12 ..
    -rwxrwxrwx  1 root root  12K May  6 23:33 file.sqlite3
```
Looks open enough :).  

1. Now, you can start this pod, like so `kubectl create -f contrib/prometheus/prometheus-all.json`.  This ReplicationController will maintain both prometheus, the server, as well as promdash, the visualization tool.  You can then configure promdash, and next time you restart the pod - you're configuration will be remain (since the promdash directory was mounted as a local docker volume).

1. Finally, you can simply access localhost:3000, which will have promdash running.  Then, add the prometheus server (locahost:9090)to as a promdash server, and create a dashboard according to the promdash directions.

## Prometheus 

You can launch prometheus easily, by simply running.

`kubectl create -f contrib/prometheus/prometheus-all.json`

Then (edit the publicIP field in prometheus-service to be a public ip on one of your kubelets), 

and run 

`kubectl create -f contrib/prometheus/prometheus-service.json`

Now, you can access the service `wget 10.0.1.89:9090`, and build graphs.

## How it works

This is a v1beta3 based, containerized prometheus ReplicationController, which scrapes endpoints which are readable on the KUBERNETES_RO service (the internal kubernetes service running in the default namespace, which is visible to all pods).

1. The KUBERNETES_RO service is already running : providing read access to the API metrics.

1. The list of services to be monitored is passed as a command line aguments in
the yaml file.

1. The startup scripts assumes that each service T will have
2 environment variables set ```T_SERVICE_HOST``` and ```T_SERVICE_PORT``` 

1. Each can be configured manually in yaml file if you want to monitor something
that is not a regular Kubernetes service.  For example, you can add comma delimted
endpoints which can be scraped like so...
```
- -t
- KUBERNETES_RO,MY_OTHER_METRIC_SERVICE
```

# Other notes

For regular Kubernetes services the env variables are set up automatically and injected at runtime. 

By default the metrics are written to a temporary location (that can be changed
in the the volumes section of the yaml file). Prometheus' UI is available 
at port 9090.

# TODO

- We should publish this image into the kube/ namespace.
- Possibly use postgre or mysql as a promdash database.
- push gateway (https://github.com/prometheus/pushgateway) setup.

[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/contrib/prometheus/README.md?pixel)]()