mirror of https://github.com/prometheus/prometheus
Browse Source
* Initial draft of migration.md * Edits. * Review feedback. * Review feedback. * Staleness link to video; add docker root example; remote config file section. * s/NB/NOTE/, remove external labels link. * More typos. * Add more details link for removed PromQL features. * s/you/your/ * Expand on prom1.8/2.0 side by side setup. * More feedback. * update links. * --query.lookback-delta flag.release-1.8.2
Tom Wilkie
7 years ago
committed by
Tobias Schmidt
1 changed files with 195 additions and 0 deletions
@ -0,0 +1,195 @@ |
|||||||
|
# Prometheus 2.0 migration guide |
||||||
|
|
||||||
|
In line with our [stability promise](https://prometheus.io/blog/2016/07/18/prometheus-1-0-released/#fine-print), |
||||||
|
the Prometheus 2.0 release contains a number of backwards incompatible changes. |
||||||
|
This document offers guidance on migrating from Prometheus 1.8 to Prometheus 2.0. |
||||||
|
|
||||||
|
## Flags |
||||||
|
|
||||||
|
The format of the Prometheus command line flags have changed. Instead of a |
||||||
|
single dash, all flags now use a double dash. Common flags (`--config.file`, |
||||||
|
`--web.listen-address` and `--web.external-url`) are still the same but beyond |
||||||
|
that, almost all the storage-related flags have been removed. |
||||||
|
|
||||||
|
Some notable flags which have been removed: |
||||||
|
- `-alertmanager.url` In Prometheus 2.0, the command line flags for configuring |
||||||
|
a static Alertmanager URL have been removed. Alertmanager must now be |
||||||
|
discovered via service discovery, see [Alertmanager service discovery](#amsd). |
||||||
|
|
||||||
|
- `-log.format` In Prometheus 2.0 logs can only be streamed to standard error. |
||||||
|
|
||||||
|
- `-query.staleness-delta` has been renamed to `--query.lookback-delta`; Prometheus |
||||||
|
2.0 introduces a new mechanism for handling staleness, see [staleness](querying/basics.md#staleness). |
||||||
|
|
||||||
|
- `-storage.local.*` Prometheus 2.0 introduces a new storage engine, as such all |
||||||
|
flags relating to the old engine have been removed. For information on the |
||||||
|
new engine, see [Storage](#storage). |
||||||
|
|
||||||
|
- `-storage.remote.*` Prometheus 2.0 has removed the already deprecated remote |
||||||
|
storage flags, and will fail to start if they are supplied. To write to |
||||||
|
InfluxDB, Graphite, or OpenTSDB use the relevant storage adapter. |
||||||
|
|
||||||
|
## Alertmanager service discovery |
||||||
|
|
||||||
|
Alertmanager service discovery was introduced in Prometheus 1.4, allowing Prometheus |
||||||
|
to dynamically discover Alertmanager replicas using the same mechanism as scrape |
||||||
|
targets. In Prometheus 2.0, the command line flags for static Alertmanager config |
||||||
|
have been removed, so the following command line flag: |
||||||
|
|
||||||
|
``` |
||||||
|
./prometheus -alertmanager.url=http://alertmanager:9093/ |
||||||
|
``` |
||||||
|
|
||||||
|
Would be replaced with the following in the `prometheus.yml` config file: |
||||||
|
|
||||||
|
```yml |
||||||
|
alerting: |
||||||
|
alertmanagers: |
||||||
|
- static_configs: |
||||||
|
- targets: |
||||||
|
- alertmanager:9093 |
||||||
|
``` |
||||||
|
|
||||||
|
You can also use all the usual Prothetheus service discovery integrations and |
||||||
|
relabeling in your Alertmanager configuration. This snippet instructs |
||||||
|
Prometheus to search for Kubernetes pods, in the `default` namespace, with the |
||||||
|
label `name: alertmanager` and with a non-empty port. |
||||||
|
|
||||||
|
```yml |
||||||
|
alerting: |
||||||
|
alertmanagers: |
||||||
|
- kubernetes_sd_configs: |
||||||
|
- role: pod |
||||||
|
tls_config: |
||||||
|
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt |
||||||
|
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token |
||||||
|
relabel_configs: |
||||||
|
- source_labels: [__meta_kubernetes_pod_label_name] |
||||||
|
regex: alertmanager |
||||||
|
action: keep |
||||||
|
- source_labels: [__meta_kubernetes_namespace] |
||||||
|
regex: default |
||||||
|
action: keep |
||||||
|
- source_labels: [__meta_kubernetes_pod_container_port_number] |
||||||
|
regex: |
||||||
|
action: drop |
||||||
|
``` |
||||||
|
|
||||||
|
## Recording rules and alerts |
||||||
|
|
||||||
|
The format for configuring alerting and recording rules has been changed to YAML. |
||||||
|
An example of a recording rule and alert in the old format: |
||||||
|
|
||||||
|
``` |
||||||
|
job:request_duration_seconds:histogram_quantile99 = |
||||||
|
histogram_quantile(0.99, sum(rate(request_duration_seconds_bucket[1m])) by (le, job)) |
||||||
|
|
||||||
|
ALERT FrontendRequestLatency |
||||||
|
IF job:request_duration_seconds:histogram_quantile99{job="frontend"} > 0.1 |
||||||
|
FOR 5m |
||||||
|
ANNOTATIONS { |
||||||
|
summary = "High frontend request latency", |
||||||
|
} |
||||||
|
``` |
||||||
|
|
||||||
|
Would look like this: |
||||||
|
|
||||||
|
```yml |
||||||
|
groups: |
||||||
|
- name: example.rules |
||||||
|
rules: |
||||||
|
- record: job:request_duration_seconds:99percentile |
||||||
|
expr: histogram_quantile(0.99, sum(rate(request_duration_seconds_bucket[1m])) |
||||||
|
BY (le, job)) |
||||||
|
- alert: FrontendRequestLatency |
||||||
|
expr: job:request_duration_seconds:99percentile{job="frontend"} > 0.1 |
||||||
|
for: 5m |
||||||
|
annotations: |
||||||
|
summary: High frontend request latency |
||||||
|
``` |
||||||
|
|
||||||
|
To help with the change, the `promtool` tool has a mode to automate the rules conversion. Given a `.rules` file, it will output a `.rules.yml` file in the |
||||||
|
new format. For example: |
||||||
|
|
||||||
|
``` |
||||||
|
$ promtool update rules example.rules |
||||||
|
``` |
||||||
|
|
||||||
|
## Storage |
||||||
|
|
||||||
|
The data format in Prometheus 2.0 has completely changed and is not backwards |
||||||
|
compatible with 1.8. To retain access to your historic monitoring data we recommend |
||||||
|
you run a non-scraping Prometheus 1.8.1 instance in parallel with your Prometheus 2.0 |
||||||
|
instance, and have the new server read existing data from the old one via the |
||||||
|
remote write protocol. |
||||||
|
|
||||||
|
Your Prometheus 1.8 instance should be started with the following flags and an |
||||||
|
empty config file (`empty.yml`): |
||||||
|
|
||||||
|
``` |
||||||
|
$ ./prometheus-1.8.1.linux-amd64/prometheus -web.listen-address ":9094" -config.file empty.yml |
||||||
|
``` |
||||||
|
|
||||||
|
NOTE: **NOTE** If you used external labels in your Prometheus 2.0 config, they need to be |
||||||
|
preserved in your Prometheus 1.8 config. |
||||||
|
Prometheus 2.0 can then be started (on the same machine) with the following flags: |
||||||
|
|
||||||
|
``` |
||||||
|
$ ./prometheus-2.0.0.linux-amd64/prometheus --config.file prometheus.yml |
||||||
|
``` |
||||||
|
|
||||||
|
Where `prometheus.yml` contains the stanza: |
||||||
|
|
||||||
|
``` |
||||||
|
remote_read: |
||||||
|
- url: "http://localhost:9094/api/v1/read" |
||||||
|
``` |
||||||
|
|
||||||
|
## PromQL |
||||||
|
|
||||||
|
The follow features have been removed from PromQL: |
||||||
|
|
||||||
|
- `drop_common_labels` function - the `without` aggregation modifier should be used |
||||||
|
instead. |
||||||
|
- `keep_common` aggregation modifier - the `by` modifier should be used instead. |
||||||
|
- `count_scalar` function - use cases are better handled by `absent()` or correct |
||||||
|
propagation of labels in operations. |
||||||
|
|
||||||
|
See [issue #3060](https://github.com/prometheus/prometheus/issues/3060) for more |
||||||
|
details. |
||||||
|
|
||||||
|
## Miscellaneous |
||||||
|
|
||||||
|
### Prometheus non-root user |
||||||
|
|
||||||
|
The Prometheus Docker image is now built to [run Prometheus |
||||||
|
as a non-root user](https://github.com/prometheus/prometheus/pull/2859). If you |
||||||
|
want the Prometheus UI/API to listen on a low port number (say, port 80), you'll |
||||||
|
need to override it. For Kubernetes, you would use the following YAML: |
||||||
|
|
||||||
|
```yml |
||||||
|
apiVersion: v1 |
||||||
|
kind: Pod |
||||||
|
metadata: |
||||||
|
name: security-context-demo-2 |
||||||
|
spec: |
||||||
|
securityContext: |
||||||
|
runAsUser: 0 |
||||||
|
... |
||||||
|
``` |
||||||
|
|
||||||
|
See [https://kubernetes.io/docs/tasks/configure-pod-container/security-context/](Configure a Security Context for a Pod or Container) |
||||||
|
for more details. |
||||||
|
|
||||||
|
If you're using Docker, then the follow snippet would be used: |
||||||
|
|
||||||
|
``` |
||||||
|
docker run -u root -p 80:80 prom/prometheus:v2.0.0-rc.2 --web.listen-address :80 |
||||||
|
``` |
||||||
|
|
||||||
|
## Prometheus lifecycle |
||||||
|
|
||||||
|
If you use the Prometheus `/-/reload` HTTP endpoint to [automatically reload your |
||||||
|
Prometheus config when it changes](configuration/configuration.md), |
||||||
|
these endpoints are disabled by default for security reasons in Prometheus 2.0. |
||||||
|
To enable them, set the `--web.enable-lifecycle` flag. |
Loading…
Reference in new issue