|
|
|
@ -30,7 +30,7 @@ This document offers guidance on migrating from Prometheus 2.x to Prometheus 3.0
|
|
|
|
|
represented as `https://example.com/metrics:443` and
|
|
|
|
|
`http://example.com/metrics:80` respectively, add them to your target URLs
|
|
|
|
|
- `agent`
|
|
|
|
|
Instead use the dedicated `--agent` cli flag.
|
|
|
|
|
Instead use the dedicated `--agent` CLI flag.
|
|
|
|
|
|
|
|
|
|
Prometheus v3 will log a warning if you continue to pass these to
|
|
|
|
|
`--enable-feature`.
|
|
|
|
@ -55,14 +55,14 @@ This document offers guidance on migrating from Prometheus 2.x to Prometheus 3.0
|
|
|
|
|
- The `.` pattern in regular expressions in PromQL matches newline characters.
|
|
|
|
|
With this change a regular expressions like `.*` matches strings that include
|
|
|
|
|
`\n`. This applies to matchers in queries and relabel configs. For example the
|
|
|
|
|
following regular expressions now match the accompanying strings, wheras in
|
|
|
|
|
following regular expressions now match the accompanying strings, whereas in
|
|
|
|
|
Prometheus v2 these combinations didn't match.
|
|
|
|
|
|
|
|
|
|
| Regex | Additional matches |
|
|
|
|
|
| ----- | ------ |
|
|
|
|
|
| ".*" | "foo\n", "Foo\nBar" |
|
|
|
|
|
| "foo.?bar" | "foo\nbar" |
|
|
|
|
|
| "foo.+bar" | "foo\nbar" |
|
|
|
|
|
| Regex | Additional matches |
|
|
|
|
|
| ----- | ------ |
|
|
|
|
|
| ".*" | "foo\n", "Foo\nBar" |
|
|
|
|
|
| "foo.?bar" | "foo\nbar" |
|
|
|
|
|
| "foo.+bar" | "foo\nbar" |
|
|
|
|
|
|
|
|
|
|
If you want Prometheus v3 to behave like v2 did, you will have to change your
|
|
|
|
|
regular expressions by replacing all `.` patterns with `[^\n]`, e.g.
|
|
|
|
@ -70,22 +70,22 @@ This document offers guidance on migrating from Prometheus 2.x to Prometheus 3.0
|
|
|
|
|
- Lookback and range selectors are left open and right closed (previously left
|
|
|
|
|
closed and right closed). This change affects queries when the evaluation time
|
|
|
|
|
perfectly aligns with the sample timestamps. For example assume querying a
|
|
|
|
|
timeseries with even spaced samples exactly 1 minute apart. Before Prometheus
|
|
|
|
|
3.x, range query with `5m` will mostly return 5 samples. But if the query
|
|
|
|
|
timeseries with evenly spaced samples exactly 1 minute apart. Before Prometheus
|
|
|
|
|
v3, a range query with `5m` would usually return 5 samples. But if the query
|
|
|
|
|
evaluation aligns perfectly with a scrape, it would return 6 samples. In
|
|
|
|
|
Prometheus 3.x queries like this will always return 5 samples.
|
|
|
|
|
This change has likely few effects for everyday use, except for some sub query
|
|
|
|
|
Prometheus v3 queries like this will always return 5 samples.
|
|
|
|
|
This change has likely few effects for everyday use, except for some subquery
|
|
|
|
|
use cases.
|
|
|
|
|
Query front-ends that align queries usually align sub-queries to multiples of
|
|
|
|
|
the step size. These sub queries will likely be affected.
|
|
|
|
|
Query front-ends that align queries usually align subqueries to multiples of
|
|
|
|
|
the step size. These subqueries will likely be affected.
|
|
|
|
|
Tests are more likely to affected. To fix those either adjust the expected
|
|
|
|
|
number of samples or extend to range by less then one sample interval.
|
|
|
|
|
number of samples or extend the range by less than one sample interval.
|
|
|
|
|
- The `holt_winters` function has been renamed to `double_exponential_smoothing`
|
|
|
|
|
and is now guarded by the `promql-experimental-functions` feature flag.
|
|
|
|
|
If you want to keep using holt_winters, you have to do both of these things:
|
|
|
|
|
- Rename holt_winters to double_exponential_smoothing in your queries.
|
|
|
|
|
If you want to keep using `holt_winters`, you have to do both of these things:
|
|
|
|
|
- Rename `holt_winters` to `double_exponential_smoothing` in your queries.
|
|
|
|
|
- Pass `--enable-feature=promql-experimental-functions` in your Prometheus
|
|
|
|
|
cli invocation..
|
|
|
|
|
CLI invocation.
|
|
|
|
|
|
|
|
|
|
## Scrape protocols
|
|
|
|
|
Prometheus v3 is more strict concerning the Content-Type header received when
|
|
|
|
@ -105,12 +105,12 @@ may now fail if this fallback protocol is not specified.
|
|
|
|
|
|
|
|
|
|
### TSDB format and downgrade
|
|
|
|
|
The TSDB format has been changed in Prometheus v2.55 in preparation for changes
|
|
|
|
|
to the index format. Consequently a Prometheus v3 tsdb can only be read by a
|
|
|
|
|
to the index format. Consequently, a Prometheus v3 TSDB can only be read by a
|
|
|
|
|
Prometheus v2.55 or newer.
|
|
|
|
|
Before upgrading to Prometheus v3 please upgrade to v2.55 first and confirm
|
|
|
|
|
Prometheus works as expected. Only then continue with the upgrade to v3.
|
|
|
|
|
|
|
|
|
|
### TSDB Storage contract
|
|
|
|
|
### TSDB storage contract
|
|
|
|
|
TSDB compatible storage is now expected to return results matching the specified
|
|
|
|
|
selectors. This might impact some third party implementations, most likely
|
|
|
|
|
implementing `remote_read`.
|
|
|
|
@ -123,7 +123,7 @@ endpoints. Furthermore, metric and label names that would have previously been
|
|
|
|
|
flagged as invalid no longer will be.
|
|
|
|
|
|
|
|
|
|
Users wishing to preserve the original validation behavior can update their
|
|
|
|
|
prometheus yaml configuration to specify the legacy validation scheme:
|
|
|
|
|
Prometheus yaml configuration to specify the legacy validation scheme:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
global:
|
|
|
|
@ -159,7 +159,7 @@ time=2024-10-24T00:03:07.542+02:00 level=INFO source=/home/user/go/src/github.co
|
|
|
|
|
|
|
|
|
|
### `le` and `quantile` label values
|
|
|
|
|
In Prometheus v3, the values of the `le` label of classic histograms and the
|
|
|
|
|
`quantile` label of summaries are normalized upon ingestions. In Prometheus v2
|
|
|
|
|
`quantile` label of summaries are normalized upon ingestion. In Prometheus v2
|
|
|
|
|
the value of these labels depended on the scrape protocol (protobuf vs text
|
|
|
|
|
format) in some situations. This led to label values changing based on the
|
|
|
|
|
scrape protocol. E.g. a metric exposed as `my_classic_hist{le="1"}` would be
|
|
|
|
|