|
|
|
@ -29,8 +29,8 @@ This document offers guidance on migrating from Prometheus 2.x to Prometheus 3.0
|
|
|
|
|
`https://example.com/metrics` or `http://exmaple.com/metrics` to be |
|
|
|
|
represented as `https://example.com/metrics:443` and |
|
|
|
|
`http://example.com/metrics:80` respectively, add them to your target URLs |
|
|
|
|
- `agent` |
|
|
|
|
Instead use the dedicated `--agent` cli flag. |
|
|
|
|
- `agent` |
|
|
|
|
Instead use the dedicated `--agent` CLI flag. |
|
|
|
|
|
|
|
|
|
Prometheus v3 will log a warning if you continue to pass these to |
|
|
|
|
`--enable-feature`. |
|
|
|
@ -55,37 +55,37 @@ This document offers guidance on migrating from Prometheus 2.x to Prometheus 3.0
|
|
|
|
|
- The `.` pattern in regular expressions in PromQL matches newline characters. |
|
|
|
|
With this change a regular expressions like `.*` matches strings that include |
|
|
|
|
`\n`. This applies to matchers in queries and relabel configs. For example the |
|
|
|
|
following regular expressions now match the accompanying strings, wheras in |
|
|
|
|
following regular expressions now match the accompanying strings, whereas in |
|
|
|
|
Prometheus v2 these combinations didn't match. |
|
|
|
|
|
|
|
|
|
| Regex | Additional matches | |
|
|
|
|
| ----- | ------ | |
|
|
|
|
| ".*" | "foo\n", "Foo\nBar" | |
|
|
|
|
| "foo.?bar" | "foo\nbar" | |
|
|
|
|
| "foo.+bar" | "foo\nbar" | |
|
|
|
|
| Regex | Additional matches | |
|
|
|
|
| ----- | ------ | |
|
|
|
|
| ".*" | "foo\n", "Foo\nBar" | |
|
|
|
|
| "foo.?bar" | "foo\nbar" | |
|
|
|
|
| "foo.+bar" | "foo\nbar" | |
|
|
|
|
|
|
|
|
|
If you want Prometheus v3 to behave like v2 did, you will have to change your |
|
|
|
|
regular expressions by replacing all `.` patterns with `[^\n]`, e.g. |
|
|
|
|
regular expressions by replacing all `.` patterns with `[^\n]`, e.g. |
|
|
|
|
`foo[^\n]*`. |
|
|
|
|
- Lookback and range selectors are left open and right closed (previously left |
|
|
|
|
closed and right closed). This change affects queries when the evaluation time |
|
|
|
|
perfectly aligns with the sample timestamps. For example assume querying a |
|
|
|
|
timeseries with even spaced samples exactly 1 minute apart. Before Prometheus |
|
|
|
|
3.x, range query with `5m` will mostly return 5 samples. But if the query |
|
|
|
|
timeseries with evenly spaced samples exactly 1 minute apart. Before Prometheus |
|
|
|
|
v3, a range query with `5m` would usually return 5 samples. But if the query |
|
|
|
|
evaluation aligns perfectly with a scrape, it would return 6 samples. In |
|
|
|
|
Prometheus 3.x queries like this will always return 5 samples. |
|
|
|
|
This change has likely few effects for everyday use, except for some sub query |
|
|
|
|
Prometheus v3 queries like this will always return 5 samples. |
|
|
|
|
This change has likely few effects for everyday use, except for some subquery |
|
|
|
|
use cases. |
|
|
|
|
Query front-ends that align queries usually align sub-queries to multiples of |
|
|
|
|
the step size. These sub queries will likely be affected. |
|
|
|
|
Query front-ends that align queries usually align subqueries to multiples of |
|
|
|
|
the step size. These subqueries will likely be affected. |
|
|
|
|
Tests are more likely to affected. To fix those either adjust the expected |
|
|
|
|
number of samples or extend to range by less then one sample interval. |
|
|
|
|
number of samples or extend the range by less than one sample interval. |
|
|
|
|
- The `holt_winters` function has been renamed to `double_exponential_smoothing` |
|
|
|
|
and is now guarded by the `promql-experimental-functions` feature flag. |
|
|
|
|
If you want to keep using holt_winters, you have to do both of these things: |
|
|
|
|
- Rename holt_winters to double_exponential_smoothing in your queries. |
|
|
|
|
If you want to keep using `holt_winters`, you have to do both of these things: |
|
|
|
|
- Rename `holt_winters` to `double_exponential_smoothing` in your queries. |
|
|
|
|
- Pass `--enable-feature=promql-experimental-functions` in your Prometheus |
|
|
|
|
cli invocation.. |
|
|
|
|
CLI invocation. |
|
|
|
|
|
|
|
|
|
## Scrape protocols |
|
|
|
|
Prometheus v3 is more strict concerning the Content-Type header received when |
|
|
|
@ -105,12 +105,12 @@ may now fail if this fallback protocol is not specified.
|
|
|
|
|
|
|
|
|
|
### TSDB format and downgrade |
|
|
|
|
The TSDB format has been changed in Prometheus v2.55 in preparation for changes |
|
|
|
|
to the index format. Consequently a Prometheus v3 tsdb can only be read by a |
|
|
|
|
to the index format. Consequently, a Prometheus v3 TSDB can only be read by a |
|
|
|
|
Prometheus v2.55 or newer. |
|
|
|
|
Before upgrading to Prometheus v3 please upgrade to v2.55 first and confirm |
|
|
|
|
Prometheus works as expected. Only then continue with the upgrade to v3. |
|
|
|
|
|
|
|
|
|
### TSDB Storage contract |
|
|
|
|
### TSDB storage contract |
|
|
|
|
TSDB compatible storage is now expected to return results matching the specified |
|
|
|
|
selectors. This might impact some third party implementations, most likely |
|
|
|
|
implementing `remote_read`. |
|
|
|
@ -123,7 +123,7 @@ endpoints. Furthermore, metric and label names that would have previously been
|
|
|
|
|
flagged as invalid no longer will be. |
|
|
|
|
|
|
|
|
|
Users wishing to preserve the original validation behavior can update their |
|
|
|
|
prometheus yaml configuration to specify the legacy validation scheme: |
|
|
|
|
Prometheus yaml configuration to specify the legacy validation scheme: |
|
|
|
|
|
|
|
|
|
``` |
|
|
|
|
global: |
|
|
|
@ -159,7 +159,7 @@ time=2024-10-24T00:03:07.542+02:00 level=INFO source=/home/user/go/src/github.co
|
|
|
|
|
|
|
|
|
|
### `le` and `quantile` label values |
|
|
|
|
In Prometheus v3, the values of the `le` label of classic histograms and the |
|
|
|
|
`quantile` label of summaries are normalized upon ingestions. In Prometheus v2 |
|
|
|
|
`quantile` label of summaries are normalized upon ingestion. In Prometheus v2 |
|
|
|
|
the value of these labels depended on the scrape protocol (protobuf vs text |
|
|
|
|
format) in some situations. This led to label values changing based on the |
|
|
|
|
scrape protocol. E.g. a metric exposed as `my_classic_hist{le="1"}` would be |
|
|
|
|