prometheus

Commit Graph

Author	SHA1	Message	Date
Björn Rabenstein	6b8573a846	Merge pull request #11687 from prometheus/beorn7/histogram histograms: Optimize query performance	2 years ago
beorn7	717a3f8e25	storage: Manually expand `genericAdd` for specific types This commit is doing what I would have expected that Go generics do for me. However, the PromQL benchmarks show a significant runtime and allocation increase with `genericAdd`, so this replaces it with hand-coded non-generic versions of it. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	551de0346f	promql: Do not return nil slices to the pool Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	817a2396cb	Name float values as "floats", not as "values" In the past, every sample value was a float, so it was fine to call a variable holding such a float "value" or "sample". With native histograms, a sample might have a histogram value. And a histogram value is still a value. Calling a float value just "value" or "sample" or "V" is therefore misleading. Over the last few commits, I already renamed many variables, but this cleans up a few more places where the changes are more invasive. Note that we do not to attempt naming in the JSON APIs or in the protobufs. That would be quite a disruption. However, internally, we can call variables as we want, and we should go with the option of avoiding misunderstandings. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	462240bc78	storage: add specialized buffers to sampleRing This utilizes the fact that most sampleRings will only contain samples of one type. In this case, the generic interface is circumvented, and a bespoke buffer for the one actually occurring sample type is used. Should a sampleRing receive a sample of a different kind later, it will transparently switch to the generic behavior. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	630bcb494b	storage: Use separate sample types for histogram vs. float Previously, we had one “polymorphous” `sample` type in the `storage` package. This commit breaks it up into `fSample`, `hSample`, and `fhSample`, each still implementing the `tsdbutil.Sample` interface. This reduces allocations in `sampleRing.Add` but inflicts the penalty of the interface wrapper, which makes things worse in total. This commit therefore just demonstrates the step taken. The next commit will tackle the interface overhead problem. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	c0879d64cf	promql: Separate `Point` into `FPoint` and `HPoint` In other words: Instead of having a “polymorphous” `Point` that can either contain a float value or a histogram value, use an `FPoint` for floats and an `HPoint` for histograms. This seemingly small change has a _lot_ of repercussions throughout the codebase. The idea here is to avoid the increase in size of `Point` arrays that happened after native histograms had been added. The higher-level data structures (`Sample`, `Series`, etc.) are still “polymorphous”. The same idea could be applied to them, but at each step the trade-offs needed to be evaluated. The idea with this change is to do the minimum necessary to get back to pre-histogram performance for functions that do not touch histograms. Here are comparisons for the `changes` function. The test data doesn't include histograms yet. Ideally, there would be no change in the benchmark result at all. First runtime v2.39 compared to directly prior to this commit: ``` name old time/op new time/op delta RangeQuery/expr=changes(a_one[1d]),steps=1-16 391µs ± 2% 542µs ± 1% +38.58% (p=0.000 n=9+8) RangeQuery/expr=changes(a_one[1d]),steps=10-16 452µs ± 2% 617µs ± 2% +36.48% (p=0.000 n=10+10) RangeQuery/expr=changes(a_one[1d]),steps=100-16 1.12ms ± 1% 1.36ms ± 2% +21.58% (p=0.000 n=8+10) RangeQuery/expr=changes(a_one[1d]),steps=1000-16 7.83ms ± 1% 8.94ms ± 1% +14.21% (p=0.000 n=10+10) RangeQuery/expr=changes(a_ten[1d]),steps=1-16 2.98ms ± 0% 3.30ms ± 1% +10.67% (p=0.000 n=9+10) RangeQuery/expr=changes(a_ten[1d]),steps=10-16 3.66ms ± 1% 4.10ms ± 1% +11.82% (p=0.000 n=10+10) RangeQuery/expr=changes(a_ten[1d]),steps=100-16 10.5ms ± 0% 11.8ms ± 1% +12.50% (p=0.000 n=8+10) RangeQuery/expr=changes(a_ten[1d]),steps=1000-16 77.6ms ± 1% 87.4ms ± 1% +12.63% (p=0.000 n=9+9) RangeQuery/expr=changes(a_hundred[1d]),steps=1-16 30.4ms ± 2% 32.8ms ± 1% +8.01% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=10-16 37.1ms ± 2% 40.6ms ± 2% +9.64% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=100-16 105ms ± 1% 117ms ± 1% +11.69% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1000-16 783ms ± 3% 876ms ± 1% +11.83% (p=0.000 n=9+10) ``` And then runtime v2.39 compared to after this commit: ``` name old time/op new time/op delta RangeQuery/expr=changes(a_one[1d]),steps=1-16 391µs ± 2% 547µs ± 1% +39.84% (p=0.000 n=9+8) RangeQuery/expr=changes(a_one[1d]),steps=10-16 452µs ± 2% 616µs ± 2% +36.15% (p=0.000 n=10+10) RangeQuery/expr=changes(a_one[1d]),steps=100-16 1.12ms ± 1% 1.26ms ± 1% +12.20% (p=0.000 n=8+10) RangeQuery/expr=changes(a_one[1d]),steps=1000-16 7.83ms ± 1% 7.95ms ± 1% +1.59% (p=0.000 n=10+8) RangeQuery/expr=changes(a_ten[1d]),steps=1-16 2.98ms ± 0% 3.38ms ± 2% +13.49% (p=0.000 n=9+10) RangeQuery/expr=changes(a_ten[1d]),steps=10-16 3.66ms ± 1% 4.02ms ± 1% +9.80% (p=0.000 n=10+9) RangeQuery/expr=changes(a_ten[1d]),steps=100-16 10.5ms ± 0% 10.8ms ± 1% +3.08% (p=0.000 n=8+10) RangeQuery/expr=changes(a_ten[1d]),steps=1000-16 77.6ms ± 1% 78.1ms ± 1% +0.58% (p=0.035 n=9+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1-16 30.4ms ± 2% 33.5ms ± 4% +10.18% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=10-16 37.1ms ± 2% 40.0ms ± 1% +7.98% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=100-16 105ms ± 1% 107ms ± 1% +1.92% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1000-16 783ms ± 3% 775ms ± 1% -1.02% (p=0.019 n=9+9) ``` In summary, the runtime doesn't really improve with this change for queries with just a few steps. For queries with many steps, this commit essentially reinstates the old performance. This is good because the many-step queries are the one that matter most (longest absolute runtime). In terms of allocations, though, this commit doesn't make a dent at all (numbers not shown). The reason is that most of the allocations happen in the sampleRingIterator (in the storage package), which has to be addressed in a separate commit. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Bartlomiej Plotka	136b48855a	Merge pull request #12259 from bboreham/labels-overwrite labels: add ScratchBuilder.Overwrite for slice implementation	2 years ago
Bryan Boreham	10cc60af01	labels: add ScratchBuilder.Overwrite for slice implementation This is a method used by some downstream projects; it was created to optimize the implementation in `labels_string.go` but we should have one for both implementations so the same code works with either. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Björn Rabenstein	8ed90b567b	Merge pull request #12234 from aknuds1/chore/improve-histogram-comments tsdb: Improve a couple of histogram documentation comments	2 years ago
Björn Rabenstein	6e0a46900b	Merge pull request #12192 from leizor/leizor/prometheus/issues/11204 Add support for native histograms to concreteSeriesIterator	2 years ago
Arve Knudsen	cca7178a12	tsdb: Improve a couple of histogram documentation comments Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
Justin Lei	f90013a5a0	Update storage/remote/codec.go Co-authored-by: Björn Rabenstein <github@rabenste.in> Signed-off-by: Justin Lei <97976793+leizor@users.noreply.github.com>	2 years ago
Justin Lei	83f43982c9	Add support for native histograms to concreteSeriesIterator Signed-off-by: Justin Lei <justin.lei@grafana.com>	2 years ago
Julien Pivotto	8dba9163f1	Merge pull request #12216 from prometheus/dependabot/go_modules/github.com/Azure/go-autorest/autorest/adal-0.9.23 build(deps): bump github.com/Azure/go-autorest/autorest/adal from 0.9.22 to 0.9.23	2 years ago
Julien Pivotto	391473141d	Check health & ready: move to flags (#12223 ) This makes it more consistent with other command like import rules. We don't have stricts rules and uniformity accross promtool unfortunately, but I think it's better to only have the http config on relevant check commands to avoid thinking Prometheus can e.g. check the config over the wire. Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>	2 years ago
Soon-Ping	6cecb87941	Generalized rule group iteration evaluation hook (#11885 ) Signed-off-by: Soon-Ping Phang <soonping@amazon.com>	2 years ago
Chris Marchbanks	044b3ecd14	Merge pull request #12203 from bboreham/raise-max-samples-per-send remote-write: raise default samples per send to 2,000	2 years ago
Ganesh Vernekar	e709b0b36e	Merge pull request #12127 from codesome/ooo-mmap-replay Update OOO min/max time properly after replaying m-map chunks	2 years ago
Ganesh Vernekar	5588cab8b2	Merge pull request #12173 from bboreham/builder-no-empty-labels labels: simplify call to get Labels from Builder	2 years ago
Alex Le	1936868e9d	Allow populate block logic in compact to be overriden outside Prometheus (#11711 ) Signed-off-by: Alex Le <leqiyue@amazon.com> Signed-off-by: Alex Le <emoc1989@gmail.com>	2 years ago
dependabot[bot]	3923e83413	build(deps): bump bufbuild/buf-setup-action from 1.13.1 to 1.16.0 (#12209 ) Bumps [bufbuild/buf-setup-action](https://github.com/bufbuild/buf-setup-action) from 1.13.1 to 1.16.0. - [Release notes](https://github.com/bufbuild/buf-setup-action/releases) - [Commits](https://github.com/bufbuild/buf-setup-action/compare/v1.13.1...v1.16.0) --- updated-dependencies: - dependency-name: bufbuild/buf-setup-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2 years ago
Nidhey Nitin Indurkar	3f7beeecc6	feat: health and readiness check of prometheus server in CLI (promtool) (#12096 ) * feat: health and readiness check of prometheus server in CLI (promtool) Signed-off-by: nidhey27 <nidhey.indurkar@infracloud.io>	2 years ago
dependabot[bot]	c8b408cf5e	build(deps): bump github.com/Azure/go-autorest/autorest/adal Bumps [github.com/Azure/go-autorest/autorest/adal](https://github.com/Azure/go-autorest) from 0.9.22 to 0.9.23. - [Release notes](https://github.com/Azure/go-autorest/releases) - [Changelog](https://github.com/Azure/go-autorest/blob/main/CHANGELOG.md) - [Commits](https://github.com/Azure/go-autorest/compare/autorest/adal/v0.9.22...autorest/adal/v0.9.23) --- updated-dependencies: - dependency-name: github.com/Azure/go-autorest/autorest/adal dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2 years ago
Julien Pivotto	4af28f8cf6	Merge pull request #12204 from prymitive/cmd_tests Use a random port in cmd/prometheus tests	2 years ago
Julien Pivotto	be07be7ff1	Merge pull request #12208 from prometheus/dependabot/github_actions/actions/cache-3.3.1 build(deps): bump actions/cache from 3.2.4 to 3.3.1	2 years ago
Julien Pivotto	b5249ba367	Merge pull request #12215 from prometheus/dependabot/go_modules/github.com/miekg/dns-1.1.53 build(deps): bump github.com/miekg/dns from 1.1.51 to 1.1.53	2 years ago
Julien Pivotto	e37e53ac54	Merge pull request #12210 from prometheus/dependabot/github_actions/actions/setup-go-4 build(deps): bump actions/setup-go from 3 to 4	2 years ago
Julien Pivotto	fa25c3b5e5	Merge pull request #12213 from prometheus/dependabot/go_modules/github.com/ionos-cloud/sdk-go/v6-6.1.5 build(deps): bump github.com/ionos-cloud/sdk-go/v6 from 6.1.4 to 6.1.5	2 years ago
Julien Pivotto	53c66ee391	Merge pull request #12211 from prometheus/dependabot/github_actions/prometheus/promci-0.1.0 build(deps): bump prometheus/promci from 0.0.2 to 0.1.0	2 years ago
Julien Pivotto	bb6d7e36bd	Merge pull request #12214 from prometheus/dependabot/go_modules/github.com/digitalocean/godo-1.98.0 build(deps): bump github.com/digitalocean/godo from 1.97.0 to 1.98.0	2 years ago
Julien Pivotto	b045f4169e	Merge pull request #12217 from prometheus/dependabot/go_modules/google.golang.org/api-0.114.0 build(deps): bump google.golang.org/api from 0.111.0 to 0.114.0	2 years ago
Julien Pivotto	a65bcb22de	Merge pull request #12218 from prometheus/dependabot/go_modules/documentation/examples/remote_storage/github.com/prometheus/prometheus-0.43.0 build(deps): bump github.com/prometheus/prometheus from 0.42.0 to 0.43.0 in /documentation/examples/remote_storage	2 years ago
Bartlomiej Plotka	b250479a65	Merge pull request #12220 from prometheus/superq/errcheck Move errcheck excludes config	2 years ago
Ganesh Vernekar	291ab4d0bc	Add Jesús Vázquez as a TSDB maintainer (#12222 ) Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
Łukasz Mierzwa	f2b9a39a48	Use a random port in cmd/prometheus tests There are a few tests that will run prometheus command. This can test if there's already something listening on port :9090 since --web.listen-address defaults to 0.0.0.0:9090. To fix that we can tell prometheus to use a random port on loopback interface. Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	2 years ago
SuperQ	8b01189327	Move errcheck excludes config Eliminate the need for a second config file for golangci-lint config file by moving the list of errcheck exclude functions into the yaml config. Signed-off-by: SuperQ <superq@gmail.com>	2 years ago
Bryan Boreham	e917202766	labels: make sure estimated size is not negative Deleted labels are remembered, even if they were not in `base` or were removed from `add`, so `base+add-del` could go negative. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	889ef998f4	remote-write: adjust MaxShards and Capacity for larger MaxSamplesPerSend Keep the target throughput and total pending samples the same. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Hayk Davtyan	408f31f786	[WebUI/ScrapePoolList] Case-insensitive search of "Scrape Pools" (#12207 ) Signed-off-by: hayk96 <hayko5999@gmail.com>	2 years ago
dependabot[bot]	ce353727f1	build(deps): bump github.com/prometheus/prometheus Bumps [github.com/prometheus/prometheus](https://github.com/prometheus/prometheus) from 0.42.0 to 0.43.0. - [Release notes](https://github.com/prometheus/prometheus/releases) - [Changelog](https://github.com/prometheus/prometheus/blob/main/CHANGELOG.md) - [Commits](https://github.com/prometheus/prometheus/compare/v0.42.0...v0.43.0) --- updated-dependencies: - dependency-name: github.com/prometheus/prometheus dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>	2 years ago
dependabot[bot]	0b7cf5b368	build(deps): bump google.golang.org/api from 0.111.0 to 0.114.0 Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.111.0 to 0.114.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.111.0...v0.114.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>	2 years ago
dependabot[bot]	6f2507656c	build(deps): bump github.com/miekg/dns from 1.1.51 to 1.1.53 Bumps [github.com/miekg/dns](https://github.com/miekg/dns) from 1.1.51 to 1.1.53. - [Release notes](https://github.com/miekg/dns/releases) - [Changelog](https://github.com/miekg/dns/blob/master/Makefile.release) - [Commits](https://github.com/miekg/dns/compare/v1.1.51...v1.1.53) --- updated-dependencies: - dependency-name: github.com/miekg/dns dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2 years ago
dependabot[bot]	bf988304d7	build(deps): bump github.com/digitalocean/godo from 1.97.0 to 1.98.0 Bumps [github.com/digitalocean/godo](https://github.com/digitalocean/godo) from 1.97.0 to 1.98.0. - [Release notes](https://github.com/digitalocean/godo/releases) - [Changelog](https://github.com/digitalocean/godo/blob/main/CHANGELOG.md) - [Commits](https://github.com/digitalocean/godo/compare/v1.97.0...v1.98.0) --- updated-dependencies: - dependency-name: github.com/digitalocean/godo dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>	2 years ago
dependabot[bot]	babdbd8add	build(deps): bump github.com/ionos-cloud/sdk-go/v6 from 6.1.4 to 6.1.5 Bumps [github.com/ionos-cloud/sdk-go/v6](https://github.com/ionos-cloud/sdk-go) from 6.1.4 to 6.1.5. - [Release notes](https://github.com/ionos-cloud/sdk-go/releases) - [Changelog](https://github.com/ionos-cloud/sdk-go/blob/master/docs/CHANGELOG.md) - [Commits](https://github.com/ionos-cloud/sdk-go/compare/v6.1.4...v6.1.5) --- updated-dependencies: - dependency-name: github.com/ionos-cloud/sdk-go/v6 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2 years ago
dependabot[bot]	a9af885de2	build(deps): bump prometheus/promci from 0.0.2 to 0.1.0 Bumps [prometheus/promci](https://github.com/prometheus/promci) from 0.0.2 to 0.1.0. - [Release notes](https://github.com/prometheus/promci/releases) - [Commits](https://github.com/prometheus/promci/compare/v0.0.2...v0.1.0) --- updated-dependencies: - dependency-name: prometheus/promci dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>	2 years ago
dependabot[bot]	e15b0a29ac	build(deps): bump actions/setup-go from 3 to 4 Bumps [actions/setup-go](https://github.com/actions/setup-go) from 3 to 4. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](https://github.com/actions/setup-go/compare/v3...v4) --- updated-dependencies: - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>	2 years ago
dependabot[bot]	2bd2a6006d	build(deps): bump actions/cache from 3.2.4 to 3.3.1 Bumps [actions/cache](https://github.com/actions/cache) from 3.2.4 to 3.3.1. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/v3.2.4...v3.3.1) --- updated-dependencies: - dependency-name: actions/cache dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>	2 years ago
Bryan Boreham	2fc2e233a6	remote-write: raise default samples per send to 2,000 Larger messages cost less, because the per-message overheads at sender and receiver are spread over more samples. Example: scraping 1.5 million series every 15 seconds will send 50 messages per second. Previous default was 500. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Ganesh Vernekar	f55ab22179	Merge pull request #12186 from codesome/remove-file Remove mistakenly added file	2 years ago

1 2 3 4 5 ...

11086 Commits (6b8573a846af602bd7c115e652cc92f869063205) All Branches Search

11086 Commits (6b8573a846af602bd7c115e652cc92f869063205)

All Branches