prometheus

Commit Graph

Author	SHA1	Message	Date
Ziqi Zhao	64dfd8a158	fix the bug of setting native histogram min bucket factor (#13846 ) * fix the bug of setting native histogram min bucket factor Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> * Add unit test for checking that min_bucket_factor is correctly applied Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> --------- Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	8 months ago
Bryan Boreham	5ed21c0d76	Merge pull request #12933 from prymitive/duplicated_samples When Prometheus scrapes a target and it sees the same time series repeated multiple times it currently silently ignores that. This change adds a test for that and fixes the scrape loop so that: * Only first sample for each unique time series is appended * Duplicated samples increment the prometheus_target_scrapes_sample_duplicate_timestamp_total metric This allows one to identify such scrape jobs and targets. Also fix some tests and benchmark.	8 months ago
Bryan Boreham	6c41ec984f	[BUGFIX] Scraping: Tolerance should be max 1% of interval Previous code set it at minimum 1%, which was not intended. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	9 months ago
Julien	fcdb6c2b14	Merge pull request #13624 from roidelapluie/tolerance Always align scrapes even if tolerance is bigger than 1% of scrape interval	9 months ago
machine424	f477e0539a	Move from golang.org/x/exp/slices into slices now that we only support Go >= 1.21 Prevent adding back golang.org/x/exp/slices. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	9 months ago
Łukasz Mierzwa	c013a3c1b5	Check of duplicated samples directly in scrapeCache.get() Avoid extra map lookup by hooking check into cache get. ``` goos: linux goarch: amd64 pkg: github.com/prometheus/prometheus/scrape cpu: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz │ main.txt │ new.txt │ │ sec/op │ sec/op vs base │ ScrapeLoopAppend-8 66.72µ ± 0% 66.89µ ± 0% ~ (p=0.879 n=50) ScrapeLoopAppendOM-8 66.61µ ± 0% 66.89µ ± 1% ~ (p=0.115 n=50) geomean 66.66µ 66.89µ +0.34% │ main.txt │ new.txt │ │ B/op │ B/op vs base │ ScrapeLoopAppend-8 20.17Ki ± 1% 20.12Ki ± 1% ~ (p=0.343 n=50) ScrapeLoopAppendOM-8 20.38Ki ± 10% 17.99Ki ± 2% -11.69% (p=0.017 n=50) geomean 20.27Ki 19.03Ki -6.14% │ main.txt │ new.txt │ │ allocs/op │ allocs/op vs base │ ScrapeLoopAppend-8 11.00 ± 0% 11.00 ± 0% ~ (p=1.000 n=50) ¹ ScrapeLoopAppendOM-8 12.00 ± 0% 12.00 ± 0% ~ (p=1.000 n=50) ¹ geomean 11.49 11.49 +0.00% ¹ all samples are equal ``` Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	9 months ago
Łukasz Mierzwa	21f8b35f5b	Move staleness tracking out of checkAddError() calls This call bloats checkAddError signature and logic, we can and should call it from the main scrape logic. Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	9 months ago
Łukasz Mierzwa	55dcaab41b	Fix TestScrapeLoopDiscardDuplicateLabels test This test calls Rollback() which is normally called from within append code. Doing so means that staleness tracking data is outdated and need to by cycled manually. Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	9 months ago
Łukasz Mierzwa	50c81bed86	Check for duplicated series on a scrape When Prometheus scrapes a target and it sees the same time series repeated multiple times it currently silently ignores that. This change adds a test for that and fixes the scrape loop so that: - Only first sample for each unique time series is appended - Duplicated samples increment the prometheus_target_scrapes_sample_duplicate_timestamp_total metric This allows one to identify such scrape jobs and targets. Benchmark results: ``` name old time/op new time/op delta ScrapeLoopAppend-8 64.8µs ± 2% 71.1µs ±20% +9.75% (p=0.000 n=10+10) ScrapeLoopAppendOM-8 64.2µs ± 1% 68.5µs ± 7% +6.71% (p=0.000 n=9+10) TargetsFromGroup/1_targets-8 14.2µs ± 1% 14.5µs ± 1% +1.99% (p=0.000 n=10+10) TargetsFromGroup/10_targets-8 149µs ± 1% 152µs ± 1% +2.05% (p=0.000 n=9+10) TargetsFromGroup/100_targets-8 1.49ms ± 4% 1.48ms ± 1% ~ (p=0.796 n=10+10) name old alloc/op new alloc/op delta ScrapeLoopAppend-8 19.9kB ± 1% 17.8kB ± 3% -10.23% (p=0.000 n=8+10) ScrapeLoopAppendOM-8 19.9kB ± 1% 18.3kB ±10% -8.14% (p=0.001 n=9+10) TargetsFromGroup/1_targets-8 2.43kB ± 0% 2.43kB ± 0% -0.15% (p=0.045 n=10+10) TargetsFromGroup/10_targets-8 24.3kB ± 0% 24.3kB ± 0% ~ (p=0.083 n=10+9) TargetsFromGroup/100_targets-8 243kB ± 0% 243kB ± 0% ~ (p=0.720 n=9+10) name old allocs/op new allocs/op delta ScrapeLoopAppend-8 9.00 ± 0% 9.00 ± 0% ~ (all equal) ScrapeLoopAppendOM-8 10.0 ± 0% 10.0 ± 0% ~ (all equal) TargetsFromGroup/1_targets-8 40.0 ± 0% 40.0 ± 0% ~ (all equal) TargetsFromGroup/10_targets-8 400 ± 0% 400 ± 0% ~ (all equal) TargetsFromGroup/100_targets-8 4.00k ± 0% 4.00k ± 0% ~ (all equal) ``` Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	9 months ago
Łukasz Mierzwa	1a8ea78207	Fix BenchmarkScrapeLoopAppendOM OpenMetrics requires EOF comment at the end of metrics body, but the makeTestMetrics() function doesn't append it. This means this benchmark tests a response with errors but I don't think that was the intention. Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	9 months ago
Bryan Boreham	5f50d974c9	scraping: reset symbol table periodically Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	9 months ago
Bryan Boreham	4e748b9cd8	scraping: re-use labels Builder in scrape report metrics Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	9 months ago
Bryan Boreham	abb3a62f04	scraping: re-use symbol table for scrape loops One symbol table for all loops in the same scrape pool, i.e. from the same job. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	9 months ago
Bryan Boreham	0403d098e1	scraping: re-use symbolTable for target discovery Call labels.NewBuilderWithSymbolTable. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	9 months ago
Bryan Boreham	9ba13de220	scraping: pass a Builder to get Target labels This saves memory allocations from making a new Builder every time. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	9 months ago
Łukasz Mierzwa	5597020a60	Use github.com/klauspost/compress for gzip and zlib klauspost/compress is a high quality drop-in replacement for common Go compression libraries. Since Prometheus sends out a lot of HTTP requests that often return compressed output having improved compression libraries helps to save cpu & memory resources. On a test Prometheus server I was able to see cpu reduction from 31 to 30 cores. Benchmark results: name old time/op new time/op delta TargetScraperGzip/metrics=1-8 69.4µs ± 4% 69.2µs ± 3% ~ (p=0.122 n=50+50) TargetScraperGzip/metrics=100-8 84.3µs ± 2% 80.9µs ± 2% -4.02% (p=0.000 n=48+46) TargetScraperGzip/metrics=1000-8 296µs ± 1% 274µs ±14% -7.35% (p=0.000 n=47+45) TargetScraperGzip/metrics=10000-8 2.06ms ± 1% 1.66ms ± 2% -19.34% (p=0.000 n=47+45) TargetScraperGzip/metrics=100000-8 20.9ms ± 2% 17.5ms ± 3% -16.50% (p=0.000 n=49+50) name old alloc/op new alloc/op delta TargetScraperGzip/metrics=1-8 6.06kB ± 0% 6.07kB ± 0% +0.24% (p=0.000 n=48+48) TargetScraperGzip/metrics=100-8 7.04kB ± 0% 6.89kB ± 0% -2.17% (p=0.000 n=49+50) TargetScraperGzip/metrics=1000-8 9.02kB ± 0% 8.35kB ± 1% -7.49% (p=0.000 n=50+50) TargetScraperGzip/metrics=10000-8 18.1kB ± 1% 16.1kB ± 2% -10.87% (p=0.000 n=47+47) TargetScraperGzip/metrics=100000-8 1.21MB ± 0% 1.01MB ± 2% -16.69% (p=0.000 n=36+50) name old allocs/op new allocs/op delta TargetScraperGzip/metrics=1-8 71.0 ± 0% 72.0 ± 0% +1.41% (p=0.000 n=50+50) TargetScraperGzip/metrics=100-8 81.0 ± 0% 76.0 ± 0% -6.17% (p=0.000 n=50+50) TargetScraperGzip/metrics=1000-8 92.0 ± 0% 83.0 ± 0% -9.78% (p=0.000 n=50+50) TargetScraperGzip/metrics=10000-8 93.0 ± 0% 91.0 ± 0% -2.15% (p=0.000 n=50+50) TargetScraperGzip/metrics=100000-8 111 ± 0% 135 ± 1% +21.89% (p=0.000 n=40+50) Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	9 months ago
Łukasz Mierzwa	92e381b8a3	Add a scrape benchmark with gzipped responses Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	9 months ago
Julien Pivotto	e22b564049	Always align scrapes even if tolerance is bigger than 1% of scrape interval When the scrape tolerance is bigger than 1% of the scrape interval, take 1% of the scrape interval as the tolerance instead of not aligning the scrape at all. Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>	9 months ago
Owen Williams	a28d7865ad	UTF-8: Add support for parsing UTF8 metric and label names This adds support for the new grammar of `{"metric_name", "l1"="val"}` to promql and some of the exposition formats. This grammar will also be valid for non-UTF-8 names. UTF-8 names will not be considered valid unless model.NameValidationScheme is changed. This does not update the go expfmt parser in text_parse.go, which will be addressed by https://github.com/prometheus/common/issues/554/. Part of https://github.com/prometheus/prometheus/issues/13095 Signed-off-by: Owen Williams <owen.williams@grafana.com>	9 months ago
Bryan Boreham	17f48f2b3b	Tests: use replacement DeepEquals in more places Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	10 months ago
Bryan Boreham	d0dee51aac	scrape tests: check NaN values directly Normally, a NaN value is never equal to any other value. Compare sample values via `Float64bits` so that NaN values which are exactly the same will compare equal. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	10 months ago
Bryan Boreham	39af788dbd	Tests: use replacement DeepEquals using go-cmp Use DeepEqual replacement using go-cmp, which is more flexible. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	10 months ago
Paweł Szulik	b0c538787d	Refactor scrape tests to use testify. Signed-off-by: Paweł Szulik <paul.szulik@gmail.com>	10 months ago
Bryan Boreham	4ad9b6df2e	Merge pull request #13336 from machine424/flakky scrape_test.go: Increase scrape interval in TestScrapeLoopCache to reduce potential flakiness.	10 months ago
Ziqi Zhao	df2a0ecf3b	Native Histograms: support `native_histogram_min_bucket_factor` in scrape_config (#13222 ) Native Histograms: support native_histogram_min_bucket_factor in scrape_config --------- Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Co-authored-by: Björn Rabenstein <github@rabenste.in>	10 months ago
machine424	2f60177203	scrape_test.go: Increase scrape interval in TestScrapeLoopCache to reduce potential flakiness Signed-off-by: machine424 <ayoubmrini424@gmail.com>	11 months ago
Julien Pivotto	0763ec841b	Merge pull request #13313 from kalpadiptyaroy/fix-quality-value-accept-header bug: Fix quality value in accept header	11 months ago
Kumar Kalpadiptya Roy	b012366c33	Issue #13268 : fix quality value in accept header Signed-off-by: Kumar Kalpadiptya Roy <kalpadiptya.roy@outlook.com>	11 months ago
Bryan Boreham	75fc8a1535	Merge pull request #13167 from bboreham/simplify-TargetsActive scrape: simplify TargetsActive function	11 months ago
Bryan Boreham	c83e1fc574	textparse: remove MetricType alias No backwards-compatibility; make a clean break. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	11 months ago
Bryan Boreham	8065bef172	Move metric type definitions to common/model They are used in multiple repos, so common is a better place for them. Several packages now don't depend on `model/textparse`, e.g. `storage/remote`. Also remove `metadata` struct from `api.go`, since it was identical to a struct in the `metadata` package. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	11 months ago
Bryan Boreham	99c17b4319	Merge pull request #13177 from bboreham/less-madness scrape: consistent function names for metadata	11 months ago
Arthur Silva Sens	5082655392	Append Created Timestamps (#12733 ) * Append created timestamps. Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com> * Log when created timestamps are ignored Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com> * Proposed changes to Append CT PR. Changes: * Changed textparse Parser interface for consistency and robustness. * Changed CT interface to be more explicit and handle validation. * Simplified test, change scrapeManager to allow testability. * Added TODOs. Signed-off-by: bwplotka <bwplotka@gmail.com> * Updates. Signed-off-by: bwplotka <bwplotka@gmail.com> * Addressed comments. Signed-off-by: bwplotka <bwplotka@gmail.com> * Refactor head_appender test Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com> * Fix linter issues Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com> * Use model.Sample in head appender test Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com> --------- Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com> Signed-off-by: bwplotka <bwplotka@gmail.com> Co-authored-by: bwplotka <bwplotka@gmail.com>	12 months ago
Filip Petkovski	10a82f87fd	Enable reusing memory when converting between histogram types The 'ToFloat' method on integer histograms currently allocates new memory each time it is called. This commit adds an optional *FloatHistogram parameter that can be used to reuse span and bucket slices. It is up to the caller to make sure the input float histogram is not used anymore after the call. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>	12 months ago
Matthieu MOREL	9c4782f1cc	golangci-lint: enable testifylint linter (#13254 ) Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	12 months ago
Björn Rabenstein	980e2895a2	Merge pull request #13129 from fatsheep9146/reduce-resolution-automatically Native Histograms: automatically reduce resolution rather than fail scrape	1 year ago
Ziqi Zhao	19ecc5dd94	add test case for bigGap Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>	1 year ago
Julien Pivotto	965e603fa7	Merge pull request #13184 from bboreham/exemplar-sort Scraping: use slices.sort for exemplars	1 year ago
Julien Pivotto	eda73dd3e5	Merge pull request #13187 from bboreham/refactor-newscrapeloop Scraping tests: refactor scrapeLoop creation	1 year ago
Bryan Boreham	3e287e0170	Scraping tests: refactor scrapeLoop creation Pull boilerplate code into a function. Where appropriate we set some config on the returned object. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	1 year ago
Bryan Boreham	784a2d2c74	Merge pull request #12992 from bboreham/single-scrape-buffer-pool Scraping: share buffer pool across all scrapes	1 year ago
Bryan Boreham	f0e1b592ab	Scraping: use slices.sort for exemplars The sort implementation using Go generics is used everywhere else in Prometheus. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	1 year ago
Paulin Todev	0102425af1	Use only one scrapeMetrics object per test. (#13051 ) The scrape loop and scrape cache should use the same instance. This brings the tests' behavior more in line with production. Signed-off-by: Paulin Todev <paulin.todev@gmail.com>	1 year ago
Bryan Boreham	9051100aba	Scraping: share buffer pool across all scrapes Previously we had one per scrapePool, and one of those per configured scraping job. Each pool holds a few unused buffers, so sharing one across all scrapePools reduces total heap memory. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	1 year ago
Bryan Boreham	34676a240e	scrape: consistent function names for metadata Too confusing to have `MetadataList` and `ListMetadata`, etc. I standardised on the ones which are in an interface. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	1 year ago
Ziqi Zhao	8fe9250f7d	optimize the logic of break the loop of reducing resolution Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>	1 year ago
Bryan Boreham	f095c33da1	scrape: simplify TargetsActive function Since everything was serialized on a single mutex, it's exactly the same if we process targets in sequence without starting goroutines. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	1 year ago
Łukasz Mierzwa	870627fbed	Add enable_compression scrape config option Currently Prometheus will always request gzip compression from the target when sending scrape requests. HTTP compression does reduce the amount of bytes sent over the wire and so is often desirable. The downside of compression is that it requires extra resources - cpu & memory. This also affects the resource usage on the target since it has to compress the response before sending it to Prometheus. This change adds a new option to the scrape job configuration block: enable_compression. The default is true so it remains the same as current Prometheus behaviour. Setting this option to false allows users to disable compression between Prometheus and the scraped target, which will require more bandwidth but it lowers the resource usage of both Prometheus and the target. Fixes #12319. Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	1 year ago
zenador	32ee1b15de	Fix error on ingesting out-of-order exemplars (#13021 ) Fix and improve ingesting exemplars for native histograms. See code comment for a detailed explanation of the algorithm. Note that this changes the current behavior for all kind of samples slightly: We now allow exemplars with the same timestamp as during the last scrape if the value or the labels have changed. Also note that we now do not ingest exemplars without timestamps for native histograms anymore. Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored-by: Björn Rabenstein <github@rabenste.in> --------- Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Signed-off-by: zenador <zenador@users.noreply.github.com> Co-authored-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored-by: Björn Rabenstein <github@rabenste.in>	1 year ago
Ziqi Zhao	10ebeb0a62	rename lastSum -> lastCount && enrich test case with expectBucketCount and expectSchema Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>	1 year ago

1 2 3 4 5 ...

323 Commits (1eb88a8723f90bd97b9e4014fe33e4edcc7477f1)