prometheus

Commit Graph

Author	SHA1	Message	Date
beorn7	7a8bb8222c	Style cleanup of all the changes in sparsehistogram so far A lot of this code was hacked together, literally during a hackathon. This commit intends not to change the code substantially, but just make the code obey the usual style practices. A (possibly incomplete) list of areas: * Generally address linter warnings. * The `pgk` directory is deprecated as per dev-summit. No new packages should be added to it. I moved the new `pkg/histogram` package to `model` anticipating what's proposed in #9478. * Make the naming of the Sparse Histogram more consistent. Including abbreviations, there were just too many names for it: SparseHistogram, Histogram, Histo, hist, his, shs, h. The idea is to call it "Histogram" in general. Only add "Sparse" if it is needed to avoid confusion with conventional Histograms (which is rare because the TSDB really has no notion of conventional Histograms). Use abbreviations only in local scope, and then really abbreviate (not just removing three out of seven letters like in "Histo"). This is in the spirit of https://github.com/golang/go/wiki/CodeReviewComments#variable-names * Several other minor name changes. * A lot of formatting of doc comments. For one, following https://github.com/golang/go/wiki/CodeReviewComments#comment-sentences , but also layout question, anticipating how things will look like when rendered by `godoc` (even where `godoc` doesn't render them right now because they are for unexported types or not a doc comment at all but just a normal code comment - consistency is queen!). * Re-enabled `TestQueryLog` and `TestEndopints` (they pass now, leaving them disabled was presumably an oversight). * Bucket iterator for histogram.Histogram is now created with a method. * HistogramChunk.iterator now allows iterator recycling. (I think @dieterbe only commented it out because he was confused by the question in the comment.) * HistogramAppender.Append panics now because we decided to treat staleness marker differently. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	fd5ea4e0b5	Merge branch 'main' into sparsehistogram	3 years ago
Bryan Boreham	92a3eeac55	Create less garbage when parsing metrics (#9299 ) * Refactor: extract function to make scrapeLoop for testing Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Add benchmarks for ScrapeLoopAppend For Prometheus and OpenMetrics Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Create less garbage when parsing metrics Exemplar escapes to heap due to being passed through text-parser interface, but we can reduce the impact by hoisting it out of the loop and resetting it after every use. (Note the cost was paid on every line even when exemplars were disabled) Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Create less garbage when parsing OpenMetrics After calling parseLVals() we always append the return value, so pass in what we want to append it to and save garbage. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Björn Rabenstein	4a85354a2c	Fix protobuf parsing of quantile-less summaries (#9277 ) Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Ganesh Vernekar	8b70e87ab9	Merge remote-tracking branch 'upstream/main' into sparse-refactor Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
beorn7	c35f138a9a	Be more specific when identifying a sparse histogram It's a prefectly valid use case to have a sparse histogram with a zero threshold of zero (i.e. only observations of exactly zero go into the zero bucket). Even if the current PoC implementation of client_golang doesn't allow that, such a case should be ingested properly. However, there is now the edge case af a sparse histogram with a zero threshold of zero and no observations yet. Such a histogram would look the same if it was meant to be a conventional histogram. For now, we ingest this case as a conventional histogram, but the final format should have means to unambiguously express if a histogram is meant to be ingested as a sparse histogram or as a conventional histogram. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	641c3ae199	protobufparse: Add support for remaining types Parser now supports summaries and legacy histograms including exemplars. It also adds the option of specifying exemplars together with a sparse histogram by simply using the legacy bucket section, too. The buckets will be ignored, but the exemplars will be ingested. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	88a6229fc4	protobufparse: add exemplar support for counters Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	9cdd9cfb8e	Add tests for protobuf parser Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	5de2df752f	Hacky implementation of protobuf parsing This "brings back" protobuf parsing, with the only goal to play with the new sparse histograms. The Prom-2.x style parser is highly adapted to the structure of the Prometheus text format (and later OpenMetrics). Some jumping through hoops is required to feed protobuf into it. This is not meant to be a model for the final implementation. It should just enable sparse histogram ingestion at a reasonable efficiency. Following known shortcomings and flaws: - No tests yet. - Summaries and legacy histograms, i.e. without sparse buckets, are ignored. - Staleness doesn't work (but this could be fixed in the appender, to be discussed). - No tricks have been tried that would be similar to the tricks the text parsers do (like direct pointers into the HTTP response body). That makes things weird here. Tricky optimizations only make sense once the final format is specified, which will almost certainly not be the old protobuf format. (Interestingly, I expect this implementation to be in fact much more efficient than the original protobuf ingestion in Prom-1.x.) - This is using a proto3 version of metrics.proto (mostly to be consistent with the other protobuf uses). However, proto3 sees no difference between an unset field. We depend on that to distinguish between an unset timestamp and the timestamp 0 (1970-01-01, 00:00:00 UTC). In this experimental code, we just assume that timestamp is never specified and therefore a timestamp of 0 always is interpreted as "not set". Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
cw9	13cd50c52e	Return error on NaN and Inf timestamps for OpenMetrics parser Signed-off-by: cw9 <chaow929@gmail.com>	3 years ago
Callum Styan	289ba11b79	Add circular in-memory exemplars storage (#6635 ) * Add circular in-memory exemplars storage Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> Signed-off-by: Martin Disibio <mdisibio@gmail.com> Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> Co-authored-by: Tom Wilkie <tom.wilkie@gmail.com> Co-authored-by: Martin Disibio <mdisibio@gmail.com> * Fix some comments, clean up exemplar metrics struct and exemplar tests. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix exemplar query api null vs empty array issue. Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> Co-authored-by: Tom Wilkie <tom.wilkie@gmail.com> Co-authored-by: Martin Disibio <mdisibio@gmail.com>	4 years ago
Julien Pivotto	6c56a1faaa	Testify: move to require (#8122 ) * Testify: move to require Moving testify to require to fail tests early in case of errors. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> * More moves Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	1282d1b39c	Refactor test assertions (#8110 ) * Refactor test assertions This pull request gets rid of assert.True where possible to use fine-grained assertions. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	4e5b1722b3	Move away from testutil, refactor imports (#8087 ) Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Ben Kochie	269e7c8091	Fix golint issues. Signed-off-by: Ben Kochie <superq@gmail.com>	5 years ago
Julien Pivotto	ed623f69e2	tsdb: don't allow ingesting empty labelsets (#6891 ) * tsdb: don't allow ingesting empty labelsets When we ingest an empty labelset in the head, further blocks can not be compacted, with the error: ``` level=error ts=2020-02-27T21:26:58.379Z caller=db.go:659 component=tsdb msg="compaction failed" err="persist head block: write compaction: add series: out-of-order series added with label set \"{}\" / prev: \"{}\"" ``` We should therefore reject those invalid empty labelsets upfront. This can be reproduced with the following: ``` cat << END > prometheus.yml scrape_configs: - job_name: 'prometheus' scrape_interval: 1s basic_auth: username: test password: test metric_relabel_configs: - regex: ".*" action: labeldrop static_configs: - targets: - 127.0.1.1:9090 END ./prometheus --storage.tsdb.min-block-duration=1m ``` And wait a few minutes. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Julien Pivotto	40b66d29ff	OpenMetrics parse: avoid setting prev token (#6781 ) We can avoid setting a prev token in the OM parser. The previous coundition that checked for prev was unreacheable. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Julien Pivotto	9398b9ac1b	OpenMetrics: force input to end with # EOF (#6505 ) Fixes #1169 Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
shreyassrivatsan	e825282dd1	Add exemplar support to the openmetrics parser (#6292 ) * Add exemplar support to the openmetrics parser Signed-off-by: Shreyas Srivatsan <shreyas@chronosphere.io>	5 years ago
Callum Styan	54e3fc82f1	Add a README.md to textparse package about how to update lexers. Signed-off-by: Callum Styan <callumstyan@gmail.com>	5 years ago
zhulongcheng	970ef41bb6	Fix exported function comments (#6002 ) Signed-off-by: zhulongcheng <zhulongcheng.dev@gmail.com>	5 years ago
Brian Brazil	94b1af1471	Update to disallow new Go 1.13 float formats. (#5984 ) Don't inadvertantly expand the acceptable ways to specify a float. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	5 years ago
Brian Brazil	c66bfce8cf	Don't panic on short input. (#5939 ) Fixes #5935 Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	5 years ago
Bartek Plotka	f0863a604e	Removed extra tsdb/testutil after merge. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>	5 years ago
Chris Marchbanks	529ccff07b	Remove all usages of stretchr/testify Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	5 years ago
Tariq Ibrahim	8fdfa8abea	refine error handling in prometheus (#5388 ) i) Uses the more idiomatic Wrap and Wrapf methods for creating nested errors. ii) Fixes some incorrect usages of fmt.Errorf where the error messages don't have any formatting directives. iii) Does away with the use of fmt package for errors in favour of pkg/errors Signed-off-by: tariqibrahim <tariq181290@gmail.com>	6 years ago
Giedrius Statkevičius	10ec2552af	promparse: sort all labels when parsing (#5372 ) * promparse: sort all labels when parsing Some label names might start with an uppercase letter in which case it needs to come before __name__. Without this it means that we are not maintaining the same invariant that all label names should be sorted in such cases. Amend the tests to check this problem automatically. Without the change the tests do not pass. Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>	6 years ago
AixesHunter	9f903fb3f7	github.com/cznic/golex has moved to modernc.org/golex (#5034 ) Signed-off-by: aixeshunter <aixeshunter@gmail.com>	6 years ago
Brian Brazil	3fdb92010e	Expand OM to OpenMetrics Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	6 years ago
Brian Brazil	9c03e11c2c	Hook OpenMetrics parser into scraping. Extend metadata api to support units. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	6 years ago
Brian Brazil	c271ef456a	Add an OpenMetrics parser. This is based on the existing parser, and has slightly better performance (few %) than it. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	6 years ago
Brian Brazil	ffe7efb411	Prepare for multiple text formats Pass content type down to text parser. Add layer of indirection in front of text parser, and rename to avoid future clashes. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	6 years ago
Brian Brazil	70c98a06f1	Handle empty HELP in text parser. (#4444 ) Fixes #4427 Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	6 years ago
Fabian Reinartz	0ff42e754e	pkg/textparse: unescape help string Signed-off-by: Fabian Reinartz <freinartz@google.com>	7 years ago
Fabian Reinartz	5fcc5b6028	pkg/textparse: support null bytes in comments and help Signed-off-by: Fabian Reinartz <freinartz@google.com>	7 years ago
Fabian Reinartz	76a4a46cb0	pkg/textparse: refactor and add metadata handling Extends the parser to allow retrieving metadata. The lexer now yields proper tokens that are fed into a hand-written parser on top. Signed-off-by: Fabian Reinartz <freinartz@google.com>	7 years ago
Filip Ochnik	ec1c6b7c29	textparse: reject empty label names	7 years ago
Julius Volz	c3d6abc8e6	Fix some lint errors (#3334 ) I left the promql ones and some others untouched as I remember that @fabxc prefers them that way.	7 years ago
Goutham Veeramachaneni	11ee713ced	Handle common escape characters properly. (#2995 ) Fixes #2974 Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	7 years ago
Julius Volz	c6f41ce391	Allow metric and label names to begin with underscores (#2961 ) While this is not recommended, it is allowed in our data model (https://prometheus.io/docs/concepts/data_model/). Fixes https://github.com/prometheus/prometheus/issues/2959	7 years ago
Goutham Veeramachaneni	b7eddbcd98	textparse: Add fuzzing and fix bug caught See https://github.com/cznic/golex/issues/11 for info on the bug Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	0e78b7e7c8	parser: support spaces everywhere Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Brian Brazil	dd07f693c8	Single quotes are not valid label value delimiters.	8 years ago
Brian Brazil	a6ca391e6e	Reject scrapes with invalid utf-8 label values.	8 years ago
Fabian Reinartz	bc7aff8cef	retrieval: extract scrape cache	8 years ago
Fabian Reinartz	a83014f53c	retrieval: fix memory leak and consumption for caches	8 years ago
Fabian Reinartz	bdc763f95f	pkg/textparse: allow null bytes in label values	8 years ago
Fabian Reinartz	ea09299ca5	pkg/textparse: handle trailing labels comma (#2752 )	8 years ago
Brian Brazil	beaa7d5a43	Move consistent NaN logic into the parser.	8 years ago

1 2

60 Commits (4a1b84f8b2df50a896f60710200c0a8f06bb9990)