prometheus

Commit Graph

Author	SHA1	Message	Date
Callum Styan	8fd73b1d28	Add Exemplar Remote Write support (#8296 ) * Write exemplars to the WAL and send them over remote write. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update example for exemplars, print data in a more obvious format. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Add metrics for remote write of exemplars. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix incorrect slices passed to send in remote write. Signed-off-by: Callum Styan <callumstyan@gmail.com> * We need to unregister the new metrics. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address review comments Signed-off-by: Callum Styan <callumstyan@gmail.com> * Order of exemplar append vs write exemplar to WAL needs to change. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Several fixes to prevent sending uninitialized or incorrect samples with an exemplar. Fix dropping exemplar for missing series. Add tests for queue_manager sending exemplars Signed-off-by: Martin Disibio <mdisibio@gmail.com> * Store both samples and exemplars in the same timeseries buffer to remove the alloc when building final request, keep sub-slices in separate buffers for re-use Signed-off-by: Martin Disibio <mdisibio@gmail.com> * Condense sample/exemplar delivery tests to parameterized sub-tests Signed-off-by: Martin Disibio <mdisibio@gmail.com> * Rename test methods for clarity now that they also handle exemplars Signed-off-by: Martin Disibio <mdisibio@gmail.com> * Rename counter variable. Fix instances where metrics were not updated correctly Signed-off-by: Martin Disibio <mdisibio@gmail.com> * Add exemplars to LoadWAL benchmark Signed-off-by: Callum Styan <callumstyan@gmail.com> * last exemplars timestamp metric needs to convert value to seconds with ms precision Signed-off-by: Callum Styan <callumstyan@gmail.com> * Process exemplar records in a separate go routine when loading the WAL. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address review comments related to clarifying comments and variable names. Also refactor sample/exemplar to enqueue prompb types. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Regenerate types proto with comments, update protoc version again. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Put remote write of exemplars behind a feature flag. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address some of Ganesh's review comments. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Move exemplar remote write feature flag to a config file field. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address Bartek's review comments. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Don't allocate exemplar buffers in queue_manager if we're not going to send exemplars over remote write. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Add ValidateExemplar function, validate exemplars when appending to head and log them all to WAL before adding them to exemplar storage. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address more reivew comments from Ganesh. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Add exemplar total label length check. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address a few last review comments Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Martin Disibio <mdisibio@gmail.com>	4 years ago
Simon Pasquier	cf269b7ddd	Update github.com/gogo/protobuf to v1.3.2 Fix for CVE-2021-3121 Signed-off-by: Simon Pasquier <spasquie@redhat.com>	4 years ago
gotjosh	4eca4dffb8	Allow metric metadata to be propagated via Remote Write. (#6815 ) * Introduce a metadata watcher Similarly to the WAL watcher, its purpose is to observe the scrape manager and pull metadata. Then, send it to a remote storage. Signed-off-by: gotjosh <josue@grafana.com> * Additional fixes after rebasing. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Rework samples/metadata metrics. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Use more descriptive variable names in MetadataWatcher collect. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix issues caused during rebasing. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix missing metric add and unneeded config code. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address some review comments. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix metrics and docs Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Replace assert with require Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Bring back max_samples_per_send metric Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix tests Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> Co-authored-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	4 years ago
Julien Pivotto	ef63d8d16d	Update vendors Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Garrett	5a9c4acfbf	Pushdown aggregator group by through read hint (#6401 ) * Pushdown aggregator group by through read hint Implement https://github.com/prometheus/prometheus/issues/6400 * add temporal aggregation pushdown support Signed-off-by: xiancli <xiancli@ebay.com>	5 years ago
Bartek Płotka	48b2c9c8ea	remote-read: streamed chunked server side; Extended protobuf; Added chunked, checksumed reader (#5703 ) Part of: https://github.com/prometheus/prometheus/issues/4517 and https://github.com/improbable-eng/thanos/issues/488 Changes: * Extended protobuf for chunked remote read and negotation. * Added checksumed, chunked Writer/Reader. * Added Server side implementation for chunked streamed remote-read. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>	5 years ago
Julius Volz	b5c833ca21	Update go.mod dependencies before release (#5883 ) * Update go.mod dependencies before release Signed-off-by: Julius Volz <julius.volz@gmail.com> * Add issue for showing query warnings in promtool Signed-off-by: Julius Volz <julius.volz@gmail.com> * Revert json-iterator back to 1.1.6 It produced errors when marshaling Point values with special float values. Signed-off-by: Julius Volz <julius.volz@gmail.com> * Fix expected step values in promtool tests after client_golang update Signed-off-by: Julius Volz <julius.volz@gmail.com> * Update generated protobuf code after proto dep updates Signed-off-by: Julius Volz <julius.volz@gmail.com>	5 years ago
Simon Pasquier	81c4248081	*: bump gRPC and protobuf dependencies (#5367 ) The goal is to remove almost all references to the golang.org/x/net/context package. github.com/gogo/protobuf => v1.2.1 google.golang.org/grpc => v1.19.1 github.com/grpc-ecosystem/grpc-gateway => v1.18.5 It also replaces github.com/cockroachdb/cmux by github.com/soheilhy/cmux because of [1] which fixes #3909 incidentally. [1] https://github.com/grpc/grpc-go/issues/2636 Signed-off-by: Simon Pasquier <spasquie@redhat.com>	6 years ago
Callum Styan	5358f76c5c	update remote write path proto so that Labels/Timeseries can't be nil (#4957 ) Signed-off-by: Callum Styan <callumstyan@gmail.com>	6 years ago
Simon Pasquier	375ad1185c	: bump gRPC dependencies (#5075 ) : bump gRPC dependencies This change updates the gRPC dependencies to more recent versions: github.com/gogo/protobuf => v1.2.0 * github.com/grpc-ecosystem/grpc-gateway => v1.6.3 * google.golang.org/grpc => v1.17.0 In addition scripts/genproto.sh leverages Go modules information instead of hardcoding SHA1 commits. This ensures that the code is generated from the exact same sources. Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Run 'make proto' in CI Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Revert tabs -> spaces change Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Fix 'make proto' step Signed-off-by: Simon Pasquier <spasquie@redhat.com> * 'go get' grpc/protobuf dependencies Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Prepopulate cache with go mod download Signed-off-by: Simon Pasquier <spasquie@redhat.com>	6 years ago
Frederic Branczyk	3287b9a4b8	Merge pull request #4579 from bwplotka/ensure-genproto-works proto: Allow reproducible proto generation.	6 years ago
Bartek Plotka	c3434a39b7	proto: Allow reproducible proto generation. Current way of setting prometheus project to generate repo is tedious. If you just go install packages described in README.md it ends up with those (non buildable code) ``` ../../../prompb/remote.pb.go:61:34: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo ../../../prompb/remote.pb.go:109:33: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo ../../../prompb/remote.pb.go:164:34: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo ../../../prompb/remote.pb.go:214:27: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo ../../../prompb/remote.pb.go:284:33: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo ``` The issue is that the guide presented in proto folder is asking to install some GO packages. It does not mention from what version and there is strong mismatch between those used to generate proto and those that are vendored and used to run the code later... The particular outdated packages in vendor are: `github.com/gogo/protobuf` and `google.golang.org/grpc`. Just updating vendor would be not enough, as it would only tmp fix. I propose adding script that will parse version from vendor.json and install correct version. Changes: - Added script that uses packages in exactly the same version the Go code is build against later. - Regenerated proto with those pinned deps. - Updated REAMDE.md - Added `make proto`Current way of setting prometheus project to generate code from proto files is tedious. If you just go install packages described in README.md, it ends up with this (non buildable code) ``` ../../../prompb/remote.pb.go:61:34: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo ../../../prompb/remote.pb.go:109:33: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo ../../../prompb/remote.pb.go:164:34: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo ../../../prompb/remote.pb.go:214:27: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo ../../../prompb/remote.pb.go:284:33: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo ``` The issue is that the guide presented in proto folder is asking to install some GO packages. It does not mention from what version and there is strong mismatch between those used to generate proto and those that are vendored and used to run the code later... The particular outdated packages in vendor are: `github.com/gogo/protobuf` and `google.golang.org/grpc`. Just updating vendor would be not enough, as it would only tmp fix. I propose adding script that will parse version from vendor.json and install correct version. Changes: - Added script that uses packages in exactly the same version the Go code is build against later. - Regenerated proto with those pinned deps. - Updated REAMDE.md - Added `make proto` Signed-off-by: Bartek Plotka <bwplotka@gmail.com>	6 years ago
Tom Wilkie	d3a1ff1abf	Reduce memory usage of remote read by reducing pointer usage. (#4655 ) Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Thomas Jackson	92c6f0c92e	Add offset to selectParams (#4226 ) * Add Start/End to SelectParams * Make remote read use the new selectParams for start/end This commit will continue sending the start/end time of the remote read query as the overarching promql time and the specific range of data that the query is intersted in receiving a response to is now part of the ReadHints (upstream discussion in #4226). * Remove unused vendored code The genproto.sh script was updated, but the code wasn't regenerated. This simply removes the vendored deps that are no longer part of the codegen output. Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>	6 years ago
Henri DF	2952387ed1	Pass query hints down into remote read query proto (#4122 ) Signed-off-by: Henri DF <henridf@gmail.com>	7 years ago
Krasi Georgiev	99e717ad18	revert ot using the gogofast plugin and regenerate grpc server	7 years ago
Goutham Veeramachaneni	3de10e3b44	Add CleanTombstones API endpoint Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	7 years ago
Krasi Georgiev	5d8f93a22a	now using only github.com/gogo/protobuf bumped all grpc-gateway packages to v1.2.2 updated and run the denproto.sh script	7 years ago
Fabian Reinartz	ccf9e62972	*: add admin grpc API	7 years ago

19 Commits (f041a6a228b1a7e34e1337e5c0974db63820c6cb)