prometheus

Commit Graph

Author	SHA1	Message	Date
beorn7	87351f2318	prompb: Modify layout of histograms Note: This is deliberately an incompatible change. Since we have never used histograms in remote read/write yet, there is no point in keeping compatibility. This _is_, however, compatible to the state in the main branch. This commit flattens the bucket message into top-level fields. This has the disadvantage of now having two triples of fields prefixed with `negative_...` or `positive_...`. However, with this structure, we save one tag on the wire. And, perhaps more importantly, we mirror the structure of the `histogram.Histogram` Go type. This commit also adjusts `repeated` fields to use names in the plural form, as it is also the case for the fields that already existed. This also adds a doc comment to `HistogramProtoToHistogram` and changes its return type to a pointer (which is more convenient and probably more efficient). Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	a38ee22110	documentation: fix remote_storage examples Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Levi Harrison	08f3ddb864	Sparse histogram remote-write support (#11001 )	2 years ago
beorn7	20a3990500	documentation: fix example dependencies The examples were still depending on an ancient prometheus version. Updating caused some dependency shenanigans, but this should work for now. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
dependabot[bot]	867d3bd78f	build(deps): bump github.com/go-kit/log (#10827 ) Bumps [github.com/go-kit/log](https://github.com/go-kit/log) from 0.2.0 to 0.2.1. - [Release notes](https://github.com/go-kit/log/releases) - [Commits](https://github.com/go-kit/log/compare/v0.2.0...v0.2.1) --- updated-dependencies: - dependency-name: github.com/go-kit/log dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	3 years ago
dependabot[bot]	7b8ed5d36b	build(deps): bump github.com/prometheus/common (#10826 ) Bumps [github.com/prometheus/common](https://github.com/prometheus/common) from 0.32.1 to 0.34.0. - [Release notes](https://github.com/prometheus/common/releases) - [Commits](https://github.com/prometheus/common/compare/v0.32.1...v0.34.0) --- updated-dependencies: - dependency-name: github.com/prometheus/common dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	3 years ago
dependabot[bot]	eef02a0334	build(deps): bump github.com/prometheus/client_golang (#10828 ) Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.12.1 to 1.12.2. - [Release notes](https://github.com/prometheus/client_golang/releases) - [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md) - [Commits](https://github.com/prometheus/client_golang/compare/v1.12.1...v1.12.2) --- updated-dependencies: - dependency-name: github.com/prometheus/client_golang dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	3 years ago
dependabot[bot]	6bd75d5ba7	build(deps): bump github.com/stretchr/testify (#10829 ) Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.7.0 to 1.7.2. - [Release notes](https://github.com/stretchr/testify/releases) - [Commits](https://github.com/stretchr/testify/compare/v1.7.0...v1.7.2) --- updated-dependencies: - dependency-name: github.com/stretchr/testify dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	3 years ago
Matthieu MOREL	12de742ae4	refactor (documentation): move from github.com/pkg/errors to 'errors' and 'fmt' (#10808 ) Signed-off-by: Matthieu MOREL <mmorel-35@users.noreply.github.com> Co-authored-by: Matthieu MOREL <mmorel-35@users.noreply.github.com>	3 years ago
Matthieu MOREL	e2ede285a2	refactor: move from io/ioutil to io and os packages (#10528 ) * refactor: move from io/ioutil to io and os packages * use fs.DirEntry instead of os.FileInfo after os.ReadDir Signed-off-by: MOREL Matthieu <matthieu.morel@cnp.fr>	3 years ago
Julien Pivotto	8cc7b7e577	Split remote storage example in its own go mod (#10244 ) This commit removes the dependency between Prometheus and influx. Note: Go keeps adding the indirect dependencies in go.mod, I can't remove them. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	3 years ago
Mateusz Gozdek	1a6c2283a3	Format Go source files using 'gofumpt -w -s -extra' Part of #9557 Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>	3 years ago
Paweł Szulik	f5563bfe95	tests: Move from t.Errorf and others. (Part 2) (#9309 ) * Refactor util tests. Signed-off-by: Paweł Szulik <paul.szulik@gmail.com>	3 years ago
Levi Harrison	b5f6f8fb36	Switched to go-kit/log Signed-off-by: Levi Harrison <git@leviharrison.dev>	4 years ago
Callum Styan	8fd73b1d28	Add Exemplar Remote Write support (#8296 ) * Write exemplars to the WAL and send them over remote write. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update example for exemplars, print data in a more obvious format. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Add metrics for remote write of exemplars. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix incorrect slices passed to send in remote write. Signed-off-by: Callum Styan <callumstyan@gmail.com> * We need to unregister the new metrics. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address review comments Signed-off-by: Callum Styan <callumstyan@gmail.com> * Order of exemplar append vs write exemplar to WAL needs to change. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Several fixes to prevent sending uninitialized or incorrect samples with an exemplar. Fix dropping exemplar for missing series. Add tests for queue_manager sending exemplars Signed-off-by: Martin Disibio <mdisibio@gmail.com> * Store both samples and exemplars in the same timeseries buffer to remove the alloc when building final request, keep sub-slices in separate buffers for re-use Signed-off-by: Martin Disibio <mdisibio@gmail.com> * Condense sample/exemplar delivery tests to parameterized sub-tests Signed-off-by: Martin Disibio <mdisibio@gmail.com> * Rename test methods for clarity now that they also handle exemplars Signed-off-by: Martin Disibio <mdisibio@gmail.com> * Rename counter variable. Fix instances where metrics were not updated correctly Signed-off-by: Martin Disibio <mdisibio@gmail.com> * Add exemplars to LoadWAL benchmark Signed-off-by: Callum Styan <callumstyan@gmail.com> * last exemplars timestamp metric needs to convert value to seconds with ms precision Signed-off-by: Callum Styan <callumstyan@gmail.com> * Process exemplar records in a separate go routine when loading the WAL. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address review comments related to clarifying comments and variable names. Also refactor sample/exemplar to enqueue prompb types. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Regenerate types proto with comments, update protoc version again. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Put remote write of exemplars behind a feature flag. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address some of Ganesh's review comments. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Move exemplar remote write feature flag to a config file field. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address Bartek's review comments. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Don't allocate exemplar buffers in queue_manager if we're not going to send exemplars over remote write. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Add ValidateExemplar function, validate exemplars when appending to head and log them all to WAL before adding them to exemplar storage. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address more reivew comments from Ganesh. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Add exemplar total label length check. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address a few last review comments Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Martin Disibio <mdisibio@gmail.com>	4 years ago
Tom Wilkie	d479151f1f	Various enhancements and refactorings for remote write receiver: - Remove unrelated changes - Refactor code out of the API module - that is already getting pretty crowded. - Don't track reference for AddFast in remote write. This has the potential to consume unlimited server-side memory if a malicious client pushes a different label set for every series. For now, its easier and safer to always use the 'slow' path. - Return 400 on out of order samples. - Use remote.DecodeWriteRequest in the remote write adapters. - Put this behing the 'remote-write-server' feature flag - Add some (very) basic docs. - Used named return & add test for commit error propagation Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	4 years ago
Julien Pivotto	4e5b1722b3	Move away from testutil, refactor imports (#8087 ) Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Marek Slabicki	8224ddec23	Capitalizing first letter of all log lines (#7043 ) Signed-off-by: Marek Slabicki <thaniri@gmail.com>	5 years ago
Vadym Martsynovskyy	a9970a47ef	Fix incorrect examples in docs Signed-off-by: Vadym Martsynovskyy <vmartsynovskyy@gmail.com>	5 years ago
Romain Baugue	95193fa027	Exhaust every request body before closing it (#5166 ) (#5479 ) From the documentation: > The default HTTP client's Transport may not > reuse HTTP/1.x "keep-alive" TCP connections if the Body is > not read to completion and closed. This effectively enable keep-alive for the fixed requests. Signed-off-by: Romain Baugue <romain.baugue@elwinar.com>	6 years ago
qinng	cc75c27580	Fix multiple response.WriteHeader calls error in remote read adapter (#5159 ) * fix multiple response.WriteHeader calls in remote read adapter * remove useless return Signed-off-by: qinng <guoruyi1@xiaomi.com>	6 years ago
Tariq Ibrahim	8fdfa8abea	refine error handling in prometheus (#5388 ) i) Uses the more idiomatic Wrap and Wrapf methods for creating nested errors. ii) Fixes some incorrect usages of fmt.Errorf where the error messages don't have any formatting directives. iii) Does away with the use of fmt package for errors in favour of pkg/errors Signed-off-by: tariqibrahim <tariq181290@gmail.com>	6 years ago
LongKB	23480bef43	Remove the duplicated words (#5251 ) Although it is spelling mistakes, it might make an affects while reading. Co-Authored-By: Nguyen Phuong An <AnNP@vn.fujitsu.com> Signed-off-by: Kim Bao Long <longkb@vn.fujitsu.com>	6 years ago
Callum Styan	5358f76c5c	update remote write path proto so that Labels/Timeseries can't be nil (#4957 ) Signed-off-by: Callum Styan <callumstyan@gmail.com>	6 years ago
Alex Yu	5dcce32ef8	update promlog to latest version (#4876 ) * update promlog to latest version Signed-off-by: Alex Yu <yu.alex96@gmail.com> * Update api tests, fix main setup Signed-off-by: Alex Yu <yu.alex96@gmail.com> * tidy go.sum Signed-off-by: Alex Yu <yu.alex96@gmail.com> * revendor prometheus/common Signed-off-by: Alex Yu <yu.alex96@gmail.com> * only initialize config; use kingpin for remote_storage_adapter Signed-off-by: Alex Yu <yu.alex96@gmail.com> * actually parse the flags Signed-off-by: Alex Yu <yu.alex96@gmail.com> * clean up imports Signed-off-by: Alex Yu <yu.alex96@gmail.com>	6 years ago
Simon Pasquier	ed19373a78	: remove use of golang.org/x/net/context (#4869 ) : remove use of golang.org/x/net/context Signed-off-by: Simon Pasquier <spasquie@redhat.com> scrape: fix TestTargetScrapeScrapeCancel Signed-off-by: Simon Pasquier <spasquie@redhat.com>	6 years ago
Tom Wilkie	d3a1ff1abf	Reduce memory usage of remote read by reducing pointer usage. (#4655 ) Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
beorn7	4fb59d1e61	Remove use of deprecated prometheus.Handler Signed-off-by: beorn7 <beorn@soundcloud.com>	6 years ago
Julius Volz	8fbe1b5133	Handle a bunch of unchecked errors (#4461 ) There are many more (mostly finalizers like Close/Stop/etc.), but most of the others seemed like one couldn't do much about them anyway. Signed-off-by: Julius Volz <julius.volz@gmail.com>	6 years ago
Harsh Agarwal	6a464ae174	expose log.level for promlog for remote_storage_adapter (#4195 ) * expose log.level for promlog for remote_storage_adapter Signed-off-by: sipian <cs15btech11019@iith.ac.in> * replace flag description Signed-off-by: Harsh Agarwal <cs15btech11019@iith.ac.in>	6 years ago
Peter Gallerani	a9d5034add	Fix missing 'msg' in remote storage adapter main.go .Log info message (#4377 ) Signed-off-by: Peter Gallerani <peter.gallerani@gmail.com>	6 years ago
ferhat elmas	ffa673f7d8	General simplifications (#3887 ) Another try as in #1516	7 years ago
Matthew Pound	e6dcc72f9a	Fix instructions on updating prometheus.yml for Remote Write Adapter Example (#3422 )	7 years ago
Krasi Georgiev	5d8f93a22a	now using only github.com/gogo/protobuf bumped all grpc-gateway packages to v1.2.2 updated and run the denproto.sh script	7 years ago
Julius Volz	099df0c5f0	Migrate "golang.org/x/net/context" -> "context" (#3333 ) In some places, where ctxhttp or gRPC are concerned, we still need to use the old contexts.	7 years ago
Fabian Reinartz	abf7c975c9	Merge branch 'master' into dev-2.0	7 years ago
Jack Neely	128b31d058	Log failure to send NaN values to remote store as Debug (#3235 ) This was a warning and can be a frequent occurrence. Let's not fill up logs unless we are asked to.	7 years ago
Goutham Veeramachaneni	3f0267c548	Merge branch 'dev-2.0' into go-kit/log Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	7 years ago
Goutham Veeramachaneni	f5aed810f9	logging: Port to common/promlog Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	7 years ago
Fabian Reinartz	d21f149745	*: migrate to go-kit/log	7 years ago
Fabian Reinartz	87918f3097	Merge branch 'master' into dev-2.0	7 years ago
Julius Volz	aa5cdcb11e	Remove extra space in log output	7 years ago
gdmello	35c952e344	Added logging for remote storage adapter (#3106 ) * Added logging for remote storage adapter on startup and on any error condition during /read or /write. * CR feedback.	7 years ago
Tom Wilkie	cf105f9d57	Update example remote adapters for change in proto location.	7 years ago
Julius Volz	e0f046396a	Fix InfluxDB retention policy usage in read adapter (#2781 )	8 years ago
Tom Wilkie	3141a6b36b	Compress remote storage requests and responses with unframed/raw snappy. (#2696 ) * Compress remote storage requests and responses with unframed/raw snappy, for compatibility with other languages. * Remove backwards compatibility code from remote_storage_adapter, update example_write_adapter * Add /documentation/examples/remote_storage/example_write_adapter/example_writer_adapter to .gitignore	8 years ago
Brian Brazil	0e0fc5a7f4	Correct example name to adapter. (#2590 )	8 years ago
Brian Brazil	c813c824d4	Separate out remote read responses. Fixes #2574	8 years ago
Julius Volz	3581057ea4	Update remote storage bridge README.md	8 years ago
Julius Volz	b391cbb808	Add InfluxDB read-back support to remote storage bridge	8 years ago
Julius Volz	815762a4ad	Move retrieval.NewHTTPClient -> httputil.NewClientFromConfig	8 years ago
Julius Volz	beb3c4b389	Remove legacy remote storage implementations This removes legacy support for specific remote storage systems in favor of only offering the generic remote write protocol. An example bridge application that translates from the generic protocol to each of those legacy backends is still provided at: documentation/examples/remote_storage/remote_storage_bridge See also https://github.com/prometheus/prometheus/issues/10 The next step in the plan is to re-add support for multiple remote storages.	8 years ago
Julius Volz	b16371595d	Add standalone remote storage bridge example In preparation for removing specific remote storage implementations, this offers an example of how to achieve the same in a separate process. Rather than having three separate bridges for OpenTSDB, InfluxDB, and Graphite, I decided to support all in one binary. For now, this is in the example documenation directory, but perhaps we will want to make a first-class project / repository out of it.	8 years ago
Julius Volz	b5163351bf	Simplify and fix remote write example After removing gRPC, this can be simplified again. Also, the configuration for the remote storage moved from flags to the config file.	8 years ago
Tom Wilkie	d83879210c	Switch back to protos over HTTP, instead of GRPC. My aim is to support the new grpc generic write path in Frankenstein. On the surface this seems easy - however I've hit a number of problems that make me think it might be better to not use grpc just yet. The explanation of the problems requires a little background. At weave, traffic to frankenstein need to go through a couple of services first, for SSL and to be authenticated. So traffic goes: internet -> frontend -> authfe -> frankenstein - The frontend is Nginx, and adds/removes SSL. Its done this way for legacy reasons, so the certs can be managed in one place, although eventually we imagine we'll merge it with authfe. All traffic from frontend is sent to authfe. - Authfe checks the auth tokens / cookie etc and then picks the service to forward the RPC to. - Frankenstein accepts the reads and does the right thing with them. First problem I hit was Nginx won't proxy http2 requests - it can accept them, but all calls downstream are http1 (see https://trac.nginx.org/nginx/ticket/923). This wasn't such a big deal, so it now looks like: internet --(grpc/http2)--> frontend --(grpc/http1)--> authfe --(grpc/http1)--> frankenstein Next problem was golang grpc server won't accept http1 requests (see https://groups.google.com/forum/#!topic/grpc-io/JnjCYGPMUms). It is possible to link a grpc server in with a normal go http mux, as long as the mux server is serving over SSL, as the golang http client & server won't do http2 over anything other than an SSL connection. This would require making all our service to service comms SSL. So I had a go a writing a grpc http1 server, and got pretty far. But is was a bit of a mess. So finally I thought I'd make a separate grpc frontend for this, running in parallel with the frontend/authfe combo on a different port - and first up I'd need a grpc reverse proxy. Ideally we'd have some nice, generic reverse proxy that only knew about a map from service names -> downstream service, and didn't need to decode & re-encode every request as it went through. It seems like this can't be done with golang's grpc library - see https://github.com/mwitkow/grpc-proxy/issues/1. And then I was surprised to find you can't do grpc from browsers! See http://www.grpc.io/faq/ - not important to us, but I'm starting to question why we decided to use grpc in the first place? It would seem we could have most of the benefits of grpc with protos over HTTP, and this wouldn't preclude moving to grpc when its a bit more mature? In fact, the grcp FAQ even admits as much: > Why is gRPC better than any binary blob over HTTP/2? > This is largely what gRPC is on the wire.	8 years ago
Julius Volz	aa3f2b7216	Generic write cleanups and changes. - fold metric name into labels - return initialization errors back to main - add snappy compression - better context handling - pre-allocation of labels - remove generic naming - other cleanups	8 years ago

1 2 3

106 Commits (1081e336a0c4004b6f817c70f0ff4f3810db80fa)