prometheus

Commit Graph

Author	SHA1	Message	Date
Harkishen Singh	cd412470d7	Consider status code 429 as recoverable errors to avoid resharding (#8237 ) * Consider status code 429 as recoverable errors to avoid resharding. * Adds support for Retry-After in backoff logic in remote storage. Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>	4 years ago
Nándor István Krácser	509000269a	remote_write: allow passing along custom HTTP headers (#8416 ) * remote_write: allow passing along custom HTTP headers Signed-off-by: Nandor Kracser <bonifaido@gmail.com> * add warning Signed-off-by: Nandor Kracser <bonifaido@gmail.com> * remote_write: add header valadtion Signed-off-by: Nandor Kracser <bonifaido@gmail.com> * extend tests for bad remote write headers Signed-off-by: Nandor Kracser <bonifaido@gmail.com> * remote_write: add note about the authorization header Signed-off-by: Nandor Kracser <bonifaido@gmail.com>	4 years ago
Julien Pivotto	4e5b1722b3	Move away from testutil, refactor imports (#8087 ) Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Siddhant Sinha	d9dcf2c90f	Fixes #7982 . Increase maxErrMsgLen value remote_write api (#8017 ) Signed-off-by: Siddhant Sinha <sid.sinha94@gmail.com>	4 years ago
Chris Marchbanks	f0f8e50567	Fix missing remote read spans (#7914 ) The remote read client needs to use the nethttp.Transport wrapper in order for spans to be instrumented properly. Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	4 years ago
Anand Sanmukhani	f4a1b3cef6	Rename `remote.newReadClient()` to `remote.NewReadClient()` (#7881 ) This should make the `NewReadClient()` exported outside the `remote` package Signed-off-by: Anand Sanmukhani <asanmukh@redhat.com>	4 years ago
Robert Fratto	2bd077ed97	expose UserAgent to make it changeable by importers (#7832 ) Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 years ago
Joe Elliott	624075eafe	Exposes remote storage http client to allow for customization (#7831 ) * Exposes remote write client to allow for customizations to the http client Signed-off-by: Joe Elliott <number101010@gmail.com> * Removed changes to vendored file Signed-off-by: Joe Elliott <number101010@gmail.com>	4 years ago
Andy Bursavich	4e6a94a27d	Invert service discovery dependencies (#7701 ) This also fixes a bug in query_log_file, which now is relative to the config file like all other paths. Signed-off-by: Andy Bursavich <abursavich@gmail.com>	4 years ago
johncming	2b75c1b199	storage/remote: rename httpclient name. (#7747 ) Signed-off-by: johncming <johncming@yahoo.com>	4 years ago
Joe Elliott	04b028f1e6	Exports recoverable error (#7689 ) Signed-off-by: Joe Elliott <number101010@gmail.com>	4 years ago
Bartlomiej Plotka	b788986717	storage: Adjusted fully storage layer support for chunk iterators: Remote read client, readyStorage, fanout. (#7059 ) * Fixed nits introduced by https://github.com/prometheus/prometheus/pull/7334 * Added ChunkQueryable implementation to fanout and readyStorage. * Added more comments. * Changed NewVerticalChunkSeriesMerger to CompactingChunkSeriesMerger, removed tiny interface by reusing VerticalSeriesMergeFunc for overlapping algorithm for both chunks and series, for both querying and compacting (!) + made sure duplicates are merged. * Added ErrChunkSeriesSet * Added Samples interface for seamless []promb.Sample to []tsdbutil.Sample conversion. * Deprecating non chunks serieset based StreamChunkedReadResponses, added chunk one. * Improved tests. * Split remote client into Write (old storage) and read. * Queryable client is now SampleAndChunkQueryable. Since we cannot use nice QueryableFunc I moved all config based options to sampleAndChunkQueryableClient to aboid boilerplate. In next commit: Changes for TSDB. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	4 years ago
Nicole J	d5a8f2afc4	Added the remote read histogram (#7334 ) change remote read queries total metric to a histogram and add read requests counter with status code Signed-off-by: njingco <jingco.nicole@gmail.com>	5 years ago
Cody Boggs	3268eac2dd	Trace Remote Write requests (#7206 ) * Trace Remote Write requests Signed-off-by: Cody Boggs <cboggs@splunk.com> * Refactor store attempts to keep code flow clearer, and avoid so many places to deal with span finishing Signed-off-by: Cody Boggs <cboggs@splunk.com>	5 years ago
Julien Pivotto	7ecd2d1c24	Jaeger: Create child span for remote read (#7187 ) * Jaeger: Create child span for remote read * Jaeger: use middleware to trace client http request Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Vasily Sliouniaev	0393b188c9	Add Jaeger (#7148 ) * Trace remote read Signed-off-by: vas <vasily.sliouniaev@jet.com> * Use jaeger Signed-off-by: vas <vasily.sliouniaev@jet.com>	5 years ago
Anand Singh Kunwar	aa61e392b2	Make remote client `Store` use passed context (#6673 ) * Remote store client's `Store` API currently doesn't use passed context, but instead just constructs a new `context.Background()` Signed-off-by: Anand Singh Kunwar <anandkunwar95@gmail.com>	5 years ago
Callum Styan	67838643ee	Add config option for remote job name (#6043 ) * Track remote write queues via a map so we don't care about index. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Support a job name for remote write/read so we can differentiate between them using the name. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Remote write/read has Name to not confuse the meaning of the field with scrape job names. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Split queue/client label into remote_name and url labels. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Don't allow for duplicate remote write/read configs. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Ensure we restart remote write queues if the hash of their config has not changed, but the remote name has changed. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Include name in remote read/write config hashes, simplify duplicates check, update test accordingly. Signed-off-by: Callum Styan <callumstyan@gmail.com>	5 years ago
陈谭军	103f26d188	fix the wrong word (#6069 ) Signed-off-by: chentanjun <2799194073@qq.com>	5 years ago
Stanislav Putrya	6141a8bd7c	Show warnings in UI if query have returned some warnings (#5964 ) * Show warnings in UI if query have returned some warnings + improve warning (error) text if query to remote was finished with error * Add prefixes for remote_read errors Signed-off-by: Stan Putrya <root.vagner@gmail.com>	5 years ago
Julius Volz	b5c833ca21	Update go.mod dependencies before release (#5883 ) * Update go.mod dependencies before release Signed-off-by: Julius Volz <julius.volz@gmail.com> * Add issue for showing query warnings in promtool Signed-off-by: Julius Volz <julius.volz@gmail.com> * Revert json-iterator back to 1.1.6 It produced errors when marshaling Point values with special float values. Signed-off-by: Julius Volz <julius.volz@gmail.com> * Fix expected step values in promtool tests after client_golang update Signed-off-by: Julius Volz <julius.volz@gmail.com> * Update generated protobuf code after proto dep updates Signed-off-by: Julius Volz <julius.volz@gmail.com>	5 years ago
Romain Baugue	95193fa027	Exhaust every request body before closing it (#5166 ) (#5479 ) From the documentation: > The default HTTP client's Transport may not > reuse HTTP/1.x "keep-alive" TCP connections if the Body is > not read to completion and closed. This effectively enable keep-alive for the fixed requests. Signed-off-by: Romain Baugue <romain.baugue@elwinar.com>	6 years ago
Tariq Ibrahim	8fdfa8abea	refine error handling in prometheus (#5388 ) i) Uses the more idiomatic Wrap and Wrapf methods for creating nested errors. ii) Fixes some incorrect usages of fmt.Errorf where the error messages don't have any formatting directives. iii) Does away with the use of fmt package for errors in favour of pkg/errors Signed-off-by: tariqibrahim <tariq181290@gmail.com>	6 years ago
Callum Styan	6f69e31398	Tail the TSDB WAL for remote_write This change switches the remote_write API to use the TSDB WAL. This should reduce memory usage and prevent sample loss when the remote end point is down. We use the new LiveReader from TSDB to tail WAL segments. Logic for finding the tracking segment is included in this PR. The WAL is tailed once for each remote_write endpoint specified. Reading from the segment is based on a ticker rather than relying on fsnotify write events, which were found to be complicated and unreliable in early prototypes. Enqueuing a sample for sending via remote_write can now block, to provide back pressure. Queues are still required to acheive parallelism and batching. We have updated the queue config based on new defaults for queue capacity and pending samples values - much smaller values are now possible. The remote_write resharding code has been updated to prevent deadlocks, and extra tests have been added for these cases. As part of this change, we attempt to guarantee that samples are not lost; however this initial version doesn't guarantee this across Prometheus restarts or non-retryable errors from the remote end (eg 400s). This changes also includes the following optimisations: - only marshal the proto request once, not once per retry - maintain a single copy of the labels for given series to reduce GC pressure Other minor tweaks: - only reshard if we've also successfully sent recently - add pending samples, latest sent timestamp, WAL events processed metrics Co-authored-by: Chris Marchbanks <csmarchbanks.com> (initial prototype) Co-authored-by: Tom Wilkie <tom.wilkie@gmail.com> (sharding changes) Signed-off-by: Callum Styan <callumstyan@gmail.com>	6 years ago
Daniele Sluijters	f25a6baedb	remote: Set User-Agent header in requests (#4891 ) Currently Prometheus requests show up with a UA of Go-http-client/1.1 which isn't super helpful. Though the X-Prometheus-Remote-* headers exist they need to be explicitly configured when logging the request in order to be able to deduce this is a request originating from Prometheus. By setting the header we remove this ambiguity and make default server logs just a bit more useful. This also updates a few other places to consistently capitalize the 'P' in the user agent, as well as ensure we set a UA to begin with. Signed-off-by: Daniele Sluijters <daenney@users.noreply.github.com>	6 years ago
Simon Pasquier	ed19373a78	: remove use of golang.org/x/net/context (#4869 ) : remove use of golang.org/x/net/context Signed-off-by: Simon Pasquier <spasquie@redhat.com> scrape: fix TestTargetScrapeScrapeCancel Signed-off-by: Simon Pasquier <spasquie@redhat.com>	6 years ago
Henri DF	ffb7836c14	Send "Accept-Encoding" header in read request (#4421 ) We should be doing this since we only accept Snappy-encoded responses. Signed-off-by: Henri DF <henridf@gmail.com>	6 years ago
Tom Wilkie	3353bbd018	Add proper unclean shutdown handling with a cancellable context. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Adam Shannon	809881d7f5	support reading basic_auth password_file for HTTP basic auth (#4077 ) Issue: https://github.com/prometheus/prometheus/issues/4076 Signed-off-by: Adam Shannon <adamkshannon@gmail.com>	7 years ago
Shubheksha Jalan	0471e64ad1	Use shared types from the `common` repo (#3674 ) * refactor: use shared types from common repo, remove util/config * vendor: add common/config * fix nit	7 years ago
Shubheksha Jalan	ec94df49d4	Refactor SD configuration to remove `config` dependency (#3629 ) * refactor: move targetGroup struct and CheckOverflow() to their own package * refactor: move auth and security related structs to a utility package, fix import error in utility package * refactor: Azure SD, remove SD struct from config * refactor: DNS SD, remove SD struct from config into dns package * refactor: ec2 SD, move SD struct from config into the ec2 package * refactor: file SD, move SD struct from config to file discovery package * refactor: gce, move SD struct from config to gce discovery package * refactor: move HTTPClientConfig and URL into util/config, fix import error in httputil * refactor: consul, move SD struct from config into consul discovery package * refactor: marathon, move SD struct from config into marathon discovery package * refactor: triton, move SD struct from config to triton discovery package, fix test * refactor: zookeeper, move SD structs from config to zookeeper discovery package * refactor: openstack, remove SD struct from config, move into openstack discovery package * refactor: kubernetes, move SD struct from config into kubernetes discovery package * refactor: notifier, use targetgroup package instead of config * refactor: tests for file, marathon, triton SD - use targetgroup package instead of config.TargetGroup * refactor: retrieval, use targetgroup package instead of config.TargetGroup * refactor: storage, use config util package * refactor: discovery manager, use targetgroup package instead of config.TargetGroup * refactor: use HTTPClient and TLS config from configUtil instead of config * refactor: tests, use targetgroup package instead of config.TargetGroup * refactor: fix tagetgroup.Group pointers that were removed by mistake * refactor: openstack, kubernetes: drop prefixes * refactor: remove import aliases forced due to vscode bug * refactor: move main SD struct out of config into discovery/config * refactor: rename configUtil to config_util * refactor: rename yamlUtil to yaml_config * refactor: kubernetes, remove prefixes * refactor: move the TargetGroup package to discovery/ * refactor: fix order of imports	7 years ago
Tobias Schmidt	434f0374f7	Refactor remote storage querier handling * Decouple remote client from ReadRecent feature. * Separate remote read filter into a small, testable function. * Use storage.Queryable interface to compose independent functionalities.	7 years ago
Krasi Georgiev	5d8f93a22a	now using only github.com/gogo/protobuf bumped all grpc-gateway packages to v1.2.2 updated and run the denproto.sh script	7 years ago
Tom Wilkie	9c3c98e8de	Revert "Port 'Don't disable HTTP keep-alives for remote storage connections.' to 2.0 (see #3173 )" This reverts commit `0997191b18`.	7 years ago
Tom Wilkie	ee011d906d	Port remote read server to 2.0.	7 years ago
Bryan Boreham	0997191b18	Port 'Don't disable HTTP keep-alives for remote storage connections.' to 2.0 (see #3173 ) Removes configurability introduced in #3160 in favour of hard-coding, per advice from @brian-brazil.	7 years ago
Tom Wilkie	3760f56c0c	remote: Expose ClientConfig type (see #3165 )	7 years ago
Tom Wilkie	16f71a7723	Port codec.go over form 1.8 branch.	7 years ago
Julius Volz	099df0c5f0	Migrate "golang.org/x/net/context" -> "context" (#3333 ) In some places, where ctxhttp or gRPC are concerned, we still need to use the old contexts.	7 years ago
Thibault Chataigner	bf4a279a91	Remote storage reads based on oldest timestamp in primary storage (#3129 ) Currently all read queries are simply pushed to remote read clients. This is fine, except for remote storage for wich it unefficient and make query slower even if remote read is unnecessary. So we need instead to compare the oldest timestamp in primary/local storage with the query range lower boundary. If the oldest timestamp is older than the mint parameter, then there is no need for remote read. This is an optionnal behavior per remote read client. Signed-off-by: Thibault Chataigner <t.chataigner@criteo.com>	7 years ago
Marc Sluiter	6a633eece1	Added go-conntrack for monitoring http connections (#3241 ) Added metrics for in- and outgoing traffic with go-conntrack.	7 years ago
Tom Wilkie	b568ace7ce	Move protos to ./prompb	7 years ago
Tom Wilkie	2dda5775e3	Initial port of remote storage to v2.	7 years ago
Fabian Reinartz	8ffc851147	Merge branch 'master' into dev-2.0	8 years ago
Julius Volz	3f23aa2cc7	Add headers to indicate remote read/write version Also add Content-Type header.	8 years ago
Julius Volz	9b33cfc457	Fix/unify context-based remote storage timeouts	8 years ago
Julius Volz	815762a4ad	Move retrieval.NewHTTPClient -> httputil.NewClientFromConfig	8 years ago
Julius Volz	eb14678a25	Make remote read/write use config.HTTPClientConfig	8 years ago
Julius Volz	02395a224d	[WIP] Remote Read	8 years ago
Julius Volz	e9476b35d5	Re-add multiple remote writers Each remote write endpoint gets its own set of relabeling rules. This is based on the (yet-to-be-merged) https://github.com/prometheus/prometheus/pull/2419, which removes legacy remote write implementations.	8 years ago

1 2

56 Commits (cdc71a1c2ac23d0ed2f7284c2a667fbfc1296414)