156 Commits (375ad1185c128e947dbf75000024088c504a0b14)

Author	SHA1	Message	Date
Krasi Georgiev	5d8f93a22a	now using only github.com/gogo/protobuf ... bumped all grpc-gateway packages to v1.2.2 updated and run the denproto.sh script	7 years ago
Tom Wilkie	1af3ef431d	s/TestRemoveLabels/TestSeriesSetFilter/	7 years ago
Tom Wilkie	9c3c98e8de	Revert "Port 'Don't disable HTTP keep-alives for remote storage connections.' to 2.0 (see #3173)" ... This reverts commit 0997191b18.	7 years ago
Tom Wilkie	746752b946	Merge external labels in order.	7 years ago
Tom Wilkie	6e4d4ea402	Initialise some counters in remote storage API.	7 years ago
Tom Wilkie	2ae04d0e79	Add license header.	7 years ago
Tom Wilkie	e8c264e47a	Add comment.	7 years ago
Tom Wilkie	ee011d906d	Port remote read server to 2.0.	7 years ago
Bryan Boreham	0997191b18	Port 'Don't disable HTTP keep-alives for remote storage connections.' to 2.0 (see #3173) ... Removes configurability introduced in #3160 in favour of hard-coding, per advice from @brian-brazil.	7 years ago
Tom Wilkie	56820726fa	Move a couple of the encoding/decoding functions into codec.go	7 years ago
Conor Broderick	08b7328669	Port Metric name validation to 2.0 (see #2975)	7 years ago
Tom Wilkie	8fe0212ff7	Port 'Make queue manager configurable.' to 2.0, see #2991	7 years ago
Tom Wilkie	3760f56c0c	remote: Expose ClientConfig type (see #3165)	7 years ago
Tom Wilkie	16f71a7723	Port codec.go over form 1.8 branch.	7 years ago
Julius Volz	099df0c5f0	Migrate "golang.org/x/net/context" -> "context" (#3333) ... In some places, where ctxhttp or gRPC are concerned, we still need to use the old contexts.	7 years ago
Julius Volz	c3d6abc8e6	Fix some lint errors (#3334) ... I left the promql ones and some others untouched as I remember that @fabxc prefers them that way.	7 years ago
Julius Volz	2846d62573	Fix staticcheck issue in test (#3331) ... staticcheck fails with: storage/remote/read_test.go:199:27: do not pass a nil Context, even if a function permits it; pass context.TODO if you are unsure about which Context to use (SA1012)	7 years ago
Brian Brazil	4a50f547c8	removeLabels needs a pointer to work. (#3326)	7 years ago
Thibault Chataigner	bf4a279a91	Remote storage reads based on oldest timestamp in primary storage (#3129) ... Currently all read queries are simply pushed to remote read clients. This is fine, except for remote storage for wich it unefficient and make query slower even if remote read is unnecessary. So we need instead to compare the oldest timestamp in primary/local storage with the query range lower boundary. If the oldest timestamp is older than the mint parameter, then there is no need for remote read. This is an optionnal behavior per remote read client. Signed-off-by: Thibault Chataigner <t.chataigner@criteo.com>	7 years ago
Julius Volz	9ef8518b37	Remove "package remote" garbage from license headers (#3304)	7 years ago
Brian Brazil	73dc96e7f5	Fix leak of ticker in remote storage queue manager.	7 years ago
Brian Brazil	ee88f0d222	Ensure all values are used or _	7 years ago
Brian Brazil	37ec2d5283	Fix off by one error in concreteSeriesSet (#3262)	7 years ago
Marc Sluiter	6a633eece1	Added go-conntrack for monitoring http connections (#3241) ... Added metrics for in- and outgoing traffic with go-conntrack.	7 years ago
Julius Volz	f7e8348a88	Re-add contexts to storage.Storage.Querier() (#3230) ... * Re-add contexts to storage.Storage.Querier() These are needed when replacing the storage by a multi-tenant implementation where the tenant is stored in the context. The 1.x query interfaces already had contexts, but they got lost in 2.x. * Convert promql.Engine to use native contexts	7 years ago
Fabian Reinartz	d21f149745	*: migrate to go-kit/log	7 years ago
Fabian Reinartz	0efecea6d4	Adapt storage APIs to uint64 references	7 years ago
Tom Wilkie	b11bc8ae24	Fix some comments.	7 years ago
Tom Wilkie	ec999ff397	Prevent number of remote write shards from going negative. ... This can happen in the situation where the system scales up the number of shards massively (to deal with some backlog), then scales it down again as the number of samples sent during the time period is less than the number received.	7 years ago
Tom Wilkie	a09acdcc5b	Make concreteSeriersIterator behave.	7 years ago
Tom Wilkie	b568ace7ce	Move protos to ./prompb	7 years ago
Tom Wilkie	db8128ceeb	Add label set as first parameter to AddFast, ingored by TSDB adapter.	7 years ago
Tom Wilkie	2dda5775e3	Initial port of remote storage to v2.	7 years ago
Fabian Reinartz	8ffc851147	Merge branch 'master' into dev-2.0	8 years ago
Julius Volz	5a896033e3	Add remote read external label handling (#2555) ... * Add remote read external label handling This implements rule 1 and 2 from https://docs.google.com/document/d/188YauRgfF0J4CYMigLsVNN34V_kUwKnApBs2dQMfBbs/edit * Use more descriptive example labels in read test * Add comment for querier.addExternalLabels() * Make argument naming in removeLabels() more generic	8 years ago
Julius Volz	3f23aa2cc7	Add headers to indicate remote read/write version ... Also add Content-Type header.	8 years ago
Julius Volz	94acd3f1d8	Add fanin tests and fix uncovered bugs	8 years ago
Julius Volz	9b33cfc457	Fix/unify context-based remote storage timeouts	8 years ago
Julius Volz	815762a4ad	Move retrieval.NewHTTPClient -> httputil.NewClientFromConfig	8 years ago
Julius Volz	eb14678a25	Make remote read/write use config.HTTPClientConfig	8 years ago
Julius Volz	406b65d0dc	Rename remote.Storage to remote.Writer	8 years ago
Julius Volz	02395a224d	[WIP] Remote Read	8 years ago
Tom Wilkie	75bb0f3253	Review feedback	8 years ago
Tom Wilkie	77cce900b8	Fix tests	8 years ago
Tom Wilkie	b48799a01e	Add license stanza	8 years ago
Tom Wilkie	9d22f030cf	Dynamically reshard the QueueManager based on observed load.	8 years ago
Tom Wilkie	1ab893c6ec	Limit 'discarding sample' logs to 1 every 10s (#2446) ... * Limit 'discarding sample' logs to 1 every 10s * Include the vendored library * Review feedback	8 years ago
Julius Volz	2f39dbc8b3	Rename StorageQueueManager -> QueueManager	8 years ago
Julius Volz	e9476b35d5	Re-add multiple remote writers ... Each remote write endpoint gets its own set of relabeling rules. This is based on the (yet-to-be-merged) https://github.com/prometheus/prometheus/pull/2419, which removes legacy remote write implementations.	8 years ago
Julius Volz	beb3c4b389	Remove legacy remote storage implementations ... This removes legacy support for specific remote storage systems in favor of only offering the generic remote write protocol. An example bridge application that translates from the generic protocol to each of those legacy backends is still provided at: documentation/examples/remote_storage/remote_storage_bridge See also https://github.com/prometheus/prometheus/issues/10 The next step in the plan is to re-add support for multiple remote storages.	8 years ago
Brian Brazil	1b8a474612	Don't clone the metric if there's no remote writes. ... The metric clone can't be further optimised, and is a non-trivial memory allocation cost so fast path it if there's no remote writes configured.	8 years ago
Julius Volz	c7932aa009	Remove gRPC leftovers in protobuf definitions	8 years ago
Brian Brazil	77605649a9	Add support for remote write relabelling. ... Switch back to a single remote writer, as we were only ever meant to have one and the relabel semantics are clearer that way.	8 years ago
Matthew Campbell	67d76e3a5d	timeseries: store varbit encoded data into cassandra	8 years ago
Tom Wilkie	4520e12440	Add HTTP Basic Auth & TLS support to the generic write path. (#1957) ... * Add config, HTTP Basic Auth and TLS support to the generic write path. - Move generic write path configuration to the config file - Factor out config.TLSConfig -> tlf.Config translation - Support TLSConfig for generic remote storage - Rename Run to Start, and make it non-blocking. - Dedupe code in httputil for TLS config. - Make remote queue metrics global.	8 years ago
Tom Wilkie	d83879210c	Switch back to protos over HTTP, instead of GRPC. ... My aim is to support the new grpc generic write path in Frankenstein. On the surface this seems easy - however I've hit a number of problems that make me think it might be better to not use grpc just yet. The explanation of the problems requires a little background. At weave, traffic to frankenstein need to go through a couple of services first, for SSL and to be authenticated. So traffic goes: internet -> frontend -> authfe -> frankenstein - The frontend is Nginx, and adds/removes SSL. Its done this way for legacy reasons, so the certs can be managed in one place, although eventually we imagine we'll merge it with authfe. All traffic from frontend is sent to authfe. - Authfe checks the auth tokens / cookie etc and then picks the service to forward the RPC to. - Frankenstein accepts the reads and does the right thing with them. First problem I hit was Nginx won't proxy http2 requests - it can accept them, but all calls downstream are http1 (see https://trac.nginx.org/nginx/ticket/923). This wasn't such a big deal, so it now looks like: internet --(grpc/http2)--> frontend --(grpc/http1)--> authfe --(grpc/http1)--> frankenstein Next problem was golang grpc server won't accept http1 requests (see https://groups.google.com/forum/#!topic/grpc-io/JnjCYGPMUms). It is possible to link a grpc server in with a normal go http mux, as long as the mux server is serving over SSL, as the golang http client & server won't do http2 over anything other than an SSL connection. This would require making all our service to service comms SSL. So I had a go a writing a grpc http1 server, and got pretty far. But is was a bit of a mess. So finally I thought I'd make a separate grpc frontend for this, running in parallel with the frontend/authfe combo on a different port - and first up I'd need a grpc reverse proxy. Ideally we'd have some nice, generic reverse proxy that only knew about a map from service names -> downstream service, and didn't need to decode & re-encode every request as it went through. It seems like this can't be done with golang's grpc library - see https://github.com/mwitkow/grpc-proxy/issues/1. And then I was surprised to find you can't do grpc from browsers! See http://www.grpc.io/faq/ - not important to us, but I'm starting to question why we decided to use grpc in the first place? It would seem we could have most of the benefits of grpc with protos over HTTP, and this wouldn't preclude moving to grpc when its a bit more mature? In fact, the grcp FAQ even admits as much: > Why is gRPC better than any binary blob over HTTP/2? > This is largely what gRPC is on the wire.	8 years ago
Tobias Schmidt	29ced0090f	Fix common english misspellings	8 years ago
Tobias Schmidt	8f3b62bfe4	Simplify struct initialization	8 years ago
Fabian Reinartz	7bd7e63f97	storage: fix struct alignment issue in test ... The uint64 `numCalls` ends up being not word-aligned on certain architectures, which makes atomic reads/writes panic.	8 years ago
Tom Wilkie	d41d91388f	Update for new generic remote storage.	8 years ago
Tom Wilkie	a6931b71e8	Rationalise retrieval metrics so we have the state (success/failed) on both samples and batches, in a consistent fashion. ... Also, report total queue capacity of all queues, i.e. capacity * shards.	8 years ago
Tom Wilkie	ece12bff93	Shard/parrallelise samples by fingerprint in StorageQueueManager ... By splitting the single queue into multiple queues and flushing each individual queue in serially (and all queues in parallel), we can guarantee to preserve the order of timestampsin samples sent to downstream systems.	8 years ago
Julius Volz	aa3f2b7216	Generic write cleanups and changes. ... - fold metric name into labels - return initialization errors back to main - add snappy compression - better context handling - pre-allocation of labels - remove generic naming - other cleanups	8 years ago
Brian Brazil	36d2c4bd0b	Add generic write path using grpc. ... This uses a new proto format, with scope for multiple samples per timeseries in future. This will allow users to pump samples out to whatever they like without having to change the core Prometheus code. There's also an example receiver to save users figuring out the boilerplate themselves.	8 years ago
Dan Milstein	764ceaa939	Add timeout to test, cap waiting at 1 second	8 years ago
Dan Milstein	007907b410	Fix one of the tests for a remote storage QueueManager ... Specifically, the TestSpawnNotMoreThanMaxConcurrentSendsGoroutines was failing on a fresh checkout of master. The test had a race condition -- it would only pass if one of the spawned goroutines happened to very quickly pull a set of samples off an internal queue. This patch rewrites the test so that it deterministically waits until all samples have been pulled off that queue. In case of errors, it also now reports on the difference between what it expected and what it found. I verified that, if the code under test is deliberately broken, the test successfully reports on that.	8 years ago
Dmitry Vorobev	273e457da4	web: return status code and error message for config resource	8 years ago
beorn7	064b57858e	Consistently use the `Seconds()` method for conversion of durations ... This also fixes one remaining case of recording integral numbers of seconds only for a metric, i.e. this will probably fix #1796.	9 years ago
Dmitry Vorobev	bd2a770015	storage/remote: Spawn not more than "maxConcurrentSends" goroutines.	9 years ago
Dmitry Savintsev	7fdb62c253	fix several minor golint style issues	9 years ago
Jonathan Boulle	38098f8c95	Add missing license headers ... Prometheus is Apache 2 licensed, and most source files have the appropriate copyright license header, but some were missing it without apparent reason. Correct that by adding it.	9 years ago
Tobias Schmidt	e82ef154ee	Remove unused code leftovers	9 years ago
Julius Volz	d3b53bd7f0	Fix comment about Graphite mapping of dimensions.	9 years ago
Fabian Reinartz	59f1e722df	Return error on sample appending	9 years ago
beorn7	ec08c9a391	Rework the way to communicate backpressure (AKA suspended ingestion) ... This gives up on the idea to communicate throuh the Append() call (by either not returning as it is now or returning an error as suggested/explored elsewhere). Here I have added a Throttled() call, which has the advantage that it can be called before a whole _batch_ of Append()'s. Scrapes will happen completely or not at all. Same for rule group evaluations. That's a highly desired behavior (as discussed elsewhere). The code is even simpler now as the whole ingestion buffer could be removed. Logging of throttled mode has been streamlined and will create at most one message per minute.	9 years ago
Corentin Chary	7b6c3e556c	Use '.' instead of '=' to separate labels from their values in Graphite ... Using .label=value. was weird to use in Graphite and didn't bring much value.	9 years ago
Corentin Chary	a2e4439086	Add support for remote storage on Graphite ... Allows to use graphite over tcp or udp. Metrics labels and values are used to construct a valid Graphite path in a way that will allow us to eventually read them back and reconstruct the metrics. For example, this metric: model.Metric{ model.MetricNameLabel: "test:metric", "testlabel": "test:value", "testlabel2": "test:value", ) Will become: test:metric.testlabel=test:value.testlabel2=test:value escape.go takes care of escaping values to match Graphite character set, it basically uses percent-encoding as a fallback wich will work pretty will in the graphite/grafana world. The remote storage module also has an optional 'prefix' parameter to prefix all metrics with a path (for example, 'prometheus.'). Graphite URLs are simply in the form tcp://host:port or udp://host:port.	9 years ago
Fabian Reinartz	e3b6ec9784	Switch to common/log	9 years ago
Julius Volz	dac26cef71	Rename global "labels" config option to "external_labels".	9 years ago
Julius Volz	eeb1da36ac	Fix InfluxDB write support to work with InfluxDB 0.9.x. ... Because the InfluxDB client library currently pulls in multiple MBs of unnecessary dependencies, I have modified and cut up the vendored version to only pull in the few pieces that are actually needed. On InfluxDB's side, this dependency issue is tracked in: https://github.com/influxdb/influxdb/issues/3447 Hopefully, it will be resolved soon. If a password is needed for InfluxDB, it may be supplied via the INFLUXDB_PW environment variable.	9 years ago
Julius Volz	5f77fce578	Improve remote storage queue manager metrics.	9 years ago
Fabian Reinartz	8fa719f778	Attach global labels to remote storage samples	9 years ago
Julius Volz	963ad82dcb	Fix "go vet" errors. ... I ignored all errors of the type "composite literal uses unkeyed fields". Most of them are wrong because of https://github.com/golang/go/issues/9171.	9 years ago
Fabian Reinartz	438e232c9b	Fix grouping of import blocks	9 years ago
Fabian Reinartz	306e8468a0	Switch from client_golang/model to common/model	9 years ago
Will Rouesnel	7810448dbe	Add proxy_url parameter to allow specifying per-job HTTP proxy servers ... Allow scrape_configs to have an optional proxy_url option which specifies a proxy to be used for all connections to hosts in that config. Internally this modifies the various client functions to take a *url.URL pointer which currently must point to an HTTP proxy (but has been left open-ended to allow the url format to be extended to support others, such as maybe SOCKS if needed).	9 years ago
Fabian Reinartz	23e77450ff	main: cleanup initialization of remote storage.	10 years ago
Fabian Reinartz	b105e26f4d	storage: remove global flags	10 years ago
Fabian Reinartz	0de6edbdfc	Move pkg/ to util/	10 years ago
Fabian Reinartz	dfaf31a1da	Move web/httputils to pkg/httputil and add DeadlineClient to it	10 years ago
Julius Volz	267fd34156	Switch Prometheus to use github.com/prometheus/log. ... This change is conceptually very simple, although the diff is large. It switches logging from "github.com/golang/glog" to "github.com/prometheus/log", while not actually changing any log messages. V(1)-style logging has been changed to be log.Debug*().	10 years ago
Julius Volz	593e565688	Allow writing to InfluxDB/OpenTSDB at the same time.	10 years ago
Julius Volz	61fb688dd9	Add experimental InfluxDB write support.	10 years ago
beorn7	be11cb2b07	Remove the sample ingestion channel. ... The one central sample ingestion channel has caused a variety of trouble. This commit removes it. Targets and rule evaluation call an Append method directly now. To incorporate multiple storage backends (like OpenTSDB), storage.Tee forks the Append into two different appenders. Note that the tsdb queue manager had its own queue anyway. It was a queue after a queue... Much queue, so overhead... Targets have their own little buffer (implemented as a channel) to avoid stalling during an http scrape. But a new scrape will only be started once the old one is fully ingested. The contraption of three pipelined ingesters was removed. A Target is an ingester itself now. Despite more logic in Target, things should be less confusing now. Also, remove lint and vet warnings in ast.go.	10 years ago
beorn7	8a1c195b54	Move emptiness check to the receivers.	10 years ago
Bjoern Rabenstein	5859b74f1b	Clean up license issues. ... - Move CONTRIBUTORS.md to the more common AUTHORS. - Added the required NOTICE file. - Changed "Prometheus Team" to "The Prometheus Authors". - Reverted the erroneous changes to the Apache License.	10 years ago
Bjoern Rabenstein	ae70eac97d	Adjust the partitioning by outcome.	10 years ago
Bjoern Rabenstein	74c143c4c9	Improve scraper shutdown time. ... - Stop target pools in parallel. - Stop individual scrapers in goroutines, too. - Timing tweaks. Change-Id: I9dff1ee18616694f14b04408eaf1625d0f989696	10 years ago
Bjoern Rabenstein	443dd33805	Improve instrumentation in storage. ... Also, fix some other minor bugs. Change-Id: If72f1c058b0f47d3e378fdf80228d7e9a8db06c7	10 years ago
Bjoern Rabenstein	b3ed9aa7a2	Clean up start-up and shut-down. ... Change-Id: Idff4bbb0a15a9f879bfbb3da5b1025179cab5e2c	10 years ago