819 Commits (5ee07b3b3a59b95cbdd007a66a6ca30df9002c86)

Author	SHA1	Message	Date
Brian Brazil	ee88f0d222	Ensure all values are used or _	7 years ago
Brian Brazil	37ec2d5283	Fix off by one error in concreteSeriesSet (#3262)	7 years ago
Marc Sluiter	6a633eece1	Added go-conntrack for monitoring http connections (#3241) ... Added metrics for in- and outgoing traffic with go-conntrack.	7 years ago
Julius Volz	f7e8348a88	Re-add contexts to storage.Storage.Querier() (#3230) ... * Re-add contexts to storage.Storage.Querier() These are needed when replacing the storage by a multi-tenant implementation where the tenant is stored in the context. The 1.x query interfaces already had contexts, but they got lost in 2.x. * Convert promql.Engine to use native contexts	7 years ago
Fabian Reinartz	7b02bfee0a	web: start web handler while TSDB is starting up	7 years ago
Fabian Reinartz	d21f149745	*: migrate to go-kit/log	7 years ago
Fabian Reinartz	0efecea6d4	Adapt storage APIs to uint64 references	7 years ago
Fabian Reinartz	0c81d5f719	storage: instantiate correct block ranges	7 years ago
Fabian Reinartz	2037778d14	vendor: update TSDB	7 years ago
Tom Wilkie	b11bc8ae24	Fix some comments.	7 years ago
Tom Wilkie	ec999ff397	Prevent number of remote write shards from going negative. ... This can happen in the situation where the system scales up the number of shards massively (to deal with some backlog), then scales it down again as the number of samples sent during the time period is less than the number received.	7 years ago
Tom Wilkie	a09acdcc5b	Make concreteSeriersIterator behave.	7 years ago
Tom Wilkie	994a7f27d6	Propagate errors through mergeSeriesSet correctly.	7 years ago
Tom Wilkie	2e0d8487e3	Return zeros if At() is called after Next() returns false.	7 years ago
Tom Wilkie	014bd31a86	Remove unnecessary whitespace changes, add comment.	7 years ago
Tom Wilkie	98ac07f86a	Add unit test for the merging on the read path.	7 years ago
Tom Wilkie	b568ace7ce	Move protos to ./prompb	7 years ago
Tom Wilkie	96e25adc8d	Introduce 'primary' storage in fanout, and have Add return the ref from the primary. ... Also, ensure all append batches are rolled back when a commit or rollback fails.	7 years ago
Tom Wilkie	db8128ceeb	Add label set as first parameter to AddFast, ingored by TSDB adapter.	7 years ago
Tom Wilkie	2dda5775e3	Initial port of remote storage to v2.	7 years ago
Fabian Reinartz	16464c3a33	Merge pull request #2910 from prometheus/adminapi ... Admin API	7 years ago
Fabian Reinartz	ccf9e62972	*: add admin grpc API	7 years ago
Goutham Veeramachaneni	243419c007	Return tsdb.ErrOutOfBounds as storage.ErrOutOfBounds ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	3069bd3996	Handle scrapes with OutOfBounds metrics better ... fixes #2894 Signed-off-by: Goutham Veeramachaneni <goutham@boomerangcommerce.com>	8 years ago
Goutham Veeramachaneni	d407bd150c	Consolidate the duration params in CLI ... * All CLI params moved to model.Duration Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	baf5b0f0fc	Fix error where we look into the future. (#2829) ... * Fix error where we look into the future. So currently we are adding values that are in the future for an older timestamp. For example, if we have [(1, 1), (150, 2)] we will end up showing [(1, 1), (2,2)]. Further it is not advisable to call .At() after Next() returns false. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Retuen early if done Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Handle Seek() where we reach the end of iterator Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Simplify code Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Brian Brazil	c02c25d5ba	Allow peeking back further in buffer.	8 years ago
Fabian Reinartz	d289dc55c3	storage: update TSDB	8 years ago
Fabian Reinartz	9b175d48cb	Add flag to disable TSDB lock file	8 years ago
Fabian Reinartz	0f3110487d	Merge remote-tracking branch 'origin/dev-2.0' into dev-2.0	8 years ago
Fabian Reinartz	37deb21c45	vendor: remove unused dependency and last ref to fabxc/tsdb	8 years ago
Brian Brazil	5c9a6ce747	Add license to files. ... This should fix CI for dev-2.0.	8 years ago
Fabian Reinartz	8ffc851147	Merge branch 'master' into dev-2.0	8 years ago
Fabian Reinartz	cfb2a7f1d5	vendor: sync organisation migration of tsdb	8 years ago
Fabian Reinartz	bbcf20ba01	web: deduplicate series in federation	8 years ago
Fabian Reinartz	4e41987bcb	storage: add deduplication function ... This adds a function to deduplicate two series sets given that duplicate series have equivalent data points.	8 years ago
Björn Rabenstein	50e4f49b7e	Merge pull request #2561 from prometheus/beorn7/storage2 ... storage: Evict unused chunk.Descs in crash recovery	8 years ago
beorn7	08fc6cbd39	storage: Evict unused chunk.Descs in crash recovery ... This is in line with the v1.5 change in paradigm to not keep chunk.Descs without chunks around after a series maintenance. It's mainly motivated by avoiding excessive amounts of RAM usage during crash recovery. The code avoids to create memory time series with zero chunk.Descs as that is prone to trigger weird effects. (Series maintenance would archive series with zero chunk.Descs, but we cannot do that here because the archive indices still have to be checked.)	8 years ago
Björn Rabenstein	1c6240fc40	Merge pull request #2559 from prometheus/beorn7/storage ... storage: Replace fpIter by sortedFPs	8 years ago
beorn7	d284ffab03	storage: Replace fpIter by sortedFPs ... The fpIter was kind of cumbersome to use and required a lock for each iteration (which wasn't even needed for the iteration at startup after loading the checkpoint). The new implementation here has an obvious penalty in memory, but it's only 8 byte per series, so 80MiB for a beefy server with 10M memory time series (which would probably need ~100GiB RAM, so the memory penalty is only 0.1% of the total memory need). The big advantage is that now series maintenance happens in order, which leads to the time between two maintenances of the same series being less random. Ideally, after each maintenance, the next maintenance would tackle the series with the largest number of non-persisted chunks. That would be quite an effort to find out or track, but with the approach here, the next maintenance will tackle the series whose previous maintenance is longest ago, which is a good approximation. While this commit won't change the _average_ number of chunks persisted per maintenance, it will reduce the mean time a given chunk has to wait for its persistence and thus reduce the steady-state number of chunks waiting for persistence. Also, the map iteration in Go is non-deterministic but not truly random. In practice, the iteration appears to be somewhat "bucketed". You can often observe a bunch of series with similar duration since their last maintenance, i.e. you see batches of series with similar number of chunks persisted per maintenance. If that batch is relatively young, a whole lot of series are maintained with very few chunks to persist. (See screenshot in PR for a better explanation.)	8 years ago
Tobias Schmidt	eac36d123e	Fix unstable fanin test (#2558)	8 years ago
Julius Volz	5a896033e3	Add remote read external label handling (#2555) ... * Add remote read external label handling This implements rule 1 and 2 from https://docs.google.com/document/d/188YauRgfF0J4CYMigLsVNN34V_kUwKnApBs2dQMfBbs/edit * Use more descriptive example labels in read test * Add comment for querier.addExternalLabels() * Make argument naming in removeLabels() more generic	8 years ago
Björn Rabenstein	e63d079b59	Merge pull request #2527 from prometheus/beorn7/storage ... storage: Evict chunks and calculate persistence pressure...	8 years ago
Julius Volz	b5b0e00923	Merge pull request #2499 from prometheus/remote-read ... Remote Read	8 years ago
beorn7	434ab2a6a3	storage: Evict chunks and calculate persistence pressure based on target heap size ... This is a fairly easy attempt to dynamically evict chunks based on the heap size. A target heap size has to be set as a command line flage, so that users can essentially say "utilize 4GiB of RAM, and please don't OOM". The -storage.local.max-chunks-to-persist and -storage.local.memory-chunks flags are deprecated by this change. Backwards compatibility is provided by ignoring -storage.local.max-chunks-to-persist and use -storage.local.memory-chunks to set the new -storage.local.target-heap-size to a reasonable (and conservative) value (both with a warning). This also makes the metrics intstrumentation more consistent (in naming and implementation) and cleans up a few quirks in the tests. Answers to anticipated comments: There is a chance that Go 1.9 will allow programs better control over the Go memory management. I don't expect those changes to be in contradiction with the approach here, but I do expect them to complement them and allow them to be more precise and controlled. In any case, once those Go changes are available, this code has to be revisted. One might be tempted to let the user specify an estimated value for the RSS usage, and then internall set a target heap size of a certain fraction of that. (In my experience, 2/3 is a fairly safe bet.) However, investigations have shown that RSS size and its relation to the heap size is really really complicated. It depends on so many factors that I wouldn't even start listing them in a commit description. It depends on many circumstances and not at least on the risk trade-off of each individual user between RAM utilization and probability of OOMing during a RAM usage peak. To not add even more to the confusion, we need to stick to the well-defined number we also use in the targeting here, the sum of the sizes of heap objects.	8 years ago
beorn7	96a303b348	storage: Use staleness delta as head chunk timeout ... Currently, if a series stops to exist, its head chunk will be kept open for an hour. That prevents it from being persisted. Which prevents it from being evicted. Which prevents the series from being archived. Most of the time, once no sample has been added to a series within the staleness limit, we can be pretty confident that this series will not receive samples anymore. The whole chain as described above can be started after 5m instead of 1h. In the relaxed case, this doesn't change a lot as the head chunk timeout is only checked during series maintenance, and usually, a series is only maintained every six hours. However, there is the typical scenario where a large service is deployed, the deoply turns out to be bad, and then it is deployed again within minutes, and quite quickly the number of time series has tripled. That's the point where the Prometheus server is stressed and switches (rightfully) into rushed mode. In that mode, time series are processed as quickly as possible, but all of that is in vein if all of those recently ended time series cannot be persisted yet for another hour. In that scenario, this change will help most, and it's exactly the scenario where help is most desperately needed.	8 years ago
Julius Volz	3f23aa2cc7	Add headers to indicate remote read/write version ... Also add Content-Type header.	8 years ago
Julius Volz	8fda83ea12	Make rules only read local data	8 years ago
Julius Volz	94acd3f1d8	Add fanin tests and fix uncovered bugs	8 years ago
Julius Volz	9b33cfc457	Fix/unify context-based remote storage timeouts	8 years ago