prometheus

Commit Graph

Author	SHA1	Message	Date
beorn7	d284ffab03	storage: Replace fpIter by sortedFPs The fpIter was kind of cumbersome to use and required a lock for each iteration (which wasn't even needed for the iteration at startup after loading the checkpoint). The new implementation here has an obvious penalty in memory, but it's only 8 byte per series, so 80MiB for a beefy server with 10M memory time series (which would probably need ~100GiB RAM, so the memory penalty is only 0.1% of the total memory need). The big advantage is that now series maintenance happens in order, which leads to the time between two maintenances of the same series being less random. Ideally, after each maintenance, the next maintenance would tackle the series with the largest number of non-persisted chunks. That would be quite an effort to find out or track, but with the approach here, the next maintenance will tackle the series whose previous maintenance is longest ago, which is a good approximation. While this commit won't change the _average_ number of chunks persisted per maintenance, it will reduce the mean time a given chunk has to wait for its persistence and thus reduce the steady-state number of chunks waiting for persistence. Also, the map iteration in Go is non-deterministic but not truly random. In practice, the iteration appears to be somewhat "bucketed". You can often observe a bunch of series with similar duration since their last maintenance, i.e. you see batches of series with similar number of chunks persisted per maintenance. If that batch is relatively young, a whole lot of series are maintained with very few chunks to persist. (See screenshot in PR for a better explanation.)	8 years ago
Tobias Schmidt	eac36d123e	Fix unstable fanin test (#2558 )	8 years ago
Conor Broderick	dafae52efa	Display total number of returned elements on console (#2532 ) Display total number of returned elements on console	8 years ago
Julius Volz	111841a230	Vendor new InfluxDB client library	8 years ago
Fabian Reinartz	e18be8d1a5	Merge pull request #2556 from prometheus/grobie/count-missed-group-executions Export number of missed rule evaluations	8 years ago
Julius Volz	3581057ea4	Update remote storage bridge README.md	8 years ago
Julius Volz	b391cbb808	Add InfluxDB read-back support to remote storage bridge	8 years ago
Tobias Schmidt	eaf33759fb	Register forgotten prometheus_evaluator_iterations_total metric	8 years ago
Tobias Schmidt	aaaba57184	Export number of missed rule evaluations In case the execution of all rules takes longer than the configured rule evaluation interval, one or more iterations will be skipped. This needs to be visible to the opterator.	8 years ago
Julius Volz	5a896033e3	Add remote read external label handling (#2555 ) * Add remote read external label handling This implements rule 1 and 2 from https://docs.google.com/document/d/188YauRgfF0J4CYMigLsVNN34V_kUwKnApBs2dQMfBbs/edit * Use more descriptive example labels in read test * Add comment for querier.addExternalLabels() * Make argument naming in removeLabels() more generic	8 years ago
Julius Volz	9cc7b393c5	Merge pull request #2548 from prometheus/sort-targets Sort targets by instance within a job	8 years ago
Julius Volz	589061919a	Merge pull request #2465 from Gouthamve/alert-metrics-2429 Better Metrics For Alerts	8 years ago
Goutham Veeramachaneni	f27ce34a13	Use Registerer to Register All Metrics * Made Metric a Gauge so that it can be registered.	8 years ago
Goutham Veeramachaneni	7ba0a9e81a	Add Comment About Initialising Counters	8 years ago
Goutham Veeramachaneni	0d0c9d5440	Move Registerer to Config Struct in Notifier	8 years ago
Fabian Reinartz	581f69d755	vendor: update tsdb and remove unused	8 years ago
Julius Volz	947c83be3b	Sort targets by instance within a job Fixes https://github.com/prometheus/prometheus/issues/2536	8 years ago
Julius Volz	336c7870ea	Merge pull request #2550 from prometheus/update-go-version ci: Update Go version to 1.8	8 years ago
Julius Volz	a44aadf4a1	ci: Update Go version to 1.8	8 years ago
Brian Brazil	8cd5aff8fe	Send instance="" with federation if instance not set. This is needed for federating non-instance level metrics, so they don't end up with the instance label of the prometheus target. Also sort external labels, so label output order is consistent.	8 years ago
Brian Brazil	d42e01b07c	Sort labelnames for federation. This makes unittests with multiple labels possible, and may be needed for performance with the new ingestion text parser.	8 years ago
Brian Brazil	dbb65846f1	Add unittest for federation external_labels behaviour	8 years ago
Goutham Veeramachaneni	5856f87be3	Update Issue Template (#2541 ) This is a comment in markdown and won't be shown while creating the issue.	8 years ago
Björn Rabenstein	29f05680a2	Merge pull request #2528 from prometheus/beorn7/storage2 main.go: Set GOGC to 40 by default	8 years ago
Björn Rabenstein	e63d079b59	Merge pull request #2527 from prometheus/beorn7/storage storage: Evict chunks and calculate persistence pressure...	8 years ago
Julius Volz	b5b0e00923	Merge pull request #2499 from prometheus/remote-read Remote Read	8 years ago
beorn7	434ab2a6a3	storage: Evict chunks and calculate persistence pressure based on target heap size This is a fairly easy attempt to dynamically evict chunks based on the heap size. A target heap size has to be set as a command line flage, so that users can essentially say "utilize 4GiB of RAM, and please don't OOM". The -storage.local.max-chunks-to-persist and -storage.local.memory-chunks flags are deprecated by this change. Backwards compatibility is provided by ignoring -storage.local.max-chunks-to-persist and use -storage.local.memory-chunks to set the new -storage.local.target-heap-size to a reasonable (and conservative) value (both with a warning). This also makes the metrics intstrumentation more consistent (in naming and implementation) and cleans up a few quirks in the tests. Answers to anticipated comments: There is a chance that Go 1.9 will allow programs better control over the Go memory management. I don't expect those changes to be in contradiction with the approach here, but I do expect them to complement them and allow them to be more precise and controlled. In any case, once those Go changes are available, this code has to be revisted. One might be tempted to let the user specify an estimated value for the RSS usage, and then internall set a target heap size of a certain fraction of that. (In my experience, 2/3 is a fairly safe bet.) However, investigations have shown that RSS size and its relation to the heap size is really really complicated. It depends on so many factors that I wouldn't even start listing them in a commit description. It depends on many circumstances and not at least on the risk trade-off of each individual user between RAM utilization and probability of OOMing during a RAM usage peak. To not add even more to the confusion, we need to stick to the well-defined number we also use in the targeting here, the sum of the sizes of heap objects.	8 years ago
Björn Rabenstein	e1a84b6256	Merge pull request #2529 from prometheus/beorn7/storage3 storage: Use staleness delta as head chunk timeout	8 years ago
beorn7	96a303b348	storage: Use staleness delta as head chunk timeout Currently, if a series stops to exist, its head chunk will be kept open for an hour. That prevents it from being persisted. Which prevents it from being evicted. Which prevents the series from being archived. Most of the time, once no sample has been added to a series within the staleness limit, we can be pretty confident that this series will not receive samples anymore. The whole chain as described above can be started after 5m instead of 1h. In the relaxed case, this doesn't change a lot as the head chunk timeout is only checked during series maintenance, and usually, a series is only maintained every six hours. However, there is the typical scenario where a large service is deployed, the deoply turns out to be bad, and then it is deployed again within minutes, and quite quickly the number of time series has tripled. That's the point where the Prometheus server is stressed and switches (rightfully) into rushed mode. In that mode, time series are processed as quickly as possible, but all of that is in vein if all of those recently ended time series cannot be persisted yet for another hour. In that scenario, this change will help most, and it's exactly the scenario where help is most desperately needed.	8 years ago
beorn7	04ccf84559	main.go: Set GOGC to 40 by default Rationale: The default value for GOGC is 100, i.e. a garbage collected is initialized once as many heap space has been allocated as was in use after the last GC was done. This ratio doesn't make a lot of sense in Prometheus, as typically about 60% of the heap is allocated for long-lived memory chunks (most of which are around for many hours if not days). Thus, short-lived heap objects are accumulated for quite some time until they finally match the large amount of memory used by bulk memory chunks and a gigantic GC cyle is invoked. With GOGC=40, we are essentially reinstating "normal" GC behavior by acknowledging that about 60% of the heap are used for long-term bulk storage. The median Prometheus production server at SoundCloud runs a GC cycle every 90 seconds. With GOGC=40, a GC cycle is run every 35 seconds (which is still not very often). However, the effective RAM usage is now reduced by about 30%. If settings are updated to utilize more RAM, the time between GC cycles goes up again (as the heap size is larger with more long-lived memory chunks, but the frequency of creating short-lived heap objects does not change). On a quite busy large Prometheus server, the timing changed from one GC run every 20s to one GC run every 12s. In the former case (just changing GOGC, leave everything else as it is), the CPU usage increases by about 10% (on a mid-size referenc server from 8.1 to 8.9). If settings are adjusted, the CPU consumptions increases more drastically (from 8 cores to 13 cores on a large reference server), despite GCs happening more rarely, presumably because a 50% larger set of memory chunks is managed now. Having more memory chunks is good in many regards, and most servers are running out of memory long before they run out of CPU cycles, so the tradeoff is overwhelmingly positive in most cases. Power users can still set the GOGC environment variable as usual, as the implementation in this commit honors an explicitly set variable.	8 years ago
Julius Volz	3f23aa2cc7	Add headers to indicate remote read/write version Also add Content-Type header.	8 years ago
Fabian Reinartz	a2e7b0b934	venodr: update tsdb and go-kit/log	8 years ago
Tobias Schmidt	6dbd779099	Merge pull request #2519 from prometheus/update-arch-diag-link Update architecture diagram link	8 years ago
Julius Volz	a20105ddb0	Update architecture diagram link	8 years ago
Julius Volz	c34257d069	Merge pull request #2518 from prometheus/update-arch-diag Remove PromDash from architecture diagram	8 years ago
Julius Volz	428e1ad42c	Remove PromDash from architecture diagram	8 years ago
Björn Rabenstein	ddcf04a768	Merge pull request #2515 from leitzler/leitzler-patch-1 Use go env to fetch GOPATH to support Go 1.8	8 years ago
Pontus Leitzler	4774d6736a	Use go env to fetch GOPATH to support Go 1.8 Go 1.8 do not require env GOPATH to be set and make will fail if it isn't set.	8 years ago
Julius Volz	8fda83ea12	Make rules only read local data	8 years ago
Julius Volz	94acd3f1d8	Add fanin tests and fix uncovered bugs	8 years ago
Julius Volz	9b33cfc457	Fix/unify context-based remote storage timeouts	8 years ago
Julius Volz	815762a4ad	Move retrieval.NewHTTPClient -> httputil.NewClientFromConfig	8 years ago
Fabian Reinartz	397f001ac5	Merge branch 'master' into dev-2.0	8 years ago
Fabian Reinartz	fc2e56c13f	vendor: update tsdb	8 years ago
Julius Volz	eb14678a25	Make remote read/write use config.HTTPClientConfig	8 years ago
Julius Volz	406b65d0dc	Rename remote.Storage to remote.Writer	8 years ago
Julius Volz	02395a224d	[WIP] Remote Read	8 years ago
Julius Volz	40e41a4776	Merge pull request #2494 from tomwilkie/remote-write-sharding Dynamically reshard the QueueManager based on observed load.	8 years ago
Julius Volz	525da88c35	Merge pull request #2479 from YKlausz/consul-tls Adding consul capability to connect via tls	8 years ago
Fabian Reinartz	0958c83d5d	Merge pull request #2511 from prometheus/fix-go-build Only truncate buildVersion if it's set	8 years ago

... 4 5 6 7 8 ...

4079 Commits (34ab7a885a6d4ec877a0f74281fcf413660d360c) All Branches Search

4079 Commits (34ab7a885a6d4ec877a0f74281fcf413660d360c)

All Branches