654 Commits (0a4d31477a75ff229ddfd95f610919c493a5a9c7)

Author	SHA1	Message	Date
beorn7	e7ac9c6863	Improvments based on review ... - Moved returns into the default section of switch statement that can only happen then. - Fix typo.	9 years ago
beorn7	199f309a39	Resurrect and rename invalid preload requests count metric. ... It is now also used in label matching, so the name of the metric changed from `prometheus_local_storage_invalid_preload_requests_total` to `non_existent_series_matches_total'.	9 years ago
beorn7	e8c1f30ab2	Merge the parallel logic of getSeriesForRange and metricForFingerprint	9 years ago
beorn7	9445c7053d	Add tests for range-limited label matching ... While doing so, improve getSeriesForRange.	9 years ago
beorn7	47e3c90f9b	Clean up error propagation ... Only return an error where callers are doing something with it except simply logging and ignoring. All the errors touched in this commit flag the storage as dirty anyway, and that fact is logged anyway. So most of what is being removed here is just log spam. As discussed earlier, the class of errors that flags the storage as dirty signals fundamental corruption, no even bubbling up a one-time warning to the user (e.g. about incomplete results) isn't helping much because _anything_ happening in the storage has to be doubted from that point on (and in fact retroactively into the past, too). Flagging the storage dirty, and alerting on it (plus marking the state in the web UI) is the only way I can see right now. As a byproduct, I cleaned up the setDirty method a bit and improved the logged errors.	9 years ago
beorn7	99854a84d7	Merge branch 'beorn7/storage6' into beorn7/storage7	9 years ago
beorn7	5e4fa96719	Merge branch 'beorn7/storage5' into beorn7/storage6	9 years ago
beorn7	b343e65907	Merge branch 'beorn7/storage4' into beorn7/storage5 ... erge is necessary,	9 years ago
beorn7	d0a4477446	Merge branch 'beorn7/storage3' into beorn7/storage4 ... Conflicts: storage/local/preload.go storage/local/storage.go storage/local/storage_test.go	9 years ago
beorn7	55eddab25f	Merge branch 'beorn7/storage2' into beorn7/storage3	9 years ago
beorn7	161eada3ad	Make chunkIterator even leaner.	9 years ago
beorn7	beb36df4bb	De-flag preloadChunksForRange ... Now there is preloadChunksForRange and preloadChunksForInstant in both, the series and the storage.	9 years ago
beorn7	836f1db04c	Improve MetricsForLabelMatchers ... WIP: This needs more tests. It now gets a from and through value, which it may opportunistically use to optimize the retrieval. With possible future range indices, this could be used in a very efficient way. This change merely applies some easy checks, which should nevertheless solve the use case of heavy rule evaluations on servers with a lot of series churn. Idea is the following: - Only archive series that are at least as old as the headChunkTimeout (which was already extremely unlikely to happen). - Then maintain a high watermark for the last archival, i.e. no archived series has a sample more recent than that watermark. - Any query that doesn't reach to a time before that watermark doesn't have to touch the archive index at all. (A production server at Soundcloud with the aforementioned series churn and heavy rule evaluations spends 50% of its CPU time in archive index lookups. Since rule evaluations usually only touch very recent values, most of those lookup should disappear with this change.) - Federation with a very broad label matcher will profit from this, too. As a byproduct, the un-needed MetricForFingerprint method was removed from the Storage interface.	9 years ago
beorn7	167b83695c	Merge branch 'beorn7/storage5' into beorn7/storage6	9 years ago
beorn7	01795382c9	Merge branch 'beorn7/storage4' into beorn7/storage5	9 years ago
beorn7	f7fc542db6	Merge branch 'master' into beorn7/storage4 ... Conflicts: storage/local/persistence.go	9 years ago
beorn7	3d86130d8c	Merge branch 'master' into beorn7/storage3	9 years ago
beorn7	1f30c8de8d	Merge branch 'master' into beorn7/storage2	9 years ago
beorn7	c13b1ecfe9	Make chunk iterators more DRY ... This finally extracts all the common code of the two chunk iterators into one. Any future chunk encodings with fast access by index can use the same iterator by simply providing an indexAccessor. Other future chunk encodings without fast index access (like Gorilla-style) can still implement the chunkIterator interface as usual.	9 years ago
beorn7	32f280a3cd	Slim down the chunkIterator interface ... For one, remove unneeded methods. Then, instead of using a channel for all values, use a bufio.Scanner-like interface. This removes the need for creating a goroutine and avoids the (unnecessary) locking performed by channel sending and receiving. This will make it much easier to write new chunk implementations (like Gorilla-style encoding).	9 years ago
beorn7	b6fdb355d7	Move dump-heads into its own tool	9 years ago
beorn7	f193f2b8ef	Add a command to promtool that dumps metadata of heads.db ... I needed this today for debugging. It can certainly be improved, but it's already quite helpful. I refactored the reading of heads.db files out of persistence, which is an improvement, too. I made minor changes to the cli package to allow outputting via the io.Writer interface.	9 years ago
beorn7	75a6b460ef	Give TestEvictAndLoadChunkDescs more time to actually evict ... Obviously, it's really bad to depend on timing here. The proper fix would be to have something like WaitForIndexing for other things to wait for, too. For now, let's see if the wait time increase fixes the issue.	9 years ago
beorn7	fc7de5374a	Quarantine series upon problem writing to the series file ... This fixes https://github.com/prometheus/prometheus/issues/1059 , but not in the obvious way (simply not updating the persist watermark, because that's actually not that simple - we don't really know what has gone wrong exactly). As any errors relevant here are most likely caused by severe and unrecoverable problems with the series file, Using the now quarantine feature is the right step. We don't really have to be worried about any inconsistent state of the series because it will be removed for good ASAP. Another plus is that we don't have to declare the whole storage dirty anymore.	9 years ago
beorn7	0ea5801e47	Handle errors caused by data corruption more gracefully ... This requires all the panic calls upon unexpected data to be converted into errors returned. This pollute the function signatures quite lot. Well, this is Go... The ideas behind this are the following: - panic only if it's a programming error. Data corruptions happen, and they are not programming errors. - If we detect a data corruption, we "quarantine" the series, essentially removing it from the database and putting its data into a separate directory for forensics. - Failure during writing to a series file is not considered corruption automatically. It will call setDirty, though, so that a crashrecovery upon the next restart will commence and check for that. - Series quarantining and setDirty calls are logged and counted in metrics, but are hidden from the user of the interfaces in interface.go, whith the notable exception of Append(). The reasoning is that we treat corruption by removing the corrupted series, i.e. a query for it will return no results on its next call anyway, so return no results right now. In the case of Append(), we want to tell the user that no data has been appended, though. Minor side effects: - Now consistently using filepath.* instead of path.*. - Introduced structured logging where I touched it. This makes things less consistent, but a complete change to structured logging would be out of scope for this PR.	9 years ago
beorn7	b6840997a7	Merge branch 'beorn7/storage2' into beorn7/storage3	9 years ago
beorn7	ce58fd357b	Merge branch 'beorn7/storage' into beorn7/storage2 ... Conflicts: storage/local/chunk.go storage/local/interface.go	9 years ago
beorn7	2581648f70	Separate iterators by offset ... Add test that exposes the problem.	9 years ago
beorn7	c740789ce3	Improve predict_linear ... Fixes https://github.com/prometheus/prometheus/issues/1401 This remove the last (and in fact bogus) use of BoundaryValues. Thus, a whole lot of unused (and arguably sub-optimal / ugly) code can be removed here, too.	9 years ago
beorn7	4b503ed9a5	Merge branch 'master' into beorn7/storage2	9 years ago
beorn7	059295332f	Merge remote-tracking branch 'origin/master' into beorn7/storage	9 years ago
beorn7	53005c3085	Merge branch 'beorn7/storage' into beorn7/storage2	9 years ago
beorn7	28e9bbc15f	Populate chunkDesc.chunkLastTime during checkpoint loading, too	9 years ago
Björn Rabenstein	a8c79f0a0c	Merge pull request #1422 from prometheus/release-0.17 ... Merge more commits from 0.17.	9 years ago
beorn7	8fa1560e48	Fix a very special case of handling the checkpoint timer	9 years ago
beorn7	41e44f6ab9	Merge branch 'master' into beorn7/storage2	9 years ago
Björn Rabenstein	d9eb624322	Merge pull request #1415 from prometheus/release-0.17 ... Forward-merge release-0.17 into master	9 years ago
beorn7	4d1f7b49b6	Fix a race condition in calculatePersistenceUrgencyScore	9 years ago
beorn7	454ecf3f52	Rework the way ranges and instants are handled ... In a way, our instants were also ranges, just with the staleness delta as range length. They are no treated equally, just that in one case, the range length is set as range, in the other the staleness delta. However, there are "real" instants where start and and time of a query is the same. In those cases, we only want to return a single value (the one closest before or at the equal start and end time). If that value is the last sample in the series, odds are we have it already in the series object. In that case, there is no need to pin or load any chunks. A special singleSampleSeriesIterator is created for that. This should greatly speed up instant queries as they happen frequently for rule evaluations.	9 years ago
beorn7	b876f8e6a5	Move lastSamplePair method up to memorySeries ... This implies a slight change of behavior as only samples added to the respective instance of a memorySeries are returned. However, this is most likely anyway what we want. Following cases: - Server has been restarted: Given the time it takes to cleanly shutdown and start up a server, the series are now stale anyway. An improved staleness handling (still to be implemented) will be based on tracking if a given target is continuing to expose samples for a given time series. In that case, we need a full scrape cycle to decide about staleness. So again, it makes sense to consider everything stale directly after a server restart. - Series unarchived due to a read request: The series is definitely stale so we don't want to return anything anyway. - Freshly created time series or series unarchived because of a sample append: That happens because appending a sample is imminent. Before the fingerprint lock is released, the series will have received a sample, and lastSamplePair will always returned the expected value.	9 years ago
beorn7	1e13f89039	Return SamplePair istead of *SamplePair consistently ... Formalize ZeroSamplePair as return value for non-existing samples. Change LastSamplePairForFingerprint to return a SamplePair (and not a pointer to it), which saves allocations in a potentially extremely frequent call.	9 years ago
beorn7	d290340367	Fix and improve chunkDesc locking	9 years ago
beorn7	0e202dacb4	Streamline series iterator creation ... This will fix issue #1035 and will also help to make issue #1264 less bad. The fundamental problem in the current code: In the preload phase, we quite accurately determine which chunks will be used for the query being executed. However, in the subsequent step of creating series iterators, the created iterators are referencing _all_ in-memory chunks in their series, even the un-pinned ones. In iterator creation, we copy a pointer to each in-memory chunk of a series into the iterator. While this creates a certain amount of allocation churn, the worst thing about it is that copying the chunk pointer out of the chunkDesc requires a mutex acquisition. (Remember that the iterator will also reference un-pinned chunks, so we need to acquire the mutex to protect against concurrent eviction.) The worst case happens if a series doesn't even contain any relevant samples for the query time range. We notice that during preloading but then we will still create a series iterator for it. But even for series that do contain relevant samples, the overhead is quite bad for instant queries that retrieve a single sample from each series, but still go through all the effort of series iterator creation. All of that is particularly bad if a series has many in-memory chunks. This commit addresses the problem from two sides: First, it merges preloading and iterator creation into one step, i.e. the preload call returns an iterator for exactly the preloaded chunks. Second, the required mutex acquisition in chunkDesc has been greatly reduced. That was enabled by a side effect of the first step, which is that the iterator is only referencing pinned chunks, so there is no risk of concurrent eviction anymore, and chunks can be accessed without mutex acquisition. To simplify the code changes for the above, the long-planned change of ValueAtTime to ValueAtOrBefore time was performed at the same time. (It should have been done first, but it kind of accidentally happened while I was in the middle of writing the series iterator changes. Sorry for that.) So far, we actively filtered the up to two values that were returned by ValueAtTime, i.e. we invested work to retrieve up to two values, and then we invested more work to throw one of them away. The SeriesIterator.BoundaryValues method can be removed once #1401 is fixed. But I really didn't want to load even more changes into this PR. Benchmarks: The BenchmarkFuzz.* benchmarks run 83% faster (i.e. about six times faster) and allocate 95% fewer bytes. The reason for that is that the benchmark reads one sample after another from the time series and creates a new series iterator for each sample read. To find out how much these improvements matter in practice, I have mirrored a beefy Prometheus server at SoundCloud that suffers from both issues #1035 and #1264. To reach steady state that would be comparable, the server needs to run for 15d. So far, it has run for 1d. The test server currently has only half as many memory time series and 60% of the memory chunks the main server has. The 90th percentile rule evaluation cycle time is ~11s on the main server and only ~3s on the test server. However, these numbers might get much closer over time. In addition to performance improvements, this commit removes about 150 LOC.	9 years ago
beorn7	ef3ab96111	Populate first and last time in the chunk descriptor earlier ... The First time is kind of trivial as we always know it when we create a new chunkDesc. The last time is only know when the chunk is closed, so we have to set it at that time. The change saves a lot of digging down into the chunk itself. Especially the last time is relative expensive as it involves the creation of an iterator. The first time access now doesn't require locking, which is also a nice gain.	9 years ago
beorn7	9a3edea477	Remove race condition from TestRetentionCutoff	9 years ago
Julius Volz	9b6d69610a	Fix various typos in comments. ... Helpfully reported by https://goreportcard.com/report/github.com/prometheus/prometheus :)	9 years ago
Fabian Reinartz	1f877f3d2a	Fix deadlock, structure target logging	9 years ago
Fabian Reinartz	59f1e722df	Return error on sample appending	9 years ago
beorn7	ec08c9a391	Rework the way to communicate backpressure (AKA suspended ingestion) ... This gives up on the idea to communicate throuh the Append() call (by either not returning as it is now or returning an error as suggested/explored elsewhere). Here I have added a Throttled() call, which has the advantage that it can be called before a whole _batch_ of Append()'s. Scrapes will happen completely or not at all. Same for rule group evaluations. That's a highly desired behavior (as discussed elsewhere). The code is even simpler now as the whole ingestion buffer could be removed. Logging of throttled mode has been streamlined and will create at most one message per minute.	9 years ago
beorn7	87ef24cd25	Add instrumentation and refactor things around "rushed mode"	9 years ago
beorn7	a2cd479058	Fix calculation of chunks to persist after restart ... Since we are not overestimating the number of chunks to persist anymore, this commit also adjusts the default value for -storage.local.memory-chunks. Update of documentation will follow.	9 years ago
beorn7	972d94433a	Introduce a hysteresis for "rushed mode" ... "Rushed mode" is formerly known as "degraded mode", which is changed with this commit, too. The name "degraded" was very misleading. Also, switch into rushed mode if we have too many chunks in memory and an at least reasonable amount of chunks to persist so that speeding up persisting chunks can help.	9 years ago
beorn7	14796bdb60	Improve chunkMaxBatchSize doc comment	9 years ago
beorn7	582af1618c	Streamline chunk writing ... This helps to avoid allocations in the same way we were already doing it during reading.	9 years ago
beorn7	99b9611351	Remove a race condition from TestRetentionCutoff	9 years ago
beorn7	3f4d22e4c7	Update doc comment ... This should have gone into a previous commit, but I forgot to save this particular file.	9 years ago
beorn7	add2ebdd56	Tolerate the lost+found directory in the data directory	9 years ago
Björn Rabenstein	6293f3a374	Merge pull request #1304 from prometheus/beorn7/storage ... Improve handling of series file truncation	9 years ago
beorn7	cb117d8346	Add a series ops metric "purge_on_request" ... It counts series deletions triggered via the API.	9 years ago
beorn7	4221c7de5c	Improve handling of series file truncation ... If only very few chunks are to be truncated from a very large series file, the rewrite of the file is a lorge overhead. With this change, a certain ratio of the file has to be dropped to make it happen. While only causing disk overhead at about the same ratio (by default 10%), it will cut down I/O by a lot in above scenario.	9 years ago
Corentin Chary	7b6c3e556c	Use '.' instead of '=' to separate labels from their values in Graphite ... Using .label=value. was weird to use in Graphite and didn't bring much value.	9 years ago
Julius Volz	75fdcf5698	Merge pull request #1197 from iksaif/master ... Add support for remote storage on Graphite	9 years ago
Corentin Chary	a2e4439086	Add support for remote storage on Graphite ... Allows to use graphite over tcp or udp. Metrics labels and values are used to construct a valid Graphite path in a way that will allow us to eventually read them back and reconstruct the metrics. For example, this metric: model.Metric{ model.MetricNameLabel: "test:metric", "testlabel": "test:value", "testlabel2": "test:value", ) Will become: test:metric.testlabel=test:value.testlabel2=test:value escape.go takes care of escaping values to match Graphite character set, it basically uses percent-encoding as a fallback wich will work pretty will in the graphite/grafana world. The remote storage module also has an optional 'prefix' parameter to prefix all metrics with a path (for example, 'prometheus.'). Graphite URLs are simply in the form tcp://host:port or udp://host:port.	9 years ago
Fabian Reinartz	33aab4169c	Anchor regexes in vector matching ... This commit makes the regex behavior of vector matching consistent with configuration and label_replace() by anchoring it. Fixes #1200	9 years ago
Fabian Reinartz	e3b6ec9784	Switch to common/log	9 years ago
Julius Volz	dac26cef71	Rename global "labels" config option to "external_labels".	9 years ago
Julius Volz	eeb1da36ac	Fix InfluxDB write support to work with InfluxDB 0.9.x. ... Because the InfluxDB client library currently pulls in multiple MBs of unnecessary dependencies, I have modified and cut up the vendored version to only pull in the few pieces that are actually needed. On InfluxDB's side, this dependency issue is tracked in: https://github.com/influxdb/influxdb/issues/3447 Hopefully, it will be resolved soon. If a password is needed for InfluxDB, it may be supplied via the INFLUXDB_PW environment variable.	9 years ago
Julius Volz	5f77fce578	Improve remote storage queue manager metrics.	9 years ago
beorn7	22d3a4311a	Increase waiting time in TestEvictAndLoadChunkDescs ... The test had become flaky with Go1.5. Theory here is that with Go1.5.x, sleeping for 10ms might not be enough to wake up another goroutine, possibly because it is used for GC. 50ms should always be enough due to GC pause guarantees with the new GC.	9 years ago
Julius Volz	af513468eb	Fix some dead code, missing error checks, shadowings. ... I applied https://medium.com/@jgautheron/quality-pipeline-for-go-projects-497e34d6567 and was greeted with a deluge of warnings, most of which were not applicable or really fixable realistically. These are some of the first ones I decided to fix.	9 years ago
beorn7	daeccdd0e9	Fix DropMetricsForFingerprints ... It now deletes the series file also for archived series. Also, fix a naming error in a doc comment.	9 years ago
Julius Volz	ffc5142c54	Merge pull request #1058 from prometheus/check-errors ... Fix error checking and logging around checkpointing.	9 years ago
Julius Volz	6774a73878	Fix error checking and logging around checkpointing.	9 years ago
Julius Volz	011faf9057	Fix typo in comment.	9 years ago
Fabian Reinartz	8fa719f778	Attach global labels to remote storage samples	9 years ago
Dieter Plaetinck	e1dacc56e6	fix comment. ... the sample doesn't get appended to the list of sampleappenders.	9 years ago
Julius Volz	744d5d5a7a	Merge pull request #1029 from prometheus/vet-fixes ... Fix "go vet" errors.	9 years ago
Julius Volz	995d3b831d	Fix most golint warnings. ... This is with `golint -min_confidence=0.5`. I left several lint warnings untouched because they were either incorrect or I felt it was better not to change them at the moment.	9 years ago
Julius Volz	963ad82dcb	Fix "go vet" errors. ... I ignored all errors of the type "composite literal uses unkeyed fields". Most of them are wrong because of https://github.com/golang/go/issues/9171.	9 years ago
Fabian Reinartz	d6b8da8d43	Switch promql types to common/model	9 years ago
Fabian Reinartz	e061595352	Move COWMetric into storage/metric package	9 years ago
Brian Brazil	fdf0d0642e	Cast value to float, as that's what the console templates expect.	9 years ago
Fabian Reinartz	1535ef1457	Replace metric.SamplePair with model.SamplePair	9 years ago
Fabian Reinartz	c9d396f476	Replace metric.LabelPair with model.LabelPair	9 years ago
Fabian Reinartz	438e232c9b	Fix grouping of import blocks	9 years ago
Fabian Reinartz	306e8468a0	Switch from client_golang/model to common/model	9 years ago
Julius Volz	f65ef1ed10	Fix wording in shutdown warning.	9 years ago
Brian Brazil	0ec71442cd	Storage: Tell users how to avoid crash recovery. ... If users see the crash recovery error, the chances are they aren't shutting down Prometheus correctly. Telling them how to do so will help them debug and fix the problem.	9 years ago
Laurie Malau	20ad403587	Don't warn/increment metric upon equal timestamps during append. ... Perhaps it would be even better to still warn in case the sample value has changed but the timestamps are equal, but we don't have efficient access to the last value.	9 years ago
Will Rouesnel	7810448dbe	Add proxy_url parameter to allow specifying per-job HTTP proxy servers ... Allow scrape_configs to have an optional proxy_url option which specifies a proxy to be used for all connections to hosts in that config. Internally this modifies the various client functions to take a *url.URL pointer which currently must point to an HTTP proxy (but has been left open-ended to allow the url format to be extended to support others, such as maybe SOCKS if needed).	9 years ago
Julius Volz	517badc21d	Only do regex lookups when there was no equality match. ... For the label matching index-based preselection phase, don't do an OR between equality and non-equality matchers. Execute only one of the two (with equality matchers preferred when present). Fixes https://github.com/prometheus/prometheus/issues/924	9 years ago
beorn7	699946bf32	Fix chunk desc loading. ... If all samples in consecutive chunks have the same timestamp, the way we used to load chunks will fail. With this change, the persist watermark is used to load the right amount of chunkDescs from disk. This bug is a possible reason for the rare storage corruption we have observed.	10 years ago
beorn7	4203849c92	Test chunkDesc eviction and loading	10 years ago
beorn7	37e12df9ff	Improve TestAppendOutOfOrder	10 years ago
beorn7	502aa9ded5	Use Has instead of Get for existence test.	10 years ago
beorn7	ff08f0b6fe	storage: ensure timestamp monotonicity within series. ... Fixes https://github.com/prometheus/prometheus/issues/481 While doing so, clean up and fix a few other things: - Fix `go vet` warnings (@fabxc to blame ;). - Fix a racey problem with unarchiving: Whenever we unarchive a series, we essentially want to do something with it. However, until we have done something with it, it appears like a series that is ready to be archived or even purged. So e.g. it would be ignored during checkpointing. With this fix, we always load the chunkDescs upon unarchiving. This is wasteful if we only want to add a new sample to an archived time series, but the (presumably more common) case where we access an archived time series in a query doesn't become more expensive. - The change above streamlined the getOrCreateSeries ond newMemorySeries flow. Also, the modTime is now always set correctly. - Fix the leveldb-backed implementation of KeyValueStore.Delete. It had the wrong behavior of still returning true, nil if a non-existing key has been passed in.	10 years ago
Julius Volz	acbc2b8cb6	storage: Fix float->uint conversions on some compilers. ... See https://github.com/prometheus/prometheus/issues/887, which will at least be partially fixed by this. From the spec https://golang.org/ref/spec#Conversions: "In all non-constant conversions involving floating-point or complex values, if the result type cannot represent the value the conversion succeeds but the result value is implementation-dependent." This ended up setting the converted values to 0 on Debian's Go 1.4.2 compiler, at least on 32-bit Debians.	10 years ago
Fabian Reinartz	3e811ad7a4	Merge pull request #827 from prometheus/fabxc/rmt-cleanup ... main: cleanup initialization of remote storage.	10 years ago
Fabian Reinartz	23e77450ff	main: cleanup initialization of remote storage.	10 years ago
beorn7	8c196c1028	Minor doc fixes.	10 years ago