127 Commits (b6f89a198284a222f23bd58b59fa93b71ed44bac)

Author	SHA1	Message	Date
beorn7	f338d791d2	storage: Several optimizations of checkpointing ... - checkpointSeriesMapAndHeads accepts a context now to allow cancelling. - If a shutdown is initiated, cancel the ongoing checkpoint. (We will create a final checkpoint anyway.) - Always wait for at least as long as the last checkpoint took before starting the next checkpoint (to cap the time spending checkpointing at 50%). - If an error has occurred during checkpointing, don't bother to sync the write. - Make sure the temporary checkpoint file is deleted, even if an error has occurred. - Clean up the checkpoint loop a bit. (The concurrent Timer.Reset(0) call might have cause a race.)	8 years ago
beorn7	4fcc73a04c	storage: Recover from corrupted indices for archived series ... An unopenable archived_fingerprint_to_timerange is simply deleted and will be rebuilt during crash recovery (wich can then take quite some time). An unopenable archived_fingerprint_to_metric is not deleted but instructions to the user are logged. A deletion has to be done by the user explicitly as it means losing all archived series (and a repair with a 3rd party tool might still be possible).	8 years ago
beorn7	f3365c4f26	storage: Mark storage as dirty if indexing fails	8 years ago
beorn7	ae286385fd	storage: Check for negative values from varint decoding ... Sadly, we have a number of places where we use varint encoding for numbers that cannot be negative. We could have saved a bit by using uvarint encoding. On the bright side, we now have a 50% chance to detect data corruption. :-/ Fixes #1800 and #2492.	8 years ago
beorn7	d771185a43	storage: Fix chunkIndexToStartSeek calculation ... With a high enough shrink ratio and enough chunks to persist, the cutoff point could be _outside_ of the file, which wreaks havoc in the storage.	8 years ago
beorn7	73bd5e4dff	Merge branch 'beorn7/storage' into beorn7/storage3	8 years ago
beorn7	46a0837816	storage: Fix offset returned by dropAndPersistChunks ... This is another corner-case that was previously never exercised because the rewriting of a series file was never prevented by the shrink ratio. Scenario: There is an existing series on disk, which is archived. If a new sample comes in for that file, a new chunk in memory is created, and the chunkDescsOffset is set to -1. If series maintenance happens before the series has at least one chunk to persist _and_ an insufficient chunks on disk is old enough for purging (so that the shrink ratio kicks in), dropAndPersistChunks would return 0, but it should return the chunk length of the series file.	8 years ago
Mitsuhiro Tanda	be8b1eb656	storage: optimize dropping chunks by using minShrinkRatio (#2397) ... storage: prevent unnecessary chunk header reading if minShrinkRatio > 0	8 years ago
beorn7	2363a90adc	storage: Do not throw away fully persisted memory series in checkpointing	8 years ago
beorn7	75282b27ba	storage: Added checks for invariants	8 years ago
beorn7	65dc8f44d3	storage: Test for errors returned by MaybePopulateLastTime	8 years ago
beorn7	4ccfc93dcf	storage: Set shrink ratio in the constructor.	8 years ago
Brian Brazil	c1b547a90e	Only checkpoint chunkdescs and series that need persisting. (#2340) ... This decreases checkpoint size by not checkpointing things that don't actually need checkpointing. This is fully compatible with the v2 checkpoint format, as it makes series appear as though the only chunksdescs in memory are those that need persisting.	8 years ago
Brian Brazil	1dcb7637f5	Add various persistence related metrics (#2333) ... Add metrics around checkpointing and persistence * Add a metric to say if checkpointing is happening, and another to track total checkpoint time and count. This breaks the existing prometheus_local_storage_checkpoint_duration_seconds by renaming it to prometheus_local_storage_checkpoint_last_duration_seconds as the former name is more appropriate for a summary. * Add metric for last checkpoint size. * Add metric for series/chunks processed by checkpoints. For long checkpoints it'd be useful to see how they're progressing. * Add metric for dirty series * Add metric for number of chunks persisted per series. You can get the number of chunks from chunk_ops, but not the matching number of series. This helps determine the size of the writes being made. * Add metric for chunks queued for persistence Chunks created includes both chunks that'll need persistence and chunks read in for queries. This only includes chunks created for persistence. * Code review comments on new persistence metrics.	8 years ago
Brian Brazil	f9e581907a	Make index queue bigger. (#2322) ... When a large Prometheus starts up fresh it can take many minutes to warmup and clear out the index queue. A larger queue means less blocking, bigger batches and cuts down startup time by ~50%.	8 years ago
Tristan Colgate	30be8e0b8a	ignore dotfiles in data directory	8 years ago
beorn7	837c029b16	storage: Fix linter issue ... Go style tries to avoid indented `else` blocks.	8 years ago
tattsun	e714079cf2	storage: fix error message (#2270) ... * storage: add error message	8 years ago
Christopher M. Luciano	148b006e25	Clarify error message when Prometheus data dir finds unexpected files	8 years ago
Julius Volz	044ebce779	Review fixups.	8 years ago
Julius Volz	ab80ced756	storage: separate chunk package, publish more names ... This is a followup to https://github.com/prometheus/prometheus/pull/2011. This publishes more of the methods and other names of the chunk code and moves the chunk code to its own package. There's some unavoidable ugliness: the chunk and chunkDesc metrics are used by both packages, so I had to move them to the chunk package. That isn't great, but I don't see how to do it better without a larger redesign of everything. Same for the evict requests and some other types.	8 years ago
Matthew Campbell	67d76e3a5d	timeseries: store varbit encoded data into cassandra	8 years ago
Matt Bostock	e618af5d0b	Storage: Add crash recovery metric 'started_dirty' ... ...to indicate when crash recovery was invoked during Prometheus startup. Fixes #1918.	8 years ago
Julius Volz	3bfec97d46	Make the storage interface higher-level. ... See discussion in https://groups.google.com/forum/#!topic/prometheus-developers/bkuGbVlvQ9g The main idea is that the user of a storage shouldn't have to deal with fingerprints anymore, and should not need to do an individual preload call for each metric. The storage interface needs to be made more high-level to not expose these details. This also makes it easier to reuse the same storage interface for remote storages later, as fewer roundtrips are required and the fingerprint concept doesn't work well across the network. NOTE: this deliberately gets rid of a small optimization in the old query Analyzer, where we dedupe instants and ranges for the same series. This should have a minor impact, as most queries do not have multiple selectors loading the same series (and at the same offset).	8 years ago
beorn7	064b57858e	Consistently use the `Seconds()` method for conversion of durations ... This also fixes one remaining case of recording integral numbers of seconds only for a metric, i.e. this will probably fix #1796.	9 years ago
Fabian Reinartz	425736a377	*: remove last remainers of non-second metrics	9 years ago
Jan van Valburg	68f3df49d0	stoarge: fix 'access denied' error on Windows ... On Windows, it is not possible to rename or delete a file that is currerntly open. This change closes the file in dropAndPersistChunks before it tries to delete it, or rename the temporary file to it.	9 years ago
beorn7	a90d645378	Checkpoint fingerprint mappings only upon shutdown ... Before, we checkpointed after every newly detected fingerprint collision, which is not a problem as long as collisions are rare. However, with a sufficient number of metrics or particular nature of the data set, there might be a lot of collisions, all to be detected upon the first set of scrapes, and then the checkpointing after each detection will take a quite long time (it's O(n²), essentially). Since we are rebuilding the fingerprint mapping during crash recovery, the previous, very conservative approach didn't even buy us anything. We only ever read from the checkpoint file after a clean shutdown, so the only time we need to write the checkpoint file is during a clean shutdown.	9 years ago
beorn7	47e3c90f9b	Clean up error propagation ... Only return an error where callers are doing something with it except simply logging and ignoring. All the errors touched in this commit flag the storage as dirty anyway, and that fact is logged anyway. So most of what is being removed here is just log spam. As discussed earlier, the class of errors that flags the storage as dirty signals fundamental corruption, no even bubbling up a one-time warning to the user (e.g. about incomplete results) isn't helping much because _anything_ happening in the storage has to be doubted from that point on (and in fact retroactively into the past, too). Flagging the storage dirty, and alerting on it (plus marking the state in the web UI) is the only way I can see right now. As a byproduct, I cleaned up the setDirty method a bit and improved the logged errors.	9 years ago
beorn7	836f1db04c	Improve MetricsForLabelMatchers ... WIP: This needs more tests. It now gets a from and through value, which it may opportunistically use to optimize the retrieval. With possible future range indices, this could be used in a very efficient way. This change merely applies some easy checks, which should nevertheless solve the use case of heavy rule evaluations on servers with a lot of series churn. Idea is the following: - Only archive series that are at least as old as the headChunkTimeout (which was already extremely unlikely to happen). - Then maintain a high watermark for the last archival, i.e. no archived series has a sample more recent than that watermark. - Any query that doesn't reach to a time before that watermark doesn't have to touch the archive index at all. (A production server at Soundcloud with the aforementioned series churn and heavy rule evaluations spends 50% of its CPU time in archive index lookups. Since rule evaluations usually only touch very recent values, most of those lookup should disappear with this change.) - Federation with a very broad label matcher will profit from this, too. As a byproduct, the un-needed MetricForFingerprint method was removed from the Storage interface.	9 years ago
beorn7	01795382c9	Merge branch 'beorn7/storage4' into beorn7/storage5	9 years ago
beorn7	f7fc542db6	Merge branch 'master' into beorn7/storage4 ... Conflicts: storage/local/persistence.go	9 years ago
beorn7	f193f2b8ef	Add a command to promtool that dumps metadata of heads.db ... I needed this today for debugging. It can certainly be improved, but it's already quite helpful. I refactored the reading of heads.db files out of persistence, which is an improvement, too. I made minor changes to the cli package to allow outputting via the io.Writer interface.	9 years ago
beorn7	fc7de5374a	Quarantine series upon problem writing to the series file ... This fixes https://github.com/prometheus/prometheus/issues/1059 , but not in the obvious way (simply not updating the persist watermark, because that's actually not that simple - we don't really know what has gone wrong exactly). As any errors relevant here are most likely caused by severe and unrecoverable problems with the series file, Using the now quarantine feature is the right step. We don't really have to be worried about any inconsistent state of the series because it will be removed for good ASAP. Another plus is that we don't have to declare the whole storage dirty anymore.	9 years ago
beorn7	0ea5801e47	Handle errors caused by data corruption more gracefully ... This requires all the panic calls upon unexpected data to be converted into errors returned. This pollute the function signatures quite lot. Well, this is Go... The ideas behind this are the following: - panic only if it's a programming error. Data corruptions happen, and they are not programming errors. - If we detect a data corruption, we "quarantine" the series, essentially removing it from the database and putting its data into a separate directory for forensics. - Failure during writing to a series file is not considered corruption automatically. It will call setDirty, though, so that a crashrecovery upon the next restart will commence and check for that. - Series quarantining and setDirty calls are logged and counted in metrics, but are hidden from the user of the interfaces in interface.go, whith the notable exception of Append(). The reasoning is that we treat corruption by removing the corrupted series, i.e. a query for it will return no results on its next call anyway, so return no results right now. In the case of Append(), we want to tell the user that no data has been appended, though. Minor side effects: - Now consistently using filepath.* instead of path.*. - Introduced structured logging where I touched it. This makes things less consistent, but a complete change to structured logging would be out of scope for this PR.	9 years ago
beorn7	28e9bbc15f	Populate chunkDesc.chunkLastTime during checkpoint loading, too	9 years ago
beorn7	ef3ab96111	Populate first and last time in the chunk descriptor earlier ... The First time is kind of trivial as we always know it when we create a new chunkDesc. The last time is only know when the chunk is closed, so we have to set it at that time. The change saves a lot of digging down into the chunk itself. Especially the last time is relative expensive as it involves the creation of an iterator. The first time access now doesn't require locking, which is also a nice gain.	9 years ago
beorn7	a2cd479058	Fix calculation of chunks to persist after restart ... Since we are not overestimating the number of chunks to persist anymore, this commit also adjusts the default value for -storage.local.memory-chunks. Update of documentation will follow.	9 years ago
beorn7	14796bdb60	Improve chunkMaxBatchSize doc comment	9 years ago
beorn7	582af1618c	Streamline chunk writing ... This helps to avoid allocations in the same way we were already doing it during reading.	9 years ago
beorn7	add2ebdd56	Tolerate the lost+found directory in the data directory	9 years ago
beorn7	4221c7de5c	Improve handling of series file truncation ... If only very few chunks are to be truncated from a very large series file, the rewrite of the file is a lorge overhead. With this change, a certain ratio of the file has to be dropped to make it happen. While only causing disk overhead at about the same ratio (by default 10%), it will cut down I/O by a lot in above scenario.	9 years ago
Fabian Reinartz	e3b6ec9784	Switch to common/log	9 years ago
Julius Volz	af513468eb	Fix some dead code, missing error checks, shadowings. ... I applied https://medium.com/@jgautheron/quality-pipeline-for-go-projects-497e34d6567 and was greeted with a deluge of warnings, most of which were not applicable or really fixable realistically. These are some of the first ones I decided to fix.	9 years ago
beorn7	daeccdd0e9	Fix DropMetricsForFingerprints ... It now deletes the series file also for archived series. Also, fix a naming error in a doc comment.	9 years ago
Julius Volz	6774a73878	Fix error checking and logging around checkpointing.	9 years ago
Fabian Reinartz	c9d396f476	Replace metric.LabelPair with model.LabelPair	9 years ago
Fabian Reinartz	438e232c9b	Fix grouping of import blocks	9 years ago
Fabian Reinartz	306e8468a0	Switch from client_golang/model to common/model	9 years ago
beorn7	699946bf32	Fix chunk desc loading. ... If all samples in consecutive chunks have the same timestamp, the way we used to load chunks will fail. With this change, the persist watermark is used to load the right amount of chunkDescs from disk. This bug is a possible reason for the rare storage corruption we have observed.	10 years ago