prometheus

Commit Graph

Author	SHA1	Message	Date
Fabian Reinartz	e061595352	Move COWMetric into storage/metric package	9 years ago
Fabian Reinartz	1535ef1457	Replace metric.SamplePair with model.SamplePair	9 years ago
Fabian Reinartz	c9d396f476	Replace metric.LabelPair with model.LabelPair	9 years ago
Fabian Reinartz	438e232c9b	Fix grouping of import blocks	9 years ago
Fabian Reinartz	306e8468a0	Switch from client_golang/model to common/model	9 years ago
Laurie Malau	20ad403587	Don't warn/increment metric upon equal timestamps during append. Perhaps it would be even better to still warn in case the sample value has changed but the timestamps are equal, but we don't have efficient access to the last value.	9 years ago
Julius Volz	517badc21d	Only do regex lookups when there was no equality match. For the label matching index-based preselection phase, don't do an OR between equality and non-equality matchers. Execute only one of the two (with equality matchers preferred when present). Fixes https://github.com/prometheus/prometheus/issues/924	9 years ago
beorn7	699946bf32	Fix chunk desc loading. If all samples in consecutive chunks have the same timestamp, the way we used to load chunks will fail. With this change, the persist watermark is used to load the right amount of chunkDescs from disk. This bug is a possible reason for the rare storage corruption we have observed.	10 years ago
beorn7	4203849c92	Test chunkDesc eviction and loading	10 years ago
beorn7	ff08f0b6fe	storage: ensure timestamp monotonicity within series. Fixes https://github.com/prometheus/prometheus/issues/481 While doing so, clean up and fix a few other things: - Fix `go vet` warnings (@fabxc to blame ;). - Fix a racey problem with unarchiving: Whenever we unarchive a series, we essentially want to do something with it. However, until we have done something with it, it appears like a series that is ready to be archived or even purged. So e.g. it would be ignored during checkpointing. With this fix, we always load the chunkDescs upon unarchiving. This is wasteful if we only want to add a new sample to an archived time series, but the (presumably more common) case where we access an archived time series in a query doesn't become more expensive. - The change above streamlined the getOrCreateSeries ond newMemorySeries flow. Also, the modTime is now always set correctly. - Fix the leveldb-backed implementation of KeyValueStore.Delete. It had the wrong behavior of still returning true, nil if a non-existing key has been passed in.	10 years ago
Fabian Reinartz	6bfb4549a6	storage: add LastSamplePairForFingerprint method	10 years ago
Fabian Reinartz	dc7d27ab9a	retrieval: add honor label handling and parametrized querying. This commit adds the honor_labels and params arguments to the scrape config. This allows to specify query parameters used by the scrapers and handling scraped labels with precedence.	10 years ago
Fabian Reinartz	1eff186555	Merge pull request #810 from prometheus/fabxc/lmatch Match empty labels.	10 years ago
Fabian Reinartz	5b91ea9b36	storage: improve label matching and allow unset matching. Matching of empty labels now also matches metrics where the label was not explicitly set to the empty string.	10 years ago
Fabian Reinartz	b105e26f4d	storage: remove global flags	10 years ago
Fabian Reinartz	5c6c0e2faa	Add storage method to delete time series	10 years ago
Fabian Reinartz	aff01e29c3	Limit retrievable samples to retention window. The storage does not delete data immediately after the retention period. We don't want to retrieve this data as it causes artifacts.	10 years ago
Fabian Reinartz	6e319532cf	Read from indexing queue during crash recovery. Change #704 introduced a regression that started reading the queue only after potential crash recovery. When more than the queue capacity was indexed, Prometheus deadlocked.	10 years ago
beorn7	3b9ab546e6	Add metrics to count inconsistencies and fp collisions.	10 years ago
Björn Rabenstein	c44e7cd105	Merge pull request #706 from prometheus/beorn7/persistence2 Improve iterator performance.	10 years ago
Fabian Reinartz	112a778922	Align int64s for atomic operations	10 years ago
beorn7	3b9c421a69	Weed out all the [Gg]et* method names. The only exception is getNumChunksToPersist to avoid naming the struct member numChunksToPersist in a weird way.	10 years ago
Julius Volz	267fd34156	Switch Prometheus to use github.com/prometheus/log. This change is conceptually very simple, although the diff is large. It switches logging from "github.com/golang/glog" to "github.com/prometheus/log", while not actually changing any log messages. V(1)-style logging has been changed to be log.Debug*().	10 years ago
beorn7	81b190bf45	Remove locking from series iterator. Cache chunk iterators.	10 years ago
Fabian Reinartz	d8440d75f1	Do not start storage processing before Start() is called.	10 years ago
beorn7	2235cec175	Handle fingerprint collisions.	10 years ago
beorn7	9820e5fe99	Use FastFingerprint where appropriate.	10 years ago
beorn7	c5fa0b90c3	Fix the case where a series in memory has 0 chunks, but chunks on disk. This is actually completely normal for a freshly unarchived series. Test added to expose.	10 years ago
beorn7	3035b8bfdd	Adaptively reduce the wait time for memory series maintenance. This will make in-memory series maintenance the faster the more chunks are waiting for persistence.	10 years ago
beorn7	6a21f73898	Fixes after review.	10 years ago
beorn7	51d35f4481	Instrument series maintenance durations.	10 years ago
beorn7	12ae6e9203	Increase resilience of the storage against data corruption - step 4. Step 4: Add a configurable sync'ing of series files after modification.	10 years ago
beorn7	11bd9ce1bd	Increase resilience of the storage against data corruption - step 3. Step 3: Remember the mtime of series files and make use of it to detect series files that are not the one the checkpoint thinks they are.	10 years ago
beorn7	e25cca823c	Increase resilience of the storage against data corruption - step 2. Step 2: Add a flag -storage.local.pedantic-checks to check every series file. Also, remove countPersistedHeadChunks channel, which is unused.	10 years ago
beorn7	3d8d8928be	Increase resilience of the storage against data corruption - step 1. Step 1: Admit the problem by turning the various "panic"s into logged errors, followed by marking the persistence as dirty.	10 years ago
beorn7	da7c0461c6	Rename persist queue len/cap to num/max chunks to persist. Remove deprecated flag storage.incoming-samples-queue-capacity.	10 years ago
beorn7	a075900f9a	Merge branch 'beorn7/persistence' into beorn7/ingestion-tweaks	10 years ago
beorn7	1d8fc7d56f	Change minor things after code review.	10 years ago
beorn7	be11cb2b07	Remove the sample ingestion channel. The one central sample ingestion channel has caused a variety of trouble. This commit removes it. Targets and rule evaluation call an Append method directly now. To incorporate multiple storage backends (like OpenTSDB), storage.Tee forks the Append into two different appenders. Note that the tsdb queue manager had its own queue anyway. It was a queue after a queue... Much queue, so overhead... Targets have their own little buffer (implemented as a channel) to avoid stalling during an http scrape. But a new scrape will only be started once the old one is fully ingested. The contraption of three pipelined ingesters was removed. A Target is an ingester itself now. Despite more logic in Target, things should be less confusing now. Also, remove lint and vet warnings in ast.go.	10 years ago
beorn7	0056eaeb4f	Redesign series maintenance and chunk persistence.	10 years ago
beorn7	5bea942d8e	Improve various things around chunk encoding. A number of mostly minor things: - Rename chunk type -> chunk encoding. - After all, do not carry around the chunk encoding to all parts of the system, but just have one place where the encoding for new chunks is set based on the flag. The new approach has caveats as well, but the polution of so many method signatures is worse. - Use the default chunk encoding for new chunks of existing series. (Previously, only new _series_ would get chunks with the default encoding.) - Use an enum for chunk encoding. (But keep the version number for the flag, for reasons discussed previously.) - Add encoding() to the chunk interface (so that a chunk knows its own encoding - no need to have that in a different top-level function). - Got rid of newFollowUpChunk (which would keep the existing encoding for all chunks of a time series). Now only use newChunk(), which will create a chunk encoding according to the flag. - Simplified transcodeAndAdd. - Reordered methods of deltaEncodedChunk and doubleDeltaEncoded chunk to match the order in the chunk interface. - Only transcode if the chunk is not yet half full. If more than half full, add a new chunk instead.	10 years ago
beorn7	13fcf1ddbc	Implement double-delta encoded chunks.	10 years ago
beorn7	5ed8f6c205	Update persistQueueLength after chunks were persisted.	10 years ago
beorn7	1db7589081	Reduce the capacity of countPersistedHeadChunks. The capacity is basically how many persisted head chunks we will count at most while doing other things, in particular checkpointing. To limit the amount of already counted head chunks, keep this number low, otherwise we will easily checkpoint too often if checkpoints take long anyway.	10 years ago
beorn7	9406afad72	Do not double-count non-persisted head chunks on loading.	10 years ago
beorn7	dbc22b972c	Check last time in head chunk for head chunk timeout, not first.	10 years ago
beorn7	edd716e63c	Fix the embarrassing bug introduced in commit `0851945`. In that commit, the 'maintainSeries' call was accidentally removed. This commit refactors things a bit so that there is now a clean 'maintainMemorySeries' and a 'maintainArchivedSeries' call. Straighten the nomenclature a bit (consistently use 'drop' for chunks and 'purge' for series/metrics). Remove the annoying 'Completed maintenance sweep through archived fingerprints' message if there were no archived fingerprints to do maintenance on.	10 years ago
beorn7	af91fb8e31	Improve persisting chunks to disk. This is done by bucketing chunks by fingerprint. If the persisting to disk falls behind, more and more chunks are in the queue. As soon as there are "double hits", we will now persist both chunks in one go, doubling the disk throughput (assuming it is limited by disk seeks). Should even more pile up so that we end wit "triple hits", we will persist those first, and so on. Even if we have millions of time series, this will still help, assuming not all of them are growing with the same speed. Series that get many samples and/or are not very compressable will accumulate chunks faster, and they will soon get double- or triple-writes. To improve the chance of double writes, -storage.local.persistence-queue-capacity could be set to a higher value. However, that will slow down shutdown a lot (as the queue has to be worked through). So we leave it to the user to set it to a really high value. A more fundamental solution would be to checkpoint not only head chunks, but also chunks still in the persist queue. That would be quite complicated for a rather limited use-case (running many time series with high ingestion rate on slow spinning disks).	10 years ago
beorn7	e22f26bc58	Move to a queue model for appending samples after all. Starting a goroutine takes 1-2µs on my laptop. From the "numbers every Go programmer should know", I had 300ns for a channel send in my mind. Turns out, on my laptop, it takes only 60ns. That's fast enough to warrant the machinery of yet another channel with a fixed set of worker goroutines feeding from it. The number chosen (8 for now) is low enough to not really afflict a measurable overhead (a big Prometheus server has >1000 goroutines running), but high enough to not make sample ingestion a bottleneck.	10 years ago
beorn7	fe518fdb28	Simplify AppendSamples by allowing it to be goroutine-unsafe.	10 years ago
beorn7	5d3cd65a5d	Improve performance of ingestion. - Parallelize AppendSamples as much as possible without breaking the contract about temporal order. - Allocate more fingerprint locker slots. - Do not run early checkpoints if we are behind on chunk persistence. - Increase fpMinWaitDuration to give the disk more time for more important things. Also, switch math.MaxInt64 and math.MinInt64 to the new constants.	10 years ago
beorn7	d2ab49c396	Make the persist queue length configurable. Also, set a much higher default value. Chunk persist requests can be quite spiky. If you collect a large number of time series that are very similar, they will tend to finish up a chunk at about the same time. There is no reason we need to back up scraping just because of that. The rationale of the new default value is "1/8 of the chunks in memory".	10 years ago
Julius Volz	9412b296d5	Remove labels on persist error counter. This fixes https://github.com/prometheus/prometheus/issues/496	10 years ago
Bjoern Rabenstein	2c8fdcbc23	Remove a deadlock during shutdown. If queries are still running when the shutdown is initiated, they will finish _during_ the shutdown. In that case, they might request chunk eviction upon unpinning their pinned chunks. That might completely fill the evict request queue _after_ draining it during storage shutdown. If that ever happens (which is the case if there are _many_ queries still running during shutdown), the affected queries will be stuck while keeping a fingerprint locked. The checkpointing can then not process that fingerprint (or one that shares the same lock). And then we are deadlocked.	10 years ago
Bjoern Rabenstein	5859b74f1b	Clean up license issues. - Move CONTRIBUTORS.md to the more common AUTHORS. - Added the required NOTICE file. - Changed "Prometheus Team" to "The Prometheus Authors". - Reverted the erroneous changes to the Apache License.	10 years ago
Julius Volz	a6bc42bc61	Minor formatting/spelling fixups.	10 years ago
Bjoern Rabenstein	0851945054	Add a heuristics to checkpoint early if there are many "dirty" series..	10 years ago
Bjoern Rabenstein	622e8350cd	Fix a bug handling freshly unarchived series. Usually, if you unarchive a series, it is to add something to it, which will create a new head chunk. However, if a series in unarchived, and before anything is added to it, it is handled by the maintenance loop, it will be archived again. In that case, we have to load the chunkDescs to know the lastTime of the series to be archived. Usually, this case will happen only rarely (as a race, has never happened so far, possibly because the locking around unarchiving and the subsequent sample append is smart enough). However, during crash recovery, we sometimes treat series as "freshly unarchived" without directly appending a sample. We might add more cases of that type later, so better deal with archiving properly and load chunkDescs if required.	10 years ago
Bjoern Rabenstein	eb932d1524	Remove a deadlock during shutdown.	10 years ago
Brian Brazil	e56786b221	Have scrape time as a pseudovariable, not a prometheus variable. This ensures it has the right timestamp, and is easier to work with. Switch sd variable away from 'outcome', using total/failed instead.	10 years ago
Julius Volz	c9618d11e8	Introduce copy-on-write for metrics in AST. This depends on changes in: https://github.com/prometheus/client_golang/tree/cow-metrics. Change-Id: I80b94833a60ddf954c7cd92fd2cfbebd8dd46142	10 years ago
Bjoern Rabenstein	674624f1c8	Completed more TODOs. - Documented checkpoint file format. - High-level description of series sanitation. - Replace fp.LoadFromString panic with an error. (Change in client_golang already submitted.) - Introduced checks for series file size where appropriate. - Removed two Law of Demeter violations. Change-Id: I555d97a2c8f4769820c2fc8bf5d6f4e160222abc	10 years ago
Bjoern Rabenstein	7d11019aa2	Squash a few trivial TODOs. - Delete unneeded file view_adapter.go. - Assessed that we still need the fingerprints in nodes (to create iterators). - Turned numMemChunkDescs into a metric. Change-Id: I29be963c795a075ec00c095f76bf26405535609d	10 years ago
Bjoern Rabenstein	14bda4180c	Changes after pair code review. Change-Id: Ib72d40f8e9027818cfbbd32a7a7201eebda07455	10 years ago
Bjoern Rabenstein	9ea808cd8b	Remove debug log line. Change-Id: Icdd2351b89f2d37ac2b615f9cf872e054c694ad1	10 years ago
Bjoern Rabenstein	bb42cc2e2d	Evict based on memory pressure. Evict recently used chunks last. Change-Id: Ie6168f0cdb3917bdc63b6fe15585dd70c1e42afe	10 years ago
Bjoern Rabenstein	d73e851b14	Tweak timing in the maintenance loop. Change-Id: I9801c4f9a22c3b3dc1ce1af81fdd9e992a4f4dd7	10 years ago
Bjoern Rabenstein	2672aa8ece	Instrument series maintenance. Change-Id: Ie4269d07ad4d23d44230c95a523088b472718e54	10 years ago
Bjoern Rabenstein	74c143c4c9	Improve scraper shutdown time. - Stop target pools in parallel. - Stop individual scrapers in goroutines, too. - Timing tweaks. Change-Id: I9dff1ee18616694f14b04408eaf1625d0f989696	10 years ago
Bjoern Rabenstein	3f61d304ce	Reorganize maintenance loop. Change-Id: Iac10f988ba3e93ffb188f49c30f92e0b6adce5a3	10 years ago
Bjoern Rabenstein	c087ee35f7	Remove archiveMtx. Change-Id: Ie8019f860bbda68621f74380c90a4e57930d3d7a	10 years ago
Bjoern Rabenstein	7af42eda65	Optimize purging. Now only purge if there is something to purge. Also, set savedFirstTime and archived time range appropriately. (Which is needed for the optimization.) Change-Id: Idcd33319a84def3ce0318d886f10c6800369e7f9	10 years ago
Bjoern Rabenstein	904acd43da	Add crash recovery. Fix the behavior if preload for non-existent series is requested. Instead of returning an error (which triggers a panic further up), simply count those incidents. They can happen regularly, we just want to know if they happen too frequently because that would mean the indexing is behind or broken. Change-Id: I4b2d1b93c4146eeea897d188063cb9574a270f8b	10 years ago
Bjoern Rabenstein	4efc60174b	Tweak and verify a few parameters. Remove TODOs accordingly. Change-Id: Ic062e13b6ae89a9135d3f14011114fe1cca1cef8	10 years ago
Bjoern Rabenstein	5f8e9617ef	Add more tests. Add an end-to-end fuzz and race test. Fix a race exposed by the above. Change-Id: Ifaa39a90cefbde8d4c29bda197cc92592ded21bb	10 years ago
Bjoern Rabenstein	d215e013b7	Fix the weird chunkDesc shuffling bug. The root cause was that after chunkDesc eviction, the offset between memory representation of chunk layout (via chunkDescs in memory) was shiftet against chunks as layed out on disk. Keeping the offset up to date is by no means trivial, so this commit is pretty involved. Also, found a race that for some reason didn't bite us so far: Persisting chunks was completel unlocked, so if chunks were purged on disk at the same time, disaster would strike. However, locking the persisting of chunk revealed interesting dead locks. Basically, never queue under the fp lock. Change-Id: I1ea9e4e71024cabbc1f9601b28e74db0c5c55db8	10 years ago
Bjoern Rabenstein	f1de5b0c4e	Run checkpointing of in-memory metrics and head chunks periodically. Checkpointing interval is now a command line flag. Along the way, several things were refactored. - Restructure the way the storage is started and stopped.. - Number of series in checkpoint is now a uint64, not a varint. (Breaks old checkpoints, needs wipe!) - More consistent naming and order of methods. Change-Id: I883d9170c9a608ee716bb0ab3d0ded8ca03760d9	10 years ago
Bjoern Rabenstein	74c9b34a5e	Improve storage instrumentation even more. Add gauge for chunks and chunkdescs in memory (backed by a global variable to be used later not only for instrumentation but also for memory management). Refactored instrumentation code once more (instrumentation.go is back :). Change-Id: Ife39947e22a48cac4982db7369c231947f446e17	10 years ago
Bjoern Rabenstein	443dd33805	Improve instrumentation in storage. Also, fix some other minor bugs. Change-Id: If72f1c058b0f47d3e378fdf80228d7e9a8db06c7	10 years ago
Bjoern Rabenstein	1936a40e75	Minor loging improvement. Change-Id: I7875d1a58ef9c5ff149f18e36f65959a4712fea2	10 years ago
Bjoern Rabenstein	096fa0f8b2	Squash a number of TODOs. - Staleness delta is no a proper function parameter and not replicated from package ast. - Named type 'chunks' replaced by explicit '[]chunk' to avoid confusion. - For the same reason, replaced 'chunkDescs' by '[]*chunkDescs'. - Verified that math.Modf is not a speed enhancement over conversion (actually 5x slower). - Renamed firstTimeField, lastTimeField into chunkFirstTime and chunkLastTime. - Verified unpin() is sufficiently goroutine-safe. - Decided not to update archivedFingerprintToTimeRange upon series truncation and added a rationale why. Change-Id: I863b8d785e5ad9f71eb63e229845eacf1bed8534	10 years ago
Bjoern Rabenstein	427c8d53a5	Fix handling of empty chunkDescs while preloading chunks. Change-Id: I73ce89fe0ef90c6eda78218e5be2cbfa0207c364	10 years ago
Bjoern Rabenstein	ecee5d8281	Fix head chunk persisting and a chunkDesc race condition. - Head chunk persisting only happens in evictOlderThan, so do it there. (With the previous code, it would never happen.) - Raw accesses to chunkDesc.chunk are now done via isEvicted (with locking). Change-Id: I48b07b56dfea4899b50df159b4ea566954396fcd	10 years ago
Bjoern Rabenstein	2b4ff620aa	Return a nop iterator for series that have been purged completely. Change-Id: I6e92cac4472486feefdecba8593c17867e8c710d	10 years ago
Julius Volz	bfa64248b7	Deal with missing series in preloading. Change-Id: Ibf3a57b329f40a3d5e0b98464a2f45d2f1bd07bf	10 years ago
Bjoern Rabenstein	b3ed9aa7a2	Clean up start-up and shut-down. Change-Id: Idff4bbb0a15a9f879bfbb3da5b1025179cab5e2c	10 years ago
Bjoern Rabenstein	4447708c9f	Fix a race in target.go. Also, fix problems in shutdown. Starting serving and shutdown still has to be cleaned up properly. It's a mess. Change-Id: I51061db12064e434066446e6fceac32741c4f84c	10 years ago
Bjoern Rabenstein	38fc24d0ed	Fix targetpool_test.go and other tests. Change-Id: I91a4dd1d39e01f174e1aaae653ce1ed7aecaa624	10 years ago
Julius Volz	7f5d3c2c29	Fix and improve the fp locker. Benchmark: $ go test -bench 'Fingerprint' -test.run 'Fingerprint' -test.cpu=1,2,4 OLD BenchmarkFingerprintLockerParallel 500000 3618 ns/op BenchmarkFingerprintLockerParallel-2 100000 12257 ns/op BenchmarkFingerprintLockerParallel-4 500000 10164 ns/op BenchmarkFingerprintLockerSerial 10000000 283 ns/op BenchmarkFingerprintLockerSerial-2 10000000 284 ns/op BenchmarkFingerprintLockerSerial-4 10000000 288 ns/op NEW BenchmarkFingerprintLockerParallel 1000000 1018 ns/op BenchmarkFingerprintLockerParallel-2 1000000 1164 ns/op BenchmarkFingerprintLockerParallel-4 2000000 910 ns/op BenchmarkFingerprintLockerSerial 50000000 56.0 ns/op BenchmarkFingerprintLockerSerial-2 50000000 47.9 ns/op BenchmarkFingerprintLockerSerial-4 50000000 54.5 ns/op Change-Id: I3c65a43822840e7e64c3c3cfe759e1de51272581	10 years ago
Bjoern Rabenstein	8fba3302bc	Bold changes to concurrency. (WIP. Probably doesn't work yet.) Change-Id: Id1537dfcca53831a1d428078a5863ece7bdf4875	10 years ago
Bjoern Rabenstein	fcdf5a8ee7	Fix bugs in chunk evict code. Also, simplify code by re-looking up metric in metric map. Change-Id: Ib2092f9184374e5a543e87d3a9f4a74fda64b193	10 years ago
Bjoern Rabenstein	7e6a03fbf9	Fix a few concurrency issues before starting to use the new fp locker. Change-Id: I8615e8816e79ef0882e123163ee590c739b79d12	10 years ago
Julius Volz	db92620163	Instrument eviction and purge durations. Change-Id: Ia5b2319363ad2644674c9b7a94162a89bcc296fb	10 years ago
Bjoern Rabenstein	01dd618a20	Fix a locking bug. Change-Id: I183780785991d0b4165ce9186f53eb8201fb3ed5	10 years ago
Bjoern Rabenstein	aea32b0b4b	Avoid redundant fingerprint calculation. Change-Id: Ief8a165dcfa5030226953346ec9dfe4a7787df1f	10 years ago
Bjoern Rabenstein	e9ff29c547	Comment/code cleanup. Change-Id: I38736e3d0fec79759a2bafa35aecf914480ff810	10 years ago
Bjoern Rabenstein	c7aad110fb	Add an indexing queue and batch the ops. Some other improvements on the way, in particular codec -> codable renaming and addition of LookupSet methods. Change-Id: I978f8f3f84ca8e4d39a9d9f152ae0ad274bbf4e2	10 years ago
Bjoern Rabenstein	71206dbc06	More code cleanups. Add license text everywhere. And others.... Change-Id: I11ccde267a2ef7eb366c4788ba7aeae14ba7545c	10 years ago
Julius Volz	f0d5d4bda3	Fix bug around index purging. Change-Id: I8cea00e03f72bbeead2cbd2d26b34d986059ced0	10 years ago
Julius Volz	630b5a087a	Also consider on-disk fingerprints during purge. This reintroduces LevelDB iterators so that we can iterate through all the on-disk fingerprints. Change-Id: I007ee4638d038d2a4461bbda27f30fcaad411474	10 years ago

1 2 3 4

160 Commits (8066de91ca26bbf268e28283aef1e6cf78eb5c17)