prometheus

Commit Graph

Author	SHA1	Message	Date
Julius Volz	a1a97ed064	Fix chunk corruption compaction bug. This fixes part 1) of https://github.com/prometheus/prometheus/issues/367 (the storing of samples with the wrong fingerprint into a compacted chunk, thus corrupting it). Change-Id: I4c36d0d2e508e37a0aba90b8ca2ecc78ee03e3f1	11 years ago
Matt T. Proud	86fcbe5bde	Retain DTO on each cycle. Change-Id: Ifc6f68f98eacb01097771d0dbf043c98bba1d518	11 years ago
Matt T. Proud	4a87c002e8	Update low-level i'faces to reflect wireformats. This commit fixes a critique of the old storage API design, whereby the input parameters were always as raw bytes and never Protocol Buffer messages that encapsulated the data, meaning every place a read or mutation was conducted needed to manually perform said translations on its own. This is taxing. Change-Id: I4786938d0d207cefb7782bd2bd96a517eead186f	11 years ago
Matt T. Proud	7910f6e863	Prevent total storage locking during memory flush. While a hack, this change should allow us to serve queries expeditiously during a flush operation. Change-Id: I9a483fd1dd2b0638ab24ace960df08773c4a5079	11 years ago
Matt T. Proud	12d5e6ca5a	Curation should not starve user-interactive ops. The background curation should be staggered to ensure that disk I/O yields to user-interactive operations in a timely manner. The lack of routine prioritization necessitates this. Change-Id: I9b498a74ccd933ffb856e06fedc167430e521d86	11 years ago
Matt T. Proud	2b42fd0068	Snapshot of no more frontier. Change-Id: Icd52da3f52bfe4529829ea70b4865ed7c9f6c446	11 years ago
Matt T. Proud	7db518d3a0	Abstract high watermark cache into standard LRU. Conflicts: storage/metric/memory.go storage/metric/tiered.go storage/metric/watermark.go Change-Id: Iab2aedbd8f83dc4ce633421bd4a55990fa026b85	11 years ago
Matt T. Proud	d74c2c54d4	Interfacification of stream. Move the stream to an interface, for a number of additional changes around it are underway. Conflicts: storage/metric/memory.go Change-Id: I4a5fc176f4a5274a64ebdb1cad52600954c463c3	11 years ago
Matt T. Proud	c262907fec	Kill interface cruft. These pieces were never used and should be thusly removed. Change-Id: I8dd151ec4c40b6d3ccffad1bb9b8b75a92e9ee37	11 years ago
Matt T. Proud	b23acccea8	Kill AppendSample interface definition. AppendSample will be repcated with AppendSamples, which will take advantage of bulks appends. This is a necessary step for indexing pipeline decoupling. Change-Id: Ia83811a87bcc89973d3b64d64b85a28710253ebc	11 years ago
Matt T. Proud	aaaf3367d6	Include forgotten imports. This fixes the build. Change-Id: Id132f4342adb9ed20116191086f157ca7f7cf515	11 years ago
Matt T. Proud	acf91f38bd	Build layered indexers. The indexers will be extracted in a short while and wrapped accordingly with these types. Change-Id: I4d1abda4e46117210babad5aa0d42f9ca1f6594f	11 years ago
Matt T. Proud	972e856d9b	Kill the curation state channel. The use of the channels for curation state were always unidiomatic. Change-Id: I1cb1d7175ebfb4faf28dff84201066278d6a0d92	11 years ago
Matt T. Proud	1ceb25b701	Publication of LevelDBMetricPersistence Fields. This will enable us to break down the onerous construction method. Change-Id: Ia89337ba39d6745af6757180af2485ec8a990a3b	11 years ago
Julius Volz	0003027dce	Add needed trailing spaces in logs.	11 years ago
Julius Volz	aa5d251f8d	Use github.com/golang/glog for all logging.	11 years ago
Matt T. Proud	a5141e4d0a	Depointerize storage conf. and chain ingester. The storage builders need to work with the assumption that they have a copy of the underlying configuration data if any mutations are made.	11 years ago
Matt T. Proud	820e551988	Code Review: Nits.	11 years ago
Matt T. Proud	a3bf2efdd5	Replace index writes with wrapped interface. This commit is the first of several and should not be regarded as the desired end state for these cleanups. What this one does it, however, is wrap the query index writing behind an interface type that can be injected into the storage stack and have its lifecycle managed separately as needed. It also would mean we can swap out underlying implementations to support remote indexing, buffering, no-op indexing very easily. In the future, most of the individual index interface members in the tiered storage will go away in favor of agents that can query and resolve what they need from the datastore without the user knowing how and why they work.	11 years ago
Matt T. Proud	52664f701a	Hot Fix: Use extracted time.	11 years ago
Matt T. Proud	38dac35b3e	Code Review: Short name consistency.	11 years ago
Matt T. Proud	a00f18d78b	Code Review: Manual re-alignment.	11 years ago
Matt T. Proud	cc989c68e1	Replace direct curation table access with wrapper.	11 years ago
Matt T. Proud	07ac921aec	Code Review: First pass.	11 years ago
Matt T. Proud	d8792cfd86	Extract HighWatermarking. Clean up the rest.	11 years ago
Matt T. Proud	f4669a812c	Extract index storage into separate types.	11 years ago
Matt T. Proud	772d3d6b11	Consolidate LevelDB storage construction. There are too many parameters to constructing a LevelDB storage instance for a construction method, so I've opted to take an idiomatic approach of embedding them in a struct for easier mediation and versioning.	11 years ago
Julius Volz	e3415e953f	Add notifications telemetry.	11 years ago
juliusv	927435d68e	Merge pull request #333 from prometheus/round-time Round time to nearest second in memory storage.	12 years ago
Julius Volz	5d88e8cc45	Round time to nearest second in memory storage. When samples get flushed to disk, they lose sub-second precision anyways. By already dropping sub-second precision, data fetched from memory vs. disk will behave the same. Later, we should consider also storing a more compact representation than time.Time in memory if we're not going to use its full precision.	12 years ago
Matt T. Proud	f7704af4f8	Code Review: Formatting comments.	12 years ago
Julius Volz	a76a797f3f	Always treat series without watermarks as too old. Current series always get watermarks written out upon append now. This drops support for old series without any watermarks by always reporting them as too old (stale) during queries.	12 years ago
Julius Volz	d2da21121c	Implement getValueRangeAtIntervalOp for faster range queries. This also short-circuits optimize() for now, since it is complex to implement for the new operator, and ops generated by the query layer already fulfill the needed invariants. We should still investigate later whether to completely delete operator optimization code or extend it to support getValueRangeAtIntervalOp operators.	12 years ago
Julius Volz	e7f049c85b	Fix expunging of empty memory series (loop var pointerization bug)	12 years ago
Julius Volz	baa5b07829	Fix condition for dropping empty memory series.	12 years ago
Matt T. Proud	30b1cf80b5	WIP - Snapshot of Moving to Client Model.	12 years ago
juliusv	42198c1f1c	Merge pull request #311 from prometheus/fix/watermarking/on-first-write Ensure new metrics are watermarked early.	12 years ago
Matt T. Proud	4137c75523	Shrink default LRU cache sizes. Observing Prometheus in production confirms we can lower these values safely.	12 years ago
Matt T. Proud	ecb9c7bb9d	Code Review: Swap ordering of elements.	12 years ago
Matt T. Proud	5daa0a09ea	Code Review: Swap ordering of watermark getting. A test for Julius.	12 years ago
Matt T. Proud	ee840904d2	Code Review: !Before -> After.	12 years ago
Matt T. Proud	2d5de99fbf	Regard in-memory series as new. This commit ensures that series that exist only in-memory and not on-disk are not regarded as too old for operation exclusion.	12 years ago
Matt T. Proud	81c406630a	Merge pull request #312 from prometheus/fix/sample-append-logging Log correct sample count when appending to disk.	12 years ago
Matt T. Proud	a1a23fbaf8	Ensure new metrics are watermarked early. With the checking of fingerprint freshness to cull stale metrics from queries, we should write watermarks early to aid in more accurate responses.	12 years ago
Julius Volz	ba8c122147	Log correct sample count when appending to disk.	12 years ago
Julius Volz	f2b4067b7b	Speedup and clean up operation optimization.	12 years ago
Julius Volz	008bc09da8	Move check for empty memory series to separate method.	12 years ago
Julius Volz	16364eda37	Drop empty series from memory after flushing.	12 years ago
Julius Volz	71199e2c93	Cache disk fingerprint->metric lookups in memory.	12 years ago
Matt T. Proud	a73f061d3c	Persist solely Protocol Buffers. An design question was open for me in the beginning was whether to serialize other types to disk, but Protocol Buffers quickly won out, which allows us to drop support for other types. This is a good start to cleaning up a lot of cruft in the storage stack and can let us eventually decouple the various moving parts into separate subsystems for easier reasoning. This commit is not strictly required, but it is a start to making the rest a lot more enjoyable to interact with.	12 years ago
juliusv	95400cb785	Merge pull request #290 from prometheus/fix/go-vet Minor "go tool vet" cleanups	12 years ago
Julius Volz	558281890b	Minor "go tool vet" cleanups	12 years ago
juliusv	615972dd01	Merge pull request #288 from prometheus/fix/curator/fallthrough-compaction-ordering Fix fallthrough compaction value ordering.	12 years ago
Matt T. Proud	86f63b078b	Fix fallthrough compaction value ordering. We discovered a regression whereby data chunks could be appended out of order if the fallthrough case was hit.	12 years ago
Julius Volz	7b9ee95030	Minor LevelDB watermark handling cleanups.	12 years ago
Julius Volz	84741b227d	Use LRU cache to avoid querying stale series.	12 years ago
Julius Volz	f98853d7b7	Fix type error in watermark list handling.	12 years ago
Matt T. Proud	ef1d5fd8a2	Introduce semaphores for tiered storage. This commit wraps the tiered storage access componnets in semaphores, since we can handle several concurrent memory reads.	12 years ago
Matt T. Proud	819045541e	Code Review: Make double-drain a panic.	12 years ago
Matt T. Proud	e217a9fb41	Race Work: Make memory arena locks more coarse. We can optimize these as needed later.	12 years ago
Matt T. Proud	beaaf386e7	Add storage state guards and transition callbacks. To ensure that we access tiered storage in the proper way, we have guards now.	12 years ago
Matt T. Proud	abb5353ade	Merge pull request #283 from prometheus/feature/storage/consult-watermark Include LRU cache for fingerprint watermarks.	12 years ago
Matt T. Proud	2c3df44af6	Ensure database access waits until it is started. This commit introduces a channel message to ensure serving state has been reached with the storage stack before anything attempts to use it.	12 years ago
Matt T. Proud	cbe2f3a7b1	Include LRU cache for fingerprint watermarks.	12 years ago
Julius Volz	51689d965d	Add debug timers to instant and range queries. This adds timers around several query-relevant code blocks. For now, the query timer stats are only logged for queries initiated through the UI. In other cases (rule evaluations), the stats are simply thrown away. My hope is that this helps us understand where queries spend time, especially in cases where they sometimes hang for unusual amounts of time.	12 years ago
Matt T. Proud	8339a189cb	Code Review: Fix seriesPresent scope. The seriesPresent scope should be constrained to the scope of a scanJob, since this is keyed to given series.	12 years ago
Matt T. Proud	fe41ce0b19	Conditionalize disk initializations. This commit conditionalizes the creation of the diskFrontier and seriesFrontier along with the iterator such that they are provisioned once something is actually required from disk.	12 years ago
Julius Volz	a8468a2e5e	Fix reversed disk flush cutoff behavior.	12 years ago
Julius Volz	eb1f956909	Revert "Revert "Ensure that all extracted samples are added to view."" This reverts commit `4b30fb86b4`.	12 years ago
Matt T. Proud	4b30fb86b4	Revert "Ensure that all extracted samples are added to view." This reverts commit `008314b5a8`. By running an automated git bisection described in https://gist.github.com/matttproud-soundcloud/22a371a8d2cba382ea64 this commit was found.	12 years ago
Julius Volz	750f862d9a	Use GetBoundaryValues() for non-counter deltas.	12 years ago
Julius Volz	f2b48b8c4a	Make getValuesAtIntervalOp consume all chunk data in one pass. This is mainly a small performance improvement, since we skip past the last extracted time immediately if it was also the last sample in the chunk, instead of trying to extract non-existent values before the chunk end again and again and only gradually approaching the end of the chunk.	12 years ago
Julius Volz	83d60bed89	extractValuesAroundTime() code simplification.	12 years ago
Julius Volz	008314b5a8	Ensure that all extracted samples are added to view. The current behavior only adds those samples to the view that are extracted by the last pass of the last processed op and throws other ones away. This is a bug. We need to append all samples that are extracted by each op pass. This also makes view.appendSamples() take an array of samples.	12 years ago
Matt T. Proud	b586801830	Code Review: Fix to-disk queue infinite growth. We discovered a bug while manually testing this branch on a live instance, whereby the to-disk queue was never actually dumped to disk.	12 years ago
Matt T. Proud	285a8b701b	Code Review: Extend lock.	12 years ago
Matt T. Proud	2526ab8c81	Code Review: Extend lock scope for appending.	12 years ago
Matt T. Proud	f994482d15	Code Review: Avenues for future improvemnet noted.	12 years ago
Matt T. Proud	298a90c143	Code Review: Initial arena size name.	12 years ago
Matt T. Proud	c07abf8521	Initial move away from skiplist.	12 years ago
Matt T. Proud	74a66fd938	Spawn grouping of fingerprints with free semaphore. The previous implementation spawned N goroutines to group samples together and would not start work until the semaphore unblocked. While this didn't leak, it polluted the scheduling space. Thusly, the routine only starts after a semaphore has been acquired.	12 years ago
Julius Volz	5b105c77fc	Repointerize fingerprints.	12 years ago
Matt T. Proud	ec5b5bae28	Fuck you, Travis.	12 years ago
Matt T. Proud	e5ac91222b	Benchmark memory arena; simplify map generation. The one-off keys have been replaced with ``model.LabelPair``, which is indexable. The performance impact is negligible, but it represents a cognitive simplification.	12 years ago
juliusv	360477f66c	Merge pull request #257 from prometheus/feature/better-memory-behaviors Pointerize memorySeriesArena.	12 years ago
Matt T. Proud	e1f20de2e9	Pointerize memorySeriesArena.	12 years ago
Matt T. Proud	8f4c7ece92	Destroy naked returns in half of corpus. The use of naked return values is frowned upon. This is the first of two bulk updates to remove them.	12 years ago
Matt T. Proud	4e0c932a4f	Simplify Encoder's encoding signature. The reality is that if we ever try to encode a Protocol Buffer and it fails, it's likely that such an error is ultimately not a runtime error and should be fixed forthwith. Thusly, we should rename ``Encoder.Encode`` to ``Encoder.MustEncode`` and drop the error return value.	12 years ago
juliusv	516101f015	Merge pull request #250 from prometheus/refactor/drop-unused-storage-setting Drop unused writeMemoryInterval	12 years ago
juliusv	9ff00b651d	Merge pull request #251 from prometheus/fix/memory-metric-mutability Fix GetMetricForFingerprint() metric mutability.	12 years ago
Bernerd Schaefer	63d9988b9c	Drop unused writeMemoryInterval	12 years ago
Julius Volz	83c60ad43a	Fix GetMetricForFingerprint() metric mutability. Some users of GetMetricForFingerprint() end up modifying the returned metric labelset. Since the memory storage's implementation of GetMetricForFingerprint() returned a pointer to the metric (and maps are reference types anyways), the external mutation propagated back into the memory storage. The fix is to make a copy of the metric before returning it.	12 years ago
Bernerd Schaefer	428d91c86f	Rename test helper files to helpers_test.go This ensures that these files are properly included only in testing.	12 years ago
juliusv	98e512d755	Merge pull request #246 from prometheus/fix/interval-value-extraction Fix and optimize getValuesAtIntervalOp data extraction.	12 years ago
Julius Volz	71a3172abb	Fix and optimize getValuesAtIntervalOp data extraction. - only the data extracted in the last loop iteration of ExtractSamples() was emitted as output - if e.g. op interval < sample interval, there were situations where the same sample was added multiple times to the output	12 years ago
Matt T. Proud	244a4a9cdb	Update to go1.1. This commit updates the documentation, Makefiles, formatting, and code semantics to support the 1.1. runtime, which includes ... 1. ``make advice``, 2. ``make format``, and 3. ``go fix`` on various targets.	12 years ago
Matt T. Proud	b224251981	Simplify compaction and expose database sizes. This commit simplifies the way that compactions across a database's keyspace occur due to reading the LevelDB internals. Secondarily it introduces the database size estimation mechanisms. Include database health and help interfaces. Add database statistics; remove status goroutines. This commit kills the use of Go routines to expose status throughout the web components of Prometheus. It also dumps raw LevelDB status on a separate /databases endpoint.	12 years ago
juliusv	92ad65ff13	Merge pull request #232 from prometheus/optimize/granular-storage-locking Synchronous memory appends and more fine-grained storage locks.	12 years ago
Matt T. Proud	1f7f89b4e3	Simplify compaction and expose database sizes. This commit simplifies the way that compactions across a database's keyspace occur due to reading the LevelDB internals. Secondarily it introduces the database size estimation mechanisms.	12 years ago
Matt T. Proud	d538b0382f	Include long-tail data deletion mechanism. This commit introduces the long-tail deletion mechanism, which will automatically cull old sample values. It is an acceptable hold-over until we get a resampling pipeline implemented. Kill legacy OS X documentation, too.	12 years ago

1 2 3 4 5 ...

307 Commits (80b3d3bf347c246a94ab9227dd17b9fd25c2d50b)