prometheus

Commit Graph

Author	SHA1	Message	Date
Matt T. Proud	a73f061d3c	Persist solely Protocol Buffers. An design question was open for me in the beginning was whether to serialize other types to disk, but Protocol Buffers quickly won out, which allows us to drop support for other types. This is a good start to cleaning up a lot of cruft in the storage stack and can let us eventually decouple the various moving parts into separate subsystems for easier reasoning. This commit is not strictly required, but it is a start to making the rest a lot more enjoyable to interact with.	12 years ago
Matt T. Proud	8f4c7ece92	Destroy naked returns in half of corpus. The use of naked return values is frowned upon. This is the first of two bulk updates to remove them.	12 years ago
Matt T. Proud	4e0c932a4f	Simplify Encoder's encoding signature. The reality is that if we ever try to encode a Protocol Buffer and it fails, it's likely that such an error is ultimately not a runtime error and should be fixed forthwith. Thusly, we should rename ``Encoder.Encode`` to ``Encoder.MustEncode`` and drop the error return value.	12 years ago
Matt T. Proud	b224251981	Simplify compaction and expose database sizes. This commit simplifies the way that compactions across a database's keyspace occur due to reading the LevelDB internals. Secondarily it introduces the database size estimation mechanisms. Include database health and help interfaces. Add database statistics; remove status goroutines. This commit kills the use of Go routines to expose status throughout the web components of Prometheus. It also dumps raw LevelDB status on a separate /databases endpoint.	12 years ago
Matt T. Proud	1f7f89b4e3	Simplify compaction and expose database sizes. This commit simplifies the way that compactions across a database's keyspace occur due to reading the LevelDB internals. Secondarily it introduces the database size estimation mechanisms.	12 years ago
Matt T. Proud	fa6a1f97d0	Expose interfaces for pruner and make pruner tool. In order to run database cleanups and diagnostics, we should have a means for pruning a database---even if LevelDB does this for us.	12 years ago
Matt T. Proud	b3e34c6658	Implement batch database sample curator. This commit introduces to Prometheus a batch database sample curator, which corroborates the high watermarks for sample series against the curation watermark table to see whether a curator of a given type needs to be run. The curator is an abstract executor, which runs various curation strategies across the database. It remarks the progress for each type of curation processor that runs for a given sample series. A curation procesor is responsible for effectuating the underlying batch changes that are request. In this commit, we introduce the CompactionProcessor, which takes several bits of runtime metadata and combine sparse sample entries in the database together to form larger groups. For instance, for a given series it would be possible to have the curator effectuate the following grouping: - Samples Older than Two Weeks: Grouped into Bunches of 10000 - Samples Older than One Week: Grouped into Bunches of 1000 - Samples Older than One Day: Grouped into Bunches of 100 - Samples Older than One Hour: Grouped into Bunches of 10 The benefits hereof of such a compaction are 1. a smaller search space in the database keyspace, 2. better employment of compression for repetious values, and 3. reduced seek times.	12 years ago
Matt T. Proud	a55602df4a	Validate diskFrontier domain for series candidate. It is the case with the benchmark tool that we thought that we generated multiple series and saved them to the disk as such, when in reality, we overwrote the fields of the outgoing metrics via Go map reference behavior. This was accidental. In the course of diagnosing this, a few errors were found: 1. ``newSeriesFrontier`` should check to see if the candidate fingerprint is within the given domain of the ``diskFrontier``. If not, as the contract in the docstring stipulates, a ``nil`` ``seriesFrontier`` should be emitted. 2. In the interests of aiding debugging, the raw LevelDB ``levigoIterator`` type now includes a helpful forensics ``String()`` method. This work produced additional cleanups: 1. ``Close() error`` with the storage stack is technically incorrect, since nowhere in the bowels of it does an error actually occur. The interface has been simplified to remove this for now.	12 years ago
Matt T. Proud	c3e3460ca6	Spin up curator run in the tests. After this commit, we'll need to add validations that it does the desired work, which we presently know that it doesn't. Given the changes I made with a plethora of renamings, I want to commit this now before it gets even larger.	12 years ago
Matt T. Proud	c53a72a894	Test data for the curator.	12 years ago
Matt T. Proud	6dcaa28806	Include LevelDB fixture generators for curator. This will help reduce common boilerplate for our test process with respect to LevelDB-related things.	12 years ago
Matt T. Proud	b2e4c88b80	Wrap LevelDB iterator operations behind interface. The LevelDB storage types return an interface type now that wraps around the underlying iterator. This both enhances testability but improves upon, in my opinion, the interface design for the LevelDB iterator. Secondarily, the resource reaping behaviors for the LevelDB iterators have been improved by dropping the externalized io.Closer object. Finally, the iterator provisioning methods provide the option for indicating whether one wants a snapshotted iterator or not.	12 years ago
Matt T. Proud	f2a30cf20c	Several important cleanups and deprecations. EachFunc is deprecated. Remove deprecated ``Pair`` and ``GetAll``. These were originally used for forensic and the old gorest impl. Nothing today in the user-facing path nor the tests uses them, especially since the advent of the ForEach protocol in the interface.	12 years ago
Matt T. Proud	8f6b55be71	Several interface cleanups. - Kill Close in Persistent and document interface. - Extract batching behavior into interface. - Kill IteratorManager, which was used for unknown reasons.	12 years ago
Julius Volz	f238b23b04	Set -leveldbFlushOnMutate to false by default.	12 years ago
Matt T. Proud	d5380897c3	Cleanups and adds performance regression.	12 years ago
Matt T. Proud	41068c2e84	Checkpoint.	12 years ago
Julius Volz	d67e4b9131	Address outstanding comments from PR/47 and other cleanups.	12 years ago
Julius Volz	e23ac0adbd	Implement bulk iteration/operation storage interface.	12 years ago
Matt T. Proud	ec3e98b365	Include Snappy in Runtime. Snappy should have been explicitly included in the runtime, for I erroneously thought that LevelDB bundled this into its runtime as-is. It turns out that this assumption is wrong, and I thought we had Snappy compression support all-along.	12 years ago
Matt T. Proud	79ba248bbe	Make LevelDB operating modes invocation-time conf. Presently our use of LevelDB and its operating modes are hardcoded into the storage stack. This pull request decouples this and re-exposes this through flags. We can now perform benchmarking and remedial tuning.	12 years ago
Matt T. Proud	ea54751431	Update import paths to new location. This repository moved from matttproud/prometheus to prometheus/prometheus, and all import paths need to be updated.	12 years ago
Matt T. Proud	3ac5d48b1a	Impl' storage i'faces and fix non-idiomatic warts. This change includes implementation of most major storage layer features, albeit some imperfect. It also includes nascent telemetry bindings, too.	12 years ago
Matt T. Proud	577acf4fe7	Exploding the storage infrastructure by contexts.	12 years ago
Matt T. Proud	15a6681651	Various cleanups. Kill LevelDB watermarks due to redundancy. General interface documentation has begun. Creating custom types for the model to prevent errors down the road. Renaming of components for easier comprehension. Exposition of interface in LevelDB. Slew of simple refactorings.	12 years ago
Matt T. Proud	c1f0d8aefd	Levigo -> LevelDB in terminology and references.	12 years ago
Matt T. Proud	6ed67b02e2	Incorporate new interface tests.	12 years ago
Matt T. Proud	44f8802ae7	Add Apache License 2.0 boilerplate.	12 years ago
Matt T. Proud	2bbdaa5790	Initial directory re-arrangement for storage.	12 years ago

29 Commits (4524a5fe370c06d0374c62d844ac6eb06cc1b164)