Commit Graph

399 Commits (1491fb9370e55266b7788e816faef86f2eb08f93)

Author SHA1 Message Date
Matt T. Proud cbe2f3a7b1 Include LRU cache for fingerprint watermarks. 2013-06-06 10:13:18 +02:00
Julius Volz 51689d965d Add debug timers to instant and range queries.
This adds timers around several query-relevant code blocks. For now, the
query timer stats are only logged for queries initiated through the UI.
In other cases (rule evaluations), the stats are simply thrown away.

My hope is that this helps us understand where queries spend time,
especially in cases where they sometimes hang for unusual amounts of
time.
2013-06-05 18:32:54 +02:00
Matt T. Proud 8339a189cb Code Review: Fix seriesPresent scope.
The seriesPresent scope should be constrained to the scope of a
scanJob, since this is keyed to given series.
2013-06-04 13:16:59 +02:00
Matt T. Proud fe41ce0b19 Conditionalize disk initializations.
This commit conditionalizes the creation of the diskFrontier and
seriesFrontier along with the iterator such that they are provisioned
once something is actually required from disk.
2013-06-04 12:53:57 +02:00
Julius Volz a8468a2e5e Fix reversed disk flush cutoff behavior. 2013-05-28 16:14:30 +02:00
Julius Volz eb1f956909 Revert "Revert "Ensure that all extracted samples are added to view.""
This reverts commit 4b30fb86b4.
2013-05-28 14:36:03 +02:00
Matt T. Proud 4b30fb86b4 Revert "Ensure that all extracted samples are added to view."
This reverts commit 008314b5a8. By
running an automated git bisection described in
https://gist.github.com/matttproud-soundcloud/22a371a8d2cba382ea64
this commit was found.
2013-05-23 13:36:22 +02:00
Julius Volz 750f862d9a Use GetBoundaryValues() for non-counter deltas. 2013-05-22 19:13:47 +02:00
Julius Volz f2b48b8c4a Make getValuesAtIntervalOp consume all chunk data in one pass.
This is mainly a small performance improvement, since we skip past the last
extracted time immediately if it was also the last sample in the chunk, instead
of trying to extract non-existent values before the chunk end again and again
and only gradually approaching the end of the chunk.
2013-05-22 18:14:45 +02:00
Julius Volz 83d60bed89 extractValuesAroundTime() code simplification. 2013-05-22 18:14:45 +02:00
Julius Volz 008314b5a8 Ensure that all extracted samples are added to view.
The current behavior only adds those samples to the view that are extracted by
the last pass of the last processed op and throws other ones away. This is a
bug. We need to append all samples that are extracted by each op pass.

This also makes view.appendSamples() take an array of samples.
2013-05-22 18:14:37 +02:00
Matt T. Proud b586801830 Code Review: Fix to-disk queue infinite growth.
We discovered a bug while manually testing this branch on a live
instance, whereby the to-disk queue was never actually dumped to
disk.
2013-05-22 17:59:53 +02:00
Matt T. Proud 285a8b701b Code Review: Extend lock. 2013-05-22 17:59:53 +02:00
Matt T. Proud 2526ab8c81 Code Review: Extend lock scope for appending. 2013-05-22 17:59:53 +02:00
Matt T. Proud f994482d15 Code Review: Avenues for future improvemnet noted. 2013-05-22 17:59:53 +02:00
Matt T. Proud 298a90c143 Code Review: Initial arena size name. 2013-05-22 17:59:53 +02:00
Matt T. Proud c07abf8521 Initial move away from skiplist. 2013-05-22 17:59:53 +02:00
Matt T. Proud 74a66fd938 Spawn grouping of fingerprints with free semaphore.
The previous implementation spawned N goroutines to group samples
together and would not start work until the semaphore unblocked.
While this didn't leak, it polluted the scheduling space.  Thusly,
the routine only starts after a semaphore has been acquired.
2013-05-21 16:11:35 +02:00
Julius Volz 5b105c77fc Repointerize fingerprints. 2013-05-21 14:28:14 +02:00
Matt T. Proud ec5b5bae28 Fuck you, Travis. 2013-05-21 09:42:00 +02:00
Matt T. Proud e5ac91222b Benchmark memory arena; simplify map generation.
The one-off keys have been replaced with ``model.LabelPair``, which is
indexable.  The performance impact is negligible, but it represents
a cognitive simplification.
2013-05-21 09:39:12 +02:00
juliusv 360477f66c Merge pull request #257 from prometheus/feature/better-memory-behaviors
Pointerize memorySeriesArena.
2013-05-16 07:36:40 -07:00
Matt T. Proud e1f20de2e9 Pointerize memorySeriesArena. 2013-05-16 17:09:28 +03:00
Matt T. Proud 8f4c7ece92 Destroy naked returns in half of corpus.
The use of naked return values is frowned upon.  This is the first
of two bulk updates to remove them.
2013-05-16 10:53:25 +03:00
Matt T. Proud 4e0c932a4f Simplify Encoder's encoding signature.
The reality is that if we ever try to encode a Protocol Buffer and it
fails, it's likely that such an error is ultimately not a runtime error
and should be fixed forthwith.  Thusly, we should rename
``Encoder.Encode`` to ``Encoder.MustEncode`` and drop the error return
value.
2013-05-16 00:54:18 +03:00
juliusv 516101f015 Merge pull request #250 from prometheus/refactor/drop-unused-storage-setting
Drop unused writeMemoryInterval
2013-05-14 08:45:59 -07:00
juliusv 9ff00b651d Merge pull request #251 from prometheus/fix/memory-metric-mutability
Fix GetMetricForFingerprint() metric mutability.
2013-05-14 08:12:45 -07:00
Bernerd Schaefer 63d9988b9c Drop unused writeMemoryInterval 2013-05-14 17:03:03 +02:00
Julius Volz 83c60ad43a Fix GetMetricForFingerprint() metric mutability.
Some users of GetMetricForFingerprint() end up modifying the returned metric
labelset. Since the memory storage's implementation of
GetMetricForFingerprint() returned a pointer to the metric (and maps are
reference types anyways), the external mutation propagated back into the memory
storage.

The fix is to make a copy of the metric before returning it.
2013-05-14 16:46:30 +02:00
Bernerd Schaefer 428d91c86f Rename test helper files to helpers_test.go
This ensures that these files are properly included only in testing.
2013-05-14 16:30:47 +02:00
juliusv 98e512d755 Merge pull request #246 from prometheus/fix/interval-value-extraction
Fix and optimize getValuesAtIntervalOp data extraction.
2013-05-14 05:55:22 -07:00
Julius Volz 71a3172abb Fix and optimize getValuesAtIntervalOp data extraction.
- only the data extracted in the last loop iteration of ExtractSamples() was
  emitted as output
- if e.g. op interval < sample interval, there were situations where the same
  sample was added multiple times to the output
2013-05-14 13:55:17 +02:00
Matt T. Proud 244a4a9cdb Update to go1.1.
This commit updates the documentation, Makefiles, formatting, and
code semantics to support the 1.1. runtime, which includes ...

1. ``make advice``,

2. ``make format``, and

3. ``go fix`` on various targets.
2013-05-14 12:39:08 +02:00
Matt T. Proud b224251981 Simplify compaction and expose database sizes.
This commit simplifies the way that compactions across a database's
keyspace occur due to reading the LevelDB internals. Secondarily it
introduces the database size estimation mechanisms.

Include database health and help interfaces.

Add database statistics; remove status goroutines.

This commit kills the use of Go routines to expose status throughout
the web components of Prometheus. It also dumps raw LevelDB status
on a separate /databases endpoint.
2013-05-14 12:29:53 +02:00
juliusv 92ad65ff13 Merge pull request #232 from prometheus/optimize/granular-storage-locking
Synchronous memory appends and more fine-grained storage locks.
2013-05-13 10:11:57 -07:00
Matt T. Proud 1f7f89b4e3 Simplify compaction and expose database sizes.
This commit simplifies the way that compactions across a database's
keyspace occur due to reading the LevelDB internals.  Secondarily it
introduces the database size estimation mechanisms.
2013-05-13 13:15:35 +02:00
Matt T. Proud d538b0382f Include long-tail data deletion mechanism.
This commit introduces the long-tail deletion mechanism, which will
automatically cull old sample values.  It is an acceptable
hold-over until we get a resampling pipeline implemented.

Kill legacy OS X documentation, too.
2013-05-13 10:54:36 +02:00
Julius Volz ce1ee444f1 Synchronous memory appends and more fine-grained storage locks.
This does two things:

1) Make TieredStorage.AppendSamples() write directly to memory instead of
   buffering to a channel first. This is needed in cases where a rule might
   immediately need the data generated by a previous rule.

2) Replace the single storage mutex by two new ones:
   - memoryMutex - needs to be locked at any time that two concurrent
                   goroutines could be accessing (via read or write) the
                   TieredStorage memoryArena.
   - memoryDeleteMutex - used to prevent any deletion of samples from
                         memoryArena as long as renderView is running and
                         assembling data from it.
   The LevelDB disk storage does not need to be protected by a mutex when
   rendering a view since renderView works off a LevelDB snapshot.

The rationale against adding memoryMutex directly to the memory storage: taking
a mutex does come with a small inherent time cost, and taking it is only
required in few places. In fact, no locking is required for the memory storage
instance which is part of a view (and not the TieredStorage).
2013-05-10 17:15:52 +02:00
Matt T. Proud fa6a1f97d0 Expose interfaces for pruner and make pruner tool.
In order to run database cleanups and diagnostics, we should have
a means for pruning a database---even if LevelDB does this for us.
2013-05-10 17:07:03 +02:00
Matt T. Proud 161c8fbf9b Include deletion processor for long-tail values.
This commit extracts the model.Values truncation behavior into the actual
tiered storage, which uses it and behaves in a peculiar way—notably the
retention of previous elements if the chunk were to ever go empty.  This is
done to enable interpolation between sparse sample values in the evaluation
cycle.  Nothing necessarily new here—just an extraction.

Now, the model.Values TruncateBefore functionality would do what a user
would expect without any surprises, which is required for the
DeletionProcessor, which may decide to split a large chunk in two if it
determines that the chunk contains the cut-off time.
2013-05-10 12:19:12 +02:00
Matt Proud 7f0d816574 Schedule the background compactors to run.
This commit introduces three background compactors, which compact
sparse samples together.

1. Older than five minutes is grouped together into chunks of 50 every 30
   minutes.

2. Older than 60 minutes is grouped together into chunks of 250 every 50
   minutes.

3. Older than one day is grouped together into chunks of 5000 every 70
   minutes.
2013-05-07 17:14:04 +02:00
Julius Volz caab131ada Repointerize TieredStorage method receiver types. 2013-05-07 15:12:33 +02:00
juliusv 89de116ea9 Merge pull request #225 from prometheus/refactor/fmt-cleanups
Slice expression simplifications.
2013-05-07 04:27:27 -07:00
Julius Volz 05afa970d2 Slice expression simplifications. 2013-05-07 13:22:29 +02:00
Matt T. Proud f897164bcf Expose TieredStorage.DiskStorage. 2013-05-07 10:26:28 +02:00
Matt T. Proud ce45787dbf Storage interface to TieredStorage.
This commit drops the Storage interface and just replaces it with a
publicized TieredStorage type.  Storage had been anticipated to be
used as a wrapper for testability but just was not used due to
practicality.  Merely overengineered.  My bad.  Anyway, we will
eventually instantiate the TieredStorage dependencies in main.go and
pass them in for more intelligent lifecycle management.

These changes will pave the way for managing the curators without
Law of Demeter violations.
2013-05-03 15:54:14 +02:00
Bernerd Schaefer 5eb9840ed7 Fix goroutine leak in leveldb.AppendSamples
The error channels in AppendSamples need to be buffered, since in the
presence of errors their values may not be consumed.
2013-05-03 12:13:05 +02:00
Matt T. Proud a3f1d81e24 Publicize a few storage components for curation.
This commit introduces the publicization of Stop and other
components, which the compaction curator shall take advantage
of.
2013-05-02 13:16:04 +02:00
Matt T. Proud 4298bab2b0 Publicize Curator and Processors.
This commit publicizes the curation and processor frameworks for
purposes of making them available in the main processor loop.
2013-05-02 12:37:24 +02:00
Julius Volz 368a792dd2 Adjust memory queue size after change to send arrays over channel. 2013-04-30 13:41:04 +02:00
juliusv b02debd69c Merge pull request #205 from prometheus/julius-channel-arrays
Send sample arrays instead of single samples over channels.
2013-04-29 09:05:05 -07:00
Julius Volz d8110fcd9c Send sample arrays instead of single samples over channels. 2013-04-29 17:24:17 +02:00
Matt T. Proud 3362bf36e2 Include curator status in web heads-up-display. 2013-04-29 12:40:33 +02:00
Matt T. Proud 6fac20c8af Harden the tests against OOMs.
This commit employs explicit memory freeing for the in-memory storage
arenas.  Secondarily, we take advantage of smaller channel buffer sizes
in the test.
2013-04-29 11:46:01 +02:00
Matt T. Proud 66bc3711ea Merge pull request #197 from prometheus/feature/storage/curation-table
Add curation remark table and refactor error mgmt.
2013-04-29 01:01:33 -07:00
Matt T. Proud d46cd089b5 Merge pull request #199 from prometheus/refactor/telemetry/api-refresh
Refresh Prometheus client API usage.
2013-04-28 22:50:30 -07:00
Matt T. Proud 3fa260f180 Complete sentence. 2013-04-28 20:26:44 +02:00
Matt T. Proud e527941b6a Use tagged struct fields. 2013-04-28 20:09:30 +02:00
Matt T. Proud a48ab34dd0 Refresh Prometheus client API usage.
The client API has been updated per https://github.com/prometheus/client_golang/pull/9.
2013-04-28 19:40:30 +02:00
Matt T. Proud 561974308d Add curation remark table and refactor error mgmt.
The curator requires the existence of a curator remark table, which
stores the progress for a given curation policy.  The tests for the
curator create an ad hoc table, but core Prometheus presently lacks
said table, which this commit adds.

Secondarily, the error handling for the LevelDB lifecycle functions
in the metric persistence have been wrapped into an UncertaintyGroup,
which mirrors some of the functions of sync.WaitGroup but adds error
capturing capability to the mix.
2013-04-28 17:26:34 +02:00
Matt T. Proud b3e34c6658 Implement batch database sample curator.
This commit introduces to Prometheus a batch database sample curator,
which corroborates the high watermarks for sample series against the
curation watermark table to see whether a curator of a given type
needs to be run.

The curator is an abstract executor, which runs various curation
strategies across the database.  It remarks the progress for each
type of curation processor that runs for a given sample series.

A curation procesor is responsible for effectuating the underlying
batch changes that are request.  In this commit, we introduce the
CompactionProcessor, which takes several bits of runtime metadata and
combine sparse sample entries in the database together to form larger
groups.  For instance, for a given series it would be possible to
have the curator effectuate the following grouping:

- Samples Older than Two Weeks: Grouped into Bunches of 10000
- Samples Older than One Week: Grouped into Bunches of 1000
- Samples Older than One Day: Grouped into Bunches of 100
- Samples Older than One Hour: Grouped into Bunches of 10

The benefits hereof of such a compaction are 1. a smaller search
space in the database keyspace, 2. better employment of compression
for repetious values, and 3. reduced seek times.
2013-04-27 17:38:18 +02:00
Julius Volz 2202cd71c9 Track alerts over time and write out alert timeseries. 2013-04-26 14:35:21 +02:00
Johannes 'fish' Ziemke 1ad41d4c00 Call closer.Close() earlier. 2013-04-25 13:29:28 +02:00
Johannes 'fish' Ziemke 22da76e8ab Close of reportTicker to exit goroutine. 2013-04-25 13:29:22 +02:00
Johannes 'fish' Ziemke 5043c6fce7 Have goroutine exit on signal via defer block. 2013-04-25 12:14:38 +02:00
juliusv af7ddc36e2 Merge pull request #176 from prometheus/optimization/view-materialization/slice-chunking
Truncate irrelevant chunk values.
2013-04-24 05:19:54 -07:00
Julius Volz 9b8c671ec9 Fixes/cleanups to renderView() samples truncation. 2013-04-24 12:42:58 +02:00
Matt T. Proud 05504d3642 WIP - Truncate irrelevant chunk values.
This does not work with the view tests.
2013-04-24 11:07:22 +02:00
Matt T. Proud a32602140e Convert the TestInstant value into UTC.
For the forthcoming Curator, we don't record timezone information in
the samples, nor do we in the curation remarks.  All times are
recorded UTC.  That said, for the test environment to better match
production, the special instant should be in UTC.
2013-04-23 18:58:39 +02:00
Matt T. Proud b1a8e51b07 Extract dto.SampleValueSeries into model.Values. 2013-04-22 13:31:11 +02:00
Matt T. Proud 422003da8e Convert trailing float64s. 2013-04-21 20:52:21 +02:00
Matt T. Proud db4ffbb262 Wrap dto.SampleKey with business logic type.
The curator work can be done easier if dto.SampleKey is no longer
directly accessed but rather has a higher level type around it that
captures a certain modicum of business logic.  This doesn't look
terribly interesting today, but it will get more so.
2013-04-21 20:38:39 +02:00
Matt T. Proud f9e99bd08a Refresh SampleValue to 64-bit floating point.
We always knew that this needed to be fixed.
2013-04-21 20:31:50 +02:00
Matt T. Proud 092c7bd88e Stochastic test support plural SampleValueSeries.
After SampleValue was refactored into SampleValueSeries, which
involves plural values under a common super key, the stochastic
test was never refreshed to reflect this reality.  We had other
tests that validated the functionality, but this one was
insufficently forward-ported.
2013-04-21 20:31:32 +02:00
Julius Volz 99dcbe0f94 Integrate memory and disk layers in view rendering. 2013-04-19 16:01:27 +02:00
Julius Volz 63625bd244 Make view use memory persistence, remove obsolete code.
This makes the memory persistence the backing store for views and
adjusts the MetricPersistence interface accordingly. It also removes
unused Get* method implementations from the LevelDB persistence so they
don't need to be adapted to the new interface. In the future, we should
rethink these interfaces.

All staleness and interpolation handling is now removed from the storage
layer and will be handled only by the query layer in the future.
2013-04-18 22:26:29 +02:00
Matt T. Proud d468271e2f Fix append queue telemetry and parameterize sizes.
The original append queue telemetry never worked, because it was
updated only upon the exit of the select statement, which would
usually liberate the queues of contents.  This has been fixed to
be reported arbitrarily.

The queue sizes are now parameterizable via flags.
2013-04-16 17:13:29 +02:00
Julius Volz 95b081f9bc Stop serving tiered storage after draining it. 2013-04-15 13:30:03 +02:00
Matt T. Proud a55602df4a Validate diskFrontier domain for series candidate.
It is the case with the benchmark tool that we thought that we
generated multiple series and saved them to the disk as such, when
in reality, we overwrote the fields of the outgoing metrics via
Go map reference behavior.  This was accidental.  In the course of
diagnosing this, a few errors were found:

1. ``newSeriesFrontier`` should check to see if the candidate fingerprint is within the given domain of the ``diskFrontier``.  If not, as the contract in the docstring stipulates, a ``nil`` ``seriesFrontier`` should be emitted.

2. In the interests of aiding debugging, the raw LevelDB ``levigoIterator`` type now includes a helpful forensics ``String()`` method.

This work produced additional cleanups:

1. ``Close() error`` with the storage stack is technically incorrect, since nowhere in the bowels of it does an error actually occur.  The interface has been simplified to remove this for now.
2013-04-09 11:47:16 +02:00
Matt T. Proud d79c932a8e Merge pull request #120 from prometheus/feature/storage/compaction
Spin up curator run in the tests.
2013-04-05 04:55:59 -07:00
Matt T. Proud c3e3460ca6 Spin up curator run in the tests.
After this commit, we'll need to add validations that it does the
desired work, which we presently know that it doesn't.  Given the
changes I made with a plethora of renamings, I want to commit this
now before it gets even larger.
2013-04-05 13:55:11 +02:00
Matt T. Proud 461da0b3a8 Merge pull request #117 from prometheus/feature/storage/compaction
Spin up storage layers for made fixtures.
2013-04-03 04:41:52 -07:00
Matt T. Proud d0ad6cbeaa Spin up storage layers for made fixtures. 2013-04-03 12:09:05 +02:00
Julius Volz c59f3fc538 Fix formatting in tiered_test.go. 2013-03-28 12:16:31 +01:00
juliusv 39826d7335 Merge pull request #107 from prometheus/julius-fix-get-fingerprints
Fix bug in GetFingerprintsForLabelSet().
2013-03-27 10:54:17 -07:00
Julius Volz 2668700e54 Fix bug in GetFingerprintsForLabelSet(). 2013-03-27 18:50:30 +01:00
Matt T. Proud c53a72a894 Test data for the curator. 2013-03-27 18:13:43 +01:00
Matt T. Proud 6dcaa28806 Include LevelDB fixture generators for curator.
This will help reduce common boilerplate for our test process
with respect to LevelDB-related things.
2013-03-27 15:13:40 +01:00
Julius Volz 55ca65aa6e More userfriendly output when we fail to create the tiered storage. 2013-03-27 11:25:05 +01:00
Matt T. Proud c4e971d7d9 Merge pull request #101 from prometheus/refactor/test/directory-extraction
Create temporary directory handler.
2013-03-26 10:46:28 -07:00
Matt T. Proud b86b0ea41a Create temporary directory handler. 2013-03-26 18:09:25 +01:00
Julius Volz 8cf2af3923 Abort view job processing on timeout. 2013-03-26 17:18:51 +01:00
Julius Volz 2b8f0b2cc7 Constantize metric name label name. 2013-03-26 16:20:23 +01:00
Julius Volz e096896932 PR comment fixups. 2013-03-26 15:28:00 +01:00
Julius Volz dd67ab115b Change GetAllMetricNames() to GetAllValuesForLabel(). 2013-03-26 14:47:07 +01:00
Julius Volz 42bdf921d1 Fetch integrated memory/disk data for simple Get* functions. 2013-03-26 14:47:07 +01:00
Julius Volz 11bb94a7e5 Implement GetAllMetricNames() for memory storage. 2013-03-26 14:47:07 +01:00
Julius Volz 991dc68d78 Rename misnamed oldestSampleTimestamp variable. 2013-03-26 11:56:10 +01:00
Matt T. Proud 3e97a3630d Include nascent curator scaffolding.
The curator doesn't do anything yet; rather, this is the type
definition including the anciliary testing scaffold.

Improve Makefile and Git developer experience.

The top-level Makefile was a bit overloaded in terms of generation of
assets and their management.  This has been offloaded into separate
Makefiles.

The Git developer experience sucked due to lack of .gitignore
policies.

Also: Fix faulty skiplist naming from old merge.
2013-03-25 19:38:14 +01:00
Matt T. Proud b2e4c88b80 Wrap LevelDB iterator operations behind interface.
The LevelDB storage types return an interface type now that wraps
around the underlying iterator.  This both enhances testability but
improves upon, in my opinion, the interface design for the LevelDB
iterator.

Secondarily, the resource reaping behaviors for the LevelDB iterators
have been improved by dropping the externalized io.Closer object.

Finally, the iterator provisioning methods provide the option for
indicating whether one wants a snapshotted iterator or not.
2013-03-25 12:57:58 +01:00
Matt T. Proud f2a30cf20c Several important cleanups and deprecations.
EachFunc is deprecated.

Remove deprecated ``Pair`` and ``GetAll``.

These were originally used for forensic and the old gorest impl.
Nothing today in the user-facing path nor the tests uses them,
especially since the advent of the ForEach protocol in the
interface.
2013-03-25 08:38:21 +01:00
Matt T. Proud 70448711ec Merge pull request #95 from prometheus/feature/persistence/batching
Several interface cleanups.
2013-03-24 00:19:46 -07:00
Matt T. Proud 8f6b55be71 Several interface cleanups.
- Kill Close in Persistent and document interface.
 - Extract batching behavior into interface.
 - Kill IteratorManager, which was used for unknown reasons.
2013-03-24 07:35:43 +01:00
Julius Volz a33d2726bc Mark range op as consumed if it receives no data points in range. 2013-03-22 11:50:02 +01:00
Julius Volz 3c9d6cb66c Add several needed persistence proxy methods to tiered storage. 2013-03-21 18:16:43 +01:00
Julius Volz 081d250929 Fix view's GetRangeValues() reverse iteration behavior. 2013-03-21 18:16:31 +01:00
Julius Volz 0be0aa59c2 Wait until storage is drained before closing the underlying leveldb. 2013-03-21 18:16:07 +01:00
Julius Volz becc278eb6 Fix two bugs in range op time advancement. 2013-03-21 18:15:52 +01:00
Matt T. Proud ceb6611957 Fix regression in subsequent range op. compactions.
We have an anomaly whereby subsequent range operations fail to be
compacted into one single range operation.  This fixes such
behavior.
2013-03-21 18:11:04 +01:00
Matt T. Proud 669abdfefe ``make format`` invocation. 2013-03-21 18:11:04 +01:00
Julius Volz bdb067b47f Implement remaining View Get* methods. 2013-03-21 18:11:04 +01:00
Julius Volz 1f42364733 Fix typo in comment. 2013-03-21 18:11:03 +01:00
Matt T. Proud 758a3f0764 Add documentation and cull junk. 2013-03-21 18:11:03 +01:00
Matt T. Proud bd8bb0edfd One additional reduction. 2013-03-21 18:11:03 +01:00
Matt T. Proud 73b463e814 Additional simplifications. 2013-03-21 18:11:03 +01:00
Matt T. Proud fd47ac570f Implied simplifications. 2013-03-21 18:11:03 +01:00
Matt T. Proud 51a0f21cf8 Interim documentation 2013-03-21 18:11:03 +01:00
Matt T. Proud b470f925b7 Extract rewriting of interval queries. 2013-03-21 18:11:03 +01:00
Matt T. Proud eb721fd220 Include note about greediest range. 2013-03-21 18:11:03 +01:00
Julius Volz e50de005f9 Populate metric in SampleSet returned from GetRangeValues() 2013-03-21 18:11:03 +01:00
Julius Volz 6001d22f87 Change Get* methods to receive fingerprints instead of metrics. 2013-03-21 18:11:03 +01:00
Julius Volz 95f8885c8a Adopt new ops sorting interface in view rendering. 2013-03-21 18:11:02 +01:00
Julius Volz 4d79dc3602 Replace renderView() by cleaner and more correct reimplementation. 2013-03-21 18:11:02 +01:00
Julius Volz e0dbc8c561 Fix edge cases in data extraction for point and interval ops. 2013-03-21 18:11:02 +01:00
Julius Volz a4361e4116 Rename extractSampleValue -> extractSampleValues. 2013-03-21 18:08:49 +01:00
Julius Volz 4e7db57e76 Fix iterator behavior in view.GetSampleAtTime() 2013-03-21 18:08:49 +01:00
Julius Volz bb9c5ed7aa Fix nil pointer exception in frontier building. 2013-03-21 18:08:48 +01:00
Matt T. Proud 896e172463 Extract time group optimizations. 2013-03-21 18:08:48 +01:00
Matt T. Proud 5a71814778 Additional greediness. 2013-03-21 18:08:48 +01:00
Matt T. Proud b00ca7e422 Refactor some greediness computations. 2013-03-21 18:08:48 +01:00
Matt T. Proud 978acd4e96 Simplify time group optimizations.
The old code performed well according to the benchmarks, but the
new code shaves 1/6th of the time off the original and with less
code.
2013-03-21 18:08:48 +01:00
Matt T. Proud d7b534e624 Update documentation. 2013-03-21 18:08:48 +01:00
Matt T. Proud 1f7ed52b46 Start writing high watermarks. 2013-03-21 18:08:48 +01:00
Julius Volz a224dda9f0 Fix diskFrontier.ContainsFingerprint() return value. 2013-03-21 18:08:48 +01:00
Matt T. Proud 47ce7ad302 Extract appending from goroutine. 2013-03-21 18:08:48 +01:00
Matt T. Proud 187cd4cdbc Extract indexing of Fingerprint to Metrics. 2013-03-21 18:08:48 +01:00
Matt T. Proud 532589f728 Extract Label Pair to Fingerprint indexing. 2013-03-21 18:08:48 +01:00
Matt T. Proud 84acfed061 Extract finding unindexed metrics. 2013-03-21 18:08:48 +01:00
Matt T. Proud 67300af137 Extract indexing to separate routine. 2013-03-21 18:08:47 +01:00
Matt T. Proud 582354f6de Fix remaining ``make advice`` issues. 2013-03-21 18:08:47 +01:00
Matt T. Proud 615e6d13d7 Run ``make format``. 2013-03-21 18:08:47 +01:00
Julius Volz caeb759ed7 Add tests for and fix getValuesAlongRangeOp value extraction. 2013-03-21 18:08:47 +01:00
Julius Volz 69a24427b7 Minor tiered storage fixups. 2013-03-21 18:08:47 +01:00
Julius Volz 3621148e7f Comment out panicking test until proper support is implemented. 2013-03-21 18:08:47 +01:00
Julius Volz e2fb497eba Add operator value extraction tests. 2013-03-21 18:08:47 +01:00
Julius Volz 12a8863582 Add data extraction methods to operator types. 2013-03-21 18:08:47 +01:00
Julius Volz 1d5df867d1 Set test time to fixed value. 2013-03-21 18:08:47 +01:00
Julius Volz 2f06b8bea6 Fix tiered storage test to trigger iterator rewinding case. 2013-03-21 18:08:47 +01:00
Julius Volz 894ecfe161 Small cleanups and comments in tiered storage. 2013-03-21 18:08:47 +01:00
Julius Volz f238b23b04 Set -leveldbFlushOnMutate to false by default. 2013-03-21 18:08:47 +01:00
Julius Volz 8939e0723a Make LevelDB chunk size a flag. 2013-03-21 18:08:46 +01:00
Julius Volz ce4f560e48 Encapsulate fingerprint frontier checks in renderView(). 2013-03-21 18:08:46 +01:00
Julius Volz 1a1cba1bb2 Address outstanding PR comments. 2013-03-21 18:08:46 +01:00
Matt T. Proud 62b5d7ce20 Oops. 2013-03-21 18:08:46 +01:00
Matt T. Proud 1e0d740f2a Conditionalize LevelDB index retrievals.
The LevelDB index retrievals could be repeated in a given operation
batch if multiple queued mutations affect the same (Label Name) singles
and (Label Name, Label Value) doubles.  This is wasteful and
inefficient, as a single retrieval suffices.  Thusly this commit
retrieves the canonical index mappings if the said mapping has not
been looked up in a given batch.
2013-03-21 18:08:46 +01:00
Matt T. Proud 34a921e16d Checkpoint. 2013-03-21 18:08:46 +01:00
Matt T. Proud 8cc5cdde0b checkpoint. 2013-03-21 18:08:46 +01:00
Matt T. Proud d5380897c3 Cleanups and adds performance regression. 2013-03-21 18:06:51 +01:00
Matt T. Proud f39b9c3c8e Checkpoint. 2013-03-21 18:06:51 +01:00
Matt T. Proud 41068c2e84 Checkpoint. 2013-03-21 18:06:51 +01:00
Matt T. Proud 13ae29b304 Initial in-memory arena implementation.
It is unbounded, and nothing uses it except for a gating flag in main.
2013-02-18 09:38:14 -06:00
Matt T. Proud efbe0e8a12 Interface simplification.
GetMetricForFingerprint(model.Fingerprint) (*Metric, error) ->
GetMetricForFingerprint(model.Fingerprint) (Metric, error)
2013-02-14 08:43:02 -08:00
Matt T. Proud f1245e8dda Interface simplifications.
GetFingerprintsForLabelName ([]*Fingerprint, error) ->
GetFingerprintsForLabelName ([]Fingerprint, error)
2013-02-14 08:19:41 -08:00
Matt T. Proud e8a733b525 Interface simplifications.
GetFingerprintsForLabelSet ([]*Fingerprint, error) ->
GetFingerprintsForLabelSet ([]Fingerprint, error)
2013-02-14 08:07:59 -08:00
Matt T. Proud f03091b139 Interface simplifications: GetRangeValues
From pointers to copies.
2013-02-13 21:11:23 -08:00
Matt T. Proud 14788cf4f3 Interface simplifications.
GetBoundaryValues() from pointers to values.
2013-02-13 21:07:42 -08:00
Matt T. Proud 56f069b3ec Interface simplifications: GetValueAtTime().
Pointer arguments to copies.
2013-02-13 21:05:01 -08:00
Matt T. Proud 1e2d6c7418 GetFingerprintsForLabelName simplifications.
``MetricPersistence.GetFingerprintsForLabelName(l *model.LabelName)``
->
``MetricPersistence.GetFingerprintsForLabelName(l model.LabelName)``
2013-02-13 17:44:35 -08:00
Matt T. Proud 900bb988c1 Simplifications of GetFingerprintsForLabelSet.
``MetricPersistence.GetFingerprintsForLabelSet(s *model.LabelSet)`` ->
``MetricPersistence.GetFingerprintsForLabelSet(s model.LabelSet)``.
2013-02-13 17:13:41 -08:00
Matt T. Proud 4fbcea73f5 MetricPersistence.AppendSample signature changes.
``MetricPersistence.AppendSample(*model.Sample)`` -> ``MetricPersistence.AppendSample(model.Sample)``.
2013-02-13 13:46:28 -08:00
Matt T. Proud 4502b49524 Swap out fingerprinting infrastructure.
All old database entries should be deleted.  :-(
2013-02-08 15:58:53 +01:00
Julius Volz d67e4b9131 Address outstanding comments from PR/47 and other cleanups. 2013-02-07 11:38:01 +01:00
Julius Volz 2659304380 Add interface for getting all metrics from a MetricsPersistence. 2013-02-06 17:06:39 +01:00
Julius Volz e23ac0adbd Implement bulk iteration/operation storage interface. 2013-02-06 17:05:23 +01:00
Matt T. Proud ec3e98b365 Include Snappy in Runtime.
Snappy should have been explicitly included in the runtime, for I
erroneously thought that LevelDB bundled this into its runtime as-is.
It turns out that this assumption is wrong, and I thought we had
Snappy compression support all-along.
2013-02-01 13:35:07 +01:00
Matt T. Proud 79ba248bbe Make LevelDB operating modes invocation-time conf.
Presently our use of LevelDB and its operating modes are hardcoded
into the storage stack.  This pull request decouples this and
re-exposes this through flags.  We can now perform benchmarking
and remedial tuning.
2013-01-27 20:28:37 +01:00
Matt T. Proud ea54751431 Update import paths to new location.
This repository moved from matttproud/prometheus to
prometheus/prometheus, and all import paths need to be updated.
2013-01-27 18:49:45 +01:00
Matt T. Proud f2ded515b7 Support versioned telemetry providers.
client_golang was updated to support full label-oriented telemetry,
which introduced interface incompatibilities with the previous
version of Prometheus.  To alleviate this, a general fetching and
processing dispatching system has been created, which discriminates
and processes according to the version of input.
2013-01-27 17:45:50 +01:00
Matt T. Proud 88d15373c5 Upgrade Prometheus to new API. 2013-01-23 17:18:45 +01:00
Julius Volz 6f937ee032 Add preliminary metric docstrings and empty base labels. 2013-01-18 01:37:50 +01:00
Matt T. Proud d9b165220e ``GetFingerprintsForLabelSet`` uses intersection.
This implementation needs to be cleaned up considerably, but it
should hopefully provide some breathing room.  Ultimately this will
be extracted out of the storage layer---specifically the LevelDB
hierarchy---to interface with a generic set of abstractions for a
storage system, but this will be where it remains for now.
2013-01-13 17:14:30 +01:00
Matt T. Proud ac438e51f6 Debug corner-case seeking anomaly. 2013-01-07 11:23:09 +01:00
Matt T. Proud be9b7942c1 Fix Stale. Pol. support in GetValueAtTime. 2013-01-06 18:36:05 +01:00
Matt T. Proud 52f52a7ee2 Include nascent instrumentation of stack. 2013-01-04 23:32:46 +01:00
Matt T. Proud 4f7adbbe7d Reduce work level for stochastic tests.
The LevelDB stochastic tests could take a long time.

Original:
```
Matt:prometheus mtp$ make test
make -C model
protoc --go_out=generated/ data.proto
go build ./...
go test ./...
warning: building out-of-date packages:
	github.com/matttproud/golang_instrumentation/maths
	github.com/matttproud/golang_instrumentation/utility
	github.com/matttproud/golang_instrumentation/metrics
	github.com/matttproud/golang_instrumentation
installing these packages with 'go test -i ./...' will speed future tests.

?   	github.com/matttproud/prometheus	[no test files]
?   	github.com/matttproud/prometheus/coding	[no test files]
ok  	github.com/matttproud/prometheus/coding/indexable	0.012s
?   	github.com/matttproud/prometheus/model	[no test files]
?   	github.com/matttproud/prometheus/model/generated	[no test files]
?   	github.com/matttproud/prometheus/retrieval	[no test files]
?   	github.com/matttproud/prometheus/storage/metric	[no test files]
ok  	github.com/matttproud/prometheus/storage/metric/leveldb	70.800s
?   	github.com/matttproud/prometheus/storage/raw	[no test files]
?   	github.com/matttproud/prometheus/storage/raw/index	[no test files]
ok  	github.com/matttproud/prometheus/storage/raw/index/leveldb	0.012s
ok  	github.com/matttproud/prometheus/storage/raw/leveldb	0.016s
ok  	github.com/matttproud/prometheus/utility	0.012s
?   	github.com/matttproud/prometheus/utility/test	[no test files]
```

to

```
Matt:prometheus mtp$ make test
make -C model
protoc --go_out=generated/ data.proto
go build ./...
go test ./...
warning: building out-of-date packages:
	github.com/matttproud/golang_instrumentation/maths
	github.com/matttproud/golang_instrumentation/utility
	github.com/matttproud/golang_instrumentation/metrics
	github.com/matttproud/golang_instrumentation
installing these packages with 'go test -i ./...' will speed future tests.

?   	github.com/matttproud/prometheus	[no test files]
?   	github.com/matttproud/prometheus/coding	[no test files]
ok  	github.com/matttproud/prometheus/coding/indexable	0.011s
?   	github.com/matttproud/prometheus/model	[no test files]
?   	github.com/matttproud/prometheus/model/generated	[no test files]
?   	github.com/matttproud/prometheus/retrieval	[no test files]
?   	github.com/matttproud/prometheus/storage/metric	[no test files]
ok  	github.com/matttproud/prometheus/storage/metric/leveldb	2.158s
?   	github.com/matttproud/prometheus/storage/raw	[no test files]
?   	github.com/matttproud/prometheus/storage/raw/index	[no test files]
ok  	github.com/matttproud/prometheus/storage/raw/index/leveldb	0.013s
ok  	github.com/matttproud/prometheus/storage/raw/leveldb	0.013s
ok  	github.com/matttproud/prometheus/utility	0.013s
?   	github.com/matttproud/prometheus/utility/test	[no test files]
```
2013-01-04 12:27:57 +01:00
Matt T. Proud 3ac5d48b1a Impl' storage i'faces and fix non-idiomatic warts.
This change includes implementation of most major storage layer
features, albeit some imperfect.  It also includes nascent telemetry
bindings, too.
2013-01-04 10:39:38 +01:00
Matt T. Proud a14dbd5bd0 Interim commit for Julius. 2012-12-19 20:34:54 +01:00
Matt T. Proud 0886592ebc New interface definition after discussion. 2012-12-12 12:53:34 +01:00
Matt T. Proud 59a708f25a Provide prototype of storage layer interfaces. 2012-12-12 12:13:27 +01:00
Matt T. Proud 6589fc92f8 Strip web services, which weren't adding value. 2012-12-12 12:04:46 +01:00
Matt T. Proud 577acf4fe7 Exploding the storage infrastructure by contexts. 2012-12-09 16:27:12 +01:00
Matt T. Proud 15a6681651 Various cleanups.
Kill LevelDB watermarks due to redundancy.

General interface documentation has begun.

Creating custom types for the model to prevent errors down the
road.

Renaming of components for easier comprehension.

Exposition of interface in LevelDB.

Slew of simple refactorings.
2012-12-08 14:03:08 +01:00
Matt T. Proud c0ce859c76 A few re-organizations. 2012-11-29 20:55:30 +01:00
Matt T. Proud 044a5b4e14 Add nascent Travis CI configuration. 2012-11-29 20:00:09 +01:00
Matt T. Proud 9f4bdaab50 Major LevelDB metric end-to-end test improvements.
Performance is enhanced through better range selection.
2012-11-28 22:48:28 +01:00
Matt T. Proud c1f0d8aefd Levigo -> LevelDB in terminology and references. 2012-11-28 20:25:19 +01:00
Matt T. Proud 6ed67b02e2 Incorporate new interface tests. 2012-11-28 18:52:04 +01:00
Matt T. Proud 44f8802ae7 Add Apache License 2.0 boilerplate. 2012-11-26 20:11:34 +01:00
Matt T. Proud 2bbdaa5790 Initial directory re-arrangement for storage. 2012-11-26 19:56:51 +01:00