prometheus

Commit Graph

Author	SHA1	Message	Date
beorn7	d121db7a65	federate: Fix PeekBack usage In most cases, there is no sample at `maxt`, so `PeekBack` has to be used. So far, `PeekBack` did not return a float histogram, and we disregarded even any returned normal histogram. This fixes both, and also tweaks the unit test to discover the problem (by using an earlier timestamp than "now" for the samples in the TSDB). Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Jesus Vazquez	e934d0f011	Merge 'main' into sparsehistogram Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>	2 years ago
Bryan Boreham	8120af22e2	benchmarks: SetBytes takes bytes per operation Where the code was multiplying bytes by number of operations, this resulted in absurdly high throughput numbers. Also, in `BenchmarkParse()`, don't run the `expfmt` case twice. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
beorn7	6f33ab2b35	Merge branch 'main' into sparsehistogram	3 years ago
detailyang	3e482c905f	fix:storage:avoid panic when iterater exhauested (#9945 ) Signed-off-by: detailyang <detailyang@gmail.com>	3 years ago
Björn Rabenstein	4ce01e9770	storage: Rename ...Values methods to At... (#9889 ) This mirrors #9888 for the richer iterators we have with histograms in the game. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Björn Rabenstein	d677aa4b29	storage: Consolidate iterator method names (Values -> At) (#9888 ) `BufferedSeriesIterator` and `MemoizedSeriesIterator` use a method called `Values` for exactly the purpose for which all other iterators of the same kind use a method called `At`. That alone is confusing, but on top of that, the `Values` method only returns a single sample, not multiple values. I assume the naming has historical reasons. This commit makes it more consistent. It is now easier to read, and now `BufferedSeriesIterator` and `MemoizedSeriesIterator` implement `chunkenc.Iterator` like many other iterators, too. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Björn Rabenstein	7e42acd3b1	tsdb: Rework iterators (#9877 ) - Pick At... method via return value of Next/Seek. - Do not clobber returned buckets. - Add partial FloatHistogram suppert. Note that the promql package is now _only_ dealing with FloatHistograms, following the idea that PromQL only knows float values. As a byproduct, I have removed the histogramSeries metric. In my understanding, series can have both float and histogram samples, so that metric doesn't make sense anymore. As another byproduct, I have converged the sampleBuf and the histogramSampleBuf in memSeries into one. The sample type stored in the sampleBuf has been extended to also contain histograms even before this commit. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	5d4db805ac	Merge branch 'main' into sparsehistogram	3 years ago
beorn7	4c28d9fac7	Move to histogram.Histogram pointers This is to avoid copying the many fields of a histogram.Histogram all the time. This also fixes a bunch of formerly broken tests. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	8f92c90897	Add TODOs and some minor tweaks Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	7a8bb8222c	Style cleanup of all the changes in sparsehistogram so far A lot of this code was hacked together, literally during a hackathon. This commit intends not to change the code substantially, but just make the code obey the usual style practices. A (possibly incomplete) list of areas: * Generally address linter warnings. * The `pgk` directory is deprecated as per dev-summit. No new packages should be added to it. I moved the new `pkg/histogram` package to `model` anticipating what's proposed in #9478. * Make the naming of the Sparse Histogram more consistent. Including abbreviations, there were just too many names for it: SparseHistogram, Histogram, Histo, hist, his, shs, h. The idea is to call it "Histogram" in general. Only add "Sparse" if it is needed to avoid confusion with conventional Histograms (which is rare because the TSDB really has no notion of conventional Histograms). Use abbreviations only in local scope, and then really abbreviate (not just removing three out of seven letters like in "Histo"). This is in the spirit of https://github.com/golang/go/wiki/CodeReviewComments#variable-names * Several other minor name changes. * A lot of formatting of doc comments. For one, following https://github.com/golang/go/wiki/CodeReviewComments#comment-sentences , but also layout question, anticipating how things will look like when rendered by `godoc` (even where `godoc` doesn't render them right now because they are for unexported types or not a doc comment at all but just a normal code comment - consistency is queen!). * Re-enabled `TestQueryLog` and `TestEndopints` (they pass now, leaving them disabled was presumably an oversight). * Bucket iterator for histogram.Histogram is now created with a method. * HistogramChunk.iterator now allows iterator recycling. (I think @dieterbe only commented it out because he was confused by the question in the comment.) * HistogramAppender.Append panics now because we decided to treat staleness marker differently. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Ganesh Vernekar	f4d3af73f0	Query histograms from TSDB and unit test for append+query (#9022 ) Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Ganesh Vernekar	04ad56d9b8	Append sparse histograms into the Head block (#9013 ) * Append sparse histograms into the Head block Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Add AtHistogram() to Iterator interface. Make HistoChunk conform to Chunk interface. Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Marco Pracucci	b92c03023d	Optimized vector selector Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Julien Pivotto	6c56a1faaa	Testify: move to require (#8122 ) * Testify: move to require Moving testify to require to fail tests early in case of errors. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> * More moves Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	1282d1b39c	Refactor test assertions (#8110 ) * Refactor test assertions This pull request gets rid of assert.True where possible to use fine-grained assertions. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	4e5b1722b3	Move away from testutil, refactor imports (#8087 ) Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Bartlomiej Plotka	b788986717	storage: Adjusted fully storage layer support for chunk iterators: Remote read client, readyStorage, fanout. (#7059 ) * Fixed nits introduced by https://github.com/prometheus/prometheus/pull/7334 * Added ChunkQueryable implementation to fanout and readyStorage. * Added more comments. * Changed NewVerticalChunkSeriesMerger to CompactingChunkSeriesMerger, removed tiny interface by reusing VerticalSeriesMergeFunc for overlapping algorithm for both chunks and series, for both querying and compacting (!) + made sure duplicates are merged. * Added ErrChunkSeriesSet * Added Samples interface for seamless []promb.Sample to []tsdbutil.Sample conversion. * Deprecating non chunks serieset based StreamChunkedReadResponses, added chunk one. * Improved tests. * Split remote client into Write (old storage) and read. * Queryable client is now SampleAndChunkQueryable. Since we cannot use nice QueryableFunc I moved all config based options to sampleAndChunkQueryableClient to aboid boilerplate. In next commit: Changes for TSDB. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	4 years ago
Bartlomiej Plotka	d5c33877f9	storage: Added Chunks{Queryable/Querier/SeriesSet/Series/Iteratable. Added generic Merge{SeriesSet/Querier} implementation. (#7005 ) * storage: Added Chunks{Queryable/Querier/SeriesSet/Series/Iteratable. Added generic Merge{SeriesSet/Querier} implementation. ## Rationales: In many places (e.g. chunk Remote read, Thanos Receive fetching chunk from TSDB), we operate on encoded chunks not samples. This means that we unnecessary decode/encode, wasting CPU, time and memory. This PR adds chunk iterator interfaces and makes the merge code to be reused between both seriesSets I will make the use of it in following PR inside tsdb itself. For now fanout implements it and mergers. All merges now also allows passing series mergers. This opens doors for custom deduplications other than TSDB vertical ones (e.g. offline one we have in Thanos). ## Changes * Added Chunk versions of all iterating methods. It all starts in Querier/ChunkQuerier. The plan is that Storage will implement both chunked and samples. * Added Seek to chunks.Iterator interface for iterating over chunks. * NewMergeChunkQuerier was added; Both this and NewMergeQuerier are now using generigMergeQuerier to share the code. Generic code was added. * Improved tests. * Added some TODO for further simplifications in next PRs. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed Brian's comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Moved s/Labeled/SeriesLabels as per Krasi suggestion. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed Krasi's comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Second iteration of Krasi comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Another round of comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Bartlomiej Plotka	34426766d8	Unify Iterator interfaces. All point to storage now. This is part of https://github.com/prometheus/prometheus/pull/5882 that can be done to simplify things. All todos I added will be fixed in follow up PRs. * querier.Querier, querier.Appender, querier.SeriesSet, and querier.Series interfaces merged with storage interface.go. All imports that. * querier.SeriesIterator replaced by chunkenc.Iterator * Added chunkenc.Iterator.Seek method and tests for xor implementation (?) * Since we properly handle SelectParams for Select methods I adjusted min max based on that. This should help in terms of performance for queries with functions like offset. * added Seek to deletedIterator and test. * storage/tsdb was removed as it was only a unnecessary glue with incompatible structs. No logic was changed, only different source of abstractions, so no need for benchmarks. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Joe Elliott	95dc59ec7e	Replaced t.Fatalf() with testutil.Assert() in buffer_test.go (#6084 ) * Added Fatal method and used it in buffer_test Signed-off-by: Joe Elliott <number101010@gmail.com> * Added period to meet contributing guidelines Signed-off-by: Joe Elliott <number101010@gmail.com> * Removed fatal testutil method. Refactored test cases to use testutil.Assert Signed-off-by: Joe Elliott <number101010@gmail.com> * Added if found condition for clarity Signed-off-by: Joe Elliott <number101010@gmail.com>	5 years ago
Chris Marchbanks	529ccff07b	Remove all usages of stretchr/testify Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	5 years ago
Alin Sinpalean	d6adfe2ae2	Use a fake SeriesIterator (that generates samples on the fly instead of using a slice) for BufferedSeriesIterator, to reduce the variance of benchmark results due to memory pressure. (#4847 ) Signed-off-by: Alin Sinpalean <alin.sinpalean@gmail.com>	6 years ago
Alin Sinpalean	e3b775b78b	Simplify BufferedSeriesIterator usage (#4294 ) * Allow for BufferedSeriesIterator instances to be created without an underlying iterator, to simplify their usage. Signed-off-by: Alin Sinpalean <alin.sinpalean@gmail.com>	6 years ago
Brian Brazil	a8c22c85cc	Correctly handle pruning wraparound after ring expansion (#3942 ) Fixes #3939	7 years ago
Tom Wilkie	da29c09dca	Some benchmarks for the mergeSeries set.	7 years ago
Goutham Veeramachaneni	baf5b0f0fc	Fix error where we look into the future. (#2829 ) * Fix error where we look into the future. So currently we are adding values that are in the future for an older timestamp. For example, if we have [(1, 1), (150, 2)] we will end up showing [(1, 1), (2,2)]. Further it is not advisable to call .At() after Next() returns false. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Retuen early if done Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Handle Seek() where we reach the end of iterator Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Simplify code Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Fabian Reinartz	0f3110487d	Merge remote-tracking branch 'origin/dev-2.0' into dev-2.0	8 years ago
Fabian Reinartz	37deb21c45	vendor: remove unused dependency and last ref to fabxc/tsdb	8 years ago
Brian Brazil	5c9a6ce747	Add license to files. This should fix CI for dev-2.0.	8 years ago
Fabian Reinartz	0ecd205794	promql: Use buffer pool for matrix allocations	8 years ago
Fabian Reinartz	bc20d93f0a	storage: rename iterator value getters to At()	8 years ago
Fabian Reinartz	7322c46b8e	storage: add mock iterator for test	8 years ago
Fabian Reinartz	0492ddbd4d	*: fully decouple tsdb, add new storage interfaces	8 years ago

35 Commits (291ab4d0bc6ea9f0e61073f303bd3982e9727b8d)