prometheus

Commit Graph

Author	SHA1	Message	Date
Matthieu MOREL	8f6cf3aabb	tsdb: use Go standard errors Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	12 months ago
Oleg Zaytsev	61daa30bb1	Pass ref to SeriesLifecycleCallback.PostDeletion (#12626 ) When a particular SeriesLifecycleCallback tries to optimize and run closer to the Head, keeping track of the HeadSeriesRef instead of the labelsets, it's impossible to handle the PostDeletion callback properly as there's no way to know which series refs were deleted from the head. This changes the callback to provide the series refs alongside the labelsets, so the implementation can choose what to do. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	1 year ago
Jesus Vazquez	c1b669bf9b	Add out-of-order sample support to the TSDB (#11075 ) * Introduce out-of-order TSDB support This implementation is based on this design doc: https://docs.google.com/document/d/1Kppm7qL9C-BJB1j6yb6-9ObG3AbdZnFUBYPNNWwDBYM/edit?usp=sharing This commit adds support to accept out-of-order ("OOO") sample into the TSDB up to a configurable time allowance. If OOO is enabled, overlapping querying are automatically enabled. Most of the additions have been borrowed from https://github.com/grafana/mimir-prometheus/ Here is the list ist of the original commits cherry picked from mimir-prometheus into this branch: - `4b2198d7ec` - `2836e5513f` - `00b379c3a5` - `ff0dc75758` - `a632c73352` - `c6f3d4ab33` - `5e8406a1d4` - `abde1e0ba1` - `e70e769889` - `df59320886` Co-authored-by: Jesus Vazquez <jesus.vazquez@grafana.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Dieter Plaetinck <dieter@grafana.com> Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * gofumpt files Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Add license header to missing files Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Fix OOO tests due to existing chunk disk mapper implementation Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Fix truncate int overflow Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Add Sync method to the WAL and update tests Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * remove useless sync Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Update minOOOTime after truncating Head * Update minOOOTime after truncating Head Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix lint Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Add a unit test Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Load OutOfOrderTimeWindow only once per appender Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Fix OOO Head LabelValues and PostingsForMatchers Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Fix replay of OOO mmap chunks Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Remove unnecessary err check Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Prevent panic with ApplyConfig Signed-off-by: Ganesh Vernekar 15064823+codesome@users.noreply.github.com Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Run OOO compaction after restart if there is OOO data from WBL Signed-off-by: Ganesh Vernekar 15064823+codesome@users.noreply.github.com Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Apply Bartek's suggestions Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Refactor OOO compaction Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Address comments and TODOs - Added a comment explaining why we need the allow overlapping compaction toggle - Clarified TSDBConfig OutOfOrderTimeWindow doc - Added an owner to all the TODOs in the code Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Run go format Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Fix remaining review comments Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix tests Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Change wbl reference when truncating ooo in TestHeadMinOOOTimeUpdate Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Fix TestWBLAndMmapReplay test failure on windows Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Address most of the feedback Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Refactor the block meta for out of order Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix windows error Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix review comments Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Ganesh Vernekar 15064823+codesome@users.noreply.github.com Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Dieter Plaetinck <dieter@grafana.com> Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>	2 years ago
Eng Zer Jun	3e67654d37	refactor: use `T.TempDir()` and `B.TempDir` to create temporary directory The directory created by `T.TempDir()` and `B.TempDir()` is automatically removed when the test and all its subtests complete. Reference: https://pkg.go.dev/testing#T.TempDir Reference: https://pkg.go.dev/testing#B.TempDir Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	3 years ago
beorn7	c954cd9d1d	Move packages out of deprecated pkg directory This creates a new `model` directory and moves all data-model related packages over there: exemplar labels relabel rulefmt textparse timestamp value All the others are more or less utilities and have been moved to `util`: gate logging modetimevfs pool runtime Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Levi Harrison	7bc11dcb06	React UI: Add Starting Screen (#8662 ) * Added walreplay API endpoint Signed-off-by: Levi Harrison <git@leviharrison.dev> * Added starting page to react-ui Signed-off-by: Levi Harrison <git@leviharrison.dev> * Documented the new endpoint Signed-off-by: Levi Harrison <git@leviharrison.dev> * Fixed typos Signed-off-by: Levi Harrison <git@leviharrison.dev> Co-authored-by: Julius Volz <julius.volz@gmail.com> * Removed logo Signed-off-by: Levi Harrison <git@leviharrison.dev> * Changed isResponding to isUnexpected Signed-off-by: Levi Harrison <git@leviharrison.dev> * Changed width of progress bar Signed-off-by: Levi Harrison <git@leviharrison.dev> * Changed width of progress bar Signed-off-by: Levi Harrison <git@leviharrison.dev> * Added DB stats object Signed-off-by: Levi Harrison <git@leviharrison.dev> * Updated starting page to work with new fields Signed-off-by: Levi Harrison <git@leviharrison.dev> * Passing nil Signed-off-by: Levi Harrison <git@leviharrison.dev> * Passing nil (pt. 2) Signed-off-by: Levi Harrison <git@leviharrison.dev> * Passing nil (pt. 3) Signed-off-by: Levi Harrison <git@leviharrison.dev> * Passing nil (and also implementing a method this time) (pt. 4) Signed-off-by: Levi Harrison <git@leviharrison.dev> * Passing nil (and also implementing a method this time) (pt. 5) Signed-off-by: Levi Harrison <git@leviharrison.dev> * Changed const to let Signed-off-by: Levi Harrison <git@leviharrison.dev> * Passing nil (pt. 6) Signed-off-by: Levi Harrison <git@leviharrison.dev> * Remove SetStats method Signed-off-by: Levi Harrison <git@leviharrison.dev> * Added comma Signed-off-by: Levi Harrison <git@leviharrison.dev> * Changed api Signed-off-by: Levi Harrison <git@leviharrison.dev> * Changed to triple equals Signed-off-by: Levi Harrison <git@leviharrison.dev> * Fixed data response types Signed-off-by: Levi Harrison <git@leviharrison.dev> * Don't return pointer Signed-off-by: Levi Harrison <git@leviharrison.dev> * Changed version Signed-off-by: Levi Harrison <git@leviharrison.dev> * Fixed interface issue Signed-off-by: Levi Harrison <git@leviharrison.dev> * Fixed pointer Signed-off-by: Levi Harrison <git@leviharrison.dev> * Fixed copying lock value error Signed-off-by: Levi Harrison <git@leviharrison.dev> Co-authored-by: Julius Volz <julius.volz@gmail.com>	4 years ago
Marco Pracucci	6248e685b2	TSDB: optimize series creation on PreCreation() failure (#8620 ) Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Dustin Hooten	b9f0baf6ff	Combine NewHead() args into a HeadOptions struct (#8452 ) * Combine NewHead() args into a HeadOptions struct Signed-off-by: Dustin Hooten <dustinhooten@gmail.com> * remove overrides params Signed-off-by: Dustin Hooten <dustinhooten@gmail.com> * address pr feedback Signed-off-by: Dustin Hooten <dustinhooten@gmail.com>	4 years ago
Marco Pracucci	db19e05d93	Add option to customise head chunks write buffer size (#8201 ) * Add option to customise head chunks write buffer size Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed tests Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Julien Pivotto	6c56a1faaa	Testify: move to require (#8122 ) * Testify: move to require Moving testify to require to fail tests early in case of errors. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> * More moves Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	4e5b1722b3	Move away from testutil, refactor imports (#8087 ) Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Javier Palomo Almena	b58a613443	Replace sync/atomic with uber-go/atomic (#7683 ) * storage: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * tsdb: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * web: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * notifier: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * cmd: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * scripts: Verify that we are not using restricted packages It checks that we are not directly importing 'sync/atomic'. Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * Reorganise imports in blocks Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * notifier/test: Apply PR suggestions Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * storage/remote: avoid storing references on newEntry Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * Revert "scripts: Verify that we are not using restricted packages" This reverts commit `278d32748e`. Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * web: Group imports accordingly Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com>	4 years ago
Ganesh Vernekar	1c99adb9fd	Callbacks for lifecycle of series in TSDB (#7159 ) * Callbacks for lifecycle of series in TSDB Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Add more comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Ganesh Vernekar	d4b9fe801f	M-map full chunks of Head from disk (#6679 ) When appending to the head and a chunk is full it is flushed to the disk and m-mapped (memory mapped) to free up memory Prom startup now happens in these stages - Iterate the m-maped chunks from disk and keep a map of series reference to its slice of mmapped chunks. - Iterate the WAL as usual. Whenever we create a new series, look for it's mmapped chunks in the map created before and add it to that series. If a head chunk is corrupted the currpted one and all chunks after that are deleted and the data after the corruption is recovered from the existing WAL which means that a corruption in m-mapped files results in NO data loss. [Mmaped chunks format](https://github.com/prometheus/prometheus/blob/master/tsdb/docs/format/head_chunks.md) - main difference is that the chunk for mmaping now also includes series reference because there is no index for mapping series to chunks. [The block chunks](https://github.com/prometheus/prometheus/blob/master/tsdb/docs/format/chunks.md) are accessed from the index which includes the offsets for the chunks in the chunks file - example - chunks of series ID have offsets 200, 500 etc in the chunk files. In case of mmaped chunks, the offsets are stored in memory and accessed from that. During WAL replay, these offsets are restored by iterating all m-mapped chunks as stated above by matching the series id present in the chunk header and offset of that chunk in that file. Prombench results _WAL Replay_ 1h Wal reply time 30% less wal reply time - 4m31 vs 3m36 2h Wal reply time 20% less wal reply time - 8m16 vs 7m _Memory During WAL Replay_ High Churn: 10-15% less RAM - 32gb vs 28gb 20% less RAM after compaction 34gb vs 27gb No Churn: 20-30% less RAM - 23gb vs 18gb 40% less RAM after compaction 32.5gb vs 20gb Screenshots are in [this comment](https://github.com/prometheus/prometheus/pull/6679#issuecomment-621678932) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Julien Pivotto	52630ad0c7	Make head Postings only return series in time range benchmark old ns/op new ns/op delta BenchmarkQuerierSelect/Head/1of1000000-8 405805161 120436132 -70.32% BenchmarkQuerierSelect/Head/10of1000000-8 403079620 120624292 -70.07% BenchmarkQuerierSelect/Head/100of1000000-8 404678647 120923522 -70.12% BenchmarkQuerierSelect/Head/1000of1000000-8 403145813 118636563 -70.57% BenchmarkQuerierSelect/Head/10000of1000000-8 405020046 125716206 -68.96% BenchmarkQuerierSelect/Head/100000of1000000-8 426305002 175808499 -58.76% BenchmarkQuerierSelect/Head/1000000of1000000-8 619002108 567013003 -8.40% BenchmarkQuerierSelect/SortedHead/1of1000000-8 1276316086 120281094 -90.58% BenchmarkQuerierSelect/SortedHead/10of1000000-8 1282631170 121836526 -90.50% BenchmarkQuerierSelect/SortedHead/100of1000000-8 1325824787 121174967 -90.86% BenchmarkQuerierSelect/SortedHead/1000of1000000-8 1271386268 121025117 -90.48% BenchmarkQuerierSelect/SortedHead/10000of1000000-8 1280223345 130838948 -89.78% BenchmarkQuerierSelect/SortedHead/100000of1000000-8 1271401620 243635515 -80.84% BenchmarkQuerierSelect/SortedHead/1000000of1000000-8 1360256090 1307744674 -3.86% BenchmarkQuerierSelect/Block/1of1000000-8 748183120 707888498 -5.39% BenchmarkQuerierSelect/Block/10of1000000-8 741084129 716317249 -3.34% BenchmarkQuerierSelect/Block/100of1000000-8 722157273 735624256 +1.86% BenchmarkQuerierSelect/Block/1000of1000000-8 727587744 731981838 +0.60% BenchmarkQuerierSelect/Block/10000of1000000-8 727518578 726860308 -0.09% BenchmarkQuerierSelect/Block/100000of1000000-8 765577046 757382386 -1.07% BenchmarkQuerierSelect/Block/1000000of1000000-8 1126722881 1084779083 -3.72% benchmark old allocs new allocs delta BenchmarkQuerierSelect/Head/1of1000000-8 4000018 24 -100.00% BenchmarkQuerierSelect/Head/10of1000000-8 4000036 82 -100.00% BenchmarkQuerierSelect/Head/100of1000000-8 4000216 625 -99.98% BenchmarkQuerierSelect/Head/1000of1000000-8 4002016 6028 -99.85% BenchmarkQuerierSelect/Head/10000of1000000-8 4020016 60037 -98.51% BenchmarkQuerierSelect/Head/100000of1000000-8 4200016 600047 -85.71% BenchmarkQuerierSelect/Head/1000000of1000000-8 6000016 6000016 +0.00% BenchmarkQuerierSelect/SortedHead/1of1000000-8 4000055 28 -100.00% BenchmarkQuerierSelect/SortedHead/10of1000000-8 4000073 87 -100.00% BenchmarkQuerierSelect/SortedHead/100of1000000-8 4000253 630 -99.98% BenchmarkQuerierSelect/SortedHead/1000of1000000-8 4002053 6036 -99.85% BenchmarkQuerierSelect/SortedHead/10000of1000000-8 4020053 60054 -98.51% BenchmarkQuerierSelect/SortedHead/100000of1000000-8 4200053 600074 -85.71% BenchmarkQuerierSelect/SortedHead/1000000of1000000-8 6000053 6000053 +0.00% BenchmarkQuerierSelect/Block/1of1000000-8 6000021 6000021 +0.00% BenchmarkQuerierSelect/Block/10of1000000-8 6000057 6000057 +0.00% BenchmarkQuerierSelect/Block/100of1000000-8 6000417 6000417 +0.00% BenchmarkQuerierSelect/Block/1000of1000000-8 6004017 6004017 +0.00% BenchmarkQuerierSelect/Block/10000of1000000-8 6040017 6040017 +0.00% BenchmarkQuerierSelect/Block/100000of1000000-8 6400017 6400017 +0.00% BenchmarkQuerierSelect/Block/1000000of1000000-8 10000018 10000018 +0.00% benchmark old bytes new bytes delta BenchmarkQuerierSelect/Head/1of1000000-8 176001177 1392 -100.00% BenchmarkQuerierSelect/Head/10of1000000-8 176002329 4368 -100.00% BenchmarkQuerierSelect/Head/100of1000000-8 176013849 33520 -99.98% BenchmarkQuerierSelect/Head/1000of1000000-8 176129056 321456 -99.82% BenchmarkQuerierSelect/Head/10000of1000000-8 177281049 3427376 -98.07% BenchmarkQuerierSelect/Head/100000of1000000-8 188801049 35055408 -81.43% BenchmarkQuerierSelect/Head/1000000of1000000-8 304001059 304001049 -0.00% BenchmarkQuerierSelect/SortedHead/1of1000000-8 229192188 2488 -100.00% BenchmarkQuerierSelect/SortedHead/10of1000000-8 229193340 5568 -100.00% BenchmarkQuerierSelect/SortedHead/100of1000000-8 229204860 35536 -99.98% BenchmarkQuerierSelect/SortedHead/1000of1000000-8 229320060 345104 -99.85% BenchmarkQuerierSelect/SortedHead/10000of1000000-8 230472060 3894672 -98.31% BenchmarkQuerierSelect/SortedHead/100000of1000000-8 241992060 40511632 -83.26% BenchmarkQuerierSelect/SortedHead/1000000of1000000-8 357192060 357192060 +0.00% BenchmarkQuerierSelect/Block/1of1000000-8 227201516 227201506 -0.00% BenchmarkQuerierSelect/Block/10of1000000-8 227203057 227203041 -0.00% BenchmarkQuerierSelect/Block/100of1000000-8 227217161 227217165 +0.00% BenchmarkQuerierSelect/Block/1000of1000000-8 227358279 227358289 +0.00% BenchmarkQuerierSelect/Block/10000of1000000-8 228769485 228769475 -0.00% BenchmarkQuerierSelect/Block/100000of1000000-8 242881487 242881477 -0.00% BenchmarkQuerierSelect/Block/1000000of1000000-8 384001705 384001705 +0.00% Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Brian Brazil	cebe36c7d5	Make head Postings only return series in time range Series() will fetch all the metadata for a series, even if it's going to be filtered later due to time ranges. For 1M series we save ~1.1s if you only needed some of the data, but take an extra ~.2s if you did want everything. benchmark old ns/op new ns/op delta BenchmarkHeadSeries/1of1000000-4 1443715987 131553480 -90.89% BenchmarkHeadSeries/10of1000000-4 1433394040 130730596 -90.88% BenchmarkHeadSeries/100of1000000-4 1437444672 131360813 -90.86% BenchmarkHeadSeries/1000of1000000-4 1438958659 132573137 -90.79% BenchmarkHeadSeries/10000of1000000-4 1438061766 145742377 -89.87% BenchmarkHeadSeries/100000of1000000-4 1455060948 281659416 -80.64% BenchmarkHeadSeries/1000000of1000000-4 1633524504 1803550153 +10.41% benchmark old allocs new allocs delta BenchmarkHeadSeries/1of1000000-4 4000055 28 -100.00% BenchmarkHeadSeries/10of1000000-4 4000073 87 -100.00% BenchmarkHeadSeries/100of1000000-4 4000253 630 -99.98% BenchmarkHeadSeries/1000of1000000-4 4002053 6036 -99.85% BenchmarkHeadSeries/10000of1000000-4 4020053 60054 -98.51% BenchmarkHeadSeries/100000of1000000-4 4200053 600074 -85.71% BenchmarkHeadSeries/1000000of1000000-4 6000053 6000094 +0.00% benchmark old bytes new bytes delta BenchmarkHeadSeries/1of1000000-4 229192184 2488 -100.00% BenchmarkHeadSeries/10of1000000-4 229193336 5568 -100.00% BenchmarkHeadSeries/100of1000000-4 229204856 35536 -99.98% BenchmarkHeadSeries/1000of1000000-4 229320056 345104 -99.85% BenchmarkHeadSeries/10000of1000000-4 230472056 3894673 -98.31% BenchmarkHeadSeries/100000of1000000-4 241992056 40511632 -83.26% BenchmarkHeadSeries/1000000of1000000-4 357192056 402380440 +12.65% Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	5 years ago
Thor	17d8c49919	made stripe size configurable (#6644 ) Signed-off-by: Thor <thansen@digitalocean.com>	5 years ago
Brian Brazil	aff9f7a9e8	Extend PostingsForMatchers benchmark to cover Blocks too. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	5 years ago
Tom Wilkie	de0a772b8e	Port tsdb to use pkg/labels. (#6326 ) * Port tsdb to use pkg/labels. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Get tests passing. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Remove useless cast. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Appease linters. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Bartek Plotka	f0863a604e	Removed extra tsdb/testutil after merge. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>	5 years ago
Ganesh Vernekar	5ecef3542d	Cleanup after merging tsdb into prometheus Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Ganesh Vernekar	7cf09b0395	Moving tsdb into its own subdirectory Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Brian Brazil	89a90fe96c	Simplify mergedPostings.Seek (#595 ) The current implementation leads to very slow behaviour when there's many lists, this no worse than n log k, where k is the number of posting lists. Adjust benchmark to catch this. Remove flattening of without lists, not needed anymore. Benchmark versus 0.4.0 (used in Prometheus 2.7): ``` benchmark old ns/op new ns/op delta BenchmarkHeadPostingForMatchers/n="1"-8 189907976 188863880 -0.55% BenchmarkHeadPostingForMatchers/n="1",j="foo"-8 113950106 110791414 -2.77% BenchmarkHeadPostingForMatchers/j="foo",n="1"-8 104965646 102388760 -2.45% BenchmarkHeadPostingForMatchers/n="1",j!="foo"-8 138743592 104424250 -24.74% BenchmarkHeadPostingForMatchers/i=~"."-8 5279594954 5206096267 -1.39% BenchmarkHeadPostingForMatchers/i=~".+"-8 8004610589 6184527719 -22.74% BenchmarkHeadPostingForMatchers/i=~""-8 2476042646 1003920432 -59.45% BenchmarkHeadPostingForMatchers/i!=""-8 7178244655 6059725323 -15.58% BenchmarkHeadPostingForMatchers/n="1",i=~".",j="foo"-8 199342649 166642946 -16.40% BenchmarkHeadPostingForMatchers/n="1",i=~".",i!="2",j="foo"-8 215774683 167515095 -22.37% BenchmarkHeadPostingForMatchers/n="1",i!=""-8 2214714769 392943663 -82.26% BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"-8 2148727410 322289262 -85.00% BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"-8 2170658009 338458171 -84.41% BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"-8 235720135 70597905 -70.05% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"-8 2190570590 343034307 -84.34% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.",j="foo"-8 2373784439 387297908 -83.68% benchmark old allocs new allocs delta BenchmarkHeadPostingForMatchers/n="1"-8 33 33 +0.00% BenchmarkHeadPostingForMatchers/n="1",j="foo"-8 33 33 +0.00% BenchmarkHeadPostingForMatchers/j="foo",n="1"-8 33 33 +0.00% BenchmarkHeadPostingForMatchers/n="1",j!="foo"-8 41 39 -4.88% BenchmarkHeadPostingForMatchers/i=~"."-8 56 56 +0.00% BenchmarkHeadPostingForMatchers/i=~".+"-8 251577 100115 -60.21% BenchmarkHeadPostingForMatchers/i=~""-8 251123 100077 -60.15% BenchmarkHeadPostingForMatchers/i!=""-8 251525 100112 -60.20% BenchmarkHeadPostingForMatchers/n="1",i=~".",j="foo"-8 42 39 -7.14% BenchmarkHeadPostingForMatchers/n="1",i=~".",i!="2",j="foo"-8 52 42 -19.23% BenchmarkHeadPostingForMatchers/n="1",i!=""-8 251069 100101 -60.13% BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"-8 251473 100101 -60.19% BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"-8 250914 100102 -60.11% BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"-8 30038 11181 -62.78% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"-8 250813 100105 -60.09% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.",j="foo"-8 281503 111260 -60.48% benchmark old bytes new bytes delta BenchmarkHeadPostingForMatchers/n="1"-8 10887600 10887600 +0.00% BenchmarkHeadPostingForMatchers/n="1",j="foo"-8 5456416 5456416 +0.00% BenchmarkHeadPostingForMatchers/j="foo",n="1"-8 5456416 5456416 +0.00% BenchmarkHeadPostingForMatchers/n="1",j!="foo"-8 5456640 5456544 -0.00% BenchmarkHeadPostingForMatchers/i=~"."-8 258254504 258254472 -0.00% BenchmarkHeadPostingForMatchers/i=~".+"-8 520126192 281554792 -45.87% BenchmarkHeadPostingForMatchers/i=~""-8 263446640 24908456 -90.55% BenchmarkHeadPostingForMatchers/i!=""-8 520121144 281553664 -45.87% BenchmarkHeadPostingForMatchers/n="1",i=~".",j="foo"-8 7062448 7062272 -0.00% BenchmarkHeadPostingForMatchers/n="1",i=~".",i!="2",j="foo"-8 7063473 7062384 -0.02% BenchmarkHeadPostingForMatchers/n="1",i!=""-8 274325656 35793776 -86.95% BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"-8 268926824 30362624 -88.71% BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"-8 268882992 30363000 -88.71% BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"-8 33193401 4269304 -87.14% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"-8 268875024 30363096 -88.71% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.",j="foo"-8 300589656 33099784 -88.99% ``` Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	6 years ago
Brian Brazil	259847a6b1	Be smarter in how we look at matchers. (#572 ) * Add unittests for PostingsForMatcher. * Selector methods are all stateless, don't need a reference. * Be smarter in how we look at matchers. Look at all matchers to see if a label can be empty. Optimise Not handling, so i!="2" is a simple lookup rather than an inverse postings list. All all the Withouts together, rather than having to subtract each from all postings. Change the pre-expand the postings logic to always do it before doing a Without only. Don't do that if it's already a list. The initial goal here was that the oft-seen pattern i=~"something.+",i!="foo",i!="bar" becomes more efficient. benchmark old ns/op new ns/op delta BenchmarkHeadPostingForMatchers/n="1"-4 5888 6160 +4.62% BenchmarkHeadPostingForMatchers/n="1",j="foo"-4 7190 6640 -7.65% BenchmarkHeadPostingForMatchers/j="foo",n="1"-4 6038 5923 -1.90% BenchmarkHeadPostingForMatchers/n="1",j!="foo"-4 6030884 4850525 -19.57% BenchmarkHeadPostingForMatchers/i=~"."-4 887377940 230329137 -74.04% BenchmarkHeadPostingForMatchers/i=~".+"-4 490316101 319931758 -34.75% BenchmarkHeadPostingForMatchers/i=~""-4 594961991 130279313 -78.10% BenchmarkHeadPostingForMatchers/i!=""-4 537542388 318751015 -40.70% BenchmarkHeadPostingForMatchers/n="1",i=~".",j="foo"-4 10460243 8565195 -18.12% BenchmarkHeadPostingForMatchers/n="1",i=~".",i!="2",j="foo"-4 44964267 8561546 -80.96% BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"-4 42244885 29137737 -31.03% BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"-4 35285834 32774584 -7.12% BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"-4 8951047 8379024 -6.39% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"-4 63813335 30672688 -51.93% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.",j="foo"-4 45381112 44924397 -1.01% Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	6 years ago
Brian Brazil	62b652fbd0	Improve Merge performance (#531 ) Use a heap for Next for merges, and pre-compute if there's many postings on the unset path. Add posting lookup benchmarks Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	6 years ago
Brian Brazil	b2d7bbd6b1	Move series fetches out of inner loop of SortedPostings. (#485 ) With 1M series: Before: BenchmarkHeadPostingForMatchers-8 1 3501996117 ns/op 61311520 B/op 78 allocs/op After: BenchmarkHeadPostingForMatchers-8 1 1403072952 ns/op 69261568 B/op 72 allocs/op This works out as 3X faster, as the above time includes other things. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	6 years ago
Ganesh Vernekar	a95323c021	Add license headers to missing files (#447 ) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Thomas Jackson	b4132df5f7	Reduce allocations for queries on `HEAD` (#417 ) Some benchmarks for HEAD and allocate the correct slice size in LabelValues , we already know what it'll be This is ~15% time improvement, and ~25% allocation improvement: ``` benchmark old ns/op new ns/op delta BenchmarkHeadPostingForMatchers-4 74452 63514 -14.69% benchmark old allocs new allocs delta BenchmarkHeadPostingForMatchers-4 20 13 -35.00% benchmark old bytes new bytes delta BenchmarkHeadPostingForMatchers-4 5425 3137 -42.18% ``` Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>	6 years ago

20 Commits (88bb05c3e8d84de147f219f6e4c891d8ba87edc2)