prometheus

Commit Graph

Author	SHA1	Message	Date
Jesus Vazquez	c1b669bf9b	Add out-of-order sample support to the TSDB (#11075 ) * Introduce out-of-order TSDB support This implementation is based on this design doc: https://docs.google.com/document/d/1Kppm7qL9C-BJB1j6yb6-9ObG3AbdZnFUBYPNNWwDBYM/edit?usp=sharing This commit adds support to accept out-of-order ("OOO") sample into the TSDB up to a configurable time allowance. If OOO is enabled, overlapping querying are automatically enabled. Most of the additions have been borrowed from https://github.com/grafana/mimir-prometheus/ Here is the list ist of the original commits cherry picked from mimir-prometheus into this branch: - `4b2198d7ec` - `2836e5513f` - `00b379c3a5` - `ff0dc75758` - `a632c73352` - `c6f3d4ab33` - `5e8406a1d4` - `abde1e0ba1` - `e70e769889` - `df59320886` Co-authored-by: Jesus Vazquez <jesus.vazquez@grafana.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Dieter Plaetinck <dieter@grafana.com> Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * gofumpt files Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Add license header to missing files Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Fix OOO tests due to existing chunk disk mapper implementation Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Fix truncate int overflow Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Add Sync method to the WAL and update tests Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * remove useless sync Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Update minOOOTime after truncating Head * Update minOOOTime after truncating Head Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix lint Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Add a unit test Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Load OutOfOrderTimeWindow only once per appender Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Fix OOO Head LabelValues and PostingsForMatchers Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Fix replay of OOO mmap chunks Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Remove unnecessary err check Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Prevent panic with ApplyConfig Signed-off-by: Ganesh Vernekar 15064823+codesome@users.noreply.github.com Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Run OOO compaction after restart if there is OOO data from WBL Signed-off-by: Ganesh Vernekar 15064823+codesome@users.noreply.github.com Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Apply Bartek's suggestions Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Refactor OOO compaction Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Address comments and TODOs - Added a comment explaining why we need the allow overlapping compaction toggle - Clarified TSDBConfig OutOfOrderTimeWindow doc - Added an owner to all the TODOs in the code Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Run go format Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Fix remaining review comments Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix tests Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Change wbl reference when truncating ooo in TestHeadMinOOOTimeUpdate Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Fix TestWBLAndMmapReplay test failure on windows Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Address most of the feedback Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Refactor the block meta for out of order Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix windows error Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix review comments Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Ganesh Vernekar 15064823+codesome@users.noreply.github.com Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Dieter Plaetinck <dieter@grafana.com> Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>	2 years ago
Matthieu MOREL	e2ede285a2	refactor: move from io/ioutil to io and os packages (#10528 ) * refactor: move from io/ioutil to io and os packages * use fs.DirEntry instead of os.FileInfo after os.ReadDir Signed-off-by: MOREL Matthieu <matthieu.morel@cnp.fr>	3 years ago
Sunil Thaha	a484a83d4a	fix: panic when checkpoint directory is empty (#9687 ) Calling `wal.NewSegmentBufReader()` without any segments would cause a `panic` resulting in prometheus crashing. This patch fixes the panic by making segmentBufReader return a EOF if there are not segments. This also means an empty checkpoint directory which should never be the case unless it has been tampered with (or has issues due to the underlying filesystem e.g. NFS) would be ignored by Prometheus and would continue to run instead of the current behaviour which is to panic. Fixes: https://github.com/prometheus/prometheus/issues/9605 Signed-off-by: Sunil Thaha <sthaha@redhat.com>	3 years ago
beorn7	a1e595edac	Fix two trivial lint warnings Not sure why those show up for me locally but not if run by the CI. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
johncming	b882d2b7c7	tsdb/wal: Avoid writing closed channel. (#9566 ) Signed-off-by: johncming <johncming@yahoo.com>	3 years ago
Mateusz Gozdek	1a6c2283a3	Format Go source files using 'gofumpt -w -s -extra' Part of #9557 Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>	3 years ago
Julien Pivotto	73255e15f6	Address golint failures from revive Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	3 years ago
Ganesh Vernekar	ee7e0071d1	Snapshot in-memory chunks on shutdown for faster restarts (#7229 ) Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Levi Harrison	b5f6f8fb36	Switched to go-kit/log Signed-off-by: Levi Harrison <git@leviharrison.dev>	4 years ago
Chris Marchbanks	7c7dafc321	Do not snappy encode if record is too large (#8790 ) Snappy cannot encode records larger than ~3.7 GB and will panic if an encoding is attempted. Check to make sure that the record is smaller than this before encoding. In the future, we could improve this behavior to still compress large records (or break them up into smaller records), but this avoids the panic for users with very large single scrape targets. Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	4 years ago
Chris Marchbanks	bedcd88343	Compress records before checking segment size (#8501 ) Right now a new segment might be created unnecessarily if the uncompressed record would not fit, but after compression (typically reducing record size in half) it would. Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	4 years ago
Marco Pracucci	63be30dcee	Fixed WAL corruption on partial writes within a page (#8125 ) * Fixed WAL corruption on partial writes Signed-off-by: Marco Pracucci <marco@pracucci.com> * Renamed variable Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed test Signed-off-by: Marco Pracucci <marco@pracucci.com> * Addressed review comments Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Julien Pivotto	4e5b1722b3	Move away from testutil, refactor imports (#8087 ) Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Ganesh Vernekar	2255b6f62f	Refactor WAL.Segments method to be part of the wal package (#6477 ) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	4 years ago
Marek Slabicki	8224ddec23	Capitalizing first letter of all log lines (#7043 ) Signed-off-by: Marek Slabicki <thaniri@gmail.com>	5 years ago
Brad Walker	3348930df5	Replace fileutil.ReadDir with ioutil.ReadDir (#7029 ) (#7033 ) * tsdb: Replace fileutil.ReadDir with ioutil.ReadDir (#7029) Signed-off-by: Brad Walker <brad@bradmwalker.com> * tsdb: Remove fileutil.ReadDir (#7029) Signed-off-by: Brad Walker <brad@bradmwalker.com>	5 years ago
Ben Kochie	269e7c8091	Fix golint issues. Signed-off-by: Ben Kochie <superq@gmail.com>	5 years ago
Ganesh Vernekar	e64a149984	Close Head in DBReadOnly.FlushWAL (#7022 ) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
zhulongcheng	e813f60fd6	tsdb: fix sequence check for WAL segments (#7032 ) Signed-off-by: zhulongcheng <zhulongcheng.dev@gmail.com>	5 years ago
李国忠	261cbab8e9	remove Unused parameter 'reg' in wal.Open function (#6941 ) Signed-off-by: fuling <fuling.lgz@alibaba-inc.com>	5 years ago
johncming	c30abf1e2b	tsdb/wal: remove unused argument. Signed-off-by: johncming <johncming@yahoo.com>	5 years ago
John McBride	669592a2c4	Exports metric for WAL write errors (#6647 ) * Exports metric for WAL write errors Signed-off-by: John McBride <jpmmcbride@gmail.com> * Correct name for counter Signed-off-by: John McBride <jpmmcbride@gmail.com> * Move WAL write failure to wal.go Signed-off-by: John McBride <jpmmcbride@gmail.com> * WAL write fail metric moved to Log for external consumers Signed-off-by: John McBride <jpmmcbride@gmail.com>	5 years ago
Dipack P Panjabi	ce7bab04dd	Compute WAL size and use it during retention size checks (#5886 ) * Compute WAL size and account for it when applying the retention settings. Signed-off-by: Dipack P Panjabi <dpanjabi@hudson-trading.com>	5 years ago
johncming	612f9cb361	tsdb/wal: pull out wal metrics separately as tsdb.DB (#5957 ) Signed-off-by: johncming <johncming@yahoo.com>	5 years ago
Erfan Besharat	9336c01dfd	Add methods to fetch page's buf data in tsdb WAL (#5967 ) * move the WAL page buf reset in its own func Signed-off-by: Erfan Besharat <erbesharat@gmail.com>	5 years ago
陈谭军	50d453b3c3	fix-up tsdb-typo (#5954 ) Signed-off-by: chentanjun <2799194073@qq.com>	5 years ago
johncming	7d43feb03f	tsdb/wal: some small refactoring for easier reading (#5930 ) Signed-off-by: johncming <johncming@yahoo.com>	5 years ago
Ganesh Vernekar	5ecef3542d	Cleanup after merging tsdb into prometheus Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Ganesh Vernekar	7cf09b0395	Moving tsdb into its own subdirectory Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago

29 Commits (2673f94eb1e2435f904ede57b2796fb9cb209766)