prometheus

Commit Graph

Author	SHA1	Message	Date
Chris Marchbanks	bedcd88343	Compress records before checking segment size (#8501 ) Right now a new segment might be created unnecessarily if the uncompressed record would not fit, but after compression (typically reducing record size in half) it would. Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	4 years ago
Li Zhijian	9602933db2	Cleanup tempfiles in tests (#8179 ) * test: cleanup tempdir for TestBlockWriter Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com> * test: cleanup tempdir for TestLogPartialWrite Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>	4 years ago
Marco Pracucci	63be30dcee	Fixed WAL corruption on partial writes within a page (#8125 ) * Fixed WAL corruption on partial writes Signed-off-by: Marco Pracucci <marco@pracucci.com> * Renamed variable Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed test Signed-off-by: Marco Pracucci <marco@pracucci.com> * Addressed review comments Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Julien Pivotto	6c56a1faaa	Testify: move to require (#8122 ) * Testify: move to require Moving testify to require to fail tests early in case of errors. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> * More moves Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Bartlomiej Plotka	3d8826a3d4	MultiError: Refactored MultiError for more concise and safe usage. (#8066 ) * MultiError: Refactored MultiError for more concise and safe usage. * Less lines * Goland IDE was marking every usage of old MultiError "potential nil" error * It was easy to forgot using Err() when error was returned, now it's safely assured on compile time. NOTE: Potentially I would rename package to merrors. (: In different PR. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed review comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fix after rebase. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	4 years ago
Julien Pivotto	1282d1b39c	Refactor test assertions (#8110 ) * Refactor test assertions This pull request gets rid of assert.True where possible to use fine-grained assertions. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	4e5b1722b3	Move away from testutil, refactor imports (#8087 ) Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Brian Brazil	073e93c768	Gracefully handle unknown WAL record types. (#8004 ) As we're looking to expand what's in the WAL, having old Prometheus servers ignore the new record types rather than treating them as corruption allows for better upgrade/downgrade paths. Adjust some tests accordingly, so they're still testing what they're meant to test. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	4 years ago
zhulongcheng	5ea6bdca1f	tsdb: sync temporary checkpoint directory before rename (#7004 ) Signed-off-by: zhulongcheng <zhulongcheng.dev@gmail.com>	4 years ago
Ganesh Vernekar	2255b6f62f	Refactor WAL.Segments method to be part of the wal package (#6477 ) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	4 years ago
gastonqiu	5a7d398d19	Chore: Log segment number when segment read failed (#7687 ) * Chore: Log segment number when segment read failed To manually fix the WAL files, it is good to know where the corrupt happened so we should log the segment number when the read failed. Related Issue #7506 Signed-off-by: gaston.qiu <gaston.qiu@umbocv.com>	4 years ago
Julien Pivotto	62805b2fe9	tsdb: test for leaks (#7566 ) Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Zhou Hao	ddedf454d0	add os.RemoveAll err verification (#7540 ) * add os.RemoveAll err verification for watcher_test Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com> * add os.RemoveAll err verification for db_test Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com> * add os.RemoveAll err verification for write_test Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com> * add os.RemoveAll err verification for queue_manager_test Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com> * tsdb/wal/watcher_test: add close operation before delete Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com>	4 years ago
Ganesh Vernekar	ea013343ca	Log when starting to create a checkpoint (#7581 ) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	4 years ago
Marek Slabicki	8224ddec23	Capitalizing first letter of all log lines (#7043 ) Signed-off-by: Marek Slabicki <thaniri@gmail.com>	5 years ago
Brad Walker	3348930df5	Replace fileutil.ReadDir with ioutil.ReadDir (#7029 ) (#7033 ) * tsdb: Replace fileutil.ReadDir with ioutil.ReadDir (#7029) Signed-off-by: Brad Walker <brad@bradmwalker.com> * tsdb: Remove fileutil.ReadDir (#7029) Signed-off-by: Brad Walker <brad@bradmwalker.com>	5 years ago
Julien Pivotto	73228b1b68	Those links should not be reverted Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Julien Pivotto	653f343547	Revert head posting optimization This reverts commit `52630ad0c7`. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Ben Kochie	269e7c8091	Fix golint issues. Signed-off-by: Ben Kochie <superq@gmail.com>	5 years ago
Ganesh Vernekar	e64a149984	Close Head in DBReadOnly.FlushWAL (#7022 ) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
zhulongcheng	e813f60fd6	tsdb: fix sequence check for WAL segments (#7032 ) Signed-off-by: zhulongcheng <zhulongcheng.dev@gmail.com>	5 years ago
Callum Styan	f802f1e8ca	Fix bug with WAL watcher and Live Reader metrics usage. (#6998 ) * Fix bug with WAL watcher and Live Reader metrics usage. Calling NewXMetrics when creating a Watcher or LiveReader results in a registration error, which we're ignoring, and as a result other than the first Watcher/Reader created, we had no metrics for either. So we would only have metrics like Watcher Records Read for the first remote write config in a users config file. Signed-off-by: Callum Styan <callumstyan@gmail.com>	5 years ago
zhulongcheng	5f5c7a4477	tsdb: sort checkpoints by segment number (#6987 ) Signed-off-by: zhulongcheng <zhulongcheng.dev@gmail.com>	5 years ago
李国忠	261cbab8e9	remove Unused parameter 'reg' in wal.Open function (#6941 ) Signed-off-by: fuling <fuling.lgz@alibaba-inc.com>	5 years ago
Julien Pivotto	52630ad0c7	Make head Postings only return series in time range benchmark old ns/op new ns/op delta BenchmarkQuerierSelect/Head/1of1000000-8 405805161 120436132 -70.32% BenchmarkQuerierSelect/Head/10of1000000-8 403079620 120624292 -70.07% BenchmarkQuerierSelect/Head/100of1000000-8 404678647 120923522 -70.12% BenchmarkQuerierSelect/Head/1000of1000000-8 403145813 118636563 -70.57% BenchmarkQuerierSelect/Head/10000of1000000-8 405020046 125716206 -68.96% BenchmarkQuerierSelect/Head/100000of1000000-8 426305002 175808499 -58.76% BenchmarkQuerierSelect/Head/1000000of1000000-8 619002108 567013003 -8.40% BenchmarkQuerierSelect/SortedHead/1of1000000-8 1276316086 120281094 -90.58% BenchmarkQuerierSelect/SortedHead/10of1000000-8 1282631170 121836526 -90.50% BenchmarkQuerierSelect/SortedHead/100of1000000-8 1325824787 121174967 -90.86% BenchmarkQuerierSelect/SortedHead/1000of1000000-8 1271386268 121025117 -90.48% BenchmarkQuerierSelect/SortedHead/10000of1000000-8 1280223345 130838948 -89.78% BenchmarkQuerierSelect/SortedHead/100000of1000000-8 1271401620 243635515 -80.84% BenchmarkQuerierSelect/SortedHead/1000000of1000000-8 1360256090 1307744674 -3.86% BenchmarkQuerierSelect/Block/1of1000000-8 748183120 707888498 -5.39% BenchmarkQuerierSelect/Block/10of1000000-8 741084129 716317249 -3.34% BenchmarkQuerierSelect/Block/100of1000000-8 722157273 735624256 +1.86% BenchmarkQuerierSelect/Block/1000of1000000-8 727587744 731981838 +0.60% BenchmarkQuerierSelect/Block/10000of1000000-8 727518578 726860308 -0.09% BenchmarkQuerierSelect/Block/100000of1000000-8 765577046 757382386 -1.07% BenchmarkQuerierSelect/Block/1000000of1000000-8 1126722881 1084779083 -3.72% benchmark old allocs new allocs delta BenchmarkQuerierSelect/Head/1of1000000-8 4000018 24 -100.00% BenchmarkQuerierSelect/Head/10of1000000-8 4000036 82 -100.00% BenchmarkQuerierSelect/Head/100of1000000-8 4000216 625 -99.98% BenchmarkQuerierSelect/Head/1000of1000000-8 4002016 6028 -99.85% BenchmarkQuerierSelect/Head/10000of1000000-8 4020016 60037 -98.51% BenchmarkQuerierSelect/Head/100000of1000000-8 4200016 600047 -85.71% BenchmarkQuerierSelect/Head/1000000of1000000-8 6000016 6000016 +0.00% BenchmarkQuerierSelect/SortedHead/1of1000000-8 4000055 28 -100.00% BenchmarkQuerierSelect/SortedHead/10of1000000-8 4000073 87 -100.00% BenchmarkQuerierSelect/SortedHead/100of1000000-8 4000253 630 -99.98% BenchmarkQuerierSelect/SortedHead/1000of1000000-8 4002053 6036 -99.85% BenchmarkQuerierSelect/SortedHead/10000of1000000-8 4020053 60054 -98.51% BenchmarkQuerierSelect/SortedHead/100000of1000000-8 4200053 600074 -85.71% BenchmarkQuerierSelect/SortedHead/1000000of1000000-8 6000053 6000053 +0.00% BenchmarkQuerierSelect/Block/1of1000000-8 6000021 6000021 +0.00% BenchmarkQuerierSelect/Block/10of1000000-8 6000057 6000057 +0.00% BenchmarkQuerierSelect/Block/100of1000000-8 6000417 6000417 +0.00% BenchmarkQuerierSelect/Block/1000of1000000-8 6004017 6004017 +0.00% BenchmarkQuerierSelect/Block/10000of1000000-8 6040017 6040017 +0.00% BenchmarkQuerierSelect/Block/100000of1000000-8 6400017 6400017 +0.00% BenchmarkQuerierSelect/Block/1000000of1000000-8 10000018 10000018 +0.00% benchmark old bytes new bytes delta BenchmarkQuerierSelect/Head/1of1000000-8 176001177 1392 -100.00% BenchmarkQuerierSelect/Head/10of1000000-8 176002329 4368 -100.00% BenchmarkQuerierSelect/Head/100of1000000-8 176013849 33520 -99.98% BenchmarkQuerierSelect/Head/1000of1000000-8 176129056 321456 -99.82% BenchmarkQuerierSelect/Head/10000of1000000-8 177281049 3427376 -98.07% BenchmarkQuerierSelect/Head/100000of1000000-8 188801049 35055408 -81.43% BenchmarkQuerierSelect/Head/1000000of1000000-8 304001059 304001049 -0.00% BenchmarkQuerierSelect/SortedHead/1of1000000-8 229192188 2488 -100.00% BenchmarkQuerierSelect/SortedHead/10of1000000-8 229193340 5568 -100.00% BenchmarkQuerierSelect/SortedHead/100of1000000-8 229204860 35536 -99.98% BenchmarkQuerierSelect/SortedHead/1000of1000000-8 229320060 345104 -99.85% BenchmarkQuerierSelect/SortedHead/10000of1000000-8 230472060 3894672 -98.31% BenchmarkQuerierSelect/SortedHead/100000of1000000-8 241992060 40511632 -83.26% BenchmarkQuerierSelect/SortedHead/1000000of1000000-8 357192060 357192060 +0.00% BenchmarkQuerierSelect/Block/1of1000000-8 227201516 227201506 -0.00% BenchmarkQuerierSelect/Block/10of1000000-8 227203057 227203041 -0.00% BenchmarkQuerierSelect/Block/100of1000000-8 227217161 227217165 +0.00% BenchmarkQuerierSelect/Block/1000of1000000-8 227358279 227358289 +0.00% BenchmarkQuerierSelect/Block/10000of1000000-8 228769485 228769475 -0.00% BenchmarkQuerierSelect/Block/100000of1000000-8 242881487 242881477 -0.00% BenchmarkQuerierSelect/Block/1000000of1000000-8 384001705 384001705 +0.00% Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
johncming	c30abf1e2b	tsdb/wal: remove unused argument. Signed-off-by: johncming <johncming@yahoo.com>	5 years ago
John McBride	669592a2c4	Exports metric for WAL write errors (#6647 ) * Exports metric for WAL write errors Signed-off-by: John McBride <jpmmcbride@gmail.com> * Correct name for counter Signed-off-by: John McBride <jpmmcbride@gmail.com> * Move WAL write failure to wal.go Signed-off-by: John McBride <jpmmcbride@gmail.com> * WAL write fail metric moved to Log for external consumers Signed-off-by: John McBride <jpmmcbride@gmail.com>	5 years ago
Julien Pivotto	201491fd1b	Remove old checkpoint dir if it still exists (#6621 ) Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Brian Brazil	ada0945b8f	Fix tsdb code and tests to work on Windows. (#6547 ) Add back Windows CI, we lost it when tsdb was merged into the prometheus repo. There's many tests failing outside tsdb, so only test tsdb for now. Fixes #6513 Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	5 years ago
Brian Brazil	536d416299	Fix tsdb code and tests to work on Windows. (#6547 ) Add back Windows CI, we lost it when tsdb was merged into the prometheus repo. There's many tests failing outside tsdb, so only test tsdb for now. Fixes #6513 Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	5 years ago
Josh Soref	91d76c8023	Spelling (#6517 ) * spelling: alertmanager Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: attributes Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: autocomplete Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: bootstrap Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: caught Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: chunkenc Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: compaction Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: corrupted Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: deletable Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: expected Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: fine-grained Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: initialized Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: iteration Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: javascript Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: multiple Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: number Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: overlapping Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: possible Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: postings Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: procedure Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: programmatic Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: queuing Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: querier Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: repairing Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: received Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: reproducible Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: retention Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: sample Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: segements Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: semantic Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: software [LICENSE] Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: staging Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: timestamp Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: unfortunately Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: uvarint Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: subsequently Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: ressamples Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	5 years ago
Callum Styan	6a24eee340	Simplify duration check for watcher WAL replay. Signed-off-by: Callum Styan <callumstyan@gmail.com>	5 years ago
Callum Styan	2d3ce3916c	Log the start and end of the WAL replay within the WAL watcher. Signed-off-by: Callum Styan <callumstyan@gmail.com>	5 years ago
Tom Wilkie	de0a772b8e	Port tsdb to use pkg/labels. (#6326 ) * Port tsdb to use pkg/labels. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Get tests passing. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Remove useless cast. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Appease linters. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Dipack P Panjabi	ce7bab04dd	Compute WAL size and use it during retention size checks (#5886 ) * Compute WAL size and account for it when applying the retention settings. Signed-off-by: Dipack P Panjabi <dpanjabi@hudson-trading.com>	5 years ago
Chris Marchbanks	8df4bca470	Garbage collect asynchronously in the WAL Watcher The WAL Watcher replays a checkpoint after it is created in order to garbage collect series that no longer exist in the WAL. Currently the garbage collection process is done serially with reading from the tip of the WAL which can cause large delays in writing samples to remote storage just after compaction occurs. This also fixes a memory leak where dropped series are not cleaned up as part of the SeriesReset process. Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	5 years ago
陈谭军	103f26d188	fix the wrong word (#6069 ) Signed-off-by: chentanjun <2799194073@qq.com>	5 years ago
AllenZMC	1e62435960	fix wrong spells in live_reader.go (#5899 ) * fix wrong spells in live_reader.go * fix wrong spells in lex.go Signed-off-by: czm <zhongming.chang@daocloud.io>	5 years ago
johncming	612f9cb361	tsdb/wal: pull out wal metrics separately as tsdb.DB (#5957 ) Signed-off-by: johncming <johncming@yahoo.com>	5 years ago
Callum Styan	3344bb5c33	Move WAL watcher code to tsdb/wal package. (#5999 ) * Move WAL watcher code to tsdb/wal package. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix tests after moving WAL watcher code. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Lint fixes. Signed-off-by: Callum Styan <callumstyan@gmail.com>	5 years ago
Erfan Besharat	9336c01dfd	Add methods to fetch page's buf data in tsdb WAL (#5967 ) * move the WAL page buf reset in its own func Signed-off-by: Erfan Besharat <erbesharat@gmail.com>	5 years ago
陈谭军	50d453b3c3	fix-up tsdb-typo (#5954 ) Signed-off-by: chentanjun <2799194073@qq.com>	5 years ago
johncming	7d43feb03f	tsdb/wal: some small refactoring for easier reading (#5930 ) Signed-off-by: johncming <johncming@yahoo.com>	5 years ago
Bartek Plotka	f0863a604e	Removed extra tsdb/testutil after merge. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>	5 years ago
Ganesh Vernekar	5ecef3542d	Cleanup after merging tsdb into prometheus Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Ganesh Vernekar	7cf09b0395	Moving tsdb into its own subdirectory Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago

44 Commits (c7a62b95cea098b677a958bf4e20084bc427aad5)