prometheus

Commit Graph

Author	SHA1	Message	Date
Ganesh Vernekar	a632c73352	Simplify how OutOfOrderTimeWindow works (#285 ) * Simplify how OutOfOrderTimeWindow works Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Update test Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
Jesus Vazquez	e70e769889	Rename OutOfOrderAllowance to OutOfOrderTimeWindow After review Allowance is perhaps a bit misleading so we've decided to replace it with a more common term like TimeWindow.	2 years ago
Ganesh Vernekar	df59320886	Add out-of-order sample support to the TSDB (#269 ) This implementation is based on this design doc: https://docs.google.com/document/d/1Kppm7qL9C-BJB1j6yb6-9ObG3AbdZnFUBYPNNWwDBYM/edit?usp=sharing This commit adds support to accept out-of-order ("OOO") sample into the TSDB up to a configurable time allowance. If OOO is enabled, overlapping querying are automatically enabled. Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Jesus Vazquez <jesus.vazquez@grafana.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Dieter Plaetinck <dieter@grafana.com>	2 years ago
Oleg Zaytsev	ac87e2d4d6	Merge remote-tracking branch 'prometheus/main' into update-prometheus	3 years ago
Oleg Zaytsev	af0f6da5cb	Fix chunk overflow appending samples at a variable rate (#10607 ) * Add a test with variable samples rate append This test overflows the chunk created in memseries, and the total amount of samples in the (only) mmapped chunk is 29, instead of the 65565 appended ones. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Cut new chunk when rate prediction was wrong When appending samples at a slow rate, and then appending at a higher rate, the prediction we made to cut a new chunk is no longer valid. Sometimes this can even cause an overflow in the chunk, if more samples than uint16 can hold are appended. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Improve comment on 2samplesPerChunk Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> Assert that all chunks have less than 240 samples Also, trigger new chunk at 240, not at more than 240 Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	3 years ago
Jesus Vazquez	48aa5cd096	Merge remote-tracking branch 'upstream/main' into jvp/merge-prometheus-main	3 years ago
Dieter Plaetinck	aa8874bc56	clarify Head.appendableMinValidTime (#10303 ) Signed-off-by: Dieter Plaetinck <dieter@grafana.com>	3 years ago
Peter Štibraný	cd86e92b74	Merge pull request #132 from grafana/reintroduce-old-chunk-mapper Partial revert of PR 109 – This reintroduces old chunk disk mapper without a queue, that is used when queue size is configured to 0.	3 years ago
Mauro Stettler	f4d628d419	resolving merge conflicts	3 years ago
Mauro Stettler	0df3489275	Write chunks via queue, predicting the refs (#10051 ) * Write chunks via queue, predicting the refs Our load tests have shown that there is a latency spike in the remote write handler whenever the head chunks need to be written, because chunkDiskMapper.WriteChunk() blocks until the chunks are written to disk. This adds a queue to the chunk disk mapper which makes the WriteChunk() method non-blocking unless the queue is full. Reads can still be served from the queue. Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * address PR feeddback Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * initialize metrics without .Add(0) Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * change isRunningMtx to normal lock Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * do not re-initialize chunkrefmap Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * update metric outside of lock scope Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * add benchmark for adding job to chunk write queue Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * remove unnecessary "success" var Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * gofumpt -extra Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * avoid WithLabelValues call in addJob Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * format comments Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * addressing PR feedback Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * rename cutExpectRef to cutAndExpectRef Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * use head.Init() instead of .initTime() Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * address PR feedback Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * PR feedback Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com> Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * update test according to PR feedback Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * replace callbackWg -> awaitCb Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * better test of truncation with empty files Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * replace callbackWg -> awaitCb Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>	3 years ago
Ganesh Vernekar	fbe3167fda	Support having an empty queue for writing m-map chunks immediately (#84 ) * Support having an empty queue for flushing m-map chunks Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Use old ChunkDiskMapper for 0 length Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix tests Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix lint Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Mauro Stettler	cc9ddf50c9	Write m-map chunks in async via a queue while predicting the m-map chunk refs (#57 ) * write chunks via queue, predicting chunkrefs * fixes * linter * cleanup * better test coverage * deduplicate operations when cutting file * better comments * fix race * improvements based on PR feedback * comments * concurrent correctness test of write queue * linter * use isStarted flag in chunk write queue * separate mutex for isStarted * fixing tests * fix race in test * PR feedback * add license headers * PR feedback * pr feedback * dont recreate workerCtrl channel * address PR feedback in high concurrency read and write test * refactor the high concurrency write queue test * pr feedback * use require.Eventually * typo * Fixing comment Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com> * use buffered chan for chunk write queue * fix races * address feedback on PR * better comment * less type casts * dont reeimplement varint * move mtx above property it protects * improve tests by using wg instead of inspecting queue * add comment * add comment * writeChunkF comment * rename isStarted -> isRunning * prevent panic when double stopping * fix variable name in comment * delete outdated comment * naming and comment fixes in TestChunkWriteQueue_WrappingAroundSizeLimit * use require.False instead of require.Equal(t, false * always enable chunk write queue * dont call callback with lock held * undo rename of curFileSequence to curFileSeq * fix accidental change * fix comment * Better syntax Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com> * Better wording in comment Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com> * better comments to explain the use of coordinator chans in test * fix test which broke to the async writing of chunks * undo previous renaming of shadowing variable * rename var threadID to workerID * rename variables based on PR feedback * rename pos -> ts * better implementation of whileNotCanceled * update querySeriesRef before using it * rename labels -> lbls * initialize the chunk write queue operations metric with all possible labels * dont return error from CutNewFile * recreate chunk ref map regularly to free * limit how often we recreate the chunk ref map * fix typo in comment * better comment Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>	3 years ago
Dieter Plaetinck	757f57e509	track OOO-ness in tsdb into a histogram (#56 ) * track OOO-ness in tsdb into a histogram This should help with building an understanding of customer requirements as far as how far back samples tend to go. Note: this has already been discussed, reviewed and approved on: https://github.com/grafana/mimir/pull/488 Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * update the test files which were not in mimir/vendor	3 years ago
Peter Štibraný	f93d38aca7	Merge remote-tracking branch 'upstream/main' into merge-upstream	3 years ago
Darshan Chaudhary	9dcf8b2208	Add the ability to disable tsdb isolation (#9270 ) * Disable isolation in isolation struct Signed-off-by: darshanime <deathbullet@gmail.com> * Run tsdb tests with isolation disabled Signed-off-by: darshanime <deathbullet@gmail.com> * Check for isolation disabled in isoState.Close() Signed-off-by: darshanime <deathbullet@gmail.com> * use t.Skip to skip isolation tests when disabled Signed-off-by: darshanime <deathbullet@gmail.com> * address review comments Signed-off-by: darshanime <deathbullet@gmail.com> * fix test for defaultIsolationState Signed-off-by: darshanime <deathbullet@gmail.com> * Change flag name. Set flag in DB. Do not init txRing. Close isoState. Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Test disabled isolation in CircleCI test_go Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Skip isolation related tests in db_test.go Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Peter Štibraný	294c155bb6	Merge remote-tracking branch 'upstream/main' into merge-upstream	3 years ago
Dieter Plaetinck	0fac9bb859	Add basic initial developer docs for TSDB (#9451 ) * Add basic initial developer docs for TSDB There's a decent amount of content already out there (blog posts, conference talks, etc), but: * when they get stale, they don't tend to get updated * they still leave me with questions that I'ld like to answer for developers (like me) who want to use, or work with, TSDB What I propose is developer docs inside the prometheus repository. Easy to find and harness the power of the community to expand it and keep it up to date. * perfect is the enemy of good. Let's have a base and incrementally improve * Markdown docs should be broad but not too deep. Source code comments can complement them, and are the ideal place for implementation details. Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * use example code that works out of the box Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * Apply suggestions from code review Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com> Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * PR feedback Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * more docs Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * PR feedback Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * Apply suggestions from code review Signed-off-by: Dieter Plaetinck <dieter@grafana.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> * Apply suggestions from code review Signed-off-by: Dieter Plaetinck <dieter@grafana.com> Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com> * feedback Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * Update tsdb/docs/usage.md Signed-off-by: Dieter Plaetinck <dieter@grafana.com> Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com> * final tweaks Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * workaround docs versioning issue Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * Move example code to real executable, testable example. Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * cleanup example test and make sure it always reproduces Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * obtain temp dir in a way that works with older Go versions Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * Fix Ganesh's comments Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Marco Pracucci	6525385b30	Merge pull request #29 from grafana/add-jitter-to-chunk-end Add jitter to head chunks flushing	3 years ago
beorn7	c954cd9d1d	Move packages out of deprecated pkg directory This creates a new `model` directory and moves all data-model related packages over there: exemplar labels relabel rulefmt textparse timestamp value All the others are more or less utilities and have been moved to `util`: gate logging modetimevfs pool runtime Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Dieter Plaetinck	cda025b5b5	TSDB: demistify SeriesRefs and ChunkRefs (#9536 ) * TSDB: demistify seriesRefs and ChunkRefs The TSDB package contains many types of series and chunk references, all shrouded in uint types. Often the same uint value may actually mean one of different types, in non-obvious ways. This PR aims to clarify the code and help navigating to relevant docs, usage, etc much quicker. Concretely: * Use appropriately named types and document their semantics and relations. * Make multiplexing and demuxing of types explicit (on the boundaries between concrete implementations and generic interfaces). * Casting between different types should be free. None of the changes should have any impact on how the code runs. TODO: Implement BlockSeriesRef where appropriate (for a future PR) Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * feedback Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * agent: demistify seriesRefs and ChunkRefs Signed-off-by: Dieter Plaetinck <dieter@grafana.com>	3 years ago
Marco Pracucci	edd05d7010	Add Head.AppendableMinValidTime() (#9643 ) Signed-off-by: Marco Pracucci <marco@pracucci.com>	3 years ago
Serge Catudal	8c3eca84db	Fix remote write receiver endpoint for exemplars (#9414 ) Signed-off-by: Serge Catudal <serge.catudal@gmail.com>	3 years ago
Bryan Boreham	2327236bb5	Decrement active_appenders metric when no samples added (#9230 ) * Decrement active_appenders metric when no samples added Also add a test that the metric is incremented and decremented as expected with and without samples. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Fix comment Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Ganesh Vernekar	ee7e0071d1	Snapshot in-memory chunks on shutdown for faster restarts (#7229 ) Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Ganesh Vernekar	8002a3ab80	Breakdown tsdb/head.go into multiple files (#9147 ) Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago

25 Commits (2836e5513f1bc591535a859f5d41154a75e7c6bc)