prometheus

Commit Graph

Author	SHA1	Message	Date
lzhfromustc	27a6e1e174	test: add buffer to channel to avoid goroutine leak (#8274 ) Signed-off-by: lzhfromustc <lzhfromustc@gmail.com>	4 years ago
Julien Pivotto	7957731339	Inline defer Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	82b5f1d8b1	Backfill: Use mmap to reuse parser code Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Atibhi Agrawal	b317b6ab9c	Backfill from OpenMetrics format (#8084 ) * get parser working Signed-off-by: aSquare14 <atibhi.a@gmail.com> * import file created Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Find min and max ts Signed-off-by: aSquare14 <atibhi.a@gmail.com> * make two passes over file and write to tsdb Signed-off-by: aSquare14 <atibhi.a@gmail.com> * print error messages Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Fix Max and Min initializer Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Start with unit tests Signed-off-by: aSquare14 <atibhi.a@gmail.com> * reset file read Signed-off-by: aSquare14 <atibhi.a@gmail.com> * align blocks to two hour range Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Add cleanup test Signed-off-by: aSquare14 <atibhi.a@gmail.com> * remove .ds_store Signed-off-by: aSquare14 <atibhi.a@gmail.com> * add license to import_test Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Fix Circle CI error Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Refactor code Move backfill from tsdb to promtool directory Signed-off-by: aSquare14 <atibhi.a@gmail.com> * fix gitignore Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Remove panic Rename ContenType Signed-off-by: aSquare14 <atibhi.a@gmail.com> * adjust mint Signed-off-by: aSquare14 <atibhi.a@gmail.com> * fix return statement Signed-off-by: aSquare14 <atibhi.a@gmail.com> * fix go modules Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Added unit test for backfill Signed-off-by: aSquare14 <atibhi.a@gmail.com> * fix CI error Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Fix file handling Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Close DB Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Close directory Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Error Handling Signed-off-by: aSquare14 <atibhi.a@gmail.com> * inline err Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Fix command line flags Signed-off-by: aSquare14 <atibhi.a@gmail.com> * add spaces before func fix pointers Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Add defer'd calls Signed-off-by: aSquare14 <atibhi.a@gmail.com> * move openmetrics.go content to backfill Signed-off-by: aSquare14 <atibhi.a@gmail.com> * changed args to flags Signed-off-by: aSquare14 <atibhi.a@gmail.com> * add tests for wrong OM files Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Added additional tests Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Add comment to warn of func reuse Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Make input required in main.go Signed-off-by: aSquare14 <atibhi.a@gmail.com> * defer blockwriter close Signed-off-by: aSquare14 <atibhi.a@gmail.com> * fix defer Signed-off-by: aSquare14 <atibhi.a@gmail.com> * defer Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Remove contentType Signed-off-by: aSquare14 <atibhi.a@gmail.com> * remove defer from backfilltest Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Fix defer remove in backfill_test Signed-off-by: aSquare14 <atibhi.a@gmail.com> * changes to fix CI errors Signed-off-by: aSquare14 <atibhi.a@gmail.com> * fix go.mod Signed-off-by: aSquare14 <atibhi.a@gmail.com> * change package name Signed-off-by: aSquare14 <atibhi.a@gmail.com> * assert->require Signed-off-by: aSquare14 <atibhi.a@gmail.com> * remove todo Signed-off-by: aSquare14 <atibhi.a@gmail.com> * fix format Signed-off-by: aSquare14 <atibhi.a@gmail.com> * fix todo Signed-off-by: aSquare14 <atibhi.a@gmail.com> * fix createblock Signed-off-by: aSquare14 <atibhi.a@gmail.com> * fix tests Signed-off-by: aSquare14 <atibhi.a@gmail.com> * fix defer Signed-off-by: aSquare14 <atibhi.a@gmail.com> * fix return Signed-off-by: aSquare14 <atibhi.a@gmail.com> * check err for anon func Signed-off-by: aSquare14 <atibhi.a@gmail.com> * change comments Signed-off-by: aSquare14 <atibhi.a@gmail.com> * update comment Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Fix for the Flush Bug Signed-off-by: aSquare14 <atibhi.a@gmail.com> * fix formatting, comments, names Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Print Blocks Signed-off-by: aSquare14 <atibhi.a@gmail.com> * cleanup Signed-off-by: aSquare14 <atibhi.a@gmail.com> * refactor test to take care of multiple samples Signed-off-by: aSquare14 <atibhi.a@gmail.com> * refactor tests Signed-off-by: aSquare14 <atibhi.a@gmail.com> * remove om Signed-off-by: aSquare14 <atibhi.a@gmail.com> * I dont know what I fixed Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Fix tests Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Fix tests, add test description, print blocks Signed-off-by: aSquare14 <atibhi.a@gmail.com> * commit after 5000 samples Signed-off-by: aSquare14 <atibhi.a@gmail.com> * reviews part 1 Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Series Count Signed-off-by: aSquare14 <atibhi.a@gmail.com> * fix CI Signed-off-by: aSquare14 <atibhi.a@gmail.com> * remove extra func Signed-off-by: aSquare14 <atibhi.a@gmail.com> * make timestamp into sec Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Reviews 2 Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Add Todo Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Fixes Signed-off-by: aSquare14 <atibhi.a@gmail.com> * fixes reviews Signed-off-by: aSquare14 <atibhi.a@gmail.com> * =0 Signed-off-by: aSquare14 <atibhi.a@gmail.com> * remove backfill.om Signed-off-by: aSquare14 <atibhi.a@gmail.com> * add global err var, remove stuff Signed-off-by: aSquare14 <atibhi.a@gmail.com> * change var name Signed-off-by: aSquare14 <atibhi.a@gmail.com> * sampleLimit pass as parameter Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Add test when number of samples greater than batch size Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Change name of batchsize Signed-off-by: aSquare14 <atibhi.a@gmail.com> * revert export Signed-off-by: aSquare14 <atibhi.a@gmail.com> * nits Signed-off-by: aSquare14 <atibhi.a@gmail.com> * remove Signed-off-by: aSquare14 <atibhi.a@gmail.com> * add comment, remove newline,consistent err Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Print Blocks Signed-off-by: aSquare14 <atibhi.a@gmail.com> * Modify comments Signed-off-by: aSquare14 <atibhi.a@gmail.com> * db.Querier Signed-off-by: aSquare14 <atibhi.a@gmail.com> * add sanity check , get maxt and mint Signed-off-by: aSquare14 <atibhi.a@gmail.com> * ci error Signed-off-by: aSquare14 <atibhi.a@gmail.com> * fix Signed-off-by: aSquare14 <atibhi.a@gmail.com> * comment change Signed-off-by: aSquare14 <atibhi.a@gmail.com> * nits Signed-off-by: aSquare14 <atibhi.a@gmail.com> * NoError Signed-off-by: aSquare14 <atibhi.a@gmail.com> * fix Signed-off-by: aSquare14 <atibhi.a@gmail.com> * fix Signed-off-by: aSquare14 <atibhi.a@gmail.com>	4 years ago
gotjosh	4eca4dffb8	Allow metric metadata to be propagated via Remote Write. (#6815 ) * Introduce a metadata watcher Similarly to the WAL watcher, its purpose is to observe the scrape manager and pull metadata. Then, send it to a remote storage. Signed-off-by: gotjosh <josue@grafana.com> * Additional fixes after rebasing. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Rework samples/metadata metrics. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Use more descriptive variable names in MetadataWatcher collect. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix issues caused during rebasing. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix missing metric add and unneeded config code. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address some review comments. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix metrics and docs Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Replace assert with require Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Bring back max_samples_per_send metric Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix tests Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> Co-authored-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	4 years ago
Julien Pivotto	6c56a1faaa	Testify: move to require (#8122 ) * Testify: move to require Moving testify to require to fail tests early in case of errors. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> * More moves Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Bartlomiej Plotka	3d8826a3d4	MultiError: Refactored MultiError for more concise and safe usage. (#8066 ) * MultiError: Refactored MultiError for more concise and safe usage. * Less lines * Goland IDE was marking every usage of old MultiError "potential nil" error * It was easy to forgot using Err() when error was returned, now it's safely assured on compile time. NOTE: Potentially I would rename package to merrors. (: In different PR. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed review comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fix after rebase. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	4 years ago
Julien Pivotto	1282d1b39c	Refactor test assertions (#8110 ) * Refactor test assertions This pull request gets rid of assert.True where possible to use fine-grained assertions. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
David Leadbeater	e7e60623ff	promtool: Calculate mint and maxt per test (#8096 ) * promtool: Calculate mint and maxt per test Previously a single test that used a later eval time would make all other tests in the file share the [mint, maxt] and potentially evaluate far more samples than needed. Fixes: #8019 Signed-off-by: David Leadbeater <dgl@dgl.cx>	4 years ago
Julien Pivotto	4e5b1722b3	Move away from testutil, refactor imports (#8087 ) Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Björn Rabenstein	71577e45eb	Merge pull request #8044 from prometheus/beorn7/metrics Instrumentation: Report valid configs in the respective metrics from the beginning	4 years ago
Arthur Silva Sens	4f45e201cc	Promtool tsdb list now prints block sizes (#7993 ) * promtool tsdb list now prints blocks' size Signed-off-by: arthursens <arthursens2005@gmail.com>	4 years ago
beorn7	0f3c1bf6cf	Report valid configs in the respective metrics from the beginning In #7399, an early validity check of the config was introduced to prevent the scenario where an invalid config is only detected after a possibly very long startup procedure. However, the respective success metrics are not updated after the initial validation so that the success metrics suggest an invalid config. If the startup procedure, like replaying the WAL, really takes very long, alerts about invalid config will trigger. This commit sets the succes metrics after initial validation. They will be set again after the "real" config (re-)load, but that shouldn't be a problem. The metric now truthfully represents whenever the config was successfully loaded, no matter if the result was then thrown away (because it was just for validation) or actually used. Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
Frederic Branczyk	da3ea43242	Merge pull request #7976 from roidelapluie/tolerance Introduce timestamp tolerance in scrapes	4 years ago
Julien Pivotto	be5ba1a62d	Fix wordings Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	4617d16b4b	Specify the removal Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	e2a2bf3c06	Add context Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	627ff84599	Adjust flag Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	6b618ecf02	Better description Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	536dfb6234	Add an experimental, hidden flag Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Frederic Branczyk	6be3ebdfe7	Merge pull request #8015 from simonpasquier/bump-k8s-deps Bump k8s dependencies + support k8s.io/klog/v2	4 years ago
Julien Pivotto	946819e16e	cmd/prometheus: Issue a warning on 32 bit archs (#8012 ) * cmd/prometheus: Issue a warning on 32 bit archs Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Simon Pasquier	9bb3555fe4	cmd/prometheus: support k8s.io/klog/v2 Signed-off-by: Simon Pasquier <spasquie@redhat.com>	4 years ago
David Leadbeater	77c784ac93	Ensure alert rules are marked as restored in unit tests (#7661 ) This makes sure the ALERTS timeseries is created when unit testing alerting rules. Signed-off-by: David Leadbeater <dgl@dgl.cx>	4 years ago
Julien Pivotto	442b3364d7	Promtool: add evaluation time to instant query (#7829 ) * Promtool: add evaluation time to instant query Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> * Apply suggestion Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Andy Bursavich	4e6a94a27d	Invert service discovery dependencies (#7701 ) This also fixes a bug in query_log_file, which now is relative to the config file like all other paths. Signed-off-by: Andy Bursavich <abursavich@gmail.com>	4 years ago
Harold Dost	21a753c4e2	Make file permissions set to allow for wider umask options. (#7782 ) 0644 -> 0666 on all non vendored code. Fixes #7717 Signed-off-by: Harold Dost <harolddost@gmail.com>	4 years ago
Julien Pivotto	d661f84748	Log duration of reloads Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Annanay	9bba8a6eae	Merge branch 'master' into appender-context Signed-off-by: Annanay <annanayagarwal@gmail.com>	4 years ago
Julien Pivotto	01e3bfcd1a	Add warnings about NFS (#7691 ) * Add warnings about NFS Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Javier Palomo Almena	b58a613443	Replace sync/atomic with uber-go/atomic (#7683 ) * storage: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * tsdb: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * web: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * notifier: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * cmd: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * scripts: Verify that we are not using restricted packages It checks that we are not directly importing 'sync/atomic'. Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * Reorganise imports in blocks Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * notifier/test: Apply PR suggestions Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * storage/remote: avoid storing references on newEntry Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * Revert "scripts: Verify that we are not using restricted packages" This reverts commit `278d32748e`. Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * web: Group imports accordingly Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com>	4 years ago
Annanay	7f98a744e5	Add context to Appender interface Signed-off-by: Annanay <annanayagarwal@gmail.com>	4 years ago
chinhnc	e05c19da5d	Display block duration in promtool list blocks command (#7653 ) * Update tsdb.go Added DURATION column to `tsdb list` command Signed-off-by: soup <chicknsoupuds@gmail.com> * Use time.Duration instead of hardcoded hour Signed-off-by: soup <chicknsoupuds@gmail.com>	4 years ago
Ben Ye	50c261502e	add tsdb cmds into promtool (#6088 ) Signed-off-by: yeya24 <yb532204897@gmail.com> update tsdb cli in makefile and promu Signed-off-by: yeya24 <yb532204897@gmail.com> remove building tsdb bin Signed-off-by: yeya24 <yb532204897@gmail.com> remove useless func Signed-off-by: yeya24 <yb532204897@gmail.com> refactor analyzeBlock Signed-off-by: yeya24 <yb532204897@gmail.com> Fix Makefile Signed-off-by: Simon Pasquier <spasquie@redhat.com>	4 years ago
Bartlomiej Plotka	a0df8a383a	promql: Removed global and add ability to have better interval for subqueries if not specified (#7628 ) * promql: Removed global and add ability to have better interval for subqueries if not specified ## Changes * Refactored tests for better hints testing * Added various TODO in places to enhance. * Moved DefaultEvalInterval global to opts with func(rangeMillis int64) int64 function instead Motivation: At Thanos we would love to have better control over the subqueries step/interval. This is important to choose proper resolution. I think having proper step also does not harm for Prometheus and remote read users. Especially on stateless querier we do not know evaluation interval and in fact putting global can be wrong to assume for Prometheus even. I think ideally we could try to have at least 3 samples within the range, the same way Prometheus UI and Grafana assumes. Anyway this interfaces allows to decide on promQL user basis. Open question: Is taking parent interval a smart move? Motivation for removing global: I spent 1h fighting with: === RUN TestEvaluations TestEvaluations: promql_test.go:31: unexpected error: error evaluating query "absent_over_time(rate(nonexistant[5m])[5m:])" (line 687): unexpected error: runtime error: integer divide by zero --- FAIL: TestEvaluations (0.32s) FAIL At the end I found that this fails on most of the versions including this master if you run this test alone. If run together with many other tests it passes. This is due to SetDefaultEvaluationInterval(1 * time.Minute) in test that is ran before TestEvaluations. Thanks to globals (: Let's fix it by dropping this global. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Added issue links for TODOs. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Removed irrelevant changes. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	4 years ago
Julien Pivotto	b83cbacbdd	Rule manager: remove blocking channel in mail (#7631 ) Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Ben Ye	e6ea798c32	promtool range query should exit when fail to parse time (#7505 ) Signed-off-by: yeya24 <yb532204897@gmail.com>	4 years ago
yeya24	797e48c1a3	support time range in promtool query labels Updated prometheus/client_golang and json-iterator/go Signed-off-by: yeya24 <yb532204897@gmail.com>	4 years ago
Frederic Branczyk	d17d88935c	rules: Use narrower interface for rule manager loading of for state (#7472 ) To load ALERT_FOR_STATE only `storage.Queryable` interface is required, so this patch uses this narrower interface for to perform this. Signed-off-by: Frederic Branczyk <fbranczyk@gmail.com>	4 years ago
Bartlomiej Plotka	b788986717	storage: Adjusted fully storage layer support for chunk iterators: Remote read client, readyStorage, fanout. (#7059 ) * Fixed nits introduced by https://github.com/prometheus/prometheus/pull/7334 * Added ChunkQueryable implementation to fanout and readyStorage. * Added more comments. * Changed NewVerticalChunkSeriesMerger to CompactingChunkSeriesMerger, removed tiny interface by reusing VerticalSeriesMergeFunc for overlapping algorithm for both chunks and series, for both querying and compacting (!) + made sure duplicates are merged. * Added ErrChunkSeriesSet * Added Samples interface for seamless []promb.Sample to []tsdbutil.Sample conversion. * Deprecating non chunks serieset based StreamChunkedReadResponses, added chunk one. * Improved tests. * Split remote client into Write (old storage) and read. * Queryable client is now SampleAndChunkQueryable. Since we cannot use nice QueryableFunc I moved all config based options to sampleAndChunkQueryableClient to aboid boilerplate. In next commit: Changes for TSDB. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	4 years ago
Harkishen Singh	70b0a34616	Exit early on invalid config file (#7399 ) * Reload config file at start Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com> * relocated config checking Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com> * change log lever Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com> * add helpful comment Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>	5 years ago
Ben Kochie	8d3c2f6829	Enable WAL compression by default (#7410 ) Enable the `--storage.tsdb.wal-compression` flag by defualt. Signed-off-by: Ben Kochie <superq@gmail.com>	5 years ago
Jordan Neufeld	268b4c29e1	Support extended durations in promtool unit tests (Fixes #6285 ) (#6297 ) * Fixed evaluation_time duration parsing in promtool unit tests (Fixes #6285) Signed-off-by: Jordan Neufeld <jordan@neufeldtech.com>	5 years ago
Arthur Silva Sens	7727b9012e	Correction of misleading help text(#5142 ) (#7231 ) * Correction of misleading help text(#5142) Signed-off-by: arthursens <arthursens2005@gmail.com>	5 years ago
Julien Pivotto	9e265aba10	Merge pull request #7225 from prometheus/release-2.18 [Merge without Squash] Merge release-2.18 back to master for 2.18.1 fixes.	5 years ago
Hongcai Ren	c7e82274c6	replace github.com/prometheus/prometheus/testutil/promlint by github.com/prometheus/client_golang/prometheus/testutil/promlint from our codebase (#7209 ) Signed-off-by: RainbowMango <renhongcai@huawei.com>	5 years ago
Julien Pivotto	645b71e9ef	Fix snapshots (#7217 ) Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Ganesh Vernekar	d4b9fe801f	M-map full chunks of Head from disk (#6679 ) When appending to the head and a chunk is full it is flushed to the disk and m-mapped (memory mapped) to free up memory Prom startup now happens in these stages - Iterate the m-maped chunks from disk and keep a map of series reference to its slice of mmapped chunks. - Iterate the WAL as usual. Whenever we create a new series, look for it's mmapped chunks in the map created before and add it to that series. If a head chunk is corrupted the currpted one and all chunks after that are deleted and the data after the corruption is recovered from the existing WAL which means that a corruption in m-mapped files results in NO data loss. [Mmaped chunks format](https://github.com/prometheus/prometheus/blob/master/tsdb/docs/format/head_chunks.md) - main difference is that the chunk for mmaping now also includes series reference because there is no index for mapping series to chunks. [The block chunks](https://github.com/prometheus/prometheus/blob/master/tsdb/docs/format/chunks.md) are accessed from the index which includes the offsets for the chunks in the chunks file - example - chunks of series ID have offsets 200, 500 etc in the chunk files. In case of mmaped chunks, the offsets are stored in memory and accessed from that. During WAL replay, these offsets are restored by iterating all m-mapped chunks as stated above by matching the series id present in the chunk header and offset of that chunk in that file. Prombench results _WAL Replay_ 1h Wal reply time 30% less wal reply time - 4m31 vs 3m36 2h Wal reply time 20% less wal reply time - 8m16 vs 7m _Memory During WAL Replay_ High Churn: 10-15% less RAM - 32gb vs 28gb 20% less RAM after compaction 34gb vs 27gb No Churn: 20-30% less RAM - 23gb vs 18gb 40% less RAM after compaction 32.5gb vs 20gb Screenshots are in [this comment](https://github.com/prometheus/prometheus/pull/6679#issuecomment-621678932) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Ben Ye	1e4e37144d	Fixed wrongly handled not ready TSDB on web and API. (#7182 ) * fix federate endpoint panic Signed-off-by: yeya24 <yb532204897@gmail.com> * Fixed all cases of not ready TSDB being wrongly handled. * Fixed issue for federation. * Ensured this will never happen again thanks to interfaces * Fixes same issue for stats. * Added tests for readiness. * Fixed bug in stats. It was: status.MaxTime = db.Head().MaxTime() status.MinTime = db.Head().MaxTime() Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed Brian's comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed Brian's comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Vasily Sliouniaev	0393b188c9	Add Jaeger (#7148 ) * Trace remote read Signed-off-by: vas <vasily.sliouniaev@jet.com> * Use jaeger Signed-off-by: vas <vasily.sliouniaev@jet.com>	5 years ago

1 2 3 4 5 ...

405 Commits (e5478983c43cb839a77c144f953b9ea25783b995)