prometheus

Commit Graph

Author	SHA1	Message	Date
gotjosh	4eca4dffb8	Allow metric metadata to be propagated via Remote Write. (#6815 ) * Introduce a metadata watcher Similarly to the WAL watcher, its purpose is to observe the scrape manager and pull metadata. Then, send it to a remote storage. Signed-off-by: gotjosh <josue@grafana.com> * Additional fixes after rebasing. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Rework samples/metadata metrics. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Use more descriptive variable names in MetadataWatcher collect. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix issues caused during rebasing. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix missing metric add and unneeded config code. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address some review comments. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix metrics and docs Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Replace assert with require Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Bring back max_samples_per_send metric Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix tests Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> Co-authored-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	4 years ago
Julien Pivotto	6c56a1faaa	Testify: move to require (#8122 ) * Testify: move to require Moving testify to require to fail tests early in case of errors. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> * More moves Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Bartlomiej Plotka	3d8826a3d4	MultiError: Refactored MultiError for more concise and safe usage. (#8066 ) * MultiError: Refactored MultiError for more concise and safe usage. * Less lines * Goland IDE was marking every usage of old MultiError "potential nil" error * It was easy to forgot using Err() when error was returned, now it's safely assured on compile time. NOTE: Potentially I would rename package to merrors. (: In different PR. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed review comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fix after rebase. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	4 years ago
Jorge Luis Betancourt	4dc755cd27	Add a metric for tracking max_samples_per_send (#8102 ) Currently there is no way of tracking the value of the `max_samples_per_send` configuration option, which is commonly tweaked when integrating with a remote write backend. Signed-off-by: Jorge Luis Betancourt Gonzalez <jorge-luis.betancourt@trivago.com>	4 years ago
Julien Pivotto	1282d1b39c	Refactor test assertions (#8110 ) * Refactor test assertions This pull request gets rid of assert.True where possible to use fine-grained assertions. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	4e5b1722b3	Move away from testutil, refactor imports (#8087 ) Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	8c9850c2bb	Remote: Do not collect non-initialized timestamp metrics (#8060 ) * Remote: Do not collect non-initialized timestamp metrics Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Frederic Branczyk	21a2e8c320	storage/remote/intern_test.go: Fix linting errors Signed-off-by: Frederic Branczyk <fbranczyk@gmail.com>	4 years ago
Siddhant Sinha	d9dcf2c90f	Fixes #7982 . Increase maxErrMsgLen value remote_write api (#8017 ) Signed-off-by: Siddhant Sinha <sid.sinha94@gmail.com>	4 years ago
Harkishen Singh	072b9649a3	Refactor vars to avoid test failures in storage/remote with -count > 1 (#7934 ) * Refactor global vars to avoid failure with run test more than once. Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com> * Register highestRecvTimestamp metric. Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com> * Use local interner vars. Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com> * Declare interner in write storage. Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>	4 years ago
Chris Marchbanks	f0f8e50567	Fix missing remote read spans (#7914 ) The remote read client needs to use the nethttp.Transport wrapper in order for spans to be instrumented properly. Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	4 years ago
Julien Pivotto	a6ee1f8517	Merge pull request #7913 from prometheus/release-2.21 Merge release 2.21 into master	4 years ago
Bartlomiej Plotka	088fcc9e48	Fixed iterator regression: Avoid using heap for each sample when iterating. (#7900 ) * Fixed iterator regression: Avoid using heap for each sample when iterating. Fixes: https://github.com/prometheus/prometheus/issues/7873 Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Check for .At() called after .Next() returned false Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * More comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	4 years ago
Bartlomiej Plotka	a399227a9f	Revert "Fixed iterator regression: Avoid using heap for each sample when iterating." This reverts commit `2c8b2c5915`.	4 years ago
Julien Pivotto	2c8b2c5915	Fixed iterator regression: Avoid using heap for each sample when iterating. Fixes: https://github.com/prometheus/prometheus/issues/7873 Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	4 years ago
Anand Sanmukhani	f4a1b3cef6	Rename `remote.newReadClient()` to `remote.NewReadClient()` (#7881 ) This should make the `NewReadClient()` exported outside the `remote` package Signed-off-by: Anand Sanmukhani <asanmukh@redhat.com>	4 years ago
showuon	dfdc358a5b	Fix the duplicated results issue from /api/v1/series (#7862 ) * Fix the duplicated results issue from /api/v1/series Signed-off-by: Luke Chen <showuon@gmail.com>	4 years ago
Robert Fratto	2bd077ed97	expose UserAgent to make it changeable by importers (#7832 ) Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 years ago
Joe Elliott	624075eafe	Exposes remote storage http client to allow for customization (#7831 ) * Exposes remote write client to allow for customizations to the http client Signed-off-by: Joe Elliott <number101010@gmail.com> * Removed changes to vendored file Signed-off-by: Joe Elliott <number101010@gmail.com>	4 years ago
Andy Bursavich	4e6a94a27d	Invert service discovery dependencies (#7701 ) This also fixes a bug in query_log_file, which now is relative to the config file like all other paths. Signed-off-by: Andy Bursavich <abursavich@gmail.com>	4 years ago
Julien Pivotto	a55c69c4c3	Apply gofmt -s on storage/remote/write_test.go (#7791 ) Noticed in https://goreportcard.com/report/github.com/prometheus/prometheus This is the only file. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Mucahit Kurt	869f1bc587	use remote.Storage in TestSampleDelivery instead of direct creation of QueueManager (#4758 ) Signed-off-by: Mucahit Kurt <mucahitkurt@gmail.com>	4 years ago
johncming	2b75c1b199	storage/remote: rename httpclient name. (#7747 ) Signed-off-by: johncming <johncming@yahoo.com>	4 years ago
Guangwen Feng	2c4a4548a8	Fix golint issue caused by incorrect func name (#7756 ) Signed-off-by: Guangwen Feng <fenggw-fnst@cn.fujitsu.com>	4 years ago
Bartlomiej Plotka	28c5cfaf0d	tsdb: Moved code merge series and iterators to differen files; cleanup. No functional changes just move! (#7714 ) I did not want to move those in previous PR to make it easier to review. Now small cleanup time for readability. (: ## Changes * Merge series goes to `storage/merge.go` leaving `fanout.go` for just fanout code. * Moved `fanout test` code from weird separate package to storage. * Unskiped one test: TestFanout_SelectSorted/chunk_querier * Moved block series set codes responsible for querying blocks to `querier.go` from `compact.go` Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	4 years ago
Julien Pivotto	9848e77678	storage/remote: remove unused code now that tsdb implements ChunkQuerier (#7715 ) * storage/remote: remove unused code now that tsdb implements ChunkQuerier Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> * encodeChunks is unused too Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Bartlomiej Plotka	e6d7cc5fa4	tsdb: Added ChunkQueryable implementations to db; unified MergeSeriesSets and vertical to single struct. (#7069 ) * tsdb: Added ChunkQueryable implementations to db; unified compactor, querier and fanout block iterating. Chained to https://github.com/prometheus/prometheus/pull/7059 * NewMerge(Chunk)Querier now takies multiple primaries allowing tsdb DB code to use it. * Added single SeriesEntry / ChunkEntry for all series implementations. * Unified all vertical, and non vertical for compact and querying to single merge series / chunk sets by reusing VerticalSeriesMergeFunc for overlapping algorithm (same logic as before) * Added block (Base/Chunk/)Querier for block querying. We then use populateAndTomb(Base/Chunk/) to iterate over chunks or samples. * Refactored endpoint tests and querier tests to include subtests. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed comments from Brian and Beorn. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fixed snapshot test and added chunk iterator support for DBReadOnly. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fixed race when iterating over Ats first. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fixed tests. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fixed populate block tests. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fixed endpoints test. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fixed test. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Added test & fixed case of head open chunk. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fixed DBReadOnly tests and bug producing 1 sample chunks. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Added cases for partial block overlap for multiple full chunks. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Added extra tests for chunk meta after compaction. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fixed small vertical merge bug and added more tests for that. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	4 years ago
Annanay	ec562f152b	Merge branch 'master' into appender-context Signed-off-by: Annanay <annanayagarwal@gmail.com>	4 years ago
Annanay	9bba8a6eae	Merge branch 'master' into appender-context Signed-off-by: Annanay <annanayagarwal@gmail.com>	4 years ago
Annanay	89129cd39a	Address comments Signed-off-by: Annanay <annanayagarwal@gmail.com>	4 years ago
Javier Palomo Almena	b58a613443	Replace sync/atomic with uber-go/atomic (#7683 ) * storage: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * tsdb: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * web: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * notifier: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * cmd: Replace usage of sync/atomic with uber-go/atomic Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * scripts: Verify that we are not using restricted packages It checks that we are not directly importing 'sync/atomic'. Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * Reorganise imports in blocks Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * notifier/test: Apply PR suggestions Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * storage/remote: avoid storing references on newEntry Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * Revert "scripts: Verify that we are not using restricted packages" This reverts commit `278d32748e`. Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com> * web: Group imports accordingly Signed-off-by: Javier Palomo <javier.palomo.almena@gmail.com>	4 years ago
Joe Elliott	04b028f1e6	Exports recoverable error (#7689 ) Signed-off-by: Joe Elliott <number101010@gmail.com>	4 years ago
Annanay	f40e4579b7	gofmt Signed-off-by: Annanay <annanayagarwal@gmail.com>	4 years ago
Annanay	7f98a744e5	Add context to Appender interface Signed-off-by: Annanay <annanayagarwal@gmail.com>	4 years ago
Zhou Hao	ddedf454d0	add os.RemoveAll err verification (#7540 ) * add os.RemoveAll err verification for watcher_test Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com> * add os.RemoveAll err verification for db_test Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com> * add os.RemoveAll err verification for write_test Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com> * add os.RemoveAll err verification for queue_manager_test Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com> * tsdb/wal/watcher_test: add close operation before delete Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com>	4 years ago
Hu Shuai	b962029031	Add a unit test for MergeLabels in storage/remote/codec.go. (#7499 ) This PR is about adding a unit test for MergeLabels in storage/remote/codec.go. Signed-off-by: Hu Shuai <hus.fnst@cn.fujitsu.com>	4 years ago
Ganesh Vernekar	a4c2ea1ca3	Merge remote-tracking branch 'upstream/master' into merge-release-2.19 Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	4 years ago
Chris Marchbanks	b299aba6cf	Fix panic when updating a remote write queue (#7452 ) Right now Queue Manager metrics are registered when the metrics struct is created, which happens before a changed queue is shutdown and the old metrics are unregistered. In the case of named queues or updates to external labels the apply config will panic due to duplicate metrics. Instead, register the metrics as part of starting the queue as we always guarantee that Stop will be called before a new Start. Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	4 years ago
Chris Marchbanks	d78656c244	Pending Samples metric includes samples in channel (#7335 ) * Pending Samples metric includes samples in channel The pending samples metric should also include samples waiting in the channels to be sent to provide a more accurate measure. In addition, make sure that the pending samples is reset to 0 anytime a queue is started as we remake all of the shards at that time. Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com> * Log the number of dropped samples on hard shutdown Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	4 years ago
Bartlomiej Plotka	b788986717	storage: Adjusted fully storage layer support for chunk iterators: Remote read client, readyStorage, fanout. (#7059 ) * Fixed nits introduced by https://github.com/prometheus/prometheus/pull/7334 * Added ChunkQueryable implementation to fanout and readyStorage. * Added more comments. * Changed NewVerticalChunkSeriesMerger to CompactingChunkSeriesMerger, removed tiny interface by reusing VerticalSeriesMergeFunc for overlapping algorithm for both chunks and series, for both querying and compacting (!) + made sure duplicates are merged. * Added ErrChunkSeriesSet * Added Samples interface for seamless []promb.Sample to []tsdbutil.Sample conversion. * Deprecating non chunks serieset based StreamChunkedReadResponses, added chunk one. * Improved tests. * Split remote client into Write (old storage) and read. * Queryable client is now SampleAndChunkQueryable. Since we cannot use nice QueryableFunc I moved all config based options to sampleAndChunkQueryableClient to aboid boilerplate. In next commit: Changes for TSDB. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	4 years ago
Nicole J	d5a8f2afc4	Added the remote read histogram (#7334 ) change remote read queries total metric to a histogram and add read requests counter with status code Signed-off-by: njingco <jingco.nicole@gmail.com>	5 years ago
Kemal Akkoyun	66dfb951c4	: Consistent Error/Warning handling for SeriesSet iterator: Allowing Async Select (#7251 ) Add errors and Warnings to SeriesSet Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Change Querier interface and refactor accordingly Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Refactor promql/engine to propagate warnings at eval stage Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Address review issues Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Make sure all the series from all Selects are pre-advanced Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Address review issues Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Separate merge series sets Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Clean Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Refactor merge querier failure handling Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Refactored and simplified fanout with improvements from incoming chunk iterator PRs. * Secondary logic is hidden, instead of weird failed series set logic we had. * Fanout is well commented * Fanout closing record all errors * MergeQuerier improved API (clearer) * deferredGenericMergeSeriesSet is not needed as we return no samples anyway for failed series sets (next = false). Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fix formatting Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Fix CI issues Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Added final tests for error handling. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed Brian's comments. * Moved hints in populate to be allocated only when needed. * Used sync.Once in secondary Querier to achieve all-or-nothing partial response logic. * Select after first Next is done will panic. NOTE: in lazySeriesSet in theory we could just panic, I think however we can totally just return error, it will panic in expand anyway. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Utilize errWithWarnings Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Fix recently introduced expansion issue Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Add tests for secondary querier error handling Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Implement lazy merge Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Add name to test cases Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Reorganize Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Address review comments Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Address review comments Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Remove redundant warnings Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Fix rebase mistake Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Bartlomiej Plotka	8f0a924be9	Merge branch 'release-2.18' into get2.18.2-to-master	5 years ago
Marco Pracucci	bc1d7d7f27	Fixed incorrect query results caused by buffer reuse in merge adapter (#7361 ) Signed-off-by: Marco Pracucci <marco@pracucci.com>	5 years ago
Marco Pracucci	1e006d84a0	Fixed incorrect query results caused by buffer reuse in merge adapter (#7361 ) Signed-off-by: Marco Pracucci <marco@pracucci.com>	5 years ago
Bert Hartmann	82c7cd320a	increase the remote write bucket range (#7323 ) * increase the remote write bucket range Increase the range of remote write buckets to capture times above 10s for laggy scenarios Buckets had been: {.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10} Buckets are now: {0.03125, 0.0625, 0.125, 0.25, 0.5, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512} Signed-off-by: Bert Hartmann <berthartm@gmail.com> * revert back to DefBuckets with addons to be backwards compatible Signed-off-by: Bert Hartmann <berthartm@gmail.com> * shuffle the buckets to maintain 2-2.5x increases Signed-off-by: Bert Hartmann <berthartm@gmail.com>	5 years ago
Nicole J	5e9bd17b1f	added the prometheus_remote_storage_remote_read_queries_total (#7328 ) * added the prometheus_remote_storage_remote_read_queries_total query Signed-off-by: njingco <jingco.nicole@gmail.com> * adjusted the help label of remoteReadQueriesTotal Signed-off-by: njingco <jingco.nicole@gmail.com>	5 years ago
Cody Boggs	3268eac2dd	Trace Remote Write requests (#7206 ) * Trace Remote Write requests Signed-off-by: Cody Boggs <cboggs@splunk.com> * Refactor store attempts to keep code flow clearer, and avoid so many places to deal with span finishing Signed-off-by: Cody Boggs <cboggs@splunk.com>	5 years ago
Chris Marchbanks	c1f9917e90	Add test for unregistering queue manager metrics Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	5 years ago
Chris Marchbanks	dfad1da296	Remove duplicate metrics in QueueManager Right now any new metrics added for remote write need to be added to both the QueueManager struct, and the queueManagerMetrics struct. Instead, use the queueManagerMetrics struct directly from QueueManager. The newQueueManagerMetrics constructor will now create the metrics for a specific queue with name and endpoint pre-populated, and a new copy of the struct will be created specifically for each queue. This also fixes a bug where prometheus_remote_storage_sent_bytes_total is not being unregistered after a queue is changed. Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	5 years ago

1 2 3 4 5 ...

1099 Commits (c8f810083d3f6a8b5750186bda13f406980c4218)