prometheus

Commit Graph

Author	SHA1	Message	Date
Mateusz Gozdek	79a753db7e	web: randomize used port in TestAgentAPIEndPoints test To make sure we hit the right target and allow running tests in parallel. Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>	3 years ago
Mateusz Gozdek	b2c933e317	web: randomize used listening ports for tests So they can be parallelized to speed up tests execution. Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>	3 years ago
Robert Fratto	bc72a718c4	Initial draft of prometheus-agent (#8785 ) * Initial draft of prometheus-agent This commit introduces a new binary, prometheus-agent, based on the Grafana Agent code. It runs a WAL-only version of prometheus without the TSDB, alerting, or rule evaluations. It is intended to be used to remote_write to Prometheus or another remote_write receiver. By default, prometheus-agent will listen on port 9095 to not collide with the prometheus default of 9090. Truncation of the WAL cooperates on a best-effort case with Remote Write. Every time the WAL is truncated, the minimum timestamp of data to truncate is determined by the lowest sent timestamp of all samples across all remote_write endpoints. This gives loose guarantees that data from the WAL will not try to be removed until the maximum sample lifetime passes or remote_write starts functionining. Signed-off-by: Robert Fratto <robertfratto@gmail.com> * add tests for Prometheus agent (#22) * add tests for Prometheus agent * add tests for Prometheus agent * rearranged tests as per the review comments * update tests for Agent * changes as per code review comments Signed-off-by: SriKrishna Paparaju <paparaju@gmail.com> * incremental changes to prometheus agent Signed-off-by: SriKrishna Paparaju <paparaju@gmail.com> * changes as per code review comments Signed-off-by: SriKrishna Paparaju <paparaju@gmail.com> * Commit feedback from code review Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Robert Fratto <robertfratto@gmail.com> * Port over some comments from grafana/agent Signed-off-by: Robert Fratto <robertfratto@gmail.com> * Rename agent.Storage to agent.DB for tsdb consistency Signed-off-by: Robert Fratto <robertfratto@gmail.com> * Consolidate agentMode ifs in cmd/prometheus/main.go Signed-off-by: Robert Fratto <robertfratto@gmail.com> * Document PreAction usage requirements better for agent mode flags Signed-off-by: Robert Fratto <robertfratto@gmail.com> * remove unnecessary defaultListenAddr Signed-off-by: Robert Fratto <robertfratto@gmail.com> * `go fmt ./tsdb/agent` and fix lint errors Signed-off-by: Robert Fratto <robertfratto@gmail.com> Co-authored-by: SriKrishna Paparaju <paparaju@gmail.com>	3 years ago
Levi Harrison	7bc11dcb06	React UI: Add Starting Screen (#8662 ) * Added walreplay API endpoint Signed-off-by: Levi Harrison <git@leviharrison.dev> * Added starting page to react-ui Signed-off-by: Levi Harrison <git@leviharrison.dev> * Documented the new endpoint Signed-off-by: Levi Harrison <git@leviharrison.dev> * Fixed typos Signed-off-by: Levi Harrison <git@leviharrison.dev> Co-authored-by: Julius Volz <julius.volz@gmail.com> * Removed logo Signed-off-by: Levi Harrison <git@leviharrison.dev> * Changed isResponding to isUnexpected Signed-off-by: Levi Harrison <git@leviharrison.dev> * Changed width of progress bar Signed-off-by: Levi Harrison <git@leviharrison.dev> * Changed width of progress bar Signed-off-by: Levi Harrison <git@leviharrison.dev> * Added DB stats object Signed-off-by: Levi Harrison <git@leviharrison.dev> * Updated starting page to work with new fields Signed-off-by: Levi Harrison <git@leviharrison.dev> * Passing nil Signed-off-by: Levi Harrison <git@leviharrison.dev> * Passing nil (pt. 2) Signed-off-by: Levi Harrison <git@leviharrison.dev> * Passing nil (pt. 3) Signed-off-by: Levi Harrison <git@leviharrison.dev> * Passing nil (and also implementing a method this time) (pt. 4) Signed-off-by: Levi Harrison <git@leviharrison.dev> * Passing nil (and also implementing a method this time) (pt. 5) Signed-off-by: Levi Harrison <git@leviharrison.dev> * Changed const to let Signed-off-by: Levi Harrison <git@leviharrison.dev> * Passing nil (pt. 6) Signed-off-by: Levi Harrison <git@leviharrison.dev> * Remove SetStats method Signed-off-by: Levi Harrison <git@leviharrison.dev> * Added comma Signed-off-by: Levi Harrison <git@leviharrison.dev> * Changed api Signed-off-by: Levi Harrison <git@leviharrison.dev> * Changed to triple equals Signed-off-by: Levi Harrison <git@leviharrison.dev> * Fixed data response types Signed-off-by: Levi Harrison <git@leviharrison.dev> * Don't return pointer Signed-off-by: Levi Harrison <git@leviharrison.dev> * Changed version Signed-off-by: Levi Harrison <git@leviharrison.dev> * Fixed interface issue Signed-off-by: Levi Harrison <git@leviharrison.dev> * Fixed pointer Signed-off-by: Levi Harrison <git@leviharrison.dev> * Fixed copying lock value error Signed-off-by: Levi Harrison <git@leviharrison.dev> Co-authored-by: Julius Volz <julius.volz@gmail.com>	4 years ago
Julien Pivotto	5b4f46a348	Add TLS and basic authentication Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Ben Kochie	5055dfbbe4	Listen on web early in startup Avoid starting up components like the TSDB if we can't bind to the web listening port. Signed-off-by: Ben Kochie <superq@gmail.com>	4 years ago
Julien Pivotto	cda52234eb	Fix panic with double close() of channel on /-/quit/ (#8166 ) Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julius Volz	3470ee1fbf	Make React UI the default, keep old UI under /classic (#8142 ) The React app's assets are now served under /assets, while all old custom web assets (including the ones for console templates) are now served from /classic/static. I tested different combinations of --web.external-url and --web.route-prefix with proxies in front, and I couldn't find a problem yet with the routing. Console templates also still work. While migrating old endpoints to /classic, I noticed that /version was being treated like a lot of the old UI pages, with readiness check handler in front of it, etc. I kept it in /version and removed that readiness wrapper, since it doesn't seem to be needed for that endpoint. Signed-off-by: Julius Volz <julius.volz@gmail.com>	4 years ago
Julien Pivotto	6c56a1faaa	Testify: move to require (#8122 ) * Testify: move to require Moving testify to require to fail tests early in case of errors. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> * More moves Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	1282d1b39c	Refactor test assertions (#8110 ) * Refactor test assertions This pull request gets rid of assert.True where possible to use fine-grained assertions. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	4e5b1722b3	Move away from testutil, refactor imports (#8087 ) Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	e208afcc95	web: Remove APIv2 (#7935 ) * web: Remove APIv2 Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	6573bf42f2	web: Close and exhaust all response bodies in tests + cleanup snapshot directory (#7850 ) * web: Close and exhaust all response bodies in tests Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> * Remove empty snapshot directory Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> * Fix lint Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Li Zhijian	369245e606	cleanup one more tempfile (#7839 ) Introduced by a recent PR(#7810) Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>	4 years ago
Li Zhijian	6f6ef1451e	Cleanup tempfiles (#7696 ) * cleanup tempfiles for web_test Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com> * cleanup tempfiles for api_test Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com> * cleanup tempfiles for file_test Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>	4 years ago
Julien Pivotto	e96d786fb8	Stop CMUX and GRPC servers even with stale connections (#7810 ) I have received a recent report for a user which confirms that sometime the GRPC server does not stop propery. It appears that there are 2 issues: 1. The cmux server can refuse to stop if there are stale connections. For that we set the ReadTimeout. 2. The GRPC server graceful stop can never finish. What this PR avoids is: ``` goroutine 227 [semacquire, 2 minutes]: sync.runtime_Semacquire(0xc00059a75c) /usr/local/go/src/runtime/sema.go:56 +0x42 sync.(WaitGroup).Wait(0xc00059a75c) /usr/local/go/src/sync/waitgroup.go:130 +0x64 google.golang.org/grpc.(Server).GracefulStop(0xc00059a600) ``` This PR stops the GRPC server after 15s. Related to the go routing dumps in #6747. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Ganesh Vernekar	d4b9fe801f	M-map full chunks of Head from disk (#6679 ) When appending to the head and a chunk is full it is flushed to the disk and m-mapped (memory mapped) to free up memory Prom startup now happens in these stages - Iterate the m-maped chunks from disk and keep a map of series reference to its slice of mmapped chunks. - Iterate the WAL as usual. Whenever we create a new series, look for it's mmapped chunks in the map created before and add it to that series. If a head chunk is corrupted the currpted one and all chunks after that are deleted and the data after the corruption is recovered from the existing WAL which means that a corruption in m-mapped files results in NO data loss. [Mmaped chunks format](https://github.com/prometheus/prometheus/blob/master/tsdb/docs/format/head_chunks.md) - main difference is that the chunk for mmaping now also includes series reference because there is no index for mapping series to chunks. [The block chunks](https://github.com/prometheus/prometheus/blob/master/tsdb/docs/format/chunks.md) are accessed from the index which includes the offsets for the chunks in the chunks file - example - chunks of series ID have offsets 200, 500 etc in the chunk files. In case of mmaped chunks, the offsets are stored in memory and accessed from that. During WAL replay, these offsets are restored by iterating all m-mapped chunks as stated above by matching the series id present in the chunk header and offset of that chunk in that file. Prombench results _WAL Replay_ 1h Wal reply time 30% less wal reply time - 4m31 vs 3m36 2h Wal reply time 20% less wal reply time - 8m16 vs 7m _Memory During WAL Replay_ High Churn: 10-15% less RAM - 32gb vs 28gb 20% less RAM after compaction 34gb vs 27gb No Churn: 20-30% less RAM - 23gb vs 18gb 40% less RAM after compaction 32.5gb vs 20gb Screenshots are in [this comment](https://github.com/prometheus/prometheus/pull/6679#issuecomment-621678932) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Ben Ye	1e4e37144d	Fixed wrongly handled not ready TSDB on web and API. (#7182 ) * fix federate endpoint panic Signed-off-by: yeya24 <yb532204897@gmail.com> * Fixed all cases of not ready TSDB being wrongly handled. * Fixed issue for federation. * Ensured this will never happen again thanks to interfaces * Fixes same issue for stats. * Added tests for readiness. * Fixed bug in stats. It was: status.MaxTime = db.Head().MaxTime() status.MinTime = db.Head().MaxTime() Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed Brian's comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed Brian's comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Ben Ye	ecda6013ed	Use only local tsdb for federation (#7096 ) Signed-off-by: yeya24 <yb532204897@gmail.com>	5 years ago
Bartlomiej Plotka	a20bebf7eb	Moved readyStorage to main. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Bartlomiej Plotka	34426766d8	Unify Iterator interfaces. All point to storage now. This is part of https://github.com/prometheus/prometheus/pull/5882 that can be done to simplify things. All todos I added will be fixed in follow up PRs. * querier.Querier, querier.Appender, querier.SeriesSet, and querier.Series interfaces merged with storage interface.go. All imports that. * querier.SeriesIterator replaced by chunkenc.Iterator * Added chunkenc.Iterator.Seek method and tests for xor implementation (?) * Since we properly handle SelectParams for Select methods I adjusted min max based on that. This should help in terms of performance for queries with functions like offset. * added Seek to deletedIterator and test. * storage/tsdb was removed as it was only a unnecessary glue with incompatible structs. No logic was changed, only different source of abstractions, so no need for benchmarks. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Harkishen Singh	489a9aa7b9	Adds normalization of localhost urls in targets page react (#6794 ) * support for globalurls in targets page react Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com> * fixed tests Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com> * removed fmts Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com> * implemented suggestions Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com> * formatted Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com> * implemented suggestions. fixed tests. Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com> * formated go code Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com> * implemented suggestions Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>	5 years ago
Ganesh Vernekar	5ecef3542d	Cleanup after merging tsdb into prometheus Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Simon Pasquier	be67b8d460	web: fix flaky TestHTTPMetrics() (#5695 ) Signed-off-by: Simon Pasquier <spasquie@redhat.com>	6 years ago
Simon Pasquier	d7f38dfdde	web: add 'code' label to HTTP metrics (#5640 ) * web: add prometheus_http_requests_total metrics Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Add unit test for requestCounter metric Signed-off-by: Simon Pasquier <spasquie@redhat.com>	6 years ago
Minh-Long Do	b26b5c9e96	Add rendering test of template based web endpoints (#5188 ) Signed-off-by: Minh-Long Do <minhlong.langos@gmail.com>	6 years ago
Simon Pasquier	6fa8de132b	web/v1/api: add tests for admin actions (#4767 ) Signed-off-by: Simon Pasquier <spasquie@redhat.com>	6 years ago
Simon Pasquier	2d7f562ed6	web: fix asset paths for Windows platforms (#4616 ) * web: fix asset paths for Windows platforms Signed-off-by: Simon Pasquier <spasquie@redhat.com> * web: add tests Signed-off-by: Simon Pasquier <spasquie@redhat.com>	6 years ago
Fabian Reinartz	e7cfc7dae5	web: restore old path prefix behavior Signed-off-by: Fabian Reinartz <freinartz@google.com>	7 years ago
Fabian Reinartz	8a4bda8d57	web: remove security headers Signed-off-by: Fabian Reinartz <freinartz@google.com>	7 years ago
Eric V	c207920500	Added HTTP security headers to web.go (#3583 ) Signed-off-by: Eric Valenzuela <evalenzuela@thousandeyes.com>	7 years ago
beorn7	94ff07b81d	Merge branch 'release-2.2' Signed-off-by: beorn7 <beorn@soundcloud.com>	7 years ago
Ganesh Vernekar	b44ce11d1b	Added test to check pathPrefix	7 years ago
Anton Tereshchenkov	4cb8f6c260	web: remove unused MetricsPath option (#3964 )	7 years ago
Krasi Georgiev	4be9a40b89	fatalf is not thread safe so using panic instead (#3680 )	7 years ago
Krasi Georgiev	c94fa731aa	bypass the proxy for the tests	7 years ago
Krasi Georgiev	0e58cda1fe	use Fatalf instead of panic	7 years ago
Krasi Georgiev	61d793cb1f	handle web_test webhandler errors.	7 years ago
Krasi Georgiev	aca8f85699	fixed the tests	7 years ago
Fabian Reinartz	53875da7fb	Merge pull request #3286 from prometheus/dev-2.0 Dev 2.0	7 years ago
Brian Brazil	8740fcc657	Use panic in goroutine in test, not t.Fatalf	7 years ago
Julien Levesy	26a067b23d	use testutil assertion helpers in web package testsuite	7 years ago
Fabian Reinartz	7a3261aa99	Merge pull request #3240 from Gouthamve/fix-admin-api web/api: Fix typo which broke the 2.0 admin APIs.	7 years ago
Fabian Reinartz	2d0b8e8b94	Merge branch 'master' into dev-2.0	7 years ago
Goutham Veeramachaneni	35aaa2002d	web/api: Add tests for v2 admin API Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	7 years ago
Fabian Reinartz	7b02bfee0a	web: start web handler while TSDB is starting up	7 years ago
Krasi Georgiev	b4b0999e7f	add prefix to the api when prometheus started with custom web.external-url Signed-off-by: Krasi Georgiev <krasi.root@gmail.com>	7 years ago
Tom Wilkie	bbc9671d50	Get profile handlers working again after #3054 and #3146 . Ensures the pprod endpoints deal with path-prefixes correctly; adds a test so we don't break it again.	7 years ago
Fabian Reinartz	d21f149745	*: migrate to go-kit/log	7 years ago
Fabian Reinartz	25f3e1c424	Merge branch 'master' into mergemaster	7 years ago

1 2

53 Commits (ddaddc2a7af235d91b710bcba1ccbec4e7cfa6b5)