The Prometheus monitoring system and time series database.
 
 
 
 
 
 
Go to file
Brian Brazil 48d25e6fe7 Reduce memory used by postings offset table.
Rather than keeping the offset of each postings list, instead
keep the nth offset of the offset of the posting list. As postings
list offsets have always been sorted, we can then get to the closest
entry before the one we want an iterate forwards.

I haven't done much tuning on the 32 number, it was chosen to try
not to read through more than a 4k page of data.

Switch to a bulk interface for fetching postings. Use it to avoid having
to re-read parts of the posting offset table when querying lots of it.

For a index with what BenchmarkHeadPostingForMatchers uses RAM
for r.postings drops from 3.79MB to 80.19kB or about 48x.
Bytes allocated go down by 30%, and suprisingly CPU usage drops by
4-6% for typical queries too.

benchmark                                                               old ns/op      new ns/op      delta
BenchmarkPostingsForMatchers/Block/n="1"-4                              35231          36673          +4.09%
BenchmarkPostingsForMatchers/Block/n="1",j="foo"-4                      563380         540627         -4.04%
BenchmarkPostingsForMatchers/Block/j="foo",n="1"-4                      536782         534186         -0.48%
BenchmarkPostingsForMatchers/Block/n="1",j!="foo"-4                     533990         541550         +1.42%
BenchmarkPostingsForMatchers/Block/i=~".*"-4                            113374598      117969608      +4.05%
BenchmarkPostingsForMatchers/Block/i=~".+"-4                            146329884      139651442      -4.56%
BenchmarkPostingsForMatchers/Block/i=~""-4                              50346510       44961127       -10.70%
BenchmarkPostingsForMatchers/Block/i!=""-4                              41261550       35356165       -14.31%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",j="foo"-4              112544418      116904010      +3.87%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",i!="2",j="foo"-4       112487086      116864918      +3.89%
BenchmarkPostingsForMatchers/Block/n="1",i!=""-4                        41094758       35457904       -13.72%
BenchmarkPostingsForMatchers/Block/n="1",i!="",j="foo"-4                41906372       36151473       -13.73%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",j="foo"-4              147262414      140424800      -4.64%
BenchmarkPostingsForMatchers/Block/n="1",i=~"1.+",j="foo"-4             28615629       27872072       -2.60%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!="2",j="foo"-4       147117177      140462403      -4.52%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!~"2.*",j="foo"-4     175096826      167902298      -4.11%

benchmark                                                               old allocs     new allocs     delta
BenchmarkPostingsForMatchers/Block/n="1"-4                              4              6              +50.00%
BenchmarkPostingsForMatchers/Block/n="1",j="foo"-4                      7              11             +57.14%
BenchmarkPostingsForMatchers/Block/j="foo",n="1"-4                      7              11             +57.14%
BenchmarkPostingsForMatchers/Block/n="1",j!="foo"-4                     15             17             +13.33%
BenchmarkPostingsForMatchers/Block/i=~".*"-4                            100010         100012         +0.00%
BenchmarkPostingsForMatchers/Block/i=~".+"-4                            200069         200040         -0.01%
BenchmarkPostingsForMatchers/Block/i=~""-4                              200072         200045         -0.01%
BenchmarkPostingsForMatchers/Block/i!=""-4                              200070         200041         -0.01%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",j="foo"-4              100013         100017         +0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",i!="2",j="foo"-4       100017         100023         +0.01%
BenchmarkPostingsForMatchers/Block/n="1",i!=""-4                        200073         200046         -0.01%
BenchmarkPostingsForMatchers/Block/n="1",i!="",j="foo"-4                200075         200050         -0.01%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",j="foo"-4              200074         200049         -0.01%
BenchmarkPostingsForMatchers/Block/n="1",i=~"1.+",j="foo"-4             111165         111150         -0.01%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!="2",j="foo"-4       200078         200055         -0.01%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!~"2.*",j="foo"-4     311282         311238         -0.01%

benchmark                                                               old bytes     new bytes     delta
BenchmarkPostingsForMatchers/Block/n="1"-4                              264           296           +12.12%
BenchmarkPostingsForMatchers/Block/n="1",j="foo"-4                      360           424           +17.78%
BenchmarkPostingsForMatchers/Block/j="foo",n="1"-4                      360           424           +17.78%
BenchmarkPostingsForMatchers/Block/n="1",j!="foo"-4                     520           552           +6.15%
BenchmarkPostingsForMatchers/Block/i=~".*"-4                            1600461       1600482       +0.00%
BenchmarkPostingsForMatchers/Block/i=~".+"-4                            24900801      17259077      -30.69%
BenchmarkPostingsForMatchers/Block/i=~""-4                              24900836      17259151      -30.69%
BenchmarkPostingsForMatchers/Block/i!=""-4                              24900760      17259048      -30.69%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",j="foo"-4              1600557       1600621       +0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",i!="2",j="foo"-4       1600717       1600813       +0.01%
BenchmarkPostingsForMatchers/Block/n="1",i!=""-4                        24900856      17259176      -30.69%
BenchmarkPostingsForMatchers/Block/n="1",i!="",j="foo"-4                24900952      17259304      -30.69%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",j="foo"-4              24900993      17259333      -30.69%
BenchmarkPostingsForMatchers/Block/n="1",i=~"1.+",j="foo"-4             3788311       3142630       -17.04%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!="2",j="foo"-4       24901137      17259509      -30.69%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!~"2.*",j="foo"-4     28693086      20405680      -28.88%

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-11 19:59:31 +00:00
.circleci Remove Travis as a CI platform (#6369) 2019-11-25 14:59:26 +01:00
.github disable github actions (#6207) 2019-10-23 23:37:53 +03:00
cmd Add time units to storage.tsdb.retention.size flag (#6365) 2019-11-30 08:00:51 +00:00
config config: add test case for scrape interval larger than timeout. (#6037) 2019-09-23 13:26:56 +02:00
console_libraries Added humanizePercentage formatting to templates (#5670) 2019-06-15 08:59:57 +01:00
consoles Update console template example metric names. (#5678) 2019-06-17 12:13:29 +01:00
discovery discovery/kubernetes: fix client metrics 2019-11-29 12:34:36 +01:00
docs Merge pull request #6442 from gotjosh/single-metric-api 2019-12-11 18:21:16 +01:00
documentation Simplify remote write dashboard in mixin. 2019-11-18 19:58:07 -08:00
notifier skip empty alerts. (#6444) 2019-12-11 10:03:50 +00:00
pkg Add exemplar support to the openmetrics parser (#6292) 2019-11-19 09:33:30 +00:00
prompb Pushdown aggregator group by through read hint (#6401) 2019-12-05 14:06:28 +00:00
promql PromQL: export lexer (#6435) 2019-12-09 19:03:31 +00:00
rules *: improve error messages when parsing bad rules (#5965) 2019-08-28 17:36:48 +02:00
scrape Make the scrape.metricMetadataStore interface public 2019-12-05 10:29:58 +00:00
scripts React UI: Serve React UI under /new (#6229) 2019-10-28 10:45:53 +01:00
storage Pushdown aggregator group by through read hint (#6401) 2019-12-05 14:06:28 +00:00
template template: remove redundant parentheses. (#5904) 2019-08-16 09:41:01 +01:00
tsdb Reduce memory used by postings offset table. 2019-12-11 19:59:31 +00:00
util util/promlint: simplify sorting function (#6448) 2019-12-11 15:45:26 +01:00
vendor upgrade aws-sdk-go (#6429) 2019-12-09 23:16:18 +00:00
web api: provide metadata for a single metric 2019-12-10 21:17:49 +00:00
.dockerignore Docker images for ARM32v7 and ARM64v8 (#5031) 2019-04-15 14:27:00 +02:00
.gitignore Add tsdb/tsdb to .gitignore 2019-11-15 12:41:37 +01:00
.golangci.yml *: enable all default linters (#5504) 2019-05-03 15:11:28 +02:00
.promu.yml Remove Travis as a CI platform (#6369) 2019-11-25 14:59:26 +01:00
CHANGELOG.md Cut 2.14.0 release (#6298) 2019-11-11 15:20:12 +01:00
CONTRIBUTING.md *: bump Go version to 1.13 (#6020) 2019-09-13 13:55:18 +01:00
Dockerfile Integrate beginning of React UI (#5694) 2019-10-17 14:38:09 +02:00
LICENSE Clean up license issues. 2015-01-21 20:07:45 +01:00
MAINTAINERS.md Update and re-work maintainers file. 2019-11-28 19:01:35 +00:00
Makefile Add unit tests for react-app (#6234) 2019-10-28 18:09:48 +01:00
Makefile.common Bump golangci-lint to v1.18.0 (#6004) 2019-09-10 17:40:08 +02:00
NOTICE Integrate beginning of React UI (#5694) 2019-10-17 14:38:09 +02:00
README.md Remove Travis as a CI platform (#6369) 2019-11-25 14:59:26 +01:00
RELEASE.md Volunteering for release shepharding (#6384) 2019-11-27 11:45:23 -07:00
VERSION Cut 2.14.0 release (#6298) 2019-11-11 15:20:12 +01:00
code-of-conduct.md Add CNCF code of conduct as the Prometheus code of conduct 2016-10-19 21:39:19 +02:00
fuzzit.sh Fix fuzzit job (#6005) 2019-09-11 09:08:20 +02:00
go.mod upgrade aws-sdk-go (#6429) 2019-12-09 23:16:18 +00:00
go.sum upgrade aws-sdk-go (#6429) 2019-12-09 23:16:18 +00:00

README.md

Prometheus

CircleCI Docker Repository on Quay Docker Pulls Go Report Card CII Best Practices fuzzit

Visit prometheus.io for the full documentation, examples and guides.

Prometheus, a Cloud Native Computing Foundation project, is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true.

Prometheus's main distinguishing features as compared to other monitoring systems are:

  • a multi-dimensional data model (timeseries defined by metric name and set of key/value dimensions)
  • a flexible query language to leverage this dimensionality
  • no dependency on distributed storage; single server nodes are autonomous
  • timeseries collection happens via a pull model over HTTP
  • pushing timeseries is supported via an intermediary gateway
  • targets are discovered via service discovery or static configuration
  • multiple modes of graphing and dashboarding support
  • support for hierarchical and horizontal federation

Architecture overview

Install

There are various ways of installing Prometheus.

Precompiled binaries

Precompiled binaries for released versions are available in the download section on prometheus.io. Using the latest production release binary is the recommended way of installing Prometheus. See the Installing chapter in the documentation for all the details.

Debian packages are available.

Docker images

Docker images are available on Quay.io or Docker Hub.

You can launch a Prometheus container for trying it out with

$ docker run --name prometheus -d -p 127.0.0.1:9090:9090 prom/prometheus

Prometheus will now be reachable at http://localhost:9090/.

Building from source

To build Prometheus from the source code yourself you need to have a working Go environment with version 1.13 or greater installed. You will also need to have Node.js and Yarn installed in order to build the frontend assets.

You can directly use the go tool to download and install the prometheus and promtool binaries into your GOPATH:

$ go get github.com/prometheus/prometheus/cmd/...
$ prometheus --config.file=your_config.yml

However, when using go get to build Prometheus, Prometheus will expect to be able to read its web assets from local filesystem directories under web/ui/static and web/ui/templates. In order for these assets to be found, you will have to run Prometheus from the root of the cloned repository. Note also that these directories do not include the new experimental React UI unless it has been built explicitly using make assets or make build.

An example of the above configuration file can be found here.

You can also clone the repository yourself and build using make build, which will compile in the web assets so that Prometheus can be run from anywhere:

$ mkdir -p $GOPATH/src/github.com/prometheus
$ cd $GOPATH/src/github.com/prometheus
$ git clone https://github.com/prometheus/prometheus.git
$ cd prometheus
$ make build
$ ./prometheus --config.file=your_config.yml

The Makefile provides several targets:

  • build: build the prometheus and promtool binaries (includes building and compiling in web assets)
  • test: run the tests
  • test-short: run the short tests
  • format: format the source code
  • vet: check the source code for common errors
  • docker: build a docker container for the current HEAD

React UI Development

For more information on building, running, and developing on the new React-based UI, see the React app's README.md.

More information

  • The source code is periodically indexed: Prometheus Core.
  • You will find a CircleCI configuration in .circleci/config.yml.
  • See the Community page for how to reach the Prometheus developers and users on various communication channels.

Contributing

Refer to CONTRIBUTING.md

License

Apache License 2.0, see LICENSE.