You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

beorn7 fc6737b7fb storage: improve index lookups ... tl;dr: This is not a fundamental solution to the indexing problem (like tindex is) but it at least avoids utilizing the intersection problem to the greatest possible amount. In more detail: Imagine the following query: nicely:aggregating:rule{job="foo",env="prod"} While it uses a nicely aggregating recording rule (which might have a very low cardinality), Prometheus still intersects the low number of fingerprints for `{__name__="nicely:aggregating:rule"}` with the many thousands of fingerprints matching `{job="foo"}` and with the millions of fingerprints matching `{env="prod"}`. This totally innocuous query is dead slow if the Prometheus server has a lot of time series with the `{env="prod"}` label. Ironically, if you make the query more complicated, it becomes blazingly fast: nicely:aggregating:rule{job=~"foo",env=~"prod"} Why so? Because Prometheus only intersects with non-Equal matchers if there are no Equal matchers. That's good in this case because it retrieves the few fingerprints for `{__name__="nicely:aggregating:rule"}` and then starts right ahead to retrieve the metric for those FPs and checking individually if they match the other matchers. This change is generalizing the idea of when to stop intersecting FPs and go into "retrieve metrics and check them individually against remaining matchers" mode: - First, sort all matchers by "expected cardinality". Matchers matching the empty string are always worst (and never used for intersections). Equal matchers are in general consider best, but by using some crude heuristics, we declare some better than others (instance labels or anything that looks like a recording rule). - Then go through the matchers until we hit a threshold of remaining FPs in the intersection. This threshold is higher if we are already in the non-Equal matcher area as intersection is even more expensive here. - Once the threshold has been reached (or we have run out of matchers that do not match the empty string), start with "retrieve metrics and check them individually against remaining matchers". A beefy server at SoundCloud was spending 67% of its CPU time in index lookups (fingerprintsForLabelPairs), serving mostly a dashboard that is exclusively built with recording rules. With this change, it spends only 35% in fingerprintsForLabelPairs. The CPU usage dropped from 26 cores to 18 cores. The median latency for query_range dropped from 14s to 50ms(!). As expected, higher percentile latency didn't improve that much because the new approach is _occasionally_ running into the worst case while the old one was _systematically_ doing so. The 99th percentile latency is now about as high as the median before (14s) while it was almost twice as high before (26s).		8 years ago
..
codable	…
fixtures/b0	…
index	Handle errors caused by data corruption more gracefully	9 years ago
storagetool	Make version informations consistent between prometheus components	9 years ago
chunk.go	storage: Make MemorySeriesStorage a public type	9 years ago
crashrecovery.go	Crash recovery: Fix an edge case.	9 years ago
delta.go	Implement Gorilla-inspired chunk encoding	9 years ago
delta_helpers.go	…
doubledelta.go	Implement Gorilla-inspired chunk encoding	9 years ago
heads.go	Merge branch 'master' into beorn7/storage4	9 years ago
instrumentation.go	storage: Make MemorySeriesStorage a public type	9 years ago
interface.go	storage: improve index lookups	8 years ago
locker.go	Update doc comments	9 years ago
locker_test.go	Add missing license headers	9 years ago
mapper.go	Checkpoint fingerprint mappings only upon shutdown	9 years ago
mapper_test.go	Checkpoint fingerprint mappings only upon shutdown	9 years ago
persistence.go	Consistently use the `Seconds()` method for conversion of durations	9 years ago
persistence_test.go	Implement Gorilla-inspired chunk encoding	9 years ago
preload.go	storage: Make MemorySeriesStorage a public type	9 years ago
series.go	storage: Make MemorySeriesStorage a public type	9 years ago
series_test.go	Never drop a still open head chunk.	9 years ago
storage.go	storage: improve index lookups	8 years ago
storage_test.go	storage: improve index lookups	8 years ago
test_helpers.go	storage: Make MemorySeriesStorage a public type	9 years ago
varbit.go	Work around compiler bug	9 years ago
varbit_helpers.go	Rename Gorilla into varbit	9 years ago
varbit_test.go	Rename Gorilla into varbit	9 years ago