2017-04-10 18:59:45 +00:00
|
|
|
// Copyright 2017 The Prometheus Authors
|
|
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
|
|
// you may not use this file except in compliance with the License.
|
|
|
|
// You may obtain a copy of the License at
|
|
|
|
//
|
|
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
//
|
|
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
// See the License for the specific language governing permissions and
|
|
|
|
// limitations under the License.
|
|
|
|
|
2017-11-30 14:34:49 +00:00
|
|
|
package index
|
2016-12-27 10:32:10 +00:00
|
|
|
|
|
|
|
import (
|
Label values with matchers by intersecting postings (#9907)
* LabelValues w/matchers by intersecting postings
Instead of iterating all matched series to find the values, this
checks if each one of the label values is present in the matched series
(postings).
Pending to be benchmarked.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Benchmark labelValuesWithMatchers
name old time/op new time/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0%
name old alloc/op new alloc/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0%
name old allocs/op new allocs/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Don't expand postings to intersect them
Using a min heap we can check whether matched postings intersect with
each one of the label values postings. This avoid expanding postings
(and thus having all of them in memory at any point).
Slightly slower than the expanding postings version for some cases, but
definitely pays the price once the cardinality grows.
Still offers 10x latency improvement where previous latencies were
reaching 1s.
Benchmark results:
name \ time/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0% 110ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0% 0.18s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0% 125ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0% 177ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0% 134ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0% 4.29µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0% 120ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0% 0.15s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0% 125.4ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0% 170ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0% 0.14s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0% 0.92ms ± 0%
name \ alloc/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0% 36.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0% 6.4kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0% 4.30kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0% 38MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0% 22.3MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0% 0MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0% 5.86kB ± 0%
name \ allocs/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0% 139.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0% 87.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0% 0.50M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0% 129.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Apply comment suggestions from the code review
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
* Change else { if } to else if
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Remove sorting of label values
We were not sorting them before, so no need to sort them now
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2021-12-28 14:59:03 +00:00
|
|
|
"container/heap"
|
|
|
|
"context"
|
2017-03-27 10:16:55 +00:00
|
|
|
"encoding/binary"
|
2017-10-05 20:22:14 +00:00
|
|
|
"fmt"
|
2017-03-27 10:16:55 +00:00
|
|
|
"math/rand"
|
2017-10-05 20:22:14 +00:00
|
|
|
"sort"
|
2021-11-05 10:01:23 +00:00
|
|
|
"strconv"
|
2016-12-27 10:32:10 +00:00
|
|
|
"testing"
|
2017-03-27 10:16:55 +00:00
|
|
|
|
Label values with matchers by intersecting postings (#9907)
* LabelValues w/matchers by intersecting postings
Instead of iterating all matched series to find the values, this
checks if each one of the label values is present in the matched series
(postings).
Pending to be benchmarked.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Benchmark labelValuesWithMatchers
name old time/op new time/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0%
name old alloc/op new alloc/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0%
name old allocs/op new allocs/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Don't expand postings to intersect them
Using a min heap we can check whether matched postings intersect with
each one of the label values postings. This avoid expanding postings
(and thus having all of them in memory at any point).
Slightly slower than the expanding postings version for some cases, but
definitely pays the price once the cardinality grows.
Still offers 10x latency improvement where previous latencies were
reaching 1s.
Benchmark results:
name \ time/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0% 110ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0% 0.18s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0% 125ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0% 177ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0% 134ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0% 4.29µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0% 120ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0% 0.15s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0% 125.4ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0% 170ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0% 0.14s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0% 0.92ms ± 0%
name \ alloc/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0% 36.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0% 6.4kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0% 4.30kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0% 38MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0% 22.3MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0% 0MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0% 5.86kB ± 0%
name \ allocs/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0% 139.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0% 87.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0% 0.50M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0% 129.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Apply comment suggestions from the code review
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
* Change else { if } to else if
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Remove sorting of label values
We were not sorting them before, so no need to sort them now
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2021-12-28 14:59:03 +00:00
|
|
|
"github.com/pkg/errors"
|
2020-10-29 09:43:23 +00:00
|
|
|
"github.com/stretchr/testify/require"
|
2020-10-22 09:00:08 +00:00
|
|
|
|
2021-11-08 14:23:17 +00:00
|
|
|
"github.com/prometheus/prometheus/model/labels"
|
2021-11-06 10:10:04 +00:00
|
|
|
"github.com/prometheus/prometheus/storage"
|
2016-12-27 10:32:10 +00:00
|
|
|
)
|
|
|
|
|
2017-09-20 16:08:57 +00:00
|
|
|
func TestMemPostings_addFor(t *testing.T) {
|
2017-11-30 14:34:49 +00:00
|
|
|
p := NewMemPostings()
|
2021-11-06 10:10:04 +00:00
|
|
|
p.m[allPostingsKey.Name] = map[string][]storage.SeriesRef{}
|
|
|
|
p.m[allPostingsKey.Name][allPostingsKey.Value] = []storage.SeriesRef{1, 2, 3, 4, 6, 7, 8}
|
2017-09-20 16:08:57 +00:00
|
|
|
|
|
|
|
p.addFor(5, allPostingsKey)
|
|
|
|
|
2021-11-06 10:10:04 +00:00
|
|
|
require.Equal(t, []storage.SeriesRef{1, 2, 3, 4, 5, 6, 7, 8}, p.m[allPostingsKey.Name][allPostingsKey.Value])
|
2017-09-20 16:08:57 +00:00
|
|
|
}
|
|
|
|
|
2017-10-05 20:22:14 +00:00
|
|
|
func TestMemPostings_ensureOrder(t *testing.T) {
|
2017-11-30 14:34:49 +00:00
|
|
|
p := NewUnorderedMemPostings()
|
2021-11-06 10:10:04 +00:00
|
|
|
p.m["a"] = map[string][]storage.SeriesRef{}
|
2017-10-05 20:22:14 +00:00
|
|
|
|
|
|
|
for i := 0; i < 100; i++ {
|
2021-11-06 10:10:04 +00:00
|
|
|
l := make([]storage.SeriesRef, 100)
|
2017-10-05 20:22:14 +00:00
|
|
|
for j := range l {
|
2021-11-06 10:10:04 +00:00
|
|
|
l[j] = storage.SeriesRef(rand.Uint64())
|
2017-10-05 20:22:14 +00:00
|
|
|
}
|
|
|
|
v := fmt.Sprintf("%d", i)
|
|
|
|
|
2018-11-02 14:27:19 +00:00
|
|
|
p.m["a"][v] = l
|
2017-10-05 20:22:14 +00:00
|
|
|
}
|
|
|
|
|
2023-04-18 10:13:05 +00:00
|
|
|
p.EnsureOrder(0)
|
2017-10-05 20:22:14 +00:00
|
|
|
|
2018-11-02 14:27:19 +00:00
|
|
|
for _, e := range p.m {
|
|
|
|
for _, l := range e {
|
|
|
|
ok := sort.SliceIsSorted(l, func(i, j int) bool {
|
|
|
|
return l[i] < l[j]
|
|
|
|
})
|
|
|
|
if !ok {
|
|
|
|
t.Fatalf("postings list %v is not sorted", l)
|
|
|
|
}
|
2017-10-05 20:22:14 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2021-11-05 10:01:23 +00:00
|
|
|
func BenchmarkMemPostings_ensureOrder(b *testing.B) {
|
|
|
|
tests := map[string]struct {
|
|
|
|
numLabels int
|
|
|
|
numValuesPerLabel int
|
|
|
|
numRefsPerValue int
|
|
|
|
}{
|
|
|
|
"many values per label": {
|
|
|
|
numLabels: 100,
|
|
|
|
numValuesPerLabel: 10000,
|
|
|
|
numRefsPerValue: 100,
|
|
|
|
},
|
|
|
|
"few values per label": {
|
|
|
|
numLabels: 1000000,
|
|
|
|
numValuesPerLabel: 1,
|
|
|
|
numRefsPerValue: 100,
|
|
|
|
},
|
|
|
|
"few refs per label value": {
|
|
|
|
numLabels: 1000,
|
|
|
|
numValuesPerLabel: 1000,
|
|
|
|
numRefsPerValue: 10,
|
|
|
|
},
|
|
|
|
}
|
|
|
|
|
|
|
|
for testName, testData := range tests {
|
|
|
|
b.Run(testName, func(b *testing.B) {
|
|
|
|
p := NewUnorderedMemPostings()
|
|
|
|
|
|
|
|
// Generate postings.
|
|
|
|
for l := 0; l < testData.numLabels; l++ {
|
|
|
|
labelName := strconv.Itoa(l)
|
2021-11-06 10:10:04 +00:00
|
|
|
p.m[labelName] = map[string][]storage.SeriesRef{}
|
2021-11-05 10:01:23 +00:00
|
|
|
|
|
|
|
for v := 0; v < testData.numValuesPerLabel; v++ {
|
2021-11-06 10:10:04 +00:00
|
|
|
refs := make([]storage.SeriesRef, testData.numRefsPerValue)
|
2021-11-05 10:01:23 +00:00
|
|
|
for j := range refs {
|
2021-11-06 10:10:04 +00:00
|
|
|
refs[j] = storage.SeriesRef(rand.Uint64())
|
2021-11-05 10:01:23 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
labelValue := strconv.Itoa(v)
|
|
|
|
p.m[labelName][labelValue] = refs
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
b.ResetTimer()
|
|
|
|
|
|
|
|
for n := 0; n < b.N; n++ {
|
2023-04-18 10:13:05 +00:00
|
|
|
p.EnsureOrder(0)
|
2021-11-05 10:01:23 +00:00
|
|
|
p.ordered = false
|
|
|
|
}
|
|
|
|
})
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2016-12-28 10:02:19 +00:00
|
|
|
func TestIntersect(t *testing.T) {
|
2019-03-21 16:23:00 +00:00
|
|
|
a := newListPostings(1, 2, 3)
|
|
|
|
b := newListPostings(2, 3, 4)
|
|
|
|
|
2021-10-22 08:06:44 +00:00
|
|
|
cases := []struct {
|
2019-03-21 16:23:00 +00:00
|
|
|
in []Postings
|
|
|
|
|
|
|
|
res Postings
|
2016-12-27 10:32:10 +00:00
|
|
|
}{
|
|
|
|
{
|
2019-03-21 16:23:00 +00:00
|
|
|
in: []Postings{},
|
|
|
|
res: EmptyPostings(),
|
2016-12-27 10:32:10 +00:00
|
|
|
},
|
|
|
|
{
|
2019-03-21 16:23:00 +00:00
|
|
|
in: []Postings{a, b, EmptyPostings()},
|
|
|
|
res: EmptyPostings(),
|
2016-12-27 10:32:10 +00:00
|
|
|
},
|
|
|
|
{
|
2019-03-21 16:23:00 +00:00
|
|
|
in: []Postings{b, a, EmptyPostings()},
|
|
|
|
res: EmptyPostings(),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
in: []Postings{EmptyPostings(), b, a},
|
|
|
|
res: EmptyPostings(),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
in: []Postings{EmptyPostings(), a, b},
|
|
|
|
res: EmptyPostings(),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
in: []Postings{a, EmptyPostings(), b},
|
|
|
|
res: EmptyPostings(),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
in: []Postings{b, EmptyPostings(), a},
|
|
|
|
res: EmptyPostings(),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
in: []Postings{b, EmptyPostings(), a, a, b, a, a, a},
|
|
|
|
res: EmptyPostings(),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
in: []Postings{
|
|
|
|
newListPostings(1, 2, 3, 4, 5),
|
|
|
|
newListPostings(6, 7, 8, 9, 10),
|
|
|
|
},
|
|
|
|
res: newListPostings(),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
in: []Postings{
|
|
|
|
newListPostings(1, 2, 3, 4, 5),
|
|
|
|
newListPostings(4, 5, 6, 7, 8),
|
|
|
|
},
|
|
|
|
res: newListPostings(4, 5),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
in: []Postings{
|
|
|
|
newListPostings(1, 2, 3, 4, 9, 10),
|
|
|
|
newListPostings(1, 4, 5, 6, 7, 8, 10, 11),
|
|
|
|
},
|
|
|
|
res: newListPostings(1, 4, 10),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
in: []Postings{
|
|
|
|
newListPostings(1),
|
|
|
|
newListPostings(0, 1),
|
|
|
|
},
|
|
|
|
res: newListPostings(1),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
in: []Postings{
|
|
|
|
newListPostings(1),
|
|
|
|
},
|
|
|
|
res: newListPostings(1),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
in: []Postings{
|
|
|
|
newListPostings(1),
|
|
|
|
newListPostings(),
|
|
|
|
},
|
|
|
|
res: newListPostings(),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
in: []Postings{
|
|
|
|
newListPostings(),
|
|
|
|
newListPostings(),
|
|
|
|
},
|
|
|
|
res: newListPostings(),
|
2016-12-27 10:32:10 +00:00
|
|
|
},
|
|
|
|
}
|
|
|
|
|
2017-04-08 15:12:29 +00:00
|
|
|
for _, c := range cases {
|
2019-03-21 16:23:00 +00:00
|
|
|
t.Run("", func(t *testing.T) {
|
|
|
|
if c.res == nil {
|
|
|
|
t.Fatal("intersect result expectancy cannot be nil")
|
|
|
|
}
|
2016-12-27 10:32:10 +00:00
|
|
|
|
2019-03-21 16:23:00 +00:00
|
|
|
expected, err := ExpandPostings(c.res)
|
2020-10-29 09:43:23 +00:00
|
|
|
require.NoError(t, err)
|
2019-03-21 16:23:00 +00:00
|
|
|
|
|
|
|
i := Intersect(c.in...)
|
|
|
|
|
|
|
|
if c.res == EmptyPostings() {
|
2020-10-29 09:43:23 +00:00
|
|
|
require.Equal(t, EmptyPostings(), i)
|
2019-03-21 16:23:00 +00:00
|
|
|
return
|
|
|
|
}
|
|
|
|
|
|
|
|
if i == EmptyPostings() {
|
|
|
|
t.Fatal("intersect unexpected result: EmptyPostings sentinel")
|
|
|
|
}
|
|
|
|
|
|
|
|
res, err := ExpandPostings(i)
|
2020-10-29 09:43:23 +00:00
|
|
|
require.NoError(t, err)
|
|
|
|
require.Equal(t, expected, res)
|
2019-03-21 16:23:00 +00:00
|
|
|
})
|
2016-12-27 10:32:10 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func TestMultiIntersect(t *testing.T) {
|
2021-10-22 08:06:44 +00:00
|
|
|
cases := []struct {
|
2021-11-06 10:10:04 +00:00
|
|
|
p [][]storage.SeriesRef
|
|
|
|
res []storage.SeriesRef
|
2016-12-27 10:32:10 +00:00
|
|
|
}{
|
|
|
|
{
|
2021-11-06 10:10:04 +00:00
|
|
|
p: [][]storage.SeriesRef{
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
{1, 2, 3, 4, 5, 6, 1000, 1001},
|
|
|
|
{2, 4, 5, 6, 7, 8, 999, 1001},
|
|
|
|
{1, 2, 5, 6, 7, 8, 1001, 1200},
|
|
|
|
},
|
2021-11-06 10:10:04 +00:00
|
|
|
res: []storage.SeriesRef{2, 5, 6, 1001},
|
2016-12-27 10:32:10 +00:00
|
|
|
},
|
2020-01-02 14:54:09 +00:00
|
|
|
// One of the reproducible cases for:
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
// https://github.com/prometheus/prometheus/issues/2616
|
|
|
|
// The initialisation of intersectPostings was moving the iterator forward
|
|
|
|
// prematurely making us miss some postings.
|
|
|
|
{
|
2021-11-06 10:10:04 +00:00
|
|
|
p: [][]storage.SeriesRef{
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
{1, 2},
|
|
|
|
{1, 2},
|
|
|
|
{1, 2},
|
|
|
|
{2},
|
|
|
|
},
|
2021-11-06 10:10:04 +00:00
|
|
|
res: []storage.SeriesRef{2},
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
},
|
2016-12-27 10:32:10 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
for _, c := range cases {
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
ps := make([]Postings, 0, len(c.p))
|
|
|
|
for _, postings := range c.p {
|
2019-03-21 16:23:00 +00:00
|
|
|
ps = append(ps, newListPostings(postings...))
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
}
|
2016-12-27 10:32:10 +00:00
|
|
|
|
2017-11-30 14:34:49 +00:00
|
|
|
res, err := ExpandPostings(Intersect(ps...))
|
2017-04-08 15:12:29 +00:00
|
|
|
|
2020-10-29 09:43:23 +00:00
|
|
|
require.NoError(t, err)
|
|
|
|
require.Equal(t, c.res, res)
|
2016-12-27 10:32:10 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2023-01-03 15:08:32 +00:00
|
|
|
func consumePostings(p Postings) error {
|
|
|
|
for p.Next() {
|
|
|
|
p.At()
|
|
|
|
}
|
|
|
|
return p.Err()
|
|
|
|
}
|
|
|
|
|
2016-12-27 10:32:10 +00:00
|
|
|
func BenchmarkIntersect(t *testing.B) {
|
2019-06-11 08:14:25 +00:00
|
|
|
t.Run("LongPostings1", func(bench *testing.B) {
|
2021-11-06 10:10:04 +00:00
|
|
|
var a, b, c, d []storage.SeriesRef
|
2016-12-27 10:32:10 +00:00
|
|
|
|
2019-06-11 08:14:25 +00:00
|
|
|
for i := 0; i < 10000000; i += 2 {
|
2021-11-06 10:10:04 +00:00
|
|
|
a = append(a, storage.SeriesRef(i))
|
2019-06-11 08:14:25 +00:00
|
|
|
}
|
|
|
|
for i := 5000000; i < 5000100; i += 4 {
|
2021-11-06 10:10:04 +00:00
|
|
|
b = append(b, storage.SeriesRef(i))
|
2019-06-11 08:14:25 +00:00
|
|
|
}
|
|
|
|
for i := 5090000; i < 5090600; i += 4 {
|
2021-11-06 10:10:04 +00:00
|
|
|
b = append(b, storage.SeriesRef(i))
|
2019-06-11 08:14:25 +00:00
|
|
|
}
|
|
|
|
for i := 4990000; i < 5100000; i++ {
|
2021-11-06 10:10:04 +00:00
|
|
|
c = append(c, storage.SeriesRef(i))
|
2019-06-11 08:14:25 +00:00
|
|
|
}
|
|
|
|
for i := 4000000; i < 6000000; i++ {
|
2021-11-06 10:10:04 +00:00
|
|
|
d = append(d, storage.SeriesRef(i))
|
2019-06-11 08:14:25 +00:00
|
|
|
}
|
2016-12-27 10:32:10 +00:00
|
|
|
|
2019-06-11 08:14:25 +00:00
|
|
|
bench.ResetTimer()
|
|
|
|
bench.ReportAllocs()
|
|
|
|
for i := 0; i < bench.N; i++ {
|
2023-01-03 15:07:22 +00:00
|
|
|
i1 := newListPostings(a...)
|
|
|
|
i2 := newListPostings(b...)
|
|
|
|
i3 := newListPostings(c...)
|
|
|
|
i4 := newListPostings(d...)
|
2023-01-03 15:08:32 +00:00
|
|
|
if err := consumePostings(Intersect(i1, i2, i3, i4)); err != nil {
|
2019-06-11 08:14:25 +00:00
|
|
|
bench.Fatal(err)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
})
|
|
|
|
|
|
|
|
t.Run("LongPostings2", func(bench *testing.B) {
|
2021-11-06 10:10:04 +00:00
|
|
|
var a, b, c, d []storage.SeriesRef
|
2016-12-27 10:32:10 +00:00
|
|
|
|
2019-06-11 08:14:25 +00:00
|
|
|
for i := 0; i < 12500000; i++ {
|
2021-11-06 10:10:04 +00:00
|
|
|
a = append(a, storage.SeriesRef(i))
|
2016-12-27 10:32:10 +00:00
|
|
|
}
|
2019-06-11 08:14:25 +00:00
|
|
|
for i := 7500000; i < 12500000; i++ {
|
2021-11-06 10:10:04 +00:00
|
|
|
b = append(b, storage.SeriesRef(i))
|
2019-06-11 08:14:25 +00:00
|
|
|
}
|
|
|
|
for i := 9000000; i < 20000000; i++ {
|
2021-11-06 10:10:04 +00:00
|
|
|
c = append(c, storage.SeriesRef(i))
|
2019-06-11 08:14:25 +00:00
|
|
|
}
|
|
|
|
for i := 10000000; i < 12000000; i++ {
|
2021-11-06 10:10:04 +00:00
|
|
|
d = append(d, storage.SeriesRef(i))
|
2019-06-11 08:14:25 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
bench.ResetTimer()
|
|
|
|
bench.ReportAllocs()
|
|
|
|
for i := 0; i < bench.N; i++ {
|
2023-01-03 15:07:22 +00:00
|
|
|
i1 := newListPostings(a...)
|
|
|
|
i2 := newListPostings(b...)
|
|
|
|
i3 := newListPostings(c...)
|
|
|
|
i4 := newListPostings(d...)
|
2023-01-03 15:08:32 +00:00
|
|
|
if err := consumePostings(Intersect(i1, i2, i3, i4)); err != nil {
|
2019-06-11 08:14:25 +00:00
|
|
|
bench.Fatal(err)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
})
|
|
|
|
|
|
|
|
// Many matchers(k >> n).
|
|
|
|
t.Run("ManyPostings", func(bench *testing.B) {
|
2023-01-03 15:07:22 +00:00
|
|
|
var lps []*ListPostings
|
|
|
|
var refs [][]storage.SeriesRef
|
2019-06-11 08:14:25 +00:00
|
|
|
|
2023-01-03 15:07:22 +00:00
|
|
|
// Create 100000 matchers(k=100000), making sure all memory allocation is done before starting the loop.
|
2019-06-11 08:14:25 +00:00
|
|
|
for i := 0; i < 100000; i++ {
|
2021-11-06 10:10:04 +00:00
|
|
|
var temp []storage.SeriesRef
|
|
|
|
for j := storage.SeriesRef(1); j < 100; j++ {
|
|
|
|
temp = append(temp, j)
|
2019-06-11 08:14:25 +00:00
|
|
|
}
|
2023-01-03 15:07:22 +00:00
|
|
|
lps = append(lps, newListPostings(temp...))
|
|
|
|
refs = append(refs, temp)
|
2019-06-11 08:14:25 +00:00
|
|
|
}
|
|
|
|
|
2023-01-03 15:07:22 +00:00
|
|
|
its := make([]Postings, len(refs))
|
2019-06-11 08:14:25 +00:00
|
|
|
bench.ResetTimer()
|
|
|
|
bench.ReportAllocs()
|
|
|
|
for i := 0; i < bench.N; i++ {
|
2023-01-03 15:07:22 +00:00
|
|
|
// Reset the ListPostings to their original values each time round the loop.
|
|
|
|
for j := range refs {
|
|
|
|
lps[j].list = refs[j]
|
|
|
|
its[j] = lps[j]
|
|
|
|
}
|
2023-01-03 15:08:32 +00:00
|
|
|
if err := consumePostings(Intersect(its...)); err != nil {
|
2019-06-11 08:14:25 +00:00
|
|
|
bench.Fatal(err)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
})
|
2016-12-27 10:32:10 +00:00
|
|
|
}
|
2016-12-28 10:02:19 +00:00
|
|
|
|
|
|
|
func TestMultiMerge(t *testing.T) {
|
2019-03-21 16:23:00 +00:00
|
|
|
i1 := newListPostings(1, 2, 3, 4, 5, 6, 1000, 1001)
|
|
|
|
i2 := newListPostings(2, 4, 5, 6, 7, 8, 999, 1001)
|
|
|
|
i3 := newListPostings(1, 2, 5, 6, 7, 8, 1001, 1200)
|
2016-12-28 10:02:19 +00:00
|
|
|
|
2019-03-21 16:23:00 +00:00
|
|
|
res, err := ExpandPostings(Merge(i1, i2, i3))
|
2020-10-29 09:43:23 +00:00
|
|
|
require.NoError(t, err)
|
2021-11-06 10:10:04 +00:00
|
|
|
require.Equal(t, []storage.SeriesRef{1, 2, 3, 4, 5, 6, 7, 8, 999, 1000, 1001, 1200}, res)
|
2016-12-28 10:02:19 +00:00
|
|
|
}
|
|
|
|
|
2017-04-21 20:08:26 +00:00
|
|
|
func TestMergedPostings(t *testing.T) {
|
2021-10-22 08:06:44 +00:00
|
|
|
cases := []struct {
|
2019-03-21 16:23:00 +00:00
|
|
|
in []Postings
|
|
|
|
|
|
|
|
res Postings
|
2016-12-28 10:02:19 +00:00
|
|
|
}{
|
|
|
|
{
|
2019-03-21 16:23:00 +00:00
|
|
|
in: []Postings{},
|
|
|
|
res: EmptyPostings(),
|
2016-12-28 10:02:19 +00:00
|
|
|
},
|
|
|
|
{
|
2019-03-21 16:23:00 +00:00
|
|
|
in: []Postings{
|
|
|
|
newListPostings(),
|
|
|
|
newListPostings(),
|
|
|
|
},
|
|
|
|
res: EmptyPostings(),
|
2016-12-28 10:02:19 +00:00
|
|
|
},
|
|
|
|
{
|
2019-03-21 16:23:00 +00:00
|
|
|
in: []Postings{
|
|
|
|
newListPostings(),
|
|
|
|
},
|
|
|
|
res: newListPostings(),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
in: []Postings{
|
|
|
|
EmptyPostings(),
|
|
|
|
EmptyPostings(),
|
|
|
|
EmptyPostings(),
|
|
|
|
EmptyPostings(),
|
|
|
|
},
|
|
|
|
res: EmptyPostings(),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
in: []Postings{
|
|
|
|
newListPostings(1, 2, 3, 4, 5),
|
|
|
|
newListPostings(6, 7, 8, 9, 10),
|
|
|
|
},
|
|
|
|
res: newListPostings(1, 2, 3, 4, 5, 6, 7, 8, 9, 10),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
in: []Postings{
|
|
|
|
newListPostings(1, 2, 3, 4, 5),
|
|
|
|
newListPostings(4, 5, 6, 7, 8),
|
|
|
|
},
|
|
|
|
res: newListPostings(1, 2, 3, 4, 5, 6, 7, 8),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
in: []Postings{
|
|
|
|
newListPostings(1, 2, 3, 4, 9, 10),
|
|
|
|
newListPostings(1, 4, 5, 6, 7, 8, 10, 11),
|
|
|
|
},
|
|
|
|
res: newListPostings(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
in: []Postings{
|
|
|
|
newListPostings(1, 2, 3, 4, 9, 10),
|
|
|
|
EmptyPostings(),
|
|
|
|
newListPostings(1, 4, 5, 6, 7, 8, 10, 11),
|
|
|
|
},
|
|
|
|
res: newListPostings(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
in: []Postings{
|
|
|
|
newListPostings(1, 2),
|
|
|
|
newListPostings(),
|
|
|
|
},
|
|
|
|
res: newListPostings(1, 2),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
in: []Postings{
|
|
|
|
newListPostings(1, 2),
|
|
|
|
EmptyPostings(),
|
|
|
|
},
|
|
|
|
res: newListPostings(1, 2),
|
2016-12-28 10:02:19 +00:00
|
|
|
},
|
|
|
|
}
|
|
|
|
|
|
|
|
for _, c := range cases {
|
2019-03-21 16:23:00 +00:00
|
|
|
t.Run("", func(t *testing.T) {
|
|
|
|
if c.res == nil {
|
|
|
|
t.Fatal("merge result expectancy cannot be nil")
|
|
|
|
}
|
2016-12-28 10:02:19 +00:00
|
|
|
|
2019-03-21 16:23:00 +00:00
|
|
|
expected, err := ExpandPostings(c.res)
|
2020-10-29 09:43:23 +00:00
|
|
|
require.NoError(t, err)
|
2019-03-21 16:23:00 +00:00
|
|
|
|
|
|
|
m := Merge(c.in...)
|
2017-04-08 15:12:29 +00:00
|
|
|
|
2019-03-21 16:23:00 +00:00
|
|
|
if c.res == EmptyPostings() {
|
2020-10-29 09:43:23 +00:00
|
|
|
require.Equal(t, EmptyPostings(), m)
|
2019-03-21 16:23:00 +00:00
|
|
|
return
|
|
|
|
}
|
|
|
|
|
|
|
|
if m == EmptyPostings() {
|
|
|
|
t.Fatal("merge unexpected result: EmptyPostings sentinel")
|
|
|
|
}
|
|
|
|
|
|
|
|
res, err := ExpandPostings(m)
|
2020-10-29 09:43:23 +00:00
|
|
|
require.NoError(t, err)
|
|
|
|
require.Equal(t, expected, res)
|
2019-03-21 16:23:00 +00:00
|
|
|
})
|
|
|
|
}
|
2017-04-21 20:08:26 +00:00
|
|
|
}
|
2017-04-08 15:12:29 +00:00
|
|
|
|
2017-04-21 20:08:26 +00:00
|
|
|
func TestMergedPostingsSeek(t *testing.T) {
|
2021-10-22 08:06:44 +00:00
|
|
|
cases := []struct {
|
2021-11-06 10:10:04 +00:00
|
|
|
a, b []storage.SeriesRef
|
2017-04-08 15:12:29 +00:00
|
|
|
|
2021-11-06 10:10:04 +00:00
|
|
|
seek storage.SeriesRef
|
2017-04-21 20:08:26 +00:00
|
|
|
success bool
|
2021-11-06 10:10:04 +00:00
|
|
|
res []storage.SeriesRef
|
2017-04-21 20:08:26 +00:00
|
|
|
}{
|
|
|
|
{
|
2021-11-06 10:10:04 +00:00
|
|
|
a: []storage.SeriesRef{2, 3, 4, 5},
|
|
|
|
b: []storage.SeriesRef{6, 7, 8, 9, 10},
|
2017-04-08 15:12:29 +00:00
|
|
|
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
seek: 1,
|
2017-04-21 20:08:26 +00:00
|
|
|
success: true,
|
2021-11-06 10:10:04 +00:00
|
|
|
res: []storage.SeriesRef{2, 3, 4, 5, 6, 7, 8, 9, 10},
|
2017-04-21 20:08:26 +00:00
|
|
|
},
|
|
|
|
{
|
2021-11-06 10:10:04 +00:00
|
|
|
a: []storage.SeriesRef{1, 2, 3, 4, 5},
|
|
|
|
b: []storage.SeriesRef{6, 7, 8, 9, 10},
|
2017-04-08 15:12:29 +00:00
|
|
|
|
2017-04-21 20:08:26 +00:00
|
|
|
seek: 2,
|
|
|
|
success: true,
|
2021-11-06 10:10:04 +00:00
|
|
|
res: []storage.SeriesRef{2, 3, 4, 5, 6, 7, 8, 9, 10},
|
2017-04-21 20:08:26 +00:00
|
|
|
},
|
|
|
|
{
|
2021-11-06 10:10:04 +00:00
|
|
|
a: []storage.SeriesRef{1, 2, 3, 4, 5},
|
|
|
|
b: []storage.SeriesRef{4, 5, 6, 7, 8},
|
2017-04-09 11:50:39 +00:00
|
|
|
|
2017-04-21 20:08:26 +00:00
|
|
|
seek: 9,
|
|
|
|
success: false,
|
|
|
|
res: nil,
|
|
|
|
},
|
|
|
|
{
|
2021-11-06 10:10:04 +00:00
|
|
|
a: []storage.SeriesRef{1, 2, 3, 4, 9, 10},
|
|
|
|
b: []storage.SeriesRef{1, 4, 5, 6, 7, 8, 10, 11},
|
2017-04-09 11:50:39 +00:00
|
|
|
|
2017-04-21 20:08:26 +00:00
|
|
|
seek: 10,
|
|
|
|
success: true,
|
2021-11-06 10:10:04 +00:00
|
|
|
res: []storage.SeriesRef{10, 11},
|
2017-04-21 20:08:26 +00:00
|
|
|
},
|
|
|
|
}
|
|
|
|
|
|
|
|
for _, c := range cases {
|
2019-03-21 16:23:00 +00:00
|
|
|
a := newListPostings(c.a...)
|
|
|
|
b := newListPostings(c.b...)
|
2017-04-21 20:08:26 +00:00
|
|
|
|
2019-01-03 16:59:52 +00:00
|
|
|
p := Merge(a, b)
|
2017-04-21 20:08:26 +00:00
|
|
|
|
2020-10-29 09:43:23 +00:00
|
|
|
require.Equal(t, c.success, p.Seek(c.seek))
|
2017-12-17 18:08:21 +00:00
|
|
|
|
|
|
|
// After Seek(), At() should be called.
|
|
|
|
if c.success {
|
|
|
|
start := p.At()
|
2017-11-30 14:34:49 +00:00
|
|
|
lst, err := ExpandPostings(p)
|
2020-10-29 09:43:23 +00:00
|
|
|
require.NoError(t, err)
|
2017-12-17 18:08:21 +00:00
|
|
|
|
2021-11-06 10:10:04 +00:00
|
|
|
lst = append([]storage.SeriesRef{start}, lst...)
|
2020-10-29 09:43:23 +00:00
|
|
|
require.Equal(t, c.res, lst)
|
2017-12-17 18:08:21 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func TestRemovedPostings(t *testing.T) {
|
2021-10-22 08:06:44 +00:00
|
|
|
cases := []struct {
|
2021-11-06 10:10:04 +00:00
|
|
|
a, b []storage.SeriesRef
|
|
|
|
res []storage.SeriesRef
|
2017-12-17 18:08:21 +00:00
|
|
|
}{
|
|
|
|
{
|
|
|
|
a: nil,
|
|
|
|
b: nil,
|
2021-11-06 10:10:04 +00:00
|
|
|
res: []storage.SeriesRef(nil),
|
2017-12-17 18:08:21 +00:00
|
|
|
},
|
|
|
|
{
|
2021-11-06 10:10:04 +00:00
|
|
|
a: []storage.SeriesRef{1, 2, 3, 4},
|
2017-12-17 18:08:21 +00:00
|
|
|
b: nil,
|
2021-11-06 10:10:04 +00:00
|
|
|
res: []storage.SeriesRef{1, 2, 3, 4},
|
2017-12-17 18:08:21 +00:00
|
|
|
},
|
|
|
|
{
|
|
|
|
a: nil,
|
2021-11-06 10:10:04 +00:00
|
|
|
b: []storage.SeriesRef{1, 2, 3, 4},
|
|
|
|
res: []storage.SeriesRef(nil),
|
2017-12-17 18:08:21 +00:00
|
|
|
},
|
|
|
|
{
|
2021-11-06 10:10:04 +00:00
|
|
|
a: []storage.SeriesRef{1, 2, 3, 4, 5},
|
|
|
|
b: []storage.SeriesRef{6, 7, 8, 9, 10},
|
|
|
|
res: []storage.SeriesRef{1, 2, 3, 4, 5},
|
2017-12-17 18:08:21 +00:00
|
|
|
},
|
|
|
|
{
|
2021-11-06 10:10:04 +00:00
|
|
|
a: []storage.SeriesRef{1, 2, 3, 4, 5},
|
|
|
|
b: []storage.SeriesRef{4, 5, 6, 7, 8},
|
|
|
|
res: []storage.SeriesRef{1, 2, 3},
|
2017-12-17 18:08:21 +00:00
|
|
|
},
|
|
|
|
{
|
2021-11-06 10:10:04 +00:00
|
|
|
a: []storage.SeriesRef{1, 2, 3, 4, 9, 10},
|
|
|
|
b: []storage.SeriesRef{1, 4, 5, 6, 7, 8, 10, 11},
|
|
|
|
res: []storage.SeriesRef{2, 3, 9},
|
2017-12-17 18:08:21 +00:00
|
|
|
},
|
|
|
|
{
|
2021-11-06 10:10:04 +00:00
|
|
|
a: []storage.SeriesRef{1, 2, 3, 4, 9, 10},
|
|
|
|
b: []storage.SeriesRef{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11},
|
|
|
|
res: []storage.SeriesRef(nil),
|
2017-12-17 18:08:21 +00:00
|
|
|
},
|
|
|
|
}
|
|
|
|
|
|
|
|
for _, c := range cases {
|
2019-03-21 16:23:00 +00:00
|
|
|
a := newListPostings(c.a...)
|
|
|
|
b := newListPostings(c.b...)
|
2017-12-17 18:08:21 +00:00
|
|
|
|
2017-11-30 14:34:49 +00:00
|
|
|
res, err := ExpandPostings(newRemovedPostings(a, b))
|
2020-10-29 09:43:23 +00:00
|
|
|
require.NoError(t, err)
|
|
|
|
require.Equal(t, c.res, res)
|
2017-12-17 18:08:21 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-02-14 13:43:04 +00:00
|
|
|
func TestRemovedNextStackoverflow(t *testing.T) {
|
2021-11-06 10:10:04 +00:00
|
|
|
var full []storage.SeriesRef
|
|
|
|
var remove []storage.SeriesRef
|
2018-02-14 13:43:04 +00:00
|
|
|
|
2021-11-06 10:10:04 +00:00
|
|
|
var i storage.SeriesRef
|
2018-02-14 13:43:04 +00:00
|
|
|
for i = 0; i < 1e7; i++ {
|
|
|
|
full = append(full, i)
|
|
|
|
remove = append(remove, i)
|
|
|
|
}
|
|
|
|
|
2019-03-21 16:23:00 +00:00
|
|
|
flp := newListPostings(full...)
|
|
|
|
rlp := newListPostings(remove...)
|
2018-02-14 13:43:04 +00:00
|
|
|
rp := newRemovedPostings(flp, rlp)
|
|
|
|
gotElem := false
|
|
|
|
for rp.Next() {
|
|
|
|
gotElem = true
|
|
|
|
}
|
2018-05-07 12:39:54 +00:00
|
|
|
|
2020-10-29 09:43:23 +00:00
|
|
|
require.NoError(t, rp.Err())
|
|
|
|
require.False(t, gotElem)
|
2018-02-14 13:43:04 +00:00
|
|
|
}
|
|
|
|
|
2017-12-17 18:08:21 +00:00
|
|
|
func TestRemovedPostingsSeek(t *testing.T) {
|
2021-10-22 08:06:44 +00:00
|
|
|
cases := []struct {
|
2021-11-06 10:10:04 +00:00
|
|
|
a, b []storage.SeriesRef
|
2017-12-17 18:08:21 +00:00
|
|
|
|
2021-11-06 10:10:04 +00:00
|
|
|
seek storage.SeriesRef
|
2017-12-17 18:08:21 +00:00
|
|
|
success bool
|
2021-11-06 10:10:04 +00:00
|
|
|
res []storage.SeriesRef
|
2017-12-17 18:08:21 +00:00
|
|
|
}{
|
|
|
|
{
|
2021-11-06 10:10:04 +00:00
|
|
|
a: []storage.SeriesRef{2, 3, 4, 5},
|
|
|
|
b: []storage.SeriesRef{6, 7, 8, 9, 10},
|
2017-12-17 18:08:21 +00:00
|
|
|
|
|
|
|
seek: 1,
|
|
|
|
success: true,
|
2021-11-06 10:10:04 +00:00
|
|
|
res: []storage.SeriesRef{2, 3, 4, 5},
|
2017-12-17 18:08:21 +00:00
|
|
|
},
|
|
|
|
{
|
2021-11-06 10:10:04 +00:00
|
|
|
a: []storage.SeriesRef{1, 2, 3, 4, 5},
|
|
|
|
b: []storage.SeriesRef{6, 7, 8, 9, 10},
|
2017-12-17 18:08:21 +00:00
|
|
|
|
|
|
|
seek: 2,
|
|
|
|
success: true,
|
2021-11-06 10:10:04 +00:00
|
|
|
res: []storage.SeriesRef{2, 3, 4, 5},
|
2017-12-17 18:08:21 +00:00
|
|
|
},
|
|
|
|
{
|
2021-11-06 10:10:04 +00:00
|
|
|
a: []storage.SeriesRef{1, 2, 3, 4, 5},
|
|
|
|
b: []storage.SeriesRef{4, 5, 6, 7, 8},
|
2017-12-17 18:08:21 +00:00
|
|
|
|
|
|
|
seek: 9,
|
|
|
|
success: false,
|
|
|
|
res: nil,
|
|
|
|
},
|
|
|
|
{
|
2021-11-06 10:10:04 +00:00
|
|
|
a: []storage.SeriesRef{1, 2, 3, 4, 9, 10},
|
|
|
|
b: []storage.SeriesRef{1, 4, 5, 6, 7, 8, 10, 11},
|
2017-12-17 18:08:21 +00:00
|
|
|
|
|
|
|
seek: 10,
|
|
|
|
success: false,
|
|
|
|
res: nil,
|
|
|
|
},
|
|
|
|
{
|
2021-11-06 10:10:04 +00:00
|
|
|
a: []storage.SeriesRef{1, 2, 3, 4, 9, 10},
|
|
|
|
b: []storage.SeriesRef{1, 4, 5, 6, 7, 8, 11},
|
2017-12-17 18:08:21 +00:00
|
|
|
|
|
|
|
seek: 4,
|
|
|
|
success: true,
|
2021-11-06 10:10:04 +00:00
|
|
|
res: []storage.SeriesRef{9, 10},
|
2017-12-17 18:08:21 +00:00
|
|
|
},
|
|
|
|
{
|
2021-11-06 10:10:04 +00:00
|
|
|
a: []storage.SeriesRef{1, 2, 3, 4, 9, 10},
|
|
|
|
b: []storage.SeriesRef{1, 4, 5, 6, 7, 8, 11},
|
2017-12-17 18:08:21 +00:00
|
|
|
|
|
|
|
seek: 5,
|
|
|
|
success: true,
|
2021-11-06 10:10:04 +00:00
|
|
|
res: []storage.SeriesRef{9, 10},
|
2017-12-17 18:08:21 +00:00
|
|
|
},
|
|
|
|
{
|
2021-11-06 10:10:04 +00:00
|
|
|
a: []storage.SeriesRef{1, 2, 3, 4, 9, 10},
|
|
|
|
b: []storage.SeriesRef{1, 4, 5, 6, 7, 8, 11},
|
2017-12-17 18:08:21 +00:00
|
|
|
|
|
|
|
seek: 10,
|
|
|
|
success: true,
|
2021-11-06 10:10:04 +00:00
|
|
|
res: []storage.SeriesRef{10},
|
2017-12-17 18:08:21 +00:00
|
|
|
},
|
|
|
|
}
|
|
|
|
|
|
|
|
for _, c := range cases {
|
2019-03-21 16:23:00 +00:00
|
|
|
a := newListPostings(c.a...)
|
|
|
|
b := newListPostings(c.b...)
|
2017-12-17 18:08:21 +00:00
|
|
|
|
|
|
|
p := newRemovedPostings(a, b)
|
|
|
|
|
2020-10-29 09:43:23 +00:00
|
|
|
require.Equal(t, c.success, p.Seek(c.seek))
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
|
|
|
|
// After Seek(), At() should be called.
|
|
|
|
if c.success {
|
|
|
|
start := p.At()
|
2017-11-30 14:34:49 +00:00
|
|
|
lst, err := ExpandPostings(p)
|
2020-10-29 09:43:23 +00:00
|
|
|
require.NoError(t, err)
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
|
2021-11-06 10:10:04 +00:00
|
|
|
lst = append([]storage.SeriesRef{start}, lst...)
|
2020-10-29 09:43:23 +00:00
|
|
|
require.Equal(t, c.res, lst)
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
}
|
2016-12-28 10:02:19 +00:00
|
|
|
}
|
|
|
|
}
|
2017-03-27 10:16:55 +00:00
|
|
|
|
|
|
|
func TestBigEndian(t *testing.T) {
|
|
|
|
num := 1000
|
|
|
|
// mock a list as postings
|
|
|
|
ls := make([]uint32, num)
|
|
|
|
ls[0] = 2
|
|
|
|
for i := 1; i < num; i++ {
|
|
|
|
ls[i] = ls[i-1] + uint32(rand.Int31n(25)) + 2
|
|
|
|
}
|
|
|
|
|
|
|
|
beLst := make([]byte, num*4)
|
|
|
|
for i := 0; i < num; i++ {
|
|
|
|
b := beLst[i*4 : i*4+4]
|
|
|
|
binary.BigEndian.PutUint32(b, ls[i])
|
|
|
|
}
|
|
|
|
|
|
|
|
t.Run("Iteration", func(t *testing.T) {
|
|
|
|
bep := newBigEndianPostings(beLst)
|
|
|
|
for i := 0; i < num; i++ {
|
2020-10-29 09:43:23 +00:00
|
|
|
require.True(t, bep.Next())
|
2021-11-06 10:10:04 +00:00
|
|
|
require.Equal(t, storage.SeriesRef(ls[i]), bep.At())
|
2017-03-27 10:16:55 +00:00
|
|
|
}
|
|
|
|
|
2020-10-29 09:43:23 +00:00
|
|
|
require.False(t, bep.Next())
|
|
|
|
require.NoError(t, bep.Err())
|
2017-03-27 10:16:55 +00:00
|
|
|
})
|
|
|
|
|
|
|
|
t.Run("Seek", func(t *testing.T) {
|
|
|
|
table := []struct {
|
|
|
|
seek uint32
|
|
|
|
val uint32
|
|
|
|
found bool
|
|
|
|
}{
|
|
|
|
{
|
|
|
|
ls[0] - 1, ls[0], true,
|
|
|
|
},
|
|
|
|
{
|
|
|
|
ls[4], ls[4], true,
|
|
|
|
},
|
|
|
|
{
|
|
|
|
ls[500] - 1, ls[500], true,
|
|
|
|
},
|
|
|
|
{
|
|
|
|
ls[600] + 1, ls[601], true,
|
|
|
|
},
|
|
|
|
{
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
ls[600] + 1, ls[601], true,
|
2017-03-27 10:16:55 +00:00
|
|
|
},
|
|
|
|
{
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
ls[600] + 1, ls[601], true,
|
2017-03-27 10:16:55 +00:00
|
|
|
},
|
|
|
|
{
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
ls[0], ls[601], true,
|
2017-03-27 10:16:55 +00:00
|
|
|
},
|
|
|
|
{
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
ls[600], ls[601], true,
|
2017-03-27 10:16:55 +00:00
|
|
|
},
|
|
|
|
{
|
|
|
|
ls[999], ls[999], true,
|
|
|
|
},
|
|
|
|
{
|
|
|
|
ls[999] + 10, ls[999], false,
|
|
|
|
},
|
|
|
|
}
|
|
|
|
|
|
|
|
bep := newBigEndianPostings(beLst)
|
|
|
|
|
|
|
|
for _, v := range table {
|
2021-11-06 10:10:04 +00:00
|
|
|
require.Equal(t, v.found, bep.Seek(storage.SeriesRef(v.seek)))
|
|
|
|
require.Equal(t, storage.SeriesRef(v.val), bep.At())
|
2020-10-29 09:43:23 +00:00
|
|
|
require.NoError(t, bep.Err())
|
2017-03-27 10:16:55 +00:00
|
|
|
}
|
|
|
|
})
|
|
|
|
}
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
|
|
|
|
func TestIntersectWithMerge(t *testing.T) {
|
2019-03-21 16:23:00 +00:00
|
|
|
// One of the reproducible cases for:
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
// https://github.com/prometheus/prometheus/issues/2616
|
2019-03-21 16:23:00 +00:00
|
|
|
a := newListPostings(21, 22, 23, 24, 25, 30)
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
|
2019-01-03 16:59:52 +00:00
|
|
|
b := Merge(
|
2019-03-21 16:23:00 +00:00
|
|
|
newListPostings(10, 20, 30),
|
|
|
|
newListPostings(15, 26, 30),
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
)
|
|
|
|
|
|
|
|
p := Intersect(a, b)
|
2017-11-30 14:34:49 +00:00
|
|
|
res, err := ExpandPostings(p)
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
|
2020-10-29 09:43:23 +00:00
|
|
|
require.NoError(t, err)
|
2021-11-06 10:10:04 +00:00
|
|
|
require.Equal(t, []storage.SeriesRef{30}, res)
|
Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect
Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Advance on Seek only when valid.
Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.
Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.
Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.
Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.
Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add test for Merge+Intersect edgecase.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
* Add comments to trivial tests.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 07:44:41 +00:00
|
|
|
}
|
2019-03-21 16:23:00 +00:00
|
|
|
|
|
|
|
func TestWithoutPostings(t *testing.T) {
|
2021-10-22 08:06:44 +00:00
|
|
|
cases := []struct {
|
2019-03-21 16:23:00 +00:00
|
|
|
base Postings
|
|
|
|
drop Postings
|
|
|
|
|
|
|
|
res Postings
|
|
|
|
}{
|
|
|
|
{
|
|
|
|
base: EmptyPostings(),
|
|
|
|
drop: EmptyPostings(),
|
|
|
|
|
|
|
|
res: EmptyPostings(),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
base: EmptyPostings(),
|
|
|
|
drop: newListPostings(1, 2),
|
|
|
|
|
|
|
|
res: EmptyPostings(),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
base: newListPostings(1, 2),
|
|
|
|
drop: EmptyPostings(),
|
|
|
|
|
|
|
|
res: newListPostings(1, 2),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
base: newListPostings(),
|
|
|
|
drop: newListPostings(),
|
|
|
|
|
|
|
|
res: newListPostings(),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
base: newListPostings(1, 2, 3),
|
|
|
|
drop: newListPostings(),
|
|
|
|
|
|
|
|
res: newListPostings(1, 2, 3),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
base: newListPostings(1, 2, 3),
|
|
|
|
drop: newListPostings(4, 5, 6),
|
|
|
|
|
|
|
|
res: newListPostings(1, 2, 3),
|
|
|
|
},
|
|
|
|
{
|
|
|
|
base: newListPostings(1, 2, 3),
|
|
|
|
drop: newListPostings(3, 4, 5),
|
|
|
|
|
|
|
|
res: newListPostings(1, 2),
|
|
|
|
},
|
|
|
|
}
|
|
|
|
|
|
|
|
for _, c := range cases {
|
|
|
|
t.Run("", func(t *testing.T) {
|
|
|
|
if c.res == nil {
|
|
|
|
t.Fatal("without result expectancy cannot be nil")
|
|
|
|
}
|
|
|
|
|
|
|
|
expected, err := ExpandPostings(c.res)
|
2020-10-29 09:43:23 +00:00
|
|
|
require.NoError(t, err)
|
2019-03-21 16:23:00 +00:00
|
|
|
|
|
|
|
w := Without(c.base, c.drop)
|
|
|
|
|
|
|
|
if c.res == EmptyPostings() {
|
2020-10-29 09:43:23 +00:00
|
|
|
require.Equal(t, EmptyPostings(), w)
|
2019-03-21 16:23:00 +00:00
|
|
|
return
|
|
|
|
}
|
|
|
|
|
|
|
|
if w == EmptyPostings() {
|
|
|
|
t.Fatal("without unexpected result: EmptyPostings sentinel")
|
|
|
|
}
|
|
|
|
|
|
|
|
res, err := ExpandPostings(w)
|
2020-10-29 09:43:23 +00:00
|
|
|
require.NoError(t, err)
|
|
|
|
require.Equal(t, expected, res)
|
2019-03-21 16:23:00 +00:00
|
|
|
})
|
|
|
|
}
|
|
|
|
}
|
2019-08-22 12:19:12 +00:00
|
|
|
|
2019-11-05 02:06:13 +00:00
|
|
|
func BenchmarkPostings_Stats(b *testing.B) {
|
|
|
|
p := NewMemPostings()
|
|
|
|
|
2021-11-06 10:10:04 +00:00
|
|
|
var seriesID storage.SeriesRef
|
2021-08-18 09:27:21 +00:00
|
|
|
|
2019-11-05 02:06:13 +00:00
|
|
|
createPostingsLabelValues := func(name, valuePrefix string, count int) {
|
|
|
|
for n := 1; n < count; n++ {
|
|
|
|
value := fmt.Sprintf("%s-%d", valuePrefix, n)
|
2021-08-18 09:27:21 +00:00
|
|
|
p.Add(seriesID, labels.FromStrings(name, value))
|
|
|
|
seriesID++
|
2019-11-05 02:06:13 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
createPostingsLabelValues("__name__", "metrics_name_can_be_very_big_and_bad", 1e3)
|
|
|
|
for i := 0; i < 20; i++ {
|
|
|
|
createPostingsLabelValues(fmt.Sprintf("host-%d", i), "metrics_name_can_be_very_big_and_bad", 1e3)
|
|
|
|
createPostingsLabelValues(fmt.Sprintf("instance-%d", i), "10.0.IP.", 1e3)
|
|
|
|
createPostingsLabelValues(fmt.Sprintf("job-%d", i), "Small_Job_name", 1e3)
|
|
|
|
createPostingsLabelValues(fmt.Sprintf("err-%d", i), "avg_namespace-", 1e3)
|
|
|
|
createPostingsLabelValues(fmt.Sprintf("team-%d", i), "team-", 1e3)
|
|
|
|
createPostingsLabelValues(fmt.Sprintf("container_name-%d", i), "pod-", 1e3)
|
|
|
|
createPostingsLabelValues(fmt.Sprintf("cluster-%d", i), "newcluster-", 1e3)
|
|
|
|
createPostingsLabelValues(fmt.Sprintf("uid-%d", i), "123412312312312311-", 1e3)
|
|
|
|
createPostingsLabelValues(fmt.Sprintf("area-%d", i), "new_area_of_work-", 1e3)
|
|
|
|
createPostingsLabelValues(fmt.Sprintf("request_id-%d", i), "owner_name_work-", 1e3)
|
|
|
|
}
|
|
|
|
b.ResetTimer()
|
|
|
|
for n := 0; n < b.N; n++ {
|
|
|
|
p.Stats("__name__")
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2023-05-19 07:36:30 +00:00
|
|
|
func TestMemPostingsStats(t *testing.T) {
|
|
|
|
// create a new MemPostings
|
|
|
|
p := NewMemPostings()
|
|
|
|
|
|
|
|
// add some postings to the MemPostings
|
|
|
|
p.Add(1, labels.FromStrings("label", "value1"))
|
|
|
|
p.Add(1, labels.FromStrings("label", "value2"))
|
|
|
|
p.Add(1, labels.FromStrings("label", "value3"))
|
|
|
|
p.Add(2, labels.FromStrings("label", "value1"))
|
|
|
|
|
|
|
|
// call the Stats method to calculate the cardinality statistics
|
|
|
|
stats := p.Stats("label")
|
|
|
|
|
|
|
|
// assert that the expected statistics were calculated
|
|
|
|
require.Equal(t, uint64(2), stats.CardinalityMetricsStats[0].Count)
|
|
|
|
require.Equal(t, "value1", stats.CardinalityMetricsStats[0].Name)
|
|
|
|
|
|
|
|
require.Equal(t, uint64(3), stats.CardinalityLabelStats[0].Count)
|
|
|
|
require.Equal(t, "label", stats.CardinalityLabelStats[0].Name)
|
|
|
|
|
|
|
|
require.Equal(t, uint64(24), stats.LabelValueStats[0].Count)
|
|
|
|
require.Equal(t, "label", stats.LabelValueStats[0].Name)
|
|
|
|
|
|
|
|
require.Equal(t, uint64(2), stats.LabelValuePairsStats[0].Count)
|
|
|
|
require.Equal(t, "label=value1", stats.LabelValuePairsStats[0].Name)
|
|
|
|
|
|
|
|
require.Equal(t, 3, stats.NumLabelPairs)
|
|
|
|
}
|
|
|
|
|
2019-08-22 12:19:12 +00:00
|
|
|
func TestMemPostings_Delete(t *testing.T) {
|
|
|
|
p := NewMemPostings()
|
|
|
|
p.Add(1, labels.FromStrings("lbl1", "a"))
|
|
|
|
p.Add(2, labels.FromStrings("lbl1", "b"))
|
|
|
|
p.Add(3, labels.FromStrings("lbl2", "a"))
|
|
|
|
|
|
|
|
before := p.Get(allPostingsKey.Name, allPostingsKey.Value)
|
2021-11-06 10:10:04 +00:00
|
|
|
p.Delete(map[storage.SeriesRef]struct{}{
|
2020-04-15 10:17:41 +00:00
|
|
|
2: {},
|
2019-08-22 12:19:12 +00:00
|
|
|
})
|
|
|
|
after := p.Get(allPostingsKey.Name, allPostingsKey.Value)
|
|
|
|
|
|
|
|
// Make sure postings gotten before the delete have the old data when
|
|
|
|
// iterated over.
|
|
|
|
expanded, err := ExpandPostings(before)
|
2020-10-29 09:43:23 +00:00
|
|
|
require.NoError(t, err)
|
2021-11-06 10:10:04 +00:00
|
|
|
require.Equal(t, []storage.SeriesRef{1, 2, 3}, expanded)
|
2019-08-22 12:19:12 +00:00
|
|
|
|
|
|
|
// Make sure postings gotten after the delete have the new data when
|
|
|
|
// iterated over.
|
|
|
|
expanded, err = ExpandPostings(after)
|
2020-10-29 09:43:23 +00:00
|
|
|
require.NoError(t, err)
|
2021-11-06 10:10:04 +00:00
|
|
|
require.Equal(t, []storage.SeriesRef{1, 3}, expanded)
|
2019-08-22 12:19:12 +00:00
|
|
|
|
|
|
|
deleted := p.Get("lbl1", "b")
|
|
|
|
expanded, err = ExpandPostings(deleted)
|
2020-10-29 09:43:23 +00:00
|
|
|
require.NoError(t, err)
|
|
|
|
require.Equal(t, 0, len(expanded), "expected empty postings, got %v", expanded)
|
2019-08-22 12:19:12 +00:00
|
|
|
}
|
Label values with matchers by intersecting postings (#9907)
* LabelValues w/matchers by intersecting postings
Instead of iterating all matched series to find the values, this
checks if each one of the label values is present in the matched series
(postings).
Pending to be benchmarked.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Benchmark labelValuesWithMatchers
name old time/op new time/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0%
name old alloc/op new alloc/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0%
name old allocs/op new allocs/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Don't expand postings to intersect them
Using a min heap we can check whether matched postings intersect with
each one of the label values postings. This avoid expanding postings
(and thus having all of them in memory at any point).
Slightly slower than the expanding postings version for some cases, but
definitely pays the price once the cardinality grows.
Still offers 10x latency improvement where previous latencies were
reaching 1s.
Benchmark results:
name \ time/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0% 110ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0% 0.18s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0% 125ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0% 177ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0% 134ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0% 4.29µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0% 120ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0% 0.15s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0% 125.4ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0% 170ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0% 0.14s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0% 0.92ms ± 0%
name \ alloc/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0% 36.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0% 6.4kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0% 4.30kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0% 38MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0% 22.3MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0% 0MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0% 5.86kB ± 0%
name \ allocs/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0% 139.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0% 87.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0% 0.50M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0% 129.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Apply comment suggestions from the code review
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
* Change else { if } to else if
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Remove sorting of label values
We were not sorting them before, so no need to sort them now
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2021-12-28 14:59:03 +00:00
|
|
|
|
|
|
|
func TestFindIntersectingPostings(t *testing.T) {
|
|
|
|
t.Run("multiple intersections", func(t *testing.T) {
|
|
|
|
p := NewListPostings([]storage.SeriesRef{10, 15, 20, 25, 30, 35, 40, 45, 50})
|
|
|
|
candidates := []Postings{
|
|
|
|
0: NewListPostings([]storage.SeriesRef{7, 13, 14, 27}), // Does not intersect.
|
|
|
|
1: NewListPostings([]storage.SeriesRef{10, 20}), // Does intersect.
|
|
|
|
2: NewListPostings([]storage.SeriesRef{29, 30, 31}), // Does intersect.
|
|
|
|
3: NewListPostings([]storage.SeriesRef{29, 30, 31}), // Does intersect (same again).
|
|
|
|
4: NewListPostings([]storage.SeriesRef{60}), // Does not intersect.
|
|
|
|
5: NewListPostings([]storage.SeriesRef{45}), // Does intersect.
|
|
|
|
6: EmptyPostings(), // Does not intersect.
|
|
|
|
}
|
|
|
|
|
|
|
|
indexes, err := FindIntersectingPostings(p, candidates)
|
|
|
|
require.NoError(t, err)
|
|
|
|
sort.Ints(indexes)
|
|
|
|
require.Equal(t, []int{1, 2, 3, 5}, indexes)
|
|
|
|
})
|
|
|
|
|
|
|
|
t.Run("no intersections", func(t *testing.T) {
|
|
|
|
p := NewListPostings([]storage.SeriesRef{10, 15, 20, 25, 30, 35, 40, 45, 50})
|
|
|
|
candidates := []Postings{
|
|
|
|
0: NewListPostings([]storage.SeriesRef{7, 13, 14, 27}), // Does not intersect.
|
|
|
|
1: NewListPostings([]storage.SeriesRef{60}), // Does not intersect.
|
|
|
|
2: EmptyPostings(), // Does not intersect.
|
|
|
|
}
|
|
|
|
|
|
|
|
indexes, err := FindIntersectingPostings(p, candidates)
|
|
|
|
require.NoError(t, err)
|
|
|
|
require.Empty(t, indexes)
|
|
|
|
})
|
|
|
|
|
|
|
|
t.Run("p is ErrPostings", func(t *testing.T) {
|
|
|
|
p := ErrPostings(context.Canceled)
|
|
|
|
candidates := []Postings{NewListPostings([]storage.SeriesRef{1})}
|
|
|
|
_, err := FindIntersectingPostings(p, candidates)
|
|
|
|
require.Error(t, err)
|
|
|
|
})
|
|
|
|
|
|
|
|
t.Run("one of the candidates is ErrPostings", func(t *testing.T) {
|
|
|
|
p := NewListPostings([]storage.SeriesRef{1})
|
|
|
|
candidates := []Postings{
|
|
|
|
NewListPostings([]storage.SeriesRef{1}),
|
|
|
|
ErrPostings(context.Canceled),
|
|
|
|
}
|
|
|
|
_, err := FindIntersectingPostings(p, candidates)
|
|
|
|
require.Error(t, err)
|
|
|
|
})
|
|
|
|
|
|
|
|
t.Run("one of the candidates fails on nth call", func(t *testing.T) {
|
|
|
|
p := NewListPostings([]storage.SeriesRef{10, 20, 30, 40, 50, 60, 70})
|
|
|
|
candidates := []Postings{
|
|
|
|
NewListPostings([]storage.SeriesRef{7, 13, 14, 27}),
|
|
|
|
&postingsFailingAfterNthCall{2, NewListPostings([]storage.SeriesRef{29, 30, 31, 40})},
|
|
|
|
}
|
|
|
|
_, err := FindIntersectingPostings(p, candidates)
|
|
|
|
require.Error(t, err)
|
|
|
|
})
|
|
|
|
|
|
|
|
t.Run("p fails on the nth call", func(t *testing.T) {
|
|
|
|
p := &postingsFailingAfterNthCall{2, NewListPostings([]storage.SeriesRef{10, 20, 30, 40, 50, 60, 70})}
|
|
|
|
candidates := []Postings{
|
|
|
|
NewListPostings([]storage.SeriesRef{7, 13, 14, 27}),
|
|
|
|
NewListPostings([]storage.SeriesRef{29, 30, 31, 40}),
|
|
|
|
}
|
|
|
|
_, err := FindIntersectingPostings(p, candidates)
|
|
|
|
require.Error(t, err)
|
|
|
|
})
|
|
|
|
}
|
|
|
|
|
|
|
|
type postingsFailingAfterNthCall struct {
|
|
|
|
ttl int
|
|
|
|
Postings
|
|
|
|
}
|
|
|
|
|
|
|
|
func (p *postingsFailingAfterNthCall) Seek(v storage.SeriesRef) bool {
|
|
|
|
p.ttl--
|
|
|
|
if p.ttl <= 0 {
|
|
|
|
return false
|
|
|
|
}
|
|
|
|
return p.Postings.Seek(v)
|
|
|
|
}
|
|
|
|
|
|
|
|
func (p *postingsFailingAfterNthCall) Next() bool {
|
|
|
|
p.ttl--
|
|
|
|
if p.ttl <= 0 {
|
|
|
|
return false
|
|
|
|
}
|
|
|
|
return p.Postings.Next()
|
|
|
|
}
|
|
|
|
|
|
|
|
func (p *postingsFailingAfterNthCall) Err() error {
|
|
|
|
if p.ttl <= 0 {
|
|
|
|
return errors.New("ttl exceeded")
|
|
|
|
}
|
|
|
|
return p.Postings.Err()
|
|
|
|
}
|
|
|
|
|
|
|
|
func TestPostingsWithIndexHeap(t *testing.T) {
|
|
|
|
t.Run("iterate", func(t *testing.T) {
|
|
|
|
h := postingsWithIndexHeap{
|
|
|
|
{index: 0, p: NewListPostings([]storage.SeriesRef{10, 20, 30})},
|
|
|
|
{index: 1, p: NewListPostings([]storage.SeriesRef{1, 5})},
|
|
|
|
{index: 2, p: NewListPostings([]storage.SeriesRef{25, 50})},
|
|
|
|
}
|
|
|
|
for _, node := range h {
|
|
|
|
node.p.Next()
|
|
|
|
}
|
2022-01-06 10:33:44 +00:00
|
|
|
heap.Init(&h)
|
Label values with matchers by intersecting postings (#9907)
* LabelValues w/matchers by intersecting postings
Instead of iterating all matched series to find the values, this
checks if each one of the label values is present in the matched series
(postings).
Pending to be benchmarked.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Benchmark labelValuesWithMatchers
name old time/op new time/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0%
name old alloc/op new alloc/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0%
name old allocs/op new allocs/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Don't expand postings to intersect them
Using a min heap we can check whether matched postings intersect with
each one of the label values postings. This avoid expanding postings
(and thus having all of them in memory at any point).
Slightly slower than the expanding postings version for some cases, but
definitely pays the price once the cardinality grows.
Still offers 10x latency improvement where previous latencies were
reaching 1s.
Benchmark results:
name \ time/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0% 110ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0% 0.18s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0% 125ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0% 177ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0% 134ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0% 4.29µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0% 120ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0% 0.15s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0% 125.4ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0% 170ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0% 0.14s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0% 0.92ms ± 0%
name \ alloc/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0% 36.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0% 6.4kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0% 4.30kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0% 38MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0% 22.3MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0% 0MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0% 5.86kB ± 0%
name \ allocs/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0% 139.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0% 87.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0% 0.50M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0% 129.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Apply comment suggestions from the code review
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
* Change else { if } to else if
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Remove sorting of label values
We were not sorting them before, so no need to sort them now
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2021-12-28 14:59:03 +00:00
|
|
|
|
|
|
|
for _, expected := range []storage.SeriesRef{1, 5, 10, 20, 25, 30, 50} {
|
2022-01-06 10:33:44 +00:00
|
|
|
require.Equal(t, expected, h.at())
|
|
|
|
require.NoError(t, h.next())
|
Label values with matchers by intersecting postings (#9907)
* LabelValues w/matchers by intersecting postings
Instead of iterating all matched series to find the values, this
checks if each one of the label values is present in the matched series
(postings).
Pending to be benchmarked.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Benchmark labelValuesWithMatchers
name old time/op new time/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0%
name old alloc/op new alloc/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0%
name old allocs/op new allocs/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Don't expand postings to intersect them
Using a min heap we can check whether matched postings intersect with
each one of the label values postings. This avoid expanding postings
(and thus having all of them in memory at any point).
Slightly slower than the expanding postings version for some cases, but
definitely pays the price once the cardinality grows.
Still offers 10x latency improvement where previous latencies were
reaching 1s.
Benchmark results:
name \ time/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0% 110ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0% 0.18s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0% 125ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0% 177ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0% 134ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0% 4.29µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0% 120ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0% 0.15s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0% 125.4ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0% 170ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0% 0.14s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0% 0.92ms ± 0%
name \ alloc/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0% 36.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0% 6.4kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0% 4.30kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0% 38MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0% 22.3MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0% 0MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0% 5.86kB ± 0%
name \ allocs/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0% 139.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0% 87.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0% 0.50M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0% 129.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Apply comment suggestions from the code review
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
* Change else { if } to else if
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Remove sorting of label values
We were not sorting them before, so no need to sort them now
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2021-12-28 14:59:03 +00:00
|
|
|
}
|
2022-01-05 10:46:43 +00:00
|
|
|
require.True(t, h.empty())
|
Label values with matchers by intersecting postings (#9907)
* LabelValues w/matchers by intersecting postings
Instead of iterating all matched series to find the values, this
checks if each one of the label values is present in the matched series
(postings).
Pending to be benchmarked.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Benchmark labelValuesWithMatchers
name old time/op new time/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0%
name old alloc/op new alloc/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0%
name old allocs/op new allocs/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Don't expand postings to intersect them
Using a min heap we can check whether matched postings intersect with
each one of the label values postings. This avoid expanding postings
(and thus having all of them in memory at any point).
Slightly slower than the expanding postings version for some cases, but
definitely pays the price once the cardinality grows.
Still offers 10x latency improvement where previous latencies were
reaching 1s.
Benchmark results:
name \ time/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0% 110ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0% 0.18s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0% 125ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0% 177ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0% 134ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0% 4.29µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0% 120ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0% 0.15s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0% 125.4ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0% 170ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0% 0.14s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0% 0.92ms ± 0%
name \ alloc/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0% 36.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0% 6.4kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0% 4.30kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0% 38MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0% 22.3MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0% 0MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0% 5.86kB ± 0%
name \ allocs/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0% 139.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0% 87.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0% 0.50M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0% 129.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Apply comment suggestions from the code review
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
* Change else { if } to else if
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Remove sorting of label values
We were not sorting them before, so no need to sort them now
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2021-12-28 14:59:03 +00:00
|
|
|
})
|
|
|
|
|
|
|
|
t.Run("pop", func(t *testing.T) {
|
|
|
|
h := postingsWithIndexHeap{
|
|
|
|
{index: 0, p: NewListPostings([]storage.SeriesRef{10, 20, 30})},
|
|
|
|
{index: 1, p: NewListPostings([]storage.SeriesRef{1, 5})},
|
|
|
|
{index: 2, p: NewListPostings([]storage.SeriesRef{25, 50})},
|
|
|
|
}
|
|
|
|
for _, node := range h {
|
|
|
|
node.p.Next()
|
|
|
|
}
|
2022-01-06 10:33:44 +00:00
|
|
|
heap.Init(&h)
|
Label values with matchers by intersecting postings (#9907)
* LabelValues w/matchers by intersecting postings
Instead of iterating all matched series to find the values, this
checks if each one of the label values is present in the matched series
(postings).
Pending to be benchmarked.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Benchmark labelValuesWithMatchers
name old time/op new time/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0%
name old alloc/op new alloc/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0%
name old allocs/op new allocs/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Don't expand postings to intersect them
Using a min heap we can check whether matched postings intersect with
each one of the label values postings. This avoid expanding postings
(and thus having all of them in memory at any point).
Slightly slower than the expanding postings version for some cases, but
definitely pays the price once the cardinality grows.
Still offers 10x latency improvement where previous latencies were
reaching 1s.
Benchmark results:
name \ time/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0% 110ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0% 0.18s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0% 125ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0% 177ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0% 134ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0% 4.29µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0% 120ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0% 0.15s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0% 125.4ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0% 170ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0% 0.14s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0% 0.92ms ± 0%
name \ alloc/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0% 36.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0% 6.4kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0% 4.30kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0% 38MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0% 22.3MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0% 0MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0% 5.86kB ± 0%
name \ allocs/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0% 139.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0% 87.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0% 0.50M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0% 129.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Apply comment suggestions from the code review
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
* Change else { if } to else if
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Remove sorting of label values
We were not sorting them before, so no need to sort them now
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2021-12-28 14:59:03 +00:00
|
|
|
|
|
|
|
for _, expected := range []storage.SeriesRef{1, 5, 10, 20} {
|
2022-01-06 10:33:44 +00:00
|
|
|
require.Equal(t, expected, h.at())
|
|
|
|
require.NoError(t, h.next())
|
Label values with matchers by intersecting postings (#9907)
* LabelValues w/matchers by intersecting postings
Instead of iterating all matched series to find the values, this
checks if each one of the label values is present in the matched series
(postings).
Pending to be benchmarked.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Benchmark labelValuesWithMatchers
name old time/op new time/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0%
name old alloc/op new alloc/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0%
name old allocs/op new allocs/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Don't expand postings to intersect them
Using a min heap we can check whether matched postings intersect with
each one of the label values postings. This avoid expanding postings
(and thus having all of them in memory at any point).
Slightly slower than the expanding postings version for some cases, but
definitely pays the price once the cardinality grows.
Still offers 10x latency improvement where previous latencies were
reaching 1s.
Benchmark results:
name \ time/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0% 110ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0% 0.18s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0% 125ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0% 177ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0% 134ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0% 4.29µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0% 120ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0% 0.15s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0% 125.4ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0% 170ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0% 0.14s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0% 0.92ms ± 0%
name \ alloc/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0% 36.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0% 6.4kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0% 4.30kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0% 38MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0% 22.3MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0% 0MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0% 5.86kB ± 0%
name \ allocs/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0% 139.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0% 87.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0% 0.50M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0% 129.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Apply comment suggestions from the code review
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
* Change else { if } to else if
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Remove sorting of label values
We were not sorting them before, so no need to sort them now
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2021-12-28 14:59:03 +00:00
|
|
|
}
|
2022-01-06 10:33:44 +00:00
|
|
|
require.Equal(t, storage.SeriesRef(25), h.at())
|
Label values with matchers by intersecting postings (#9907)
* LabelValues w/matchers by intersecting postings
Instead of iterating all matched series to find the values, this
checks if each one of the label values is present in the matched series
(postings).
Pending to be benchmarked.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Benchmark labelValuesWithMatchers
name old time/op new time/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0%
name old alloc/op new alloc/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0%
name old allocs/op new allocs/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Don't expand postings to intersect them
Using a min heap we can check whether matched postings intersect with
each one of the label values postings. This avoid expanding postings
(and thus having all of them in memory at any point).
Slightly slower than the expanding postings version for some cases, but
definitely pays the price once the cardinality grows.
Still offers 10x latency improvement where previous latencies were
reaching 1s.
Benchmark results:
name \ time/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0% 110ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0% 0.18s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0% 125ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0% 177ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0% 134ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0% 4.29µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0% 120ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0% 0.15s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0% 125.4ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0% 170ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0% 0.14s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0% 0.92ms ± 0%
name \ alloc/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0% 36.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0% 6.4kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0% 4.30kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0% 38MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0% 22.3MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0% 0MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0% 5.86kB ± 0%
name \ allocs/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0% 139.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0% 87.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0% 0.50M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0% 129.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Apply comment suggestions from the code review
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
* Change else { if } to else if
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Remove sorting of label values
We were not sorting them before, so no need to sort them now
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2021-12-28 14:59:03 +00:00
|
|
|
node := heap.Pop(&h).(postingsWithIndex)
|
|
|
|
require.Equal(t, 2, node.index)
|
|
|
|
require.Equal(t, storage.SeriesRef(25), node.p.At())
|
|
|
|
})
|
|
|
|
}
|