2018-10-09 13:09:44 +00:00
|
|
|
// Copyright 2018 The Prometheus Authors
|
2018-10-04 13:52:03 +00:00
|
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
|
|
// you may not use this file except in compliance with the License.
|
|
|
|
// You may obtain a copy of the License at
|
|
|
|
//
|
|
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
//
|
|
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
// See the License for the specific language governing permissions and
|
|
|
|
// limitations under the License.
|
|
|
|
|
|
|
|
package textparse
|
|
|
|
|
|
|
|
import (
|
2024-10-18 15:12:31 +00:00
|
|
|
"errors"
|
|
|
|
"fmt"
|
2018-10-05 16:11:16 +00:00
|
|
|
"mime"
|
|
|
|
|
2023-11-22 14:39:21 +00:00
|
|
|
"github.com/prometheus/common/model"
|
|
|
|
|
2021-11-08 14:23:17 +00:00
|
|
|
"github.com/prometheus/prometheus/model/exemplar"
|
Style cleanup of all the changes in sparsehistogram so far
A lot of this code was hacked together, literally during a
hackathon. This commit intends not to change the code substantially,
but just make the code obey the usual style practices.
A (possibly incomplete) list of areas:
* Generally address linter warnings.
* The `pgk` directory is deprecated as per dev-summit. No new packages should
be added to it. I moved the new `pkg/histogram` package to `model`
anticipating what's proposed in #9478.
* Make the naming of the Sparse Histogram more consistent. Including
abbreviations, there were just too many names for it: SparseHistogram,
Histogram, Histo, hist, his, shs, h. The idea is to call it "Histogram" in
general. Only add "Sparse" if it is needed to avoid confusion with
conventional Histograms (which is rare because the TSDB really has no notion
of conventional Histograms). Use abbreviations only in local scope, and then
really abbreviate (not just removing three out of seven letters like in
"Histo"). This is in the spirit of
https://github.com/golang/go/wiki/CodeReviewComments#variable-names
* Several other minor name changes.
* A lot of formatting of doc comments. For one, following
https://github.com/golang/go/wiki/CodeReviewComments#comment-sentences
, but also layout question, anticipating how things will look like
when rendered by `godoc` (even where `godoc` doesn't render them
right now because they are for unexported types or not a doc comment
at all but just a normal code comment - consistency is queen!).
* Re-enabled `TestQueryLog` and `TestEndopints` (they pass now,
leaving them disabled was presumably an oversight).
* Bucket iterator for histogram.Histogram is now created with a
method.
* HistogramChunk.iterator now allows iterator recycling. (I think
@dieterbe only commented it out because he was confused by the
question in the comment.)
* HistogramAppender.Append panics now because we decided to treat
staleness marker differently.
Signed-off-by: beorn7 <beorn@grafana.com>
2021-10-09 13:57:07 +00:00
|
|
|
"github.com/prometheus/prometheus/model/histogram"
|
2021-11-08 14:23:17 +00:00
|
|
|
"github.com/prometheus/prometheus/model/labels"
|
2018-10-04 13:52:03 +00:00
|
|
|
)
|
|
|
|
|
2024-10-15 14:23:27 +00:00
|
|
|
// Parser parses samples from a byte slice of samples in different exposition formats.
|
2018-10-04 13:52:03 +00:00
|
|
|
type Parser interface {
|
2021-06-29 21:45:23 +00:00
|
|
|
// Series returns the bytes of a series with a simple float64 as a
|
|
|
|
// value, the timestamp if set, and the value of the current sample.
|
2018-10-04 13:52:03 +00:00
|
|
|
Series() ([]byte, *int64, float64)
|
|
|
|
|
2021-06-29 21:45:23 +00:00
|
|
|
// Histogram returns the bytes of a series with a sparse histogram as a
|
Style cleanup of all the changes in sparsehistogram so far
A lot of this code was hacked together, literally during a
hackathon. This commit intends not to change the code substantially,
but just make the code obey the usual style practices.
A (possibly incomplete) list of areas:
* Generally address linter warnings.
* The `pgk` directory is deprecated as per dev-summit. No new packages should
be added to it. I moved the new `pkg/histogram` package to `model`
anticipating what's proposed in #9478.
* Make the naming of the Sparse Histogram more consistent. Including
abbreviations, there were just too many names for it: SparseHistogram,
Histogram, Histo, hist, his, shs, h. The idea is to call it "Histogram" in
general. Only add "Sparse" if it is needed to avoid confusion with
conventional Histograms (which is rare because the TSDB really has no notion
of conventional Histograms). Use abbreviations only in local scope, and then
really abbreviate (not just removing three out of seven letters like in
"Histo"). This is in the spirit of
https://github.com/golang/go/wiki/CodeReviewComments#variable-names
* Several other minor name changes.
* A lot of formatting of doc comments. For one, following
https://github.com/golang/go/wiki/CodeReviewComments#comment-sentences
, but also layout question, anticipating how things will look like
when rendered by `godoc` (even where `godoc` doesn't render them
right now because they are for unexported types or not a doc comment
at all but just a normal code comment - consistency is queen!).
* Re-enabled `TestQueryLog` and `TestEndopints` (they pass now,
leaving them disabled was presumably an oversight).
* Bucket iterator for histogram.Histogram is now created with a
method.
* HistogramChunk.iterator now allows iterator recycling. (I think
@dieterbe only commented it out because he was confused by the
question in the comment.)
* HistogramAppender.Append panics now because we decided to treat
staleness marker differently.
Signed-off-by: beorn7 <beorn@grafana.com>
2021-10-09 13:57:07 +00:00
|
|
|
// value, the timestamp if set, and the histogram in the current sample.
|
2022-08-25 15:07:41 +00:00
|
|
|
// Depending on the parsed input, the function returns an (integer) Histogram
|
|
|
|
// or a FloatHistogram, with the respective other return value being nil.
|
|
|
|
Histogram() ([]byte, *int64, *histogram.Histogram, *histogram.FloatHistogram)
|
2021-06-29 21:45:23 +00:00
|
|
|
|
2018-10-04 13:52:03 +00:00
|
|
|
// Help returns the metric name and help text in the current entry.
|
|
|
|
// Must only be called after Next returned a help entry.
|
|
|
|
// The returned byte slices become invalid after the next call to Next.
|
|
|
|
Help() ([]byte, []byte)
|
|
|
|
|
|
|
|
// Type returns the metric name and type in the current entry.
|
|
|
|
// Must only be called after Next returned a type entry.
|
|
|
|
// The returned byte slices become invalid after the next call to Next.
|
2023-12-12 12:14:36 +00:00
|
|
|
Type() ([]byte, model.MetricType)
|
2018-10-04 13:52:03 +00:00
|
|
|
|
2018-10-04 16:57:47 +00:00
|
|
|
// Unit returns the metric name and unit in the current entry.
|
|
|
|
// Must only be called after Next returned a unit entry.
|
|
|
|
// The returned byte slices become invalid after the next call to Next.
|
|
|
|
Unit() ([]byte, []byte)
|
|
|
|
|
2018-10-04 13:52:03 +00:00
|
|
|
// Comment returns the text of the current comment.
|
|
|
|
// Must only be called after Next returned a comment entry.
|
|
|
|
// The returned byte slice becomes invalid after the next call to Next.
|
|
|
|
Comment() []byte
|
|
|
|
|
|
|
|
// Metric writes the labels of the current sample into the passed labels.
|
|
|
|
// It returns the string from which the metric was parsed.
|
2024-10-15 14:23:27 +00:00
|
|
|
// The values of the "le" labels of classic histograms and "quantile" labels
|
|
|
|
// of summaries should follow the OpenMetrics formatting rules.
|
2018-10-04 13:52:03 +00:00
|
|
|
Metric(l *labels.Labels) string
|
|
|
|
|
2019-11-19 09:33:30 +00:00
|
|
|
// Exemplar writes the exemplar of the current sample into the passed
|
2023-07-13 12:16:10 +00:00
|
|
|
// exemplar. It can be called repeatedly to retrieve multiple exemplars
|
|
|
|
// for the same sample. It returns false once all exemplars are
|
|
|
|
// retrieved (including the case where no exemplars exist at all).
|
2019-11-19 09:33:30 +00:00
|
|
|
Exemplar(l *exemplar.Exemplar) bool
|
|
|
|
|
2023-12-11 08:43:42 +00:00
|
|
|
// CreatedTimestamp returns the created timestamp (in milliseconds) for the
|
|
|
|
// current sample. It returns nil if it is unknown e.g. if it wasn't set,
|
|
|
|
// if the scrape protocol or metric type does not support created timestamps.
|
2024-10-07 11:17:44 +00:00
|
|
|
// Assume the CreatedTimestamp returned pointer is only valid until
|
|
|
|
// the Next iteration.
|
2023-12-11 08:43:42 +00:00
|
|
|
CreatedTimestamp() *int64
|
2023-10-18 18:04:02 +00:00
|
|
|
|
2024-02-29 15:40:53 +00:00
|
|
|
// Next advances the parser to the next sample.
|
|
|
|
// It returns (EntryInvalid, io.EOF) if no samples were read.
|
2018-10-04 13:52:03 +00:00
|
|
|
Next() (Entry, error)
|
|
|
|
}
|
|
|
|
|
2024-10-18 15:12:31 +00:00
|
|
|
// extractMediaType returns the mediaType of a required parser. It tries first to
|
|
|
|
// extract a valid and supported mediaType from contentType. If that fails,
|
|
|
|
// the provided fallbackType (possibly an empty string) is returned, together with
|
|
|
|
// an error. fallbackType is used as-is without further validation.
|
|
|
|
func extractMediaType(contentType, fallbackType string) (string, error) {
|
2022-02-08 09:57:56 +00:00
|
|
|
if contentType == "" {
|
2024-10-18 15:12:31 +00:00
|
|
|
if fallbackType == "" {
|
|
|
|
return "", errors.New("non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target")
|
|
|
|
}
|
|
|
|
return fallbackType, fmt.Errorf("non-compliant scrape target sending blank Content-Type, using fallback_scrape_protocol %q", fallbackType)
|
2022-02-08 09:57:56 +00:00
|
|
|
}
|
|
|
|
|
2024-10-18 15:12:31 +00:00
|
|
|
// We have a contentType, parse it.
|
2018-10-05 16:11:16 +00:00
|
|
|
mediaType, _, err := mime.ParseMediaType(contentType)
|
2021-06-29 21:45:23 +00:00
|
|
|
if err != nil {
|
2024-10-18 15:12:31 +00:00
|
|
|
if fallbackType == "" {
|
|
|
|
retErr := fmt.Errorf("cannot parse Content-Type %q and no fallback_scrape_protocol for target", contentType)
|
|
|
|
return "", errors.Join(retErr, err)
|
|
|
|
}
|
|
|
|
retErr := fmt.Errorf("could not parse received Content-Type %q, using fallback_scrape_protocol %q", contentType, fallbackType)
|
|
|
|
return fallbackType, errors.Join(retErr, err)
|
|
|
|
}
|
|
|
|
|
|
|
|
// We have a valid media type, either we recognise it and can use it
|
|
|
|
// or we have to error.
|
|
|
|
switch mediaType {
|
|
|
|
case "application/openmetrics-text", "application/vnd.google.protobuf", "text/plain":
|
|
|
|
return mediaType, nil
|
|
|
|
}
|
|
|
|
// We're here because we have no recognised mediaType.
|
|
|
|
if fallbackType == "" {
|
|
|
|
return "", fmt.Errorf("received unsupported Content-Type %q and no fallback_scrape_protocol specified for target", contentType)
|
2021-06-29 21:45:23 +00:00
|
|
|
}
|
2024-10-18 15:12:31 +00:00
|
|
|
return fallbackType, fmt.Errorf("received unsupported Content-Type %q, using fallback_scrape_protocol %q", contentType, fallbackType)
|
|
|
|
}
|
|
|
|
|
|
|
|
// New returns a new parser of the byte slice.
|
|
|
|
//
|
|
|
|
// This function no longer guarantees to return a valid parser.
|
|
|
|
//
|
|
|
|
// It only returns a valid parser if the supplied contentType and fallbackType allow.
|
|
|
|
// An error may also be returned if fallbackType had to be used or there was some
|
|
|
|
// other error parsing the supplied Content-Type.
|
|
|
|
// If the returned parser is nil then the scrape must fail.
|
|
|
|
func New(b []byte, contentType, fallbackType string, parseClassicHistograms, skipOMCTSeries bool, st *labels.SymbolTable) (Parser, error) {
|
|
|
|
mediaType, err := extractMediaType(contentType, fallbackType)
|
|
|
|
// err may be nil or something we want to warn about.
|
|
|
|
|
2021-06-29 21:45:23 +00:00
|
|
|
switch mediaType {
|
|
|
|
case "application/openmetrics-text":
|
2024-10-02 10:52:03 +00:00
|
|
|
return NewOpenMetricsParser(b, st, func(o *openMetricsParserOptions) {
|
|
|
|
o.SkipCTSeries = skipOMCTSeries
|
2024-10-18 15:12:31 +00:00
|
|
|
}), err
|
2021-06-29 21:45:23 +00:00
|
|
|
case "application/vnd.google.protobuf":
|
2024-10-18 15:12:31 +00:00
|
|
|
return NewProtobufParser(b, parseClassicHistograms, st), err
|
|
|
|
case "text/plain":
|
|
|
|
return NewPromParser(b, st), err
|
2021-06-29 21:45:23 +00:00
|
|
|
default:
|
2024-10-18 15:12:31 +00:00
|
|
|
return nil, err
|
2018-10-05 16:11:16 +00:00
|
|
|
}
|
2018-10-04 13:52:03 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// Entry represents the type of a parsed entry.
|
|
|
|
type Entry int
|
|
|
|
|
|
|
|
const (
|
2021-06-29 21:45:23 +00:00
|
|
|
EntryInvalid Entry = -1
|
|
|
|
EntryType Entry = 0
|
|
|
|
EntryHelp Entry = 1
|
lint: Revamp our linting rules, mostly around doc comments
Several things done here:
- Set `max-issues-per-linter` to 0 so that we actually see all linter
warnings and not just 50 per linter. (As we also set
`max-same-issues` to 0, I assume this was the intention from the
beginning.)
- Stop using the golangci-lint default excludes (by setting
`exclude-use-default: false`. Those are too generous and don't match
our style conventions. (I have re-added some of the excludes
explicitly in this commit. See below.)
- Re-add the `errcheck` exclusion we have used so far via the
defaults.
- Exclude the signature requirement `govet` has for `Seek` methods
because we use non-standard `Seek` methods a lot. (But we keep other
requirements, while the default excludes completely disabled the
check for common method segnatures.)
- Exclude warnings about missing doc comments on exported symbols. (We
used to be pretty adamant about doc comments, but stopped that at
some point in the past. By now, we have about 500 missing doc
comments. We may consider reintroducing this check, but that's
outside of the scope of this commit. The default excludes of
golangci-lint essentially ignore doc comments completely.)
- By stop using the default excludes, we now get warnings back on
malformed doc comments. That's the most impactful change in this
commit. It does not enforce doc comments (again), but _if_ there is
a doc comment, it has to have the recommended form. (Most of the
changes in this commit are fixing this form.)
- Improve wording/spelling of some comments in .golangci.yml, and
remove an outdated comment.
- Leave `package-comments` inactive, but add a TODO asking if we
should change that.
- Add a new sub-linter `comment-spacings` (and fix corresponding
comments), which avoids missing spaces after the leading `//`.
Signed-off-by: beorn7 <beorn@grafana.com>
2024-08-22 11:59:36 +00:00
|
|
|
EntrySeries Entry = 2 // EntrySeries marks a series with a simple float64 as value.
|
2021-06-29 21:45:23 +00:00
|
|
|
EntryComment Entry = 3
|
|
|
|
EntryUnit Entry = 4
|
lint: Revamp our linting rules, mostly around doc comments
Several things done here:
- Set `max-issues-per-linter` to 0 so that we actually see all linter
warnings and not just 50 per linter. (As we also set
`max-same-issues` to 0, I assume this was the intention from the
beginning.)
- Stop using the golangci-lint default excludes (by setting
`exclude-use-default: false`. Those are too generous and don't match
our style conventions. (I have re-added some of the excludes
explicitly in this commit. See below.)
- Re-add the `errcheck` exclusion we have used so far via the
defaults.
- Exclude the signature requirement `govet` has for `Seek` methods
because we use non-standard `Seek` methods a lot. (But we keep other
requirements, while the default excludes completely disabled the
check for common method segnatures.)
- Exclude warnings about missing doc comments on exported symbols. (We
used to be pretty adamant about doc comments, but stopped that at
some point in the past. By now, we have about 500 missing doc
comments. We may consider reintroducing this check, but that's
outside of the scope of this commit. The default excludes of
golangci-lint essentially ignore doc comments completely.)
- By stop using the default excludes, we now get warnings back on
malformed doc comments. That's the most impactful change in this
commit. It does not enforce doc comments (again), but _if_ there is
a doc comment, it has to have the recommended form. (Most of the
changes in this commit are fixing this form.)
- Improve wording/spelling of some comments in .golangci.yml, and
remove an outdated comment.
- Leave `package-comments` inactive, but add a TODO asking if we
should change that.
- Add a new sub-linter `comment-spacings` (and fix corresponding
comments), which avoids missing spaces after the leading `//`.
Signed-off-by: beorn7 <beorn@grafana.com>
2024-08-22 11:59:36 +00:00
|
|
|
EntryHistogram Entry = 5 // EntryHistogram marks a series with a native histogram as a value.
|
2018-10-04 13:52:03 +00:00
|
|
|
)
|