mirror of https://github.com/k3s-io/k3s
270 lines
12 KiB
Markdown
270 lines
12 KiB
Markdown
|
<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->
|
|||
|
|
|
|||
|
|
<!-- BEGIN STRIP_FOR_RELEASE -->
|
|||
|
|
|
|||
|
|
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
|||
|
|
width="25" height="25">
|
|||
|
|
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
|||
|
|
width="25" height="25">
|
|||
|
|
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
|||
|
|
width="25" height="25">
|
|||
|
|
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
|||
|
|
width="25" height="25">
|
|||
|
|
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
|||
|
|
width="25" height="25">
|
|||
|
|
|
|||
|
|
<h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2>
|
|||
|
|
|
|||
|
|
If you are using a released version of Kubernetes, you should
|
|||
|
|
refer to the docs that go with that version.
|
|||
|
|
|
|||
|
|
Documentation for other releases can be found at
|
|||
|
|
[releases.k8s.io](http://releases.k8s.io).
|
|||
|
|
</strong>
|
|||
|
|
--
|
|||
|
|
|
|||
|
|
<!-- END STRIP_FOR_RELEASE -->
|
|||
|
|
|
|||
|
|
<!-- END MUNGE: UNVERSIONED_WARNING -->
|
|||
|
|
|
|||
|
|
# CRI: Log management for container stdout/stderr streams
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Goals and non-goals
|
|||
|
|
|
|||
|
|
Container Runtime Interface (CRI) is an ongoing project to allow container
|
|||
|
|
runtimes to integrate with kubernetes via a newly-defined API. The goal of this
|
|||
|
|
proposal is to define how container's *stdout/stderr* log streams should be
|
|||
|
|
handled in CRI.
|
|||
|
|
|
|||
|
|
The explicit non-goal is to define how (non-stdout/stderr) application logs
|
|||
|
|
should be handled. Collecting and managing arbitrary application logs is a
|
|||
|
|
long-standing issue [1] in kubernetes and is worth a proposal of its own. Even
|
|||
|
|
though this proposal does not touch upon these logs, the direction of
|
|||
|
|
this proposal is aligned with one of the most-discussed solutions, logging
|
|||
|
|
volumes [1], for general logging management.
|
|||
|
|
|
|||
|
|
*In this proposal, “logs” refer to the stdout/stderr streams of the
|
|||
|
|
containers, unless specified otherwise.*
|
|||
|
|
|
|||
|
|
Previous CRI logging issues:
|
|||
|
|
- Tracking issue: https://github.com/kubernetes/kubernetes/issues/30709
|
|||
|
|
- Proposal (by @tmrtfs): https://github.com/kubernetes/kubernetes/pull/33111
|
|||
|
|
|
|||
|
|
The scope of this proposal is narrower than the #33111 proposal, and hopefully
|
|||
|
|
this will encourage a more focused discussion.
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Background
|
|||
|
|
|
|||
|
|
Below is a brief overview of logging in kubernetes with docker, which is the
|
|||
|
|
only container runtime with fully functional integration today.
|
|||
|
|
|
|||
|
|
**Log lifecycle and management**
|
|||
|
|
|
|||
|
|
Docker supports various logging drivers (e.g., syslog, journal, and json-file),
|
|||
|
|
and allows users to configure the driver by passing flags to the docker daemon
|
|||
|
|
at startup. Kubernetes defaults to the "json-file" logging driver, in which
|
|||
|
|
docker writes the stdout/stderr streams to a file in the json format as shown
|
|||
|
|
below.
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
{“log”: “The actual log line”, “stream”: “stderr”, “time”: “2016-10-05T00:00:30.082640485Z”}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Docker deletes the log files when the container is removed, and a cron-job (or
|
|||
|
|
systemd timer-based job) on the node is responsible to rotate the logs (using
|
|||
|
|
`logrotate`). To preserve the logs for introspection and debuggability, kubelet
|
|||
|
|
keeps the terminated container until the pod object has been deleted from the
|
|||
|
|
apiserver.
|
|||
|
|
|
|||
|
|
**Container log retrieval**
|
|||
|
|
|
|||
|
|
The kubernetes CLI tool, kubectl, allows users to access the container logs
|
|||
|
|
using [`kubectl logs`]
|
|||
|
|
(http://kubernetes.io/docs/user-guide/kubectl/kubectl_logs/) command.
|
|||
|
|
`kubectl logs` supports flags such as `--since` that requires understanding of
|
|||
|
|
the format and the metadata (i.e., timestamps) of the logs. In the current
|
|||
|
|
implementation, kubelet calls `docker logs` with parameters to return the log
|
|||
|
|
content. As of now, docker only supports `log` operations for the “journal” and
|
|||
|
|
“json-file” drivers [2]. In other words, *the support of `kubectl logs` is not
|
|||
|
|
universal in all kuernetes deployments*.
|
|||
|
|
|
|||
|
|
**Cluster logging support**
|
|||
|
|
|
|||
|
|
In a production cluster, logs are usually collected, aggregated, and shipped to
|
|||
|
|
a remote store where advanced analysis/search/archiving functions are
|
|||
|
|
supported. In kubernetes, the default cluster-addons includes a per-node log
|
|||
|
|
collection daemon, `fluentd`. To facilitate the log collection, kubelet creates
|
|||
|
|
symbolic links to all the docker containers logs under `/var/log/containers`
|
|||
|
|
with pod and container metadata embedded in the filename.
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
/var/log/containers/<pod_name>_<pod_namespace>_<container_name>-<container_id>.log`
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The fluentd daemon watches the `/var/log/containers/` directory and extract the
|
|||
|
|
metadata associated with the log from the path. Note that this integration
|
|||
|
|
requires kubelet to know where the container runtime stores the logs, and will
|
|||
|
|
not be directly applicable to CRI.
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Requirements
|
|||
|
|
|
|||
|
|
1. **Provide ways for CRI-compliant runtimes to support all existing logging
|
|||
|
|
features, i.e., `kubectl logs`.**
|
|||
|
|
|
|||
|
|
2. **Allow kubelet to manage the lifecycle of the logs to pave the way for
|
|||
|
|
better disk management in the future.** This implies that the lifecycle
|
|||
|
|
of containers and their logs need to be decoupled.
|
|||
|
|
|
|||
|
|
3. **Allow log collectors to easily integrate with Kubernetes across
|
|||
|
|
different container runtimes while preserving efficient storage and
|
|||
|
|
retrieval.**
|
|||
|
|
|
|||
|
|
Requirement (1) provides opportunities for runtimes to continue support
|
|||
|
|
`kubectl logs --since` and related features. Note that even though such
|
|||
|
|
features are only supported today for a limited set of log drivers, this is an
|
|||
|
|
important usability tool for a fresh, basic kubernetes cluster, and should not
|
|||
|
|
be overlooked. Requirement (2) stems from the fact that disk is managed by
|
|||
|
|
kubelet as a node-level resource (not per-pod) today, hence it is difficult to
|
|||
|
|
delegate to the runtime by enforcing per-pod disk quota policy. In addition,
|
|||
|
|
container disk quota is not well supported yet, and such limitation may not
|
|||
|
|
even be well-perceived by users. Requirement (1) is crucial to the kubernetes'
|
|||
|
|
extensibility and usability across all deployments.
|
|||
|
|
|
|||
|
|
## Proposed solution
|
|||
|
|
|
|||
|
|
This proposal intends to satisfy the requirements by
|
|||
|
|
|
|||
|
|
1. Enforce where the container logs should be stored on the host
|
|||
|
|
filesystem. Both kubelet and the log collector can interact with
|
|||
|
|
the log files directly.
|
|||
|
|
|
|||
|
|
2. Ask the runtime to decorate the logs in a format that kubelet understands.
|
|||
|
|
|
|||
|
|
**Log directories and structures**
|
|||
|
|
|
|||
|
|
Kubelet will be configured with a root directory (e.g., `/var/log/pods` or
|
|||
|
|
`/var/lib/kubelet/logs/) to store all container logs. Below is an example of a
|
|||
|
|
path to the log of a container in a pod.
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
/var/log/pods/<podUID>/<containerName>_<instance#>.log
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
In CRI, this is implemented by setting the pod-level log directory when
|
|||
|
|
creating the pod sandbox, and passing the relative container log path
|
|||
|
|
when creating a container.
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
PodSandboxConfig.LogDirectory: /var/log/pods/<podUID>/
|
|||
|
|
ContainerConfig.LogPath: <containerName>_<instance#>.log
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Because kubelet determines where the logs are stores and can access them
|
|||
|
|
directly, this meets requirement (1). As for requirement (2), the log collector
|
|||
|
|
can easily extract basic pod metadata (e.g., pod UID, container name) from
|
|||
|
|
the paths, and watch the directly for any changes. In the future, we can
|
|||
|
|
extend this by maintaining a metada file in the pod directory.
|
|||
|
|
|
|||
|
|
**Log format**
|
|||
|
|
|
|||
|
|
The runtime should decorate each log entry with a RFC 3339Nano timestamp
|
|||
|
|
prefix, the stream type (i.e., "stdout" or "stderr"), and ends with a newline.
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
2016-10-06T00:17:09.669794202Z stdout The content of the log entry 1
|
|||
|
|
2016-10-06T00:17:10.113242941Z stderr The content of the log entry 2
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
With the knowledge, kubelet can parses the logs and serve them for `kubectl
|
|||
|
|
logs` requests. This meets requirement (3). Note that the format is defined
|
|||
|
|
deliberately simple to provide only information necessary to serve the requests.
|
|||
|
|
We do not intend for kubelet to host various logging plugins. It is also worth
|
|||
|
|
mentioning again that the scope of this proposal is restricted to stdout/stderr
|
|||
|
|
streams of the container, and we impose no restriction to the logging format of
|
|||
|
|
arbitrary container logs.
|
|||
|
|
|
|||
|
|
**Who should rotate the logs?**
|
|||
|
|
|
|||
|
|
We assume that a separate task (e.g., cron job) will be configured on the node
|
|||
|
|
to rotate the logs periodically, similar to today’s implementation.
|
|||
|
|
|
|||
|
|
We do not rule out the possibility of letting kubelet or a per-node daemon
|
|||
|
|
(`DaemonSet`) to take up the responsibility, or even declare rotation policy
|
|||
|
|
in the kubernetes API as part of the `PodSpec`, but it is beyond the scope of
|
|||
|
|
the this proposal.
|
|||
|
|
|
|||
|
|
**What about non-supported log formats?**
|
|||
|
|
|
|||
|
|
If a runtime chooses to store logs in non-supported formats, it essentially
|
|||
|
|
opts out of `kubectl logs` features, which is backed by kubelet today. It is
|
|||
|
|
assumed that the user can rely on the advanced, cluster logging infrastructure
|
|||
|
|
to examine the logs.
|
|||
|
|
|
|||
|
|
It is also possible that in the future, `kubectl logs` can contact the cluster
|
|||
|
|
logging infrastructure directly to serve logs [1a]. Note that this does not
|
|||
|
|
eliminate the need to store the logs on the node locally for reliability.
|
|||
|
|
|
|||
|
|
|
|||
|
|
**How can existing runtimes (docker/rkt) comply to the logging requirements?**
|
|||
|
|
|
|||
|
|
In the short term, the ongoing docker-CRI integration [3] will support the
|
|||
|
|
proposed solution only partially by (1) creating symbolic links for kubelet
|
|||
|
|
to access, but not manage the logs, and (2) add support for json format in
|
|||
|
|
kubelet. A more sophisticated solution that either involves using a custom
|
|||
|
|
plugin or launching a separate process to copy and decorate the log will be
|
|||
|
|
considered as a mid-term solution.
|
|||
|
|
|
|||
|
|
For rkt, implementation will rely on providing external file-descriptors for
|
|||
|
|
stdout/stderr to applications via systemd [4]. Those streams are currently
|
|||
|
|
managed by a journald sidecar, which collects stream outputs and store them
|
|||
|
|
in the journal file of the pod. This will replaced by a custom sidecar which
|
|||
|
|
can produce logs in the format expected by this specification and can handle
|
|||
|
|
clients attaching as well.
|
|||
|
|
|
|||
|
|
## Alternatives
|
|||
|
|
|
|||
|
|
There are ad-hoc solutions/discussions that addresses one or two of the
|
|||
|
|
requirements, but no comprehensive solution for CRI specifically has been
|
|||
|
|
proposed so far (with the excpetion of @tmrtfs's proposal
|
|||
|
|
[#33111](https://github.com/kubernetes/kubernetes/pull/33111), which has a much
|
|||
|
|
wider scope). It has come up in discussions that kubelet can delegate all the
|
|||
|
|
logging management to the runtime to allow maximum flexibility. However, it is
|
|||
|
|
difficult for this approach to meet either requirement (1) or (2), without
|
|||
|
|
defining complex logging API.
|
|||
|
|
|
|||
|
|
There are also possibilities to implement the current proposal by imposing the
|
|||
|
|
log file paths, while leveraging the runtime to access and/or manage logs. This
|
|||
|
|
requires the runtime to expose knobs in CRI to retrieve, remove, and examine
|
|||
|
|
the disk usage of logs. The upside of this approach is that kubelet needs not
|
|||
|
|
mandate the logging format, assuming runtime already includes plugins for
|
|||
|
|
various logging formats. Unfortunately, this is not true for existing runtimes
|
|||
|
|
such as docker, which supports log retrieval only for a very limited number of
|
|||
|
|
log drivers [2]. On the other hand, the downside is that we would be enforcing
|
|||
|
|
more requirements on the runtime through log storage location on the host, and
|
|||
|
|
a potentially premature logging API that may change as the disk management
|
|||
|
|
evolves.
|
|||
|
|
|
|||
|
|
## References
|
|||
|
|
|
|||
|
|
[1] Log management issues:
|
|||
|
|
- a. https://github.com/kubernetes/kubernetes/issues/17183
|
|||
|
|
- b. https://github.com/kubernetes/kubernetes/issues/24677
|
|||
|
|
- c. https://github.com/kubernetes/kubernetes/pull/13010
|
|||
|
|
|
|||
|
|
[2] Docker logging drivers:
|
|||
|
|
- https://docs.docker.com/engine/admin/logging/overview/
|
|||
|
|
|
|||
|
|
[3] Docker CRI integration:
|
|||
|
|
- https://github.com/kubernetes/kubernetes/issues/31459
|
|||
|
|
|
|||
|
|
[4] rkt support: https://github.com/systemd/systemd/pull/4179
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
|
|||
|
|
[]()
|
|||
|
|
<!-- END MUNGE: GENERATED_ANALYTICS -->
|