mirror of https://github.com/k3s-io/k3s
270 lines
12 KiB
Markdown
270 lines
12 KiB
Markdown
<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->
|
||
|
||
<!-- BEGIN STRIP_FOR_RELEASE -->
|
||
|
||
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
||
width="25" height="25">
|
||
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
||
width="25" height="25">
|
||
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
||
width="25" height="25">
|
||
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
||
width="25" height="25">
|
||
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
||
width="25" height="25">
|
||
|
||
<h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2>
|
||
|
||
If you are using a released version of Kubernetes, you should
|
||
refer to the docs that go with that version.
|
||
|
||
Documentation for other releases can be found at
|
||
[releases.k8s.io](http://releases.k8s.io).
|
||
</strong>
|
||
--
|
||
|
||
<!-- END STRIP_FOR_RELEASE -->
|
||
|
||
<!-- END MUNGE: UNVERSIONED_WARNING -->
|
||
|
||
# CRI: Log management for container stdout/stderr streams
|
||
|
||
|
||
## Goals and non-goals
|
||
|
||
Container Runtime Interface (CRI) is an ongoing project to allow container
|
||
runtimes to integrate with kubernetes via a newly-defined API. The goal of this
|
||
proposal is to define how container's *stdout/stderr* log streams should be
|
||
handled in CRI.
|
||
|
||
The explicit non-goal is to define how (non-stdout/stderr) application logs
|
||
should be handled. Collecting and managing arbitrary application logs is a
|
||
long-standing issue [1] in kubernetes and is worth a proposal of its own. Even
|
||
though this proposal does not touch upon these logs, the direction of
|
||
this proposal is aligned with one of the most-discussed solutions, logging
|
||
volumes [1], for general logging management.
|
||
|
||
*In this proposal, “logs” refer to the stdout/stderr streams of the
|
||
containers, unless specified otherwise.*
|
||
|
||
Previous CRI logging issues:
|
||
- Tracking issue: https://github.com/kubernetes/kubernetes/issues/30709
|
||
- Proposal (by @tmrtfs): https://github.com/kubernetes/kubernetes/pull/33111
|
||
|
||
The scope of this proposal is narrower than the #33111 proposal, and hopefully
|
||
this will encourage a more focused discussion.
|
||
|
||
|
||
## Background
|
||
|
||
Below is a brief overview of logging in kubernetes with docker, which is the
|
||
only container runtime with fully functional integration today.
|
||
|
||
**Log lifecycle and management**
|
||
|
||
Docker supports various logging drivers (e.g., syslog, journal, and json-file),
|
||
and allows users to configure the driver by passing flags to the docker daemon
|
||
at startup. Kubernetes defaults to the "json-file" logging driver, in which
|
||
docker writes the stdout/stderr streams to a file in the json format as shown
|
||
below.
|
||
|
||
```
|
||
{“log”: “The actual log line”, “stream”: “stderr”, “time”: “2016-10-05T00:00:30.082640485Z”}
|
||
```
|
||
|
||
Docker deletes the log files when the container is removed, and a cron-job (or
|
||
systemd timer-based job) on the node is responsible to rotate the logs (using
|
||
`logrotate`). To preserve the logs for introspection and debuggability, kubelet
|
||
keeps the terminated container until the pod object has been deleted from the
|
||
apiserver.
|
||
|
||
**Container log retrieval**
|
||
|
||
The kubernetes CLI tool, kubectl, allows users to access the container logs
|
||
using [`kubectl logs`]
|
||
(http://kubernetes.io/docs/user-guide/kubectl/kubectl_logs/) command.
|
||
`kubectl logs` supports flags such as `--since` that requires understanding of
|
||
the format and the metadata (i.e., timestamps) of the logs. In the current
|
||
implementation, kubelet calls `docker logs` with parameters to return the log
|
||
content. As of now, docker only supports `log` operations for the “journal” and
|
||
“json-file” drivers [2]. In other words, *the support of `kubectl logs` is not
|
||
universal in all kuernetes deployments*.
|
||
|
||
**Cluster logging support**
|
||
|
||
In a production cluster, logs are usually collected, aggregated, and shipped to
|
||
a remote store where advanced analysis/search/archiving functions are
|
||
supported. In kubernetes, the default cluster-addons includes a per-node log
|
||
collection daemon, `fluentd`. To facilitate the log collection, kubelet creates
|
||
symbolic links to all the docker containers logs under `/var/log/containers`
|
||
with pod and container metadata embedded in the filename.
|
||
|
||
```
|
||
/var/log/containers/<pod_name>_<pod_namespace>_<container_name>-<container_id>.log`
|
||
```
|
||
|
||
The fluentd daemon watches the `/var/log/containers/` directory and extract the
|
||
metadata associated with the log from the path. Note that this integration
|
||
requires kubelet to know where the container runtime stores the logs, and will
|
||
not be directly applicable to CRI.
|
||
|
||
|
||
## Requirements
|
||
|
||
1. **Provide ways for CRI-compliant runtimes to support all existing logging
|
||
features, i.e., `kubectl logs`.**
|
||
|
||
2. **Allow kubelet to manage the lifecycle of the logs to pave the way for
|
||
better disk management in the future.** This implies that the lifecycle
|
||
of containers and their logs need to be decoupled.
|
||
|
||
3. **Allow log collectors to easily integrate with Kubernetes across
|
||
different container runtimes while preserving efficient storage and
|
||
retrieval.**
|
||
|
||
Requirement (1) provides opportunities for runtimes to continue support
|
||
`kubectl logs --since` and related features. Note that even though such
|
||
features are only supported today for a limited set of log drivers, this is an
|
||
important usability tool for a fresh, basic kubernetes cluster, and should not
|
||
be overlooked. Requirement (2) stems from the fact that disk is managed by
|
||
kubelet as a node-level resource (not per-pod) today, hence it is difficult to
|
||
delegate to the runtime by enforcing per-pod disk quota policy. In addition,
|
||
container disk quota is not well supported yet, and such limitation may not
|
||
even be well-perceived by users. Requirement (1) is crucial to the kubernetes'
|
||
extensibility and usability across all deployments.
|
||
|
||
## Proposed solution
|
||
|
||
This proposal intends to satisfy the requirements by
|
||
|
||
1. Enforce where the container logs should be stored on the host
|
||
filesystem. Both kubelet and the log collector can interact with
|
||
the log files directly.
|
||
|
||
2. Ask the runtime to decorate the logs in a format that kubelet understands.
|
||
|
||
**Log directories and structures**
|
||
|
||
Kubelet will be configured with a root directory (e.g., `/var/log/pods` or
|
||
`/var/lib/kubelet/logs/) to store all container logs. Below is an example of a
|
||
path to the log of a container in a pod.
|
||
|
||
```
|
||
/var/log/pods/<podUID>/<containerName>_<instance#>.log
|
||
```
|
||
|
||
In CRI, this is implemented by setting the pod-level log directory when
|
||
creating the pod sandbox, and passing the relative container log path
|
||
when creating a container.
|
||
|
||
```
|
||
PodSandboxConfig.LogDirectory: /var/log/pods/<podUID>/
|
||
ContainerConfig.LogPath: <containerName>_<instance#>.log
|
||
```
|
||
|
||
Because kubelet determines where the logs are stores and can access them
|
||
directly, this meets requirement (1). As for requirement (2), the log collector
|
||
can easily extract basic pod metadata (e.g., pod UID, container name) from
|
||
the paths, and watch the directly for any changes. In the future, we can
|
||
extend this by maintaining a metada file in the pod directory.
|
||
|
||
**Log format**
|
||
|
||
The runtime should decorate each log entry with a RFC 3339Nano timestamp
|
||
prefix, the stream type (i.e., "stdout" or "stderr"), and ends with a newline.
|
||
|
||
```
|
||
2016-10-06T00:17:09.669794202Z stdout The content of the log entry 1
|
||
2016-10-06T00:17:10.113242941Z stderr The content of the log entry 2
|
||
```
|
||
|
||
With the knowledge, kubelet can parses the logs and serve them for `kubectl
|
||
logs` requests. This meets requirement (3). Note that the format is defined
|
||
deliberately simple to provide only information necessary to serve the requests.
|
||
We do not intend for kubelet to host various logging plugins. It is also worth
|
||
mentioning again that the scope of this proposal is restricted to stdout/stderr
|
||
streams of the container, and we impose no restriction to the logging format of
|
||
arbitrary container logs.
|
||
|
||
**Who should rotate the logs?**
|
||
|
||
We assume that a separate task (e.g., cron job) will be configured on the node
|
||
to rotate the logs periodically, similar to today’s implementation.
|
||
|
||
We do not rule out the possibility of letting kubelet or a per-node daemon
|
||
(`DaemonSet`) to take up the responsibility, or even declare rotation policy
|
||
in the kubernetes API as part of the `PodSpec`, but it is beyond the scope of
|
||
the this proposal.
|
||
|
||
**What about non-supported log formats?**
|
||
|
||
If a runtime chooses to store logs in non-supported formats, it essentially
|
||
opts out of `kubectl logs` features, which is backed by kubelet today. It is
|
||
assumed that the user can rely on the advanced, cluster logging infrastructure
|
||
to examine the logs.
|
||
|
||
It is also possible that in the future, `kubectl logs` can contact the cluster
|
||
logging infrastructure directly to serve logs [1a]. Note that this does not
|
||
eliminate the need to store the logs on the node locally for reliability.
|
||
|
||
|
||
**How can existing runtimes (docker/rkt) comply to the logging requirements?**
|
||
|
||
In the short term, the ongoing docker-CRI integration [3] will support the
|
||
proposed solution only partially by (1) creating symbolic links for kubelet
|
||
to access, but not manage the logs, and (2) add support for json format in
|
||
kubelet. A more sophisticated solution that either involves using a custom
|
||
plugin or launching a separate process to copy and decorate the log will be
|
||
considered as a mid-term solution.
|
||
|
||
For rkt, implementation will rely on providing external file-descriptors for
|
||
stdout/stderr to applications via systemd [4]. Those streams are currently
|
||
managed by a journald sidecar, which collects stream outputs and store them
|
||
in the journal file of the pod. This will replaced by a custom sidecar which
|
||
can produce logs in the format expected by this specification and can handle
|
||
clients attaching as well.
|
||
|
||
## Alternatives
|
||
|
||
There are ad-hoc solutions/discussions that addresses one or two of the
|
||
requirements, but no comprehensive solution for CRI specifically has been
|
||
proposed so far (with the excpetion of @tmrtfs's proposal
|
||
[#33111](https://github.com/kubernetes/kubernetes/pull/33111), which has a much
|
||
wider scope). It has come up in discussions that kubelet can delegate all the
|
||
logging management to the runtime to allow maximum flexibility. However, it is
|
||
difficult for this approach to meet either requirement (1) or (2), without
|
||
defining complex logging API.
|
||
|
||
There are also possibilities to implement the current proposal by imposing the
|
||
log file paths, while leveraging the runtime to access and/or manage logs. This
|
||
requires the runtime to expose knobs in CRI to retrieve, remove, and examine
|
||
the disk usage of logs. The upside of this approach is that kubelet needs not
|
||
mandate the logging format, assuming runtime already includes plugins for
|
||
various logging formats. Unfortunately, this is not true for existing runtimes
|
||
such as docker, which supports log retrieval only for a very limited number of
|
||
log drivers [2]. On the other hand, the downside is that we would be enforcing
|
||
more requirements on the runtime through log storage location on the host, and
|
||
a potentially premature logging API that may change as the disk management
|
||
evolves.
|
||
|
||
## References
|
||
|
||
[1] Log management issues:
|
||
- a. https://github.com/kubernetes/kubernetes/issues/17183
|
||
- b. https://github.com/kubernetes/kubernetes/issues/24677
|
||
- c. https://github.com/kubernetes/kubernetes/pull/13010
|
||
|
||
[2] Docker logging drivers:
|
||
- https://docs.docker.com/engine/admin/logging/overview/
|
||
|
||
[3] Docker CRI integration:
|
||
- https://github.com/kubernetes/kubernetes/issues/31459
|
||
|
||
[4] rkt support: https://github.com/systemd/systemd/pull/4179
|
||
|
||
|
||
|
||
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
|
||
[]()
|
||
<!-- END MUNGE: GENERATED_ANALYTICS -->
|