k3s/docs/proposals/runtime-pod-cache.md

<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->

<!-- BEGIN STRIP_FOR_RELEASE -->

<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
     width="25" height="25">
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
     width="25" height="25">
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
     width="25" height="25">
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
     width="25" height="25">
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
     width="25" height="25">

<h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2>

If you are using a released version of Kubernetes, you should
refer to the docs that go with that version.

Documentation for other releases can be found at
[releases.k8s.io](http://releases.k8s.io).
</strong>
--

<!-- END STRIP_FOR_RELEASE -->

<!-- END MUNGE: UNVERSIONED_WARNING -->

# Kubelet: Runtime Pod Cache

This proposal builds on top of the Pod Lifecycle Event Generator (PLEG) proposed
in [#12802](https://issues.k8s.io/12802). It assumes that Kubelet subscribes to
the pod lifecycle event stream to eliminate periodic polling of pod
states. Please see [#12802](https://issues.k8s.io/12802). for the motivation and
design concept for PLEG.

Runtime pod cache is an in-memory cache which stores the *status* of
all pods, and is maintained by PLEG. It serves as a single source of
truth for internal pod status, freeing Kubelet from querying the
container runtime.

## Motivation

With PLEG, Kubelet no longer needs to perform comprehensive state
checking for all pods periodically. It only instructs a pod worker to
start syncing when there is a change of its pod status. Nevertheless,
during each sync, a pod worker still needs to construct the pod status
by examining all containers (whether dead or alive) in the pod, due to
the lack of the caching of previous states. With the integration of
pod cache, we can further improve Kubelet's CPU usage by

 1. Lowering the number of concurrent requests to the container
    runtime since pod workers no longer have to query the runtime
    individually.
 2. Lowering the total number of inspect requests because there is no
    need to inspect containers with no state changes.

***Don't we already have a [container runtime cache]
(https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/container/runtime_cache.go)?***

The runtime cache is an optimization that reduces the number of `GetPods()`
calls from the workers. However,

 * The cache does not store all information necessary for a worker to
   complete a sync (e.g., `docker inspect`); workers still need to inspect
   containers individually to generate `api.PodStatus`.
 * Workers sometimes need to bypass the cache in order to retrieve the
   latest pod state.

This proposal generalizes the cache and instructs PLEG to populate the cache, so
that the content is always up-to-date.

**Why can't each worker cache its own pod status?**

The short answer is yes, they can. The longer answer is that localized
caching limits the use of the cache content -- other components cannot
access it. This often leads to caching at multiple places and/or passing
objects around, complicating the control flow.

## Runtime Pod Cache

![pod cache](pod-cache.png)

Pod cache stores the `PodStatus` for all pods on the node. `PodStatus` encompasses
all the information required from the container runtime to generate
`api.PodStatus` for a pod.

```go
// PodStatus represents the status of the pod and its containers.
// api.PodStatus can be derived from examining PodStatus and api.Pod.
type PodStatus struct {
    ID types.UID
    Name string
    Namespace string
    IP string
    ContainerStatuses []*ContainerStatus
}

// ContainerStatus represents the status of a container.
type ContainerStatus struct {
    ID ContainerID
    Name string
    State ContainerState
    CreatedAt time.Time
    StartedAt time.Time
    FinishedAt time.Time
    ExitCode int
    Image string
    ImageID string
    Hash uint64
    RestartCount int
    Reason string
    Message string
}
```

`PodStatus` is defined in the container runtime interface, hence is
runtime-agnostic.

PLEG is responsible for updating the entries pod cache, hence always keeping
the cache up-to-date.

1. Detect change of container state
2. Inspect the pod for details
3. Update the pod cache with the new PodStatus
  - If there is no real change of the pod entry, do nothing
  - Otherwise, generate and send out the corresponding pod lifecycle event

Note that in (3), PLEG can check if there is any disparity between the old
and the new pod entry to filter out duplicated events if needed.

### Evict cache entries

Note that the cache represents all the pods/containers known by the container
runtime. A cache entry should only be evicted if the pod is no longer visible
by the container runtime. PLEG is responsible for deleting entries in the
cache.

### Generate `api.PodStatus`

Because pod cache stores the up-to-date `PodStatus` of the pods, Kubelet can
generate the `api.PodStatus` by interpreting the cache entry at any
time. To avoid sending intermediate status (e.g., while a pod worker
is restarting a container), we will instruct the pod worker to generate a new
status at the beginning of each sync.

### Cache contention

Cache contention should not be a problem when the number of pods is
small. When Kubelet scales, we can always shard the pods by ID to
reduce contention.

### Disk management

The pod cache is not capable to fulfill the needs of container/image garbage
collectors as they may demand more than pod-level information. These components
will still need to query the container runtime directly at times. We may
consider extending the cache for these use cases, but they are beyond the scope
of this proposal.


## Impact on Pod Worker Control Flow

A pod worker may perform various operations (e.g., start/kill a container)
during a sync. They will expect to see the results of such operations reflected
in the cache in the next sync. Alternately, they can bypass the cache and
query the container runtime directly to get the latest status. However, this
is not desirable since the cache is introduced exactly to eliminate unnecessary,
concurrent queries. Therefore, a pod worker should be blocked until all expected
results have been updated to the cache by PLEG.

Depending on the type of PLEG (see [#12802](https://issues.k8s.io/12802)) in
use, the methods to check whether a requirement is met can differ. For a
PLEG that solely relies on relisting, a pod worker can simply wait until the
relist timestamp is newer than the end of the worker's last sync. On the other
hand, if pod worker knows what events to expect, they can also block until the
events are observed.

It should be noted that `api.PodStatus` will only be generated by the pod
worker *after* the cache has been updated. This means that the perceived
responsiveness of Kubelet (from querying the API server) will be affected by
how soon the cache can be populated. For the pure-relisting PLEG, the relist
period can become the bottleneck. On the other hand, A PLEG which watches the
upstream event stream (and knows how what events to expect) is not restricted
by such periods and should improve Kubelet's perceived responsiveness.

## TODOs for v1.2

 - Redefine container runtime types ([#12619](https://issues.k8s.io/12619)):
   and introduce `PodStatus`. Refactor dockertools and rkt to use the new type.

 - Add cache and instruct PLEG to populate it.

 - Refactor Kubelet to use the cache.

 - Deprecate the old runtime cache.


<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/runtime-pod-cache.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->