mirror of https://github.com/k3s-io/k3s
265 lines
10 KiB
Markdown
265 lines
10 KiB
Markdown
<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->
|
|
|
|
<!-- BEGIN STRIP_FOR_RELEASE -->
|
|
|
|
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
|
width="25" height="25">
|
|
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
|
width="25" height="25">
|
|
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
|
width="25" height="25">
|
|
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
|
width="25" height="25">
|
|
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
|
width="25" height="25">
|
|
|
|
<h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2>
|
|
|
|
If you are using a released version of Kubernetes, you should
|
|
refer to the docs that go with that version.
|
|
|
|
<!-- TAG RELEASE_LINK, added by the munger automatically -->
|
|
<strong>
|
|
The latest release of this document can be found
|
|
[here](http://releases.k8s.io/release-1.4/docs/proposals/volume-provisioning.md).
|
|
|
|
Documentation for other releases can be found at
|
|
[releases.k8s.io](http://releases.k8s.io).
|
|
</strong>
|
|
--
|
|
|
|
<!-- END STRIP_FOR_RELEASE -->
|
|
|
|
<!-- END MUNGE: UNVERSIONED_WARNING -->
|
|
|
|
## Abstract
|
|
|
|
Real Kubernetes clusters have a variety of volumes which differ widely in
|
|
size, iops performance, retention policy, and other characteristics.
|
|
Administrators need a way to dynamically provision volumes of these different
|
|
types to automatically meet user demand.
|
|
|
|
A new mechanism called 'storage classes' is proposed to provide this
|
|
capability.
|
|
|
|
## Motivation
|
|
|
|
In Kubernetes 1.2, an alpha form of limited dynamic provisioning was added
|
|
that allows a single volume type to be provisioned in clouds that offer
|
|
special volume types.
|
|
|
|
In Kubernetes 1.3, a label selector was added to persistent volume claims to
|
|
allow administrators to create a taxonomy of volumes based on the
|
|
characteristics important to them, and to allow users to make claims on those
|
|
volumes based on those characteristics. This allows flexibility when claiming
|
|
existing volumes; the same flexibility is needed when dynamically provisioning
|
|
volumes.
|
|
|
|
After gaining experience with dynamic provisioning after the 1.2 release, we
|
|
want to create a more flexible feature that allows configuration of how
|
|
different storage classes are provisioned and supports provisioning multiple
|
|
types of volumes within a single cloud.
|
|
|
|
### Out-of-tree provisioners
|
|
|
|
One of our goals is to enable administrators to create out-of-tree
|
|
provisioners, that is, provisioners whose code does not live in the Kubernetes
|
|
project. Our experience since the 1.2 release with dynamic provisioning has
|
|
shown that it is impossible to anticipate every aspect and manner of
|
|
provisioning that administrators will want to perform. The proposed design
|
|
should not prevent future work to allow out-of-tree provisioners.
|
|
|
|
## Design
|
|
|
|
This design represents the minimally viable changes required to provision based on storage class configuration. Additional incremental features may be added as a separate effort.
|
|
|
|
We propose that:
|
|
|
|
1. For the base impelementation storage class and volume selectors are mutually exclusive.
|
|
|
|
2. An api object will be incubated in storage.k8s.io/v1beta1 to hold the a `StorageClass`
|
|
API resource. Each StorageClass object contains parameters required by the provisioner to provision volumes of that class. These parameters are opaque to the user.
|
|
|
|
3. `PersistentVolume.Spec.Class` attribute is added to volumes. This attribute
|
|
is optional and specifies which `StorageClass` instance represents
|
|
storage characteristics of a particular PV.
|
|
|
|
During incubation, `Class` is an annotation and not
|
|
actual attribute.
|
|
|
|
4. `PersistentVolume` instances do not require labels by the provisioner.
|
|
|
|
5. `PersistentVolumeClaim.Spec.Class` attribute is added to claims. This
|
|
attribute specifies that only a volume with equal
|
|
`PersistentVolume.Spec.Class` value can satisfy a claim.
|
|
|
|
During incubation, `Class` is just an annotation and not
|
|
actual attribute.
|
|
|
|
6. The existing provisioner plugin implementations be modified to accept
|
|
parameters as specified via `StorageClass`.
|
|
|
|
7. The persistent volume controller modified to invoke provisioners using `StorageClass` configuration and bind claims with `PersistentVolumeClaim.Spec.Class` to volumes with equivalent `PersistentVolume.Spec.Class`
|
|
|
|
8. The existing alpha dynamic provisioning feature be phased out in the
|
|
next release.
|
|
|
|
### Controller workflow for provisioning volumes
|
|
|
|
1. When a new claim is submitted, the controller attempts to find an existing
|
|
volume that will fulfill the claim.
|
|
|
|
1. If the claim has non-empty `claim.Spec.Class`, only PVs with the same
|
|
`pv.Spec.Class` are considered.
|
|
|
|
2. If the claim has empty `claim.Spec.Class`, only PVs with an unset `pv.Spec.Class` are considered.
|
|
|
|
All "considered" volumes are evaluated and the
|
|
smallest matching volume is bound to the claim.
|
|
|
|
2. If no volume is found for the claim and `claim.Spec.Class` is not set or is
|
|
empty string dynamic provisioning is disabled.
|
|
|
|
3. If `claim.Spec.Class` is set the controller tries to find instance of StorageClass with this name. If no
|
|
such StorageClass is found, the controller goes back to step 1. and
|
|
periodically retries finding a matching volume or storage class again until
|
|
a match is found. The claim is `Pending` during this period.
|
|
|
|
4. With StorageClass instance, the controller finds volume plugin specified by
|
|
StorageClass.Provisioner.
|
|
|
|
5. All provisioners are in-tree; they implement an interface called
|
|
`ProvisionableVolumePlugin`, which has a method called `NewProvisioner`
|
|
that returns a new provisioner.
|
|
|
|
6. The controller calls volume plugin `Provision` with Parameters from the `StorageClass` configuration object.
|
|
|
|
7. If `Provision` returns an error, the controller generates an event on the
|
|
claim and goes back to step 1., i.e. it will retry provisioning periodically
|
|
|
|
8. If `Provision` returns no error, the controller creates the returned
|
|
`api.PersistentVolume`, fills its `Class` attribute with `claim.Spec.Class`
|
|
and makes it already bound to the claim
|
|
|
|
1. If the create operation for the `api.PersistentVolume` fails, it is
|
|
retried
|
|
|
|
2. If the create operation does not succeed in reasonable time, the
|
|
controller attempts to delete the provisioned volume and creates an event
|
|
on the claim
|
|
|
|
Existing behavior is un-changed for claims that do not specify `claim.Spec.Class`.
|
|
|
|
### `StorageClass` API
|
|
|
|
A new API group should hold the API for storage classes, following the pattern
|
|
of autoscaling, metrics, etc. To allow for future storage-related APIs, we
|
|
should call this new API group `storage.k8s.io` and incubate in storage.k8s.io/v1beta1.
|
|
|
|
Storage classes will be represented by an API object called `StorageClass`:
|
|
|
|
```go
|
|
package storage
|
|
|
|
// StorageClass describes the parameters for a class of storage for
|
|
// which PersistentVolumes can be dynamically provisioned.
|
|
//
|
|
// StorageClasses are non-namespaced; the name of the storage class
|
|
// according to etcd is in ObjectMeta.Name.
|
|
type StorageClass struct {
|
|
unversioned.TypeMeta `json:",inline"`
|
|
ObjectMeta `json:"metadata,omitempty"`
|
|
|
|
// Provisioner indicates the type of the provisioner.
|
|
Provisioner string `json:"provisioner,omitempty"`
|
|
|
|
// Parameters for dynamic volume provisioner.
|
|
Parameters map[string]string `json:"parameters,omitempty"`
|
|
}
|
|
|
|
```
|
|
|
|
`PersistentVolumeClaimSpec` and `PersistentVolumeSpec` both get Class attribute
|
|
(the existing annotation is used during incubation):
|
|
|
|
```go
|
|
type PersistentVolumeClaimSpec struct {
|
|
// Name of requested storage class. If non-empty, only PVs with this
|
|
// pv.Spec.Class will be considered for binding and if no such PV is
|
|
// available, StorageClass with this name will be used to dynamically
|
|
// provision the volume.
|
|
Class string
|
|
...
|
|
}
|
|
|
|
type PersistentVolumeSpec struct {
|
|
// Name of StorageClass instance that this volume belongs to.
|
|
Class string
|
|
...
|
|
}
|
|
```
|
|
|
|
Storage classes are natural to think of as a global resource, since they:
|
|
|
|
1. Align with PersistentVolumes, which are a global resource
|
|
2. Are administrator controlled
|
|
|
|
### Provisioning configuration
|
|
|
|
With the scheme outlined above the provisioner creates PVs using parameters specified in the `StorageClass` object.
|
|
|
|
### Provisioner interface changes
|
|
|
|
`struct volume.VolumeOptions` (containing parameters for a provisioner plugin)
|
|
will be extended to contain StorageClass.Parameters.
|
|
|
|
The existing provisioner implementations will be modified to accept the StorageClass configuration object.
|
|
|
|
### PV Controller Changes
|
|
|
|
The persistent volume controller will be modified to implement the new
|
|
workflow described in this proposal. The changes will be limited to the
|
|
`provisionClaimOperation` method, which is responsible for invoking the
|
|
provisioner and to favor existing volumes before provisioning a new one.
|
|
|
|
## Examples
|
|
|
|
### AWS provisioners with distinct QoS
|
|
|
|
This example shows two storage classes, "aws-fast" and "aws-slow".
|
|
|
|
```
|
|
apiVersion: v1
|
|
kind: StorageClass
|
|
metadata:
|
|
name: aws-fast
|
|
provisioner: kubernetes.io/aws-ebs
|
|
parameters:
|
|
zone: us-east-1b
|
|
type: ssd
|
|
|
|
|
|
apiVersion: v1
|
|
kind: StorageClass
|
|
metadata:
|
|
name: aws-slow
|
|
provisioner: kubernetes.io/aws-ebs
|
|
parameters:
|
|
zone: us-east-1b
|
|
type: spinning
|
|
```
|
|
|
|
# Additional Implementation Details
|
|
|
|
0. Annotation `volume.alpha.kubernetes.io/storage-class` is used instead of `claim.Spec.Class` and `volume.Spec.Class` during incubation.
|
|
|
|
1. `claim.Spec.Selector` and `claim.Spec.Class` are mutually exclusive. User can either match existing volumes with `Selector` XOR match existing volumes with `Class` and get dynamic provisioning by using `Class`. This simplifies initial PR and also provisioners.
|
|
|
|
# Cloud Providers
|
|
|
|
Since the `volume.alpha.kubernetes.io/storage-class` is in use a `StorageClass` must be defined to support provisioning. No default is assumed as before.
|
|
|
|
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
|
|
[]()
|
|
<!-- END MUNGE: GENERATED_ANALYTICS -->
|