# vSphere Volume - [Prerequisites](#prerequisites) - [Examples](#examples) - [Volumes](#volumes) - [Persistent Volumes](#persistent-volumes) - [Storage Class](#storage-class) - [Virtual SAN policy support inside Kubernetes](#virtual-san-policy-support-inside-kubernetes) - [Stateful Set](#stateful-set) ## Prerequisites - Kubernetes with vSphere Cloud Provider configured. For cloudprovider configuration please refer [vSphere getting started guide](http://kubernetes.io/docs/getting-started-guides/vsphere/). ## Examples ### Volumes 1. Create VMDK. First ssh into ESX and then use following command to create vmdk, ```shell vmkfstools -c 2G /vmfs/volumes/datastore1/volumes/myDisk.vmdk ``` 2. Create Pod which uses 'myDisk.vmdk'. See example ```yaml apiVersion: v1 kind: Pod metadata: name: test-vmdk spec: containers: - image: gcr.io/google_containers/test-webserver name: test-container volumeMounts: - mountPath: /test-vmdk name: test-volume volumes: - name: test-volume # This VMDK volume must already exist. vsphereVolume: volumePath: "[datastore1] volumes/myDisk" fsType: ext4 ``` [Download example](vsphere-volume-pod.yaml?raw=true) Creating the pod: ``` bash $ kubectl create -f examples/volumes/vsphere/vsphere-volume-pod.yaml ``` Verify that pod is running: ```bash $ kubectl get pods test-vmdk NAME READY STATUS RESTARTS AGE test-vmdk 1/1 Running 0 48m ``` ### Persistent Volumes 1. Create VMDK. First ssh into ESX and then use following command to create vmdk, ```shell vmkfstools -c 2G /vmfs/volumes/datastore1/volumes/myDisk.vmdk ``` 2. Create Persistent Volume. See example: ```yaml apiVersion: v1 kind: PersistentVolume metadata: name: pv0001 spec: capacity: storage: 2Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain vsphereVolume: volumePath: "[datastore1] volumes/myDisk" fsType: ext4 ``` In the above example datastore1 is located in the root folder. If datastore is member of Datastore Cluster or located in sub folder, the folder path needs to be provided in the VolumePath as below. ```yaml vsphereVolume: VolumePath: "[DatastoreCluster/datastore1] volumes/myDisk" ``` [Download example](vsphere-volume-pv.yaml?raw=true) Creating the persistent volume: ``` bash $ kubectl create -f examples/volumes/vsphere/vsphere-volume-pv.yaml ``` Verifying persistent volume is created: ``` bash $ kubectl describe pv pv0001 Name: pv0001 Labels: Status: Available Claim: Reclaim Policy: Retain Access Modes: RWO Capacity: 2Gi Message: Source: Type: vSphereVolume (a Persistent Disk resource in vSphere) VolumePath: [datastore1] volumes/myDisk FSType: ext4 No events. ``` 3. Create Persistent Volume Claim. See example: ```yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: pvc0001 spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi ``` [Download example](vsphere-volume-pvc.yaml?raw=true) Creating the persistent volume claim: ``` bash $ kubectl create -f examples/volumes/vsphere/vsphere-volume-pvc.yaml ``` Verifying persistent volume claim is created: ``` bash $ kubectl describe pvc pvc0001 Name: pvc0001 Namespace: default Status: Bound Volume: pv0001 Labels: Capacity: 2Gi Access Modes: RWO No events. ``` 3. Create Pod which uses Persistent Volume Claim. See example: ```yaml apiVersion: v1 kind: Pod metadata: name: pvpod spec: containers: - name: test-container image: gcr.io/google_containers/test-webserver volumeMounts: - name: test-volume mountPath: /test-vmdk volumes: - name: test-volume persistentVolumeClaim: claimName: pvc0001 ``` [Download example](vsphere-volume-pvcpod.yaml?raw=true) Creating the pod: ``` bash $ kubectl create -f examples/volumes/vsphere/vsphere-volume-pvcpod.yaml ``` Verifying pod is created: ``` bash $ kubectl get pod pvpod NAME READY STATUS RESTARTS AGE pvpod 1/1 Running 0 48m ``` ### Storage Class __Note: Here you don't need to create vmdk it is created for you.__ 1. Create Storage Class. Example 1: ```yaml kind: StorageClass apiVersion: storage.k8s.io/v1beta1 metadata: name: fast provisioner: kubernetes.io/vsphere-volume parameters: diskformat: zeroedthick fstype: ext3 ``` [Download example](vsphere-volume-sc-fast.yaml?raw=true) You can also specify the datastore in the Storageclass as shown in example 2. The volume will be created on the datastore specified in the storage class. This field is optional. If not specified as shown in example 1, the volume will be created on the datastore specified in the vsphere config file used to initialize the vSphere Cloud Provider. Example 2: ```yaml kind: StorageClass apiVersion: storage.k8s.io/v1beta1 metadata: name: fast provisioner: kubernetes.io/vsphere-volume parameters: diskformat: zeroedthick datastore: VSANDatastore ``` If datastore is member of DataStore Cluster or within some sub folder, the datastore folder path needs to be provided in the datastore parameter as below. ```yaml parameters: datastore: DatastoreCluster/VSANDatastore ``` [Download example](vsphere-volume-sc-with-datastore.yaml?raw=true) Creating the storageclass: ``` bash $ kubectl create -f examples/volumes/vsphere/vsphere-volume-sc-fast.yaml ``` Verifying storage class is created: ``` bash $ kubectl describe storageclass fast Name: fast IsDefaultClass: No Annotations: Provisioner: kubernetes.io/vsphere-volume Parameters: diskformat=zeroedthick,fstype=ext3 No events. ``` 2. Create Persistent Volume Claim. See example: ```yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: pvcsc001 annotations: volume.beta.kubernetes.io/storage-class: fast spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi ``` [Download example](vsphere-volume-pvcsc.yaml?raw=true) Creating the persistent volume claim: ``` bash $ kubectl create -f examples/volumes/vsphere/vsphere-volume-pvcsc.yaml ``` Verifying persistent volume claim is created: ``` bash $ kubectl describe pvc pvcsc001 Name: pvcsc001 Namespace: default StorageClass: fast Status: Bound Volume: pvc-83295256-f8e0-11e6-8263-005056b2349c Labels: Capacity: 2Gi Access Modes: RWO Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1m 1m 1 persistentvolume-controller Normal ProvisioningSucceeded Successfully provisioned volume pvc-83295256-f8e0-11e6-8263-005056b2349c using kubernetes.io/vsphere-volume ``` Persistent Volume is automatically created and is bounded to this pvc. Verifying persistent volume claim is created: ``` bash $ kubectl describe pv pvc-83295256-f8e0-11e6-8263-005056b2349c Name: pvc-83295256-f8e0-11e6-8263-005056b2349c Labels: StorageClass: fast Status: Bound Claim: default/pvcsc001 Reclaim Policy: Delete Access Modes: RWO Capacity: 2Gi Message: Source: Type: vSphereVolume (a Persistent Disk resource in vSphere) VolumePath: [datastore1] kubevols/kubernetes-dynamic-pvc-83295256-f8e0-11e6-8263-005056b2349c.vmdk FSType: ext3 No events. ``` __Note: VMDK is created inside ```kubevols``` folder in datastore which is mentioned in 'vsphere' cloudprovider configuration. The cloudprovider config is created during setup of Kubernetes cluster on vSphere.__ 3. Create Pod which uses Persistent Volume Claim with storage class. See example: ```yaml apiVersion: v1 kind: Pod metadata: name: pvpod spec: containers: - name: test-container image: gcr.io/google_containers/test-webserver volumeMounts: - name: test-volume mountPath: /test-vmdk volumes: - name: test-volume persistentVolumeClaim: claimName: pvcsc001 ``` [Download example](vsphere-volume-pvcscpod.yaml?raw=true) Creating the pod: ``` bash $ kubectl create -f examples/volumes/vsphere/vsphere-volume-pvcscpod.yaml ``` Verifying pod is created: ``` bash $ kubectl get pod pvpod NAME READY STATUS RESTARTS AGE pvpod 1/1 Running 0 48m ``` ### Virtual SAN policy support inside Kubernetes Vsphere Infrastructure(VI) Admins will have the ability to specify custom Virtual SAN Storage Capabilities during dynamic volume provisioning. You can now define storage requirements, such as performance and availability, in the form of storage capabilities during dynamic volume provisioning. The storage capability requirements are converted into a Virtual SAN policy which are then pushed down to the Virtual SAN layer when a persistent volume (virtual disk) is being created. The virtual disk is distributed across the Virtual SAN datastore to meet the requirements. The official [VSAN policy documentation](https://pubs.vmware.com/vsphere-65/index.jsp?topic=%2Fcom.vmware.vsphere.virtualsan.doc%2FGUID-08911FD3-2462-4C1C-AE81-0D4DBC8F7990.html) describes in detail about each of the individual storage capabilities that are supported by VSAN. The user can specify these storage capabilities as part of storage class defintion based on his application needs. The policy settings can be one or more of the following: * *hostFailuresToTolerate*: represents NumberOfFailuresToTolerate * *diskStripes*: represents NumberofDiskStripesPerObject * *objectSpaceReservation*: represents ObjectSpaceReservation * *cacheReservation*: represents FlashReadCacheReservation * *iopsLimit*: represents IOPSLimitForObject * *forceProvisioning*: represents if volume must be Force Provisioned __Note: Here you don't need to create persistent volume it is created for you.__ 1. Create Storage Class. Example 1: ```yaml kind: StorageClass apiVersion: storage.k8s.io/v1beta1 metadata: name: fast provisioner: kubernetes.io/vsphere-volume parameters: diskformat: zeroedthick hostFailuresToTolerate: "2" cachereservation: "20" ``` [Download example](vsphere-volume-sc-vsancapabilities.yaml?raw=true) Here a persistent volume will be created with the Virtual SAN capabilities - hostFailuresToTolerate to 2 and cachereservation is 20% read cache reserved for storage object. Also the persistent volume will be *zeroedthick* disk. The official [VSAN policy documentation](https://pubs.vmware.com/vsphere-65/index.jsp?topic=%2Fcom.vmware.vsphere.virtualsan.doc%2FGUID-08911FD3-2462-4C1C-AE81-0D4DBC8F7990.html) describes in detail about each of the individual storage capabilities that are supported by VSAN and can be configured on the virtual disk. You can also specify the datastore in the Storageclass as shown in example 2. The volume will be created on the datastore specified in the storage class. This field is optional. If not specified as shown in example 1, the volume will be created on the datastore specified in the vsphere config file used to initialize the vSphere Cloud Provider. Example 2: ```yaml kind: StorageClass apiVersion: storage.k8s.io/v1beta1 metadata: name: fast provisioner: kubernetes.io/vsphere-volume parameters: diskformat: zeroedthick datastore: VSANDatastore hostFailuresToTolerate: "2" cachereservation: "20" ``` [Download example](vsphere-volume-sc-vsancapabilities-with-datastore.yaml?raw=true) __Note: If you do not apply a storage policy during dynamic provisioning on a VSAN datastore, it will use a default Virtual SAN policy.__ Creating the storageclass: ``` bash $ kubectl create -f examples/volumes/vsphere/vsphere-volume-sc-vsancapabilities.yaml ``` Verifying storage class is created: ``` bash $ kubectl describe storageclass fast Name: fast Annotations: Provisioner: kubernetes.io/vsphere-volume Parameters: diskformat=zeroedthick, hostFailuresToTolerate="2", cachereservation="20" No events. ``` 2. Create Persistent Volume Claim. See example: ```yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: pvcsc-vsan annotations: volume.beta.kubernetes.io/storage-class: fast spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi ``` [Download example](vsphere-volume-pvcsc.yaml?raw=true) Creating the persistent volume claim: ``` bash $ kubectl create -f examples/volumes/vsphere/vsphere-volume-pvcsc.yaml ``` Verifying persistent volume claim is created: ``` bash $ kubectl describe pvc pvcsc-vsan Name: pvcsc-vsan Namespace: default Status: Bound Volume: pvc-80f7b5c1-94b6-11e6-a24f-005056a79d2d Labels: Capacity: 2Gi Access Modes: RWO No events. ``` Persistent Volume is automatically created and is bounded to this pvc. Verifying persistent volume claim is created: ``` bash $ kubectl describe pv pvc-80f7b5c1-94b6-11e6-a24f-005056a79d2d Name: pvc-80f7b5c1-94b6-11e6-a24f-005056a79d2d Labels: Status: Bound Claim: default/pvcsc-vsan Reclaim Policy: Delete Access Modes: RWO Capacity: 2Gi Message: Source: Type: vSphereVolume (a Persistent Disk resource in vSphere) VolumePath: [VSANDatastore] kubevols/kubernetes-dynamic-pvc-80f7b5c1-94b6-11e6-a24f-005056a79d2d.vmdk FSType: ext4 No events. ``` __Note: VMDK is created inside ```kubevols``` folder in datastore which is mentioned in 'vsphere' cloudprovider configuration. The cloudprovider config is created during setup of Kubernetes cluster on vSphere.__ 3. Create Pod which uses Persistent Volume Claim with storage class. See example: ```yaml apiVersion: v1 kind: Pod metadata: name: pvpod spec: containers: - name: test-container image: gcr.io/google_containers/test-webserver volumeMounts: - name: test-volume mountPath: /test volumes: - name: test-volume persistentVolumeClaim: claimName: pvcsc-vsan ``` [Download example](vsphere-volume-pvcscpod.yaml?raw=true) Creating the pod: ``` bash $ kubectl create -f examples/volumes/vsphere/vsphere-volume-pvcscpod.yaml ``` Verifying pod is created: ``` bash $ kubectl get pod pvpod NAME READY STATUS RESTARTS AGE pvpod 1/1 Running 0 48m ``` ### Stateful Set vSphere volumes can be consumed by Stateful Sets. 1. Create a storage class that will be used by the ```volumeClaimTemplates``` of a Stateful Set. See example: ```yaml kind: StorageClass apiVersion: storage.k8s.io/v1beta1 metadata: name: thin-disk provisioner: kubernetes.io/vsphere-volume parameters: diskformat: thin ``` [Download example](simple-storageclass.yaml) 2. Create a Stateful set that consumes storage from the Storage Class created. See example: ```yaml --- apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx spec: ports: - port: 80 name: web clusterIP: None selector: app: nginx --- apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: web spec: serviceName: "nginx" replicas: 14 template: metadata: labels: app: nginx spec: containers: - name: nginx image: gcr.io/google_containers/nginx-slim:0.8 ports: - containerPort: 80 name: web volumeMounts: - name: www mountPath: /usr/share/nginx/html volumeClaimTemplates: - metadata: name: www annotations: volume.beta.kubernetes.io/storage-class: thin-disk spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gi ``` This will create Persistent Volume Claims for each replica and provision a volume for each claim if an existing volume could be bound to the claim. [Download example](simple-statefulset.yaml) [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/examples/volumes/vsphere/README.md?pixel)]()