github
/
k3s
mirror of https://github.com/k3s-io/k3s
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
k3s/pkg/kubelet/volumemanager/reconciler/reconciler_test.go

478 lines
15 KiB
Go
Raw Normal View History

Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
/*
Remove "All rights reserved" from all the headers.
2016-06-03 00:25:58 +00:00
Copyright 2016 The Kubernetes Authors.
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package reconciler
import (
Controller wait for attach and exponential backoff Modify attach/detach controller to keep track of volumes to report attached in Node VolumeToAttach status. Modify kubelet volume manager to wait for volume to show up in Node VolumeToAttach status. Implement exponential backoff for errors in volume manager and attach detach controller
2016-06-16 06:48:04 +00:00
"fmt"
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
"testing"
"time"
"github.com/stretchr/testify/assert"
refactor: use metav1.ObjectMeta in other types
2017-01-17 03:38:19 +00:00
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
start the apimachinery repo
2017-01-11 14:09:48 +00:00
"k8s.io/apimachinery/pkg/runtime"
k8stypes "k8s.io/apimachinery/pkg/types"
"k8s.io/apimachinery/pkg/util/sets"
"k8s.io/apimachinery/pkg/util/wait"
move testing/core to client-go
2017-01-25 20:07:10 +00:00
core "k8s.io/client-go/testing"
move client/record
2017-01-30 18:39:54 +00:00
"k8s.io/client-go/tools/record"
cmd/kubelet
2016-11-18 20:50:58 +00:00
"k8s.io/kubernetes/pkg/api/v1"
rename /release_1_5 to /clientset
2016-12-14 01:18:17 +00:00
"k8s.io/kubernetes/pkg/client/clientset_generated/clientset/fake"
This change supports robust kubelet volume cleanup Currently kubelet volume management works on the concept of desired and actual world of states. The volume manager periodically compares the two worlds and perform volume mount/unmount and/or attach/detach operations. When kubelet restarts, the cache of those two worlds are gone. Although desired world can be recovered through apiserver, actual world can not be recovered which may cause some volumes cannot be cleaned up if their information is deleted by apiserver. This change adds the reconstruction of the actual world by reading the pod directories from disk. The reconstructed volume information is added to both desired world and actual world if it cannot be found in either world. The rest logic would be as same as before, desired world populator may clean up the volume entry if it is no longer in apiserver, and then volume manager should invoke unmount to clean it up.
2016-06-23 19:46:21 +00:00
"k8s.io/kubernetes/pkg/kubelet/config"
Reorganize volume controllers and manager
2016-07-02 01:50:25 +00:00
"k8s.io/kubernetes/pkg/kubelet/volumemanager/cache"
Make volume unmount more robust using exclusive mount w/ O_EXCL
2016-07-06 17:42:56 +00:00
"k8s.io/kubernetes/pkg/util/mount"
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
"k8s.io/kubernetes/pkg/volume"
volumetesting "k8s.io/kubernetes/pkg/volume/testing"
"k8s.io/kubernetes/pkg/volume/util/operationexecutor"
"k8s.io/kubernetes/pkg/volume/util/volumehelper"
)
const (
// reconcilerLoopSleepDuration is the amount of time the reconciler loop
// waits between successive executions
Fix volume states out of sync problem after kubelet restarts When kubelet restarts, all the information about the volumes will be gone from actual/desired states. When update node status with mounted volumes, the volume list might be empty although there are still volumes are mounted and in turn causing master to detach those volumes since they are not in the mounted volumes list. This fix is to make sure only update mounted volumes list after reconciler starts sync states process. This sync state process will scan the existing volume directories and reconstruct actual states if they are missing. This PR also fixes the problem during orphaned pods' directories. In case of the pod directory is unmounted but has not yet deleted (e.g., interrupted with kubelet restarts), clean up routine will delete the directory so that the pod directoriy could be cleaned up (it is safe to delete directory since it is no longer mounted) The third issue this PR fixes is that during reconstruct volume in actual state, mounter could not be nil since it is required for creating container.VolumeMap. If it is nil, it might cause nil pointer exception in kubelet. Details are in proposal PR #33203
2016-09-27 23:12:57 +00:00
reconcilerLoopSleepDuration time.Duration = 0 * time.Millisecond
reconcilerSyncStatesSleepPeriod time.Duration = 10 * time.Minute
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
// waitForAttachTimeout is the maximum amount of time a
// operationexecutor.Mount call will wait for a volume to be attached.
Use strongly-typed types.NodeName for a node name We had another bug where we confused the hostname with the NodeName. To avoid this happening again, and to make the code more self-documenting, we use types.NodeName (a typedef alias for string) whenever we are referring to the Node.Name. A tedious but mechanical commit therefore, to change all uses of the node name to use types.NodeName Also clean up some of the (many) places where the NodeName is referred to as a hostname (not true on AWS), or an instanceID (not true on GCE), etc.
2016-07-16 06:10:29 +00:00
waitForAttachTimeout time.Duration = 1 * time.Second
nodeName k8stypes.NodeName = k8stypes.NodeName("mynodename")
kubeletPodsDir string = "fake-dir"
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
)
// Calls Run()
// Verifies there are no calls to attach, detach, mount, unmount, etc.
func Test_Run_Positive_DoNothing(t *testing.T) {
// Arrange
volumePluginMgr, fakePlugin := volumetesting.GetTestVolumePluginMgr(t)
dsw := cache.NewDesiredStateOfWorld(volumePluginMgr)
asw := cache.NewActualStateOfWorld(nodeName, volumePluginMgr)
Controller wait for attach and exponential backoff Modify attach/detach controller to keep track of volumes to report attached in Node VolumeToAttach status. Modify kubelet volume manager to wait for volume to show up in Node VolumeToAttach status. Implement exponential backoff for errors in volume manager and attach detach controller
2016-06-16 06:48:04 +00:00
kubeClient := createTestClient()
Add Events for operation_executor to show status of mounts, failed or successful
2016-06-21 16:13:23 +00:00
fakeRecorder := &record.FakeRecorder{}
Add unit tests for MountVolume() of operation executor
2016-12-21 19:39:47 +00:00
oex := operationexecutor.NewOperationExecutor(operationexecutor.NewOperationGenerator(kubeClient, volumePluginMgr, fakeRecorder, false /* checkNodeCapabilitiesBeforeMount */))
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
reconciler := NewReconciler(
Controller wait for attach and exponential backoff Modify attach/detach controller to keep track of volumes to report attached in Node VolumeToAttach status. Modify kubelet volume manager to wait for volume to show up in Node VolumeToAttach status. Implement exponential backoff for errors in volume manager and attach detach controller
2016-06-16 06:48:04 +00:00
kubeClient,
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
false, /* controllerAttachDetachEnabled */
reconcilerLoopSleepDuration,
Fix volume states out of sync problem after kubelet restarts When kubelet restarts, all the information about the volumes will be gone from actual/desired states. When update node status with mounted volumes, the volume list might be empty although there are still volumes are mounted and in turn causing master to detach those volumes since they are not in the mounted volumes list. This fix is to make sure only update mounted volumes list after reconciler starts sync states process. This sync state process will scan the existing volume directories and reconstruct actual states if they are missing. This PR also fixes the problem during orphaned pods' directories. In case of the pod directory is unmounted but has not yet deleted (e.g., interrupted with kubelet restarts), clean up routine will delete the directory so that the pod directoriy could be cleaned up (it is safe to delete directory since it is no longer mounted) The third issue this PR fixes is that during reconstruct volume in actual state, mounter could not be nil since it is required for creating container.VolumeMap. If it is nil, it might cause nil pointer exception in kubelet. Details are in proposal PR #33203
2016-09-27 23:12:57 +00:00
reconcilerSyncStatesSleepPeriod,
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
waitForAttachTimeout,
nodeName,
dsw,
asw,
Make volume unmount more robust using exclusive mount w/ O_EXCL
2016-07-06 17:42:56 +00:00
oex,
This change supports robust kubelet volume cleanup Currently kubelet volume management works on the concept of desired and actual world of states. The volume manager periodically compares the two worlds and perform volume mount/unmount and/or attach/detach operations. When kubelet restarts, the cache of those two worlds are gone. Although desired world can be recovered through apiserver, actual world can not be recovered which may cause some volumes cannot be cleaned up if their information is deleted by apiserver. This change adds the reconstruction of the actual world by reading the pod directories from disk. The reconstructed volume information is added to both desired world and actual world if it cannot be found in either world. The rest logic would be as same as before, desired world populator may clean up the volume entry if it is no longer in apiserver, and then volume manager should invoke unmount to clean it up.
2016-06-23 19:46:21 +00:00
&mount.FakeMounter{},
volumePluginMgr,
kubeletPodsDir)
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
// Act
This change supports robust kubelet volume cleanup Currently kubelet volume management works on the concept of desired and actual world of states. The volume manager periodically compares the two worlds and perform volume mount/unmount and/or attach/detach operations. When kubelet restarts, the cache of those two worlds are gone. Although desired world can be recovered through apiserver, actual world can not be recovered which may cause some volumes cannot be cleaned up if their information is deleted by apiserver. This change adds the reconstruction of the actual world by reading the pod directories from disk. The reconstructed volume information is added to both desired world and actual world if it cannot be found in either world. The rest logic would be as same as before, desired world populator may clean up the volume entry if it is no longer in apiserver, and then volume manager should invoke unmount to clean it up.
2016-06-23 19:46:21 +00:00
runReconciler(reconciler)
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
// Assert
assert.NoError(t, volumetesting.VerifyZeroAttachCalls(fakePlugin))
assert.NoError(t, volumetesting.VerifyZeroWaitForAttachCallCount(fakePlugin))
assert.NoError(t, volumetesting.VerifyZeroMountDeviceCallCount(fakePlugin))
assert.NoError(t, volumetesting.VerifyZeroSetUpCallCount(fakePlugin))
assert.NoError(t, volumetesting.VerifyZeroTearDownCallCount(fakePlugin))
assert.NoError(t, volumetesting.VerifyZeroDetachCallCount(fakePlugin))
}
// Populates desiredStateOfWorld cache with one volume/pod.
// Calls Run()
// Verifies there is are attach/mount/etc calls and no detach/unmount calls.
func Test_Run_Positive_VolumeAttachAndMount(t *testing.T) {
// Arrange
volumePluginMgr, fakePlugin := volumetesting.GetTestVolumePluginMgr(t)
dsw := cache.NewDesiredStateOfWorld(volumePluginMgr)
asw := cache.NewActualStateOfWorld(nodeName, volumePluginMgr)
Controller wait for attach and exponential backoff Modify attach/detach controller to keep track of volumes to report attached in Node VolumeToAttach status. Modify kubelet volume manager to wait for volume to show up in Node VolumeToAttach status. Implement exponential backoff for errors in volume manager and attach detach controller
2016-06-16 06:48:04 +00:00
kubeClient := createTestClient()
Add Events for operation_executor to show status of mounts, failed or successful
2016-06-21 16:13:23 +00:00
fakeRecorder := &record.FakeRecorder{}
Add unit tests for MountVolume() of operation executor
2016-12-21 19:39:47 +00:00
oex := operationexecutor.NewOperationExecutor(operationexecutor.NewOperationGenerator(kubeClient, volumePluginMgr, fakeRecorder, false /* checkNodeCapabilitiesBeforeMount */))
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
reconciler := NewReconciler(
Controller wait for attach and exponential backoff Modify attach/detach controller to keep track of volumes to report attached in Node VolumeToAttach status. Modify kubelet volume manager to wait for volume to show up in Node VolumeToAttach status. Implement exponential backoff for errors in volume manager and attach detach controller
2016-06-16 06:48:04 +00:00
kubeClient,
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
false, /* controllerAttachDetachEnabled */
reconcilerLoopSleepDuration,
Fix volume states out of sync problem after kubelet restarts When kubelet restarts, all the information about the volumes will be gone from actual/desired states. When update node status with mounted volumes, the volume list might be empty although there are still volumes are mounted and in turn causing master to detach those volumes since they are not in the mounted volumes list. This fix is to make sure only update mounted volumes list after reconciler starts sync states process. This sync state process will scan the existing volume directories and reconstruct actual states if they are missing. This PR also fixes the problem during orphaned pods' directories. In case of the pod directory is unmounted but has not yet deleted (e.g., interrupted with kubelet restarts), clean up routine will delete the directory so that the pod directoriy could be cleaned up (it is safe to delete directory since it is no longer mounted) The third issue this PR fixes is that during reconstruct volume in actual state, mounter could not be nil since it is required for creating container.VolumeMap. If it is nil, it might cause nil pointer exception in kubelet. Details are in proposal PR #33203
2016-09-27 23:12:57 +00:00
reconcilerSyncStatesSleepPeriod,
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
waitForAttachTimeout,
nodeName,
dsw,
asw,
Make volume unmount more robust using exclusive mount w/ O_EXCL
2016-07-06 17:42:56 +00:00
oex,
This change supports robust kubelet volume cleanup Currently kubelet volume management works on the concept of desired and actual world of states. The volume manager periodically compares the two worlds and perform volume mount/unmount and/or attach/detach operations. When kubelet restarts, the cache of those two worlds are gone. Although desired world can be recovered through apiserver, actual world can not be recovered which may cause some volumes cannot be cleaned up if their information is deleted by apiserver. This change adds the reconstruction of the actual world by reading the pod directories from disk. The reconstructed volume information is added to both desired world and actual world if it cannot be found in either world. The rest logic would be as same as before, desired world populator may clean up the volume entry if it is no longer in apiserver, and then volume manager should invoke unmount to clean it up.
2016-06-23 19:46:21 +00:00
&mount.FakeMounter{},
volumePluginMgr,
kubeletPodsDir)
cmd/kubelet
2016-11-18 20:50:58 +00:00
pod := &v1.Pod{
refactor: use metav1.ObjectMeta in other types
2017-01-17 03:38:19 +00:00
ObjectMeta: metav1.ObjectMeta{
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
Name: "pod1",
UID: "pod1uid",
},
cmd/kubelet
2016-11-18 20:50:58 +00:00
Spec: v1.PodSpec{
Volumes: []v1.Volume{
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
{
Name: "volume-name",
cmd/kubelet
2016-11-18 20:50:58 +00:00
VolumeSource: v1.VolumeSource{
GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
PDName: "fake-device1",
},
},
},
},
},
}
volumeSpec := &volume.Spec{Volume: &pod.Spec.Volumes[0]}
podName := volumehelper.GetUniquePodName(pod)
Mark VolumeInUse before checking if it is Attached Ensure that kublet marks VolumeInUse before checking if it is Attached. Also ensures that the attach/detach controller always fetches a fresh copy of the node object before detach (instead ofKubelet relying on node informer cache).
2016-06-27 00:33:01 +00:00
generatedVolumeName, err := dsw.AddPodToVolume(
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
podName, pod, volumeSpec, volumeSpec.Name(), "" /* volumeGidValue */)
// Assert
if err != nil {
t.Fatalf("AddPodToVolume failed. Expected: <no error> Actual: <%v>", err)
}
// Act
This change supports robust kubelet volume cleanup Currently kubelet volume management works on the concept of desired and actual world of states. The volume manager periodically compares the two worlds and perform volume mount/unmount and/or attach/detach operations. When kubelet restarts, the cache of those two worlds are gone. Although desired world can be recovered through apiserver, actual world can not be recovered which may cause some volumes cannot be cleaned up if their information is deleted by apiserver. This change adds the reconstruction of the actual world by reading the pod directories from disk. The reconstructed volume information is added to both desired world and actual world if it cannot be found in either world. The rest logic would be as same as before, desired world populator may clean up the volume entry if it is no longer in apiserver, and then volume manager should invoke unmount to clean it up.
2016-06-23 19:46:21 +00:00
runReconciler(reconciler)
Mark VolumeInUse before checking if it is Attached Ensure that kublet marks VolumeInUse before checking if it is Attached. Also ensures that the attach/detach controller always fetches a fresh copy of the node object before detach (instead ofKubelet relying on node informer cache).
2016-06-27 00:33:01 +00:00
waitForMount(t, fakePlugin, generatedVolumeName, asw)
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
// Assert
assert.NoError(t, volumetesting.VerifyAttachCallCount(
1 /* expectedAttachCallCount */, fakePlugin))
assert.NoError(t, volumetesting.VerifyWaitForAttachCallCount(
1 /* expectedWaitForAttachCallCount */, fakePlugin))
assert.NoError(t, volumetesting.VerifyMountDeviceCallCount(
1 /* expectedMountDeviceCallCount */, fakePlugin))
assert.NoError(t, volumetesting.VerifySetUpCallCount(
1 /* expectedSetUpCallCount */, fakePlugin))
assert.NoError(t, volumetesting.VerifyZeroTearDownCallCount(fakePlugin))
assert.NoError(t, volumetesting.VerifyZeroDetachCallCount(fakePlugin))
}
// Populates desiredStateOfWorld cache with one volume/pod.
// Enables controllerAttachDetachEnabled.
// Calls Run()
// Verifies there is one mount call and no unmount calls.
// Verifies there are no attach/detach calls.
func Test_Run_Positive_VolumeMountControllerAttachEnabled(t *testing.T) {
// Arrange
volumePluginMgr, fakePlugin := volumetesting.GetTestVolumePluginMgr(t)
dsw := cache.NewDesiredStateOfWorld(volumePluginMgr)
asw := cache.NewActualStateOfWorld(nodeName, volumePluginMgr)
Controller wait for attach and exponential backoff Modify attach/detach controller to keep track of volumes to report attached in Node VolumeToAttach status. Modify kubelet volume manager to wait for volume to show up in Node VolumeToAttach status. Implement exponential backoff for errors in volume manager and attach detach controller
2016-06-16 06:48:04 +00:00
kubeClient := createTestClient()
Add Events for operation_executor to show status of mounts, failed or successful
2016-06-21 16:13:23 +00:00
fakeRecorder := &record.FakeRecorder{}
Add unit tests for MountVolume() of operation executor
2016-12-21 19:39:47 +00:00
oex := operationexecutor.NewOperationExecutor(operationexecutor.NewOperationGenerator(kubeClient, volumePluginMgr, fakeRecorder, false /* checkNodeCapabilitiesBeforeMount */))
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
reconciler := NewReconciler(
Controller wait for attach and exponential backoff Modify attach/detach controller to keep track of volumes to report attached in Node VolumeToAttach status. Modify kubelet volume manager to wait for volume to show up in Node VolumeToAttach status. Implement exponential backoff for errors in volume manager and attach detach controller
2016-06-16 06:48:04 +00:00
kubeClient,
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
true, /* controllerAttachDetachEnabled */
reconcilerLoopSleepDuration,
Fix volume states out of sync problem after kubelet restarts When kubelet restarts, all the information about the volumes will be gone from actual/desired states. When update node status with mounted volumes, the volume list might be empty although there are still volumes are mounted and in turn causing master to detach those volumes since they are not in the mounted volumes list. This fix is to make sure only update mounted volumes list after reconciler starts sync states process. This sync state process will scan the existing volume directories and reconstruct actual states if they are missing. This PR also fixes the problem during orphaned pods' directories. In case of the pod directory is unmounted but has not yet deleted (e.g., interrupted with kubelet restarts), clean up routine will delete the directory so that the pod directoriy could be cleaned up (it is safe to delete directory since it is no longer mounted) The third issue this PR fixes is that during reconstruct volume in actual state, mounter could not be nil since it is required for creating container.VolumeMap. If it is nil, it might cause nil pointer exception in kubelet. Details are in proposal PR #33203
2016-09-27 23:12:57 +00:00
reconcilerSyncStatesSleepPeriod,
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
waitForAttachTimeout,
nodeName,
dsw,
asw,
Make volume unmount more robust using exclusive mount w/ O_EXCL
2016-07-06 17:42:56 +00:00
oex,
This change supports robust kubelet volume cleanup Currently kubelet volume management works on the concept of desired and actual world of states. The volume manager periodically compares the two worlds and perform volume mount/unmount and/or attach/detach operations. When kubelet restarts, the cache of those two worlds are gone. Although desired world can be recovered through apiserver, actual world can not be recovered which may cause some volumes cannot be cleaned up if their information is deleted by apiserver. This change adds the reconstruction of the actual world by reading the pod directories from disk. The reconstructed volume information is added to both desired world and actual world if it cannot be found in either world. The rest logic would be as same as before, desired world populator may clean up the volume entry if it is no longer in apiserver, and then volume manager should invoke unmount to clean it up.
2016-06-23 19:46:21 +00:00
&mount.FakeMounter{},
volumePluginMgr,
kubeletPodsDir)
cmd/kubelet
2016-11-18 20:50:58 +00:00
pod := &v1.Pod{
refactor: use metav1.ObjectMeta in other types
2017-01-17 03:38:19 +00:00
ObjectMeta: metav1.ObjectMeta{
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
Name: "pod1",
UID: "pod1uid",
},
cmd/kubelet
2016-11-18 20:50:58 +00:00
Spec: v1.PodSpec{
Volumes: []v1.Volume{
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
{
Name: "volume-name",
cmd/kubelet
2016-11-18 20:50:58 +00:00
VolumeSource: v1.VolumeSource{
GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
PDName: "fake-device1",
},
},
},
},
},
}
volumeSpec := &volume.Spec{Volume: &pod.Spec.Volumes[0]}
podName := volumehelper.GetUniquePodName(pod)
Mark VolumeInUse before checking if it is Attached Ensure that kublet marks VolumeInUse before checking if it is Attached. Also ensures that the attach/detach controller always fetches a fresh copy of the node object before detach (instead ofKubelet relying on node informer cache).
2016-06-27 00:33:01 +00:00
generatedVolumeName, err := dsw.AddPodToVolume(
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
podName, pod, volumeSpec, volumeSpec.Name(), "" /* volumeGidValue */)
cmd/kubelet
2016-11-18 20:50:58 +00:00
dsw.MarkVolumesReportedInUse([]v1.UniqueVolumeName{generatedVolumeName})
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
// Assert
if err != nil {
t.Fatalf("AddPodToVolume failed. Expected: <no error> Actual: <%v>", err)
}
// Act
This change supports robust kubelet volume cleanup Currently kubelet volume management works on the concept of desired and actual world of states. The volume manager periodically compares the two worlds and perform volume mount/unmount and/or attach/detach operations. When kubelet restarts, the cache of those two worlds are gone. Although desired world can be recovered through apiserver, actual world can not be recovered which may cause some volumes cannot be cleaned up if their information is deleted by apiserver. This change adds the reconstruction of the actual world by reading the pod directories from disk. The reconstructed volume information is added to both desired world and actual world if it cannot be found in either world. The rest logic would be as same as before, desired world populator may clean up the volume entry if it is no longer in apiserver, and then volume manager should invoke unmount to clean it up.
2016-06-23 19:46:21 +00:00
runReconciler(reconciler)
Mark VolumeInUse before checking if it is Attached Ensure that kublet marks VolumeInUse before checking if it is Attached. Also ensures that the attach/detach controller always fetches a fresh copy of the node object before detach (instead ofKubelet relying on node informer cache).
2016-06-27 00:33:01 +00:00
waitForMount(t, fakePlugin, generatedVolumeName, asw)
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
// Assert
assert.NoError(t, volumetesting.VerifyZeroAttachCalls(fakePlugin))
assert.NoError(t, volumetesting.VerifyWaitForAttachCallCount(
1 /* expectedWaitForAttachCallCount */, fakePlugin))
assert.NoError(t, volumetesting.VerifyMountDeviceCallCount(
1 /* expectedMountDeviceCallCount */, fakePlugin))
assert.NoError(t, volumetesting.VerifySetUpCallCount(
1 /* expectedSetUpCallCount */, fakePlugin))
assert.NoError(t, volumetesting.VerifyZeroTearDownCallCount(fakePlugin))
assert.NoError(t, volumetesting.VerifyZeroDetachCallCount(fakePlugin))
}
// Populates desiredStateOfWorld cache with one volume/pod.
// Calls Run()
// Verifies there is one attach/mount/etc call and no detach calls.
// Deletes volume/pod from desired state of world.
// Verifies detach/unmount calls are issued.
func Test_Run_Positive_VolumeAttachMountUnmountDetach(t *testing.T) {
// Arrange
volumePluginMgr, fakePlugin := volumetesting.GetTestVolumePluginMgr(t)
dsw := cache.NewDesiredStateOfWorld(volumePluginMgr)
asw := cache.NewActualStateOfWorld(nodeName, volumePluginMgr)
Controller wait for attach and exponential backoff Modify attach/detach controller to keep track of volumes to report attached in Node VolumeToAttach status. Modify kubelet volume manager to wait for volume to show up in Node VolumeToAttach status. Implement exponential backoff for errors in volume manager and attach detach controller
2016-06-16 06:48:04 +00:00
kubeClient := createTestClient()
Add Events for operation_executor to show status of mounts, failed or successful
2016-06-21 16:13:23 +00:00
fakeRecorder := &record.FakeRecorder{}
Add unit tests for MountVolume() of operation executor
2016-12-21 19:39:47 +00:00
oex := operationexecutor.NewOperationExecutor(operationexecutor.NewOperationGenerator(kubeClient, volumePluginMgr, fakeRecorder, false /* checkNodeCapabilitiesBeforeMount */))
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
reconciler := NewReconciler(
Controller wait for attach and exponential backoff Modify attach/detach controller to keep track of volumes to report attached in Node VolumeToAttach status. Modify kubelet volume manager to wait for volume to show up in Node VolumeToAttach status. Implement exponential backoff for errors in volume manager and attach detach controller
2016-06-16 06:48:04 +00:00
kubeClient,
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
false, /* controllerAttachDetachEnabled */
reconcilerLoopSleepDuration,
Fix volume states out of sync problem after kubelet restarts When kubelet restarts, all the information about the volumes will be gone from actual/desired states. When update node status with mounted volumes, the volume list might be empty although there are still volumes are mounted and in turn causing master to detach those volumes since they are not in the mounted volumes list. This fix is to make sure only update mounted volumes list after reconciler starts sync states process. This sync state process will scan the existing volume directories and reconstruct actual states if they are missing. This PR also fixes the problem during orphaned pods' directories. In case of the pod directory is unmounted but has not yet deleted (e.g., interrupted with kubelet restarts), clean up routine will delete the directory so that the pod directoriy could be cleaned up (it is safe to delete directory since it is no longer mounted) The third issue this PR fixes is that during reconstruct volume in actual state, mounter could not be nil since it is required for creating container.VolumeMap. If it is nil, it might cause nil pointer exception in kubelet. Details are in proposal PR #33203
2016-09-27 23:12:57 +00:00
reconcilerSyncStatesSleepPeriod,
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
waitForAttachTimeout,
nodeName,
dsw,
asw,
Make volume unmount more robust using exclusive mount w/ O_EXCL
2016-07-06 17:42:56 +00:00
oex,
This change supports robust kubelet volume cleanup Currently kubelet volume management works on the concept of desired and actual world of states. The volume manager periodically compares the two worlds and perform volume mount/unmount and/or attach/detach operations. When kubelet restarts, the cache of those two worlds are gone. Although desired world can be recovered through apiserver, actual world can not be recovered which may cause some volumes cannot be cleaned up if their information is deleted by apiserver. This change adds the reconstruction of the actual world by reading the pod directories from disk. The reconstructed volume information is added to both desired world and actual world if it cannot be found in either world. The rest logic would be as same as before, desired world populator may clean up the volume entry if it is no longer in apiserver, and then volume manager should invoke unmount to clean it up.
2016-06-23 19:46:21 +00:00
&mount.FakeMounter{},
volumePluginMgr,
kubeletPodsDir)
cmd/kubelet
2016-11-18 20:50:58 +00:00
pod := &v1.Pod{
refactor: use metav1.ObjectMeta in other types
2017-01-17 03:38:19 +00:00
ObjectMeta: metav1.ObjectMeta{
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
Name: "pod1",
UID: "pod1uid",
},
cmd/kubelet
2016-11-18 20:50:58 +00:00
Spec: v1.PodSpec{
Volumes: []v1.Volume{
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
{
Name: "volume-name",
cmd/kubelet
2016-11-18 20:50:58 +00:00
VolumeSource: v1.VolumeSource{
GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
PDName: "fake-device1",
},
},
},
},
},
}
volumeSpec := &volume.Spec{Volume: &pod.Spec.Volumes[0]}
podName := volumehelper.GetUniquePodName(pod)
generatedVolumeName, err := dsw.AddPodToVolume(
podName, pod, volumeSpec, volumeSpec.Name(), "" /* volumeGidValue */)
// Assert
if err != nil {
t.Fatalf("AddPodToVolume failed. Expected: <no error> Actual: <%v>", err)
}
// Act
This change supports robust kubelet volume cleanup Currently kubelet volume management works on the concept of desired and actual world of states. The volume manager periodically compares the two worlds and perform volume mount/unmount and/or attach/detach operations. When kubelet restarts, the cache of those two worlds are gone. Although desired world can be recovered through apiserver, actual world can not be recovered which may cause some volumes cannot be cleaned up if their information is deleted by apiserver. This change adds the reconstruction of the actual world by reading the pod directories from disk. The reconstructed volume information is added to both desired world and actual world if it cannot be found in either world. The rest logic would be as same as before, desired world populator may clean up the volume entry if it is no longer in apiserver, and then volume manager should invoke unmount to clean it up.
2016-06-23 19:46:21 +00:00
runReconciler(reconciler)
Mark VolumeInUse before checking if it is Attached Ensure that kublet marks VolumeInUse before checking if it is Attached. Also ensures that the attach/detach controller always fetches a fresh copy of the node object before detach (instead ofKubelet relying on node informer cache).
2016-06-27 00:33:01 +00:00
waitForMount(t, fakePlugin, generatedVolumeName, asw)
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
// Assert
assert.NoError(t, volumetesting.VerifyAttachCallCount(
1 /* expectedAttachCallCount */, fakePlugin))
assert.NoError(t, volumetesting.VerifyWaitForAttachCallCount(
1 /* expectedWaitForAttachCallCount */, fakePlugin))
assert.NoError(t, volumetesting.VerifyMountDeviceCallCount(
1 /* expectedMountDeviceCallCount */, fakePlugin))
assert.NoError(t, volumetesting.VerifySetUpCallCount(
1 /* expectedSetUpCallCount */, fakePlugin))
assert.NoError(t, volumetesting.VerifyZeroTearDownCallCount(fakePlugin))
assert.NoError(t, volumetesting.VerifyZeroDetachCallCount(fakePlugin))
// Act
dsw.DeletePodFromVolume(podName, generatedVolumeName)
Mark VolumeInUse before checking if it is Attached Ensure that kublet marks VolumeInUse before checking if it is Attached. Also ensures that the attach/detach controller always fetches a fresh copy of the node object before detach (instead ofKubelet relying on node informer cache).
2016-06-27 00:33:01 +00:00
waitForDetach(t, fakePlugin, generatedVolumeName, asw)
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
// Assert
assert.NoError(t, volumetesting.VerifyTearDownCallCount(
1 /* expectedTearDownCallCount */, fakePlugin))
assert.NoError(t, volumetesting.VerifyDetachCallCount(
1 /* expectedDetachCallCount */, fakePlugin))
}
// Populates desiredStateOfWorld cache with one volume/pod.
// Enables controllerAttachDetachEnabled.
// Calls Run()
// Verifies one mount call is made and no unmount calls.
// Deletes volume/pod from desired state of world.
// Verifies one unmount call is made.
// Verifies there are no attach/detach calls made.
func Test_Run_Positive_VolumeUnmountControllerAttachEnabled(t *testing.T) {
// Arrange
volumePluginMgr, fakePlugin := volumetesting.GetTestVolumePluginMgr(t)
dsw := cache.NewDesiredStateOfWorld(volumePluginMgr)
asw := cache.NewActualStateOfWorld(nodeName, volumePluginMgr)
Controller wait for attach and exponential backoff Modify attach/detach controller to keep track of volumes to report attached in Node VolumeToAttach status. Modify kubelet volume manager to wait for volume to show up in Node VolumeToAttach status. Implement exponential backoff for errors in volume manager and attach detach controller
2016-06-16 06:48:04 +00:00
kubeClient := createTestClient()
Add Events for operation_executor to show status of mounts, failed or successful
2016-06-21 16:13:23 +00:00
fakeRecorder := &record.FakeRecorder{}
Add unit tests for MountVolume() of operation executor
2016-12-21 19:39:47 +00:00
oex := operationexecutor.NewOperationExecutor(operationexecutor.NewOperationGenerator(kubeClient, volumePluginMgr, fakeRecorder, false /* checkNodeCapabilitiesBeforeMount */))
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
reconciler := NewReconciler(
Controller wait for attach and exponential backoff Modify attach/detach controller to keep track of volumes to report attached in Node VolumeToAttach status. Modify kubelet volume manager to wait for volume to show up in Node VolumeToAttach status. Implement exponential backoff for errors in volume manager and attach detach controller
2016-06-16 06:48:04 +00:00
kubeClient,
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
true, /* controllerAttachDetachEnabled */
reconcilerLoopSleepDuration,
Fix volume states out of sync problem after kubelet restarts When kubelet restarts, all the information about the volumes will be gone from actual/desired states. When update node status with mounted volumes, the volume list might be empty although there are still volumes are mounted and in turn causing master to detach those volumes since they are not in the mounted volumes list. This fix is to make sure only update mounted volumes list after reconciler starts sync states process. This sync state process will scan the existing volume directories and reconstruct actual states if they are missing. This PR also fixes the problem during orphaned pods' directories. In case of the pod directory is unmounted but has not yet deleted (e.g., interrupted with kubelet restarts), clean up routine will delete the directory so that the pod directoriy could be cleaned up (it is safe to delete directory since it is no longer mounted) The third issue this PR fixes is that during reconstruct volume in actual state, mounter could not be nil since it is required for creating container.VolumeMap. If it is nil, it might cause nil pointer exception in kubelet. Details are in proposal PR #33203
2016-09-27 23:12:57 +00:00
reconcilerSyncStatesSleepPeriod,
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
waitForAttachTimeout,
nodeName,
dsw,
asw,
Make volume unmount more robust using exclusive mount w/ O_EXCL
2016-07-06 17:42:56 +00:00
oex,
This change supports robust kubelet volume cleanup Currently kubelet volume management works on the concept of desired and actual world of states. The volume manager periodically compares the two worlds and perform volume mount/unmount and/or attach/detach operations. When kubelet restarts, the cache of those two worlds are gone. Although desired world can be recovered through apiserver, actual world can not be recovered which may cause some volumes cannot be cleaned up if their information is deleted by apiserver. This change adds the reconstruction of the actual world by reading the pod directories from disk. The reconstructed volume information is added to both desired world and actual world if it cannot be found in either world. The rest logic would be as same as before, desired world populator may clean up the volume entry if it is no longer in apiserver, and then volume manager should invoke unmount to clean it up.
2016-06-23 19:46:21 +00:00
&mount.FakeMounter{},
volumePluginMgr,
kubeletPodsDir)
cmd/kubelet
2016-11-18 20:50:58 +00:00
pod := &v1.Pod{
refactor: use metav1.ObjectMeta in other types
2017-01-17 03:38:19 +00:00
ObjectMeta: metav1.ObjectMeta{
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
Name: "pod1",
UID: "pod1uid",
},
cmd/kubelet
2016-11-18 20:50:58 +00:00
Spec: v1.PodSpec{
Volumes: []v1.Volume{
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
{
Name: "volume-name",
cmd/kubelet
2016-11-18 20:50:58 +00:00
VolumeSource: v1.VolumeSource{
GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
PDName: "fake-device1",
},
},
},
},
},
}
volumeSpec := &volume.Spec{Volume: &pod.Spec.Volumes[0]}
podName := volumehelper.GetUniquePodName(pod)
generatedVolumeName, err := dsw.AddPodToVolume(
podName, pod, volumeSpec, volumeSpec.Name(), "" /* volumeGidValue */)
// Assert
if err != nil {
t.Fatalf("AddPodToVolume failed. Expected: <no error> Actual: <%v>", err)
}
// Act
This change supports robust kubelet volume cleanup Currently kubelet volume management works on the concept of desired and actual world of states. The volume manager periodically compares the two worlds and perform volume mount/unmount and/or attach/detach operations. When kubelet restarts, the cache of those two worlds are gone. Although desired world can be recovered through apiserver, actual world can not be recovered which may cause some volumes cannot be cleaned up if their information is deleted by apiserver. This change adds the reconstruction of the actual world by reading the pod directories from disk. The reconstructed volume information is added to both desired world and actual world if it cannot be found in either world. The rest logic would be as same as before, desired world populator may clean up the volume entry if it is no longer in apiserver, and then volume manager should invoke unmount to clean it up.
2016-06-23 19:46:21 +00:00
runReconciler(reconciler)
cmd/kubelet
2016-11-18 20:50:58 +00:00
dsw.MarkVolumesReportedInUse([]v1.UniqueVolumeName{generatedVolumeName})
Mark VolumeInUse before checking if it is Attached Ensure that kublet marks VolumeInUse before checking if it is Attached. Also ensures that the attach/detach controller always fetches a fresh copy of the node object before detach (instead ofKubelet relying on node informer cache).
2016-06-27 00:33:01 +00:00
waitForMount(t, fakePlugin, generatedVolumeName, asw)
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
// Assert
assert.NoError(t, volumetesting.VerifyZeroAttachCalls(fakePlugin))
assert.NoError(t, volumetesting.VerifyWaitForAttachCallCount(
1 /* expectedWaitForAttachCallCount */, fakePlugin))
assert.NoError(t, volumetesting.VerifyMountDeviceCallCount(
1 /* expectedMountDeviceCallCount */, fakePlugin))
assert.NoError(t, volumetesting.VerifySetUpCallCount(
1 /* expectedSetUpCallCount */, fakePlugin))
assert.NoError(t, volumetesting.VerifyZeroTearDownCallCount(fakePlugin))
assert.NoError(t, volumetesting.VerifyZeroDetachCallCount(fakePlugin))
// Act
dsw.DeletePodFromVolume(podName, generatedVolumeName)
Mark VolumeInUse before checking if it is Attached Ensure that kublet marks VolumeInUse before checking if it is Attached. Also ensures that the attach/detach controller always fetches a fresh copy of the node object before detach (instead ofKubelet relying on node informer cache).
2016-06-27 00:33:01 +00:00
waitForDetach(t, fakePlugin, generatedVolumeName, asw)
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
// Assert
assert.NoError(t, volumetesting.VerifyTearDownCallCount(
1 /* expectedTearDownCallCount */, fakePlugin))
assert.NoError(t, volumetesting.VerifyZeroDetachCallCount(fakePlugin))
}
Mark VolumeInUse before checking if it is Attached Ensure that kublet marks VolumeInUse before checking if it is Attached. Also ensures that the attach/detach controller always fetches a fresh copy of the node object before detach (instead ofKubelet relying on node informer cache).
2016-06-27 00:33:01 +00:00
func waitForMount(
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
t *testing.T,
fakePlugin *volumetesting.FakeVolumePlugin,
cmd/kubelet
2016-11-18 20:50:58 +00:00
volumeName v1.UniqueVolumeName,
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
asw cache.ActualStateOfWorld) {
err := retryWithExponentialBackOff(
time.Duration(5*time.Millisecond),
func() (bool, error) {
mountedVolumes := asw.GetMountedVolumes()
Mark VolumeInUse before checking if it is Attached Ensure that kublet marks VolumeInUse before checking if it is Attached. Also ensures that the attach/detach controller always fetches a fresh copy of the node object before detach (instead ofKubelet relying on node informer cache).
2016-06-27 00:33:01 +00:00
for _, mountedVolume := range mountedVolumes {
if mountedVolume.VolumeName == volumeName {
return true, nil
}
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
}
return false, nil
},
)
if err != nil {
Mark VolumeInUse before checking if it is Attached Ensure that kublet marks VolumeInUse before checking if it is Attached. Also ensures that the attach/detach controller always fetches a fresh copy of the node object before detach (instead ofKubelet relying on node informer cache).
2016-06-27 00:33:01 +00:00
t.Fatalf("Timed out waiting for volume %q to be attached.", volumeName)
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
}
}
func waitForDetach(
t *testing.T,
fakePlugin *volumetesting.FakeVolumePlugin,
cmd/kubelet
2016-11-18 20:50:58 +00:00
volumeName v1.UniqueVolumeName,
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
asw cache.ActualStateOfWorld) {
err := retryWithExponentialBackOff(
time.Duration(5*time.Millisecond),
func() (bool, error) {
Mark VolumeInUse before checking if it is Attached Ensure that kublet marks VolumeInUse before checking if it is Attached. Also ensures that the attach/detach controller always fetches a fresh copy of the node object before detach (instead ofKubelet relying on node informer cache).
2016-06-27 00:33:01 +00:00
if asw.VolumeExists(volumeName) {
return false, nil
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
}
Mark VolumeInUse before checking if it is Attached Ensure that kublet marks VolumeInUse before checking if it is Attached. Also ensures that the attach/detach controller always fetches a fresh copy of the node object before detach (instead ofKubelet relying on node informer cache).
2016-06-27 00:33:01 +00:00
return true, nil
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
},
)
if err != nil {
Mark VolumeInUse before checking if it is Attached Ensure that kublet marks VolumeInUse before checking if it is Attached. Also ensures that the attach/detach controller always fetches a fresh copy of the node object before detach (instead ofKubelet relying on node informer cache).
2016-06-27 00:33:01 +00:00
t.Fatalf("Timed out waiting for volume %q to be detached.", volumeName)
Introduce new kubelet volume manager This commit adds a new volume manager in kubelet that synchronizes volume mount/unmount (and attach/detach, if attach/detach controller is not enabled). This eliminates the race conditions between the pod creation loop and the orphaned volumes loops. It also removes the unmount/detach from the `syncPod()` path so volume clean up never blocks the `syncPod` loop.
2016-05-30 02:22:22 +00:00
}
}
func retryWithExponentialBackOff(initialDuration time.Duration, fn wait.ConditionFunc) error {
backoff := wait.Backoff{
Duration: initialDuration,
Factor: 3,
Jitter: 0,
Steps: 6,
}
return wait.ExponentialBackoff(backoff, fn)
}
Controller wait for attach and exponential backoff Modify attach/detach controller to keep track of volumes to report attached in Node VolumeToAttach status. Modify kubelet volume manager to wait for volume to show up in Node VolumeToAttach status. Implement exponential backoff for errors in volume manager and attach detach controller
2016-06-16 06:48:04 +00:00
func createTestClient() *fake.Clientset {
fakeClient := &fake.Clientset{}
fakeClient.AddReactor("get", "nodes",
func(action core.Action) (bool, runtime.Object, error) {
cmd/kubelet
2016-11-18 20:50:58 +00:00
return true, &v1.Node{
refactor: use metav1.ObjectMeta in other types
2017-01-17 03:38:19 +00:00
ObjectMeta: metav1.ObjectMeta{Name: string(nodeName)},
cmd/kubelet
2016-11-18 20:50:58 +00:00
Status: v1.NodeStatus{
VolumesAttached: []v1.AttachedVolume{
Controller wait for attach and exponential backoff Modify attach/detach controller to keep track of volumes to report attached in Node VolumeToAttach status. Modify kubelet volume manager to wait for volume to show up in Node VolumeToAttach status. Implement exponential backoff for errors in volume manager and attach detach controller
2016-06-16 06:48:04 +00:00
{
Name: "fake-plugin/volume-name",
DevicePath: "fake/path",
},
}},
cmd/kubelet
2016-11-18 20:50:58 +00:00
Spec: v1.NodeSpec{ExternalID: string(nodeName)},
Controller wait for attach and exponential backoff Modify attach/detach controller to keep track of volumes to report attached in Node VolumeToAttach status. Modify kubelet volume manager to wait for volume to show up in Node VolumeToAttach status. Implement exponential backoff for errors in volume manager and attach detach controller
2016-06-16 06:48:04 +00:00
}, nil
})
fakeClient.AddReactor("*", "*", func(action core.Action) (bool, runtime.Object, error) {
return true, nil, fmt.Errorf("no reaction implemented for %s", action)
})
return fakeClient
}
This change supports robust kubelet volume cleanup Currently kubelet volume management works on the concept of desired and actual world of states. The volume manager periodically compares the two worlds and perform volume mount/unmount and/or attach/detach operations. When kubelet restarts, the cache of those two worlds are gone. Although desired world can be recovered through apiserver, actual world can not be recovered which may cause some volumes cannot be cleaned up if their information is deleted by apiserver. This change adds the reconstruction of the actual world by reading the pod directories from disk. The reconstructed volume information is added to both desired world and actual world if it cannot be found in either world. The rest logic would be as same as before, desired world populator may clean up the volume entry if it is no longer in apiserver, and then volume manager should invoke unmount to clean it up.
2016-06-23 19:46:21 +00:00
func runReconciler(reconciler Reconciler) {
sourcesReady := config.NewSourcesReady(func(_ sets.String) bool { return false })
go reconciler.Run(sourcesReady, wait.NeverStop)
}