Merge pull request #64098 from jacobwgillespie/fix-e2e-complete-pods-in-kube-system-ns-failure

Automatic merge from submit-queue (batch tested with PRs 63283, 64032, 64159, 64126, 64098). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Fix running e2e tests with completed kube-system pods

As per @timothysc's suggestion, this PR is the same as PR #53222, but with the CLA signed and with a comment explaining why completed pods are skipped.  Credit for finding this fix goes to @dlosev.

From the original PR:

> Currently e2e runner fails during BeforeSuite execution if there are pods in kube-system namespace in 'Completed' state.
>
> Exactly this issue was fixed previously in #37325 by adding new option "skipSucceeded" to ignore 'Completed' pods. Then, in 2e3cd93#diff-eb7b79470992813ea1905e96c298b47bL557 that option was removed and code was adjusted to ignore 'Completed' pods all the time. But in 0bf96a3 that code was accidentally removed as redundant.
>
> This fix restores removed code.

```release-note
NONE
```

test/e2e/framework/util.go

@@ -704,10 +704,9 @@ func WaitForPodsRunningReady(c clientset.Interface, ns string, minPods, allowedN
 			case res && err == nil:
 				nOk++
 			case pod.Status.Phase == v1.PodSucceeded:
-				Logf("The status of Pod %s is Succeeded which is unexpected", pod.ObjectMeta.Name)
-				badPods = append(badPods, pod)
+				Logf("The status of Pod %s is Succeeded, skipping waiting", pod.ObjectMeta.Name)
 				// it doesn't make sense to wait for this pod
-				return false, errors.New("unexpected Succeeded pod state")
+				continue
 			case pod.Status.Phase != v1.PodFailed:
 				Logf("The status of Pod %s is %s (Ready = false), waiting for it to be either Running (with Ready = true) or Failed", pod.ObjectMeta.Name, pod.Status.Phase)
 				notReady++