Automatic merge from submit-queue
Install go-bindata in cross-build image
Another follow-up to #25584.
We need `go-bindata` to create `test/e2e/generated`, and downloading it with `go get` at build time is painful for a variety of reasons. We can just include it in the cross-build image and not worry about it, especially as it updates very infrequently.
This fixes `hack/update-generated-protobuf.sh` as well.
cc @jayunit100 @soltysh
Automatic merge from submit-queue
Resolve docker-daemon cgroup issue for both systemd and non-systemd node for node e2e tests
Fixed https://github.com/kubernetes/kubernetes/issues/29827
cc/ @coufon this should unblock your pr: #29764
I validated both containervm image and coreos image, and works as expected.
This is also required for adding gci image to node e2e test infrastructure.
Automatic merge from submit-queue
Node E2E: Add serial jenkins job.
This PR added a jenkins job for serial test. It will run all serial test one by one.
This will be useful for https://github.com/kubernetes/kubernetes/pull/29809.
@coufon @yujuhong @dchen1107
/cc @kubernetes/sig-node
Automatic merge from submit-queue
E2E & Node E2E: Add exec util in framework
For #29081.
Based on #29092 and #29494.
For first commit is a squashed commit of all old commits.
**The last 2 commits are new.**
This PR added exec util in framework, and moved `privileged.go` and `kubelet_etc_hosts` into `common` directory.
@vishh @timstclair
/cc @kubernetes/sig-node
Automatic merge from submit-queue
Node E2E: Make node e2e parallel
For https://github.com/kubernetes/kubernetes/issues/29081.
Fix https://github.com/kubernetes/kubernetes/issues/26215.
Based on https://github.com/kubernetes/kubernetes/pull/28807, https://github.com/kubernetes/kubernetes/pull/29020, will rebase after they are merged.
**Only the last commit is new.**
We are going to move more tests into the node e2e test. However, currently node e2e test only run sequentially, the test duration will increase quickly when we add more test.
This PR makes the node e2e test run in parallel so as to shorten test duration, so that we can add more test to improve the test coverage.
* If you run the test locally with `make test-e2e-node`, it will use `-p` ginkgo flag, which uses `(cores-1)` parallel test nodes by default.
* If you run the test remotely or in the Jenkin, the parallelism will be controlled by the environment variable `PARALLELISM`. The default value is `8`, which is reasonable for our test node (n1-standard-1).
Before this PR, it took **833.592s** to run all test on my desktop.
With this PR, it only takes **234.058s** to run.
The pull request node e2e run with this PR takes **232.327s**.
The pull request node e2e run for other PRs takes **673.810s**.
/cc @kubernetes/sig-node
Automatic merge from submit-queue
Adding GCI to node e2e.
Depends on https://github.com/kubernetes/kubernetes/pull/29486
Adding the dev release as of now since stable and beta run docker v1.9.1
which is incompatible with kubelet.
bindata and yaml, Gobindata automation
bindata utils for generating, go generate
match server version
gitignore for dirty, ca, rbase, KUBE_ROOT, buildfix
(rebased jul-25,29)
Automatic merge from submit-queue
Fix killing child sudo process in e2e_node tests
Fixes#29211.
The context is we are trying to kill a process started as `sudo kube-apiserver`, but `sudo` ignores signals from the same process group. Applying `Setpgid` means the `sudo kill` process won't be in the same process group, so will not fall foul of this nifty feature.
I also took the liberty of removing some code setting `Pdeathsig` because it claims to be doing something in the same area, but actually it doesn't do that at all. The setting is applied to the forked process, i.e. `sudo`, and it means the `sudo` will get killed if we (`e2e_node.test`) die. This (a) isn't what the comment says and (b) doesn't help because sending SIGKILL to the sudo process leaves sudo's child alive.
I didn't use the "hack for linux-only" approach because I think `Setpgid` is available on all platforms that `e2e_node` builds on.
This is the default for etcd2, but etcd3 only listens on 2379.
Specifying the ports keeps things consistent no matter which
version the user has installed.
Automatic merge from submit-queue
Syncing imaging pulling backoff logic
- Syncing the backoff logic in the parallel image puller and the sequential image puller to prepare for merging the two pullers into one.
- Moving image error definitions under kubelet/images
Automatic merge from submit-queue
Change some node e2e test to use the prepull image framework.
Fix https://github.com/kubernetes/kubernetes/issues/28868.
Node e2e test framework pre-pulls all images in [image_list.go](bc2f223f5a/test/e2e_node/image_list.go)
All node e2e test should use image from the "image_list". If a test needs new image, we should update the image_list to include the new image.
/cc @kubernetes/sig-node to notice people to use `image_list` when adding test. :)
Automatic merge from submit-queue
Start namespace controller in node e2e
Fix https://github.com/kubernetes/kubernetes/issues/28320.
Based on https://github.com/kubernetes/kubernetes/pull/28807, only the last 2 commits are new.
Before this PR, there was no namespace controller running in node e2e test infrastructure. We can not enable the [`delete-namespace`](f2ddd60eb9/test/e2e/framework/test_context.go (L109)) flag in the test framework.
So after the test running, there will be running pod left on the test node. This seems to be acceptable in our test infrastructure because we create an new instance each time.
However, in 1.4 we may want to provide part of the test as node conformance test to the user, they definitely don't want the test to leave tons of pods on their node after test running.
Currently, there is no easy way to only start namespace controller in kube-controller-manager (confirmed with @mikedanese), so in this PR I started a "uncontainerized" one in the test infrastructure.
This PR:
* Started the namespace controller in the node e2e test infrastructure and enable the automatic namespace deletion.
* Change the privileged test to use framework (@yujuhong), so that all node e2e tests are using the framework and test pods will be cleaned up by namespace controller.
/cc @kubernetes/sig-node
Automatic merge from submit-queue
Don't repeat the program name in healthCheckCommand.String()
The name is in both `Path` and `Args[0]`, so start printing args at 1.
Also refactor to avoid an extra space character in the output.
I pondered whether `healthCheckCommand.String()` should check if the slice is empty, to avoid a panic, but it didn't check for `Cmd==nil` before.
Fixes#29107
Automatic merge from submit-queue
Change the docker validation node e2e test to use gci-canary-test
This PR changed the continuous docker validation node e2e test to use the image config file introduced in https://github.com/kubernetes/kubernetes/pull/28708. @euank
This PR also changed the gci image family from `gci-preview-test` to `gci-canary-test`. @wonderfly
Automatic merge from submit-queue
Node E2E: Make it possible to share test between e2e and node e2e
This PR is part of the plan to improve node e2e test coverage.
* Now to improve test coverage, we have to copy test from e2e to node e2e.
* When adding a new test, we have to decide its destiny at the very beginning - whether it is a node e2e or e2e.
This PR makes it possible to share test between e2e and node e2e.
By leveraging the mechanism of ginkgo, as long as we can import the test package in the test suite, the corresponding `Describe` will be run to initialize the global variable `_`, and the test will be inserted into the test suite. (See https://github.com/onsi/composition-ginkgo-example)
In the future, we just need to use the framework to write the test, and put the test into `test/e2e/node`, then it will be automatically shared by the 2 test suites.
This PR:
1) Refactored the framework to make it automatically differentiate e2e and node e2e (Mainly refactored the `PodClient` and the apiserver client initialization).
2) Created a new directory `test/e2e/node` and make it shared by e2e and node e2e.
3) Moved `container_probe.go` into `test/e2e/node` to verify the change.
@kubernetes/sig-node
[]()
Automatic merge from submit-queue
Fix a bug in mirror pod node e2e test.
Fixed a bug in test/e2e_node/mirror_pod_test.go. The function 'checkMirrorPodDisappear' returns nil even when the pod does not disappear. It should return a non-nil error.
@Random-Liu
Automatic merge from submit-queue
Support terminal resizing for exec/attach/run
```release-note
Add support for terminal resizing for exec, attach, and run. Note that for Docker, exec sessions
inherit the environment from the primary process, so if the container was created with tty=false,
that means the exec session's TERM variable will default to "dumb". Users can override this by
setting TERM=xterm (or whatever is appropriate) to get the correct "smart" terminal behavior.
```
Fixes#13585
Add support for terminal resizing for exec, attach, and run. Note that for Docker, exec sessions
inherit the environment from the primary process, so if the container was created with tty=false,
that means the exec session's TERM variable will default to "dumb". Users can override this by
setting TERM=xterm (or whatever is appropriate) to get the correct "smart" terminal behavior.
This allows us to start building real dependencies into Makefile.
Leave old hack/* scripts in place but advise to use 'make'. There are a few
rules that call things like 'go run' or 'build/*' that I left as-is for now.
Automatic merge from submit-queue
node_e2e: configure gce images via config file
This file provides the abiliy to specify image project on a per-image
basis and is more extensible for future changes.
For backwards compatibility and local development convenience, the
existing flags are kept and should work.
The eventual goal is to be able to source some images, such as the CoreOS one (and possibly containervm one) from their upstream projects and do all new configuration changes via a cloud-init key added to the image config.
This PR is a first step there. A following PR will add a config key of `cloud-init` or `user-data` and migrate the CoreOS e2e to use that.
This motivation is driven by the fact that currently the changes needed for the CoreOS image can all be done quickly in cloud-init and this will make it much easier to update the image and ensure that changes are applied consistently.
/cc @timstclair @vishh @yifan-gu @pwittrock
This file provides the abiliy to specify image project on a per-image
basis and is more extensible for future changes.
For backwards compatibility and local development convenience, the
existing flags are kept and should work.
Automatic merge from submit-queue
Node E2E: Disable kubenet for local node e2e test.
After https://github.com/kubernetes/kubernetes/pull/28196, we must manually setup cni and nsenter in local node to run `make test_e2e_node`, which may not be necessary for local development.
I've tried to move cni downloading logic into `BeforeSuite`, however it is still hard to figure out who should install nsenter, manually installed by every developer? in the `setup_host.sh` script? in `BeforeSuite`?
This PR:
* Added a flag to disable kubenet and disabled kubenet in local test.
* Cleaned up the CNI installation logic a bit.
/cc @yujuhong @freehan
[]()
Automatic merge from submit-queue
Fix node e2e issues on selinux enabled systems
It fixes following 3 node e2es:
```
[Fail] [k8s.io] Container Runtime Conformance Test container runtime conformance blackbox test when starting a container that exits [It] it should run with the expected status [Conformance]
/root/upstream-code/gocode/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:114
[Fail] [k8s.io] Kubelet metrics api when querying /stats/summary [It] it should report resource usage through the stats api
/root/upstream-code/gocode/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:158
```
```
[Fail] [k8s.io] Container Runtime Conformance Test container runtime conformance blackbox test when starting a container that exits [It] should report termination message if TerminationMessagePath is set [Conformance]
/root/upstream-code/gocode/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:150
```
@kubernetes/rh-cluster-infra
Automatic merge from submit-queue
Fix node confomance test
Fixes https://github.com/kubernetes/kubernetes/issues/28255, https://github.com/kubernetes/kubernetes/issues/28250, https://github.com/kubernetes/kubernetes/issues/28341.
The main reason of the flake is that in the failed test expects the `PodPhase` to keep `Pending`. It did `Eventually` check and `Consistently` check for 5 seconds. However, the default `PodPhase` is `Pending`, when the check passes, the `PodStatus` could still be in default state.
After that, the test expects the container status to be `Waiting`, which may not be the case, because the default `ContainerStatuses` is empty, and the pod could still be in the default state.
This PR changes the test to ensure `ContainerStatuses` first and then check the `PodPhase` after that.
Mark P1 because the test fails relatively frequently and does block some PRs.
@pwittrock
/cc @liangchenye @ncdc
[]()
Automatic merge from submit-queue
Use slices of items to clean up after tests
Fixes#27582.
We used to maintain a pointer variable for each process to kill after the
tests finish. @lavalamp suggested using a slice instead, which is a much
cleaner solution. This implements @lavalamp's suggestion and also extends
the idea to tracking directories that need to be removed after the tests finish.
This also means that we should no longer check for nil `killCmd`s inside
`func (k *killCmd) Kill() error {...}` (see #27582 and #27589). If a nil
`killCmd` makes it in there, something is bad elsewhere and we want to see
the nil pointer exception immediately.
Mentioning @timstclair and @euank wrt the original issue/PR.
[Fail] [k8s.io] Kubelet metrics api when querying /stats/summary [It] it should report resource usage through the stats api
/root/upstream-code/gocode/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:158
[Fail] [k8s.io] Container Runtime Conformance Test container runtime conformance blackbox test when starting a container that exits [It] should report termination message if TerminationMessagePath is set [Conformance]
/root/upstream-code/gocode/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:150
[Fail] [k8s.io] Container Runtime Conformance Test container runtime conformance blackbox test when starting a container that exits [It] it should run with the expected status [Conformance]
/root/upstream-code/gocode/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:114
Fixes#27582
We used to maintain a pointer variable for each process to kill after the
tests finish. @lavalamp suggested using a slice instead, which is a much
cleaner solution. This implements @lavalamp's suggestion and also extends
the idea to tracking directories that need to be removed after the tests finish.
This also means that we should no longer check for nil `killCmd`s inside
`func (k *killCmd) Kill() error {...}` (see #27582 and #27589). If a nil
`killCmd` makes it in there, something is bad elsewhere and we want to see
the nil pointer exception immediately.
Automatic merge from submit-queue
Remove duplicated nginx image. Use nginx-slim instead
This PR removes the image `gcr.io/google_containers/nginx:1.7.9` and uses `gcr.io/google_containers/nginx-slim:0.7`.
Besides removing the duplication `1.7.9` is 16 months old.
Automatic merge from submit-queue
e2e_node: lower the log verbosity level
The current level is so high that the logs are almost unreadable.
This fixes#27593
[k8s.io] Kubelet metrics api when querying /stats/summary [It] it should report resource usage through the stats api
And the logs show following error:
Jun 21 15:57:13 localhost journal: tee: /test-empty-dir-mnt: Is a directory
Automatic merge from submit-queue
Add image pulling node e2e
Fixes#27007.
Based on #27309, will rebase after #27309 gets merged.
This PR added all tests mentioned in #27007:
* Pull an image from invalid registry;
* Pull an invalid image from gcr;
* Pull an image from gcr;
* Pull an image from docker hub;
* Pull an image needs auth with/without secrets.
For the imagePullSecrets test, I created a new gcloud project "authenticated-image-pulling", and the service account in the code only has "Storage Object Viewer" permission.
/cc @pwittrock @vishh
[]()
Automatic merge from submit-queue
Fix node e2e coreos kubelet cgroup detection
Fixes#26979#26431
The root issue, as best I can tell, is that cgroup detection does not work when the kubelet is started under an ssh session and the systemd `*Accounting` variables are set. I added additional logging and noted some differences in the cgroup slice names between those cadvisor returns and the kubelet detects for itself.
This difference does not occur if the kubelet is properly running under a unit. That environment is also a more common and sane environment.
See also discussion in #26903
cc @derekwaynecarr @vishh @pwittrock
Automatic merge from submit-queue
Port the downward api test to the node e2e suite
Also extend the framework to allow a custom client config loading function, so
that the node e2e suite can reuse the same framework across tests.
This fixes#26609
/cc @timstclair @pwittrock
This reverts commit 2494c77972.
The previous commit, which launches the kubelet under `systemd-run`,
fixes an error in detecting the kubelet's cgroup stats.
Fixes#26979
Also includes other improvements:
- Makefile rule to run tests against remote instance using existing host or image
- Makefile will reuse an instance created from an image if it was not torn down
- Runner starts gce instances in parallel with building source
- Runner uses instance ip instead of hostname so that it doesn't need to resolve
- Runner supports cleaning up files and processes on an instance without stopping / deleting it
- Runner runs tests using `ginkgo` binary to support running tests in parallel
Automatic merge from submit-queue
Mark runtime conformance tests as flaky and not run them in jenkins CI.
For #26809
@pwittrock As discussed offline, marking runtime tests as flaky for now. I'm not sure if those tests are required. Testing docker in every Kubernetes PR is un-necessary.
These tests can be run periodically in a separate CI. AFAIK, these tests don't seem to exercise any kube features.
Automatic merge from submit-queue
Update Node e2e Core OS image to run systemd with CPU & Memory accounting enabled by default
cc @derekwaynecarr
For #26289
Automatic merge from submit-queue
Enable node e2e accounting on systemd
Updated the e2e setup.sh script to enable cpu and memory accounting.
Related to https://github.com/kubernetes/kubernetes/issues/26198
/cc @pwittrock
- Exit non-0 if infrastructure failures happen
- Exit 0 if no infrastructure failures happen regardless of test results
(Jenkins will use junit.xml to determine test results)
Automatic merge from submit-queue
Support per-test-environment ginkgo flags for node e2e tests to facilitate skipping miss behaving tests in PR builder
We had an issue today where some node e2e tests were timing out in the pr builder. We want to be able to skip tests in the pr builder and leave them running in the CI if this happens again.
[]()
Automatic merge from submit-queue
Make Privileged pods node e2e use the framework
Made the test more readable along the way with more logs. This should help us triage failures/flakes in the future.
#24577
Automatic merge from submit-queue
Use pause image depending on the server's platform when testing
Removed all pause image constant strings, now the pause image is chosen by arch. Part of the effort of making e2e arch-agnostic.
The pause image name and version is also now only in two places, and it's documented to bump both
Also removed "amd64" constants in the code. Such constants should be replaced by `runtime.GOARCH` or by looking up the server platform
Fixes: #22876 and #15140
Makes it easier for: #25730
Related: #17981
This is for `v1.3`
@ixdy @thockin @vishh @kubernetes/sig-testing @andyzheng0831 @pensu
- Add junit test reported
- Write etcd.log, kubelet.log and kube-apiserver.log to files instead of stdout
- Scp artifacts to the jenkins WORKSPACE
Fixes#25966
Automatic merge from submit-queue
kubelet/cadvisor: Refactor cadvisor disk stat/usage interfaces.
basically
1) cadvisor struct will know what runtime the kubelet is, passed in via additional argument to New()
2) rename cadvisor wrapper function to DockerImagesFsInfo() to ImagesFsInfo() and have linux implementation choose a label based on the runtime inside the cadvisor struct
2a) mock/fake/unsupported modified to take the same additional argument in New()
3) kubelet's wrapper for the cadvisor wrapper is renamed in parallel
4) make all tests use new interface
Automatic merge from submit-queue
Add a timeout to the node e2e Ginkgo test runner
Also add a few debugging statements to indicate progress.
Should help prevent #25639, since we'll timeout tests before Jenkins times out the build.
Automatic merge from submit-queue
Reimplement 'pause' in C - smaller footprint all around
Statically links against musl. Size of amd64 binary is 3560 bytes.
I couldn't test the arm binary since I have no hardware to test it on, though I assume we want it to work on a raspberry pi.
This PR also adds the gcc5/musl cross compiling image used to build the binaries.
@thockin
Correct the display of the output when e.g. starting the kubelet fails in the node e2e tests. This
change makes it display a string instead of an array of decimal values.
Automatic merge from submit-queue
Framework support for node e2e.
This should let us port existing e2e tests to the node e2e suite, if the tests are node specific.
- Add keep-alive to ssh connection
- Don't try to stop services on image-based runs
- Increase jenkins ci timeout to 90 minutes to accomadate unpredictable go build times
- Remove spammy log statement
Automatic merge from submit-queue
Fix PullImage and add corresponding node e2e test
Fixes#24101. This is a bug introduced by #23506, since ref #23563.
The root cause of #24101 is described [here](https://github.com/kubernetes/kubernetes/issues/24101#issuecomment-208547623).
This PR
1) Fixes#24101 by decoding the messages returned during pulling image, and return error if any of the messages contains error.
2) Add the node e2e test to detect this kind of failure.
3) Get present check out of `ConformanceImage.Remove()` and `ConformanceImage.Pull()`. Because sometimes we may expect error to occur in `PullImage()` and `RemoveImage()`, but even that doesn't happen, the `Present()` check will still return error and let the test pass.
@yujuhong @freehan @liangchenye
Also /cc @resouer, because he is doing the image related functions refactoring.
Automatic merge from submit-queue
e2e_node: port privileged pod tests from test/e2e/priviliged.go
The ported test is functionally the same as the original test.
The main difference between the two tests is that the original test relies on
`kubectl` to exec into the container, while the latter directly uses the REST
client of the apiserver. This avoids the need to copy kubectl to the node under
test.
Automatic merge from submit-queue
Do not throw creation errors for containers that fail immediately after being started
Fixes (hopefully) #23607
cc @dchen1107
The ported test is functionally the same as the original test.
The main difference between the two tests is that the original test relies on
`kubectl` to exec into the container, while the latter directly uses the REST
client of the apiserver. This avoids the need to copy kubectl to the node under
test.
- Add README.md for node e2e tests
- Add support for --cleanup=false to leave test files on remote hosts and temporary instances for debugging
- Add ubuntu trusty instances for docker 1.8 and docker 1.9 to jenkins pr builder
- Disable coreos-beta for jenkins ci since it is failing - need to investigate
- support allocating gce instances from images and running tests against them
- set --hostname-override to match node name
- add jenkins script to source to reproduce jenkins build locally
- Add Makefile targets
- Start services in the test harness and connect locally
- Build test into binary and copy to remote host to start services
- Use tar to copy binaries to remote hosts to simplify design
* Metrics will not be expose until they are hooked up to a handler
* Metrics are not cached and expose a dos vector, this must be fixed before release or the stats should not be exposed through an api endpoint
- Improve documentation and method naming
- Fix command this is run remotely
- Never reschedule the busybox logging test pod since it is supposed to terminate
- Update log test condition retrylogic to correctly retry instead of failing the test
- localhost -> 127.0.0.1 to work on coreos
- give name to etcd to work on coreos
- allow using full hostname for nodename for coreos
Bryan Boreham