Automatic merge from submit-queue
Make GCI nodes mount non tmpfs, ext* & bind mounts using an external mounter
This PR downloads the stage1 & gci-mounter ACIs as part of cluster bring up instead of downloading them dynamically from gcr.io, which was the cause for #36206.
I have also optimized the containerized mounter to pre-load the mounter image once to avoid fetch latency while using it.
Original PR which got reverted: https://github.com/kubernetes/kubernetes/pull/35821
```release-note
GCI nodes use an external mounter script to mount NFS & GlusterFS storage volumes
```
@mtaufen Node e2e is not re-enabled in this PR.
cc @jingxu97
Automatic merge from submit-queue
Update rkt version on GCI nodes to v1.18.0
v1.18.0 avoids outputting debug information by default which happens to
pollute events and kubelet logs.
Automatic merge from submit-queue
Run flaky tests in parallel
We should try to emulate the main CI environment in the flaky test suite so that it is clear when a test can be moved out of the flaky suite. Since a common source of flakes is unintended interactions between tests running in parallel, we should run the flaky suite in parallel to better detect such flakes.
Automatic merge from submit-queue
Update GCI_VERSION to gci-dev-55-8866-0-0
Update GCI base image:
Change log:
* Built-in kubernetes updated to v1.4.0
* Enabled VXLAN and IP_SET config options in kernel to support some networking tools
* OpenSSL CVE fixes
```release-note
Update GCI base image:
* Enabled VXLAN and IP_SET config options in kernel to support some networking tools (ebtools)
* OpenSSL CVE fixes
```
cc/ @kubernetes/goog-image cc/ @dchen1107
Changelog:
* Built-in kubernetes updated to v1.4.0
* Enabled VXLAN and IP_SET config options in kernel to support some networking tools
* OpenSSL CVE fixes
Automatic merge from submit-queue
CRI: Add serial and benchmark test suite.
For https://github.com/kubernetes/kubernetes/issues/31459.
The serial test result will be shown on test-grid.
The benchmark test result will be shown [node-perf-dash](http://node-perf-dash.k8s.io/#/builds)
This PR also changes the cri validation test to use the same gci image with node e2e instead of the canary image. The docker version is still 1.11.2.
@yujuhong @feiskyer @yifan-gu
/cc @kubernetes/sig-node
Automatic merge from submit-queue
Update the containervm image to the latest one (container-v1-3-v20160…
Node e2e is running with old containervm image which only has docker 1.9.1. This pr fixed such issue.
Automatic merge from submit-queue
Fix the properties file for node e2e cri validation.
I fixed this locally before, but accidentally missed in the PR. Sorry about that.
This time, I've tried myself, it should work.
@yujuhong
Automatic merge from submit-queue
Bump up GCI version.
```release-note
Upgrading Container-VM base image for k8s on GCE. Brief changelog as follows:
- Fixed performance regression in veth device driver
- Docker and related binaries are statically linked
- Fixed the issue of systemd being oom-killable
```
Fixes#32596
This needs a cherrypick into v1.4 release branch because it is fixing v1.4 release blocking issues. This patch is easy and safe to rollback in case of emergencies.
@vishh can you please review?
Fixes#32596 and many other issues.
cc/ @kubernetes/goog-image FYI
Brief changelog compared to gci-dev-54-8743-3-0:
- Fixed performance regression in veth device driver
- Docker and related binaries are statically linked
- Fixed the issue of systemd being oom-killable
- Updated built-in kubelet version to 1.3.7
- add ethtool and ebtables binaries expected by kubelet
Fixes#32596
Automatic merge from submit-queue
Add node e2e density test using 60 QPS for benchmark
This PR adds a new benchmark node e2e density test which sets Kubelet API QPS limit from default 5 to 60, through ConfigMap.
The latency caused by API QPS limit is as large as ~30% when creating a large batch of pods (e.g. 105). It makes the pod startup latency, as well creation throughput underestimated. This test helps us to know the real performance of Kubelet core.
Automatic merge from submit-queue
Automated Docker Validation: Change wrong name in perf config.
The config key `containervm-density*` is improper, remove it.
/cc @coufon
Automatic merge from submit-queue
Plumb --feature-gates from TEST_ARGS to components in node e2e tests
This means you can set `TEST_ARGS` on the command line, in a `.properties` config for a Jenkins job, etc, to toggle gated features. For example:
`TEST_ARGS='--feature-gates=DynamicKubeletConfig=true'`
/cc @vishh @jlowdermilk
Prior to this change, a K8s branch (master as well as release) was
pinned to a GCI milestone. It would pick up the latest GCI release on
that milestone at the time of cluster creation. The rationale was the
K8s users would automatically get the bug fixes in newer versions of
GCI. However in practice, it makes the runtime environment
non-deterministic, and lack of continuous e2e tests mean we would run
into breakages sooner or later.
With this change, each K8s release will pick a specific version
of GCI by default (similar to how the Debian-based container-vm gets used).
Users can override the default version through KUBE_GCE_MASTER_IMAGE and
KUBE_GCE_NODE_IMAGE environment variables.
We expect the default GCI version will be updated relatively frequently stay
updated with newer GCI releases. We can also automate the process to
automatically bump the hard-coded GCI version in future.
This commit enables the dynamic kubelet configuration feature for the
node e2e Jenkins serial tests, which is where the test for dynamic kubelet
configuration currently runs.
Automatic merge from submit-queue
test/node-e2e: Update CoreOS update disabling
Previously in this saga... #25004
This disables update-engine and locksmithd with ignition instead of
cloud-init so that they're really totally 100% disabled. Our ignition guy promises.
Pretty much every way of disabling them with cloud-init is mildly racy.
Fixes#31633
I think @vishh can say "I told you so" after the comment on https://github.com/kubernetes/kubernetes/pull/30023#discussion-diff-73431324 .. he was right, but it turns out "stop" there doesn't really work either because of the mess that is cloud-init. Fortunately, converting our cloud-init to json and calling it "ignition" works quite well 😄
Testing done: I ssh'd in and verified that yes, they're disabled. I didn't wait on the e2e tests to pass, so we'll let this PR check that.
This disables update-engine and locksmithd with ignition instead of
cloud-init so that they're really totally 100% disabled.
Pretty much every way of disabling them with cloud-init is mildly racy.
Fixes#31633
Automatic merge from submit-queue
Node Conformance Test: Refactor node e2e framework
For #30122, #30174.
Based on #30348.
**Please only review the last 3 commits.**
This PR is part of our roadmap to package node conformance test.
The 1st commit is from #30348, it removed unnecessary dependencies in the node e2e test framework, because we've statically linked these dependencies.
The PR refactored the node e2e framework. Moving different utilities into different packages under `pkg/`.
We need to do this because:
1) Files like e2e_remote.go and e2e_build.go should only be used by runner, but they were compiled into the test suite because they were placed in the same package. The worst thing is that it will introduce some never used flags in the test suite binary.
2) Make the directory structure more clear. Only test should be placed in `test/e2e_node`, other utilities should be placed in different packages in `pkg/`.
@dchen1107 @vishh
/cc @kubernetes/sig-node @kubernetes/sig-testing
Automatic merge from submit-queue
add throughput in perf data and disable --cgroups-per-qos
This PR adds throughput data to printed perf data for benchmark. It also disables --cgrous-per-qos in jenkinds-benchmark.properties.
Automatic merge from submit-queue
Node Conformance Test: Remove unnecessary binary copy
For #30122, #30174.
This PR removed unnecessary dependencies in the node e2e test framework, because we've statically linked these dependencies.
@dchen1107 @vishh
/cc @kubernetes/sig-node @kubernetes/sig-testing
Automatic merge from submit-queue
skip benchmark in jenkins serial test
This PR changes jenkins-serial.properties to skip benchmark tests (with tag [Benchmark]) in jenkins serial tests. It also add more comments in run_e2e.go.
Automatic merge from submit-queue
Add benchmark to jenkins
This PR contains the following changes:
1. Add more tests in density benchmark test;
2. Add the peak value (100%) in latency and CPU usage statistic data;
3. Move the Ginkgo focus flag from e2e_remote.go to run_e2e.go;
4. Support running benchmark in run_e2e.go. The benchmark configuration file is an extension of image configuration. Each item requires additional GCE machine type (e.g. n1-standard-1, default value will be used if empty) and test names (Ginkgo focus regex strings). A test item is regarded as benchmark if the tests field is non-empty.
Automatic merge from submit-queue
Unblock iterative development on pod-level cgroups
In order to allow forward progress on this feature, it takes the commits from #28017#29049 and then it globally disables the flag that allows these features to be exercised in the kubelet. The flag can be re-added to the kubelet when its actually ready.
/cc @vishh @dubstack @kubernetes/rh-cluster-infra
It can run tests against multiple existing images that match a regex.
GCI images will be using a regex.
Signed-off-by: Vishnu kannan <vishnuk@google.com>
Automatic merge from submit-queue
node_e2e: Use upstream CoreOS image directly
.. and update it to the latest alpha
This will make updating the CoreOS image in the future much simpler since it won't involve project-copying, manual-baking, or so on.
cc @pwittrock @vishh @bboreham @yifan-gu
Automatic merge from submit-queue
Node E2E: Add serial jenkins job.
This PR added a jenkins job for serial test. It will run all serial test one by one.
This will be useful for https://github.com/kubernetes/kubernetes/pull/29809.
@coufon @yujuhong @dchen1107
/cc @kubernetes/sig-node
Automatic merge from submit-queue
Node E2E: Make node e2e parallel
For https://github.com/kubernetes/kubernetes/issues/29081.
Fix https://github.com/kubernetes/kubernetes/issues/26215.
Based on https://github.com/kubernetes/kubernetes/pull/28807, https://github.com/kubernetes/kubernetes/pull/29020, will rebase after they are merged.
**Only the last commit is new.**
We are going to move more tests into the node e2e test. However, currently node e2e test only run sequentially, the test duration will increase quickly when we add more test.
This PR makes the node e2e test run in parallel so as to shorten test duration, so that we can add more test to improve the test coverage.
* If you run the test locally with `make test-e2e-node`, it will use `-p` ginkgo flag, which uses `(cores-1)` parallel test nodes by default.
* If you run the test remotely or in the Jenkin, the parallelism will be controlled by the environment variable `PARALLELISM`. The default value is `8`, which is reasonable for our test node (n1-standard-1).
Before this PR, it took **833.592s** to run all test on my desktop.
With this PR, it only takes **234.058s** to run.
The pull request node e2e run with this PR takes **232.327s**.
The pull request node e2e run for other PRs takes **673.810s**.
/cc @kubernetes/sig-node
Automatic merge from submit-queue
Adding GCI to node e2e.
Depends on https://github.com/kubernetes/kubernetes/pull/29486
Adding the dev release as of now since stable and beta run docker v1.9.1
which is incompatible with kubelet.
bindata and yaml, Gobindata automation
bindata utils for generating, go generate
match server version
gitignore for dirty, ca, rbase, KUBE_ROOT, buildfix
(rebased jul-25,29)
Automatic merge from submit-queue
Change the docker validation node e2e test to use gci-canary-test
This PR changed the continuous docker validation node e2e test to use the image config file introduced in https://github.com/kubernetes/kubernetes/pull/28708. @euank
This PR also changed the gci image family from `gci-preview-test` to `gci-canary-test`. @wonderfly
Automatic merge from submit-queue
node_e2e: configure gce images via config file
This file provides the abiliy to specify image project on a per-image
basis and is more extensible for future changes.
For backwards compatibility and local development convenience, the
existing flags are kept and should work.
The eventual goal is to be able to source some images, such as the CoreOS one (and possibly containervm one) from their upstream projects and do all new configuration changes via a cloud-init key added to the image config.
This PR is a first step there. A following PR will add a config key of `cloud-init` or `user-data` and migrate the CoreOS e2e to use that.
This motivation is driven by the fact that currently the changes needed for the CoreOS image can all be done quickly in cloud-init and this will make it much easier to update the image and ensure that changes are applied consistently.
/cc @timstclair @vishh @yifan-gu @pwittrock
This file provides the abiliy to specify image project on a per-image
basis and is more extensible for future changes.
For backwards compatibility and local development convenience, the
existing flags are kept and should work.
This reverts commit 2494c77972.
The previous commit, which launches the kubelet under `systemd-run`,
fixes an error in detecting the kubelet's cgroup stats.
Fixes#26979
Also includes other improvements:
- Makefile rule to run tests against remote instance using existing host or image
- Makefile will reuse an instance created from an image if it was not torn down
- Runner starts gce instances in parallel with building source
- Runner uses instance ip instead of hostname so that it doesn't need to resolve
- Runner supports cleaning up files and processes on an instance without stopping / deleting it
- Runner runs tests using `ginkgo` binary to support running tests in parallel
Automatic merge from submit-queue
Mark runtime conformance tests as flaky and not run them in jenkins CI.
For #26809
@pwittrock As discussed offline, marking runtime tests as flaky for now. I'm not sure if those tests are required. Testing docker in every Kubernetes PR is un-necessary.
These tests can be run periodically in a separate CI. AFAIK, these tests don't seem to exercise any kube features.
- Add junit test reported
- Write etcd.log, kubelet.log and kube-apiserver.log to files instead of stdout
- Scp artifacts to the jenkins WORKSPACE
Fixes#25966
- Add README.md for node e2e tests
- Add support for --cleanup=false to leave test files on remote hosts and temporary instances for debugging
- Add ubuntu trusty instances for docker 1.8 and docker 1.9 to jenkins pr builder
- Disable coreos-beta for jenkins ci since it is failing - need to investigate
- support allocating gce instances from images and running tests against them
- set --hostname-override to match node name
- add jenkins script to source to reproduce jenkins build locally
Yu-Ju Hong
