mirror of https://github.com/k3s-io/k3s
Merge pull request #23353 from spiffxp/conformance-docs
Automatic merge from submit-queue Update conformance test policy Addresses #14913 /cc @kubernetes/sig-testingpull/6/head
k8s-merge-robot
commit
be8ce6c385
|
|
@ -1,30 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
# Copyright 2015 The Kubernetes Authors All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Conformance Tests a running Kubernetes cluster.
|
||||
# Validates that the cluster was deployed, is accessible, and at least
|
||||
# satisfies end-to-end tests marked as being a part of the conformance suite.
|
||||
# Emphasis on broad coverage and being non-destructive over thoroughness.
|
||||
|
||||
set -o errexit
|
||||
set -o nounset
|
||||
set -o pipefail
|
||||
|
||||
KUBE_ROOT=$(dirname "${BASH_SOURCE}")/..
|
||||
|
||||
TEST_ARGS="$@"
|
||||
|
||||
exec "${KUBE_ROOT}/hack/conformance-test.sh" ${TEST_ARGS}
|
||||
|
|
@ -31,7 +31,7 @@ TEST_ARGS="$@"
|
|||
|
||||
KUBE_ROOT=$(cd "$(dirname "${BASH_SOURCE}")/../../.." && pwd)
|
||||
|
||||
TEST_CMD="KUBECONFIG=~/.kube/config hack/conformance-test.sh"
|
||||
TEST_CMD="KUBERNETES_CONFORMANCE_TEST=y KUBECONFIG=~/.kube/config go run hack/e2e.go --test --test_args=\"--ginkgo.focus=\\[Conformance\\]\""
|
||||
if [ -n "${CONFORMANCE_BRANCH}" ]; then
|
||||
# create a CONFORMANCE_BRANCH clone in a subdirectory
|
||||
TEST_CMD="
|
||||
|
|
|
|||
|
|
@ -77,12 +77,15 @@ go run hack/e2e.go -v --pushup
|
|||
# Run all tests
|
||||
go run hack/e2e.go -v --test
|
||||
|
||||
# Run tests matching the regex "\[Conformance\]" (the conformance tests)
|
||||
go run hack/e2e.go -v -test --test_args="--ginkgo.focus=\[Conformance\]"
|
||||
# Run tests matching the regex "\[Feature:Performance\]"
|
||||
go run hack/e2e.go -v -test --test_args="--ginkgo.focus=\[Feature:Performance\]"
|
||||
|
||||
# Conversely, exclude tests that match the regex "Pods.*env"
|
||||
go run hack/e2e.go -v -test --test_args="--ginkgo.focus=Pods.*env"
|
||||
|
||||
# Run tests in parallel, skip any that must be run serially
|
||||
GINKGO_PARALLEL=y go run hack/e2e.go --v --test --test_args="--ginkgo.skip=\[Serial\]"
|
||||
|
||||
# Flags can be combined, and their actions will take place in this order:
|
||||
# --build, --push|--up|--pushup, --test|--tests=..., --down
|
||||
#
|
||||
|
|
@ -96,9 +99,6 @@ KUBERNETES_PROVIDER=aws go run hack/e2e.go -v --build --pushup --test --down
|
|||
# kubectl output.
|
||||
go run hack/e2e.go -v -ctl='get events'
|
||||
go run hack/e2e.go -v -ctl='delete pod foobar'
|
||||
|
||||
# Alternately, if you have the e2e cluster up and no desire to see the event stream, you can run ginkgo-e2e.sh directly:
|
||||
hack/ginkgo-e2e.sh --ginkgo.focus=\[Conformance\]
|
||||
```
|
||||
|
||||
The tests are built into a single binary which can be run used to deploy a Kubernetes system or run tests against an already-deployed Kubernetes system. See `go run hack/e2e.go --help` (or the flag definitions in `hack/e2e.go`) for more options, such as reusing an existing cluster.
|
||||
|
|
@ -208,13 +208,31 @@ We are working on implementing clearer partitioning of our e2e tests to make run
|
|||
|
||||
### Conformance tests
|
||||
|
||||
Finally, `[Conformance]` tests are tests we expect to pass on **any** Kubernetes cluster. The `[Conformance]` label does not supersede any other labels. `[Conformance]` test policies are a work-in-progress (see #18162).
|
||||
Finally, `[Conformance]` tests represent a subset of the e2e-tests we expect to pass on **any** Kubernetes cluster. The `[Conformance]` label does not supersede any other labels.
|
||||
|
||||
End-to-end testing, as described above, is for [development distributions](writing-a-getting-started-guide.md). A conformance test is used on a [versioned distro](writing-a-getting-started-guide.md). (Links WIP)
|
||||
As each new release of Kubernetes providers new functionality, the subset of tests necessary to demonstrate conformance grows with each release. Conformance is thus considered versioned, with the same backwards compatibility guarantees as laid out in [our versioning policy](../design/versioning.md#supported-releases). Conformance tests for a given version should be run off of the release branch that corresponds to that version. Thus `v1.2` conformance tests would be run from the head of the `release-1.2` branch. eg:
|
||||
|
||||
The conformance test runs a subset of the e2e-tests against a manually-created cluster. It does not require support for up/push/down and other operations. To run a conformance test, you need to know the IP of the master for your cluster and the authorization arguments to use. The conformance test is intended to run against a cluster at a specific binary release of Kubernetes. See [conformance-test.sh](http://releases.k8s.io/HEAD/hack/conformance-test.sh).
|
||||
- A v1.3 development cluster should pass v1.1, v1.2 conformance tests
|
||||
- A v1.2 cluster should pass v1.1, v1.2 conformance tests
|
||||
- A v1.1 cluster should pass v1.0, v1.1 conformance tests, and fail v1.2 conformance tests
|
||||
|
||||
### Defining what Conformance means
|
||||
Conformance tests are designed to be run with no cloud provider configured. Conformance tests can be run against clusters that have not been created with `hack/e2e.go`, just provide a kubeconfig with the appropriate endpoint and credentials.
|
||||
|
||||
```sh
|
||||
# setup for conformance tests
|
||||
export KUBECONFIG=/path/to/kubeconfig
|
||||
export KUBERNETES_CONFORMANCE_TEST=y
|
||||
|
||||
# run all conformance tests
|
||||
go run hack/e2e.go -v --test_args="--ginkgo.focus=\[Conformance\]"
|
||||
|
||||
# run all parallel-safe conformance tests in parallel
|
||||
GINKGO_PARALLEL=y go run hack/e2e.go --v --test --test_args="--ginkgo.focus=\[Conformance\] --ginkgo.skip=\[Serial\]"
|
||||
# ... and finish up with remaining tests in serial
|
||||
go run hack/e2e.go --v --test --test_args="--ginkgo.focus=\[Serial\].*\[Conformance\]"
|
||||
```
|
||||
|
||||
### Defining Conformance Subset
|
||||
|
||||
It is impossible to define the entire space of Conformance tests without knowing the future, so instead, we define the compliment of conformance tests, below.
|
||||
|
||||
|
|
|
|||
|
|
@ -174,55 +174,7 @@ hack/test-integration.sh # Run all integration tests.
|
|||
|
||||
## End-to-End tests
|
||||
|
||||
* e2e tests build kubernetes and deploy a cluster of nodes.
|
||||
- Generally on a specific cloud provider.
|
||||
* Access gcr.io images
|
||||
* Access a specific, non-latest image tag (unless testing pulling).
|
||||
* Tests may not flake due to intermittent issues.
|
||||
* Use ginko to desribe steps.
|
||||
- See [should run a job to completion when tasks succeed](../../test/e2e/job.go)
|
||||
* Use [NewDefaultFramework](../../test/e2e/framework.go)
|
||||
- Contains clients, namespace and auto resource cleanup
|
||||
* See [coding conventions](coding-conventions.md).
|
||||
|
||||
### e2e test philosophy
|
||||
|
||||
In general passing unit and integration tests should provide sufficient
|
||||
confidence to allow code to merge. If that is not the case,
|
||||
please *invest more time adding unit and integration test coverage*.
|
||||
These tests run faster and have a smaller failure domain.
|
||||
|
||||
However, end-to-end (e2e) tests provide maximum confidence that
|
||||
the system is working in exchange for reduced performance and a
|
||||
higher debugging cost.
|
||||
|
||||
e2e tests deploy a real kubernetes cluster of real nodes on a concrete provider
|
||||
such as GCE. The tests then manipulate the cluster in certain ways and
|
||||
assert the expected results.
|
||||
|
||||
For a more in depth discussion please read [End-to-End Testing in Kubernetes](e2e-tests.md).
|
||||
|
||||
### Running e2e tests
|
||||
|
||||
```sh
|
||||
cd kubernetes
|
||||
go run hack/e2e.go -v --build --up --test --down
|
||||
|
||||
# Change code, run unit and integration tests
|
||||
# Push to an existing cluster, or bring up a cluster if it's down.
|
||||
go run hack/e2e.go -v --pushup
|
||||
|
||||
# Run all tests on an already up cluster
|
||||
go run hack/e2e.go -v --test
|
||||
|
||||
# Run only conformance tests
|
||||
go run hack/e2e.go -v -test --test_args="--ginkgo.focus=\[Conformance\]"
|
||||
|
||||
# Run tests on a specific provider
|
||||
KUBERNETES_PROVIDER=aws go run hack/e2e.go --build --pushup --test --down
|
||||
```
|
||||
|
||||
For a more in depth discussion please read [End-to-End Testing in Kubernetes](e2e-tests.md).
|
||||
Please refer to [End-to-End Testing in Kubernetes](e2e-tests.md).
|
||||
|
||||
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
|
||||
[]()
|
||||
|
|
|
|||
|
|
@ -68,7 +68,7 @@ These guidelines say *what* to do. See the Rationale section for *why*.
|
|||
own repo.
|
||||
- Add or update a row in [The Matrix](../../docs/getting-started-guides/README.md).
|
||||
- State the binary version of Kubernetes that you tested clearly in your Guide doc.
|
||||
- Setup a cluster and run the [conformance test](development.md#conformance-testing) against it, and report the
|
||||
- Setup a cluster and run the [conformance tests](e2e-tests.md#conformance-tests) against it, and report the
|
||||
results in your PR.
|
||||
- Versioned distros should typically not modify or add code in `cluster/`. That is just scripts for developer
|
||||
distros.
|
||||
|
|
|
|||
|
|
@ -1,87 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
# Copyright 2015 The Kubernetes Authors All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# When run as described below, the conformance test tests whether a cluster
|
||||
# supports key features for Kubernetes version 1.0.
|
||||
|
||||
# Instructions:
|
||||
# - Setup a Kubernetes cluster with $NUM_NODES nodes (defined below).
|
||||
# - Provide a Kubeconfig file whose current context is set to the
|
||||
# cluster to be tested, and with suitable auth setting.
|
||||
# - Specify the location of that kubeconfig with, e.g.:
|
||||
# declare -x KUBECONFIG="$HOME/.kube/config"
|
||||
# - Make sure only essential pods are running and there are no failed/pending pods.
|
||||
# - Go to a git tree that contains the kubernetes source.
|
||||
# - git clone git://github.com/kubernetes/kubernetes.git
|
||||
# - Checkout the upstream/conformance-test-v1 branch
|
||||
# - git checkout upstream/conformance-test-v1
|
||||
# - The working tree will be in a "detached HEAD" state.
|
||||
# - Make binaries needed by e2e
|
||||
# make clean
|
||||
# make quick-release
|
||||
# - Run the test and capture output:
|
||||
# hack/conformance-test.sh 2>&1 | tee conformance.$(date +%FT%T%z).log
|
||||
#
|
||||
|
||||
# About the conformance test:
|
||||
# The conformance test checks whether a kubernetes cluster supports
|
||||
# a minimum set of features to be called "Kubernetes". It is similar
|
||||
# to `hack/e2e.go` but it differs in that:
|
||||
# - hack/e2e.go is intended to test a cluster with binaries built at HEAD,
|
||||
# while this conformance test does not care what version the binaries are.
|
||||
# - this means the user needs to setup a cluster first.
|
||||
# - this means the user does not need to write any cluster/... scripts. Custom
|
||||
# clusters can be tested.
|
||||
# - hack/e2e.go is intended to run e2e tests built at HEAD, while
|
||||
# this conformance test is intended to be run e2e tests built at a particular
|
||||
# version. This ensures that all conformance testees run the same set of tests,
|
||||
# regardless of when they test for conformance.
|
||||
# - it excludes certain e2e tests:
|
||||
# - tests that are specific to certain cloud providers
|
||||
# - tests of optional features, such as volume types.
|
||||
# - tests of performance, scale, or reliability
|
||||
# - known flaky tests.
|
||||
# TODO: when preparing to release a new major or minor version of Kubernetes,
|
||||
# create a new conformance-test-vX.Y branch, update mentions of that branch in this file,
|
||||
# reevaluate the set of e2e tests,
|
||||
# update documentation at docs/getting-started-guides/README.md to have
|
||||
# a new column for conformance at that new version, and notify
|
||||
# community.
|
||||
|
||||
TEST_ARGS="$@"
|
||||
|
||||
: ${KUBECONFIG:?"Must set KUBECONFIG before running conformance test."}
|
||||
echo "Conformance test using current-context of ${KUBECONFIG}"
|
||||
echo -n "Conformance test run date:"
|
||||
date
|
||||
echo -n "Conformance test SHA:"
|
||||
HEAD_SHA=$(git rev-parse HEAD)
|
||||
echo $HEAD_SHA
|
||||
echo "Conformance test version tag(s):"
|
||||
git tag --points-at $HEAD_SHA
|
||||
echo
|
||||
echo "Conformance test checking conformance with Kubernetes version 1.0"
|
||||
|
||||
# It runs a whitelist of tests: all tests which are flagged with [Conformance]
|
||||
# somewhere in the description (i.e. either in the Describe part or the It part).
|
||||
# The list of tagged conformance tests can be retrieved by:
|
||||
#
|
||||
# NUM_NODES=4 KUBERNETES_CONFORMANCE_TEST="y" \
|
||||
# hack/ginkgo-e2e.sh -ginkgo.focus='\[Conformance\]' -ginkgo.dryRun=true
|
||||
|
||||
declare -x KUBERNETES_CONFORMANCE_TEST="y"
|
||||
declare -x NUM_NODES=4
|
||||
exec hack/ginkgo-e2e.sh -ginkgo.focus='\[Conformance\]' ${TEST_ARGS}
|
||||
Loading…
Reference in New Issue