From e87204c064c8ecde594d8d3bc7fb3409cac9c0a4 Mon Sep 17 00:00:00 2001 From: Derek Nola Date: Thu, 12 Aug 2021 12:42:31 -0700 Subject: [PATCH] Added new testing documentation (#3823) * Added new testing documentation Signed-off-by: Derek Nola --- CONTRIBUTING.md | 2 + tests/TESTING.md | 103 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 105 insertions(+) create mode 100644 tests/TESTING.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index adb137e78f..49294ec02a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -10,6 +10,8 @@ If you're interested in contributing documentation, please note the following: - Doc issues are raised in this repository, and they are tracked under the `kind/documentation` label. - Pull requests are submitted to the K3s documentation source in the [Rancher docs repository.](https://github.com/rancher/docs/) The K3s docs content is in the `content/k3s/` directory. +If you're interested in contributing new tests, please see the `TESTING.md` in the tests directory. + ## Pull request and git commit guidance ### Opening PRs and organizing commits diff --git a/tests/TESTING.md b/tests/TESTING.md new file mode 100644 index 0000000000..892eed036f --- /dev/null +++ b/tests/TESTING.md @@ -0,0 +1,103 @@ +# Testing Standards in K3s + +Go testing in K3s comes in 3 forms: Unit, Integration, and End-to-End (E2E). This +document will explain *when* each test should be written and *how* each test should be +generated, formatted, and run. + +Note: all shell commands given are relateive to the root k3s repo directory. +___ + +## Unit Tests + +Unit tests should be written when a component or function of a package needs testing. +Unit tests should be used for "white box" testing. + +### Framework + +All unit tests in K3s follow a [Table Driven Test](https://github.com/golang/go/wiki/TableDrivenTests) style. Specifically, K3s unit tests are automatically generated using the [gotests](https://github.com/cweill/gotests) tool. This is built into the Go vscode extension, has documented integrations for other popular editors, or can be run via command line. Additionally, a set of custom templates are provided to extend the generated test's functionality. To use these templates, call: + +```bash +gotests --template_dir=/contrib/gotests_templates +``` + +Or in vscode, edit the Go extension setting `Go: Generate Tests Flags` +and add `--template_dir=/contrib/gotests_templates` as an item. + +To facilitate unit test creation, see `tests/util/runtime.go` helper functions. + +### Format + +All unit tests should be placed within the package of the file they test. +All unit test files should be named: `_test.go`. +All unit test functions should be named: `Test_Unit` or `Test_Unit_`. +See the [etcd unit test](https://github.com/k3s-io/k3s/blob/master/pkg/etcd/etcd_test.go) as an example. + +### Running + +```bash +go test ./pkg/... -run Unit +``` + +Note: As unit tests call functions directly, they are the primary drivers of K3s's code coverage +metric. + +___ + +## Integration Tests + +Integration tests should be used to test a specific functionality of k3s that exists across multiple Go packages, either via exported function calls, or more often, CLI comands. +Integration tests should be used for "black box" testing. + +### Framework + +All integration tests in K3s follow a [Behavior Diven Development (BDD)](https://en.wikipedia.org/wiki/Behavior-driven_development) style. Specifically, K3s uses [Ginkgo](https://onsi.github.io/ginkgo/) and [Gomega](https://onsi.github.io/gomega/) to drive the tests. +To generate an initial test, the command `ginkgo bootstrap` can be used. + +To facilitate K3s CLI testing, see `tests/util/cmd.go` helper functions. + +### Format + +Integration tests can be placed in two areas: + +1. Next to the go package they intend to test. +2. In `tests/integration/` for package agnostic testing. + +Package specific integration tests should use the `_test` package. +Package agnostic integration tests should use the `integration` package. +All integration test files should be named: `_int_test.go` +All integration test functions should be named: `Test_Integration`. +See the [etcd snapshot test](https://github.com/k3s-io/k3s/blob/master/pkg/etcd/etcd_int_test.go) as a package specific example. +See the [local storage test](https://github.com/k3s-io/k3s/blob/master/tests/integration/localstorage_int_test.go) as a package agnostic example. + +### Running + +```bash +go test ./pkg/... ./tests/... -run Integration +``` + +___ + +## End-to-End (E2E) Tests + +End-to-end tests utilize [Ginkgo](https://onsi.github.io/ginkgo/) and [Gomega](https://onsi.github.io/gomega/) like the integration tests, but rely on separate testing utilities and are self-contained within the `test/e2e` directory. E2E tests cover complete K3s single and multi-cluster configuration and administration: bringup, update, teardown etc. +E2E tests are run nightly as part of K3s quality assurance (QA). + +### Format + +All E2E tests should be placed under the `e2e` package. +All E2E test functions should be named: `Test_E2E`. +See the [upgrade cluster test](https://github.com/k3s-io/k3s/blob/master/tests/e2e/upgradecluster_test.go) as an example. + +### Running + +Generally, E2E tests are run as a nightly Jenkins job for QA. They can still be run locally but additional setup may be required. + +```bash +go test ./tests/... -run E2E +``` + +## Contributing New Or Updated Tests + +___ +We gladly accept new and updated tests of all types. If you wish to create +a new test or update an existing test, please submit a PR with a title that includes the words ` (Created/Updated)`.