k3s/tests/e2e
Brad Davidson 81dda9d626 Add command output to test failure message
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2024-12-06 11:45:34 -08:00
..
amd64_resource_files Add e2e test for advanced fields in services 2024-10-09 12:32:27 +02:00
btrfs Replace deprecated ruby function 2024-05-13 09:41:28 +02:00
cis_amd64_resource_files Add scripts to run e2e test using ansible (#5134) 2022-08-05 09:16:10 -07:00
dualstack Update to newer OS images for install testing (#10681) 2024-08-12 09:26:19 -07:00
embeddedmirror Update to newer OS images for install testing (#10681) 2024-08-12 09:26:19 -07:00
externalip Update to newer OS images for install testing (#10681) 2024-08-12 09:26:19 -07:00
privateregistry Add rewrite test to privateregistry e2e 2024-12-03 11:58:51 -08:00
rootless Update to newer OS images for install testing (#10681) 2024-08-12 09:26:19 -07:00
rotateca Update to newer OS images for install testing (#10681) 2024-08-12 09:26:19 -07:00
s3 Update to newer OS images for install testing (#10681) 2024-08-12 09:26:19 -07:00
scripts Reduce the number of GH api request for E2E nightly (#11148) 2024-10-23 09:36:06 -07:00
secretsencryption Update to newer OS images for install testing (#10681) 2024-08-12 09:26:19 -07:00
secretsencryption_old Update to newer OS images for install testing (#10681) 2024-08-12 09:26:19 -07:00
snapshotrestore If no etcd was deployed, fail etcd-snapshot with a useful error 2024-11-28 09:40:42 +01:00
splitserver Fix secrets-encrypt reencrypt timeout error (#11385) 2024-12-05 09:11:22 -08:00
startup Update to newer OS images for install testing (#10681) 2024-08-12 09:26:19 -07:00
svcpoliciesandfirewall Add e2e test for advanced fields in services 2024-10-09 12:32:27 +02:00
tailscale Update to newer OS images for install testing (#10681) 2024-08-12 09:26:19 -07:00
token Update to newer OS images for install testing (#10681) 2024-08-12 09:26:19 -07:00
upgradecluster Add MariaDB to E2E (#10724) 2024-09-19 08:40:02 -07:00
validatecluster Add command output to test failure message 2024-12-06 11:45:34 -08:00
wasm Update to newer OS images for install testing (#10681) 2024-08-12 09:26:19 -07:00
README.md Add ubuntu 24.04 apt command for e2e test 2024-11-26 11:45:04 -03:00
e2e_test_playbook.yaml Add scripts to run e2e test using ansible (#5134) 2022-08-05 09:16:10 -07:00
testutils.go Fix GenKubeConfigFile, move from "cat" command to "scp" 2024-11-12 15:54:37 -08:00
vagrantdefaults.rb Reduce the number of GH api request for E2E nightly (#11148) 2024-10-23 09:36:06 -07:00

README.md

End-to-End (E2E) Tests

E2E tests cover multi-node K3s configuration and administration: bringup, update, teardown etc. across a wide range of operating systems. E2E tests are run nightly as part of K3s quality assurance (QA).

Framework

End-to-end tests utilize Ginkgo and Gomega like the integration tests, but rely on Vagrant to provide the underlying cluster configuration.

Currently tested operating systems are:

Format

All E2E tests should be placed under tests/e2e/<TEST_NAME>.
All E2E test functions should be named: Test_E2E<TEST_NAME>.
A E2E test consists of two parts:

  1. Vagrantfile: a vagrant file which describes and configures the VMs upon which the cluster and test will run
  2. <TEST_NAME>.go: A go test file which calls vagrant up and controls the actual testing of the cluster

See the validate cluster test as an example.

Setup

To run the E2E tests, you must first install the following:

  • Vagrant
  • Libvirt
  • Vagrant plugins

Vagrant

Download the latest version (currently 2.2.19) of Vagrant from the website. Do not use built-in packages, they often old or do not include the required ruby library extensions necessary to get certain plugins working.

Libvirt

Follow the OS specific guides to install libvirt/qemu on your host:

  • openSUSE
  • ubuntu 20.04
  • ubuntu 22.04:
    sudo apt install ruby-libvirt qemu libvirt-daemon-system libvirt-clients ebtables dnsmasq-base libxslt-dev libxml2-dev libvirt-dev zlib1g-dev ruby-dev libguestfs-tools
    
  • ubuntu 24.04:
    sudo apt install ruby-libvirt qemu-kvm libvirt-daemon-system libvirt-clients ebtables dnsmasq-base libxslt-dev libxml2-dev libvirt-dev zlib1g-dev ruby-dev libguestfs-tools
    
  • debian
  • fedora

Vagrant plugins

Install the necessary vagrant plugins with the following command:

vagrant plugin install vagrant-libvirt vagrant-scp vagrant-k3s vagrant-reload

Kubectl

For linux

   curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
   sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

If it does not work, or you are on a different system, check the official tutorial

Running

Generally, E2E tests are run as a nightly Jenkins job for QA. They can still be run locally but additional setup may be required. By default, all E2E tests are designed with libvirt as the underlying VM provider. Instructions for installing libvirt and its associated vagrant plugin, vagrant-libvirt can be found here. VirtualBox is also supported as a backup VM provider.

Once setup is complete, all E2E tests can be run with:

go test -timeout=15m ./tests/e2e/... -run E2E

Tests can be run individually with:

go test -timeout=15m ./tests/e2e/validatecluster/... -run E2E
#or
go test -timeout=15m ./tests/e2e/... -run E2EClusterValidation

Additionally, to generate junit reporting for the tests, the Ginkgo CLI is used. Installation instructions can be found here.

To run the all E2E tests and generate JUnit testing reports:

ginkgo --junit-report=result.xml ./tests/e2e/...

Note: The go test default timeout is 10 minutes, thus the -timeout flag should be used. The ginkgo default timeout is 1 hour, no timeout flag is needed.

Debugging

In the event of a test failure, the cluster and VMs are retained in their broken state. Startup logs are retained in vagrant.log.
To see a list of nodes: vagrant status
To ssh into a node: vagrant ssh <NODE>
Once you are done/ready to restart the test, use vagrant destroy -f to remove the broken cluster.