mirror of https://github.com/k3s-io/k3s
1e2105808b
Automatic merge from submit-queue Enable "kick the tires" support for Nvidia GPUs in COS This PR provides an installation daemonset that will install Nvidia CUDA drivers on Google Container Optimized OS (COS). User space libraries and debug utilities from the Nvidia driver installation are made available on the host in a special directory on the host - * `/home/kubernetes/bin/nvidia/lib` for libraries * `/home/kubernetes/bin/nvidia/bin` for debug utilities Containers that run CUDA applications on COS are expected to consume the libraries and debug utilities (if necessary) from the host directories using `HostPath` volumes. Note: This solution requires updating Pod Spec across distros. This is a known issue and will be addressed in the future. Until then CUDA workloads will not be portable. This PR updates the COS base image version to m59. This is coupled with this PR for the following reasons: 1. Driver installation requires disabling a kernel feature in COS. 2. The kernel API for disabling this interface changed across COS versions 3. If the COS image update is not handled in this PR, then a subsequent COS image update will break GPU integration and will require an update to the installation scripts in this PR. 4. Instead of having to post `3` PRs, one each for adding the basic installer, updating COS to m59, and then updating the installer again, this PR combines all the changes to reduce review overhead and latency, and additional noise that will be created when GPU tests break. **Try out this PR** 1. Get Quota for GPUs in any region 2. `export `KUBE_GCE_ZONE=<zone-with-gpus>` KUBE_NODE_OS_DISTRIBUTION=gci` 3. `NODE_ACCELERATORS="type=nvidia-tesla-k80,count=1" cluster/kube-up.sh` 4. `kubectl create -f cluster/gce/gci/nvidia-gpus/cos-installer-daemonset.yaml` 5. Run your CUDA app in a pod. **Another option is to run a e2e manually to try out this PR** 1. Get Quota for GPUs in any region 2. export `KUBE_GCE_ZONE=<zone-with-gpus>` KUBE_NODE_OS_DISTRIBUTION=gci 3. `NODE_ACCELERATORS="type=nvidia-tesla-k80,count=1"` 4. `go run hack/e2e.go -- --up` 5. `hack/ginkgo-e2e.sh --ginkgo.focus="\[Feature:GPU\]"` The e2e will install the drivers automatically using the daemonset and then run test workloads to validate driver integration. TODO: - [x] Update COS image version to m59 release. - [x] Remove sleep from the install script and add it to the daemonset - [x] Add an e2e that will run the daemonset and run a sample CUDA app on COS clusters. - [x] Setup a test project with necessary quota to run GPU tests against HEAD to start with https://github.com/kubernetes/test-infra/pull/2759 - [x] Update node e2e serial configs to install nvidia drivers on COS by default |
||
|---|---|---|
| .. | ||
| boilerplate | ||
| cmd/teststale | ||
| e2e-internal | ||
| gen-swagger-doc | ||
| jenkins | ||
| lib | ||
| make-rules | ||
| testdata | ||
| verify-flags | ||
| .linted_packages | ||
| BUILD | ||
| OWNERS | ||
| autogenerated_placeholder.txt | ||
| benchmark-go.sh | ||
| build-cross.sh | ||
| build-go.sh | ||
| build-ui.sh | ||
| cherry_pick_pull.sh | ||
| dev-build-and-push.sh | ||
| dev-build-and-up.sh | ||
| dev-push-hyperkube.sh | ||
| e2e-node-test.sh | ||
| e2e.go | ||
| e2e_test.go | ||
| federated-ginkgo-e2e.sh | ||
| generate-bindata.sh | ||
| generate-docs.sh | ||
| get-build.sh | ||
| ginkgo-e2e.sh | ||
| godep-restore.sh | ||
| godep-save.sh | ||
| grab-profiles.sh | ||
| install-etcd.sh | ||
| list-feature-tests.sh | ||
| local-up-cluster.sh | ||
| lookup_pull.py | ||
| print-workspace-status.sh | ||
| run-in-gopath.sh | ||
| test-cmd.sh | ||
| test-go.sh | ||
| test-integration.sh | ||
| test-update-storage-objects.sh | ||
| update-all.sh | ||
| update-api-reference-docs.sh | ||
| update-bazel.sh | ||
| update-codecgen.sh | ||
| update-codegen.sh | ||
| update-federation-api-reference-docs.sh | ||
| update-federation-generated-swagger-docs.sh | ||
| update-federation-openapi-spec.sh | ||
| update-federation-swagger-spec.sh | ||
| update-generated-docs.sh | ||
| update-generated-protobuf-dockerized.sh | ||
| update-generated-protobuf.sh | ||
| update-generated-runtime-dockerized.sh | ||
| update-generated-runtime.sh | ||
| update-generated-swagger-docs.sh | ||
| update-godep-licenses.sh | ||
| update-gofmt.sh | ||
| update-openapi-spec.sh | ||
| update-staging-client-go.sh | ||
| update-staging-godeps.sh | ||
| update-swagger-spec.sh | ||
| update-translations.sh | ||
| update_owners.py | ||
| verify-all.sh | ||
| verify-api-groups.sh | ||
| verify-api-reference-docs.sh | ||
| verify-bazel.sh | ||
| verify-boilerplate.sh | ||
| verify-cli-conventions.sh | ||
| verify-codecgen.sh | ||
| verify-codegen.sh | ||
| verify-description.sh | ||
| verify-federation-api-reference-docs.sh | ||
| verify-federation-generated-swagger-docs.sh | ||
| verify-federation-openapi-spec.sh | ||
| verify-federation-swagger-spec.sh | ||
| verify-flags-underscore.py | ||
| verify-generated-docs.sh | ||
| verify-generated-protobuf.sh | ||
| verify-generated-runtime.sh | ||
| verify-generated-swagger-docs.sh | ||
| verify-godep-licenses.sh | ||
| verify-godeps.sh | ||
| verify-gofmt.sh | ||
| verify-golint.sh | ||
| verify-govet.sh | ||
| verify-import-boss.sh | ||
| verify-linkcheck.sh | ||
| verify-no-vendor-cycles.sh | ||
| verify-openapi-spec.sh | ||
| verify-pkg-names.sh | ||
| verify-readonly-packages.sh | ||
| verify-staging-client-go.sh | ||
| verify-staging-godeps.sh | ||
| verify-staging-imports.sh | ||
| verify-swagger-spec.sh | ||
| verify-symbols.sh | ||
| verify-test-images.sh | ||
| verify-test-owners.sh | ||