a3f40dd8df
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Fixes the races around devicemanager Allocate() and endpoint deletion. There is a race in predicateAdmitHandler Admit() that getNodeAnyWayFunc() could get Node with non-zero deviceplugin resource allocatable for a non-existing endpoint. That race can happen when a device plugin fails, but is more likely when kubelet restarts as with the current registration model, there is a time gap between kubelet restart and device plugin re-registration. During this time window, even though devicemanager could have removed the resource initially during GetCapacity() call, Kubelet may overwrite the device plugin resource capacity/allocatable with the old value when node update from the API server comes in later. This could cause a pod to be started without proper device runtime config set. To solve this problem, introduce endpointStopGracePeriod. When a device plugin fails, don't immediately remove the endpoint but set stopTime in its endpoint. During kubelet restart, create endpoints with stopTime set for any checkpointed registered resource. The endpoint is considered to be in stopGracePeriod if its stoptime is set. This allows us to track what resources should be handled by devicemanager during the time gap. When an endpoint's stopGracePeriod expires, we remove the endpoint and its resource. This allows the resource to be exported through other channels (e.g., by directly updating node status through API server) if there is such use case. Currently endpointStopGracePeriod is set as 5 minutes. Given that an endpoint is no longer immediately removed upon disconnection, mark all its devices unhealthy so that we can signal the resource allocatable change to the scheduler to avoid scheduling more pods to the node. When a device plugin endpoint is in stopGracePeriod, pods requesting the corresponding resource will fail admission handler. Tested: Ran GPUDevicePlugin e2e_node test 100 times and all passed now. **What this PR does / why we need it**: **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes https://github.com/kubernetes/kubernetes/issues/60176 **Special notes for your reviewer**: **Release note**: ```release-note Fixes the races around devicemanager Allocate() and endpoint deletion. ``` |
||
|---|---|---|
| .github | ||
| Godeps | ||
| api | ||
| build | ||
| cluster | ||
| cmd | ||
| docs | ||
| examples | ||
| hack | ||
| logo | ||
| pkg | ||
| plugin | ||
| staging | ||
| test | ||
| third_party | ||
| translations | ||
| vendor | ||
| .bazelrc | ||
| .generated_files | ||
| .gitattributes | ||
| .gitignore | ||
| .kazelcfg.json | ||
| BUILD.bazel | ||
| CHANGELOG-1.2.md | ||
| CHANGELOG-1.3.md | ||
| CHANGELOG-1.4.md | ||
| CHANGELOG-1.5.md | ||
| CHANGELOG-1.6.md | ||
| CHANGELOG-1.7.md | ||
| CHANGELOG-1.8.md | ||
| CHANGELOG-1.9.md | ||
| CHANGELOG-1.10.md | ||
| CHANGELOG.md | ||
| CONTRIBUTING.md | ||
| LICENSE | ||
| Makefile | ||
| Makefile.generated_files | ||
| OWNERS | ||
| OWNERS_ALIASES | ||
| README.md | ||
| SUPPORT.md | ||
| WORKSPACE | ||
| code-of-conduct.md | ||
| labels.yaml | ||
README.md
Kubernetes
Kubernetes is an open source system for managing containerized applications across multiple hosts; providing basic mechanisms for deployment, maintenance, and scaling of applications.
Kubernetes builds upon a decade and a half of experience at Google running production workloads at scale using a system called Borg, combined with best-of-breed ideas and practices from the community.
Kubernetes is hosted by the Cloud Native Computing Foundation (CNCF). If you are a company that wants to help shape the evolution of technologies that are container-packaged, dynamically-scheduled and microservices-oriented, consider joining the CNCF. For details about who's involved and how Kubernetes plays a role, read the CNCF announcement.
To start using Kubernetes
See our documentation on kubernetes.io.
Try our interactive tutorial.
Take a free course on Scalable Microservices with Kubernetes.
To start developing Kubernetes
The community repository hosts all information about building Kubernetes from source, how to contribute code and documentation, who to contact about what, etc.
If you want to build Kubernetes right away there are two options:
You have a working Go environment.
$ go get -d k8s.io/kubernetes
$ cd $GOPATH/src/k8s.io/kubernetes
$ make
You have a working Docker environment.
$ git clone https://github.com/kubernetes/kubernetes
$ cd kubernetes
$ make quick-release
For the full story, head over to the developer's documentation.
Support
If you need support, start with the troubleshooting guide, and work your way through the process that we've outlined.
That said, if you have questions, reach out to us one way or another.