Automatic merge from submit-queue (batch tested with PRs 55103, 56036, 56186). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Removed opaque integer resources (deprecated in v1.8)
**What this PR does / why we need it**:
* Remove opaque integer resources (OIR) support from the code base. This feature was deprecated in v1.8 and replaced by Extended Resources (ER).
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#55102
**Release note**:
```release-note
Remove opaque integer resources (OIR) support (deprecated in v1.8.)
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Implement volume resize for cinder
**What this PR does / why we need it**:
resize for cinder
xref: [resize proposal](https://github.com/kubernetes/community/pull/657)
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: xref https://github.com/kubernetes/community/pull/657
Follow up: #49727
**Special notes for your reviewer**:
**Release note**:
```release-note
Implement volume resize for cinder
```
wip, assign to myself first
/assign @NickrenREN
Automatic merge from submit-queue (batch tested with PRs 55812, 55752, 55447, 55848, 50984). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Make versioned types for webhook admission config
Versioned webhook admission config type as promised in https://github.com/kubernetes/kubernetes/pull/54414.
@kubernetes/sig-api-machinery-pr-reviews
@ericchiang as promised. fyi.
```yaml
kind: AdmissionConfiguration
apiVersion: apiserver.k8s.io/v1alpha1
plugins:
- name: GenericAdmissionWebhook
configuration:
kind: WebhookAdmission
apiVersion: apiserver.config.k8s.io/v1alpha1
kubeConfigFile: /path/to/my/file
```
`ADMISSION_CONTROL_CONFIG_FILE=../foo.yaml hack/local-up-cluster.sh`
Automatic merge from submit-queue (batch tested with PRs 56128, 56004, 56083, 55833, 56042). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Suppress the warning when a pod in binding cannot be expired
**What this PR does / why we need it**:
I have a scheduler extender, which implements the `Bind` call and takes several minutes to respond to that call. The scheduler log was full of the following error.
```
W1120 10:23:09.691188 99720 cache.go:442] Couldn't expire cache for pod default/xxx. Binding is still in progress.
```
The TTL for a pod to be expired in the scheduler cache is 30 seconds. But it's also possible that the binding (which is done asynchronously) can take longer than 30 seconds.
2cbb07a439/plugin/pkg/scheduler/factory/factory.go (L143)
The go routine that checks whether a pod has been expired is triggered every second.
2cbb07a439/plugin/pkg/scheduler/schedulercache/cache.go (L33)
So, it will print the the following warning every seconds until the pod gets expired.
2cbb07a439/plugin/pkg/scheduler/schedulercache/cache.go (L442-L443)
I think it's a valid for the binding to take more than one second, so we should downgrade this to an info to avoid polluting the scheduler log.
**Release note**:
```release-note
None
```
/sig scheduling
/assign @bsalamat
/cc @vishh
Automatic merge from submit-queue (batch tested with PRs 54316, 53400, 55933, 55786, 55794). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add support to take nominated pods into account during scheduling to avoid starvation of higher priority pods
**What this PR does / why we need it**:
When a pod preempts lower priority pods, the preemptor gets a "nominated node name" annotation. We call such a pod a nominated pod. This PR adds the logic to take such nominated pods into account when scheduling other pods on the same node that the nominated pod is expected to run. This is needed to avoid starvation of preemptor pods. Otherwise, lower priority pods may fill up the space freed after preemption before the preemptor gets a chance to get scheduled.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#54501
**Special notes for your reviewer**: This PR is built on top of #55109 and includes all the changes there as well.
**Release note**:
```release-note
Add support to take nominated pods into account during scheduling to avoid starvation of higher priority pods.
```
/sig scheduling
ref/ #47604
Automatic merge from submit-queue (batch tested with PRs 55938, 56055, 53385, 55796, 55922). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
admission: make admission metrics compositional
Metrics emission of admission plugins and the admission chain can be implemented compositionally, i.e. completely independently from the chain logic. This PR does that, moves the whole metrics code into a sub-package to contain complexity. The plumbing logic for the emitted metrics finally is cleanly done in the apiserver bootstrapping code, instead of being totally interleaved with the core admission logic.
Ratio:
- considerably less complexity
- admission plugins are compositional, including the chain. We cannot assume that there is only one chain at the outside of the admission plugin structure. Downstream projects might have more complex admission chains, i.e. multiple chain object nested.
- addition of metrics is plumbing and should be in the apiserver plumbing code. This makes it much easier to reason about the security critical admission chain.
Follow-up of #55183 and based on #55919.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add ExtendedResourceToleration admission controller.
/kind feature
/sig scheduling
/area hw-accelerators
There's elaborate discussion on this in #55080. In short, we would like to enable cluster operators and/or cloud providers to create dedicated nodes with extended resources (like GPUs, FPGAs etc.) that are reserved for pods requesting such resources. [Taints is the kubernetes concept to create dedicated nodes.](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/#example-use-cases) If the cluster operator or cloud provider wants to create dedicated node pools, they are expected to taint the nodes containing extended resources with the key equal to the name of the resource and effect equal to NoSchedule. If they do that, only pods that have a toleration for such a taint can be scheduled there. To make it easy for the user, this admission controller when enabled, automatically adds a toleration with key `example.com/device`, operator `Exists` and effect `NoSchedule` if an extended resource of name `example.com/device` is requested.
**Release note**:
```release-note
Add ExtendedResourceToleration admission controller. This facilitates creation of dedicated nodes with extended resources. If operators want to create dedicated nodes with extended resources (like GPUs, FPGAs etc.), they are expected to taint the node with extended resource name as the key. This admission controller, if enabled, automatically adds tolerations for such taints to pods requesting extended resources, so users don't have to manually add these tolerations.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Move of unreachable taint key out of alpha
**What this PR does / why we need it**:
Move of unreachable taint key out of alpha, which already happened in community doc.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#54198
**Special notes for your reviewer**:
Please see #54198 for the context of this inconsistency.
**Release note**:
```release-note
Move unreachable taint key out of alpha.
Please note the existing pods with the alpha toleration should be updated by user himself to tolerate the GA taint.
```
Automatic merge from submit-queue (batch tested with PRs 51192, 55010). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Refactoring of priority function(CaculateSpreadPriority) by using map/reduce pattern
**What this PR does / why we need it**:
Ref #24246. exactly ref https://github.com/kubernetes/kubernetes/issues/51455, the PR aim to unify priority functions(deprecated) by using map/reduce pattern.
This is the first step, my todo list(WIP):
- interpod-affnity priority funciton refactoring
- the priority funcitons register pattern
- deprecated priority function definition and all related logic. etc.
**Which issue this PR fixes**:
no issue, just unify the priority functions pattern.
**Special notes for your reviewer**:
none
**Release note**:
none
Automatic merge from submit-queue (batch tested with PRs 54556, 55379, 55881, 55891, 55705). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Ensure config file exists before loading configuration in imagePolicyWebhook
**What this PR does / why we need it**:
ensure config file exists before loading configuration in imagePolicyWebhook
**Which issue(s) this PR fixes**
Fixes#55378
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 55254, 55525, 50108, 54674, 55263). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Refactor Priority Reduce functions
- Reducing the duplicated reduce function by introducing a reduce
function generator that generates common reduce functions.
- Remove logs from reduce function, so it's purely calculating scores.
- Optimize the reduce functions by removing unnecessary conversion to
float64.
**Release note**:
```release-note
None
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Adding unit tests to methods of 'scheduler/algorithm/priorities/util'
What this PR does / why we need it:
1) Adding unit tests to methods of 'non_zero.go'
2) Adding unit tests to methods of 'util.go'
3) Adding/Modifing unit tests to methods of 'topologies.go'
thank you!
This test would never fail because Annotations were empty in all cases!
This was missed when Tolerations were moved from Annotations to be part of Pod.Spec in #38957.
Automatic merge from submit-queue (batch tested with PRs 53780, 55663, 55321, 52421, 55659). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
remove duplicated import
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 53780, 55663, 55321, 52421, 55659). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
add hostip and protocol to the hostport predicates
**What this PR does / why we need it**:
This PR adds "hostIP and protocol" to scheduler hostport predicate procedure
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
fix#51950
**Special notes for your reviewer**:
- [x] basic implementation, need review
- [x] e2e test
- [x] update doc (will be done in seperate PR)
**Release note**:
```release-note
add hostIP and protocol to the original hostport predicates procedure in scheduler.
```
Automatic merge from submit-queue (batch tested with PRs 54005, 55127, 53850, 55486, 53440). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Enforce use of fixed size int types in the API
Changes 'int' to 'int32', enforced by import_known_versions_test
Follow up to PR #53402
**What this PR does / why we need it**:
This PR changes a few fields within the API from 'int' to 'int32' and is now enforced by import_known_versions_test. We need this so integer fields are the same size regardless of $GOARCH.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue (batch tested with PRs 54005, 55127, 53850, 55486, 53440). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
aggregate cluster roles
xref https://github.com/kubernetes/community/pull/1219https://github.com/kubernetes/features/issues/502
This is a pull with API types, a controller, and a demonstration of how to move admin, edit, and view. Once we agree on the shape, I'll
I added
```yaml
aggregationRule:
clusterRoleSelectors:
- matchLabels:
rbac.authorization.k8s.io/aggregate-to-admin: true
```
to the `ClusterRole`. A controller then goes and gathers all the matching ClusterRoles and sets the `rules` to the union of matching cluster roles.
@kubernetes/sig-auth-pr-reviews
```release-note
RBAC ClusterRoles can now select other roles to aggregate
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add test case in metadata_test.go
**What this PR does / why we need it**:
Add test case in metadata_test.go
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#55567
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 54987, 55221, 54099, 55144, 54215). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
not calculate new priority when user update other spec of a pod
motivation of this change:
If we update the priority, pod validation mechanism will prevent this update
request, this is not expected.
**Release note**:
```
Priority admission controller: not calculate new priority when user update a pod
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
remove redundant code in admission initializer
**What this PR does / why we need it**:
remove unused return error in `k8s.io\kubernetes\staging\src\k8s.io\apiserver\pkg\admission\initializer\initializer.go`
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 53651, 55252). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
split serviceaccount admission into mutation and validation
Splits the SA admission plugin into mutation and validation halves.
I had some trouble with e2e tests before, so this got split out of another pull.
Automatic merge from submit-queue (batch tested with PRs 53747, 54528, 55279, 55251, 55311). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
[trivil] fix comment in kube-scheduler
**Release note**:
```
NONE
```
Automatic merge from submit-queue (batch tested with PRs 53747, 54528, 55279, 55251, 55311). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
update podtolerations admission to mutate and validate separately
Updates the podtolerations admission plugin to mutate and validate separately. This also fixes the bug where the toleration whitelist isn't respected for memory pressure.
```release-note-action-required
The `node.kubernetes.io/memory-pressure` taint now respects the configured whitelist. If you need to use it, you'll have to add it to the whitelist.
```
Automatic merge from submit-queue (batch tested with PRs 54493, 52501, 55172, 54780, 54819). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix divide by zero issue in calculating spread priority for zones
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Allow HPA to get custom metrics
**What this PR does / why we need it**:
This PR allows HPA to read custom metrics.
**Release note**:
```release-note
Allow HPA to read custom metrics.
```
- Reducing the duplicated reduce function by introducing a reduce
function generator that generates common reduce functions.
- Remove logs from reduce function, so it's purely calculating scores.
- Optimize the reduce functions by removing unnecessary conversion to
float64.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix 'Schedulercache is corrupted' error
Fixes#50916
If an Assume()ed pod is Add()ed with a different nodeName, the podStates view of the pod is not corrected to reflect the actual nodeName. On the next Update(), the scheduler observes the mismatch and process exits.
```release-note
Fixed 'Schedulercache is corrupted' error in kube-scheduler
```
Automatic merge from submit-queue (batch tested with PRs 53592, 52562, 55175, 55213). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Refactor kube-scheduler config API, command, and server setup
Refactor the kube-scheduler configuration API, command setup, and server setup according to the guidelines established in #32215 and using the kube-proxy refactor (#34727) as a model of a well factored component adhering to said guidelines.
* Config API: clarify meaning and use of algorithm source by replacing modality derived from bools and string emptiness checks with an explicit AlgorithmSource type hierarchy.
* Config API: consolidate client connection config with common structs.
* Config API: split and simplify healthz/metrics server configuration.
* Config API: clarify leader election configuration.
* Config API: improve defaulting.
* CLI: deprecate all flags except `--config`.
* CLI: port all flags to new config API.
* CLI: refactor to match kube-proxy Cobra command style.
* Server: refactor away configurator.go to clarify application wiring.
* Server: refactor to more clearly separate wiring/setup from running.
Fixes https://github.com/kubernetes/kubernetes/issues/52428.
@kubernetes/api-reviewers
@kubernetes/sig-cluster-lifecycle-pr-reviews
@kubernetes/sig-scheduling-pr-reviews
/cc @ncdc @timothysc @bsalamat
```release-note
The kube-scheduler command now supports a `--config` flag which is the location of a file containing a serialized scheduler configuration. Most other kube-scheduler flags are now deprecated.
```
Automatic merge from submit-queue (batch tested with PRs 53273, 55058, 55237, 50140). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
split some admission plugins into mutation and validation halves
Splits the podnodeselector, serviceaccount, and priority admission plugins into validating and mutating admission plugins.
@kubernetes/sig-api-machinery-pr-reviews
Refactor the kube-scheduler configuration API, command setup, and server
setup according to the guidelines established in #32215 and using the
kube-proxy refactor (#34727) as a model of a well factored component
adhering to said guidelines.
* Config API: clarify meaning and use of algorithm source by replacing
modality derived from bools and string emptiness checks with an explicit
AlgorithmSource type hierarchy.
* Config API: consolidate client connection config with common structs.
* Config API: split and simplify healthz/metrics server configuration.
* Config API: clarify leader election configuration.
* Config API: improve defaulting.
* CLI: deprecate all flags except `--config`.
* CLI: port all flags to new config API.
* CLI: refactor to match kube-proxy Cobra command style.
* Server: refactor away configurator.go to clarify application wiring.
* Server: refactor to more clearly separate wiring/setup from running.
Fixes#52428.
Automatic merge from submit-queue (batch tested with PRs 55114, 52976, 54871, 55122, 55140). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
haveSame is suboptimal, fix it as well as the name
**What this PR does / why we need it**: Make the util function simpler, also faster by reducing a for loop.
**Special notes for your reviewer**:
**Release note**:
```release-note
None
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add comment for preempt sunction.
**What this PR does / why we need it**:
Add explain for preempt sunction.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
make easy validation admission plugins into validators
This switches "easy" admission plugins that need to be validators to be validators. I also did one simple plugin to do both mutation and validation for practice.
@kubernetes/sig-api-machinery-pr-reviews @caesarxuchao
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add admission handler for device resources allocation
**What this PR does / why we need it**:
Add admission handler for device resources allocation to fail fast during pod creation
**Which issue this PR fixes**
fixes#51592
**Special notes for your reviewer**:
@jiayingz Sorry, there is something wrong with my branch in #51895. And I think the existing comments in the PR might be too long for others to view. So I closed it and opened the new one, as we have basically reach an agreement on the implement :)
I have covered the functionality and unit test part here, and would set about the e2e part ASAP
/cc @jiayingz @vishh @RenaudWasTaken
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
admission: wire through validating-only admission
Based on https://github.com/kubernetes/kubernetes/pull/54232.
This is important and required for beta because it affects the shape of the webhook admission plugins that are going to be produced and is needed to make sure that our existing chain continues to properly verify and protect the API objects based on their final state after webhook admission mutators run.
We discussed this in the October 11 API machinery call with @erictune and @caesarxuchao and we agreed to do this as a requirement for beta. See this part of the recording: https://www.youtube.com/watch?v=mrgDPHyr4VY#t=325 .
Automatic merge from submit-queue (batch tested with PRs 54800, 53898, 54812, 54921, 53558). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Allow override of cluster level (default, whitelist) tolerations by namespace level empty (default, whitelist) tolerations.
Currently In PodTolerationRestriction admission plugin, if namespace level default and whitelist of tolerations are nil or empty, they do not override cluster level default and whitelist tolerations.
This PR fixes the plugin to not override cluster level tolerations only when namespace level toleration are nil. IOW, if namespace level toleration are empty, they override cluster level tolerations. To be more clear, if following annotations are set to empty, they override cluster level tolerations.
```
scheduler.alpha.kubernetes.io/defaultTolerations : ""
scheduler.alpha.kubernetes.io/tolerationsWhitelist: ""
```
This behavior is inline with PodNodeSelector admission plugin too.
@sjenning @derekwaynecarr
**Release Note**:
```release-note
In PodTolerationRestriction admisson plugin, if namespace level tolerations are empty, now they override cluster level tolerations.
```
Tim Allclair