Automatic merge from submit-queue
AWS: check validity of KSM key before creating a new encrypted disk.
AWS CreateVolume call does not check if referenced encryption key actually exists and returns a valid new AWS EBS volume even though an invalid key was specified. Later on it removes the EBS silently when its encryption fails.
To work around this buggy behavior we manually check that the key exists before calling CreateVolume.
Fixes#48438
/sig aws
Please review carefully. Can we safely assume that Kubernetes controller-manager can read encryption keys?
```release-note
aws: Kubernetes now checks existence of provided KSM (Key Management Service) key before creating an encrypted AWS EBS.
```
Automatic merge from submit-queue
e2e: Add tests for network tiers in GCE
This test depends on #51301, which adds the new feature. Only the `e2e: Add tests for network tiers in GCE` commit is new.
#51301 should pass this new test.
Automatic merge from submit-queue
Add Google cloud KMS service for envelope encryption transformer
This adds the required pieces which will allow addition of KMS based encryption providers (envelope transformer).
For now, we will be implementing it using Google Cloud KMS, but the code should make it easy to add support for any other such provider which can expose Decrypt and Encrypt calls.
Writing tests for Google Cloud KMS Service may cause a significant overhead to the testing framework. It has been tested locally and on GKE though.
Upcoming after this PR:
* Complete implementation of the envelope transformer, which uses LRU cache to maintain decrypted DEKs in memory.
* Track key version to assist in data re-encryption after a KEK rotation.
Development branch containing the changes described above: https://github.com/sakshamsharma/kubernetes/pull/4
Envelope transformer used by this PR was merged in #49350
Concerns #48522
Planned configuration:
```
kind: EncryptionConfig
apiVersion: v1
resources:
- resources:
- secrets
providers:
- kms:
cachesize: 100
configfile: gcp-cloudkms.conf
name: gcp-cloudkms
- identity: {}
```
gcp-cloudkms.conf:
```
[GoogleCloudKMS]
kms-location: global
kms-keyring: google-container-engine
kms-cryptokey: example-key
```
Automatic merge from submit-queue (batch tested with PRs 51298, 51510, 51511)
GCE: Add a fake forwarding rule service
Also add more methods to the address service. These
will be used for testing soon.
Automatic merge from submit-queue (batch tested with PRs 50919, 51410, 50099, 51300, 50296)
GCE: Read networkProjectID param
Fixes#48515
/assign bowei
The first commit is the original PR cherrypicked. The master's kubelet isn't provided a cloud config path, so the project is retrieved via instance metadata. In the GKE case, this project cannot be retrieved by the master and caused an error.
**Release note**:
```release-note
NONE
```
AWS CreateVolume call does not check if referenced encryption key actually
exists and returns a valid new AWS EBS volume even though an invalid key
was specified. Later on it removes the EBS silently when its encryption fails.
To work around this buggy behavior we manually check that the key exists
before calling CreateVolume.
Automatic merge from submit-queue
Implement GetZoneByProviderID and GetZoneByNodeName for openstack
This is part of #50926
cc @wlan0
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 51174, 51363, 51087, 51382, 51388)
Add InstanceExistsByProviderID to cloud provider interface for CCM
**What this PR does / why we need it**:
Currently, [`MonitorNode()`](02b520f0a4/pkg/controller/cloud/nodecontroller.go (L240)) in the node controller checks with the CCM if a node still exists by calling `ExternalID(nodeName)`. `ExternalID` is supposed to return the provider id of a node which is not supported on every cloud. This means that any clouds who cannot infer the provider id by the node name from a remote location will never remove nodes that no longer exist.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#50985
**Special notes for your reviewer**:
We'll want to create a subsequent issue to track the implementation of these two new methods in the cloud providers.
**Release note**:
```release-note
Adds `InstanceExists` and `InstanceExistsByProviderID` to cloud provider interface for the cloud controller manager
```
/cc @wlan0 @thockin @andrewsykim @luxas @jhorwit2
/area cloudprovider
/sig cluster-lifecycle
Automatic merge from submit-queue (batch tested with PRs 51235, 50819, 51274, 50972, 50504)
Support for specifying external LoadBalancerIP on openstack
1. Support ServiceAnnotationLoadBalancerFloatingNetworkId for LB v1
2. Support for specifying external LoadBalancerIP on openstack
Add ServiceAnnotationLoadBalancerInternal annotation to distinguish
between internal LoadBalancerIP and external LoadBalancerIP.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
Fix#50851
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 51114, 51233, 51024, 51053, 51197)
Add AddAliasToInstance() to gce cloud provider
- Adds AddAliasToInstance() to the GCE cloud provider.
- Adds field "secondary-range-name" to the gce.conf configuration file.
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 51244, 50559, 49770, 51194, 50901)
Fix the matching rule of instance ProviderID
Url.Parse() can't parse ProviderID which contains ':///'.
This PR use regexp to match ProviderID.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
Fix#49769
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 51224, 51191, 51158, 50669, 51222)
Change the FakeCloudAddressService to store Alpha objects internally
The change assumes the compute Alpha object is the superset of the v1
object. By storing the Alpha objects internally in the fake, we can
convert them to Beta and v1 to test different functions.
The change assumes the compute Alpha object is the superset of the v1
object. By storing the Alpha objects internally in the fake, we can
convert them to Beta and v1 to test different functions.
- Adds AddAliasToInstance() to the GCE cloud provider.
- Adds field "secondary-range-name" to the gce.conf configuration file.
```release-note
NONE
```
Automatic merge from submit-queue
cloudprovider/openstack bug fix: don't try to append pool id if pool doesn't exist
**What this PR does / why we need it**:
This fixes a bug in the OpenStack cloud provider that could cause a panic.
Consider what will happen in the current `LbaasV2.EnsureLoadBalancerDeleted` code if `nil, ErrNotFound` is returned by `getPoolByListenerID`.
Automatic merge from submit-queue (batch tested with PRs 38947, 50239, 51115, 51094, 51116)
Mark the volumes as detached when node does not exist
If node does not exist, node's volumes will be detached
automatically and become available. So mark them detached and do not return err.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
#50200
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 50967, 50505, 50706, 51033, 51028)
teach gce cloud to handle alpha/beta operations v2
Alternative to #50704
This one feels cleaner. BUT, type assertion problems cannot be exposed at compile time.
Please let me know what you think. This will set the precedence for consuming GCE alpha/beta API.
cc: @thockin @yujuhong @saad-ali @MrHohn
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 50893, 50913, 50963, 50629, 50640)
gce external LB: add a function to verify the requested IP address
Factor out the logic for verifying the user-requested IP for better
readability and testing. Also rename a few variables for clarity.
If user specify floating-network-id by annotation rather than cloud
provider file, openstack cloud provider don't delete floatingip when
deleting LoadBalancer service.
Automatic merge from submit-queue (batch tested with PRs 50255, 50885)
AWS: Arbitrarily choose first (lexicographically) subnet in AZ
When there is more than one subnet for an AZ on AWS choose arbitrarily
chose the first one lexicographically for consistency.
**What this PR does / why we need it**:
If two subnets were to be used appear in the same aws az which one is chosen is currently not consistent. This could lead to difficulty in diagnosing issues.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#45983
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue (batch tested with PRs 50303, 50856)
Make route-controller list only relevant routes instead of all of them
Ref https://github.com/kubernetes/kubernetes/issues/50854 (somewhat related issue)
IIUC from the code, route-controller memory is mainly being used in storing routes and nodes (also CIDRs, but that's not much).
This should help reduce that memory usage (particularly when running in a project with large no. of routes), by moving filtering to server-side.
For e.g in kubernetes-scale project we have ~5000 routes (each about 600B) => 3 MB of routes
This doesn't help with reducing time to list the routes as filtering is also linear.
cc @kubernetes/sig-scalability-misc @wojtek-t @gmarek
Automatic merge from submit-queue
Mark volume as detached when node does not exist for vsphere
If node does not exist, node's volumes will be detached
automatically and become available. So mark them detached and
return false without error.
Fix#50266
**Special notes for your reviewer**:
/assign @jingxu97
**Release note**:
```release-note
NONE
```
Kubernetes Submit Queue