Commit Graph

1055 Commits (6d5b2ef49e7f2f1dad51ec077a66b536e5329350)

Author SHA1 Message Date
Justin Santa Barbara 7e5c6877d7 Fixes per code review 2016-08-25 22:51:04 -04:00
Vipul Sabhaya 808f18439d Make a vSphere cluster the failure_zone
* Currently the vSphere cloud provider treats Datacenter as the failure
  Zone.  This doesn't necessarily work since in the current implemention
  Kubernetes nodes cannot span Datacenters.
* This change introduces Clusters as the failure zone, while treating
  Datacenters as Regions
* Also updated tests for Zones
2016-08-25 15:37:24 -07:00
Kubernetes Submit Queue 49ff2e8831 Merge pull request #31115 from jsafrane/add-constants
Automatic merge from submit-queue

Add constants and documentation around AWS magic numbers

Also, bumped max IOPS/GB to 50, it changed from 30 since last time I checked.

Source: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html

@kubernetes/sig-storage
2016-08-24 12:59:50 -07:00
Kubernetes Submit Queue f96b8fc85a Merge pull request #31334 from jsafrane/fix-aws-encryption
Automatic merge from submit-queue

Fix AWS reporting "The parameter KmsKeyId requires the parameter Encrypted to be set."

- use aws.String/Int/Bool functions
- don't set the key to empty string, use nil instead

@justinsb @kubernetes/sig-storage
2016-08-24 10:55:33 -07:00
Justin Santa Barbara 6a1f892c1d AWS: Sanity checks after volume attach
In the light of issue #29324, double check that the volume was attached
correctly where we expect it, before returning.

Issue #29324
2016-08-24 13:00:38 -04:00
Justin Santa Barbara 81240da858 AWS: move volume attachment map to cloud level
The problem is that attachments are now done on the master, and we are
only caching the attachment map persistently for the local instance.  So
there is now a race, because the attachment map is cleared every time.

Issue #29324
2016-08-24 13:00:33 -04:00
Kubernetes Submit Queue 3544f8a717 Merge pull request #29836 from rootfs/azuredd
Automatic merge from submit-queue

support Azure data disk volume

This is a WIP of supporting azure data disk volume. Will add test and dynamic provisioning support once #29006 is merged

replace #25915
fix #23259

@kubernetes/sig-storage 
@colemickens @brendandburns
2016-08-24 02:49:54 -07:00
Jan Safranek 8cd5e263b8 Fix AWS reporting "The parameter KmsKeyId requires the parameter Encrypted to be set."
- use aws.String/Int/Bool functions
- don't set the key to empty string, use nil instead
2016-08-24 10:05:07 +02:00
Kubernetes Submit Queue e427ab0baa Merge pull request #30649 from dagnello/openstack-lbaasv2-multiport
Automatic merge from submit-queue

fix Openstack provider to allow more than one service port for lbaas v2

This resolves bug #30477 where if a service defines multiple ports for load balancer, the plugin will fail with multiple ports are not supported.

@anguslees @jianhuiz
2016-08-23 22:36:09 -07:00
Girish Kalele b82c028f77 GCE Cloud provider changes for ESIPP
Add feature gate (ExternalTrafficLocalOnly) for alpha feature
2016-08-23 16:16:39 -07:00
Abrar Shivani 551cea9600 Fix: dynamic provisioning for vSphere 2016-08-23 15:48:47 -07:00
Davide Agnello 97fab82552 Openstack provider allowing more than one service port for lbaas v2 2016-08-23 10:17:32 -07:00
Huamin Chen dea4b0226d support Azure data disk volume
Signed-off-by: Huamin Chen <hchen@redhat.com>
2016-08-23 13:23:07 +00:00
Kubernetes Submit Queue c5d56ea356 Merge pull request #30535 from abrarshivani/vsphere_attach_detach_interface
Automatic merge from submit-queue

Implements Attacher Plugin Interface for vSphere

This PR does the following,

Fixes #29028 (vsphere volume should implement attacher interface):  Implements Attacher Plugin Interface for vSphere. 
See file: 
pkg/volume/vsphere_volume/vsphere_volume.go. - Removed attach and detach calls from SetupAt and TearDownAt.
pkg/volume/vsphere_volume/attacher.go. - Implements Attacher & Detacher Plugin Interface for vSphere. (Ref :- GCE_PD & AWS attacher.go)
pkg/cloudproviders/provider/vsphere.go - Added DiskIsAttach method.

The vSphere plugin code needs clean up. (ex: The code for getting vSphere instance is repeated in file pkg/cloudprovider/providers/vsphere.go). I will fix this in next PR.
2016-08-23 05:13:12 -07:00
Kubernetes Submit Queue 119c0028ee Merge pull request #30904 from rootfs/cinder-prov2
Automatic merge from submit-queue

support storage class in Cinder provisioner

replace #30876

@kubernetes/sig-storage @jsafrane
2016-08-22 11:06:45 -07:00
Kubernetes Submit Queue bfafb6f425 Merge pull request #30695 from krancour/manage-elb-attributes
Automatic merge from submit-queue

AWS: More ELB attributes via service annotations

Replaces #25015 and addresses all of @justinsb's feedback therein. This is a new PR because I was unable to reopen #25015 to amend it.

I noticed recently that there is existing (but undocumented) precedent for the AWS cloud provider to manage ELB-specifc load balancer configuration based on service annotations.  In particular, one can _already_ designate an ELB as "internal" or enable PROXY protocol.

This PR extends this capability to the management of ELB attributes, which includes the following items:
* Access logs:
    * Enabled / disabled
    * Emit interval
    * S3 bucket name
    * S3 bucket prefix
* Connection draining:
    * Enabled / disabled
    * Timeout
* Connection:
    * Idle timeout
* Cross-zone load balancing:
    * Enabled / disabled

Some of these are possibly more useful than others.  Use cases that immediately come to mind:

* Enabling cross-zone load balancing is potentially useful for "Ubernetes Light," or anyone otherwise attempting to spread worker nodes around multiple AZs.
* Increasing idle timeout is useful for the benefit of anyone dealing with long-running requests. An example I personally care about would be git pushes to Deis' builder component.
2016-08-22 10:24:12 -07:00
Jan Safranek a596668de7 Add constants and documentation aroung AWS magic numbers
Also, remove check for IOPS per GB, AWS checks it on its own.
2016-08-22 15:30:47 +02:00
Huamin Chen 259bce370e support storage class in Cinder provisioner
Signed-off-by: Huamin Chen <hchen@redhat.com>
2016-08-22 09:28:29 -04:00
Kubernetes Submit Queue 364d696fd5 Merge pull request #30563 from knarz/master
Automatic merge from submit-queue

AWS: Support HTTP->HTTP mode for ELB

**What this PR does / why we need it**:

Right now it is not possible to create an AWS ELB that listens for HTTP and where the backend pod also listens for HTTP.
I asked @justinsb in slack and he said that this seems to be an oversight, so I'd like to use this PR as a step towards solving this.

**Special notes for your reviewer**:

I've only added a simple unit test. Are any integration tests needed? I'm not familiar with the code base.

cc @therc
2016-08-22 00:54:44 -07:00
Abrar Shivani e89ad04422 Implements Attacher Plugin Interface for vSphere 2016-08-19 00:28:55 -07:00
markturansky 9a2645aa5e add encryption to aws provisioner and cloud provider 2016-08-18 15:42:44 -04:00
Jan Safranek d94220810e GCE changes for the new provisioning model 2016-08-18 10:36:50 +02:00
Jan Safranek 4b97db202c AWS changes for new provisioning model 2016-08-18 10:36:49 +02:00
Kent Rancourt 96dad1f0f3 Add support for managing ELB attributes with service annotations 2016-08-16 13:07:49 -04:00
Sander van Harmelen 7c3e644162 Enable managing public IP’s and work with projects
This commit adds logic for allocating and associating a public IP, if the `—load-balancer-ip` option is not used. It will do proper management of IP’s that are allocated by this provider, so IP’s that are no longer needed/used will also be released again.

Additionally the provider can now also work with CloudStack projects and advanced (VPC) networks.

Lastly the Zone interface now returns an actual zone (supplied by the cloud config), a few logical errors are fixed and the first few tests are added.

All the functionality is extensively tested against both basic and advanced (VPC) networks.
2016-08-13 10:23:17 +02:00
ngtuna 52cb7b0755 Initial CloudStack provider 2016-08-13 09:40:23 +02:00
Sascha Hanse 9a111fffc8 enables the aws-load-balancer-backend-protocol annotion to be used without a cert to be able to create an HTTP->HTTP ELB 2016-08-13 02:30:35 +02:00
Kubernetes Submit Queue 7df59f75cd Merge pull request #29726 from anguslees/lb-autodetect
Automatic merge from submit-queue

openstack: Autodetect LBaaS v1 vs v2

```release-note
* openstack: autodetect LBaaS v1/v2 by querying for available extensions.  For most installs, this effectively changes the default from v1 to v2.  Existing installs can add "lb-version = v1" to the provider config file to continue to use v1.
```

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.kubernetes.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.kubernetes.io/reviews/kubernetes/kubernetes/29726)
<!-- Reviewable:end -->
2016-08-12 09:02:42 -07:00
Angus Lees e4c354c329 openstack: Autodetect LBaaS v1 vs v2
This removes the need to manually specify the version in all but unusual
cases.

For most installs this will effectively flip the default from
v1 (deprecated) to v2 so conservative existing installs may want to
manually configure "lb-version = v1" before upgrading.
2016-08-12 10:33:11 +10:00
Kubernetes Submit Queue 2537f66f0e Merge pull request #29230 from luxas/goimport
Automatic merge from submit-queue

Run goimport for the whole repo

While removing GOMAXPROC and running goimports, I noticed quite a lot of other files also needed a goimport format. Didn't commit `*.generated.go`, `*.deepcopy.go` or files in `vendor`

This is more for testing if it builds.
The only strange thing here is the gopkg.in/gcfg.v1 => github.com/scalingdata/gcfg replace.
cc @jfrazelle @thockin
2016-08-05 16:22:01 -07:00
Rohith 0da5f50b03 - fixing the spelling mistakes 2016-08-04 10:17:59 +01:00
Abrar Shivani 87e7535e94 - Updated vmware/govmomi godep (Needs for vsan support)
- Fix unmount for vsanDatastore
- Add support for vsan datastore
2016-08-03 16:37:56 -07:00
Davide Agnello 19642aa4bd vSphere Volume Attach limit bug
- Cannot attach scsi devices to slot #7
- Limit of 4 scsi controllers per vm MAX
2016-08-02 14:49:09 -07:00
Lucas Käldström c88a07ce1a Run goimports 2016-08-02 15:12:39 +03:00
k8s-merge-robot c3129af2d3 Merge pull request #28222 from ammeon/openstack-reorg
Automatic merge from submit-queue

Re-org of the openstack cloud provider
2016-07-29 15:54:13 -07:00
k8s-merge-robot 5b7f7e7bd3 Merge pull request #29365 from lixiaobing10051267/masterLen
Automatic merge from submit-queue

len(vmList) output format not correct

len(vmList) output format not correct, not "%s", is "%d".
2016-07-27 02:41:58 -07:00
Cole Mickens 2ebffb431d implement azure cloudprovider 2016-07-26 14:50:33 -07:00
Cole Mickens 6ad9dc659f add clusterName to Loadbalancer methods 2016-07-26 14:50:33 -07:00
k8s-merge-robot b851e8d699 Merge pull request #29363 from lixiaobing10051267/masterV1
Automatic merge from submit-queue

TestLoadBalancer() test v1 not v2

TestLoadBalancer() should test v1 and TestLoadBalancerV2() test v2, but In TestLoadBalancerV() there are codes:
cfg.LoadBalancer.LBVersion = "v2"
2016-07-25 14:15:27 -07:00
k8s-merge-robot cbe8cd58a9 Merge pull request #29490 from justinsb/fix_cloudprovider_contract_comments
Automatic merge from submit-queue

Correctly document cloudprovider Instances contract
2016-07-23 11:10:54 -07:00
saadali 89fd358c52 Assume volume detached if node doesn't exist
Fixes #29358
2016-07-22 22:07:32 -07:00
Justin Santa Barbara aa9f2b2cda AWS: Deprecate a few functions in favor of aws-sdk-go
We have a few functions that predate aws-sdk-go, but they have natural
equivalents in aws-sdk-go.  Document them as deprecated, and replace
the implementation with the equivalent in aws-sdk-go to make it obvious
that they are the same.
2016-07-22 22:08:20 -04:00
Justin Santa Barbara 29e95431c5 Correctly document cloudprovider Instances contract
The requirement that ExternalID returns InstanceNotFound when the
instance not found was incorrectly documented on InstanceID and
InstanceType.  This requirement arises from the node controller, which
is the only place that checks for the InstanceNotFound error.
2016-07-22 21:59:31 -04:00
lixiaobing10051267 0e8dd302d1 len(vmList) output format not correct 2016-07-21 16:57:37 +08:00
lixiaobing10051267 54341caa87 TestLoadBalancer() test v1 not v2 2016-07-21 15:30:37 +08:00
k8s-merge-robot 10211f4df2 Merge pull request #29268 from lixiaobing10051267/masterExpected2
Automatic merge from submit-queue

Information is opposite to real meaning to express

master is not equal to expectedMaster, the meaning should be the master is unexpected:
	master, err := mesosCloud.Master(clusterName)
	if master != expectedMaster {
		t.Fatalf("Master returns the expected value: (expected: %#v, actual: %#v", expectedMaster, master)
2016-07-20 11:11:51 -07:00
k8s-merge-robot a0da4153b6 Merge pull request #29260 from lixiaobing10051267/masterErr
Automatic merge from submit-queue

Modify err output format from %s to %v

t.Errorf err output format should be %v
2016-07-20 11:11:46 -07:00
k8s-merge-robot 60f9ce8a41 Merge pull request #29253 from lixiaobing10051267/masterLBname
Automatic merge from submit-queue

format number not consistent with real variable number

glog.Infof format number not consistent with real variable number, should add %s for second var because loadBalancerName is string:
func (c *Cloud) ensureLoadBalancer(namespacedName types.NamespacedName, loadBalancerName string, ...
2016-07-20 11:11:42 -07:00
lixiaobing10051267 025135de6a Information is opposite to real meaning to express 2016-07-20 16:54:19 +08:00
lixiaobing10051267 e3bff25dbb Modify err output format from %s to %v 2016-07-20 15:06:47 +08:00
k8s-merge-robot a3110dcb41 Merge pull request #28417 from kevensen/awszonefix
Automatic merge from submit-queue

AWS: Added experimental option to skip zone check

This pull request resolves #28380.  In the vast majority of cases, it is appropriate to validate the AWS region against a known set of regions.  However, there is the edge case where this is undesirable as Kubernetes may be deployed in an AWS-like environment where the region is not one of the known regions.

By adding the optional **DisableStrictZoneCheck true** to the **[Global]** section in the aws.conf file (e.g. /etc/aws/aws.conf) one can bypass the ragion validation.
2016-07-19 21:03:28 -07:00
lixiaobing10051267 6de7bc0085 format number not consistent with real variable number 2016-07-20 11:14:45 +08:00
Kenneth D. Evensen d69fe11c09
Fixing gofmt errors 2016-07-19 16:50:55 -04:00
Davanum Srinivas ee8507a5ae Use Infof/Warningf when appropriate
When we use a format string, we should use Infof/Warningf instead
of Info/Warning
2016-07-19 12:10:53 -04:00
k8s-merge-robot 73a3d48dc8 Merge pull request #29196 from lixiaobing10051267/masterDiskName
Automatic merge from submit-queue

glog.Warning output content not complete, lack of string fomat "%s"
2016-07-19 07:31:24 -07:00
lixiaobing10051267 1a01308356 glog.Warning output content not complete, lack of string fomat "%s" 2016-07-19 20:35:52 +08:00
Wojciech Tyczynski 84f7ceb80a Extend logging for GCE load balancer creation 2016-07-19 13:38:51 +02:00
k8s-merge-robot fa174bcdaf Merge pull request #29042 from dims/fixup-imports
Automatic merge from submit-queue

Use Go canonical import paths

Add canonical imports only in existing doc.go files.
https://golang.org/doc/go1.4#canonicalimports

Fixes #29014
2016-07-18 07:23:38 -07:00
Prashanth Balasubramanian a9426a19c9 Don't recreate lb cloud resources on kcm restart 2016-07-17 21:59:22 -07:00
Davanum Srinivas 2b0ed014b7 Use Go canonical import paths
Add canonical imports only in existing doc.go files.
https://golang.org/doc/go1.4#canonicalimports

Fixes #29014
2016-07-16 13:48:21 -04:00
joe2far 5a8445a79a Re-org of the openstack cloud provider 2016-07-14 12:41:45 +01:00
joe2far 5ead89b5bb Fixed several typos 2016-07-13 15:06:24 +01:00
Quinton Hoole 791dd215d2 Deprecate the term "Ubernetes" in favor of "Cluster Federation" and "Multi-AZ Clusters" 2016-07-06 15:42:56 -07:00
Kenneth D. Evensen 274411b94e
Adding comments 2016-07-02 16:54:41 -04:00
Kenneth D. Evensen 7e4af9a66b
Added flag to skip zone check 2016-07-02 05:58:25 -04:00
David McMahon ef0c9f0c5b Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
dkalleg 49436012ba vSphere provider - Adding config for working dir
This allows the user the set "working-dir" in their vsphere.cfg file.
The value should be a path in the vSphere datastore in which the
provider will look for vms.
2016-06-27 11:45:39 -07:00
k8s-merge-robot 56db230455 Merge pull request #27331 from hpcloud/hpe/vsphere-get-id
Automatic merge from submit-queue

vSphere provider - Getting node data by ip instead of uuid

To get the uuid we need the service to be running as root. This change
allows us to run the controller-manager and api server as non-root.
2016-06-25 20:57:50 -07:00
Rudi Chiarito 8db551f674 golint fixes for aws cloudprovider 2016-06-24 17:06:38 -04:00
k8s-merge-robot 3a29aa7941 Merge pull request #27496 from hpcloud/hpe/vsphere-scsidriver
Automatic merge from submit-queue

Adding SCSI controller type filter for vSphere disk attach

Hot plug of disks to a SCSI controller of type lsilogic doesn't work as expected. When a device is detached from the controller, it fails to remove the device from the /dev path which makes the subsequent attaches to the node to fail. With scsi controller types lsilogic-sas or paravirtual this seems to work well. This patch filters the existing controller for these types, and if it doesn't find one, it creates a new controller for disk attach.

This PR is dependent on https://github.com/kubernetes/kubernetes/pull/26658 (1st commit) also targeting this for 1.3
2016-06-23 08:09:43 -07:00
k8s-merge-robot b7ac841e7f Merge pull request #27810 from hpcloud/lbaas-v2-liberty-fix
Automatic merge from submit-queue

Removing name field from Member for compatibility with OpenStack Liberty

In OpenStack Mitaka, the name field for members was added as an optional field but does not exist in Liberty.  Therefore the current implementation for lbaas v2 will not work in Liberty.
2016-06-22 11:22:48 -07:00
k8s-merge-robot 07471cf90f Merge pull request #27553 from justinsb/pvc_zone_spreading_2
Automatic merge from submit-queue

AWS/GCE: Spread PetSet volume creation across zones, create GCE volumes in non-master zones

Long term we plan on integrating this into the scheduler, but in the
short term we use the volume name to place it onto a zone.
    
We hash the volume name so we don't bias to the first few zones.
    
If the volume name "looks like" a PetSet volume name (ending with
-<number>) then we use the number as an offset.  In that case we hash
the base name.
2016-06-22 01:22:16 -07:00
k8s-merge-robot 48f2b11c48 Merge pull request #27829 from zmerlynn/fix-tp-max-2
Automatic merge from submit-queue

GCE provider: Create TargetPool with 200 instances, then update with rest

GCE provider: Create TargetPool with 200 instances, then update with rest
 
Tested with 2000 nodes, this actually meets the GCE API specifications (which is nutty). Previous PR (#25178) was based on a mistaken understanding of a poorly documented set of limitations, and even poorer testing, for which I am embarassed.

Also includes the revert of #25178 (review commits separately).

[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/.github/PULL_REQUEST_TEMPLATE.md?pixel)]()
2016-06-22 00:48:30 -07:00
Zach Loafman f63ac1962c GCE provider: Create TargetPool with 200 instances, then update with rest
Tested with 2000 nodes, this actually meets the GCE API specifications
(which is nutty). Previous PR (#25178) was based on a mistaken
understanding of a poorly documented set of limitations, and even
poorer testing, for which I am embarassed.
2016-06-21 18:15:02 -07:00
Davide Agnello 53805caa60 Removing name field from Member for compatibility with OpenStack Liberty
In OpenStack Mitaka, the name field for members was added as an optional
field but does not exist in Liberty.  Therefore the current
implementation for lbaas v2 will not work in Liberty.
2016-06-21 15:29:33 -07:00
Justin Santa Barbara 404c501c0d AWS: Add missing error check for #27774
There was an error check missing, which seems likely to have caused 27774

Issue #27774
2016-06-21 15:37:18 -04:00
Justin Santa Barbara dd94997619 Add comments & misc review fixes
Lots of comments describing the heuristics, how it fits together and the
limitations.

In particular, we can't guarantee correct volume placement if the set of
zones is changing between allocating volumes.
2016-06-21 15:22:16 -04:00
Zach Loafman ebc65f1dd3 Revert "GCE: Allow nodes to exceed target pool maximums"
This reverts commit faf0c44429.
2016-06-21 09:54:56 -07:00
Zach Loafman dd4dae4a57 GCE provider: Limit Filter calls to regexps rather than insane blobs
Filters can't exceed 4k, and GET requests against the GCE API are also
limited, so these break down in different ways at different cluster
counts. Fix it by introducing an advisory node-instance-prefix
configuration in the GCE provider that can hint the
EnsureLoadBalancer/UpdateLoadBalancer code (and the firewall
creation/update code). If it's not there, or wrong (a hostname that's
registered violates it), just ignore it and grab the whole project.
2016-06-21 08:34:48 -07:00
k8s-merge-robot 554b7010fa Merge pull request #27677 from justinsb/fix_24254
Automatic merge from submit-queue

AWS: Enable ICMP Type 3 Code 4 for ELBs

This enables MTU discovery.

Fixes #24254
2016-06-20 11:07:40 -07:00
Abitha Palaniappan 4a5ade213c Adding scsi controller type filter while attaching disks
Hot attach of disk to a scsi controller will work only if the
controller type is lsilogic-sas or paravirtual.This patch filters
the existing controller for these types, if it doesn't find one it
creates a new scsi controller.
2016-06-20 09:54:55 -07:00
k8s-merge-robot 4fcbc0ada7 Merge pull request #26658 from hpcloud/hpe/vsphere-vol-bugfixes
Automatic merge from submit-queue

Fixing vSphere Volume plugin bugs

This PR fixes #26646 and targeted for 1.3
2016-06-19 21:06:13 -07:00
k8s-merge-robot 536ed2843e Merge pull request #27628 from justinsb/fix_27534
Automatic merge from submit-queue

AWS volumes: Use /dev/xvdXX names with EC2

We are using HVM style names, which cannot be paravirtual style names.

See
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html

This also fixes problems introduced when moving volume mounting to KCM.

Fix #27534
2016-06-19 13:17:09 -07:00
k8s-merge-robot 090e6ff660 Merge pull request #27625 from goltermann/spell2
Automatic merge from submit-queue

Fix several spelling errors in comments

What the title says
2016-06-18 23:08:28 -07:00
Justin Santa Barbara fddc9d61fa AWS: Enable ICMP Type 3 Code 4 for ELBs
This enables MTU discovery.

Fixes #24254
2016-06-18 21:52:10 -04:00
k8s-merge-robot 7ea6705519 Merge pull request #25929 from jingxu97/bug-notag
Automatic merge from submit-queue

refuse to create a firewall rule with no target tag

fixes #25145

This modification in gce.firewallObject() will return error when trying
to create or update firewall rule if no node tag can be found. Also add
unit test for this modification.
2016-06-17 23:56:17 -07:00
Justin Santa Barbara 9c2566572d GCE Multizone: Allow volumes to be created in non-master zone
We had a long-lasting bug which prevented creation of volumes in
non-master zones, because the cloudprovider in the volume label
admission controller is not initialized with the multizone setting
(issue #27656).

This implements a simple workaround: if the volume is created with the
failure-domain zone label, we look for the volume in that zone.  This is
more efficient, avoids introducing a new semantic, and allows users (and
the dynamic provisioner) to create volumes in non-master zones.

Fixes #27657
2016-06-17 23:27:41 -04:00
Justin Santa Barbara e711cbf912 GCE/AWS: Spread PetSet volume creation across zones
Long term we plan on integrating this into the scheduler, but in the
short term we use the volume name to place it onto a zone.

We hash the volume name so we don't bias to the first few zones.

If the volume name "looks like" a PetSet volume name (ending with
-<number>) then we use the number as an offset.  In that case we hash
the base name.

Fixes #27256
2016-06-17 23:27:31 -04:00
goltermann 218645b346 Fix several spelling errors in comments. 2016-06-17 10:41:18 -07:00
Justin Santa Barbara 3af950f8f4 AWS volumes: Use /dev/xvdXX names with EC2
We are using HVM style names, which cannot be paravirtual style names.

See
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html

This also fixes problems introduced when moving volume mounting to KCM.

Fix #27534
2016-06-17 13:09:26 -04:00
dkalleg d95e767601 vSphere provider - Getting node data by ip instead of uuid
To get the uuid we need the service to be running as root. This change
allows us to run the controller-manager and api server as non-root.
2016-06-15 13:50:20 -07:00
Abitha Palaniappan 6a8cec1c5c Fix vSphere Volume plugin bugs
- replaces probeVolume with scsiHostRescan to scan hot attached disks
 - fixes substring match of UUID returned from AttachDisk
 - changes DetachDisk to take volumePath argument instead of diskID
 - fixes delayed failure at mount rather than attach disk
 - removes cloning of virtual disk in AttachDisk
2016-06-13 17:20:55 -07:00
Zach Loafman 49b3936829 GCE provider: Fix minor cosmetic logging issue
Log the pollOp, not the base op.
2016-06-13 14:53:12 -07:00
k8s-merge-robot 6d32eba4fd Merge pull request #26900 from Clarifai/aws-instance-cache
Automatic merge from submit-queue

AWS: cache instances during service reload to avoid rate limiting on restart

Fixes #25610 by reducing redundant calls to DescribeInstances()
 
```release-note
* The AWS cloudprovider will cache results from DescribeInstances() if the set of nodes hasn't changed
```

Also move int/stringSlicesEqual from servicecontroller.go to pkg/util/slice
2016-06-12 14:24:25 -07:00
k8s-merge-robot 5d8710196b Merge pull request #26962 from zmerlynn/log-gce-better
Automatic merge from submit-queue

GCE provider: Log full contents of long operations

[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/.github/PULL_REQUEST_TEMPLATE.md?pixel)]()Dump JSON of long running (>1m) GCE operations.
2016-06-12 10:46:01 -07:00
Rudi Chiarito e29709df73 AWS: cache values from getInstancesByNodeName() 2016-06-11 13:46:06 -04:00
k8s-merge-robot 4793372a85 Merge pull request #25888 from rootfs/attacher-aws-cinder
Automatic merge from submit-queue

implement EBS and Cinder attacher/detacher 

follow up with #21709

@kubernetes/sig-storage
2016-06-10 05:39:22 -07:00
k8s-merge-robot a2703a3e63 Merge pull request #26976 from Clarifai/elb
Automatic merge from submit-queue

AWS: support mixed plaintext/encrypted ports in ELBs via service.beta.kubernetes.io/aws-load-balancer-ssl-ports annotation

Fixes #26268

Implements the second SSL ELB annotation, per #24978

`service.beta.kubernetes.io/aws-load-balancer-ssl-ports=*` (comma-separated list of port numbers or e.g. `https`)

If not specified, all ports are secure (SSL or HTTPS).
2016-06-10 00:12:24 -07:00
k8s-merge-robot cbde2ec8c2 Merge pull request #25987 from hpcloud/openstack-lbaas-v2
Automatic merge from submit-queue

LBaaS v2 Support for Openstack Cloud Provider Plugin

Resolves #19774.

This work is based on Gophercloud support for LBaaS v2 currently in review (this will have to merge first):
https://github.com/rackspace/gophercloud/pull/575

These changes includes the addition of a new loadbalancer configuration option:  **LBVersion**.  If this configuration attribute is missing or anything other than "v2", lbaas v1 implementation will be used.
2016-06-09 18:32:35 -07:00
Davide Agnello eb32bf55ef Adding support for lbaas-v2 in OpenStack Cloud Provider 2016-06-08 18:16:59 -07:00
Huamin Chen d1e0a13924 support AWS and Cinder attacher
Signed-off-by: Huamin Chen <hchen@redhat.com>
2016-06-08 12:56:24 +00:00
Jan Safranek 5cd5ae8d82 Add GCE attacher unit tests. 2016-06-08 13:53:04 +02:00
Jan Safranek 2b342c1e76 Add interface to abstract GCE volume operations.
We want to write unit test with fake GCE.
2016-06-08 12:37:08 +02:00
Rudi Chiarito 4ff9e9319f AWS: support mixed plaintext/encrypted ports in ELBs
Fixes #26268

Implements the second SSL ELB annotation, per #24978

service.beta.kubernetes.io/aws-load-balancer-ssl-ports=* (or e.g. https)

If not specified, all ports are secure (SSL or HTTPS).
2016-06-07 18:39:53 -04:00
Zach Loafman ce6537aa48 GCE provider: Log full contents of long operations
Dump JSON of long running (>1m) GCE operations.
2016-06-07 06:58:43 -07:00
Andrew Williams 01d9cddda5 Add Amazon ELB proxy protocol support
Add ELB proxy protocol support via the annotation
"service.beta.kubernetes.io/aws-load-balancer-proxy-protocol". This
allows servers like Nginx and Haproxy to retrieve the real IP address of
a remote client.
2016-05-31 10:33:16 -05:00
Quinton Hoole 4983183cdd Merge pull request #26020 from quinton-hoole/2016-05-21-google-cloud-dns
Google Cloud DNS dnsprovider.
2016-05-28 07:55:21 -07:00
Quinton Hoole 0af78dcce7 Google Cloud DNS dnsprovider rewrite/refactor for unit tests. Successfully Tested against both real backend and stubbed backend. 2016-05-27 15:22:43 -07:00
Jing Xu 9a66dc7282 refuse to create a firewall rule with no target tag
Implements #25145
This modification in gce.firewallObject() will return error when trying
to create or update firewall rule if no node tag can be found. Also add
unit test for this modification.
2016-05-26 17:33:39 -07:00
Minhan Xia a1bd33f510 promote sourceRange into service spec 2016-05-26 10:42:30 -07:00
Alex Mohr 0c20ae7e90 Merge pull request #26331 from Clarifai/kitchensink
AWS: Fix long-standing bug in stringSetToPointers
2016-05-26 10:41:34 -07:00
Rudi Chiarito ca8699e83e Fix long-standing bug in aws.stringSetToPointers
Instead of N pointers, we were returning N null pointers, followed by the real
thing. It's not clear why we didn't trip on this until now, maybe there is a
new server-side check for empty subnetID strings.
2016-05-26 10:56:37 -04:00
Zach Loafman 34d69f4497 Revert "GCE provider: Rate limit all API calls"
This reverts commit 9b5cdfb705.
2016-05-25 14:23:46 -07:00
Zach Loafman 4adc542667 Revert "Log requests to GCE"
This reverts commit 55fdc1c036.
2016-05-25 14:23:41 -07:00
Wojciech Tyczynski 55fdc1c036 Log requests to GCE 2016-05-24 18:14:37 +02:00
Zach Loafman 9b5cdfb705 GCE provider: Rate limit all API calls
Instead of just rate limits to operation polling, send all API calls
through a rate limited RoundTripper.

This isn't a perfect solution, since the QPS is obviously getting
split between different controllers, etc., but it's also spread across
different APIs, which, in practice, rate limit differently.

Fixes #26119 (hopefully)
2016-05-23 23:23:53 -07:00
Abitha Palaniappan 95c009dbdb Adding vSphere Volume support for vSphere Cloud Provider 2016-05-21 11:00:14 -07:00
k8s-merge-robot d33fa39abf Merge pull request #23254 from jsafrane/devel/ulimited-aws-devices
Automatic merge from submit-queue

AWS: Move enforcement of attached AWS device limit from kubelet to scheduler

Limit of nr. of attached EBS volumes to a node is now enforced by scheduler. It can be adjusted by `KUBE_MAX_PD_VOLS` env. variable there. Therefore we don't need the same check in kubelet. If the system admin wants to attach more, we should allow it.

Kubelet limit is now 650 attached volumes ('ba'..'zz').

Note that the scheduler counts only *pods* assigned to a node. When a pod is deleted and a new pod is scheduled on a node, kubelet start (slowly) detaching the old volume and (slowly) attaching the new volume. Depending on AWS speed **it may happen that more than KUBE_MAX_PD_VOLS volumes are actually attached to a node for some time!** Kubelet will clean it up in few seconds / minutes (both attach/detach is quite slow).

Fixes #22994
2016-05-19 06:13:42 -07:00
dagnello 52811375d1 Adding error check when read instance id failed 2016-05-13 22:18:55 -07:00
k8s-merge-robot 24c46acd16 Merge pull request #24369 from Clarifai/ecr
Automatic merge from submit-queue

AWS: Allow cross-region image pulling with ECR

Fixes #23298
Definitely should be in the release notes; should maybe get merged in 1.2 along with #23594 after some soaking. Documentation changes to follow.

cc @justinsb @erictune @rata @miguelfrde

This is step two. We now create long-lived, lazy ECR providers in all regions.
When first used, they will create the actual ECR providers doing the work
behind the scenes, namely talking to ECR in the region where the image lives,
rather than the one our instance is running in.

Also:
- moved the list of AWS regions out of the AWS cloudprovider and into the
credentialprovider, then exported it from there.
- improved logging

Behold, running in us-east-1:

```
aws_credentials.go:127] Creating ecrProvider for us-west-2
aws_credentials.go:63] AWS request: ecr:GetAuthorizationToken in us-west-2
aws_credentials.go:217] Adding credentials for user AWS in us-west-2
Successfully pulled image "123456789012.dkr.ecr.us-west-2.amazonaws.com/test:latest"
```

*"One small step for a pod, one giant leap for Kube-kind."*

<!-- Reviewable:start -->
---
This change is [<img src="http://reviewable.k8s.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](http://reviewable.k8s.io/reviews/kubernetes/kubernetes/24369)
<!-- Reviewable:end -->
2016-05-13 15:15:45 -07:00
k8s-merge-robot 4ac4e0f6a2 Merge pull request #23495 from Clarifai/elb
Automatic merge from submit-queue

AWS: SSL support for ELB listeners through annotations

In the API, ports have only either TCP or UDP as their protocols, but ELB distinguishes HTTPS->HTTP[S]? from SSL->(SSL|TCP). 

Per #24978, this is implemented through two separate annotations:

`service.beta.kubernetes.io/aws-load-balancer-ssl-cert=arn:aws:acm:us-east-1:123456789012:certificate/12345678-1234-1234-1234-123456789012`
`service.beta.kubernetes.io/aws-load-balancer-backend-protocol=(https|http|ssl|tcp)`

Mixing plain-text and encrypted listeners will be in a separate PR, implementing #24978's `aws-load-balancer-ssl-ports=LIST`
2016-05-12 07:12:35 -07:00
David Oppenheimer 4ae37df7b7 Remove myself from a bunch of OWNERS files, as I am too overloaded
to do so many code reviews right now.
2016-05-11 13:34:51 -07:00
k8s-merge-robot e1fa044dba Merge pull request #24703 from hpcloud/hpe/vsphere-instance
Automatic merge from submit-queue

vSphere Cloud Provider Implementation 

This is the first PR towards implementation for vSphere cloud provider support in Kubernetes (ref. issue #23932).
2016-05-11 04:32:16 -07:00
Rudi Chiarito 6e6ea46182 Include changes from feedback
Use constructor for ecrProvider
Rename package to "credentials" like golint requests
Don't wrap the lazy provider with a caching provider
Add immedita compile-time interface conformance checks for the interfaces
Added comments
2016-05-10 12:03:40 -04:00
Rudi Chiarito eea29e8851 Allow cross-region image pulling with AWS' ECR
This is step two. We now create long-lived, lazy ECR providers in all regions.
When first used, they will create the actual ECR providers doing the work
behind the scenes, namely talking to ECR in the region where the image lives,
rather than the one our instance is running in.

Also:

- moved the list of AWS regions out of the AWS cloudprovider and into the
credentialprovider, then exported it from there.
- improved logging

Behold, running in us-east-1:

```
aws_credentials.go:127] Creating ecrProvider for us-west-2
aws_credentials.go:63] AWS request: ecr:GetAuthorizationToken in us-west-2
aws_credentials.go:217] Adding credentials for user AWS in us-west-2
Successfully pulled image 123456789012.dkr.ecr.us-west-2.amazonaws.com/test:latest"
```

*"One small step for a pod, one giant leap for Kube-kind."*
2016-05-10 12:03:39 -04:00
Rudi Chiarito 59334408a6 Change default when no BE proto given, add test for that
Also improve error message when BE proto is wrong
2016-05-10 11:53:44 -04:00
Rudi Chiarito e19c069b9d Add comment, rename getListener to buildListener 2016-05-10 11:40:34 -04:00
Zach Loafman faf0c44429 GCE: Allow nodes to exceed target pool maximums
If we would exceeded the TargetPool API maximums, instead just
randomly select some subsection of the nodes to include in the TP
instead.
2016-05-09 21:44:58 -07:00
dagnello f7b3cf39d6 Adding OWNERS file for vSphere cloud-provider package
also updating license file for Govmomi library
2016-05-09 08:55:42 -07:00
dagnello 4fd02f54ef Retreive VM Name from Running Instance
When vSphere cloud provider object is instantiated, the VM name of the
Node where this object is being create in needs to be set.  This patch
also includes vSphere as part of the cloud provider package.
2016-05-09 08:31:18 -07:00
dagnello c999cbaf8d VSphere Instance NodeAddresses ExternalID and InstanceID
This patch includes implementation for the following Instance object
interfaces:
* NodeAddresses
* ExternalID
* InstanceID

Also minor refactoring in overall Instance implementation.
2016-05-09 08:31:18 -07:00
Abitha Palaniappan 0d7d7fba60 Adding vSphere cloud provider support for Instance List 2016-05-09 08:31:18 -07:00
dagnello 88f6a6d66b Adding Zones Support for vSphere Cloud Provider 2016-05-09 08:31:18 -07:00
dagnello 051aa190e9 Adding vSphere Instances Scaffolding 2016-05-09 08:31:18 -07:00
Abitha Palaniappan c2045e8ee0 Add vSphere cloud provider skeleton 2016-05-09 08:31:18 -07:00
Alex Robinson 3d435b56be Merge pull request #25225 from a-robinson/fix
Actually populate nodeTags field in GCE provider struct
2016-05-05 14:35:01 -07:00
Alex Robinson 15874cd717 Actually populate nodeTags field in GCE provider struct 2016-05-05 21:26:54 +00:00
Alex Robinson b75fa08aec GCE: Prefer preconfigured node tags for firewalls, if available 2016-05-04 22:49:06 +00:00
Rudi Chiarito 898df1f52b Fix API fields to use new int32 sizes 2016-05-02 19:20:50 -04:00
Rudi Chiarito 61471965d8 Split annotation in two 2016-05-02 19:18:02 -04:00
Rudi Chiarito 7b7dd7861f Add support for HTTPS->HTTP ELB listeners through annotations
Moved listener creation to a separate function, which had the nice
side effect of allowing tests (added eight cases).
2016-05-02 19:18:01 -04:00
k8s-merge-robot 928990730e Merge pull request #24457 from leokhoa/master
Automatic merge from submit-queue

AWS: Add support for ap-northeast-2 region (Seoul)

This PR does:
- Support AWS Seoul region: ap-northeast-2. 
Currently, I can not setup Kubernetes on AWS Seoul.  
Error Messages: 

> 
> ip-10-0-0-50 core # docker logs 0697db
> I0419 07:57:44.569174       1 aws.go:466] Zone not specified in configuration file; querying AWS metadata service
> F0419 07:57:44.570380       1 controllermanager.go:279] Cloud provider could not be initialized: could not init cloud provider "aws": not a valid AWS zone (unknown region): ap-northeast-2a
2016-05-02 09:46:47 -07:00
Clayton Coleman fdb110c859
Fix the rest of the code 2016-04-29 17:12:10 -04:00
zhouhaibing089 bf1a3f99c0 Uncomment the code that cause by #19254 2016-04-25 23:21:31 +08:00
k8s-merge-robot 06160b6abe Merge pull request #22023 from mkulke/rackspace-improvements
Automatic merge from submit-queue

Rackspace improvements (OpenStack Cinder)

This adds PV support via Cinder on Rackspace clusters. Rackspace Cloud Block Storage is pretty much vanilla OpenStack Cinder, so there is no need for a separate Volume Plugin. Instead I refactored the Cinder/OpenStack interaction a bit (by introducing a CinderProvider Interface and moving the device path detection logic to the OpenStack part).

Right now this is limited to `AttachDisk` and `DetachDisk`. Creation and deletion of Block Storage is not in scope of this PR.

Also the `ExternalID` and `InstanceID` cloud provider methods have been implemented for Rackspace.
2016-04-21 16:38:13 -07:00
kulke ba4d74f3c7 Added Block Storage support to Rackspace provider, improved Node discovery. 2016-04-21 10:31:37 +02:00
goltermann 3fa6c6f6d9 Enable vet 2016-04-20 09:48:24 -07:00
Alex Robinson 23a8401c32 Merge pull request #24452 from a-robinson/dpv
Fix gce.getDiskByNameUnknownZone logic.
2016-04-19 11:58:04 -07:00
k8s-merge-robot db28f73c3b Merge pull request #24282 from goltermann/spelling
Automatic merge from submit-queue

Fix misspellings in comments
2016-04-19 03:47:07 -07:00
Khoa Le ce771effc6 Added AWS Seoul region 2016-04-19 15:20:39 +07:00
Alex Robinson d69ccd5c8c Fix gce.getDiskByNameUnknownZone logic. 2016-04-18 22:55:59 -07:00
k8s-merge-robot 1087162548 Merge pull request #23841 from dagnello/update-openstack-cloudprovider-volume-tests
Automatic merge from submit-queue

Extend OpenStack Cloud Provider Volume Tests

Adding coverage for:
getVolume
AttachDisk
DetachDisk
2016-04-16 17:44:09 -07:00
goltermann c226c9435b Fix misspellings in comments.
https://goreportcard.com/report/k8s.io/kubernetes#misspell
2016-04-14 13:57:45 -07:00
k8s-merge-robot 62d65f3ce6 Merge pull request #23770 from dagnello/update-openstack-cloudprovider-instance-tests
Automatic merge from submit-queue

Extend OpenStack Cloud Provider Instance Tests

Adding OpenStack cloud provider Instance coverage for:
ExternalID
InstanceID
2016-04-14 04:26:42 -07:00
Jeff Lowdermilk 9be85fbaf0 Merge pull request #23541 from bprashanth/fw_default
Default firewall port to TCP when unspecified.
2016-04-08 09:32:00 -07:00
dagnello cd7e99af1c Add OpenStack Cloud Provider Instance Tests
Adding OpenStack cloud provider instance unit tests for externalID and
instanceID.
2016-04-04 17:06:17 -07:00
dagnello 28590781c1 Extend OpenStack Cloud Provider Volume Tests
Adding test coverage for:
getVolume
AttachDisk
DetachDisk
2016-04-04 17:01:42 -07:00
zhouhaibing089 4758f9953b fix the gcfg typo 2016-03-29 23:32:11 +08:00
k8s-merge-robot 0c06f31cb8 Merge pull request #23340 from justinsb/fix_23339
Auto commit by PR queue bot
2016-03-29 05:04:27 -07:00
Prashanth Balasubramanian 7436fc6261 Default firewall port to TCP when unspecified. 2016-03-28 11:06:19 -07:00
k8s-merge-robot e44ad7a083 Merge pull request #22735 from resouer/throttle-dev
Auto commit by PR queue bot
2016-03-26 06:44:48 -07:00
k8s-merge-robot 4e4ad61260 Merge pull request #23366 from goltermann/vet
Auto commit by PR queue bot
2016-03-24 21:50:56 -07:00
harry 8472cfa214 Refactor throttle into util pkg
Fix missing throttle.go
2016-03-25 08:32:23 +08:00
Jan Safranek e4dc6709de Remove limit of attached AWS devices from kubelet.
Limit of nr. of attached EBS volumes to a node is now enforced by scheduler. It
can be adjusted by KUBE_MAX_PD_VOLS env. variable there.

Therefore we don't need the same check in kubelet. If the system admin wants to
attach more, we should allow it.

Kubelet limit is now 650 attached volumes ('ba'..'zz').
2016-03-23 12:07:16 +01:00
Chris Batey and James Ravn be9ce30897 Change LoadBalancer methods to take api.Service
This is a better abstraction than passing in specific pieces of the
Service that each of the cloudproviders may or may not need. For
instance, many of the providers don't need a region, yet this is passed
in. Similarly many of the providers want a string IP for the load
balancer, but it passes in a converted net ip. Affinity is unused by
AWS. A provider change may also require adding a new parameter which has
an effect on all other cloud provider implementations.

Further, this will simplify adding provider specific load balancer
options, such as with labels or some other metadata. For example, we
could add labels for configuring the details of an AWS elastic load
balancer, such as idle timeout on connections, whether it is
internal or external, cross-zone load balancing, and so on.

Authors: @chbatey, @jsravn
2016-03-23 10:48:11 +00:00
goltermann 34d4eaea08 Fixing several (but not all) go vet errors. Most are around string formatting, or unreachable code. 2016-03-22 17:26:50 -07:00
Justin Santa Barbara 59013f5507 AWS: Fix problems with >2 security groups
The previous logic was incorrect; if we saw two untagged security groups
before seeing the first tagged security, we would incorrectly return an
error.

Fix #23339
2016-03-22 13:00:14 -04:00
k8s-merge-robot 8c02a46c4d Merge pull request #22486 from thockin/update-gcfg-dep
Auto commit by PR queue bot
2016-03-21 18:47:21 -07:00
k8s-merge-robot 0a28a38110 Merge pull request #22280 from justinsb/fix_error_message_formats
Auto commit by PR queue bot
2016-03-21 16:53:38 -07:00
k8s-merge-robot 0fe049f9ff Merge pull request #23019 from alex-mohr/oplimit
Auto commit by PR queue bot
2016-03-19 02:26:56 -07:00
k8s-merge-robot ad7a0da5af Merge pull request #23089 from saad-ali/addZoneToErr
Auto commit by PR queue bot
2016-03-17 15:07:28 -07:00
saadali 26e3d7a27a Add zone to GCE "disk not found" error message 2016-03-16 17:31:12 -07:00
deads2k ab03317d96 support CIDRs in NO_PROXY 2016-03-16 16:22:54 -04:00
Tim Hockin a073c80e45 Use newer home for gcfg package
Switch from obsolete "github.com/scalingdata/gcfg" to "gopkg.in/gcfg.v1".
2016-03-16 08:42:08 -07:00
Alex Mohr 6dc63f805f Add a rate limiter to the GCE cloudprovider
It will poll for operation completion with at most 10 qps to avoid
triggering GCE's rate limits.
2016-03-15 15:47:32 -07:00
Jan Safranek f270cb1b9b Allow 39 atached EBS devices.
AWS has soft support limit for 40 attached EBS devices. Assuming there is just
one root device, use the rest for persistent volumes.

The devices will have name /dev/xvdba - /dev/xvdcm, leaving /dev/sda - /dev/sdz
to the system.

Also, add better error handling and propagate error
"Too many EBS volumes attached to node XYZ" to a pod.
2016-03-15 17:28:59 +01:00
k8s-merge-robot bf1bb5d309 Merge pull request #22899 from justinsb/aws_fix_e2e
Auto commit by PR queue bot
2016-03-14 17:37:13 -07:00
Justin Santa Barbara 3e12e8b7cc AWS e2e: don't try to build a full cloudprovider in e2e
We have previously tried building a full cloudprovider in e2e for AWS;
this wasn't the best idea, because e2e runs on a different machine than
normal operations, and often doesn't even run in AWS.  In turn, this
meant that the cloudprovider had to do extra work and have extra code,
which we would like to get rid of.  Indeed, I got rid of some code which
tolerated not running in AWS, and this broke e2e.
2016-03-12 06:14:45 -05:00
k8s-merge-robot 45064e19d1 Merge pull request #22793 from justinsb/fix_22792
Auto commit by PR queue bot
2016-03-11 20:37:25 -08:00
k8s-merge-robot 26e309c753 Merge pull request #22784 from justinsb/fix_17626
Auto commit by PR queue bot
2016-03-11 20:04:13 -08:00
k8s-merge-robot de7193a095 Merge pull request #22788 from justinsb/fix_22786
Auto commit by PR queue bot
2016-03-11 15:13:06 -08:00
Justin Santa Barbara e40595fa57 AWS volumes: Release disk from attaching map on error
If AWS gives us an actual error (vs just timing out), we know the disk
did not attach, and so we can remove it immediately from the attaching
map.
2016-03-11 11:40:39 -05:00
Justin Santa Barbara 16730aba96 AWS: Tag created EBS volumes with our cluster tag
Fix #22792
2016-03-10 08:13:50 -05:00
Justin Santa Barbara 79b2b7edef AWS EBS: Remove the attached volumes cache
There are known issues with the attached-volume state cache that just aren't
possible to fix with the current interface.

Replace it with a map of the active attach jobs (that was the original
requirement, to avoid a nasty race condition).

This costs us an extra DescribeInstance call on attach/detach, but that
seems worth it if it ends this class of bugs.

Fix #15073
2016-03-10 07:50:35 -05:00
Justin Santa Barbara 0921af4aca AWS: Don't pass empty filters to AWS requests
It gives an error: `The filter 'null' is invalid`

Instead of a zero-length filter list, provide a nil value.

Fix #22786
2016-03-10 07:22:31 -05:00
Justin Santa Barbara 7c82fe7389 AWS: Increase timeout deleting ELB; log remaining security groups
Either ELB is slow to delete (in which case the bumped timeout will
help), or the security groups are otherwise blocked (in which case
logging them will help us track this down).

Fix #17626
2016-03-10 06:57:13 -05:00
Justin Santa Barbara cb818a01d0 AWS: Fix some error messages
Some error messages had incorrect spacing.  Prefer Warningf to Warning,
and fix some of those problems.
2016-03-08 06:29:29 -05:00
Justin Santa Barbara 02e79b9e52 AWS: If we have no subnets, bail out early
We know the ELB call will fail, so we error out early rather than
hitting the API.  Preserves rate limit quota, and also allows us to give
a more self-evident message.

Fix #21993
2016-03-06 09:41:29 -05:00
Justin Santa Barbara 5cf837452b AWS: Fix problems identifying subnets for internal ELBs
We tacitly supported this before, but we broke this with the
public-subnet detection.

Fix #22527
2016-03-06 09:41:29 -05:00
Justin Santa Barbara 43e6602c42 AWS: Fix test failure introduced by rebase 2016-03-05 08:11:30 -05:00
Justin Santa Barbara cff564b1a6 AWS: Remove dead code and fix up comments 2016-03-05 08:09:40 -05:00
Justin Santa Barbara f8e6098e4d AWS: Update tests for refactoring 2016-03-05 08:09:40 -05:00
Justin Santa Barbara af9efa02b4 AWS: Remove getSelfAWSInstance, use field directly
Now that we always populate the local instance, we don't need a getter.
2016-03-05 08:09:40 -05:00
Justin Santa Barbara 8c492c7536 AWS: Don't store the AZ on the cloud
Now we have Ubernetes-Lite, an AWSCloud can span multiple AZs.
2016-03-05 08:09:40 -05:00
Justin Santa Barbara ddb5072a54 AWS: Don't pretend getSelfAWSInstance can return an error
It can't any more; this simplifies calling code.
2016-03-05 08:09:40 -05:00
Justin Santa Barbara 40d0afbb1b AWS: Capture VPC ID into AWSCloud, avoiding requeries
By storing the VPC ID on AWSCloud, we avoid the need to requery or to
pass it around.
2016-03-05 08:09:40 -05:00
Justin Santa Barbara 00b666f853 AWS: Rename getInfo -> describeInstance/describeVolume
Makes it clearer that we are making an AWS API call
2016-03-05 08:09:40 -05:00
Justin Santa Barbara efa68a3590 AWS: Build awsInstance as part of cloud provider creation
We need getSelfAWSInstance to be working anyway; we might as well build
it early, and then we can use its methods to extract e.g. the VPC ID
2016-03-05 08:09:40 -05:00
Justin Santa Barbara 0375fa057f AWS: Refactor newAWSInstance
Now that we can't build an awsInstance from metadata, because of the
PrivateDnsName issue, we might as well simplify the arguments.

Create a 'placeholder' method though - newAWSInstanceFromMetadata - that
documents the desire to use metadata, shows how we would get it, but
links to the bug which explains why we can't use it.
2016-03-05 08:09:40 -05:00
k8s-merge-robot a80f6a7ea6 Merge pull request #21905 from justinsb/aws_wrap_security_group_error
Auto commit by PR queue bot
2016-03-05 01:25:38 -08:00
k8s-merge-robot 264f5786ea Merge pull request #21987 from justinsb/fixx_21895
Auto commit by PR queue bot
2016-03-04 17:07:57 -08:00
k8s-merge-robot 40778f3d2c Merge pull request #22194 from chbatey/testify-for-aws-provider
Auto commit by PR queue bot
2016-03-04 02:29:32 -08:00
Justin Santa Barbara 62e34da125 AWS: Remove extra load balancer security group ingress rules
The ingress CIDRs are going to be dynamic, and in general we don't want
to leave old ingress rules around.

Fix #21895
2016-03-03 18:55:49 -05:00
k8s-merge-robot de72b6be1b Merge pull request #21907 from justinsb/load_balancer_source_ranges
Auto commit by PR queue bot
2016-03-03 14:10:47 -08:00
Justin Santa Barbara cb92133dfa LB Source Ranges: Move validation into API layer
Had to move other things around too to avoid a weird api ->
cloudprovider dependency.

Also adding fixes per code reviews.

(This is a squash of the previously approved commits)
2016-03-03 10:27:17 -05:00
k8s-merge-robot d81d823ca5 Merge pull request #22393 from eparis/blunderbuss
Auto commit by PR queue bot
2016-03-02 18:51:56 -08:00
Eric Paris 5e5a823294 Move blunderbuss assignees into tree 2016-03-02 20:46:32 -05:00
Christopher Batey aebd4c95e1 Use testify for AWS provider
This has two main advantages:

* The use of the mock package to verify API calls against the aws SDK
* Nicer error messages for asserts without having to use if statements
2016-03-01 14:32:45 +00:00
k8s-merge-robot 6e6550a105 Merge pull request #21989 from justinsb/fix_21986
Auto commit by PR queue bot
2016-03-01 03:51:43 -08:00
Justin Santa Barbara 818925cc25 Openstack null-support for load balancer source
We return an error if the user specifies a non 0.0.0.0/0 load balancer
source restriction on OpenStack, where we can't enforce the restriction
(currently).
2016-02-29 19:32:15 -05:00
Justin Santa Barbara 49e1149227 AWS: Add support for load balancer source ranges
This refactors #21431 to pull a lot of the code into cloudprovider so it
can be reused by AWS.

It also changes the name of the annotation to be non-GCE specific:
service.beta.kubernetes.io/load-balancer-source-ranges

Fix #21651
2016-02-29 19:32:08 -05:00
k8s-merge-robot fe03c663d9 Merge pull request #22094 from alex-mohr/routes
Auto commit by PR queue bot
2016-02-29 05:46:51 -08:00
k8s-merge-robot 99eaf73f0c Merge pull request #22019 from sky-uk/aws-handle-implicit-routing-tables
Auto commit by PR queue bot
2016-02-27 06:53:41 -08:00
k8s-merge-robot 394d5da23c Merge pull request #21319 from Clarifai/ensure-lb-servicename
Auto commit by PR queue bot
2016-02-27 02:03:14 -08:00
Vishnu kannan 85efe33c16 Use local metadata server, if available, for GCE compute API invocations.
Signed-off-by: Vishnu kannan <vishnuk@google.com>
2016-02-26 16:54:22 -08:00
Alex Mohr 0816fa2072 Add support for more than 500 results to GCE cloud provider API calls
for Instance.List and Routes.List which we will definitely have
more than 500 of when supporting 1000 nodes.

Add TODOs for other GCE List API calls to do similar fixes.

Add more logging to GCE's routecontroller.go when creating or deleting routes.
2016-02-26 16:03:01 -08:00
James Ravn f568b6511a Handle aws implicit and shared routing tables
Fix the AWS subnet lookup that checks if a subnet is public, which was
missing a few cases:

- Subnets without explicit routing tables, which use the main VPC
  routing table.
- Routing tables not tagged with KubernetesCluster. The filter for this
  is now removed.
2016-02-25 22:52:26 +00:00
Justin Santa Barbara 1cdfc9ad84 AWS: Find the correct security group by looking at tags
Like everything else AWS, we differentiate between k8s-owned security
groups and k8s-not-owned security groups using tags.

When we are setting up the ingress rule for ELBs, pick the security
group that is tagged over any others.

We continue to tolerate a single security group being untagged, but
having multiple security groups without tagging is now an error, as it
leads to undefined behaviour.

We also log at startup if the cluster tag is not defined.

Fix #21986
2016-02-25 11:20:58 -05:00
k8s-merge-robot 2a58c0062d Merge pull request #17913 from jtblin/jtblin/17912-pick-public-subnets
Auto commit by PR queue bot
2016-02-24 23:48:15 -08:00
Justin Santa Barbara e50ae40301 AWS: Wrap AWS error when failing to create security group ingress
All AWS errors should be wrapped in a user-friendly error before
returning.
2016-02-24 14:13:44 -05:00
Rudi Chiarito af4507b1ae Add service name to descriptions for GCE LB resources
Follow up from #20731. I have no way of testing this.

There's an additional group of functions (Get|Delete|Reserve)GlobalStaticIP that can create an IP without the
service description, but those are not called anywhere in the Kubernetes codebase and are probably for the
Ingress project. I'm leaving those alone for now.
2016-02-24 10:21:31 -05:00
k8s-merge-robot 9c1d8bf99d Merge pull request #21399 from sky-uk/disable-ingress-sg
Auto commit by PR queue bot
2016-02-24 00:05:47 -08:00
James Ravn and Yoseph Samuel 9f62e81be5 Disable aws node security group ingress creation
Add aws cloud config:

    [global]
    disableSecurityGroupIngress = true

The aws provider creates an inbound rule per load balancer on the node
security group. However, this can quickly run into the AWS security
group rule limit of 50.

This disables the automatic ingress creation. It requires that the user
has setup a rule that allows inbound traffic on kubelet ports from the
local VPC subnet (so load balancers can access it). E.g.  `10.82.0.0/16
30000-32000`.

Limits: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_Limits.html#vpc-limits-security-groups

Authors: @jsravn, @balooo
2016-02-23 15:24:50 +00:00
Chris Batey, James Ravn and Yoseph Samuel 087ff78cf9 Only find running aws hosts by nodename
When finding instance by node name in AWS, only retrieve running
instances.  Otherwise terminated, old nodes can show up with the same
tag when rebuilding nodes in the cluster.

Another improvement made is to filter instances by the node names
provided, rather than selecting all instances and filtering in code.

Authors: @jsravn, @chbatey, @balooo
2016-02-23 14:47:16 +00:00
k8s-merge-robot 9470a7e61c Merge pull request #21627 from justinsb/fix_11324
Auto commit by PR queue bot
2016-02-23 03:45:10 -08:00
Justin Santa Barbara 7e69426b8b AWS: Only warn about nameless instances if they are running
Otherwise this is just confusing logspam.

Fix #20912
2016-02-22 10:27:13 -05:00
k8s-merge-robot 84891cabad Merge pull request #19335 from justinsb/aws_delay_when_requestlimitexceeded
Auto commit by PR queue bot
2016-02-21 18:38:18 -08:00
Justin Santa Barbara f8af47b645 AWS: Pass globals into Backoff struct
Thanks for the suggestion bprashanth!
2016-02-21 20:17:22 -05:00
k8s-merge-robot 2e3053a204 Merge pull request #21431 from freehan/sourcerange
Auto commit by PR queue bot
2016-02-21 16:14:42 -08:00
Justin Santa Barbara b269e8f43c AWS: Delay all AWS calls when we observe RequestLimitExceeded errors
This applies a cross-request time delay when we observe
RequestLimitExceeded errors, unlike the default library behaviour which
only applies a *per-request* backoff.

Issue #12121
2016-02-21 15:09:42 -05:00
Justin Santa Barbara 22d719018a AWS: Recover if tags missing on security group
In the AWS API (generally) we tag things we create, and then we filter
to find them.  However, creation & tagging are typically two separate
calls.  So there is a chance that we will create an object, but fail to
tag it.

We fix this (done here in the case of security groups, but we can do
this more generally) by retrieving the resource without a tag filter.
If the retrieved resource has the correct tags, great.  If it has the
tags for another cluster, that's a problem, and we raise an error.  If
it has no tags at all, we add the tags.

This only works where the resource is uniquely named (or we can
otherwise retrieve it uniquely).  For security groups, the SG name comes
from the service UUID, so that's unique.

Fixes #11324
2016-02-20 14:58:19 -05:00
k8s-merge-robot f08a8f23c1 Merge pull request #20959 from justinsb/fix_20911
Auto commit by PR queue bot
2016-02-20 00:56:53 -08:00
k8s-merge-robot 6be4417aff Merge pull request #17649 from jtblin/jtblin/17647-aws-elb-creation-no-tags
Auto commit by PR queue bot
2016-02-18 19:41:15 -08:00
Minhan Xia 7ffb123abe add source range support for loadbalancer on gce 2016-02-18 17:05:02 -08:00
Jerome Touffe-Blin 62a8b3d44c Fix #17912 - pick public subnets only on ELB creation 2016-02-16 23:16:38 +11:00
Rudi Chiarito da4ba61232 Fix AWS IPPermission check for case with preexisting groups and ranges 2016-02-15 11:41:18 -05:00
k8s-merge-robot d6b4ff3884 Merge pull request #20909 from Clarifai/instance-type-label
Auto commit by PR queue bot
2016-02-13 18:51:42 -08:00
Rudi Chiarito b3863eae82 Add instance-type label to cloud providers
Fully implemented for AWS and GCE
2016-02-12 15:02:03 -05:00
Jan Safranek 1d0b1c227b Add PV.Name into names of generated GCE/AWS/OSP volumes.
Volume names have now format <cluster-name>-dynamic-<pv-name>.

pv-name is guaranteed to be unique in Kubernetes cluster, adding
<cluster-name> ensures we don't conflict with any running cluster
in the cloud project (kube-controller-manager --cluster-name=XXX).

'kubernetes' is the default cluster name.
2016-02-12 09:46:59 +01:00
Justin Santa Barbara a0093eb503 e2e: Don't try to create a UDP LoadBalancer on AWS
AWS doesn't support type=LoadBalancer with UDP services.  For now, we
simply skip over the test with type=LoadBalancer on AWS for the UDP
service.

Fix #20911
2016-02-10 12:31:18 -05:00
k8s-merge-robot 9520bb5ddf Merge pull request #20731 from Clarifai/ensure-lb-servicename
Auto commit by PR queue bot
2016-02-10 01:25:07 -08:00
k8s-merge-robot 2ec49efd54 Merge pull request #19945 from Clarifai/fix-formatting
Auto commit by PR queue bot
2016-02-09 16:05:00 -08:00
David Pratt 57782459e6 Log missing public IP from AWS metadata. 2016-02-09 12:06:02 -06:00
David Pratt adde7f548f Fix AWS kubelet registration.
This commit allows the AWS cloud provider plugin to work on EC2 instances
that do not have a public IP. The EC2 metadata service returns a 404 for the
'public-ipv4' endpoint for private instances, and the plugin was bubbling this
up as a fatal error.
2016-02-09 11:46:53 -06:00
Rudi Chiarito 5874b0cb9d Pass namespaced service name to cloudprovider's EnsureLoadBalancer
Also has an AWS implementation that plugs the service name into the ELB and SG.
Log the service name under GCE and OpenStack.
Fixes #20668
2016-02-09 06:50:53 -05:00
k8s-merge-robot 318705feb9 Merge pull request #20378 from jtblin/jtblin/11543-aws-cloud-provider-private-hosted-dns-zone-via-api
Auto commit by PR queue bot
2016-02-08 23:43:59 -08:00
k8s-merge-robot fec0d127b3 Merge pull request #15938 from justinsb/aws_ebs_cleanup
Auto commit by PR queue bot
2016-02-08 21:42:52 -08:00
Tim Hockin fecb71420c Demote static IPs ASAP for easier cleanup
This exposed bugs in the IP promotion/demotion logic.
2016-02-06 21:15:06 -08:00
k8s-merge-robot 257c3ad776 Merge pull request #20153 from sky-uk/fix-sg-comparison
Auto commit by PR queue bot
2016-02-06 12:25:26 -08:00
k8s-merge-robot 1b52e0ec3a Merge pull request #20210 from jsafrane/devel/gce-tags
Auto commit by PR queue bot
2016-02-05 21:36:25 -08:00