History

Kubernetes Submit Queue 5a54555f59 Merge pull request #63049 from andrewsykim/kcm-nodeipam Automatic merge from submit-queue (batch tested with PRs 63049, 59731). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. re-enable nodeipam in kube-controller-manager What this PR does / why we need it: Re-enables nodeipam controller for external clouds. Also does a small refactor so that we don't need to pass in `allocateNodeCidr` into the controller. In v1.10 we made a change (`9187b343e1 (diff-f11913dc67d80d36b3d06a93f61c49cf)` in https://github.com/kubernetes/kubernetes/pull/57492) where nodeipam would be disabled for any cluster that sets `--cloud-provider=external`. The original intention behind this was that the nodeipam controller is cloud specific for some clouds (only GCE at the moment) so it should be moved to the CCM (cloud controller manager). After some discussions with wg-cloud-provider it makes sense to re-enable nodeipam controller in KCM and have GCE CCM enable its own cloud-specific IPAM controller as part of [Initialize()](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/cloud.go#L33-L35). This would allow for GCE to run nodeipam in both KCM (by setting --cloud-provider=gce and --allocate-node-cidr) and in the CCM (once implemented in `Initialize()`) without disabling nodeipam in the KCM for all external clouds and avoids having to implement nodeipam in CCM. Which issue(s) this PR fixes (optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged): Fixes # Special notes for your reviewer: Release note: ```release-note Re-enable nodeipam controller for external clouds. ```		2018-05-11 11:07:12 -07:00
..
BUILD	update tests to be specific about the versions they are testing instead of floating	2018-05-01 13:18:41 -04:00
README.md	Support custom test configurations	2018-04-02 10:29:36 -07:00
cloud.go	…
ipam_test.go	Merge pull request #63049 from andrewsykim/kcm-nodeipam	2018-05-11 11:07:12 -07:00
main_test.go	Support custom test configurations	2018-04-02 10:29:36 -07:00
results.go	Support custom test configurations	2018-04-02 10:29:36 -07:00
test-performance.sh	Support custom test configurations	2018-04-02 10:29:36 -07:00
util.go	update tests to be specific about the versions they are testing instead of floating	2018-05-01 13:18:41 -04:00

README.md

IPAM Performance Test

Motivation

We wanted to be able to test the behavior of the IPAM controller's under various scenarios, by mocking and monitoring the edges that the controller interacts with. This has the following goals:

Save time on testing
To simulate various behaviors cheaply
To observe and model the ideal behavior of the IPAM controller code

Currently the test runs through the 4 different IPAM controller modes for cases where the kube API QPS is a) equal to and b) significantly less than the number of nodes being added to observe and quantify behavior.

How to run

# In kubernetes root path
make generated_files

cd test/integration/ipamperf
./test-performance.sh

The runner scripts support a few different options:

./test-performance.sh -h
usage: ./test-performance.sh [-h] [-d] [-r <pattern>] [-o <filename>]
usage: ./test-performance.sh <options>
 -h display this help message
 -d enable debug logs in tests
 -r <pattern> regex pattern to match for tests
 -o <filename> file to write JSON formatted results to
 -p <id> enable cpu and memory profiles, output written to mem-<id>.out and cpu-<id>.out
 -c enable custom test configuration
 -a <name> allocator name, one of RangeAllocator, CloudAllocator, IPAMFromCluster, IPAMFromCloud
 -k <num> api server qps for allocator
 -n <num> number of nodes to simulate
 -m <num> api server qps for node creation
 -l <num> gce cloud endpoint qps

The tests follow the pattern TestPerformance/{AllocatorType}-KubeQPS{X}-Nodes{Y}, where AllocatorType is one of

RangeAllocator
IPAMFromCluster
CloudAllocator
IPAMFromCloud

and X represents the QPS configured for the kubernetes API client, and Y is the number of nodes to create.

The -d flags set the -v level for glog to 6, enabling nearly all of the debug logs in the code.

So to run the test for CloudAllocator with 10 nodes, one can run

./test-performance.sh -r /CloudAllocator.*Nodes10$

At the end of the test, a JSON format of the results for all the tests run is printed. Passing the -o option allows for also saving this JSON to a named file.

Profiling the code

It's possible to get the CPU and memory profiles of code during test execution by using the -p option. The CPU and memory profiles are generated in the same directory with the file names set to cpu-<id>.out and cpu-<id>.out, where <id> is the argument value. Typicall pattern is to put in the number of nodes being simulated as the id, or 'all' in case running the full suite.

Custom Test Configuration

It's also possible to run a custom test configuration by passing the -c option. With this option, it then possible to specify the number of nodes to simulate and the API server qps values for creation, IPAM allocation and cloud endpoint, along with the allocator name to run. The defaults values for the qps parmeters are 30 for IPAM allocation, 100 for node creation and 30 for the cloud endpoint, and the default allocator is the RangeAllocator.

Code Organization

The core of the tests are defined in ipam_test.go, using the t.Run() helper to control parallelism as we want to able to start the master once. cloud.go contains the mock of the cloud server endpoint and can be configured to behave differently as needed by the various modes. The tracking of the node behavior and creation of the test results data is in results.go.