* Move registries.yaml handling out to rancher/wharfie
* Add system-default-registry support
* Add CLI support for kubelet image credential providers
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
Problem:
Only the client CA is passed to the kube-controller-manager and
therefore CSRs with the signer name "kubernetes.io/kubelet-serving" are
signed with the client CA. Serving certificates must be signed with the
server CA otherwise e.g. "kubectl logs" fails with the error message
"x509: certificate signed by unknown authority".
Solution:
Instead of providing only one CA via the kube-controller-manager
parameter "--cluster-signing-cert-file", the corresponding CA for every
signer is set with the parameters
"--cluster-signing-kube-apiserver-client-cert-file",
"--cluster-signing-kubelet-client-cert-file",
"--cluster-signing-kubelet-serving-cert-file", and
"--cluster-signing-legacy-unknown-cert-file".
Signed-off-by: Siegfried Weber <mail@siegfriedweber.net>
The kube-apiserver cert should have the same SANs in the same order,
excluding the extra user-configured SANs since this will only be used
in-cluster.
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
* Update Kubernetes to v1.21.0
* Update to golang v1.16.2
* Update dependent modules to track with upstream
* Switch to upstream flannel
* Track changes to upstream cloud-controller-manager and FeatureGates
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
Now rootless mode can be used with cgroup v2 resource limitations.
A pod is executed in a cgroup like "/user.slice/user-1001.slice/user@1001.service/k3s-rootless.service/kubepods/podd0eb6921-c81a-4214-b36c-d3b9bb212fac/63b5a253a1fd4627da16bfce9bec58d72144cf30fe833e0ca9a6d60ebf837475".
This is accomplished by running `kubelet` in a cgroup namespace, and enabling `cgroupfs` driver for the cgroup hierarchy delegated by systemd.
To enable cgroup v2 resource limitation, `k3s server --rootless` needs to be launched as `systemctl --user` service.
Please see the comment lines in `k3s-rootless.service` for the usage.
Running `k3s server --rootless` via a terminal is not supported.
When it really needs to be launched via a terminal, `systemd-run --user -p Delegate --tty` needs to be prepended to create a systemd scope.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
* Add functionality for etcd snapshot/restore to and from S3 compatible backends.
* Update etcd restore functionality to extract and write certificates and configs from snapshot.
Adds support for retagging images to appear to have been sourced from
one or more additional registries as they are imported from the tarball.
This is intended to support RKE2 use cases with system-default-registry
where the images need to appear to have been pulled from a registry
other than docker.io.
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
Fix issue 900
cgroup2 support was introduced in PR 2584, but got broken in f3de60ff31
It was failing with "F1210 19:13:37.305388 4955 server.go:181] cannot set feature gate SupportPodPidsLimit to false, feature is locked to true"
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
We're not setting ``--service-account-issuer` to a https URL, which causes an
error message at startup when the feature gate is enabled. From the
docs on that flag:
> If this option is not a valid URI per the OpenID Discovery 1.0 spec, the
> ServiceAccountIssuerDiscovery feature will remain disabled, even if the
> feature gate is set to true. It is highly recommended that this value
> comply with the OpenID spec:
> https://openid.net/specs/openid-connect-discovery-1_0.html. In practice,
> this means that service-account-issuer must be an https URL. It is also
> highly recommended that this URL be capable of serving OpenID discovery
> documents at {service-account-issuer}/.well-known/openid-configuration.
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
As per documentation, the cloud-provider flag should not be passed to
controller-manager when using cloud-controller. However, the legacy
cloud-related controllers still need to be explicitly disabled to
prevent errors from being logged.
Fixing this also prevents controller-manager from creating the
cloud-controller-manager service account that needed extra RBAC.
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
Resolves warning 2 from #2471.
As per https://github.com/kubernetes/cloud-provider/issues/12 the
ClusterID requirement was never really followed through on, so the
flag is probably going to be removed in the future.
One side-effect of this is that the core k8s cloud-controller-manager
also wants to watch nodes, and needs RBAC to do so.
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
* Set etcd timeouts using values from k8s instead of etcdctl
Fix for one of the warnings from #2303
* Use etcd zap logger instead of deprecated capsnlog
Fix for one of the warnings from #2303
* Remove member self-promotion code paths
* Add learner promotion tracking code
* Fix RaftAppliedIndex progress check
* Remove ErrGRPCKeyNotFound check
This is not used by v3 API - it just returns a response with 0 KVs.
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
Since we're replacing the k3s rolebindings.yaml in rke2, we should allow
renaming this so that we can use the white-labeled name downstream.
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
* cli: add --selinux flag to agent/server sub-cmds
Introduces --selinux flag to affirmatively enable SELinux in containerd.
Deprecates --disable-selinux flag which now defaults to true which
auto-detection of SELinux configuration for containerd is no longer
supported. Specifying both --selinux and --disable-selinux will result
in an error message encouraging you to pick a side.
* Update pkg/agent/containerd/containerd.go
update log warning message about enabled selinux host but disabled runtime
Co-authored-by: Brad Davidson <brad@oatmail.org>
Signed-off-by: Jacob Blain Christen <jacob@rancher.com>
Before this change, k3s configured the scheduler and controller's
insecure ports to listen on 0.0.0.0. Those ports include pprof, which
provides a DoS vector at the very least.
These ports are only enabled for componentstatus checks in the first
place, and componentstatus is hardcoded to only do the check on
localhost anyway (see
https://github.com/kubernetes/kubernetes/blob/v1.18.2/pkg/registry/core/rest/storage_core.go#L341-L344),
so there shouldn't be any downside to switching them to listen only on
localhost.
This will watch for the following kube-apiserver-arg variables and apply
them to the k3s kube-apiserver https listener.
--kube-apiserver-arg=tls-cipher-suites=XXXXXXX
--kube-apiserver-arg=tls-min-version=XXXXXXX
In rke2 everything is a static pod so this causes a chicken and egg situation
in which we need the kubelet running before the kube-apiserver can be
launched. By starting the apiserver in the background this allows us to
do this odd bootstrapping.
In k3s today the kubernetes API and the /v1-k3s API are combined into
one http server. In rke2 we are running unmodified, non-embedded Kubernetes
and as such it is preferred to run k8s and the /v1-k3s API on different
ports. The /v1-k3s API port is called the SupervisorPort in the code.
To support this separation of ports a new shim was added on the client in
then pkg/agent/proxy package that will launch two load balancers instead
of just one load balancer. One load balancer for 6443 and the other
for 9345 (which is the supervisor port).
The containerd flag was accidentally added to kubelet and is
deprecated, but needed for cadvisor to properly connect with
the k3s containerd socket, so adding for now.
On my environment, the name=systemd entry in /proc/self/cgroup
looks like:
13:name=systemd:/docker/917b388b40c70b17a3283d852d38bfcdc84d1bf8242e32a779eacd98a610e499
Kubelet periodically complains like:
E0802 06:42:52.667123 1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/917b388b40c70b17a3283d852d38bfcdc84d1bf8242e32a779eacd98a610e499/kube-proxy": failed to get cgroup stats for "/docker/917b388b40c70b17a3283d852d38bfcdc84d1bf8242e32a779eacd98a610e499/kube-proxy": failed to get container info for "/docker/917b388b40c70b17a3283d852d38bfcdc84d1bf8242e32a779eacd98a610e499/kube-proxy": unknown container "/docker/917b388b40c70b17a3283d852d38bfcdc84d1bf8242e32a779eacd98a610e499/kube-proxy"
Allow the kubelet resolv-conf flag to be set, or automatically
discovered from /etc/resolv.conf & /run/systemd/resolve/resolv.conf if
no loopback devices are present, or create our own which points to
nameserver 8.8.8.8
Brad Davidson