The metrics structure passed to volume stat calculator
can contain real stats on subset of metrics fields. For example,
the metrics structure filled by a CSI driver can have
either INODES or BYTES filled, IOW it a valid return.
In such cases the volume stat calculator panic with below
trace:
0516 21:36:19.013143 14452 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/home/hchiramm/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/home/hchiramm/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/home/hchiramm/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/panic.go:522
/usr/local/go/src/runtime/panic.go:82
/usr/local/go/src/runtime/signal_unix.go:390
/home/hchiramm/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/api/resource/quantity.go:697
/home/hchiramm/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/api/resource/quantity.go:685
/home/hchiramm/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/volume_stat_calculator.go:144
/home/hchiramm/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/volume_stat_calculator.go:125
/home/hchiramm/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/volume_stat_calculator.go:65
/home/hchiramm/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152
/home/hchiramm/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153
/usr/local/go/src/runtime/asm_amd64.s:1337
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
When adding a new etcd member the etcd cluster can enter a state
of vote, where any new members added at the exact same time will
fail with an error right away.
Implement exponential backoff retry around the MemberAdd call.
This solves a kubeadm problem when concurrently joining
control-plane nodes with stacked etcd members.
From experiment, a few retries with milliseconds apart are
sufficient to achieve the concurrent join of a 3xCP cluster.
Apply the same backoff to MemberRemove in case the concurrent
removal of members fails for similar reasons.
Automated cherry pick of #79094: fix kubelet can not delete orphaned pod directory when "/var/lib/kubelet/pods" directory symbolically links to another device's directory
The way that ReplicaCalculator signals it's desire to not change the
current scale is by returning the current scale. However the current
scale is from scale.Status.Replicas and can be larger than
scale.Spec.Replicas (e.g. during Deployment rollout with configured
surge). This causes a positive feedback loop because
scale.Status.Replicas is written back into scale.Spec.Replicas,
further increasing the current scale.
This PR fixes the feedback loop by plumbing the replica count from
spec through horizontal.go and replica_calculator.go so the calculator
can punt with the right value.
This fixes the issue where "childFD" from syscall.Openat is assigned to
a local variable inside the for loop, instead of the correct one in the
function scope. This results in that when trying to close the "childFD"
in the function scope, it will be equal to "-1", instead of the correct
value.
Kubernetes Prow Robot