7826407a2e
Make sure there are no duplicates in etcd member list ( #4025 )
...
* Make sure there are no duplicates in etcd member list
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* fix node names with hyphens
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* use full server name for etcd node name
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
3 years ago
086ca8ba6a
Fix premature etcd shutdown when joining an existing cluster
...
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
3 years ago
928b8531c3
[master] Add `etcd-member-management` controller to K3s ( #4001 )
...
* Initial leader elected etcd member management controller
* Bump etcd to v3.5.0-k3s2
Signed-off-by: Chris Kim <oats87g@gmail.com>
3 years ago
b4d8c641c6
Add exposed metrics listener instead of replacing loopback listener
...
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
3 years ago
29c8b238e5
Replace klog with non-exiting fork
...
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
3 years ago
741ba95b04
Migrate sqlite data to etcd when initializing the cluster
...
Signed-off-by: Darren Shepherd <darren@rancher.com>
3 years ago
a1ec43e0b7
feat: add option to disable s3 over https
...
Signed-off-by: Devin Buhl <devin.kray@gmail.com>
3 years ago
b8add39b07
Bump kine for metrics/tls changes
...
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
3 years ago
e95b75409a
Fix lint failures
...
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
3 years ago
872855015c
Update etcd to v3.5.0
...
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
3 years ago
b23955e835
Fix URL pruning when joining an etcd member ( #3832 )
...
* Fix URL pruning when joining an etcd member
Problem:
Existing member clientURLs were checked if they contain the joining
node's IP. In some edge cases this would prune valid URLs when the
joining IP is a substring match of the only existing member's IP.
Because of this, it was impossible to e.g. join 10.0.0.2 to an existing
node that has an IP of 10.0.0.2X or 10.0.0.2XX:
level=fatal msg="starting kubernetes: preparing server: start managed database:
joining etcd cluster: etcdclient: no available endpoints"
Solution:
Fixed by properly parsing the URLs and comparing the IPs for equality
instead of substring match.
Signed-off-by: Malte Starostik <info@stellaware.de>
3 years ago
dcf0657b20
account for an s3 folder when listing objects ( #3807 )
...
* account for an s3 folder when listing objects
3 years ago
b4eca61aeb
Prevent snapshot commands from creating empty snapshot directory ( #3783 )
...
Signed-off-by: dereknola <derek.nola@suse.com>
3 years ago
bc96ffb5f3
Fix Node stuck at deletion ( #3771 )
...
* fix Node stuck at deletion
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* fix Node stuck at deletion
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
3 years ago
21c8a33647
Introduction of Integration Tests ( #3695 )
...
* Commit of new etcd snapshot integration tests.
* Updated integration github action to not run on doc changes.
* Update Drone runner to only run unit tests
Signed-off-by: dereknola <derek.nola@suse.com>
3 years ago
bba49ea447
Fix to allow prune to correctly cleanup custom named snapshots ( #3649 )
...
Signed-off-by: dereknola <derek.nola@suse.com>
3 years ago
f5fbb9a9a8
Export cli server flags and etcd restoration functions ( #3527 )
...
* Export cli server flags and etfd restoration functions
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* export S3
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
3 years ago
ecbf17e2ed
move object channel defer close to goroutine
...
Signed-off-by: Brian Downs <brian.downs@gmail.com>
4 years ago
254b52077e
add retention default and wire in s3 prune
...
Signed-off-by: Brian Downs <brian.downs@gmail.com>
4 years ago
6ee28214fa
Add the ability to prune etcd snapshots ( #3310 )
...
* add prune subcommand to force rentention policy enforcement
4 years ago
bcd8b67db4
Add the ability to list etcd snapshots ( #3303 )
...
* add ability to list local and s3 etcd snapshots
4 years ago
e998cd110d
Add the ability to delete an etcd snapshot locally or from S3 ( #3277 )
...
* Add the ability to delete a given set of etcd snapshots from the CLI for locally stored and S3 store snapshots.
4 years ago
beb0d8397a
reference node name when needed
...
Signed-off-by: Brian Downs <brian.downs@gmail.com>
4 years ago
c5ad71ce0b
Collect and Store etcd Snapshots and Metadata ( #3239 )
...
* Add the ability to store local etcd snapshots and etcd snapshots stored in an S3 compatible object store in a ConfigMap.
4 years ago
66ed6efd57
Resolve local retention issue when S3 in use.
...
Remove early return preventing local retention policy to be enforced
resulting in N number of snapshots being stored.
Signed-off-by: Brian Downs <brian.downs@gmail.com>
4 years ago
73df65d93a
remove etcd data dir when etcd is disabled ( #3059 )
...
* remove etcd data dir when etcd is disabled
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* fix comment
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* more fixes
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* use debug instead of info logs
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
4 years ago
7c99f8645d
Have Bootstrap Data Stored in etcd at Completed Start ( #3038 )
...
* have state stored in etcd at completed start and remove unneeded code
4 years ago
7cdfaad6ce
Always use static ports for client load-balancers ( #3026 )
...
* Always use static ports for the load-balancers
This fixes an issue where RKE2 kube-proxy daemonset pods were failing to
communicate with the apiserver when RKE2 was restarted because the
load-balancer used a different port every time it started up.
This also changes the apiserver load-balancer port to be 1 below the
supervisor port instead of 1 above it. This makes the apiserver port
consistent at 6443 across servers and agents on RKE2.
Additional fixes below were required to successfully test and use this change
on etcd-only nodes.
* Actually add lb-server-port flag to CLI
* Fix nil pointer when starting server with --disable-etcd but no --server
* Don't try to use full URI as initial load-balancer endpoint
* Fix etcd load-balancer pool updates
* Update dynamiclistener to fix cert updates on etcd-only nodes
* Handle recursive initial server URL in load balancer
* Don't run the deploy controller on etcd-only nodes
4 years ago
4d1f9eda9d
Etcd Snapshot/Restore to/from S3 Compatible Backends ( #2902 )
...
* Add functionality for etcd snapshot/restore to and from S3 compatible backends.
* Update etcd restore functionality to extract and write certificates and configs from snapshot.
4 years ago
d6124981d5
remove etcd member if disable etcd is passed
...
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
4 years ago
5749f66aa3
Add disable flags for control components ( #2900 )
...
* Add disable flags to control components
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* golint
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* more fixes
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* fixes to disable flags
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* Add comments to functions
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* Fix joining problem
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* more fixes
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* golint
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* fix ticker
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* fix role labels
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* more fixes
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
4 years ago
071de833ae
Fix typo in field tag
...
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
4 years ago
06fda7accf
Add functionality to bind custom IP address for Etcd metrics endpoint ( #2750 )
...
* Add functionality to bind custom IP address for Etcd metrics endpoint
Signed-off-by: yuriydzobak <yurii.dzobak@lotusflare.com>
4 years ago
13229019f8
Add ability to perform an etcd on-demand snapshot via cli ( #2819 )
...
* add ability to perform an etcd on-demand snapshot via cli
4 years ago
86f68d5d62
change etcd dir permission if it exists
...
Signed-off-by: MonzElmasry <menna.elmasry@rancher.com>
4 years ago
fadc5a8057
Add tombstone file to etcd and catch errc etcd channel ( #2592 )
...
* Add tombstone file to embedded etcd
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* go mod update
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* fixes
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* more fixes
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* more changes
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* gofmt and goimports
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* go mod update
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* go lint
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* go lint
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* go mod tidy
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
4 years ago
523ccaf3f2
Merge pull request #2448 from MonzElmasry/new_b
...
Make etcd use node private ip
4 years ago
e8436cc76b
Make etcd use node private ip
...
Signed-off-by: MonzElmasry <menna.elmasry@rancher.com>
4 years ago
fcd18d1b6e
skip node delete from removed member ( #2413 )
...
* skip node delete from removed member
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* use grpc errors
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* go imports
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* exit if node is the etcd that being removed
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
4 years ago
de18528412
Make etcd voting members responsible for managing learners ( #2399 )
...
* Set etcd timeouts using values from k8s instead of etcdctl
Fix for one of the warnings from #2303
* Use etcd zap logger instead of deprecated capsnlog
Fix for one of the warnings from #2303
* Remove member self-promotion code paths
* Add learner promotion tracking code
* Fix RaftAppliedIndex progress check
* Remove ErrGRPCKeyNotFound check
This is not used by v3 API - it just returns a response with 0 KVs.
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
4 years ago
373449ec0a
Allow for multiple etcd snapshot restoration ( #2307 )
...
* add reset tmp file
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* go imports
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* fix multiple lines string
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* fix typo
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* use resetFile function
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
4 years ago
703ba5cde7
Add a bunch of doc comments
...
Also change identical error messages to clarify where problems are
occurring.
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
4 years ago
f59e8fc21b
Fix etcd directory permissions
...
Silences warning on startup about insecure directory permissions
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
4 years ago
ee99660a96
Rename etcd directory helpers to reduce confusion about which datadir we're talking about
...
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
4 years ago
97eb28a01a
Remove unnecessary listener arg from managed DB setup
...
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
4 years ago
42bba04651
Skip etcd snapshots if the local endpoint is still a learner ( #2295 )
...
* Don't take snapshots if the local endpoint is still a learner
* Configure timeouts for etcd client dialer
4 years ago
ba70c41cce
Initial Logging Output Update ( #2246 )
...
This attempts to update logging statements to make them consistent
through out the code base. It also adds additional context to messages
where possible, simplifies messages, and updates level where necessary.
4 years ago
46fe57d7e9
reset etcd name on cluster reset ( #2284 )
...
* reset etcd name on cluster reset
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* gofmt
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
4 years ago
866dc94cea
Galal hussein etcd backup restore ( #2154 )
...
* Add etcd snapshot and restore
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* fix error logs
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* goimports
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* fix flag describtion
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* Add disable snapshot and retention
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* use creation time for snapshot retention
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* unexport method, update var name
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* adjust snapshot flags
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* update var name, string concat
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* revert previous change, create constants
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* update
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* updates
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* type assertion error checking
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* update
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* update
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* update
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* pr remediation
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* pr remediation
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* pr remediation
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* pr remediation
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* pr remediation
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* updates
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* updates
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* simplify logic, remove unneeded function
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* update flags
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* update flags
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* add comment
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* exit on restore completion, update flag names, move retention check
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* exit on restore completion, update flag names, move retention check
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* exit on restore completion, update flag names, move retention check
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* update disable snapshots flag and field names
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* move function
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* update field names
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* update var and field names
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* update var and field names
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* update defaultSnapshotIntervalMinutes to 12 like rke
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* update directory perms
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* update etc-snapshot-dir usage
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* update interval to 12 hours
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* fix usage typo
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* add cron
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* add cron
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* add cron
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* wire in cron
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* wire in cron
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* wire in cron
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* wire in cron
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* wire in cron
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* wire in cron
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* wire in cron
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* update deps target to work, add build/data target for creation, and generate
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* remove dead make targets
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* error handling, cluster reset functionality
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* error handling, cluster reset functionality
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* update
Signed-off-by: Brian Downs <brian.downs@gmail.com>
* remove intermediate dapper file
Signed-off-by: Brian Downs <brian.downs@gmail.com>
Co-authored-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
4 years ago
169ee63907
Add etcd members as learners ( #2066 )
...
* Add etcd members as learners
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
* Ignore errors in promote member
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
4 years ago
Hussein Galal