github
/
consul
mirror of https://github.com/hashicorp/consul
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
consul/agent/agent_test.go

4883 lines
130 KiB
Go
Raw Normal View History

Adding more agent tests
2013-12-23 21:52:10 +00:00
package agent
import (
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
"bytes"
agent: fix several data races and bugs related to node-local alias checks (#5876) The observed bug was that a full restart of a consul datacenter (servers and clients) in conjunction with a restart of a connect-flavored application with bring-your-own-service-registration logic would very frequently cause the envoy sidecar service check to never reflect the aliased service. Over the course of investigation several bugs and unfortunate interactions were corrected: (1) local.CheckState objects were only shallow copied, but the key piece of data that gets read and updated is one of the things not copied (the underlying Check with a Status field). When the stock code was run with the race detector enabled this highly-relevant-to-the-test-scenario field was found to be racy. Changes: a) update the existing Clone method to include the Check field b) copy-on-write when those fields need to change rather than incrementally updating them in place. This made the observed behavior occur slightly less often. (2) If anything about how the runLocal method for node-local alias check logic was ever flawed, there was no fallback option. Those checks are purely edge-triggered and failure to properly notice a single edge transition would leave the alias check incorrect until the next flap of the aliased check. The change was to introduce a fallback timer to act as a control loop to double check the alias check matches the aliased check every minute (borrowing the duration from the non-local alias check logic body). This made the observed behavior eventually go away when it did occur. (3) Originally I thought there were two main actions involved in the data race: A. The act of adding the original check (from disk recovery) and its first health evaluation. B. The act of the HTTP API requests coming in and resetting the local state when re-registering the same services and checks. It took awhile for me to realize that there's a third action at work: C. The goroutines associated with the original check and the later checks. The actual sequence of actions that was causing the bad behavior was that the API actions result in the original check to be removed and re-added _without waiting for the original goroutine to terminate_. This means for brief windows of time during check definition edits there are two goroutines that can be sending updates for the alias check status. In extremely unlikely scenarios the original goroutine sees the aliased check start up in `critical` before being removed but does not get the notification about the nearly immediate update of that check to `passing`. This is interlaced wit the new goroutine coming up, initializing its base case to `passing` from the current state and then listening for new notifications of edge triggers. If the original goroutine "finishes" its update, it then commits one more write into the local state of `critical` and exits leaving the alias check no longer reflecting the underlying check. The correction here is to enforce that the old goroutines must terminate before spawning the new one for alias checks.
2019-05-24 18:36:56 +00:00
"context"
agent: enable reloading of tls config (#5419) This PR introduces reloading tls configuration. Consul will now be able to reload the TLS configuration which previously required a restart. It is not yet possible to turn TLS ON or OFF with these changes. Only when TLS is already turned on, the configuration can be reloaded. Most importantly the certificates and CAs.
2019-03-13 09:29:06 +00:00
"crypto/tls"
Add an AutoEncrypt “integration” test Also fix a bug where Consul could segfault if TLS was enabled but no client certificate was provided. How no one has reported this as a problem I am not sure.
2020-06-30 17:52:57 +00:00
"crypto/x509"
Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.
2020-06-10 20:47:35 +00:00
"encoding/base64"
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
"encoding/json"
Adding more agent tests
2013-12-23 21:52:10 +00:00
"fmt"
"io/ioutil"
checks: when a service does not exists in an alias, consider it failing (#7384) In current implementation of Consul, check alias cannot determine if a service exists or not. Because a service without any check is semantically considered as passing, so when no healthchecks are found for an agent, the check was considered as passing. But this make little sense as the current implementation does not make any difference between: * a non-existing service (passing) * a service without any check (passing as well) In order to make it work, we have to ensure that when a check did not find any healthcheck, the service does indeed exists. If it does not, lets consider the check as failing.
2020-06-04 12:50:52 +00:00
"math/rand"
Implement advertise_addrs for SerfLan, SerfWan and RPC. Fixes #550. This will make it possible to configure the advertised adresses for SerfLan, SerfWan and RPC. It will enable multiple consul clients on a single host which is very useful in a container environment. This option might override advertise_addr and advertise_addr_wan depending on the configuration. It will be configureable with advertise_addrs. Example: { "advertise_addrs": { "serf_lan": "10.0.120.91:4424", "serf_wan": "201.20.10.61:4423", "rpc": "10.20.10.61:4424" } }
2015-06-05 11:44:42 +00:00
"net"
Adds enable_agent_tls_for_checks configuration option which allows (#3661) HTTP health checks for services requiring 2-way TLS to be checked using the agent's credentials.
2017-11-08 02:22:09 +00:00
"net/http"
"net/http/httptest"
Adding more agent tests
2013-12-23 21:52:10 +00:00
"os"
agent: test loading keyring files for client and server
2014-09-18 05:31:32 +00:00
"path/filepath"
agent: fix several data races and bugs related to node-local alias checks (#5876) The observed bug was that a full restart of a consul datacenter (servers and clients) in conjunction with a restart of a connect-flavored application with bring-your-own-service-registration logic would very frequently cause the envoy sidecar service check to never reflect the aliased service. Over the course of investigation several bugs and unfortunate interactions were corrected: (1) local.CheckState objects were only shallow copied, but the key piece of data that gets read and updated is one of the things not copied (the underlying Check with a Status field). When the stock code was run with the race detector enabled this highly-relevant-to-the-test-scenario field was found to be racy. Changes: a) update the existing Clone method to include the Check field b) copy-on-write when those fields need to change rather than incrementally updating them in place. This made the observed behavior occur slightly less often. (2) If anything about how the runLocal method for node-local alias check logic was ever flawed, there was no fallback option. Those checks are purely edge-triggered and failure to properly notice a single edge transition would leave the alias check incorrect until the next flap of the aliased check. The change was to introduce a fallback timer to act as a control loop to double check the alias check matches the aliased check every minute (borrowing the duration from the non-local alias check logic body). This made the observed behavior eventually go away when it did occur. (3) Originally I thought there were two main actions involved in the data race: A. The act of adding the original check (from disk recovery) and its first health evaluation. B. The act of the HTTP API requests coming in and resetting the local state when re-registering the same services and checks. It took awhile for me to realize that there's a third action at work: C. The goroutines associated with the original check and the later checks. The actual sequence of actions that was causing the bad behavior was that the API actions result in the original check to be removed and re-added _without waiting for the original goroutine to terminate_. This means for brief windows of time during check definition edits there are two goroutines that can be sending updates for the alias check status. In extremely unlikely scenarios the original goroutine sees the aliased check start up in `critical` before being removed but does not get the notification about the nearly immediate update of that check to `passing`. This is interlaced wit the new goroutine coming up, initializing its base case to `passing` from the current state and then listening for new notifications of edge triggers. If the original goroutine "finishes" its update, it then commits one more write into the local state of `critical` and exits leaving the alias check no longer reflecting the underlying check. The correction here is to enforce that the old goroutines must terminate before spawning the new one for alias checks.
2019-05-24 18:36:56 +00:00
"strconv"
Fix test import
2017-01-24 02:11:13 +00:00
"strings"
Added ratelimit to handle throtling cache (#8226) This implements a solution for #7863 It does: Add a new config cache.entry_fetch_rate to limit the number of calls/s for a given cache entry, default value = rate.Inf Add cache.entry_fetch_max_burst size of rate limit (default value = 2) The new configuration now supports the following syntax for instance to allow 1 query every 3s: command line HCL: -hcl 'cache = { entry_fetch_rate = 0.333}' in JSON { "cache": { "entry_fetch_rate": 0.333 } }
2020-07-27 21:11:11 +00:00
"sync"
Adding more agent tests
2013-12-23 21:52:10 +00:00
"testing"
"time"
agent: Fixing port collision in tests
2014-11-19 19:51:25 +00:00
Ensure certificates retrieved through the cache get persisted with auto-config (#8409)
2020-07-30 15:37:18 +00:00
"github.com/golang/protobuf/jsonpb"
wan federation via mesh gateways (#6884) This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch: There are several distinct chunks of code that are affected: * new flags and config options for the server * retry join WAN is slightly different * retry join code is shared to discover primary mesh gateways from secondary datacenters * because retry join logic runs in the *agent* and the results of that operation for primary mesh gateways are needed in the *server* there are some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur at multiple layers of abstraction just to pass the data down to the right layer. * new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers * the function signature for RPC dialing picked up a new required field (the node name of the destination) * several new RPCs for manipulating a FederationState object: `FederationState:{Apply,Get,List,ListMeshGateways}` * 3 read-only internal APIs for debugging use to invoke those RPCs from curl * raft and fsm changes to persist these FederationStates * replication for FederationStates as they are canonically stored in the Primary and replicated to the Secondaries. * a special derivative of anti-entropy that runs in secondaries to snapshot their local mesh gateway `CheckServiceNodes` and sync them into their upstream FederationState in the primary (this works in conjunction with the replication to distribute addresses for all mesh gateways in all DCs to all other DCs) * a "gateway locator" convenience object to make use of this data to choose the addresses of gateways to use for any given RPC or gossip operation to a remote DC. This gets data from the "retry join" logic in the agent and also directly calls into the FSM. * RPC (`:8300`) on the server sniffs the first byte of a new connection to determine if it's actually doing native TLS. If so it checks the ALPN header for protocol determination (just like how the existing system uses the type-byte marker). * 2 new kinds of protocols are exclusively decoded via this native TLS mechanism: one for ferrying "packet" operations (udp-like) from the gossip layer and one for "stream" operations (tcp-like). The packet operations re-use sockets (using length-prefixing) to cut down on TLS re-negotiation overhead. * the server instances specially wrap the `memberlist.NetTransport` when running with gateway federation enabled (in a `wanfed.Transport`). The general gist is that if it tries to dial a node in the SAME datacenter (deduced by looking at the suffix of the node name) there is no change. If dialing a DIFFERENT datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh gateways to eventually end up in a server's :8300 port. * a new flag when launching a mesh gateway via `consul connect envoy` to indicate that the servers are to be exposed. This sets a special service meta when registering the gateway into the catalog. * `proxycfg/xds` notice this metadata blob to activate additional watches for the FederationState objects as well as the location of all of the consul servers in that datacenter. * `xds:` if the extra metadata is in place additional clusters are defined in a DC to bulk sink all traffic to another DC's gateways. For the current datacenter we listen on a wildcard name (`server.<dc>.consul`) that load balances all servers as well as one mini-cluster per node (`<node>.server.<dc>.consul`) * the `consul tls cert create` command got a new flag (`-node`) to help create an additional SAN in certs that can be used with this flavor of federation.
2020-03-09 20:59:02 +00:00
"github.com/google/tcpproxy"
agent: cache notifications work after error if the underlying RPC returns index=1 (#6547) Fixes #6521 Ensure that initial failures to fetch an agent cache entry using the notify API where the underlying RPC returns a synthetic index of 1 correctly recovers when those RPCs resume working. The bug in the Cache.notifyBlockingQuery used to incorrectly "fix" the index for the next query from 0 to 1 for all queries, when it should have not done so for queries that errored. Also fixed some things that made debugging difficult: - config entry read/list endpoints send back QueryMeta headers - xds event loops don't swallow the cache notification errors
2019-09-26 15:42:17 +00:00
"github.com/hashicorp/consul/agent/cache"
cachetype "github.com/hashicorp/consul/agent/cache-types"
Decouple the code that executes checks from the agent
2017-10-25 09:18:07 +00:00
"github.com/hashicorp/consul/agent/checks"
Persist proxies from config files Also change how loadProxies works. Now it will load all persisted proxies into a map, then when loading config file proxies will look up the previous proxy token in that map.
2018-07-17 20:16:43 +00:00
"github.com/hashicorp/consul/agent/config"
Fixed many tests after rebase. Some still failing and seem unrelated to any connect changes.
2018-05-10 16:04:33 +00:00
"github.com/hashicorp/consul/agent/connect"
agent: move agent/consul/structs to agent/structs
2017-07-06 10:34:00 +00:00
"github.com/hashicorp/consul/agent/structs"
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
"github.com/hashicorp/consul/api"
Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.
2020-06-10 20:47:35 +00:00
"github.com/hashicorp/consul/internal/go-sso/oidcauth/oidcauthtest"
wan federation via mesh gateways (#6884) This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch: There are several distinct chunks of code that are affected: * new flags and config options for the server * retry join WAN is slightly different * retry join code is shared to discover primary mesh gateways from secondary datacenters * because retry join logic runs in the *agent* and the results of that operation for primary mesh gateways are needed in the *server* there are some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur at multiple layers of abstraction just to pass the data down to the right layer. * new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers * the function signature for RPC dialing picked up a new required field (the node name of the destination) * several new RPCs for manipulating a FederationState object: `FederationState:{Apply,Get,List,ListMeshGateways}` * 3 read-only internal APIs for debugging use to invoke those RPCs from curl * raft and fsm changes to persist these FederationStates * replication for FederationStates as they are canonically stored in the Primary and replicated to the Secondaries. * a special derivative of anti-entropy that runs in secondaries to snapshot their local mesh gateway `CheckServiceNodes` and sync them into their upstream FederationState in the primary (this works in conjunction with the replication to distribute addresses for all mesh gateways in all DCs to all other DCs) * a "gateway locator" convenience object to make use of this data to choose the addresses of gateways to use for any given RPC or gossip operation to a remote DC. This gets data from the "retry join" logic in the agent and also directly calls into the FSM. * RPC (`:8300`) on the server sniffs the first byte of a new connection to determine if it's actually doing native TLS. If so it checks the ALPN header for protocol determination (just like how the existing system uses the type-byte marker). * 2 new kinds of protocols are exclusively decoded via this native TLS mechanism: one for ferrying "packet" operations (udp-like) from the gossip layer and one for "stream" operations (tcp-like). The packet operations re-use sockets (using length-prefixing) to cut down on TLS re-negotiation overhead. * the server instances specially wrap the `memberlist.NetTransport` when running with gateway federation enabled (in a `wanfed.Transport`). The general gist is that if it tries to dial a node in the SAME datacenter (deduced by looking at the suffix of the node name) there is no change. If dialing a DIFFERENT datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh gateways to eventually end up in a server's :8300 port. * a new flag when launching a mesh gateway via `consul connect envoy` to indicate that the servers are to be exposed. This sets a special service meta when registering the gateway into the catalog. * `proxycfg/xds` notice this metadata blob to activate additional watches for the FederationState objects as well as the location of all of the consul servers in that datacenter. * `xds:` if the extra metadata is in place additional clusters are defined in a DC to bulk sink all traffic to another DC's gateways. For the current datacenter we listen on a wildcard name (`server.<dc>.consul`) that load balances all servers as well as one mini-cluster per node (`<node>.server.<dc>.consul`) * the `consul tls cert create` command got a new flag (`-node`) to help create an additional SAN in certs that can be used with this flavor of federation.
2020-03-09 20:59:02 +00:00
"github.com/hashicorp/consul/ipaddr"
Update TestAgent_GetCoordinate The old test case was a very specific regresion test for a case that is no longer possible. Replaced with a new test that checks the default coordinate is returned.
2020-06-24 16:56:46 +00:00
"github.com/hashicorp/consul/lib"
Ensure certificates retrieved through the cache get persisted with auto-config (#8409)
2020-07-30 15:37:18 +00:00
"github.com/hashicorp/consul/proto/pbautoconf"
agent: fix several data races and bugs related to node-local alias checks (#5876) The observed bug was that a full restart of a consul datacenter (servers and clients) in conjunction with a restart of a connect-flavored application with bring-your-own-service-registration logic would very frequently cause the envoy sidecar service check to never reflect the aliased service. Over the course of investigation several bugs and unfortunate interactions were corrected: (1) local.CheckState objects were only shallow copied, but the key piece of data that gets read and updated is one of the things not copied (the underlying Check with a Status field). When the stock code was run with the race detector enabled this highly-relevant-to-the-test-scenario field was found to be racy. Changes: a) update the existing Clone method to include the Check field b) copy-on-write when those fields need to change rather than incrementally updating them in place. This made the observed behavior occur slightly less often. (2) If anything about how the runLocal method for node-local alias check logic was ever flawed, there was no fallback option. Those checks are purely edge-triggered and failure to properly notice a single edge transition would leave the alias check incorrect until the next flap of the aliased check. The change was to introduce a fallback timer to act as a control loop to double check the alias check matches the aliased check every minute (borrowing the duration from the non-local alias check logic body). This made the observed behavior eventually go away when it did occur. (3) Originally I thought there were two main actions involved in the data race: A. The act of adding the original check (from disk recovery) and its first health evaluation. B. The act of the HTTP API requests coming in and resetting the local state when re-registering the same services and checks. It took awhile for me to realize that there's a third action at work: C. The goroutines associated with the original check and the later checks. The actual sequence of actions that was causing the bad behavior was that the API actions result in the original check to be removed and re-added _without waiting for the original goroutine to terminate_. This means for brief windows of time during check definition edits there are two goroutines that can be sending updates for the alias check status. In extremely unlikely scenarios the original goroutine sees the aliased check start up in `critical` before being removed but does not get the notification about the nearly immediate update of that check to `passing`. This is interlaced wit the new goroutine coming up, initializing its base case to `passing` from the current state and then listening for new notifications of edge triggers. If the original goroutine "finishes" its update, it then commits one more write into the local state of `critical` and exits leaving the alias check no longer reflecting the underlying check. The correction here is to enforce that the old goroutines must terminate before spawning the new one for alias checks.
2019-05-24 18:36:56 +00:00
"github.com/hashicorp/consul/sdk/freeport"
Move internal/ to sdk/ (#5568) * Move internal/ to sdk/ * Add a readme to the SDK folder
2019-03-27 12:54:56 +00:00
"github.com/hashicorp/consul/sdk/testutil"
"github.com/hashicorp/consul/sdk/testutil/retry"
wan federation via mesh gateways (#6884) This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch: There are several distinct chunks of code that are affected: * new flags and config options for the server * retry join WAN is slightly different * retry join code is shared to discover primary mesh gateways from secondary datacenters * because retry join logic runs in the *agent* and the results of that operation for primary mesh gateways are needed in the *server* there are some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur at multiple layers of abstraction just to pass the data down to the right layer. * new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers * the function signature for RPC dialing picked up a new required field (the node name of the destination) * several new RPCs for manipulating a FederationState object: `FederationState:{Apply,Get,List,ListMeshGateways}` * 3 read-only internal APIs for debugging use to invoke those RPCs from curl * raft and fsm changes to persist these FederationStates * replication for FederationStates as they are canonically stored in the Primary and replicated to the Secondaries. * a special derivative of anti-entropy that runs in secondaries to snapshot their local mesh gateway `CheckServiceNodes` and sync them into their upstream FederationState in the primary (this works in conjunction with the replication to distribute addresses for all mesh gateways in all DCs to all other DCs) * a "gateway locator" convenience object to make use of this data to choose the addresses of gateways to use for any given RPC or gossip operation to a remote DC. This gets data from the "retry join" logic in the agent and also directly calls into the FSM. * RPC (`:8300`) on the server sniffs the first byte of a new connection to determine if it's actually doing native TLS. If so it checks the ALPN header for protocol determination (just like how the existing system uses the type-byte marker). * 2 new kinds of protocols are exclusively decoded via this native TLS mechanism: one for ferrying "packet" operations (udp-like) from the gossip layer and one for "stream" operations (tcp-like). The packet operations re-use sockets (using length-prefixing) to cut down on TLS re-negotiation overhead. * the server instances specially wrap the `memberlist.NetTransport` when running with gateway federation enabled (in a `wanfed.Transport`). The general gist is that if it tries to dial a node in the SAME datacenter (deduced by looking at the suffix of the node name) there is no change. If dialing a DIFFERENT datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh gateways to eventually end up in a server's :8300 port. * a new flag when launching a mesh gateway via `consul connect envoy` to indicate that the servers are to be exposed. This sets a special service meta when registering the gateway into the catalog. * `proxycfg/xds` notice this metadata blob to activate additional watches for the FederationState objects as well as the location of all of the consul servers in that datacenter. * `xds:` if the extra metadata is in place additional clusters are defined in a DC to bulk sink all traffic to another DC's gateways. For the current datacenter we listen on a wildcard name (`server.<dc>.consul`) that load balances all servers as well as one mini-cluster per node (`<node>.server.<dc>.consul`) * the `consul tls cert create` command got a new flag (`-node`) to help create an additional SAN in certs that can be used with this flavor of federation.
2020-03-09 20:59:02 +00:00
"github.com/hashicorp/consul/testrpc"
Adds basic support for node IDs.
2017-01-18 06:20:11 +00:00
"github.com/hashicorp/consul/types"
Expose HTTP-based paths through Connect proxy (#6446) Fixes: #5396 This PR adds a proxy configuration stanza called expose. These flags register listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only listening on the loopback interface, while still accepting traffic from non Connect-enabled services. Under expose there is a boolean checks flag that would automatically expose all registered HTTP and gRPC check paths. This stanza also accepts a paths list to expose individual paths. The primary use case for this functionality would be to expose paths for third parties like Prometheus or the kubelet. Listeners for requests to exposed paths are be configured dynamically at run time. Any time a proxy, or check can be registered, a listener can also be created. In this initial implementation requests to these paths are not authenticated/encrypted.
2019-09-26 02:55:52 +00:00
"github.com/hashicorp/go-uuid"
Update TestAgent_GetCoordinate The old test case was a very specific regresion test for a case that is no longer possible. Replaced with a new test that checks the default coordinate is returned.
2020-06-24 16:56:46 +00:00
"github.com/hashicorp/serf/coordinate"
wan federation via mesh gateways (#6884) This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch: There are several distinct chunks of code that are affected: * new flags and config options for the server * retry join WAN is slightly different * retry join code is shared to discover primary mesh gateways from secondary datacenters * because retry join logic runs in the *agent* and the results of that operation for primary mesh gateways are needed in the *server* there are some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur at multiple layers of abstraction just to pass the data down to the right layer. * new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers * the function signature for RPC dialing picked up a new required field (the node name of the destination) * several new RPCs for manipulating a FederationState object: `FederationState:{Apply,Get,List,ListMeshGateways}` * 3 read-only internal APIs for debugging use to invoke those RPCs from curl * raft and fsm changes to persist these FederationStates * replication for FederationStates as they are canonically stored in the Primary and replicated to the Secondaries. * a special derivative of anti-entropy that runs in secondaries to snapshot their local mesh gateway `CheckServiceNodes` and sync them into their upstream FederationState in the primary (this works in conjunction with the replication to distribute addresses for all mesh gateways in all DCs to all other DCs) * a "gateway locator" convenience object to make use of this data to choose the addresses of gateways to use for any given RPC or gossip operation to a remote DC. This gets data from the "retry join" logic in the agent and also directly calls into the FSM. * RPC (`:8300`) on the server sniffs the first byte of a new connection to determine if it's actually doing native TLS. If so it checks the ALPN header for protocol determination (just like how the existing system uses the type-byte marker). * 2 new kinds of protocols are exclusively decoded via this native TLS mechanism: one for ferrying "packet" operations (udp-like) from the gossip layer and one for "stream" operations (tcp-like). The packet operations re-use sockets (using length-prefixing) to cut down on TLS re-negotiation overhead. * the server instances specially wrap the `memberlist.NetTransport` when running with gateway federation enabled (in a `wanfed.Transport`). The general gist is that if it tries to dial a node in the SAME datacenter (deduced by looking at the suffix of the node name) there is no change. If dialing a DIFFERENT datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh gateways to eventually end up in a server's :8300 port. * a new flag when launching a mesh gateway via `consul connect envoy` to indicate that the servers are to be exposed. This sets a special service meta when registering the gateway into the catalog. * `proxycfg/xds` notice this metadata blob to activate additional watches for the FederationState objects as well as the location of all of the consul servers in that datacenter. * `xds:` if the extra metadata is in place additional clusters are defined in a DC to bulk sink all traffic to another DC's gateways. For the current datacenter we listen on a wildcard name (`server.<dc>.consul`) that load balances all servers as well as one mini-cluster per node (`<node>.server.<dc>.consul`) * the `consul tls cert create` command got a new flag (`-node`) to help create an additional SAN in certs that can be used with this flavor of federation.
2020-03-09 20:59:02 +00:00
"github.com/hashicorp/serf/serf"
Persist proxy state through agent restart
2018-05-14 20:55:24 +00:00
"github.com/stretchr/testify/assert"
agent: fix crash that could happen if proxy was nil on load
2018-05-02 18:51:47 +00:00
"github.com/stretchr/testify/require"
Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.
2020-06-10 20:47:35 +00:00
"gopkg.in/square/go-jose.v2/jwt"
Adding more agent tests
2013-12-23 21:52:10 +00:00
)
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
func getService(a *TestAgent, id string) *structs.NodeService {
return a.State.Service(structs.NewServiceID(id, nil))
}
func getCheck(a *TestAgent, id types.CheckID) *structs.HealthCheck {
return a.State.Check(structs.NewCheckID(id, nil))
}
func requireServiceExists(t *testing.T, a *TestAgent, id string) *structs.NodeService {
t.Helper()
svc := getService(a, id)
require.NotNil(t, svc, "missing service %q", id)
return svc
}
func requireServiceMissing(t *testing.T, a *TestAgent, id string) {
t.Helper()
require.Nil(t, getService(a, id), "have service %q (expected missing)", id)
}
func requireCheckExists(t *testing.T, a *TestAgent, id types.CheckID) *structs.HealthCheck {
t.Helper()
chk := getCheck(a, id)
require.NotNil(t, chk, "missing check %q", id)
return chk
}
func requireCheckMissing(t *testing.T, a *TestAgent, id types.CheckID) {
t.Helper()
require.Nil(t, getCheck(a, id), "have check %q (expected missing)", id)
}
func requireCheckExistsMap(t *testing.T, m interface{}, id types.CheckID) {
t.Helper()
require.Contains(t, m, structs.NewCheckID(id, nil), "missing check %q", id)
}
func requireCheckMissingMap(t *testing.T, m interface{}, id types.CheckID) {
t.Helper()
require.NotContains(t, m, structs.NewCheckID(id, nil), "have check %q (expected missing)", id)
}
test: add test for starting/stopping lots of agents
2017-05-23 14:04:53 +00:00
func TestAgent_MultiStartStop(t *testing.T) {
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
for i := 0; i < 10; i++ {
test: add test for starting/stopping lots of agents
2017-05-23 14:04:53 +00:00
t.Run("", func(t *testing.T) {
test: start secondary agents also via TestAgent
2017-05-31 08:56:19 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
test: add test for starting/stopping lots of agents
2017-05-23 14:04:53 +00:00
time.Sleep(250 * time.Millisecond)
a.Shutdown()
})
}
}
Fixed many tests after rebase. Some still failing and seem unrelated to any connect changes.
2018-05-10 16:04:33 +00:00
func TestAgent_ConnectClusterIDConfig(t *testing.T) {
tests := []struct {
name string
hcl string
wantClusterID string
remove funky panic/recover in agent tests (#6442)
2019-09-04 20:59:11 +00:00
wantErr bool
Fixed many tests after rebase. Some still failing and seem unrelated to any connect changes.
2018-05-10 16:04:33 +00:00
}{
{
name: "default TestAgent has fixed cluster id",
hcl: "",
wantClusterID: connect.TestClusterID,
},
{
Fix broken tests from PR merge related to proxy secure defaults
2018-06-13 08:34:20 +00:00
name: "no cluster ID specified sets to test ID",
Fixed many tests after rebase. Some still failing and seem unrelated to any connect changes.
2018-05-10 16:04:33 +00:00
hcl: "connect { enabled = true }",
Fix broken tests from PR merge related to proxy secure defaults
2018-06-13 08:34:20 +00:00
wantClusterID: connect.TestClusterID,
Fixed many tests after rebase. Some still failing and seem unrelated to any connect changes.
2018-05-10 16:04:33 +00:00
},
{
Make invalid clusterID be fatal
2018-05-22 14:11:13 +00:00
name: "non-UUID cluster_id is fatal",
hcl: `connect {
enabled = true
ca_config {
cluster_id = "fake-id"
}
}`,
Fixed many tests after rebase. Some still failing and seem unrelated to any connect changes.
2018-05-10 16:04:33 +00:00
wantClusterID: "",
remove funky panic/recover in agent tests (#6442)
2019-09-04 20:59:11 +00:00
wantErr: true,
Fixed many tests after rebase. Some still failing and seem unrelated to any connect changes.
2018-05-10 16:04:33 +00:00
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
Convert the remaining calls to NewTestAgentWithFields After removing the t.Name() parameter with sed, convert the last few tests which use a custom name to call NewTestAgentWithFields instead.
2020-03-31 20:12:33 +00:00
a := TestAgent{HCL: tt.hcl}
Remove t.Name() from TestAgent.Name And re-add the name to the logger so that log messages from different agents in a single can be identified.
2020-03-30 20:05:27 +00:00
err := a.Start(t)
remove funky panic/recover in agent tests (#6442)
2019-09-04 20:59:11 +00:00
if tt.wantErr {
if err == nil {
t.Fatal("expected error, got nil")
refactor & add better retry logic to NewTestAgent (#6363) Fixes #6361
2019-09-03 22:05:51 +00:00
}
remove funky panic/recover in agent tests (#6442)
2019-09-04 20:59:11 +00:00
return // don't run the rest of the test
Make invalid clusterID be fatal
2018-05-22 14:11:13 +00:00
}
remove funky panic/recover in agent tests (#6442)
2019-09-04 20:59:11 +00:00
if !tt.wantErr && err != nil {
t.Fatal(err)
Make invalid clusterID be fatal
2018-05-22 14:11:13 +00:00
}
remove funky panic/recover in agent tests (#6442)
2019-09-04 20:59:11 +00:00
defer a.Shutdown()
cfg := a.consulConfig()
assert.Equal(t, tt.wantClusterID, cfg.CAConfig.ClusterID)
Fixed many tests after rebase. Some still failing and seem unrelated to any connect changes.
2018-05-10 16:04:33 +00:00
})
}
}
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
func TestAgent_StartStop(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
Make invalid clusterID be fatal
2018-05-22 14:11:13 +00:00
defer a.Shutdown()
Adding more agent tests
2013-12-23 21:52:10 +00:00
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
if err := a.Leave(); err != nil {
Adding more agent tests
2013-12-23 21:52:10 +00:00
t.Fatalf("err: %v", err)
}
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
if err := a.Shutdown(); err != nil {
Adding more agent tests
2013-12-23 21:52:10 +00:00
t.Fatalf("err: %v", err)
}
select {
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
case <-a.ShutdownCh():
Adding more agent tests
2013-12-23 21:52:10 +00:00
default:
t.Fatalf("should be closed")
}
}
agent: beginning refactor
2015-01-16 01:24:15 +00:00
func TestAgent_RPCPing(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
add wait to TestAgent_RPCPing
2019-02-22 02:47:19 +00:00
testrpc.WaitForTestAgent(t, a.RPC, "dc1")
Adding more agent tests
2013-12-23 21:52:10 +00:00
var out struct{}
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
if err := a.RPC("Status.Ping", struct{}{}, &out); err != nil {
Adding more agent tests
2013-12-23 21:52:10 +00:00
t.Fatalf("err: %v", err)
}
}
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
Adds support for agent-side ACL token management via API instead of config files. (#3324) * Adds token store and removes all runtime use of config for ACL tokens. * Adds a new API for changing agent tokens on the fly.
2017-07-26 18:03:43 +00:00
func TestAgent_TokenStore(t *testing.T) {
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
acl_token = "user"
acl_agent_token = "agent"
acl_agent_master_token = "master"`,
)
Adds support for agent-side ACL token management via API instead of config files. (#3324) * Adds token store and removes all runtime use of config for ACL tokens. * Adds a new API for changing agent tokens on the fly.
2017-07-26 18:03:43 +00:00
defer a.Shutdown()
if got, want := a.tokens.UserToken(), "user"; got != want {
t.Fatalf("got %q want %q", got, want)
}
if got, want := a.tokens.AgentToken(), "agent"; got != want {
t.Fatalf("got %q want %q", got, want)
}
if got, want := a.tokens.IsAgentMasterToken("master"), true; got != want {
t.Fatalf("got %v want %v", got, want)
}
}
Makes reap time configurable for LAN and WAN.
2016-04-11 05:46:07 +00:00
func TestAgent_ReconnectConfigSettings(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Makes reap time configurable for LAN and WAN.
2016-04-11 05:46:07 +00:00
func() {
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
Makes reap time configurable for LAN and WAN.
2016-04-11 05:46:07 +00:00
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
lan := a.consulConfig().SerfLANConfig.ReconnectTimeout
Makes reap time configurable for LAN and WAN.
2016-04-11 05:46:07 +00:00
if lan != 3*24*time.Hour {
t.Fatalf("bad: %s", lan.String())
}
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
wan := a.consulConfig().SerfWANConfig.ReconnectTimeout
Makes reap time configurable for LAN and WAN.
2016-04-11 05:46:07 +00:00
if wan != 3*24*time.Hour {
t.Fatalf("bad: %s", wan.String())
}
}()
func() {
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
reconnect_timeout = "24h"
reconnect_timeout_wan = "36h"
`)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
Makes reap time configurable for LAN and WAN.
2016-04-11 05:46:07 +00:00
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
lan := a.consulConfig().SerfLANConfig.ReconnectTimeout
Sets an anti-footgun floor for the configurable reap time.
2016-04-11 06:31:16 +00:00
if lan != 24*time.Hour {
Makes reap time configurable for LAN and WAN.
2016-04-11 05:46:07 +00:00
t.Fatalf("bad: %s", lan.String())
}
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
wan := a.consulConfig().SerfWANConfig.ReconnectTimeout
Sets an anti-footgun floor for the configurable reap time.
2016-04-11 06:31:16 +00:00
if wan != 36*time.Hour {
Makes reap time configurable for LAN and WAN.
2016-04-11 05:46:07 +00:00
t.Fatalf("bad: %s", wan.String())
}
}()
}
Fix #4515: Segfault when serf_wan port was -1 but reconnect_time_wan was set (#4531) Fixes #4515 This just slightly refactors the logic to only attempt to set the serf wan reconnect timeout when the rest of the serf wan settings are configured - thus avoiding a segfault.
2018-08-17 18:44:25 +00:00
func TestAgent_ReconnectConfigWanDisabled(t *testing.T) {
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
Fix #4515: Segfault when serf_wan port was -1 but reconnect_time_wan was set (#4531) Fixes #4515 This just slightly refactors the logic to only attempt to set the serf wan reconnect timeout when the rest of the serf wan settings are configured - thus avoiding a segfault.
2018-08-17 18:44:25 +00:00
ports { serf_wan = -1 }
reconnect_timeout_wan = "36h"
`)
defer a.Shutdown()
// This is also testing that we dont panic like before #4515
require.Nil(t, a.consulConfig().SerfWANConfig)
}
Adds a new -disable-host-node-id option to help when testing with containers. Fixes #2877.
2017-04-13 05:05:38 +00:00
func TestAgent_setupNodeID(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
node_id = ""
`)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
Adds basic support for node IDs.
2017-01-18 06:20:11 +00:00
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
cfg := a.config
Adds basic support for node IDs.
2017-01-18 06:20:11 +00:00
// The auto-assigned ID should be valid.
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
id := a.consulConfig().NodeID
Adds basic support for node IDs.
2017-01-18 06:20:11 +00:00
if _, err := uuid.ParseUUID(string(id)); err != nil {
t.Fatalf("err: %v", err)
}
Adds a test to make sure we get the same ID on the second run.
2017-01-18 17:52:34 +00:00
// Running again should get the same ID (persisted in the file).
test: rename *Config vars to 'cfg' 'c' is ambigous since Command also uses this and we want to use 'config' as a package name.
2017-05-22 11:03:59 +00:00
cfg.NodeID = ""
if err := a.setupNodeID(cfg); err != nil {
Adds a test to make sure we get the same ID on the second run.
2017-01-18 17:52:34 +00:00
t.Fatalf("err: %v", err)
}
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
if newID := a.consulConfig().NodeID; id != newID {
Adds a test to make sure we get the same ID on the second run.
2017-01-18 17:52:34 +00:00
t.Fatalf("bad: %q vs %q", id, newID)
}
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
// Set an invalid ID via.Config.
test: rename *Config vars to 'cfg' 'c' is ambigous since Command also uses this and we want to use 'config' as a package name.
2017-05-22 11:03:59 +00:00
cfg.NodeID = types.NodeID("nope")
err := a.setupNodeID(cfg)
Adds basic support for node IDs.
2017-01-18 06:20:11 +00:00
if err == nil || !strings.Contains(err.Error(), "uuid string is wrong length") {
t.Fatalf("err: %v", err)
}
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
// Set a valid ID via.Config.
Adds basic support for node IDs.
2017-01-18 06:20:11 +00:00
newID, err := uuid.GenerateUUID()
if err != nil {
t.Fatalf("err: %v", err)
}
test: rename *Config vars to 'cfg' 'c' is ambigous since Command also uses this and we want to use 'config' as a package name.
2017-05-22 11:03:59 +00:00
cfg.NodeID = types.NodeID(strings.ToUpper(newID))
if err := a.setupNodeID(cfg); err != nil {
Adds basic support for node IDs.
2017-01-18 06:20:11 +00:00
t.Fatalf("err: %v", err)
}
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
if id := a.consulConfig().NodeID; string(id) != newID {
Adds basic support for node IDs.
2017-01-18 06:20:11 +00:00
t.Fatalf("bad: %q vs. %q", id, newID)
}
// Set an invalid ID via the file.
test: rename *Config vars to 'cfg' 'c' is ambigous since Command also uses this and we want to use 'config' as a package name.
2017-05-22 11:03:59 +00:00
fileID := filepath.Join(cfg.DataDir, "node-id")
Adds basic support for node IDs.
2017-01-18 06:20:11 +00:00
if err := ioutil.WriteFile(fileID, []byte("adf4238a!882b!9ddc!4a9d!5b6758e4159e"), 0600); err != nil {
t.Fatalf("err: %v", err)
}
test: rename *Config vars to 'cfg' 'c' is ambigous since Command also uses this and we want to use 'config' as a package name.
2017-05-22 11:03:59 +00:00
cfg.NodeID = ""
err = a.setupNodeID(cfg)
Adds basic support for node IDs.
2017-01-18 06:20:11 +00:00
if err == nil || !strings.Contains(err.Error(), "uuid is improperly formatted") {
t.Fatalf("err: %v", err)
}
// Set a valid ID via the file.
Forces user-supplied node IDs to lower case for consistency.
2017-03-14 02:51:56 +00:00
if err := ioutil.WriteFile(fileID, []byte("ADF4238a-882b-9ddc-4a9d-5b6758e4159e"), 0600); err != nil {
Adds basic support for node IDs.
2017-01-18 06:20:11 +00:00
t.Fatalf("err: %v", err)
}
test: rename *Config vars to 'cfg' 'c' is ambigous since Command also uses this and we want to use 'config' as a package name.
2017-05-22 11:03:59 +00:00
cfg.NodeID = ""
if err := a.setupNodeID(cfg); err != nil {
Adds basic support for node IDs.
2017-01-18 06:20:11 +00:00
t.Fatalf("err: %v", err)
}
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
if id := a.consulConfig().NodeID; string(id) != "adf4238a-882b-9ddc-4a9d-5b6758e4159e" {
Adds basic support for node IDs.
2017-01-18 06:20:11 +00:00
t.Fatalf("bad: %q vs. %q", id, newID)
}
}
Adds a new -disable-host-node-id option to help when testing with containers. Fixes #2877.
2017-04-13 05:05:38 +00:00
func TestAgent_makeNodeID(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
node_id = ""
`)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
Adds a new -disable-host-node-id option to help when testing with containers. Fixes #2877.
2017-04-13 05:05:38 +00:00
// We should get a valid host-based ID initially.
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
id, err := a.makeNodeID()
Adds a new -disable-host-node-id option to help when testing with containers. Fixes #2877.
2017-04-13 05:05:38 +00:00
if err != nil {
t.Fatalf("err: %v", err)
}
Add unconvert linter To find unnecessary type convertions
2020-04-16 17:35:28 +00:00
if _, err := uuid.ParseUUID(id); err != nil {
Adds a new -disable-host-node-id option to help when testing with containers. Fixes #2877.
2017-04-13 05:05:38 +00:00
t.Fatalf("err: %v", err)
}
Changes host-based node IDs from opt-out to opt-in. (#3187)
2017-06-24 16:36:53 +00:00
// Calling again should yield a random ID by default.
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
another, err := a.makeNodeID()
Adds a new -disable-host-node-id option to help when testing with containers. Fixes #2877.
2017-04-13 05:05:38 +00:00
if err != nil {
t.Fatalf("err: %v", err)
}
Changes host-based node IDs from opt-out to opt-in. (#3187)
2017-06-24 16:36:53 +00:00
if id == another {
Adds a new -disable-host-node-id option to help when testing with containers. Fixes #2877.
2017-04-13 05:05:38 +00:00
t.Fatalf("bad: %s vs %s", id, another)
}
Changes host-based node IDs from opt-out to opt-in. (#3187)
2017-06-24 16:36:53 +00:00
// Turn on host-based IDs and try again. We should get the same ID
// each time (and a different one from the random one above).
Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.
2020-06-10 20:47:35 +00:00
a.GetConfig().DisableHostNodeID = false
Changes host-based node IDs from opt-out to opt-in. (#3187)
2017-06-24 16:36:53 +00:00
id, err = a.makeNodeID()
Adds a new -disable-host-node-id option to help when testing with containers. Fixes #2877.
2017-04-13 05:05:38 +00:00
if err != nil {
t.Fatalf("err: %v", err)
}
if id == another {
t.Fatalf("bad: %s vs %s", id, another)
}
Changes host-based node IDs from opt-out to opt-in. (#3187)
2017-06-24 16:36:53 +00:00
// Calling again should yield the host-based ID.
another, err = a.makeNodeID()
if err != nil {
t.Fatalf("err: %v", err)
}
if id != another {
t.Fatalf("bad: %s vs %s", id, another)
}
Adds a new -disable-host-node-id option to help when testing with containers. Fixes #2877.
2017-04-13 05:05:38 +00:00
}
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
func TestAgent_AddService(t *testing.T) {
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
t.Run("normal", func(t *testing.T) {
t.Parallel()
testAgent_AddService(t, "")
})
t.Run("service manager", func(t *testing.T) {
t.Parallel()
testAgent_AddService(t, "enable_central_service_config = true")
})
}
func testAgent_AddService(t *testing.T, extraHCL string) {
t.Helper()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
node_name = "node1"
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
`+extraHCL)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
tests := []struct {
desc string
srv *structs.NodeService
agent: add default weights to service in local state to prevent AE churn (#5126) * Add default weights when adding a service with no weights to local state to prevent constant AE re-sync. This fix was contributed by @42wim in https://github.com/hashicorp/consul/pull/5096 but was merged against the wrong base. This adds it to master and adds a test to cover the behaviour. * Fix tests that broke due to comparing internal state which now has default weights
2019-01-08 10:13:49 +00:00
wantSrv func(ns *structs.NodeService)
agent: move structs into consul/structs pkg * CheckDefinition * ServiceDefinition * CheckType
2017-06-15 16:46:06 +00:00
chkTypes []*structs.CheckType
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
healthChks map[string]*structs.HealthCheck
}{
{
"one check",
&structs.NodeService{
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
ID: "svcid1",
Service: "svcname1",
Tags: []string{"tag1"},
Weights: nil, // nil weights...
Port: 8100,
EnterpriseMeta: *structs.DefaultEnterpriseMeta(),
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
},
agent: add default weights to service in local state to prevent AE churn (#5126) * Add default weights when adding a service with no weights to local state to prevent constant AE re-sync. This fix was contributed by @42wim in https://github.com/hashicorp/consul/pull/5096 but was merged against the wrong base. This adds it to master and adds a test to cover the behaviour. * Fix tests that broke due to comparing internal state which now has default weights
2019-01-08 10:13:49 +00:00
// ... should be populated to avoid "IsSame" returning true during AE.
func(ns *structs.NodeService) {
ns.Weights = &structs.Weights{
Passing: 1,
Warning: 1,
}
},
agent: move structs into consul/structs pkg * CheckDefinition * ServiceDefinition * CheckType
2017-06-15 16:46:06 +00:00
[]*structs.CheckType{
Enable gofmt simplify Code changes done automatically with 'gofmt -s -w'
2020-06-16 17:19:31 +00:00
{
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
CheckID: "check1",
Name: "name1",
TTL: time.Minute,
Notes: "note1",
},
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
},
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
map[string]*structs.HealthCheck{
Enable gofmt simplify Code changes done automatically with 'gofmt -s -w'
2020-06-16 17:19:31 +00:00
"check1": {
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
Node: "node1",
CheckID: "check1",
Name: "name1",
Status: "critical",
Notes: "note1",
ServiceID: "svcid1",
ServiceName: "svcname1",
ServiceTags: []string{"tag1"},
Type: "ttl",
EnterpriseMeta: *structs.DefaultEnterpriseMeta(),
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
},
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
},
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
},
{
"multiple checks",
&structs.NodeService{
ID: "svcid2",
Service: "svcname2",
agent: add default weights to service in local state to prevent AE churn (#5126) * Add default weights when adding a service with no weights to local state to prevent constant AE re-sync. This fix was contributed by @42wim in https://github.com/hashicorp/consul/pull/5096 but was merged against the wrong base. This adds it to master and adds a test to cover the behaviour. * Fix tests that broke due to comparing internal state which now has default weights
2019-01-08 10:13:49 +00:00
Weights: &structs.Weights{
Passing: 2,
Warning: 1,
},
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
Tags: []string{"tag2"},
Port: 8200,
EnterpriseMeta: *structs.DefaultEnterpriseMeta(),
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
},
agent: add default weights to service in local state to prevent AE churn (#5126) * Add default weights when adding a service with no weights to local state to prevent constant AE re-sync. This fix was contributed by @42wim in https://github.com/hashicorp/consul/pull/5096 but was merged against the wrong base. This adds it to master and adds a test to cover the behaviour. * Fix tests that broke due to comparing internal state which now has default weights
2019-01-08 10:13:49 +00:00
nil, // No change expected
agent: move structs into consul/structs pkg * CheckDefinition * ServiceDefinition * CheckType
2017-06-15 16:46:06 +00:00
[]*structs.CheckType{
Enable gofmt simplify Code changes done automatically with 'gofmt -s -w'
2020-06-16 17:19:31 +00:00
{
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
CheckID: "check1",
Name: "name1",
TTL: time.Minute,
Notes: "note1",
},
Enable gofmt simplify Code changes done automatically with 'gofmt -s -w'
2020-06-16 17:19:31 +00:00
{
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
CheckID: "check-noname",
TTL: time.Minute,
},
Enable gofmt simplify Code changes done automatically with 'gofmt -s -w'
2020-06-16 17:19:31 +00:00
{
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
Name: "check-noid",
TTL: time.Minute,
},
Enable gofmt simplify Code changes done automatically with 'gofmt -s -w'
2020-06-16 17:19:31 +00:00
{
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
TTL: time.Minute,
},
},
map[string]*structs.HealthCheck{
Enable gofmt simplify Code changes done automatically with 'gofmt -s -w'
2020-06-16 17:19:31 +00:00
"check1": {
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
Node: "node1",
CheckID: "check1",
Name: "name1",
Status: "critical",
Notes: "note1",
ServiceID: "svcid2",
ServiceName: "svcname2",
ServiceTags: []string{"tag2"},
Type: "ttl",
EnterpriseMeta: *structs.DefaultEnterpriseMeta(),
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
},
Enable gofmt simplify Code changes done automatically with 'gofmt -s -w'
2020-06-16 17:19:31 +00:00
"check-noname": {
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
Node: "node1",
CheckID: "check-noname",
Name: "Service 'svcname2' check",
Status: "critical",
ServiceID: "svcid2",
ServiceName: "svcname2",
ServiceTags: []string{"tag2"},
Type: "ttl",
EnterpriseMeta: *structs.DefaultEnterpriseMeta(),
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
},
Enable gofmt simplify Code changes done automatically with 'gofmt -s -w'
2020-06-16 17:19:31 +00:00
"service:svcid2:3": {
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
Node: "node1",
CheckID: "service:svcid2:3",
Name: "check-noid",
Status: "critical",
ServiceID: "svcid2",
ServiceName: "svcname2",
ServiceTags: []string{"tag2"},
Type: "ttl",
EnterpriseMeta: *structs.DefaultEnterpriseMeta(),
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
},
Enable gofmt simplify Code changes done automatically with 'gofmt -s -w'
2020-06-16 17:19:31 +00:00
"service:svcid2:4": {
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
Node: "node1",
CheckID: "service:svcid2:4",
Name: "Service 'svcname2' check",
Status: "critical",
ServiceID: "svcid2",
ServiceName: "svcname2",
ServiceTags: []string{"tag2"},
Type: "ttl",
EnterpriseMeta: *structs.DefaultEnterpriseMeta(),
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
},
},
},
}
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
for _, tt := range tests {
t.Run(tt.desc, func(t *testing.T) {
// check the service registration
t.Run(tt.srv.ID, func(t *testing.T) {
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
err := a.AddService(tt.srv, tt.chkTypes, false, "", ConfigSourceLocal)
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
if err != nil {
t.Fatalf("err: %v", err)
}
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
got := getService(a, tt.srv.ID)
agent: add default weights to service in local state to prevent AE churn (#5126) * Add default weights when adding a service with no weights to local state to prevent constant AE re-sync. This fix was contributed by @42wim in https://github.com/hashicorp/consul/pull/5096 but was merged against the wrong base. This adds it to master and adds a test to cover the behaviour. * Fix tests that broke due to comparing internal state which now has default weights
2019-01-08 10:13:49 +00:00
// Make a copy since the tt.srv points to the one in memory in the local
// state still so changing it is a tautology!
want := *tt.srv
if tt.wantSrv != nil {
tt.wantSrv(&want)
}
require.Equal(t, &want, got)
require.True(t, got.IsSame(&want))
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
})
// check the health checks
for k, v := range tt.healthChks {
t.Run(k, func(t *testing.T) {
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
got := getCheck(a, types.CheckID(k))
agent: add default weights to service in local state to prevent AE churn (#5126) * Add default weights when adding a service with no weights to local state to prevent constant AE re-sync. This fix was contributed by @42wim in https://github.com/hashicorp/consul/pull/5096 but was merged against the wrong base. This adds it to master and adds a test to cover the behaviour. * Fix tests that broke due to comparing internal state which now has default weights
2019-01-08 10:13:49 +00:00
require.Equal(t, v, got)
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
})
}
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
// check the ttl checks
for k := range tt.healthChks {
t.Run(k+" ttl", func(t *testing.T) {
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
chk := a.checkTTLs[structs.NewCheckID(types.CheckID(k), nil)]
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
if chk == nil {
t.Fatal("got nil want TTL check")
}
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if got, want := string(chk.CheckID.ID), k; got != want {
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
t.Fatalf("got CheckID %v want %v", got, want)
}
if got, want := chk.TTL, time.Minute; got != want {
t.Fatalf("got TTL %v want %v", got, want)
}
})
}
})
agent: pass notes field through for checks inside of service definitions. Fixes #449
2014-11-07 02:24:04 +00:00
}
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
}
agent: avoid reverting any check updates that occur while a service is being added or the config is reloaded (#6144)
2019-07-17 19:06:50 +00:00
func TestAgent_AddServices_AliasUpdateCheckNotReverted(t *testing.T) {
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
t.Run("normal", func(t *testing.T) {
t.Parallel()
testAgent_AddServices_AliasUpdateCheckNotReverted(t, "")
})
t.Run("service manager", func(t *testing.T) {
t.Parallel()
testAgent_AddServices_AliasUpdateCheckNotReverted(t, "enable_central_service_config = true")
})
}
func testAgent_AddServices_AliasUpdateCheckNotReverted(t *testing.T, extraHCL string) {
t.Helper()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
agent: avoid reverting any check updates that occur while a service is being added or the config is reloaded (#6144)
2019-07-17 19:06:50 +00:00
node_name = "node1"
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
`+extraHCL)
agent: avoid reverting any check updates that occur while a service is being added or the config is reloaded (#6144)
2019-07-17 19:06:50 +00:00
defer a.Shutdown()
// It's tricky to get an UpdateCheck call to be timed properly so it lands
// right in the middle of an addServiceInternal call so we cheat a bit and
// rely upon alias checks to do that work for us. We add enough services
// that probabilistically one of them is going to end up properly in the
// critical section.
//
// The first number I picked here (10) surprisingly failed every time prior
// to PR #6144 solving the underlying problem.
const numServices = 10
services := make([]*structs.ServiceDefinition, numServices)
checkIDs := make([]types.CheckID, numServices)
checks: when a service does not exists in an alias, consider it failing (#7384) In current implementation of Consul, check alias cannot determine if a service exists or not. Because a service without any check is semantically considered as passing, so when no healthchecks are found for an agent, the check was considered as passing. But this make little sense as the current implementation does not make any difference between: * a non-existing service (passing) * a service without any check (passing as well) In order to make it work, we have to ensure that when a check did not find any healthcheck, the service does indeed exists. If it does not, lets consider the check as failing.
2020-06-04 12:50:52 +00:00
services[0] = &structs.ServiceDefinition{
ID: "fake",
Name: "fake",
Port: 8080,
Checks: []*structs.CheckType{},
}
for i := 1; i < numServices; i++ {
agent: avoid reverting any check updates that occur while a service is being added or the config is reloaded (#6144)
2019-07-17 19:06:50 +00:00
name := fmt.Sprintf("web-%d", i)
services[i] = &structs.ServiceDefinition{
ID: name,
Name: name,
Port: 8080 + i,
Checks: []*structs.CheckType{
Enable gofmt simplify Code changes done automatically with 'gofmt -s -w'
2020-06-16 17:19:31 +00:00
{
agent: avoid reverting any check updates that occur while a service is being added or the config is reloaded (#6144)
2019-07-17 19:06:50 +00:00
Name: "alias-for-fake-service",
AliasService: "fake",
},
},
}
checkIDs[i] = types.CheckID("service:" + name)
}
// Add all of the services quickly as you might do from config file snippets.
for _, service := range services {
ns := service.NodeService()
chkTypes, err := service.CheckTypes()
require.NoError(t, err)
require.NoError(t, a.AddService(ns, chkTypes, false, service.Token, ConfigSourceLocal))
}
retry.Run(t, func(r *retry.R) {
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
gotChecks := a.State.Checks(nil)
agent: avoid reverting any check updates that occur while a service is being added or the config is reloaded (#6144)
2019-07-17 19:06:50 +00:00
for id, check := range gotChecks {
require.Equal(r, "passing", check.Status, "check %q is wrong", id)
require.Equal(r, "No checks found.", check.Output, "check %q is wrong", id)
}
})
}
checks: when a service does not exists in an alias, consider it failing (#7384) In current implementation of Consul, check alias cannot determine if a service exists or not. Because a service without any check is semantically considered as passing, so when no healthchecks are found for an agent, the check was considered as passing. But this make little sense as the current implementation does not make any difference between: * a non-existing service (passing) * a service without any check (passing as well) In order to make it work, we have to ensure that when a check did not find any healthcheck, the service does indeed exists. If it does not, lets consider the check as failing.
2020-06-04 12:50:52 +00:00
func test_createAlias(t *testing.T, agent *TestAgent, chk *structs.CheckType, expectedResult string) func(r *retry.R) {
t.Helper()
serviceNum := rand.Int()
srv := &structs.NodeService{
Service: fmt.Sprintf("serviceAlias-%d", serviceNum),
Tags: []string{"tag1"},
Port: 8900 + serviceNum,
}
if srv.ID == "" {
srv.ID = fmt.Sprintf("serviceAlias-%d", serviceNum)
}
chk.Status = api.HealthWarning
if chk.CheckID == "" {
chk.CheckID = types.CheckID(fmt.Sprintf("check-%d", serviceNum))
}
err := agent.AddService(srv, []*structs.CheckType{chk}, false, "", ConfigSourceLocal)
assert.NoError(t, err)
return func(r *retry.R) {
t.Helper()
found := false
for _, c := range agent.State.CheckStates(structs.WildcardEnterpriseMeta()) {
if c.Check.CheckID == chk.CheckID {
found = true
assert.Equal(t, expectedResult, c.Check.Status, "Check state should be %s, was %s in %#v", expectedResult, c.Check.Status, c.Check)
tests: use constructor instead init (#8024)
2020-06-04 20:59:06 +00:00
srvID := structs.NewServiceID(srv.ID, structs.WildcardEnterpriseMeta())
if err := agent.Agent.State.RemoveService(srvID); err != nil {
checks: when a service does not exists in an alias, consider it failing (#7384) In current implementation of Consul, check alias cannot determine if a service exists or not. Because a service without any check is semantically considered as passing, so when no healthchecks are found for an agent, the check was considered as passing. But this make little sense as the current implementation does not make any difference between: * a non-existing service (passing) * a service without any check (passing as well) In order to make it work, we have to ensure that when a check did not find any healthcheck, the service does indeed exists. If it does not, lets consider the check as failing.
2020-06-04 12:50:52 +00:00
fmt.Println("[DEBUG] Fail to remove service", srvID, ", err:=", err)
}
fmt.Println("[DEBUG] Service Removed", srvID, ", err:=", err)
break
}
}
assert.True(t, found)
}
}
// TestAgent_CheckAliasRPC test the Alias Check to be properly sync remotely
// and locally.
// It contains a few hacks such as unlockIndexOnNode because watch performed
// in CheckAlias.runQuery() waits for 1 min, so Shutdoww the agent might take time
// So, we ensure the agent will update regularilly the index
func TestAgent_CheckAliasRPC(t *testing.T) {
t.Helper()
a := NewTestAgent(t, `
node_name = "node1"
`)
srv := &structs.NodeService{
ID: "svcid1",
Service: "svcname1",
Tags: []string{"tag1"},
Port: 8100,
}
unlockIndexOnNode := func() {
// We ensure to not block and update Agent's index
srv.Tags = []string{fmt.Sprintf("tag-%s", time.Now())}
assert.NoError(t, a.waitForUp())
err := a.AddService(srv, []*structs.CheckType{}, false, "", ConfigSourceLocal)
assert.NoError(t, err)
}
shutdownAgent := func() {
// This is to be sure Alias Checks on remote won't be blocked during 1 min
unlockIndexOnNode()
fmt.Println("[DEBUG] STARTING shutdown for TestAgent_CheckAliasRPC", time.Now())
go a.Shutdown()
unlockIndexOnNode()
fmt.Println("[DEBUG] DONE shutdown for TestAgent_CheckAliasRPC", time.Now())
}
defer shutdownAgent()
testrpc.WaitForTestAgent(t, a.RPC, "dc1")
assert.NoError(t, a.waitForUp())
err := a.AddService(srv, []*structs.CheckType{}, false, "", ConfigSourceLocal)
assert.NoError(t, err)
retry.Run(t, func(r *retry.R) {
t.Helper()
var args structs.NodeSpecificRequest
args.Datacenter = "dc1"
args.Node = "node1"
args.AllowStale = true
var out structs.IndexedNodeServices
err := a.RPC("Catalog.NodeServices", &args, &out)
assert.NoError(r, err)
foundService := false
tests: use constructor instead init (#8024)
2020-06-04 20:59:06 +00:00
lookup := structs.NewServiceID("svcid1", structs.WildcardEnterpriseMeta())
checks: when a service does not exists in an alias, consider it failing (#7384) In current implementation of Consul, check alias cannot determine if a service exists or not. Because a service without any check is semantically considered as passing, so when no healthchecks are found for an agent, the check was considered as passing. But this make little sense as the current implementation does not make any difference between: * a non-existing service (passing) * a service without any check (passing as well) In order to make it work, we have to ensure that when a check did not find any healthcheck, the service does indeed exists. If it does not, lets consider the check as failing.
2020-06-04 12:50:52 +00:00
for _, srv := range out.NodeServices.Services {
sid := srv.CompoundServiceID()
if lookup.Matches(&sid) {
foundService = true
}
}
assert.True(r, foundService, "could not find svcid1 in %#v", out.NodeServices.Services)
})
checks := make([](func(*retry.R)), 0)
checks = append(checks, test_createAlias(t, a, &structs.CheckType{
Name: "Check_Local_Ok",
AliasService: "svcid1",
}, api.HealthPassing))
checks = append(checks, test_createAlias(t, a, &structs.CheckType{
Name: "Check_Local_Fail",
AliasService: "svcidNoExistingID",
}, api.HealthCritical))
checks = append(checks, test_createAlias(t, a, &structs.CheckType{
Name: "Check_Remote_Host_Ok",
AliasNode: "node1",
AliasService: "svcid1",
}, api.HealthPassing))
checks = append(checks, test_createAlias(t, a, &structs.CheckType{
Name: "Check_Remote_Host_Non_Existing_Service",
AliasNode: "node1",
AliasService: "svcidNoExistingID",
}, api.HealthCritical))
// We wait for max 5s for all checks to be in sync
{
for i := 0; i < 50; i++ {
unlockIndexOnNode()
allNonWarning := true
for _, chk := range a.State.Checks(structs.WildcardEnterpriseMeta()) {
if chk.Status == api.HealthWarning {
allNonWarning = false
}
}
if allNonWarning {
break
} else {
time.Sleep(100 * time.Millisecond)
}
}
}
for _, toRun := range checks {
unlockIndexOnNode()
retry.Run(t, toRun)
}
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
func TestAgent_AddServiceNoExec(t *testing.T) {
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
t.Run("normal", func(t *testing.T) {
t.Parallel()
testAgent_AddServiceNoExec(t, "")
})
t.Run("service manager", func(t *testing.T) {
t.Parallel()
testAgent_AddServiceNoExec(t, "enable_central_service_config = true")
})
}
func testAgent_AddServiceNoExec(t *testing.T, extraHCL string) {
t.Helper()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
node_name = "node1"
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
`+extraHCL)
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
defer a.Shutdown()
testrpc.WaitForTestAgent(t, a.RPC, "dc1")
srv := &structs.NodeService{
ID: "svcid1",
Service: "svcname1",
Tags: []string{"tag1"},
Port: 8100,
}
chk := &structs.CheckType{
ScriptArgs: []string{"exit", "0"},
Interval: 15 * time.Second,
}
err := a.AddService(srv, []*structs.CheckType{chk}, false, "", ConfigSourceLocal)
if err == nil || !strings.Contains(err.Error(), "Scripts are disabled on this agent") {
t.Fatalf("err: %v", err)
}
err = a.AddService(srv, []*structs.CheckType{chk}, false, "", ConfigSourceRemote)
if err == nil || !strings.Contains(err.Error(), "Scripts are disabled on this agent") {
t.Fatalf("err: %v", err)
}
}
func TestAgent_AddServiceNoRemoteExec(t *testing.T) {
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
t.Run("normal", func(t *testing.T) {
t.Parallel()
testAgent_AddServiceNoRemoteExec(t, "")
})
t.Run("service manager", func(t *testing.T) {
t.Parallel()
testAgent_AddServiceNoRemoteExec(t, "enable_central_service_config = true")
})
}
func testAgent_AddServiceNoRemoteExec(t *testing.T, extraHCL string) {
t.Helper()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
node_name = "node1"
enable_local_script_checks = true
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
`+extraHCL)
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
defer a.Shutdown()
testrpc.WaitForTestAgent(t, a.RPC, "dc1")
srv := &structs.NodeService{
ID: "svcid1",
Service: "svcname1",
Tags: []string{"tag1"},
Port: 8100,
}
chk := &structs.CheckType{
ScriptArgs: []string{"exit", "0"},
Interval: 15 * time.Second,
}
err := a.AddService(srv, []*structs.CheckType{chk}, false, "", ConfigSourceRemote)
if err == nil || !strings.Contains(err.Error(), "Scripts are disabled on this agent") {
t.Fatalf("err: %v", err)
}
}
Added ratelimit to handle throtling cache (#8226) This implements a solution for #7863 It does: Add a new config cache.entry_fetch_rate to limit the number of calls/s for a given cache entry, default value = rate.Inf Add cache.entry_fetch_max_burst size of rate limit (default value = 2) The new configuration now supports the following syntax for instance to allow 1 query every 3s: command line HCL: -hcl 'cache = { entry_fetch_rate = 0.333}' in JSON { "cache": { "entry_fetch_rate": 0.333 } }
2020-07-27 21:11:11 +00:00
func TestCacheRateLimit(test *testing.T) {
test.Parallel()
tests := []struct {
// count := number of updates performed (1 every 10ms)
count int
// rateLimit rate limiting of cache
rateLimit float64
// Minimum number of updates to see from a cache perspective
// We add a value with tolerance to work even on a loaded CI
minUpdates int
}{
// 250 => we have a test running for at least 2.5s
{250, 0.5, 1},
{250, 1, 1},
{300, 2, 2},
}
for _, currentTest := range tests {
test.Run(fmt.Sprintf("rate_limit_at_%v", currentTest.rateLimit), func(t *testing.T) {
tt := currentTest
t.Parallel()
a := NewTestAgent(t, fmt.Sprintf("cache = { entry_fetch_rate = %v, entry_fetch_max_burst = 1 }", tt.rateLimit))
defer a.Shutdown()
testrpc.WaitForTestAgent(t, a.RPC, "dc1")
var wg sync.WaitGroup
stillProcessing := true
injectService := func(i int) {
srv := &structs.NodeService{
Service: "redis",
ID: "redis",
Port: 1024 + i,
Address: fmt.Sprintf("10.0.1.%d", i%255),
}
err := a.AddService(srv, []*structs.CheckType{}, false, "", ConfigSourceRemote)
require.Nil(t, err)
}
runUpdates := func() {
wg.Add(tt.count)
for i := 0; i < tt.count; i++ {
time.Sleep(10 * time.Millisecond)
injectService(i)
wg.Done()
}
stillProcessing = false
}
getIndex := func(t *testing.T, oldIndex int) int {
req, err := http.NewRequest("GET", fmt.Sprintf("/v1/health/service/redis?cached&wait=5s&index=%d", oldIndex), nil)
require.NoError(t, err)
resp := httptest.NewRecorder()
Fix some broken code in master There were several PRs that while all passed CI independently, when they all got merged into the same branch caused compilation errors in test code. The main changes that caused issues where changing agent/cache.Cache.New to require a concrete options struct instead of a pointer. This broke the cert monitor tests and the catalog_list_services_test.go. Another change was made to unembed the http.Server from the agent.HTTPServer struct. That coupled with another change to add a test to ensure cache rate limiting coming from HTTP requests was working as expected caused compilation failures.
2020-07-28 13:50:10 +00:00
a.srv.handler(false).ServeHTTP(resp, req)
Added ratelimit to handle throtling cache (#8226) This implements a solution for #7863 It does: Add a new config cache.entry_fetch_rate to limit the number of calls/s for a given cache entry, default value = rate.Inf Add cache.entry_fetch_max_burst size of rate limit (default value = 2) The new configuration now supports the following syntax for instance to allow 1 query every 3s: command line HCL: -hcl 'cache = { entry_fetch_rate = 0.333}' in JSON { "cache": { "entry_fetch_rate": 0.333 } }
2020-07-27 21:11:11 +00:00
// Key doesn't actually exist so we should get 404
if got, want := resp.Code, http.StatusOK; got != want {
t.Fatalf("bad response code got %d want %d", got, want)
}
index, err := strconv.Atoi(resp.Header().Get("X-Consul-Index"))
require.NoError(t, err)
return index
}
{
start := time.Now()
injectService(0)
// Get the first index
index := getIndex(t, 0)
require.Greater(t, index, 2)
go runUpdates()
numberOfUpdates := 0
for stillProcessing {
oldIndex := index
index = getIndex(t, oldIndex)
require.GreaterOrEqual(t, index, oldIndex, "index must be increasing only")
numberOfUpdates++
}
elapsed := time.Since(start)
qps := float64(time.Second) * float64(numberOfUpdates) / float64(elapsed)
summary := fmt.Sprintf("received %v updates in %v aka %f qps, target max was: %f qps", numberOfUpdates, elapsed, qps, tt.rateLimit)
// We must never go beyond the limit, we give 10% margin to avoid having values like 1.05 instead of 1 due to precision of clock
require.LessOrEqual(t, qps, 1.1*tt.rateLimit, fmt.Sprintf("it should never get more requests than ratelimit, had: %s", summary))
// We must have at least being notified a few times
require.GreaterOrEqual(t, numberOfUpdates, tt.minUpdates, fmt.Sprintf("It should have received a minimum of %d updates, had: %s", tt.minUpdates, summary))
}
wg.Wait()
})
}
}
Add support for dual stack IPv4/IPv6 network (#6640) * Use consts for well known tagged adress keys * Add ipv4 and ipv6 tagged addresses for node lan and wan * Add ipv4 and ipv6 tagged addresses for service lan and wan * Use IPv4 and IPv6 address in DNS
2020-01-17 14:54:17 +00:00
func TestAddServiceIPv4TaggedDefault(t *testing.T) {
t.Helper()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
Add support for dual stack IPv4/IPv6 network (#6640) * Use consts for well known tagged adress keys * Add ipv4 and ipv6 tagged addresses for node lan and wan * Add ipv4 and ipv6 tagged addresses for service lan and wan * Use IPv4 and IPv6 address in DNS
2020-01-17 14:54:17 +00:00
defer a.Shutdown()
testrpc.WaitForTestAgent(t, a.RPC, "dc1")
srv := &structs.NodeService{
Service: "my_service",
ID: "my_service_id",
Port: 8100,
Address: "10.0.1.2",
}
err := a.AddService(srv, []*structs.CheckType{}, false, "", ConfigSourceRemote)
require.Nil(t, err)
ns := a.State.Service(structs.NewServiceID("my_service_id", nil))
require.NotNil(t, ns)
svcAddr := structs.ServiceAddress{Address: srv.Address, Port: srv.Port}
require.Equal(t, svcAddr, ns.TaggedAddresses[structs.TaggedAddressLANIPv4])
require.Equal(t, svcAddr, ns.TaggedAddresses[structs.TaggedAddressWANIPv4])
_, ok := ns.TaggedAddresses[structs.TaggedAddressLANIPv6]
require.False(t, ok)
_, ok = ns.TaggedAddresses[structs.TaggedAddressWANIPv6]
require.False(t, ok)
}
func TestAddServiceIPv6TaggedDefault(t *testing.T) {
t.Helper()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
Add support for dual stack IPv4/IPv6 network (#6640) * Use consts for well known tagged adress keys * Add ipv4 and ipv6 tagged addresses for node lan and wan * Add ipv4 and ipv6 tagged addresses for service lan and wan * Use IPv4 and IPv6 address in DNS
2020-01-17 14:54:17 +00:00
defer a.Shutdown()
testrpc.WaitForTestAgent(t, a.RPC, "dc1")
srv := &structs.NodeService{
Service: "my_service",
ID: "my_service_id",
Port: 8100,
Address: "::5",
}
err := a.AddService(srv, []*structs.CheckType{}, false, "", ConfigSourceRemote)
require.Nil(t, err)
ns := a.State.Service(structs.NewServiceID("my_service_id", nil))
require.NotNil(t, ns)
svcAddr := structs.ServiceAddress{Address: srv.Address, Port: srv.Port}
require.Equal(t, svcAddr, ns.TaggedAddresses[structs.TaggedAddressLANIPv6])
require.Equal(t, svcAddr, ns.TaggedAddresses[structs.TaggedAddressWANIPv6])
_, ok := ns.TaggedAddresses[structs.TaggedAddressLANIPv4]
require.False(t, ok)
_, ok = ns.TaggedAddresses[structs.TaggedAddressWANIPv4]
require.False(t, ok)
}
func TestAddServiceIPv4TaggedSet(t *testing.T) {
t.Helper()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
Add support for dual stack IPv4/IPv6 network (#6640) * Use consts for well known tagged adress keys * Add ipv4 and ipv6 tagged addresses for node lan and wan * Add ipv4 and ipv6 tagged addresses for service lan and wan * Use IPv4 and IPv6 address in DNS
2020-01-17 14:54:17 +00:00
defer a.Shutdown()
testrpc.WaitForTestAgent(t, a.RPC, "dc1")
srv := &structs.NodeService{
Service: "my_service",
ID: "my_service_id",
Port: 8100,
Address: "10.0.1.2",
TaggedAddresses: map[string]structs.ServiceAddress{
structs.TaggedAddressWANIPv4: {
Address: "10.100.200.5",
Port: 8100,
},
},
}
err := a.AddService(srv, []*structs.CheckType{}, false, "", ConfigSourceRemote)
require.Nil(t, err)
ns := a.State.Service(structs.NewServiceID("my_service_id", nil))
require.NotNil(t, ns)
svcAddr := structs.ServiceAddress{Address: srv.Address, Port: srv.Port}
require.Equal(t, svcAddr, ns.TaggedAddresses[structs.TaggedAddressLANIPv4])
require.Equal(t, structs.ServiceAddress{Address: "10.100.200.5", Port: 8100}, ns.TaggedAddresses[structs.TaggedAddressWANIPv4])
_, ok := ns.TaggedAddresses[structs.TaggedAddressLANIPv6]
require.False(t, ok)
_, ok = ns.TaggedAddresses[structs.TaggedAddressWANIPv6]
require.False(t, ok)
}
func TestAddServiceIPv6TaggedSet(t *testing.T) {
t.Helper()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
Add support for dual stack IPv4/IPv6 network (#6640) * Use consts for well known tagged adress keys * Add ipv4 and ipv6 tagged addresses for node lan and wan * Add ipv4 and ipv6 tagged addresses for service lan and wan * Use IPv4 and IPv6 address in DNS
2020-01-17 14:54:17 +00:00
defer a.Shutdown()
testrpc.WaitForTestAgent(t, a.RPC, "dc1")
srv := &structs.NodeService{
Service: "my_service",
ID: "my_service_id",
Port: 8100,
Address: "::5",
TaggedAddresses: map[string]structs.ServiceAddress{
structs.TaggedAddressWANIPv6: {
Address: "::6",
Port: 8100,
},
},
}
err := a.AddService(srv, []*structs.CheckType{}, false, "", ConfigSourceRemote)
require.Nil(t, err)
ns := a.State.Service(structs.NewServiceID("my_service_id", nil))
require.NotNil(t, ns)
svcAddr := structs.ServiceAddress{Address: srv.Address, Port: srv.Port}
require.Equal(t, svcAddr, ns.TaggedAddresses[structs.TaggedAddressLANIPv6])
require.Equal(t, structs.ServiceAddress{Address: "::6", Port: 8100}, ns.TaggedAddresses[structs.TaggedAddressWANIPv6])
_, ok := ns.TaggedAddresses[structs.TaggedAddressLANIPv4]
require.False(t, ok)
_, ok = ns.TaggedAddresses[structs.TaggedAddressWANIPv4]
require.False(t, ok)
}
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
func TestAgent_RemoveService(t *testing.T) {
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
t.Run("normal", func(t *testing.T) {
t.Parallel()
testAgent_RemoveService(t, "")
})
t.Run("service manager", func(t *testing.T) {
t.Parallel()
testAgent_RemoveService(t, "enable_central_service_config = true")
})
}
func testAgent_RemoveService(t *testing.T, extraHCL string) {
t.Helper()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, extraHCL)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
// Remove a service that doesn't exist
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.RemoveService(structs.NewServiceID("redis", nil)); err != nil {
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
t.Fatalf("err: %v", err)
}
Add tests to remove service/check without an ID
2015-01-27 09:10:56 +00:00
// Remove without an ID
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.RemoveService(structs.NewServiceID("", nil)); err == nil {
Add tests to remove service/check without an ID
2015-01-27 09:10:56 +00:00
t.Fatalf("should have errored")
}
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
// Removing a service with a single check works
{
srv := &structs.NodeService{
ID: "memcache",
Service: "memcache",
Port: 8000,
}
Enable gofmt simplify Code changes done automatically with 'gofmt -s -w'
2020-06-16 17:19:31 +00:00
chkTypes := []*structs.CheckType{{TTL: time.Minute}}
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddService(srv, chkTypes, false, "", ConfigSourceLocal); err != nil {
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
t.Fatalf("err: %v", err)
}
agent: use service ID field to determine associated health checks during deregister
2015-05-07 22:30:01 +00:00
// Add a check after the fact with a specific check ID
agent: move structs into consul/structs pkg * CheckDefinition * ServiceDefinition * CheckType
2017-06-15 16:46:06 +00:00
check := &structs.CheckDefinition{
agent: use service ID field to determine associated health checks during deregister
2015-05-07 22:30:01 +00:00
ID: "check2",
Name: "check2",
ServiceID: "memcache",
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
TTL: time.Minute,
agent: use service ID field to determine associated health checks during deregister
2015-05-07 22:30:01 +00:00
}
hc := check.HealthCheck("node1")
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddCheck(hc, check.CheckType(), false, "", ConfigSourceLocal); err != nil {
agent: use service ID field to determine associated health checks during deregister
2015-05-07 22:30:01 +00:00
t.Fatalf("err: %s", err)
}
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.RemoveService(structs.NewServiceID("memcache", nil)); err != nil {
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
t.Fatalf("err: %s", err)
}
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
require.Nil(t, a.State.Check(structs.NewCheckID("service:memcache", nil)), "have memcache check")
require.Nil(t, a.State.Check(structs.NewCheckID("check2", nil)), "have check2 check")
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
}
// Removing a service with multiple checks works
{
Fix checks removal when removing service (#5457) Fix my recently discovered issue described here: #5456
2019-03-14 15:02:49 +00:00
// add a service to remove
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
srv := &structs.NodeService{
ID: "redis",
Service: "redis",
Port: 8000,
}
agent: move structs into consul/structs pkg * CheckDefinition * ServiceDefinition * CheckType
2017-06-15 16:46:06 +00:00
chkTypes := []*structs.CheckType{
Enable gofmt simplify Code changes done automatically with 'gofmt -s -w'
2020-06-16 17:19:31 +00:00
{TTL: time.Minute},
{TTL: 30 * time.Second},
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddService(srv, chkTypes, false, "", ConfigSourceLocal); err != nil {
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
t.Fatalf("err: %v", err)
}
Fix checks removal when removing service (#5457) Fix my recently discovered issue described here: #5456
2019-03-14 15:02:49 +00:00
// add another service that wont be affected
srv = &structs.NodeService{
ID: "mysql",
Service: "mysql",
Port: 3306,
}
chkTypes = []*structs.CheckType{
Enable gofmt simplify Code changes done automatically with 'gofmt -s -w'
2020-06-16 17:19:31 +00:00
{TTL: time.Minute},
{TTL: 30 * time.Second},
Fix checks removal when removing service (#5457) Fix my recently discovered issue described here: #5456
2019-03-14 15:02:49 +00:00
}
if err := a.AddService(srv, chkTypes, false, "", ConfigSourceLocal); err != nil {
t.Fatalf("err: %v", err)
}
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
// Remove the service
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.RemoveService(structs.NewServiceID("redis", nil)); err != nil {
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
t.Fatalf("err: %v", err)
}
// Ensure we have a state mapping
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireServiceMissing(t, a, "redis")
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
// Ensure checks were removed
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireCheckMissing(t, a, "service:redis:1")
requireCheckMissing(t, a, "service:redis:2")
requireCheckMissingMap(t, a.checkTTLs, "service:redis:1")
requireCheckMissingMap(t, a.checkTTLs, "service:redis:2")
Fix checks removal when removing service (#5457) Fix my recently discovered issue described here: #5456
2019-03-14 15:02:49 +00:00
// check the mysql service is unnafected
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireCheckExistsMap(t, a.checkTTLs, "service:mysql:1")
requireCheckExists(t, a, "service:mysql:1")
requireCheckExistsMap(t, a.checkTTLs, "service:mysql:2")
requireCheckExists(t, a, "service:mysql:2")
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
}
}
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
func TestAgent_RemoveServiceRemovesAllChecks(t *testing.T) {
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
t.Run("normal", func(t *testing.T) {
t.Parallel()
testAgent_RemoveServiceRemovesAllChecks(t, "")
})
t.Run("service manager", func(t *testing.T) {
t.Parallel()
testAgent_RemoveServiceRemovesAllChecks(t, "enable_central_service_config = true")
})
}
func testAgent_RemoveServiceRemovesAllChecks(t *testing.T, extraHCL string) {
t.Helper()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
node_name = "node1"
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
`+extraHCL)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
svc := &structs.NodeService{ID: "redis", Service: "redis", Port: 8000, EnterpriseMeta: *structs.DefaultEnterpriseMeta()}
agent: move structs into consul/structs pkg * CheckDefinition * ServiceDefinition * CheckType
2017-06-15 16:46:06 +00:00
chk1 := &structs.CheckType{CheckID: "chk1", Name: "chk1", TTL: time.Minute}
chk2 := &structs.CheckType{CheckID: "chk2", Name: "chk2", TTL: 2 * time.Minute}
Store check type in catalog (#6561)
2019-10-17 18:33:11 +00:00
hchk1 := &structs.HealthCheck{
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
Node: "node1",
CheckID: "chk1",
Name: "chk1",
Status: "critical",
ServiceID: "redis",
ServiceName: "redis",
Type: "ttl",
EnterpriseMeta: *structs.DefaultEnterpriseMeta(),
Store check type in catalog (#6561)
2019-10-17 18:33:11 +00:00
}
hchk2 := &structs.HealthCheck{Node: "node1",
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
CheckID: "chk2",
Name: "chk2",
Status: "critical",
ServiceID: "redis",
ServiceName: "redis",
Type: "ttl",
EnterpriseMeta: *structs.DefaultEnterpriseMeta(),
Store check type in catalog (#6561)
2019-10-17 18:33:11 +00:00
}
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
// register service with chk1
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddService(svc, []*structs.CheckType{chk1}, false, "", ConfigSourceLocal); err != nil {
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
t.Fatal("Failed to register service", err)
}
// verify chk1 exists
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireCheckExists(t, a, "chk1")
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
// update the service with chk2
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddService(svc, []*structs.CheckType{chk2}, false, "", ConfigSourceLocal); err != nil {
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
t.Fatal("Failed to update service", err)
}
// check that both checks are there
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
require.Equal(t, hchk1, getCheck(a, "chk1"))
require.Equal(t, hchk2, getCheck(a, "chk2"))
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
// Remove service
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.RemoveService(structs.NewServiceID("redis", nil)); err != nil {
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
t.Fatal("Failed to remove service", err)
}
// Check that both checks are gone
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireCheckMissing(t, a, "chk1")
requireCheckMissing(t, a, "chk2")
agent: support custom check id and name This patch adds support for a custom check id and name when registering a service. This is achieved by adding a CheckID and a Name field to the CheckType structure which is used to register checks with a service and when returning health check definitions. CheckDefinition is a superset of CheckType which duplicates some of the fields of CheckType. This patch decouples these two structures by removing the embedding of CheckType in CheckDefinition. Fixes #3047
2017-05-15 19:49:13 +00:00
}
Adds a more robust unit test for index churn.
2018-02-07 04:35:38 +00:00
// TestAgent_IndexChurn is designed to detect a class of issues where
// we would have unnecessary catalog churn from anti-entropy. See issues
// #3259, #3642, #3845, and #3866.
func TestAgent_IndexChurn(t *testing.T) {
Adds a before/after test for #3845.
2018-02-06 00:18:29 +00:00
t.Parallel()
Adds a more robust unit test for index churn.
2018-02-07 04:35:38 +00:00
t.Run("no tags", func(t *testing.T) {
verifyIndexChurn(t, nil)
})
t.Run("with tags", func(t *testing.T) {
verifyIndexChurn(t, []string{"foo", "bar"})
})
}
// verifyIndexChurn registers some things and runs anti-entropy a bunch of times
// in a row to make sure there are no index bumps.
func verifyIndexChurn(t *testing.T, tags []string) {
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
Adds a before/after test for #3845.
2018-02-06 00:18:29 +00:00
defer a.Shutdown()
Implementation of Weights Data structures (#4468) * Implementation of Weights Data structures Adding this datastructure will allow us to resolve the issues #1088 and #4198 This new structure defaults to values: ``` { Passing: 1, Warning: 0 } ``` Which means, use weight of 0 for a Service in Warning State while use Weight 1 for a Healthy Service. Thus it remains compatible with previous Consul versions. * Implemented weights for DNS SRV Records * DNS properly support agents with weight support while server does not (backwards compatibility) * Use Warning value of Weights of 1 by default When using DNS interface with only_passing = false, all nodes with non-Critical healthcheck used to have a weight value of 1. While having weight.Warning = 0 as default value, this is probably a bad idea as it breaks ascending compatibility. Thus, we put a default value of 1 to be consistent with existing behaviour. * Added documentation for new weight field in service description * Better documentation about weights as suggested by @banks * Return weight = 1 for unknown Check states as suggested by @banks * Fixed typo (of -> or) in error message as requested by @mkeeler * Fixed unstable unit test TestRetryJoin * Fixed unstable tests * Fixed wrong Fatalf format in `testrpc/wait.go` * Added notes regarding DNS SRV lookup limitations regarding number of instances * Documentation fixes and clarification regarding SRV records with weights as requested by @banks * Rephrase docs
2018-09-07 14:30:47 +00:00
weights := &structs.Weights{
Passing: 1,
Warning: 1,
}
PR to fix TestAgent_IndexChurn and TestPreparedQuery_Wrapper. (#4512) * Fixes TestAgent_IndexChurn * Fixes TestPreparedQuery_Wrapper * Increased sleep in agent_test for IndexChurn to 500ms * Made the comment about joinWAN operation much less of a cliffhanger
2018-08-09 16:40:07 +00:00
// Ensure we have a leader before we start adding the services
testrpc.WaitForLeader(t, a.RPC, "dc1")
Adds a before/after test for #3845.
2018-02-06 00:18:29 +00:00
svc := &structs.NodeService{
ID: "redis",
Service: "redis",
Port: 8000,
Adds a more robust unit test for index churn.
2018-02-07 04:35:38 +00:00
Tags: tags,
Implementation of Weights Data structures (#4468) * Implementation of Weights Data structures Adding this datastructure will allow us to resolve the issues #1088 and #4198 This new structure defaults to values: ``` { Passing: 1, Warning: 0 } ``` Which means, use weight of 0 for a Service in Warning State while use Weight 1 for a Healthy Service. Thus it remains compatible with previous Consul versions. * Implemented weights for DNS SRV Records * DNS properly support agents with weight support while server does not (backwards compatibility) * Use Warning value of Weights of 1 by default When using DNS interface with only_passing = false, all nodes with non-Critical healthcheck used to have a weight value of 1. While having weight.Warning = 0 as default value, this is probably a bad idea as it breaks ascending compatibility. Thus, we put a default value of 1 to be consistent with existing behaviour. * Added documentation for new weight field in service description * Better documentation about weights as suggested by @banks * Return weight = 1 for unknown Check states as suggested by @banks * Fixed typo (of -> or) in error message as requested by @mkeeler * Fixed unstable unit test TestRetryJoin * Fixed unstable tests * Fixed wrong Fatalf format in `testrpc/wait.go` * Added notes regarding DNS SRV lookup limitations regarding number of instances * Documentation fixes and clarification regarding SRV records with weights as requested by @banks * Rephrase docs
2018-09-07 14:30:47 +00:00
Weights: weights,
Adds a before/after test for #3845.
2018-02-06 00:18:29 +00:00
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddService(svc, nil, true, "", ConfigSourceLocal); err != nil {
Adds a before/after test for #3845.
2018-02-06 00:18:29 +00:00
t.Fatalf("err: %v", err)
}
chk := &structs.HealthCheck{
CheckID: "redis-check",
Adds a more robust unit test for index churn.
2018-02-07 04:35:38 +00:00
Name: "Service-level check",
Adds a before/after test for #3845.
2018-02-06 00:18:29 +00:00
ServiceID: "redis",
Status: api.HealthCritical,
}
chkt := &structs.CheckType{
TTL: time.Hour,
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddCheck(chk, chkt, true, "", ConfigSourceLocal); err != nil {
Adds a before/after test for #3845.
2018-02-06 00:18:29 +00:00
t.Fatalf("err: %v", err)
}
Adds a more robust unit test for index churn.
2018-02-07 04:35:38 +00:00
chk = &structs.HealthCheck{
CheckID: "node-check",
Name: "Node-level check",
Status: api.HealthCritical,
}
chkt = &structs.CheckType{
TTL: time.Hour,
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddCheck(chk, chkt, true, "", ConfigSourceLocal); err != nil {
Adds a more robust unit test for index churn.
2018-02-07 04:35:38 +00:00
t.Fatalf("err: %v", err)
}
Adds a before/after test for #3845.
2018-02-06 00:18:29 +00:00
if err := a.sync.State.SyncFull(); err != nil {
t.Fatalf("err: %v", err)
}
Adds a more robust unit test for index churn.
2018-02-07 04:35:38 +00:00
Adds a before/after test for #3845.
2018-02-06 00:18:29 +00:00
args := &structs.ServiceSpecificRequest{
Datacenter: "dc1",
ServiceName: "redis",
}
Adds a more robust unit test for index churn.
2018-02-07 04:35:38 +00:00
var before structs.IndexedCheckServiceNodes
PR to fix TestAgent_IndexChurn and TestPreparedQuery_Wrapper. (#4512) * Fixes TestAgent_IndexChurn * Fixes TestPreparedQuery_Wrapper * Increased sleep in agent_test for IndexChurn to 500ms * Made the comment about joinWAN operation much less of a cliffhanger
2018-08-09 16:40:07 +00:00
// This sleep is so that the serfHealth check is added to the agent
// A value of 375ms is sufficient enough time to ensure the serfHealth
// check is added to an agent. 500ms so that we don't see flakiness ever.
time.Sleep(500 * time.Millisecond)
Adds a more robust unit test for index churn.
2018-02-07 04:35:38 +00:00
if err := a.RPC("Health.ServiceNodes", args, &before); err != nil {
Adds a before/after test for #3845.
2018-02-06 00:18:29 +00:00
t.Fatalf("err: %v", err)
}
PR to fix TestAgent_IndexChurn and TestPreparedQuery_Wrapper. (#4512) * Fixes TestAgent_IndexChurn * Fixes TestPreparedQuery_Wrapper * Increased sleep in agent_test for IndexChurn to 500ms * Made the comment about joinWAN operation much less of a cliffhanger
2018-08-09 16:40:07 +00:00
for _, name := range before.Nodes[0].Checks {
Allow users to configure either unstructured or JSON logging (#7130) * hclog Allow users to choose between unstructured and JSON logging
2020-01-28 23:50:41 +00:00
a.logger.Debug("Registered node", "node", name.Name)
PR to fix TestAgent_IndexChurn and TestPreparedQuery_Wrapper. (#4512) * Fixes TestAgent_IndexChurn * Fixes TestPreparedQuery_Wrapper * Increased sleep in agent_test for IndexChurn to 500ms * Made the comment about joinWAN operation much less of a cliffhanger
2018-08-09 16:40:07 +00:00
}
Adds a more robust unit test for index churn.
2018-02-07 04:35:38 +00:00
if got, want := len(before.Nodes), 1; got != want {
t.Fatalf("got %d want %d", got, want)
}
if got, want := len(before.Nodes[0].Checks), 3; /* incl. serfHealth */ got != want {
Adds a before/after test for #3845.
2018-02-06 00:18:29 +00:00
t.Fatalf("got %d want %d", got, want)
}
for i := 0; i < 10; i++ {
Allow users to configure either unstructured or JSON logging (#7130) * hclog Allow users to choose between unstructured and JSON logging
2020-01-28 23:50:41 +00:00
a.logger.Info("Sync in progress", "iteration", i+1)
Adds a before/after test for #3845.
2018-02-06 00:18:29 +00:00
if err := a.sync.State.SyncFull(); err != nil {
t.Fatalf("err: %v", err)
}
}
PR to fix TestAgent_IndexChurn and TestPreparedQuery_Wrapper. (#4512) * Fixes TestAgent_IndexChurn * Fixes TestPreparedQuery_Wrapper * Increased sleep in agent_test for IndexChurn to 500ms * Made the comment about joinWAN operation much less of a cliffhanger
2018-08-09 16:40:07 +00:00
// If this test fails here this means that the Consul-X-Index
// has changed for the RPC, which means that idempotent ops
// are not working as intended.
Adds a more robust unit test for index churn.
2018-02-07 04:35:38 +00:00
var after structs.IndexedCheckServiceNodes
if err := a.RPC("Health.ServiceNodes", args, &after); err != nil {
Adds a before/after test for #3845.
2018-02-06 00:18:29 +00:00
t.Fatalf("err: %v", err)
}
Replace goe/verify.Values with testify/require.Equal (#7993) * testing: replace most goe/verify.Values with require.Equal One difference between these two comparisons is that go/verify considers nil slices/maps to be equal to empty slices/maps, where as testify/require does not, and does not appear to provide any way to enable that behaviour. Because of this difference some expected values were changed from empty slices to nil slices, and some calls to verify.Values were left. * Remove github.com/pascaldekloe/goe/verify Reduce the number of assertion packages we use from 2 to 1
2020-06-02 16:41:25 +00:00
require.Equal(t, before, after)
Adds a before/after test for #3845.
2018-02-06 00:18:29 +00:00
}
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
func TestAgent_AddCheck(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
enable_script_checks = true
`)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
health := &structs.HealthCheck{
Node: "foo",
CheckID: "mem",
Name: "memory util",
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
Status: api.HealthCritical,
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
}
agent: move structs into consul/structs pkg * CheckDefinition * ServiceDefinition * CheckType
2017-06-15 16:46:06 +00:00
chk := &structs.CheckType{
Remove the script field from checks in favor of args
2018-05-08 22:31:53 +00:00
ScriptArgs: []string{"exit", "0"},
Interval: 15 * time.Second,
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
err := a.AddCheck(health, chk, false, "", ConfigSourceLocal)
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
if err != nil {
t.Fatalf("err: %v", err)
}
// Ensure we have a check mapping
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
sChk := requireCheckExists(t, a, "mem")
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
Allow specifying a status field in the agent/service/register and agent/check/register endpoints. This status must be one of the valid check statuses: 'passing', 'warning', 'critical', 'unknown'. If the status field is not present or the empty string, the default of 'critical' is used.
2015-04-12 00:53:48 +00:00
// Ensure our check is in the right state
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
if sChk.Status != api.HealthCritical {
Allow specifying a status field in the agent/service/register and agent/check/register endpoints. This status must be one of the valid check statuses: 'passing', 'warning', 'critical', 'unknown'. If the status field is not present or the empty string, the default of 'critical' is used.
2015-04-12 00:53:48 +00:00
t.Fatalf("check not critical")
}
// Ensure a TTL is setup
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireCheckExistsMap(t, a.checkMonitors, "mem")
Allow specifying a status field in the agent/service/register and agent/check/register endpoints. This status must be one of the valid check statuses: 'passing', 'warning', 'critical', 'unknown'. If the status field is not present or the empty string, the default of 'critical' is used.
2015-04-12 00:53:48 +00:00
}
func TestAgent_AddCheck_StartPassing(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
enable_script_checks = true
`)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
Allow specifying a status field in the agent/service/register and agent/check/register endpoints. This status must be one of the valid check statuses: 'passing', 'warning', 'critical', 'unknown'. If the status field is not present or the empty string, the default of 'critical' is used.
2015-04-12 00:53:48 +00:00
health := &structs.HealthCheck{
Node: "foo",
CheckID: "mem",
Name: "memory util",
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
Status: api.HealthPassing,
Allow specifying a status field in the agent/service/register and agent/check/register endpoints. This status must be one of the valid check statuses: 'passing', 'warning', 'critical', 'unknown'. If the status field is not present or the empty string, the default of 'critical' is used.
2015-04-12 00:53:48 +00:00
}
agent: move structs into consul/structs pkg * CheckDefinition * ServiceDefinition * CheckType
2017-06-15 16:46:06 +00:00
chk := &structs.CheckType{
Remove the script field from checks in favor of args
2018-05-08 22:31:53 +00:00
ScriptArgs: []string{"exit", "0"},
Interval: 15 * time.Second,
Allow specifying a status field in the agent/service/register and agent/check/register endpoints. This status must be one of the valid check statuses: 'passing', 'warning', 'critical', 'unknown'. If the status field is not present or the empty string, the default of 'critical' is used.
2015-04-12 00:53:48 +00:00
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
err := a.AddCheck(health, chk, false, "", ConfigSourceLocal)
Allow specifying a status field in the agent/service/register and agent/check/register endpoints. This status must be one of the valid check statuses: 'passing', 'warning', 'critical', 'unknown'. If the status field is not present or the empty string, the default of 'critical' is used.
2015-04-12 00:53:48 +00:00
if err != nil {
t.Fatalf("err: %v", err)
}
// Ensure we have a check mapping
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
sChk := requireCheckExists(t, a, "mem")
Allow specifying a status field in the agent/service/register and agent/check/register endpoints. This status must be one of the valid check statuses: 'passing', 'warning', 'critical', 'unknown'. If the status field is not present or the empty string, the default of 'critical' is used.
2015-04-12 00:53:48 +00:00
// Ensure our check is in the right state
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
if sChk.Status != api.HealthPassing {
Allow specifying a status field in the agent/service/register and agent/check/register endpoints. This status must be one of the valid check statuses: 'passing', 'warning', 'critical', 'unknown'. If the status field is not present or the empty string, the default of 'critical' is used.
2015-04-12 00:53:48 +00:00
t.Fatalf("check not passing")
}
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
// Ensure a TTL is setup
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireCheckExistsMap(t, a.checkMonitors, "mem")
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
}
agent: Adding minimum check interval. Fixes #64.
2014-04-21 21:42:42 +00:00
func TestAgent_AddCheck_MinInterval(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
enable_script_checks = true
`)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: Adding minimum check interval. Fixes #64.
2014-04-21 21:42:42 +00:00
health := &structs.HealthCheck{
Node: "foo",
CheckID: "mem",
Name: "memory util",
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
Status: api.HealthCritical,
agent: Adding minimum check interval. Fixes #64.
2014-04-21 21:42:42 +00:00
}
agent: move structs into consul/structs pkg * CheckDefinition * ServiceDefinition * CheckType
2017-06-15 16:46:06 +00:00
chk := &structs.CheckType{
Remove the script field from checks in favor of args
2018-05-08 22:31:53 +00:00
ScriptArgs: []string{"exit", "0"},
Interval: time.Microsecond,
agent: Adding minimum check interval. Fixes #64.
2014-04-21 21:42:42 +00:00
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
err := a.AddCheck(health, chk, false, "", ConfigSourceLocal)
agent: Adding minimum check interval. Fixes #64.
2014-04-21 21:42:42 +00:00
if err != nil {
t.Fatalf("err: %v", err)
}
// Ensure we have a check mapping
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireCheckExists(t, a, "mem")
agent: Adding minimum check interval. Fixes #64.
2014-04-21 21:42:42 +00:00
// Ensure a TTL is setup
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if mon, ok := a.checkMonitors[structs.NewCheckID("mem", nil)]; !ok {
agent: Adding minimum check interval. Fixes #64.
2014-04-21 21:42:42 +00:00
t.Fatalf("missing mem monitor")
Decouple the code that executes checks from the agent
2017-10-25 09:18:07 +00:00
} else if mon.Interval != checks.MinInterval {
agent: Adding minimum check interval. Fixes #64.
2014-04-21 21:42:42 +00:00
t.Fatalf("bad mem monitor interval")
}
}
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
func TestAgent_AddCheck_MissingService(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
enable_script_checks = true
`)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
health := &structs.HealthCheck{
Node: "foo",
CheckID: "baz",
Name: "baz check 1",
ServiceID: "baz",
}
agent: move structs into consul/structs pkg * CheckDefinition * ServiceDefinition * CheckType
2017-06-15 16:46:06 +00:00
chk := &structs.CheckType{
Remove the script field from checks in favor of args
2018-05-08 22:31:53 +00:00
ScriptArgs: []string{"exit", "0"},
Interval: time.Microsecond,
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
err := a.AddCheck(health, chk, false, "", ConfigSourceLocal)
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err == nil || err.Error() != fmt.Sprintf("ServiceID %q does not exist", structs.ServiceIDString("baz", nil)) {
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
t.Fatalf("expected service id error, got: %v", err)
}
}
agent: test check state restoration from AddCheck
2015-06-06 00:33:34 +00:00
func TestAgent_AddCheck_RestoreState(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: test check state restoration from AddCheck
2015-06-06 00:33:34 +00:00
// Create some state and persist it
Decouple the code that executes checks from the agent
2017-10-25 09:18:07 +00:00
ttl := &checks.CheckTTL{
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
CheckID: structs.NewCheckID("baz", nil),
agent: test check state restoration from AddCheck
2015-06-06 00:33:34 +00:00
TTL: time.Minute,
}
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
err := a.persistCheckState(ttl, api.HealthPassing, "yup")
agent: test check state restoration from AddCheck
2015-06-06 00:33:34 +00:00
if err != nil {
t.Fatalf("err: %s", err)
}
// Build and register the check definition and initial state
health := &structs.HealthCheck{
Node: "foo",
CheckID: "baz",
Name: "baz check 1",
}
agent: move structs into consul/structs pkg * CheckDefinition * ServiceDefinition * CheckType
2017-06-15 16:46:06 +00:00
chk := &structs.CheckType{
agent: test check state restoration from AddCheck
2015-06-06 00:33:34 +00:00
TTL: time.Minute,
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
err = a.AddCheck(health, chk, false, "", ConfigSourceLocal)
agent: test check state restoration from AddCheck
2015-06-06 00:33:34 +00:00
if err != nil {
t.Fatalf("err: %s", err)
}
// Ensure the check status was restored during registration
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
check := requireCheckExists(t, a, "baz")
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
if check.Status != api.HealthPassing {
agent: test check state restoration from AddCheck
2015-06-06 00:33:34 +00:00
t.Fatalf("bad: %#v", check)
}
if check.Output != "yup" {
t.Fatalf("bad: %#v", check)
}
}
Adds new config to make script checks opt-in, updates documentation. (#3284)
2017-07-17 18:20:35 +00:00
func TestAgent_AddCheck_ExecDisable(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Adds new config to make script checks opt-in, updates documentation. (#3284)
2017-07-17 18:20:35 +00:00
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
Adds new config to make script checks opt-in, updates documentation. (#3284)
2017-07-17 18:20:35 +00:00
health := &structs.HealthCheck{
Node: "foo",
CheckID: "mem",
Name: "memory util",
Status: api.HealthCritical,
}
chk := &structs.CheckType{
Remove the script field from checks in favor of args
2018-05-08 22:31:53 +00:00
ScriptArgs: []string{"exit", "0"},
Interval: 15 * time.Second,
Adds new config to make script checks opt-in, updates documentation. (#3284)
2017-07-17 18:20:35 +00:00
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
err := a.AddCheck(health, chk, false, "", ConfigSourceLocal)
Tweaks the error when scripts are disabled. This will hopefully help people self-serve if they upgrade without accounting for this.
2017-07-20 05:15:04 +00:00
if err == nil || !strings.Contains(err.Error(), "Scripts are disabled on this agent") {
Adds new config to make script checks opt-in, updates documentation. (#3284)
2017-07-17 18:20:35 +00:00
t.Fatalf("err: %v", err)
}
// Ensure we don't have a check mapping
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireCheckMissing(t, a, "mem")
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
err = a.AddCheck(health, chk, false, "", ConfigSourceRemote)
if err == nil || !strings.Contains(err.Error(), "Scripts are disabled on this agent") {
t.Fatalf("err: %v", err)
}
// Ensure we don't have a check mapping
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireCheckMissing(t, a, "mem")
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
}
func TestAgent_AddCheck_ExecRemoteDisable(t *testing.T) {
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
enable_local_script_checks = true
`)
defer a.Shutdown()
testrpc.WaitForTestAgent(t, a.RPC, "dc1")
health := &structs.HealthCheck{
Node: "foo",
CheckID: "mem",
Name: "memory util",
Status: api.HealthCritical,
}
chk := &structs.CheckType{
ScriptArgs: []string{"exit", "0"},
Interval: 15 * time.Second,
}
err := a.AddCheck(health, chk, false, "", ConfigSourceRemote)
if err == nil || !strings.Contains(err.Error(), "Scripts are disabled on this agent from remote calls") {
t.Fatalf("err: %v", err)
}
// Ensure we don't have a check mapping
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireCheckMissing(t, a, "mem")
Adds new config to make script checks opt-in, updates documentation. (#3284)
2017-07-17 18:20:35 +00:00
}
Add gRPC health-check #3073
2017-12-27 04:35:22 +00:00
func TestAgent_AddCheck_GRPC(t *testing.T) {
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
Add gRPC health-check #3073
2017-12-27 04:35:22 +00:00
defer a.Shutdown()
health := &structs.HealthCheck{
Node: "foo",
CheckID: "grpchealth",
Name: "grpc health checking protocol",
Status: api.HealthCritical,
}
chk := &structs.CheckType{
GRPC: "localhost:12345/package.Service",
Interval: 15 * time.Second,
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
err := a.AddCheck(health, chk, false, "", ConfigSourceLocal)
Add gRPC health-check #3073
2017-12-27 04:35:22 +00:00
if err != nil {
t.Fatalf("err: %v", err)
}
// Ensure we have a check mapping
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
sChk := requireCheckExists(t, a, "grpchealth")
Add gRPC health-check #3073
2017-12-27 04:35:22 +00:00
// Ensure our check is in the right state
if sChk.Status != api.HealthCritical {
t.Fatalf("check not critical")
}
// Ensure a check is setup
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireCheckExistsMap(t, a.checkGRPCs, "grpchealth")
Add gRPC health-check #3073
2017-12-27 04:35:22 +00:00
}
agent: fix several data races and bugs related to node-local alias checks (#5876) The observed bug was that a full restart of a consul datacenter (servers and clients) in conjunction with a restart of a connect-flavored application with bring-your-own-service-registration logic would very frequently cause the envoy sidecar service check to never reflect the aliased service. Over the course of investigation several bugs and unfortunate interactions were corrected: (1) local.CheckState objects were only shallow copied, but the key piece of data that gets read and updated is one of the things not copied (the underlying Check with a Status field). When the stock code was run with the race detector enabled this highly-relevant-to-the-test-scenario field was found to be racy. Changes: a) update the existing Clone method to include the Check field b) copy-on-write when those fields need to change rather than incrementally updating them in place. This made the observed behavior occur slightly less often. (2) If anything about how the runLocal method for node-local alias check logic was ever flawed, there was no fallback option. Those checks are purely edge-triggered and failure to properly notice a single edge transition would leave the alias check incorrect until the next flap of the aliased check. The change was to introduce a fallback timer to act as a control loop to double check the alias check matches the aliased check every minute (borrowing the duration from the non-local alias check logic body). This made the observed behavior eventually go away when it did occur. (3) Originally I thought there were two main actions involved in the data race: A. The act of adding the original check (from disk recovery) and its first health evaluation. B. The act of the HTTP API requests coming in and resetting the local state when re-registering the same services and checks. It took awhile for me to realize that there's a third action at work: C. The goroutines associated with the original check and the later checks. The actual sequence of actions that was causing the bad behavior was that the API actions result in the original check to be removed and re-added _without waiting for the original goroutine to terminate_. This means for brief windows of time during check definition edits there are two goroutines that can be sending updates for the alias check status. In extremely unlikely scenarios the original goroutine sees the aliased check start up in `critical` before being removed but does not get the notification about the nearly immediate update of that check to `passing`. This is interlaced wit the new goroutine coming up, initializing its base case to `passing` from the current state and then listening for new notifications of edge triggers. If the original goroutine "finishes" its update, it then commits one more write into the local state of `critical` and exits leaving the alias check no longer reflecting the underlying check. The correction here is to enforce that the old goroutines must terminate before spawning the new one for alias checks.
2019-05-24 18:36:56 +00:00
func TestAgent_RestoreServiceWithAliasCheck(t *testing.T) {
// t.Parallel() don't even think about making this parallel
// This test is very contrived and tests for the absence of race conditions
// related to the implementation of alias checks. As such it is slow,
// serial, full of sleeps and retries, and not generally a great test to
// run all of the time.
//
// That said it made it incredibly easy to root out various race conditions
// quite successfully.
//
// The original set of races was between:
//
// - agent startup reloading Services and Checks from disk
// - API requests to also re-register those same Services and Checks
// - the goroutines for the as-yet-to-be-stopped CheckAlias goroutines
if os.Getenv("SLOWTEST") != "1" {
t.Skip("skipping slow test; set SLOWTEST=1 to run")
return
}
// We do this so that the agent logs and the informational messages from
// the test itself are interwoven properly.
logf := func(t *testing.T, a *TestAgent, format string, args ...interface{}) {
Allow users to configure either unstructured or JSON logging (#7130) * hclog Allow users to choose between unstructured and JSON logging
2020-01-28 23:50:41 +00:00
a.logger.Info("testharness: " + fmt.Sprintf(format, args...))
agent: fix several data races and bugs related to node-local alias checks (#5876) The observed bug was that a full restart of a consul datacenter (servers and clients) in conjunction with a restart of a connect-flavored application with bring-your-own-service-registration logic would very frequently cause the envoy sidecar service check to never reflect the aliased service. Over the course of investigation several bugs and unfortunate interactions were corrected: (1) local.CheckState objects were only shallow copied, but the key piece of data that gets read and updated is one of the things not copied (the underlying Check with a Status field). When the stock code was run with the race detector enabled this highly-relevant-to-the-test-scenario field was found to be racy. Changes: a) update the existing Clone method to include the Check field b) copy-on-write when those fields need to change rather than incrementally updating them in place. This made the observed behavior occur slightly less often. (2) If anything about how the runLocal method for node-local alias check logic was ever flawed, there was no fallback option. Those checks are purely edge-triggered and failure to properly notice a single edge transition would leave the alias check incorrect until the next flap of the aliased check. The change was to introduce a fallback timer to act as a control loop to double check the alias check matches the aliased check every minute (borrowing the duration from the non-local alias check logic body). This made the observed behavior eventually go away when it did occur. (3) Originally I thought there were two main actions involved in the data race: A. The act of adding the original check (from disk recovery) and its first health evaluation. B. The act of the HTTP API requests coming in and resetting the local state when re-registering the same services and checks. It took awhile for me to realize that there's a third action at work: C. The goroutines associated with the original check and the later checks. The actual sequence of actions that was causing the bad behavior was that the API actions result in the original check to be removed and re-added _without waiting for the original goroutine to terminate_. This means for brief windows of time during check definition edits there are two goroutines that can be sending updates for the alias check status. In extremely unlikely scenarios the original goroutine sees the aliased check start up in `critical` before being removed but does not get the notification about the nearly immediate update of that check to `passing`. This is interlaced wit the new goroutine coming up, initializing its base case to `passing` from the current state and then listening for new notifications of edge triggers. If the original goroutine "finishes" its update, it then commits one more write into the local state of `critical` and exits leaving the alias check no longer reflecting the underlying check. The correction here is to enforce that the old goroutines must terminate before spawning the new one for alias checks.
2019-05-24 18:36:56 +00:00
}
dataDir := testutil.TempDir(t, "agent") // we manage the data dir
cfg := `
server = false
bootstrap = false
enable_central_service_config = false
data_dir = "` + dataDir + `"
`
Rename NewTestAgentWithFields to StartTestAgent This function now only starts the agent. Using: git grep -l 'StartTestAgent(t, true,' | \ xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'
2020-03-31 20:24:39 +00:00
a := StartTestAgent(t, TestAgent{HCL: cfg, DataDir: dataDir})
agent: fix several data races and bugs related to node-local alias checks (#5876) The observed bug was that a full restart of a consul datacenter (servers and clients) in conjunction with a restart of a connect-flavored application with bring-your-own-service-registration logic would very frequently cause the envoy sidecar service check to never reflect the aliased service. Over the course of investigation several bugs and unfortunate interactions were corrected: (1) local.CheckState objects were only shallow copied, but the key piece of data that gets read and updated is one of the things not copied (the underlying Check with a Status field). When the stock code was run with the race detector enabled this highly-relevant-to-the-test-scenario field was found to be racy. Changes: a) update the existing Clone method to include the Check field b) copy-on-write when those fields need to change rather than incrementally updating them in place. This made the observed behavior occur slightly less often. (2) If anything about how the runLocal method for node-local alias check logic was ever flawed, there was no fallback option. Those checks are purely edge-triggered and failure to properly notice a single edge transition would leave the alias check incorrect until the next flap of the aliased check. The change was to introduce a fallback timer to act as a control loop to double check the alias check matches the aliased check every minute (borrowing the duration from the non-local alias check logic body). This made the observed behavior eventually go away when it did occur. (3) Originally I thought there were two main actions involved in the data race: A. The act of adding the original check (from disk recovery) and its first health evaluation. B. The act of the HTTP API requests coming in and resetting the local state when re-registering the same services and checks. It took awhile for me to realize that there's a third action at work: C. The goroutines associated with the original check and the later checks. The actual sequence of actions that was causing the bad behavior was that the API actions result in the original check to be removed and re-added _without waiting for the original goroutine to terminate_. This means for brief windows of time during check definition edits there are two goroutines that can be sending updates for the alias check status. In extremely unlikely scenarios the original goroutine sees the aliased check start up in `critical` before being removed but does not get the notification about the nearly immediate update of that check to `passing`. This is interlaced wit the new goroutine coming up, initializing its base case to `passing` from the current state and then listening for new notifications of edge triggers. If the original goroutine "finishes" its update, it then commits one more write into the local state of `critical` and exits leaving the alias check no longer reflecting the underlying check. The correction here is to enforce that the old goroutines must terminate before spawning the new one for alias checks.
2019-05-24 18:36:56 +00:00
defer os.RemoveAll(dataDir)
defer a.Shutdown()
testCtx, testCancel := context.WithCancel(context.Background())
defer testCancel()
sdk: add freelist tracking and ephemeral port range skipping to freeport This should cut down on test flakiness. Problems handled: - If you had enough parallel test cases running, the former circular approach to handling the port block could hand out the same port to multiple cases before they each had a chance to bind them, leading to one of the two tests to fail. - The freeport library would allocate out of the ephemeral port range. This has been corrected for Linux (which should cover CI). - The library now waits until a formerly-in-use port is verified to be free before putting it back into circulation.
2019-08-27 21:16:41 +00:00
testHTTPServer, returnPort := launchHTTPCheckServer(t, testCtx)
defer func() {
testHTTPServer.Close()
returnPort()
}()
agent: fix several data races and bugs related to node-local alias checks (#5876) The observed bug was that a full restart of a consul datacenter (servers and clients) in conjunction with a restart of a connect-flavored application with bring-your-own-service-registration logic would very frequently cause the envoy sidecar service check to never reflect the aliased service. Over the course of investigation several bugs and unfortunate interactions were corrected: (1) local.CheckState objects were only shallow copied, but the key piece of data that gets read and updated is one of the things not copied (the underlying Check with a Status field). When the stock code was run with the race detector enabled this highly-relevant-to-the-test-scenario field was found to be racy. Changes: a) update the existing Clone method to include the Check field b) copy-on-write when those fields need to change rather than incrementally updating them in place. This made the observed behavior occur slightly less often. (2) If anything about how the runLocal method for node-local alias check logic was ever flawed, there was no fallback option. Those checks are purely edge-triggered and failure to properly notice a single edge transition would leave the alias check incorrect until the next flap of the aliased check. The change was to introduce a fallback timer to act as a control loop to double check the alias check matches the aliased check every minute (borrowing the duration from the non-local alias check logic body). This made the observed behavior eventually go away when it did occur. (3) Originally I thought there were two main actions involved in the data race: A. The act of adding the original check (from disk recovery) and its first health evaluation. B. The act of the HTTP API requests coming in and resetting the local state when re-registering the same services and checks. It took awhile for me to realize that there's a third action at work: C. The goroutines associated with the original check and the later checks. The actual sequence of actions that was causing the bad behavior was that the API actions result in the original check to be removed and re-added _without waiting for the original goroutine to terminate_. This means for brief windows of time during check definition edits there are two goroutines that can be sending updates for the alias check status. In extremely unlikely scenarios the original goroutine sees the aliased check start up in `critical` before being removed but does not get the notification about the nearly immediate update of that check to `passing`. This is interlaced wit the new goroutine coming up, initializing its base case to `passing` from the current state and then listening for new notifications of edge triggers. If the original goroutine "finishes" its update, it then commits one more write into the local state of `critical` and exits leaving the alias check no longer reflecting the underlying check. The correction here is to enforce that the old goroutines must terminate before spawning the new one for alias checks.
2019-05-24 18:36:56 +00:00
registerServicesAndChecks := func(t *testing.T, a *TestAgent) {
// add one persistent service with a simple check
require.NoError(t, a.AddService(
&structs.NodeService{
ID: "ping",
Service: "ping",
Port: 8000,
},
[]*structs.CheckType{
Enable gofmt simplify Code changes done automatically with 'gofmt -s -w'
2020-06-16 17:19:31 +00:00
{
agent: fix several data races and bugs related to node-local alias checks (#5876) The observed bug was that a full restart of a consul datacenter (servers and clients) in conjunction with a restart of a connect-flavored application with bring-your-own-service-registration logic would very frequently cause the envoy sidecar service check to never reflect the aliased service. Over the course of investigation several bugs and unfortunate interactions were corrected: (1) local.CheckState objects were only shallow copied, but the key piece of data that gets read and updated is one of the things not copied (the underlying Check with a Status field). When the stock code was run with the race detector enabled this highly-relevant-to-the-test-scenario field was found to be racy. Changes: a) update the existing Clone method to include the Check field b) copy-on-write when those fields need to change rather than incrementally updating them in place. This made the observed behavior occur slightly less often. (2) If anything about how the runLocal method for node-local alias check logic was ever flawed, there was no fallback option. Those checks are purely edge-triggered and failure to properly notice a single edge transition would leave the alias check incorrect until the next flap of the aliased check. The change was to introduce a fallback timer to act as a control loop to double check the alias check matches the aliased check every minute (borrowing the duration from the non-local alias check logic body). This made the observed behavior eventually go away when it did occur. (3) Originally I thought there were two main actions involved in the data race: A. The act of adding the original check (from disk recovery) and its first health evaluation. B. The act of the HTTP API requests coming in and resetting the local state when re-registering the same services and checks. It took awhile for me to realize that there's a third action at work: C. The goroutines associated with the original check and the later checks. The actual sequence of actions that was causing the bad behavior was that the API actions result in the original check to be removed and re-added _without waiting for the original goroutine to terminate_. This means for brief windows of time during check definition edits there are two goroutines that can be sending updates for the alias check status. In extremely unlikely scenarios the original goroutine sees the aliased check start up in `critical` before being removed but does not get the notification about the nearly immediate update of that check to `passing`. This is interlaced wit the new goroutine coming up, initializing its base case to `passing` from the current state and then listening for new notifications of edge triggers. If the original goroutine "finishes" its update, it then commits one more write into the local state of `critical` and exits leaving the alias check no longer reflecting the underlying check. The correction here is to enforce that the old goroutines must terminate before spawning the new one for alias checks.
2019-05-24 18:36:56 +00:00
HTTP: testHTTPServer.URL,
Method: "GET",
Interval: 5 * time.Second,
Timeout: 1 * time.Second,
},
},
true, "", ConfigSourceLocal,
))
// add one persistent sidecar service with an alias check in the manner
// of how sidecar_service would add it
require.NoError(t, a.AddService(
&structs.NodeService{
ID: "ping-sidecar-proxy",
Service: "ping-sidecar-proxy",
Port: 9000,
},
[]*structs.CheckType{
Enable gofmt simplify Code changes done automatically with 'gofmt -s -w'
2020-06-16 17:19:31 +00:00
{
agent: fix several data races and bugs related to node-local alias checks (#5876) The observed bug was that a full restart of a consul datacenter (servers and clients) in conjunction with a restart of a connect-flavored application with bring-your-own-service-registration logic would very frequently cause the envoy sidecar service check to never reflect the aliased service. Over the course of investigation several bugs and unfortunate interactions were corrected: (1) local.CheckState objects were only shallow copied, but the key piece of data that gets read and updated is one of the things not copied (the underlying Check with a Status field). When the stock code was run with the race detector enabled this highly-relevant-to-the-test-scenario field was found to be racy. Changes: a) update the existing Clone method to include the Check field b) copy-on-write when those fields need to change rather than incrementally updating them in place. This made the observed behavior occur slightly less often. (2) If anything about how the runLocal method for node-local alias check logic was ever flawed, there was no fallback option. Those checks are purely edge-triggered and failure to properly notice a single edge transition would leave the alias check incorrect until the next flap of the aliased check. The change was to introduce a fallback timer to act as a control loop to double check the alias check matches the aliased check every minute (borrowing the duration from the non-local alias check logic body). This made the observed behavior eventually go away when it did occur. (3) Originally I thought there were two main actions involved in the data race: A. The act of adding the original check (from disk recovery) and its first health evaluation. B. The act of the HTTP API requests coming in and resetting the local state when re-registering the same services and checks. It took awhile for me to realize that there's a third action at work: C. The goroutines associated with the original check and the later checks. The actual sequence of actions that was causing the bad behavior was that the API actions result in the original check to be removed and re-added _without waiting for the original goroutine to terminate_. This means for brief windows of time during check definition edits there are two goroutines that can be sending updates for the alias check status. In extremely unlikely scenarios the original goroutine sees the aliased check start up in `critical` before being removed but does not get the notification about the nearly immediate update of that check to `passing`. This is interlaced wit the new goroutine coming up, initializing its base case to `passing` from the current state and then listening for new notifications of edge triggers. If the original goroutine "finishes" its update, it then commits one more write into the local state of `critical` and exits leaving the alias check no longer reflecting the underlying check. The correction here is to enforce that the old goroutines must terminate before spawning the new one for alias checks.
2019-05-24 18:36:56 +00:00
Name: "Connect Sidecar Aliasing ping",
AliasService: "ping",
},
},
true, "", ConfigSourceLocal,
))
}
retryUntilCheckState := func(t *testing.T, a *TestAgent, checkID string, expectedStatus string) {
t.Helper()
retry.Run(t, func(r *retry.R) {
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
chk := requireCheckExists(t, a, types.CheckID(checkID))
if chk.Status != expectedStatus {
logf(t, a, "check=%q expected status %q but got %q", checkID, expectedStatus, chk.Status)
r.Fatalf("check=%q expected status %q but got %q", checkID, expectedStatus, chk.Status)
agent: fix several data races and bugs related to node-local alias checks (#5876) The observed bug was that a full restart of a consul datacenter (servers and clients) in conjunction with a restart of a connect-flavored application with bring-your-own-service-registration logic would very frequently cause the envoy sidecar service check to never reflect the aliased service. Over the course of investigation several bugs and unfortunate interactions were corrected: (1) local.CheckState objects were only shallow copied, but the key piece of data that gets read and updated is one of the things not copied (the underlying Check with a Status field). When the stock code was run with the race detector enabled this highly-relevant-to-the-test-scenario field was found to be racy. Changes: a) update the existing Clone method to include the Check field b) copy-on-write when those fields need to change rather than incrementally updating them in place. This made the observed behavior occur slightly less often. (2) If anything about how the runLocal method for node-local alias check logic was ever flawed, there was no fallback option. Those checks are purely edge-triggered and failure to properly notice a single edge transition would leave the alias check incorrect until the next flap of the aliased check. The change was to introduce a fallback timer to act as a control loop to double check the alias check matches the aliased check every minute (borrowing the duration from the non-local alias check logic body). This made the observed behavior eventually go away when it did occur. (3) Originally I thought there were two main actions involved in the data race: A. The act of adding the original check (from disk recovery) and its first health evaluation. B. The act of the HTTP API requests coming in and resetting the local state when re-registering the same services and checks. It took awhile for me to realize that there's a third action at work: C. The goroutines associated with the original check and the later checks. The actual sequence of actions that was causing the bad behavior was that the API actions result in the original check to be removed and re-added _without waiting for the original goroutine to terminate_. This means for brief windows of time during check definition edits there are two goroutines that can be sending updates for the alias check status. In extremely unlikely scenarios the original goroutine sees the aliased check start up in `critical` before being removed but does not get the notification about the nearly immediate update of that check to `passing`. This is interlaced wit the new goroutine coming up, initializing its base case to `passing` from the current state and then listening for new notifications of edge triggers. If the original goroutine "finishes" its update, it then commits one more write into the local state of `critical` and exits leaving the alias check no longer reflecting the underlying check. The correction here is to enforce that the old goroutines must terminate before spawning the new one for alias checks.
2019-05-24 18:36:56 +00:00
}
logf(t, a, "check %q has reached desired status %q", checkID, expectedStatus)
})
}
registerServicesAndChecks(t, a)
time.Sleep(1 * time.Second)
retryUntilCheckState(t, a, "service:ping", api.HealthPassing)
retryUntilCheckState(t, a, "service:ping-sidecar-proxy", api.HealthPassing)
logf(t, a, "==== POWERING DOWN ORIGINAL ====")
require.NoError(t, a.Shutdown())
time.Sleep(1 * time.Second)
futureHCL := cfg + `
node_id = "` + string(a.Config.NodeID) + `"
node_name = "` + a.Config.NodeName + `"
`
restartOnce := func(idx int, t *testing.T) {
t.Helper()
// Reload and retain former NodeID and data directory.
Rename NewTestAgentWithFields to StartTestAgent This function now only starts the agent. Using: git grep -l 'StartTestAgent(t, true,' | \ xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'
2020-03-31 20:24:39 +00:00
a2 := StartTestAgent(t, TestAgent{HCL: futureHCL, DataDir: dataDir})
agent: fix several data races and bugs related to node-local alias checks (#5876) The observed bug was that a full restart of a consul datacenter (servers and clients) in conjunction with a restart of a connect-flavored application with bring-your-own-service-registration logic would very frequently cause the envoy sidecar service check to never reflect the aliased service. Over the course of investigation several bugs and unfortunate interactions were corrected: (1) local.CheckState objects were only shallow copied, but the key piece of data that gets read and updated is one of the things not copied (the underlying Check with a Status field). When the stock code was run with the race detector enabled this highly-relevant-to-the-test-scenario field was found to be racy. Changes: a) update the existing Clone method to include the Check field b) copy-on-write when those fields need to change rather than incrementally updating them in place. This made the observed behavior occur slightly less often. (2) If anything about how the runLocal method for node-local alias check logic was ever flawed, there was no fallback option. Those checks are purely edge-triggered and failure to properly notice a single edge transition would leave the alias check incorrect until the next flap of the aliased check. The change was to introduce a fallback timer to act as a control loop to double check the alias check matches the aliased check every minute (borrowing the duration from the non-local alias check logic body). This made the observed behavior eventually go away when it did occur. (3) Originally I thought there were two main actions involved in the data race: A. The act of adding the original check (from disk recovery) and its first health evaluation. B. The act of the HTTP API requests coming in and resetting the local state when re-registering the same services and checks. It took awhile for me to realize that there's a third action at work: C. The goroutines associated with the original check and the later checks. The actual sequence of actions that was causing the bad behavior was that the API actions result in the original check to be removed and re-added _without waiting for the original goroutine to terminate_. This means for brief windows of time during check definition edits there are two goroutines that can be sending updates for the alias check status. In extremely unlikely scenarios the original goroutine sees the aliased check start up in `critical` before being removed but does not get the notification about the nearly immediate update of that check to `passing`. This is interlaced wit the new goroutine coming up, initializing its base case to `passing` from the current state and then listening for new notifications of edge triggers. If the original goroutine "finishes" its update, it then commits one more write into the local state of `critical` and exits leaving the alias check no longer reflecting the underlying check. The correction here is to enforce that the old goroutines must terminate before spawning the new one for alias checks.
2019-05-24 18:36:56 +00:00
defer a2.Shutdown()
a = nil
// reregister during standup; we use an adjustable timing to try and force a race
sleepDur := time.Duration(idx+1) * 500 * time.Millisecond
time.Sleep(sleepDur)
logf(t, a2, "re-registering checks and services after a delay of %v", sleepDur)
for i := 0; i < 20; i++ { // RACE RACE RACE!
registerServicesAndChecks(t, a2)
time.Sleep(50 * time.Millisecond)
}
time.Sleep(1 * time.Second)
retryUntilCheckState(t, a2, "service:ping", api.HealthPassing)
logf(t, a2, "giving the alias check a chance to notice...")
time.Sleep(5 * time.Second)
retryUntilCheckState(t, a2, "service:ping-sidecar-proxy", api.HealthPassing)
}
for i := 0; i < 20; i++ {
name := "restart-" + strconv.Itoa(i)
ok := t.Run(name, func(t *testing.T) {
restartOnce(i, t)
})
require.True(t, ok, name+" failed")
}
}
sdk: add freelist tracking and ephemeral port range skipping to freeport This should cut down on test flakiness. Problems handled: - If you had enough parallel test cases running, the former circular approach to handling the port block could hand out the same port to multiple cases before they each had a chance to bind them, leading to one of the two tests to fail. - The freeport library would allocate out of the ephemeral port range. This has been corrected for Linux (which should cover CI). - The library now waits until a formerly-in-use port is verified to be free before putting it back into circulation.
2019-08-27 21:16:41 +00:00
func launchHTTPCheckServer(t *testing.T, ctx context.Context) (srv *httptest.Server, returnPortsFn func()) {
ports := freeport.MustTake(1)
agent: fix several data races and bugs related to node-local alias checks (#5876) The observed bug was that a full restart of a consul datacenter (servers and clients) in conjunction with a restart of a connect-flavored application with bring-your-own-service-registration logic would very frequently cause the envoy sidecar service check to never reflect the aliased service. Over the course of investigation several bugs and unfortunate interactions were corrected: (1) local.CheckState objects were only shallow copied, but the key piece of data that gets read and updated is one of the things not copied (the underlying Check with a Status field). When the stock code was run with the race detector enabled this highly-relevant-to-the-test-scenario field was found to be racy. Changes: a) update the existing Clone method to include the Check field b) copy-on-write when those fields need to change rather than incrementally updating them in place. This made the observed behavior occur slightly less often. (2) If anything about how the runLocal method for node-local alias check logic was ever flawed, there was no fallback option. Those checks are purely edge-triggered and failure to properly notice a single edge transition would leave the alias check incorrect until the next flap of the aliased check. The change was to introduce a fallback timer to act as a control loop to double check the alias check matches the aliased check every minute (borrowing the duration from the non-local alias check logic body). This made the observed behavior eventually go away when it did occur. (3) Originally I thought there were two main actions involved in the data race: A. The act of adding the original check (from disk recovery) and its first health evaluation. B. The act of the HTTP API requests coming in and resetting the local state when re-registering the same services and checks. It took awhile for me to realize that there's a third action at work: C. The goroutines associated with the original check and the later checks. The actual sequence of actions that was causing the bad behavior was that the API actions result in the original check to be removed and re-added _without waiting for the original goroutine to terminate_. This means for brief windows of time during check definition edits there are two goroutines that can be sending updates for the alias check status. In extremely unlikely scenarios the original goroutine sees the aliased check start up in `critical` before being removed but does not get the notification about the nearly immediate update of that check to `passing`. This is interlaced wit the new goroutine coming up, initializing its base case to `passing` from the current state and then listening for new notifications of edge triggers. If the original goroutine "finishes" its update, it then commits one more write into the local state of `critical` and exits leaving the alias check no longer reflecting the underlying check. The correction here is to enforce that the old goroutines must terminate before spawning the new one for alias checks.
2019-05-24 18:36:56 +00:00
port := ports[0]
addr := net.JoinHostPort("127.0.0.1", strconv.Itoa(port))
var lc net.ListenConfig
listener, err := lc.Listen(ctx, "tcp", addr)
require.NoError(t, err)
handler := http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("OK\n"))
})
sdk: add freelist tracking and ephemeral port range skipping to freeport This should cut down on test flakiness. Problems handled: - If you had enough parallel test cases running, the former circular approach to handling the port block could hand out the same port to multiple cases before they each had a chance to bind them, leading to one of the two tests to fail. - The freeport library would allocate out of the ephemeral port range. This has been corrected for Linux (which should cover CI). - The library now waits until a formerly-in-use port is verified to be free before putting it back into circulation.
2019-08-27 21:16:41 +00:00
srv = &httptest.Server{
agent: fix several data races and bugs related to node-local alias checks (#5876) The observed bug was that a full restart of a consul datacenter (servers and clients) in conjunction with a restart of a connect-flavored application with bring-your-own-service-registration logic would very frequently cause the envoy sidecar service check to never reflect the aliased service. Over the course of investigation several bugs and unfortunate interactions were corrected: (1) local.CheckState objects were only shallow copied, but the key piece of data that gets read and updated is one of the things not copied (the underlying Check with a Status field). When the stock code was run with the race detector enabled this highly-relevant-to-the-test-scenario field was found to be racy. Changes: a) update the existing Clone method to include the Check field b) copy-on-write when those fields need to change rather than incrementally updating them in place. This made the observed behavior occur slightly less often. (2) If anything about how the runLocal method for node-local alias check logic was ever flawed, there was no fallback option. Those checks are purely edge-triggered and failure to properly notice a single edge transition would leave the alias check incorrect until the next flap of the aliased check. The change was to introduce a fallback timer to act as a control loop to double check the alias check matches the aliased check every minute (borrowing the duration from the non-local alias check logic body). This made the observed behavior eventually go away when it did occur. (3) Originally I thought there were two main actions involved in the data race: A. The act of adding the original check (from disk recovery) and its first health evaluation. B. The act of the HTTP API requests coming in and resetting the local state when re-registering the same services and checks. It took awhile for me to realize that there's a third action at work: C. The goroutines associated with the original check and the later checks. The actual sequence of actions that was causing the bad behavior was that the API actions result in the original check to be removed and re-added _without waiting for the original goroutine to terminate_. This means for brief windows of time during check definition edits there are two goroutines that can be sending updates for the alias check status. In extremely unlikely scenarios the original goroutine sees the aliased check start up in `critical` before being removed but does not get the notification about the nearly immediate update of that check to `passing`. This is interlaced wit the new goroutine coming up, initializing its base case to `passing` from the current state and then listening for new notifications of edge triggers. If the original goroutine "finishes" its update, it then commits one more write into the local state of `critical` and exits leaving the alias check no longer reflecting the underlying check. The correction here is to enforce that the old goroutines must terminate before spawning the new one for alias checks.
2019-05-24 18:36:56 +00:00
Listener: listener,
Config: &http.Server{Handler: handler},
}
srv.Start()
sdk: add freelist tracking and ephemeral port range skipping to freeport This should cut down on test flakiness. Problems handled: - If you had enough parallel test cases running, the former circular approach to handling the port block could hand out the same port to multiple cases before they each had a chance to bind them, leading to one of the two tests to fail. - The freeport library would allocate out of the ephemeral port range. This has been corrected for Linux (which should cover CI). - The library now waits until a formerly-in-use port is verified to be free before putting it back into circulation.
2019-08-27 21:16:41 +00:00
return srv, func() { freeport.Return(ports) }
agent: fix several data races and bugs related to node-local alias checks (#5876) The observed bug was that a full restart of a consul datacenter (servers and clients) in conjunction with a restart of a connect-flavored application with bring-your-own-service-registration logic would very frequently cause the envoy sidecar service check to never reflect the aliased service. Over the course of investigation several bugs and unfortunate interactions were corrected: (1) local.CheckState objects were only shallow copied, but the key piece of data that gets read and updated is one of the things not copied (the underlying Check with a Status field). When the stock code was run with the race detector enabled this highly-relevant-to-the-test-scenario field was found to be racy. Changes: a) update the existing Clone method to include the Check field b) copy-on-write when those fields need to change rather than incrementally updating them in place. This made the observed behavior occur slightly less often. (2) If anything about how the runLocal method for node-local alias check logic was ever flawed, there was no fallback option. Those checks are purely edge-triggered and failure to properly notice a single edge transition would leave the alias check incorrect until the next flap of the aliased check. The change was to introduce a fallback timer to act as a control loop to double check the alias check matches the aliased check every minute (borrowing the duration from the non-local alias check logic body). This made the observed behavior eventually go away when it did occur. (3) Originally I thought there were two main actions involved in the data race: A. The act of adding the original check (from disk recovery) and its first health evaluation. B. The act of the HTTP API requests coming in and resetting the local state when re-registering the same services and checks. It took awhile for me to realize that there's a third action at work: C. The goroutines associated with the original check and the later checks. The actual sequence of actions that was causing the bad behavior was that the API actions result in the original check to be removed and re-added _without waiting for the original goroutine to terminate_. This means for brief windows of time during check definition edits there are two goroutines that can be sending updates for the alias check status. In extremely unlikely scenarios the original goroutine sees the aliased check start up in `critical` before being removed but does not get the notification about the nearly immediate update of that check to `passing`. This is interlaced wit the new goroutine coming up, initializing its base case to `passing` from the current state and then listening for new notifications of edge triggers. If the original goroutine "finishes" its update, it then commits one more write into the local state of `critical` and exits leaving the alias check no longer reflecting the underlying check. The correction here is to enforce that the old goroutines must terminate before spawning the new one for alias checks.
2019-05-24 18:36:56 +00:00
}
agent: run alias checks
2018-06-30 13:38:56 +00:00
func TestAgent_AddCheck_Alias(t *testing.T) {
t.Parallel()
agent: use the correct ACL token for alias checks
2018-07-12 17:17:53 +00:00
require := require.New(t)
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
agent: run alias checks
2018-06-30 13:38:56 +00:00
defer a.Shutdown()
health := &structs.HealthCheck{
Node: "foo",
CheckID: "aliashealth",
Name: "Alias health check",
Status: api.HealthCritical,
}
chk := &structs.CheckType{
AliasService: "foo",
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
err := a.AddCheck(health, chk, false, "", ConfigSourceLocal)
agent: use the correct ACL token for alias checks
2018-07-12 17:17:53 +00:00
require.NoError(err)
agent: run alias checks
2018-06-30 13:38:56 +00:00
// Ensure we have a check mapping
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
sChk := requireCheckExists(t, a, "aliashealth")
agent: use the correct ACL token for alias checks
2018-07-12 17:17:53 +00:00
require.Equal(api.HealthCritical, sChk.Status)
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
chkImpl, ok := a.checkAliases[structs.NewCheckID("aliashealth", nil)]
agent: use the correct ACL token for alias checks
2018-07-12 17:17:53 +00:00
require.True(ok, "missing aliashealth check")
require.Equal("", chkImpl.RPCReq.Token)
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
cs := a.State.CheckState(structs.NewCheckID("aliashealth", nil))
agent: use the correct ACL token for alias checks
2018-07-12 17:17:53 +00:00
require.NotNil(cs)
require.Equal("", cs.Token)
}
func TestAgent_AddCheck_Alias_setToken(t *testing.T) {
t.Parallel()
require := require.New(t)
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
agent: use the correct ACL token for alias checks
2018-07-12 17:17:53 +00:00
defer a.Shutdown()
health := &structs.HealthCheck{
Node: "foo",
CheckID: "aliashealth",
Name: "Alias health check",
Status: api.HealthCritical,
}
chk := &structs.CheckType{
AliasService: "foo",
agent: run alias checks
2018-06-30 13:38:56 +00:00
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
err := a.AddCheck(health, chk, false, "foo", ConfigSourceLocal)
agent: use the correct ACL token for alias checks
2018-07-12 17:17:53 +00:00
require.NoError(err)
agent: run alias checks
2018-06-30 13:38:56 +00:00
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
cs := a.State.CheckState(structs.NewCheckID("aliashealth", nil))
agent: use the correct ACL token for alias checks
2018-07-12 17:17:53 +00:00
require.NotNil(cs)
require.Equal("foo", cs.Token)
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
chkImpl, ok := a.checkAliases[structs.NewCheckID("aliashealth", nil)]
agent: use the correct ACL token for alias checks
2018-07-12 17:17:53 +00:00
require.True(ok, "missing aliashealth check")
require.Equal("foo", chkImpl.RPCReq.Token)
}
func TestAgent_AddCheck_Alias_userToken(t *testing.T) {
t.Parallel()
require := require.New(t)
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
agent: use the correct ACL token for alias checks
2018-07-12 17:17:53 +00:00
acl_token = "hello"
`)
defer a.Shutdown()
health := &structs.HealthCheck{
Node: "foo",
CheckID: "aliashealth",
Name: "Alias health check",
Status: api.HealthCritical,
}
chk := &structs.CheckType{
AliasService: "foo",
agent: run alias checks
2018-06-30 13:38:56 +00:00
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
err := a.AddCheck(health, chk, false, "", ConfigSourceLocal)
agent: use the correct ACL token for alias checks
2018-07-12 17:17:53 +00:00
require.NoError(err)
agent: run alias checks
2018-06-30 13:38:56 +00:00
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
cs := a.State.CheckState(structs.NewCheckID("aliashealth", nil))
agent: use the correct ACL token for alias checks
2018-07-12 17:17:53 +00:00
require.NotNil(cs)
require.Equal("", cs.Token) // State token should still be empty
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
chkImpl, ok := a.checkAliases[structs.NewCheckID("aliashealth", nil)]
agent: use the correct ACL token for alias checks
2018-07-12 17:17:53 +00:00
require.True(ok, "missing aliashealth check")
require.Equal("hello", chkImpl.RPCReq.Token) // Check should use the token
}
func TestAgent_AddCheck_Alias_userAndSetToken(t *testing.T) {
t.Parallel()
require := require.New(t)
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
agent: use the correct ACL token for alias checks
2018-07-12 17:17:53 +00:00
acl_token = "hello"
`)
defer a.Shutdown()
health := &structs.HealthCheck{
Node: "foo",
CheckID: "aliashealth",
Name: "Alias health check",
Status: api.HealthCritical,
agent: run alias checks
2018-06-30 13:38:56 +00:00
}
agent: use the correct ACL token for alias checks
2018-07-12 17:17:53 +00:00
chk := &structs.CheckType{
AliasService: "foo",
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
err := a.AddCheck(health, chk, false, "goodbye", ConfigSourceLocal)
agent: use the correct ACL token for alias checks
2018-07-12 17:17:53 +00:00
require.NoError(err)
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
cs := a.State.CheckState(structs.NewCheckID("aliashealth", nil))
agent: use the correct ACL token for alias checks
2018-07-12 17:17:53 +00:00
require.NotNil(cs)
require.Equal("goodbye", cs.Token)
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
chkImpl, ok := a.checkAliases[structs.NewCheckID("aliashealth", nil)]
agent: use the correct ACL token for alias checks
2018-07-12 17:17:53 +00:00
require.True(ok, "missing aliashealth check")
require.Equal("goodbye", chkImpl.RPCReq.Token)
agent: run alias checks
2018-06-30 13:38:56 +00:00
}
Adds new config to make script checks opt-in, updates documentation. (#3284)
2017-07-17 18:20:35 +00:00
func TestAgent_RemoveCheck(t *testing.T) {
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
enable_script_checks = true
`)
Adds new config to make script checks opt-in, updates documentation. (#3284)
2017-07-17 18:20:35 +00:00
defer a.Shutdown()
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
// Remove check that doesn't exist
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.RemoveCheck(structs.NewCheckID("mem", nil), false); err != nil {
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
t.Fatalf("err: %v", err)
}
Add tests to remove service/check without an ID
2015-01-27 09:10:56 +00:00
// Remove without an ID
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.RemoveCheck(structs.NewCheckID("", nil), false); err == nil {
Add tests to remove service/check without an ID
2015-01-27 09:10:56 +00:00
t.Fatalf("should have errored")
}
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
health := &structs.HealthCheck{
Node: "foo",
CheckID: "mem",
Name: "memory util",
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
Status: api.HealthCritical,
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
}
agent: move structs into consul/structs pkg * CheckDefinition * ServiceDefinition * CheckType
2017-06-15 16:46:06 +00:00
chk := &structs.CheckType{
Remove the script field from checks in favor of args
2018-05-08 22:31:53 +00:00
ScriptArgs: []string{"exit", "0"},
Interval: 15 * time.Second,
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
err := a.AddCheck(health, chk, false, "", ConfigSourceLocal)
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
if err != nil {
t.Fatalf("err: %v", err)
}
// Remove check
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.RemoveCheck(structs.NewCheckID("mem", nil), false); err != nil {
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
t.Fatalf("err: %v", err)
}
// Ensure we have a check mapping
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireCheckMissing(t, a, "mem")
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
// Ensure a TTL is setup
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireCheckMissingMap(t, a.checkMonitors, "mem")
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
}
Adds enable_agent_tls_for_checks configuration option which allows (#3661) HTTP health checks for services requiring 2-way TLS to be checked using the agent's credentials.
2017-11-08 02:22:09 +00:00
func TestAgent_HTTPCheck_TLSSkipVerify(t *testing.T) {
t.Parallel()
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "GOOD")
})
server := httptest.NewTLSServer(handler)
defer server.Close()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
Adds enable_agent_tls_for_checks configuration option which allows (#3661) HTTP health checks for services requiring 2-way TLS to be checked using the agent's credentials.
2017-11-08 02:22:09 +00:00
defer a.Shutdown()
health := &structs.HealthCheck{
Node: "foo",
CheckID: "tls",
Name: "tls check",
Status: api.HealthCritical,
}
chk := &structs.CheckType{
HTTP: server.URL,
Interval: 20 * time.Millisecond,
TLSSkipVerify: true,
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
err := a.AddCheck(health, chk, false, "", ConfigSourceLocal)
Adds enable_agent_tls_for_checks configuration option which allows (#3661) HTTP health checks for services requiring 2-way TLS to be checked using the agent's credentials.
2017-11-08 02:22:09 +00:00
if err != nil {
t.Fatalf("err: %v", err)
}
retry.Run(t, func(r *retry.R) {
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
status := getCheck(a, "tls")
Adds enable_agent_tls_for_checks configuration option which allows (#3661) HTTP health checks for services requiring 2-way TLS to be checked using the agent's credentials.
2017-11-08 02:22:09 +00:00
if status.Status != api.HealthPassing {
r.Fatalf("bad: %v", status.Status)
}
if !strings.Contains(status.Output, "GOOD") {
r.Fatalf("bad: %v", status.Output)
}
})
}
func TestAgent_HTTPCheck_EnableAgentTLSForChecks(t *testing.T) {
t.Parallel()
run := func(t *testing.T, ca string) {
Rename NewTestAgentWithFields to StartTestAgent This function now only starts the agent. Using: git grep -l 'StartTestAgent(t, true,' | \ xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'
2020-03-31 20:24:39 +00:00
a := StartTestAgent(t, TestAgent{
Adds enable_agent_tls_for_checks configuration option which allows (#3661) HTTP health checks for services requiring 2-way TLS to be checked using the agent's credentials.
2017-11-08 02:22:09 +00:00
UseTLS: true,
HCL: `
enable_agent_tls_for_checks = true
verify_incoming = true
server_name = "consul.test"
key_file = "../test/client_certs/server.key"
cert_file = "../test/client_certs/server.crt"
` + ca,
test: ensure all TestAgent constructions use a constructor (#6443) ensure all TestAgent constructions use a constructor to get start retries + test logs going to the right place Fixes #6435
2019-09-05 17:24:36 +00:00
})
Adds enable_agent_tls_for_checks configuration option which allows (#3661) HTTP health checks for services requiring 2-way TLS to be checked using the agent's credentials.
2017-11-08 02:22:09 +00:00
defer a.Shutdown()
health := &structs.HealthCheck{
Node: "foo",
CheckID: "tls",
Name: "tls check",
Status: api.HealthCritical,
}
url := fmt.Sprintf("https://%s/v1/agent/self", a.srv.ln.Addr().String())
chk := &structs.CheckType{
HTTP: url,
Interval: 20 * time.Millisecond,
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
err := a.AddCheck(health, chk, false, "", ConfigSourceLocal)
Adds enable_agent_tls_for_checks configuration option which allows (#3661) HTTP health checks for services requiring 2-way TLS to be checked using the agent's credentials.
2017-11-08 02:22:09 +00:00
if err != nil {
t.Fatalf("err: %v", err)
}
retry.Run(t, func(r *retry.R) {
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
status := getCheck(a, "tls")
Adds enable_agent_tls_for_checks configuration option which allows (#3661) HTTP health checks for services requiring 2-way TLS to be checked using the agent's credentials.
2017-11-08 02:22:09 +00:00
if status.Status != api.HealthPassing {
r.Fatalf("bad: %v", status.Status)
}
if !strings.Contains(status.Output, "200 OK") {
r.Fatalf("bad: %v", status.Output)
}
})
}
// We need to test both methods of passing the CA info to ensure that
// we propagate all the fields correctly. All the other fields are
// covered by the HCL in the test run function.
tests := []struct {
desc string
config string
}{
{"ca_file", `ca_file = "../test/client_certs/rootca.crt"`},
{"ca_path", `ca_path = "../test/client_certs/path"`},
}
for _, tt := range tests {
t.Run(tt.desc, func(t *testing.T) {
run(t, tt.config)
})
}
}
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
func TestAgent_updateTTLCheck(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
Support for maximum size for Output of checks (#5233) * Support for maximum size for Output of checks This PR allows users to limit the size of output produced by checks at the agent and check level. When set at the agent level, it will limit the output for all checks monitored by the agent. When set at the check level, it can override the agent max for a specific check but only if it is lower than the agent max. Default value is 4k, and input must be at least 1.
2019-06-26 15:43:25 +00:00
checkBufSize := 100
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
health := &structs.HealthCheck{
Node: "foo",
CheckID: "mem",
Name: "memory util",
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
Status: api.HealthCritical,
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
}
agent: move structs into consul/structs pkg * CheckDefinition * ServiceDefinition * CheckType
2017-06-15 16:46:06 +00:00
chk := &structs.CheckType{
Support for maximum size for Output of checks (#5233) * Support for maximum size for Output of checks This PR allows users to limit the size of output produced by checks at the agent and check level. When set at the agent level, it will limit the output for all checks monitored by the agent. When set at the check level, it can override the agent max for a specific check but only if it is lower than the agent max. Default value is 4k, and input must be at least 1.
2019-06-26 15:43:25 +00:00
TTL: 15 * time.Second,
OutputMaxSize: checkBufSize,
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
}
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
// Add check and update it.
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
err := a.AddCheck(health, chk, false, "", ConfigSourceLocal)
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
if err != nil {
t.Fatalf("err: %v", err)
}
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.updateTTLCheck(structs.NewCheckID("mem", nil), api.HealthPassing, "foo"); err != nil {
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
t.Fatalf("err: %v", err)
}
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
// Ensure we have a check mapping.
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
status := getCheck(a, "mem")
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
if status.Status != api.HealthPassing {
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
t.Fatalf("bad: %v", status)
}
Store check output in dedicated field. Fixes #59.
2014-04-21 23:20:22 +00:00
if status.Output != "foo" {
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
t.Fatalf("bad: %v", status)
}
Support for maximum size for Output of checks (#5233) * Support for maximum size for Output of checks This PR allows users to limit the size of output produced by checks at the agent and check level. When set at the agent level, it will limit the output for all checks monitored by the agent. When set at the check level, it can override the agent max for a specific check but only if it is lower than the agent max. Default value is 4k, and input must be at least 1.
2019-06-26 15:43:25 +00:00
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.updateTTLCheck(structs.NewCheckID("mem", nil), api.HealthCritical, strings.Repeat("--bad-- ", 5*checkBufSize)); err != nil {
Support for maximum size for Output of checks (#5233) * Support for maximum size for Output of checks This PR allows users to limit the size of output produced by checks at the agent and check level. When set at the agent level, it will limit the output for all checks monitored by the agent. When set at the check level, it can override the agent max for a specific check but only if it is lower than the agent max. Default value is 4k, and input must be at least 1.
2019-06-26 15:43:25 +00:00
t.Fatalf("err: %v", err)
}
// Ensure we have a check mapping.
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
status = getCheck(a, "mem")
Support for maximum size for Output of checks (#5233) * Support for maximum size for Output of checks This PR allows users to limit the size of output produced by checks at the agent and check level. When set at the agent level, it will limit the output for all checks monitored by the agent. When set at the check level, it can override the agent max for a specific check but only if it is lower than the agent max. Default value is 4k, and input must be at least 1.
2019-06-26 15:43:25 +00:00
if status.Status != api.HealthCritical {
t.Fatalf("bad: %v", status)
}
if len(status.Output) > checkBufSize*2 {
t.Fatalf("bad: %v", len(status.Output))
}
agent: Adding methods to register services and checks
2014-01-30 21:39:02 +00:00
}
agent: auto-register the consul service on server nodes
2014-10-14 22:05:41 +00:00
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
func TestAgent_PersistService(t *testing.T) {
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
t.Run("normal", func(t *testing.T) {
t.Parallel()
testAgent_PersistService(t, "")
})
t.Run("service manager", func(t *testing.T) {
t.Parallel()
testAgent_PersistService(t, "enable_central_service_config = true")
})
}
func testAgent_PersistService(t *testing.T, extraHCL string) {
t.Helper()
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
dataDir := testutil.TempDir(t, "agent") // we manage the data dir
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
defer os.RemoveAll(dataDir)
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
cfg := `
server = false
bootstrap = false
data_dir = "` + dataDir + `"
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
` + extraHCL
Rename NewTestAgentWithFields to StartTestAgent This function now only starts the agent. Using: git grep -l 'StartTestAgent(t, true,' | \ xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'
2020-03-31 20:24:39 +00:00
a := StartTestAgent(t, TestAgent{HCL: cfg, DataDir: dataDir})
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
svc := &structs.NodeService{
ID: "redis",
Service: "redis",
Tags: []string{"foo"},
Port: 8000,
}
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
file := filepath.Join(a.Config.DataDir, servicesDir, stringHash(svc.ID))
agent: prefer config over persisted services/checks (#497)
2014-11-25 03:24:32 +00:00
// Check is not persisted unless requested
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddService(svc, nil, false, "", ConfigSourceLocal); err != nil {
agent: prefer config over persisted services/checks (#497)
2014-11-25 03:24:32 +00:00
t.Fatalf("err: %v", err)
}
if _, err := os.Stat(file); err == nil {
t.Fatalf("should not persist")
}
// Persists to file if requested
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddService(svc, nil, true, "mytoken", ConfigSourceLocal); err != nil {
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
t.Fatalf("err: %v", err)
}
if _, err := os.Stat(file); err != nil {
t.Fatalf("err: %s", err)
}
agent: restore tokens for services and checks in config
2015-04-28 19:44:46 +00:00
expected, err := json.Marshal(persistedService{
agent: use an additional parameter for passing tokens
2015-05-05 00:36:17 +00:00
Token: "mytoken",
agent: test coverage loading service/check tokens from persisted files
2015-04-28 05:46:01 +00:00
Service: svc,
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
Source: "local",
agent: test coverage loading service/check tokens from persisted files
2015-04-28 05:46:01 +00:00
})
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
if err != nil {
t.Fatalf("err: %s", err)
}
content, err := ioutil.ReadFile(file)
if err != nil {
t.Fatalf("err: %s", err)
}
if !bytes.Equal(expected, content) {
t.Fatalf("bad: %s", string(content))
}
agent: allow persisted services to be updated on disk
2015-05-06 05:08:03 +00:00
// Updates service definition on disk
svc.Port = 8001
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddService(svc, nil, true, "mytoken", ConfigSourceLocal); err != nil {
agent: allow persisted services to be updated on disk
2015-05-06 05:08:03 +00:00
t.Fatalf("err: %v", err)
}
expected, err = json.Marshal(persistedService{
Token: "mytoken",
Service: svc,
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
Source: "local",
agent: allow persisted services to be updated on disk
2015-05-06 05:08:03 +00:00
})
if err != nil {
t.Fatalf("err: %s", err)
}
content, err = ioutil.ReadFile(file)
if err != nil {
t.Fatalf("err: %s", err)
}
if !bytes.Equal(expected, content) {
t.Fatalf("bad: %s", string(content))
}
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
a.Shutdown()
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
// Should load it back during later start
Rename NewTestAgentWithFields to StartTestAgent This function now only starts the agent. Using: git grep -l 'StartTestAgent(t, true,' | \ xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'
2020-03-31 20:24:39 +00:00
a2 := StartTestAgent(t, TestAgent{HCL: cfg, DataDir: dataDir})
test: start secondary agents also via TestAgent
2017-05-31 08:56:19 +00:00
defer a2.Shutdown()
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
restored := a2.State.ServiceState(structs.NewServiceID(svc.ID, nil))
local state: tests compile
2017-08-28 12:17:13 +00:00
if restored == nil {
t.Fatalf("service %q missing", svc.ID)
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
}
local state: tests compile
2017-08-28 12:17:13 +00:00
if got, want := restored.Token, "mytoken"; got != want {
t.Fatalf("got token %q want %q", got, want)
agent: test coverage loading service/check tokens from persisted files
2015-04-28 05:46:01 +00:00
}
local state: fix failing tests
2017-08-28 12:17:14 +00:00
if got, want := restored.Service.Port, 8001; got != want {
local state: tests compile
2017-08-28 12:17:13 +00:00
t.Fatalf("got port %d want %d", got, want)
agent: allow persisted services to be updated on disk
2015-05-06 05:08:03 +00:00
}
agent: allow config reload to modify checks/services persistence This change consolidates loading services and checks from both config and persisted state into methods on the agent. As part of this, we introduce optional persistence when calling RemoveCheck/RemoveService. Fixes a bug where config reloads would kill persisted services/checks. Also fixes an edge case: 1. A service or check is registered via the HTTP API 2. A new service or check definition with the same ID is added to config 3. Config is reloaded The desired behavior (which this implements) is: 1. All services and checks deregistered in memory 2. All services and checks in config are registered first 3. All persisted checks are restored using the same logic as the agent start sequence, which prioritizes config over persisted, and removes any persistence files if new config counterparts are present.
2014-11-26 07:58:02 +00:00
}
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
agent: restore tokens for services and checks in config
2015-04-28 19:44:46 +00:00
func TestAgent_persistedService_compat(t *testing.T) {
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
t.Run("normal", func(t *testing.T) {
t.Parallel()
testAgent_persistedService_compat(t, "")
})
t.Run("service manager", func(t *testing.T) {
t.Parallel()
testAgent_persistedService_compat(t, "enable_central_service_config = true")
})
}
func testAgent_persistedService_compat(t *testing.T, extraHCL string) {
t.Helper()
agent: backwards compat for persisted services from pre-0.5.1
2015-04-28 19:18:41 +00:00
// Tests backwards compatibility of persisted services from pre-0.5.1
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, extraHCL)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: backwards compat for persisted services from pre-0.5.1
2015-04-28 19:18:41 +00:00
svc := &structs.NodeService{
Add support for dual stack IPv4/IPv6 network (#6640) * Use consts for well known tagged adress keys * Add ipv4 and ipv6 tagged addresses for node lan and wan * Add ipv4 and ipv6 tagged addresses for service lan and wan * Use IPv4 and IPv6 address in DNS
2020-01-17 14:54:17 +00:00
ID: "redis",
Service: "redis",
Tags: []string{"foo"},
Port: 8000,
TaggedAddresses: map[string]structs.ServiceAddress{},
Weights: &structs.Weights{Passing: 1, Warning: 1},
EnterpriseMeta: *structs.DefaultEnterpriseMeta(),
agent: backwards compat for persisted services from pre-0.5.1
2015-04-28 19:18:41 +00:00
}
// Encode the NodeService directly. This is what previous versions
// would serialize to the file (without the wrapper)
encoded, err := json.Marshal(svc)
if err != nil {
t.Fatalf("err: %s", err)
}
// Write the content to the file
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
file := filepath.Join(a.Config.DataDir, servicesDir, stringHash(svc.ID))
agent: backwards compat for persisted services from pre-0.5.1
2015-04-28 19:18:41 +00:00
if err := os.MkdirAll(filepath.Dir(file), 0700); err != nil {
t.Fatalf("err: %s", err)
}
if err := ioutil.WriteFile(file, encoded, 0600); err != nil {
t.Fatalf("err: %s", err)
}
// Load the services
agent: configuration reload preserves check's statuses for services (#7345) This fixes issue #7318 Between versions 1.5.2 and 1.5.3, a regression has been introduced regarding health of services. A patch #6144 had been issued for HealthChecks of nodes, but not for healthchecks of services. What happened when a reload was: 1. save all healthcheck statuses 2. cleanup everything 3. add new services with healthchecks In step 3, the state of healthchecks was taken into account locally, so at step 3, but since we cleaned up at step 2, state was lost. This PR introduces the snap parameter, so step 3 can use information from step 1
2020-03-09 11:59:41 +00:00
if err := a.loadServices(a.Config, nil); err != nil {
agent: backwards compat for persisted services from pre-0.5.1
2015-04-28 19:18:41 +00:00
t.Fatalf("err: %s", err)
}
// Ensure the service was restored
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
result := requireServiceExists(t, a, "redis")
agent: add default weights to service in local state to prevent AE churn (#5126) * Add default weights when adding a service with no weights to local state to prevent constant AE re-sync. This fix was contributed by @42wim in https://github.com/hashicorp/consul/pull/5096 but was merged against the wrong base. This adds it to master and adds a test to cover the behaviour. * Fix tests that broke due to comparing internal state which now has default weights
2019-01-08 10:13:49 +00:00
require.Equal(t, svc, result)
agent: backwards compat for persisted services from pre-0.5.1
2015-04-28 19:18:41 +00:00
}
agent: allow config reload to modify checks/services persistence This change consolidates loading services and checks from both config and persisted state into methods on the agent. As part of this, we introduce optional persistence when calling RemoveCheck/RemoveService. Fixes a bug where config reloads would kill persisted services/checks. Also fixes an edge case: 1. A service or check is registered via the HTTP API 2. A new service or check definition with the same ID is added to config 3. Config is reloaded The desired behavior (which this implements) is: 1. All services and checks deregistered in memory 2. All services and checks in config are registered first 3. All persisted checks are restored using the same logic as the agent start sequence, which prioritizes config over persisted, and removes any persistence files if new config counterparts are present.
2014-11-26 07:58:02 +00:00
func TestAgent_PurgeService(t *testing.T) {
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
t.Run("normal", func(t *testing.T) {
t.Parallel()
testAgent_PurgeService(t, "")
})
t.Run("service manager", func(t *testing.T) {
t.Parallel()
testAgent_PurgeService(t, "enable_central_service_config = true")
})
}
func testAgent_PurgeService(t *testing.T, extraHCL string) {
t.Helper()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, extraHCL)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: allow config reload to modify checks/services persistence This change consolidates loading services and checks from both config and persisted state into methods on the agent. As part of this, we introduce optional persistence when calling RemoveCheck/RemoveService. Fixes a bug where config reloads would kill persisted services/checks. Also fixes an edge case: 1. A service or check is registered via the HTTP API 2. A new service or check definition with the same ID is added to config 3. Config is reloaded The desired behavior (which this implements) is: 1. All services and checks deregistered in memory 2. All services and checks in config are registered first 3. All persisted checks are restored using the same logic as the agent start sequence, which prioritizes config over persisted, and removes any persistence files if new config counterparts are present.
2014-11-26 07:58:02 +00:00
svc := &structs.NodeService{
ID: "redis",
Service: "redis",
Tags: []string{"foo"},
Port: 8000,
}
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
file := filepath.Join(a.Config.DataDir, servicesDir, stringHash(svc.ID))
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddService(svc, nil, true, "", ConfigSourceLocal); err != nil {
agent: allow config reload to modify checks/services persistence This change consolidates loading services and checks from both config and persisted state into methods on the agent. As part of this, we introduce optional persistence when calling RemoveCheck/RemoveService. Fixes a bug where config reloads would kill persisted services/checks. Also fixes an edge case: 1. A service or check is registered via the HTTP API 2. A new service or check definition with the same ID is added to config 3. Config is reloaded The desired behavior (which this implements) is: 1. All services and checks deregistered in memory 2. All services and checks in config are registered first 3. All persisted checks are restored using the same logic as the agent start sequence, which prioritizes config over persisted, and removes any persistence files if new config counterparts are present.
2014-11-26 07:58:02 +00:00
t.Fatalf("err: %v", err)
}
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
// Exists
if _, err := os.Stat(file); err != nil {
t.Fatalf("err: %s", err)
}
agent: allow config reload to modify checks/services persistence This change consolidates loading services and checks from both config and persisted state into methods on the agent. As part of this, we introduce optional persistence when calling RemoveCheck/RemoveService. Fixes a bug where config reloads would kill persisted services/checks. Also fixes an edge case: 1. A service or check is registered via the HTTP API 2. A new service or check definition with the same ID is added to config 3. Config is reloaded The desired behavior (which this implements) is: 1. All services and checks deregistered in memory 2. All services and checks in config are registered first 3. All persisted checks are restored using the same logic as the agent start sequence, which prioritizes config over persisted, and removes any persistence files if new config counterparts are present.
2014-11-26 07:58:02 +00:00
// Not removed
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.removeService(structs.NewServiceID(svc.ID, nil), false); err != nil {
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
t.Fatalf("err: %s", err)
}
agent: allow config reload to modify checks/services persistence This change consolidates loading services and checks from both config and persisted state into methods on the agent. As part of this, we introduce optional persistence when calling RemoveCheck/RemoveService. Fixes a bug where config reloads would kill persisted services/checks. Also fixes an edge case: 1. A service or check is registered via the HTTP API 2. A new service or check definition with the same ID is added to config 3. Config is reloaded The desired behavior (which this implements) is: 1. All services and checks deregistered in memory 2. All services and checks in config are registered first 3. All persisted checks are restored using the same logic as the agent start sequence, which prioritizes config over persisted, and removes any persistence files if new config counterparts are present.
2014-11-26 07:58:02 +00:00
if _, err := os.Stat(file); err != nil {
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
t.Fatalf("err: %s", err)
}
agent: allow config reload to modify checks/services persistence This change consolidates loading services and checks from both config and persisted state into methods on the agent. As part of this, we introduce optional persistence when calling RemoveCheck/RemoveService. Fixes a bug where config reloads would kill persisted services/checks. Also fixes an edge case: 1. A service or check is registered via the HTTP API 2. A new service or check definition with the same ID is added to config 3. Config is reloaded The desired behavior (which this implements) is: 1. All services and checks deregistered in memory 2. All services and checks in config are registered first 3. All persisted checks are restored using the same logic as the agent start sequence, which prioritizes config over persisted, and removes any persistence files if new config counterparts are present.
2014-11-26 07:58:02 +00:00
Improve logging when deregistering a nonexistent service (#2492) Log a warning instead of a success message when attempting to deregister a nonexistent service. In Consul 0.8 this can be changed to giving an error outright, but for now we can keep the idempotent delete behavior.
2016-11-09 21:56:54 +00:00
// Re-add the service
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddService(svc, nil, true, "", ConfigSourceLocal); err != nil {
Improve logging when deregistering a nonexistent service (#2492) Log a warning instead of a success message when attempting to deregister a nonexistent service. In Consul 0.8 this can be changed to giving an error outright, but for now we can keep the idempotent delete behavior.
2016-11-09 21:56:54 +00:00
t.Fatalf("err: %v", err)
}
agent: allow config reload to modify checks/services persistence This change consolidates loading services and checks from both config and persisted state into methods on the agent. As part of this, we introduce optional persistence when calling RemoveCheck/RemoveService. Fixes a bug where config reloads would kill persisted services/checks. Also fixes an edge case: 1. A service or check is registered via the HTTP API 2. A new service or check definition with the same ID is added to config 3. Config is reloaded The desired behavior (which this implements) is: 1. All services and checks deregistered in memory 2. All services and checks in config are registered first 3. All persisted checks are restored using the same logic as the agent start sequence, which prioritizes config over persisted, and removes any persistence files if new config counterparts are present.
2014-11-26 07:58:02 +00:00
// Removed
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.removeService(structs.NewServiceID(svc.ID, nil), true); err != nil {
agent: allow config reload to modify checks/services persistence This change consolidates loading services and checks from both config and persisted state into methods on the agent. As part of this, we introduce optional persistence when calling RemoveCheck/RemoveService. Fixes a bug where config reloads would kill persisted services/checks. Also fixes an edge case: 1. A service or check is registered via the HTTP API 2. A new service or check definition with the same ID is added to config 3. Config is reloaded The desired behavior (which this implements) is: 1. All services and checks deregistered in memory 2. All services and checks in config are registered first 3. All persisted checks are restored using the same logic as the agent start sequence, which prioritizes config over persisted, and removes any persistence files if new config counterparts are present.
2014-11-26 07:58:02 +00:00
t.Fatalf("err: %s", err)
}
if _, err := os.Stat(file); !os.IsNotExist(err) {
t.Fatalf("bad: %#v", err)
}
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
}
agent: prefer config over persisted services/checks (#497)
2014-11-25 03:24:32 +00:00
func TestAgent_PurgeServiceOnDuplicate(t *testing.T) {
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
t.Run("normal", func(t *testing.T) {
t.Parallel()
testAgent_PurgeServiceOnDuplicate(t, "")
})
t.Run("service manager", func(t *testing.T) {
t.Parallel()
testAgent_PurgeServiceOnDuplicate(t, "enable_central_service_config = true")
})
}
func testAgent_PurgeServiceOnDuplicate(t *testing.T, extraHCL string) {
t.Helper()
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
dataDir := testutil.TempDir(t, "agent") // we manage the data dir
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
defer os.RemoveAll(dataDir)
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
cfg := `
data_dir = "` + dataDir + `"
server = false
bootstrap = false
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
` + extraHCL
Rename NewTestAgentWithFields to StartTestAgent This function now only starts the agent. Using: git grep -l 'StartTestAgent(t, true,' | \ xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'
2020-03-31 20:24:39 +00:00
a := StartTestAgent(t, TestAgent{HCL: cfg, DataDir: dataDir})
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: prefer config over persisted services/checks (#497)
2014-11-25 03:24:32 +00:00
svc1 := &structs.NodeService{
ID: "redis",
Service: "redis",
Tags: []string{"foo"},
Port: 8000,
}
// First persist the service
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
require.NoError(t, a.AddService(svc1, nil, true, "", ConfigSourceLocal))
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
a.Shutdown()
agent: prefer config over persisted services/checks (#497)
2014-11-25 03:24:32 +00:00
// Try bringing the agent back up with the service already
// existing in the config
Rename NewTestAgentWithFields to StartTestAgent This function now only starts the agent. Using: git grep -l 'StartTestAgent(t, true,' | \ xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'
2020-03-31 20:24:39 +00:00
a2 := StartTestAgent(t, TestAgent{Name: "Agent2", HCL: cfg + `
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
service = {
id = "redis"
name = "redis"
tags = ["bar"]
port = 9000
}
test: ensure all TestAgent constructions use a constructor (#6443) ensure all TestAgent constructions use a constructor to get start retries + test logs going to the right place Fixes #6435
2019-09-05 17:24:36 +00:00
`, DataDir: dataDir})
test: start secondary agents also via TestAgent
2017-05-31 08:56:19 +00:00
defer a2.Shutdown()
agent: prefer config over persisted services/checks (#497)
2014-11-25 03:24:32 +00:00
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
sid := svc1.CompoundServiceID()
file := filepath.Join(a.Config.DataDir, servicesDir, sid.StringHash())
_, err := os.Stat(file)
require.Error(t, err, "should have removed persisted service")
result := requireServiceExists(t, a, "redis")
require.NotEqual(t, []string{"bar"}, result.Tags)
require.NotEqual(t, 9000, result.Port)
agent: prefer config over persisted services/checks (#497)
2014-11-25 03:24:32 +00:00
}
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
func TestAgent_PersistCheck(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
dataDir := testutil.TempDir(t, "agent") // we manage the data dir
cfg := `
data_dir = "` + dataDir + `"
server = false
bootstrap = false
enable_script_checks = true
`
Rename NewTestAgentWithFields to StartTestAgent This function now only starts the agent. Using: git grep -l 'StartTestAgent(t, true,' | \ xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'
2020-03-31 20:24:39 +00:00
a := StartTestAgent(t, TestAgent{HCL: cfg, DataDir: dataDir})
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
defer os.RemoveAll(dataDir)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
check := &structs.HealthCheck{
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
Node: a.config.NodeName,
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
CheckID: "mem",
Name: "memory check",
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
Status: api.HealthPassing,
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
}
agent: move structs into consul/structs pkg * CheckDefinition * ServiceDefinition * CheckType
2017-06-15 16:46:06 +00:00
chkType := &structs.CheckType{
Remove the script field from checks in favor of args
2018-05-08 22:31:53 +00:00
ScriptArgs: []string{"/bin/true"},
Interval: 10 * time.Second,
agent: persist CheckType with health checks
2014-11-29 20:25:01 +00:00
}
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
cid := check.CompoundCheckID()
file := filepath.Join(a.Config.DataDir, checksDir, cid.StringHash())
agent: prefer config over persisted services/checks (#497)
2014-11-25 03:24:32 +00:00
// Not persisted if not requested
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
require.NoError(t, a.AddCheck(check, chkType, false, "", ConfigSourceLocal))
_, err := os.Stat(file)
require.Error(t, err, "should not persist")
agent: prefer config over persisted services/checks (#497)
2014-11-25 03:24:32 +00:00
// Should persist if requested
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
require.NoError(t, a.AddCheck(check, chkType, true, "mytoken", ConfigSourceLocal))
_, err = os.Stat(file)
require.NoError(t, err)
agent: restore tokens for services and checks in config
2015-04-28 19:44:46 +00:00
expected, err := json.Marshal(persistedCheck{
Check: check,
ChkType: chkType,
agent: use an additional parameter for passing tokens
2015-05-05 00:36:17 +00:00
Token: "mytoken",
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
Source: "local",
agent: restore tokens for services and checks in config
2015-04-28 19:44:46 +00:00
})
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
require.NoError(t, err)
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
content, err := ioutil.ReadFile(file)
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
require.NoError(t, err)
require.Equal(t, expected, content)
agent: allow persisted services to be updated on disk
2015-05-06 05:08:03 +00:00
// Updates the check definition on disk
check.Name = "mem1"
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
require.NoError(t, a.AddCheck(check, chkType, true, "mytoken", ConfigSourceLocal))
agent: allow persisted services to be updated on disk
2015-05-06 05:08:03 +00:00
expected, err = json.Marshal(persistedCheck{
Check: check,
ChkType: chkType,
Token: "mytoken",
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
Source: "local",
agent: allow persisted services to be updated on disk
2015-05-06 05:08:03 +00:00
})
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
require.NoError(t, err)
agent: allow persisted services to be updated on disk
2015-05-06 05:08:03 +00:00
content, err = ioutil.ReadFile(file)
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
require.NoError(t, err)
require.Equal(t, expected, content)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
a.Shutdown()
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
// Should load it back during later start
Rename NewTestAgentWithFields to StartTestAgent This function now only starts the agent. Using: git grep -l 'StartTestAgent(t, true,' | \ xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'
2020-03-31 20:24:39 +00:00
a2 := StartTestAgent(t, TestAgent{Name: "Agent2", HCL: cfg, DataDir: dataDir})
test: start secondary agents also via TestAgent
2017-05-31 08:56:19 +00:00
defer a2.Shutdown()
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
result := requireCheckExists(t, a2, check.CheckID)
require.Equal(t, api.HealthCritical, result.Status)
require.Equal(t, "mem1", result.Name)
agent: persist CheckType with health checks
2014-11-29 20:25:01 +00:00
// Should have restored the monitor
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireCheckExistsMap(t, a2.checkMonitors, check.CheckID)
chkState := a2.State.CheckState(structs.NewCheckID(check.CheckID, nil))
require.NotNil(t, chkState)
require.Equal(t, "mytoken", chkState.Token)
agent: allow config reload to modify checks/services persistence This change consolidates loading services and checks from both config and persisted state into methods on the agent. As part of this, we introduce optional persistence when calling RemoveCheck/RemoveService. Fixes a bug where config reloads would kill persisted services/checks. Also fixes an edge case: 1. A service or check is registered via the HTTP API 2. A new service or check definition with the same ID is added to config 3. Config is reloaded The desired behavior (which this implements) is: 1. All services and checks deregistered in memory 2. All services and checks in config are registered first 3. All persisted checks are restored using the same logic as the agent start sequence, which prioritizes config over persisted, and removes any persistence files if new config counterparts are present.
2014-11-26 07:58:02 +00:00
}
func TestAgent_PurgeCheck(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: allow config reload to modify checks/services persistence This change consolidates loading services and checks from both config and persisted state into methods on the agent. As part of this, we introduce optional persistence when calling RemoveCheck/RemoveService. Fixes a bug where config reloads would kill persisted services/checks. Also fixes an edge case: 1. A service or check is registered via the HTTP API 2. A new service or check definition with the same ID is added to config 3. Config is reloaded The desired behavior (which this implements) is: 1. All services and checks deregistered in memory 2. All services and checks in config are registered first 3. All persisted checks are restored using the same logic as the agent start sequence, which prioritizes config over persisted, and removes any persistence files if new config counterparts are present.
2014-11-26 07:58:02 +00:00
check := &structs.HealthCheck{
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
Node: a.Config.NodeName,
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
CheckID: "mem",
Name: "memory check",
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
Status: api.HealthPassing,
agent: allow config reload to modify checks/services persistence This change consolidates loading services and checks from both config and persisted state into methods on the agent. As part of this, we introduce optional persistence when calling RemoveCheck/RemoveService. Fixes a bug where config reloads would kill persisted services/checks. Also fixes an edge case: 1. A service or check is registered via the HTTP API 2. A new service or check definition with the same ID is added to config 3. Config is reloaded The desired behavior (which this implements) is: 1. All services and checks deregistered in memory 2. All services and checks in config are registered first 3. All persisted checks are restored using the same logic as the agent start sequence, which prioritizes config over persisted, and removes any persistence files if new config counterparts are present.
2014-11-26 07:58:02 +00:00
}
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
file := filepath.Join(a.Config.DataDir, checksDir, checkIDHash(check.CheckID))
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddCheck(check, nil, true, "", ConfigSourceLocal); err != nil {
agent: allow config reload to modify checks/services persistence This change consolidates loading services and checks from both config and persisted state into methods on the agent. As part of this, we introduce optional persistence when calling RemoveCheck/RemoveService. Fixes a bug where config reloads would kill persisted services/checks. Also fixes an edge case: 1. A service or check is registered via the HTTP API 2. A new service or check definition with the same ID is added to config 3. Config is reloaded The desired behavior (which this implements) is: 1. All services and checks deregistered in memory 2. All services and checks in config are registered first 3. All persisted checks are restored using the same logic as the agent start sequence, which prioritizes config over persisted, and removes any persistence files if new config counterparts are present.
2014-11-26 07:58:02 +00:00
t.Fatalf("err: %v", err)
}
// Not removed
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.RemoveCheck(structs.NewCheckID(check.CheckID, nil), false); err != nil {
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
t.Fatalf("err: %s", err)
}
agent: allow config reload to modify checks/services persistence This change consolidates loading services and checks from both config and persisted state into methods on the agent. As part of this, we introduce optional persistence when calling RemoveCheck/RemoveService. Fixes a bug where config reloads would kill persisted services/checks. Also fixes an edge case: 1. A service or check is registered via the HTTP API 2. A new service or check definition with the same ID is added to config 3. Config is reloaded The desired behavior (which this implements) is: 1. All services and checks deregistered in memory 2. All services and checks in config are registered first 3. All persisted checks are restored using the same logic as the agent start sequence, which prioritizes config over persisted, and removes any persistence files if new config counterparts are present.
2014-11-26 07:58:02 +00:00
if _, err := os.Stat(file); err != nil {
t.Fatalf("err: %s", err)
}
// Removed
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.RemoveCheck(structs.NewCheckID(check.CheckID, nil), true); err != nil {
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
t.Fatalf("err: %s", err)
}
agent: allow config reload to modify checks/services persistence This change consolidates loading services and checks from both config and persisted state into methods on the agent. As part of this, we introduce optional persistence when calling RemoveCheck/RemoveService. Fixes a bug where config reloads would kill persisted services/checks. Also fixes an edge case: 1. A service or check is registered via the HTTP API 2. A new service or check definition with the same ID is added to config 3. Config is reloaded The desired behavior (which this implements) is: 1. All services and checks deregistered in memory 2. All services and checks in config are registered first 3. All persisted checks are restored using the same logic as the agent start sequence, which prioritizes config over persisted, and removes any persistence files if new config counterparts are present.
2014-11-26 07:58:02 +00:00
if _, err := os.Stat(file); !os.IsNotExist(err) {
t.Fatalf("bad: %#v", err)
}
agent: first pass at local service and check persistence
2014-11-24 08:36:03 +00:00
}
agent: prefer config over persisted services/checks (#497)
2014-11-25 03:24:32 +00:00
func TestAgent_PurgeCheckOnDuplicate(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
nodeID := NodeID()
dataDir := testutil.TempDir(t, "agent")
Rename NewTestAgentWithFields to StartTestAgent This function now only starts the agent. Using: git grep -l 'StartTestAgent(t, true,' | \ xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'
2020-03-31 20:24:39 +00:00
a := StartTestAgent(t, TestAgent{
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
DataDir: dataDir,
HCL: `
node_id = "` + nodeID + `"
node_name = "Node ` + nodeID + `"
data_dir = "` + dataDir + `"
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
server = false
bootstrap = false
enable_script_checks = true
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
`})
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
defer os.RemoveAll(dataDir)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: prefer config over persisted services/checks (#497)
2014-11-25 03:24:32 +00:00
check1 := &structs.HealthCheck{
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
Node: a.Config.NodeName,
CheckID: "mem",
Name: "memory check",
Status: api.HealthPassing,
EnterpriseMeta: *structs.DefaultEnterpriseMeta(),
agent: prefer config over persisted services/checks (#497)
2014-11-25 03:24:32 +00:00
}
// First persist the check
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddCheck(check1, nil, true, "", ConfigSourceLocal); err != nil {
agent: prefer config over persisted services/checks (#497)
2014-11-25 03:24:32 +00:00
t.Fatalf("err: %v", err)
}
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
a.Shutdown()
agent: prefer config over persisted services/checks (#497)
2014-11-25 03:24:32 +00:00
// Start again with the check registered in config
Rename NewTestAgentWithFields to StartTestAgent This function now only starts the agent. Using: git grep -l 'StartTestAgent(t, true,' | \ xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'
2020-03-31 20:24:39 +00:00
a2 := StartTestAgent(t, TestAgent{
Remove t.Name() from TestAgent.Name And re-add the name to the logger so that log messages from different agents in a single can be identified.
2020-03-30 20:05:27 +00:00
Name: "Agent2",
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
DataDir: dataDir,
HCL: `
node_id = "` + nodeID + `"
node_name = "Node ` + nodeID + `"
data_dir = "` + dataDir + `"
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
server = false
bootstrap = false
enable_script_checks = true
check = {
id = "mem"
name = "memory check"
notes = "my cool notes"
Remove the script field from checks in favor of args
2018-05-08 22:31:53 +00:00
args = ["/bin/check-redis.py"]
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
interval = "30s"
}
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
`})
test: start secondary agents also via TestAgent
2017-05-31 08:56:19 +00:00
defer a2.Shutdown()
agent: prefer config over persisted services/checks (#497)
2014-11-25 03:24:32 +00:00
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
cid := check1.CompoundCheckID()
file := filepath.Join(dataDir, checksDir, cid.StringHash())
agent: prefer config over persisted services/checks (#497)
2014-11-25 03:24:32 +00:00
if _, err := os.Stat(file); err == nil {
t.Fatalf("should have removed persisted check")
}
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
result := requireCheckExists(t, a2, "mem")
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
expected := &structs.HealthCheck{
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
Node: a2.Config.NodeName,
CheckID: "mem",
Name: "memory check",
Status: api.HealthCritical,
Notes: "my cool notes",
EnterpriseMeta: *structs.DefaultEnterpriseMeta(),
agent: prefer config over persisted services/checks (#497)
2014-11-25 03:24:32 +00:00
}
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
require.Equal(t, expected, result)
agent: prefer config over persisted services/checks (#497)
2014-11-25 03:24:32 +00:00
}
agent: test service and check unloading
2015-01-08 06:26:40 +00:00
agent: restore tokens for services and checks in config
2015-04-28 19:44:46 +00:00
func TestAgent_loadChecks_token(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
check = {
id = "rabbitmq"
name = "rabbitmq"
token = "abc123"
ttl = "10s"
}
`)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: restore tokens for services and checks in config
2015-04-28 19:44:46 +00:00
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireCheckExists(t, a, "rabbitmq")
require.Equal(t, "abc123", a.State.CheckToken(structs.NewCheckID("rabbitmq", nil)))
agent: restore tokens for services and checks in config
2015-04-28 19:44:46 +00:00
}
agent: test service and check unloading
2015-01-08 06:26:40 +00:00
func TestAgent_unloadChecks(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: test service and check unloading
2015-01-08 06:26:40 +00:00
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
// First register a service
svc := &structs.NodeService{
ID: "redis",
Service: "redis",
Tags: []string{"foo"},
Port: 8000,
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddService(svc, nil, false, "", ConfigSourceLocal); err != nil {
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
t.Fatalf("err: %v", err)
}
// Register a check
agent: test service and check unloading
2015-01-08 06:26:40 +00:00
check1 := &structs.HealthCheck{
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
Node: a.Config.NodeName,
agent: support multiple checks per service
2015-01-14 01:52:17 +00:00
CheckID: "service:redis",
agent: test service and check unloading
2015-01-08 06:26:40 +00:00
Name: "redischeck",
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
Status: api.HealthPassing,
agent: test service and check unloading
2015-01-08 06:26:40 +00:00
ServiceID: "redis",
ServiceName: "redis",
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddCheck(check1, nil, false, "", ConfigSourceLocal); err != nil {
agent: test service and check unloading
2015-01-08 06:26:40 +00:00
t.Fatalf("err: %s", err)
}
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireCheckExists(t, a, check1.CheckID)
agent: test service and check unloading
2015-01-08 06:26:40 +00:00
// Unload all of the checks
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
if err := a.unloadChecks(); err != nil {
agent: test service and check unloading
2015-01-08 06:26:40 +00:00
t.Fatalf("err: %s", err)
}
// Make sure it was unloaded
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireCheckMissing(t, a, check1.CheckID)
agent: test service and check unloading
2015-01-08 06:26:40 +00:00
}
agent: restore tokens for services and checks in config
2015-04-28 19:44:46 +00:00
func TestAgent_loadServices_token(t *testing.T) {
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
t.Run("normal", func(t *testing.T) {
t.Parallel()
testAgent_loadServices_token(t, "")
})
t.Run("service manager", func(t *testing.T) {
t.Parallel()
testAgent_loadServices_token(t, "enable_central_service_config = true")
})
}
func testAgent_loadServices_token(t *testing.T, extraHCL string) {
t.Helper()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
service = {
id = "rabbitmq"
name = "rabbitmq"
port = 5672
token = "abc123"
}
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
`+extraHCL)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: restore tokens for services and checks in config
2015-04-28 19:44:46 +00:00
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireServiceExists(t, a, "rabbitmq")
if token := a.State.ServiceToken(structs.NewServiceID("rabbitmq", nil)); token != "abc123" {
agent: restore tokens for services and checks in config
2015-04-28 19:44:46 +00:00
t.Fatalf("bad: %s", token)
}
}
Add SidecarService Syntax sugar to Service Definition (#4686) * Added new Config for SidecarService in ServiceDefinitions. * WIP: all the code needed for SidecarService is written... none of it is tested other than config :). Need API updates too. * Test coverage for the new sidecarServiceFromNodeService method. * Test API registratrion with SidecarService * Recursive Key Translation :facepalm: * Add tests for nested sidecar defintion arrays to ensure they are translated correctly * Use dedicated internal state rather than Service Meta for tracking sidecars for deregistration. Add tests for deregistration. * API struct for agent register. No other endpoint should be affected yet. * Additional test cases to cover updates to API registrations
2018-09-27 13:33:12 +00:00
func TestAgent_loadServices_sidecar(t *testing.T) {
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
t.Run("normal", func(t *testing.T) {
t.Parallel()
testAgent_loadServices_sidecar(t, "")
})
t.Run("service manager", func(t *testing.T) {
t.Parallel()
testAgent_loadServices_sidecar(t, "enable_central_service_config = true")
})
}
func testAgent_loadServices_sidecar(t *testing.T, extraHCL string) {
t.Helper()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
Add SidecarService Syntax sugar to Service Definition (#4686) * Added new Config for SidecarService in ServiceDefinitions. * WIP: all the code needed for SidecarService is written... none of it is tested other than config :). Need API updates too. * Test coverage for the new sidecarServiceFromNodeService method. * Test API registratrion with SidecarService * Recursive Key Translation :facepalm: * Add tests for nested sidecar defintion arrays to ensure they are translated correctly * Use dedicated internal state rather than Service Meta for tracking sidecars for deregistration. Add tests for deregistration. * API struct for agent register. No other endpoint should be affected yet. * Additional test cases to cover updates to API registrations
2018-09-27 13:33:12 +00:00
service = {
id = "rabbitmq"
name = "rabbitmq"
port = 5672
token = "abc123"
connect = {
sidecar_service {}
}
}
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
`+extraHCL)
Add SidecarService Syntax sugar to Service Definition (#4686) * Added new Config for SidecarService in ServiceDefinitions. * WIP: all the code needed for SidecarService is written... none of it is tested other than config :). Need API updates too. * Test coverage for the new sidecarServiceFromNodeService method. * Test API registratrion with SidecarService * Recursive Key Translation :facepalm: * Add tests for nested sidecar defintion arrays to ensure they are translated correctly * Use dedicated internal state rather than Service Meta for tracking sidecars for deregistration. Add tests for deregistration. * API struct for agent register. No other endpoint should be affected yet. * Additional test cases to cover updates to API registrations
2018-09-27 13:33:12 +00:00
defer a.Shutdown()
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
svc := requireServiceExists(t, a, "rabbitmq")
if token := a.State.ServiceToken(structs.NewServiceID("rabbitmq", nil)); token != "abc123" {
Add SidecarService Syntax sugar to Service Definition (#4686) * Added new Config for SidecarService in ServiceDefinitions. * WIP: all the code needed for SidecarService is written... none of it is tested other than config :). Need API updates too. * Test coverage for the new sidecarServiceFromNodeService method. * Test API registratrion with SidecarService * Recursive Key Translation :facepalm: * Add tests for nested sidecar defintion arrays to ensure they are translated correctly * Use dedicated internal state rather than Service Meta for tracking sidecars for deregistration. Add tests for deregistration. * API struct for agent register. No other endpoint should be affected yet. * Additional test cases to cover updates to API registrations
2018-09-27 13:33:12 +00:00
t.Fatalf("bad: %s", token)
}
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireServiceExists(t, a, "rabbitmq-sidecar-proxy")
if token := a.State.ServiceToken(structs.NewServiceID("rabbitmq-sidecar-proxy", nil)); token != "abc123" {
Add SidecarService Syntax sugar to Service Definition (#4686) * Added new Config for SidecarService in ServiceDefinitions. * WIP: all the code needed for SidecarService is written... none of it is tested other than config :). Need API updates too. * Test coverage for the new sidecarServiceFromNodeService method. * Test API registratrion with SidecarService * Recursive Key Translation :facepalm: * Add tests for nested sidecar defintion arrays to ensure they are translated correctly * Use dedicated internal state rather than Service Meta for tracking sidecars for deregistration. Add tests for deregistration. * API struct for agent register. No other endpoint should be affected yet. * Additional test cases to cover updates to API registrations
2018-09-27 13:33:12 +00:00
t.Fatalf("bad: %s", token)
}
// Sanity check rabbitmq service should NOT have sidecar info in state since
// it's done it's job and should be a registration syntax sugar only.
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
assert.Nil(t, svc.Connect.SidecarService)
Add SidecarService Syntax sugar to Service Definition (#4686) * Added new Config for SidecarService in ServiceDefinitions. * WIP: all the code needed for SidecarService is written... none of it is tested other than config :). Need API updates too. * Test coverage for the new sidecarServiceFromNodeService method. * Test API registratrion with SidecarService * Recursive Key Translation :facepalm: * Add tests for nested sidecar defintion arrays to ensure they are translated correctly * Use dedicated internal state rather than Service Meta for tracking sidecars for deregistration. Add tests for deregistration. * API struct for agent register. No other endpoint should be affected yet. * Additional test cases to cover updates to API registrations
2018-09-27 13:33:12 +00:00
}
func TestAgent_loadServices_sidecarSeparateToken(t *testing.T) {
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
t.Run("normal", func(t *testing.T) {
t.Parallel()
testAgent_loadServices_sidecarSeparateToken(t, "")
})
t.Run("service manager", func(t *testing.T) {
t.Parallel()
testAgent_loadServices_sidecarSeparateToken(t, "enable_central_service_config = true")
})
}
func testAgent_loadServices_sidecarSeparateToken(t *testing.T, extraHCL string) {
t.Helper()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
Add SidecarService Syntax sugar to Service Definition (#4686) * Added new Config for SidecarService in ServiceDefinitions. * WIP: all the code needed for SidecarService is written... none of it is tested other than config :). Need API updates too. * Test coverage for the new sidecarServiceFromNodeService method. * Test API registratrion with SidecarService * Recursive Key Translation :facepalm: * Add tests for nested sidecar defintion arrays to ensure they are translated correctly * Use dedicated internal state rather than Service Meta for tracking sidecars for deregistration. Add tests for deregistration. * API struct for agent register. No other endpoint should be affected yet. * Additional test cases to cover updates to API registrations
2018-09-27 13:33:12 +00:00
service = {
id = "rabbitmq"
name = "rabbitmq"
port = 5672
token = "abc123"
connect = {
sidecar_service {
token = "789xyz"
}
}
}
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
`+extraHCL)
Add SidecarService Syntax sugar to Service Definition (#4686) * Added new Config for SidecarService in ServiceDefinitions. * WIP: all the code needed for SidecarService is written... none of it is tested other than config :). Need API updates too. * Test coverage for the new sidecarServiceFromNodeService method. * Test API registratrion with SidecarService * Recursive Key Translation :facepalm: * Add tests for nested sidecar defintion arrays to ensure they are translated correctly * Use dedicated internal state rather than Service Meta for tracking sidecars for deregistration. Add tests for deregistration. * API struct for agent register. No other endpoint should be affected yet. * Additional test cases to cover updates to API registrations
2018-09-27 13:33:12 +00:00
defer a.Shutdown()
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireServiceExists(t, a, "rabbitmq")
if token := a.State.ServiceToken(structs.NewServiceID("rabbitmq", nil)); token != "abc123" {
Add SidecarService Syntax sugar to Service Definition (#4686) * Added new Config for SidecarService in ServiceDefinitions. * WIP: all the code needed for SidecarService is written... none of it is tested other than config :). Need API updates too. * Test coverage for the new sidecarServiceFromNodeService method. * Test API registratrion with SidecarService * Recursive Key Translation :facepalm: * Add tests for nested sidecar defintion arrays to ensure they are translated correctly * Use dedicated internal state rather than Service Meta for tracking sidecars for deregistration. Add tests for deregistration. * API struct for agent register. No other endpoint should be affected yet. * Additional test cases to cover updates to API registrations
2018-09-27 13:33:12 +00:00
t.Fatalf("bad: %s", token)
}
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireServiceExists(t, a, "rabbitmq-sidecar-proxy")
if token := a.State.ServiceToken(structs.NewServiceID("rabbitmq-sidecar-proxy", nil)); token != "789xyz" {
Add SidecarService Syntax sugar to Service Definition (#4686) * Added new Config for SidecarService in ServiceDefinitions. * WIP: all the code needed for SidecarService is written... none of it is tested other than config :). Need API updates too. * Test coverage for the new sidecarServiceFromNodeService method. * Test API registratrion with SidecarService * Recursive Key Translation :facepalm: * Add tests for nested sidecar defintion arrays to ensure they are translated correctly * Use dedicated internal state rather than Service Meta for tracking sidecars for deregistration. Add tests for deregistration. * API struct for agent register. No other endpoint should be affected yet. * Additional test cases to cover updates to API registrations
2018-09-27 13:33:12 +00:00
t.Fatalf("bad: %s", token)
}
}
Improve Connect with Prepared Queries (#5291) Given a query like: ``` { "Name": "tagged-connect-query", "Service": { "Service": "foo", "Tags": ["tag"], "Connect": true } } ``` And a Consul configuration like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": {} }, "tags": ["tag"] ] } ``` If you executed the query it would always turn up with 0 results. This was because the sidecar service was being created without any tags. You could instead make your config look like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": { "tags": ["tag"] } }, "tags": ["tag"] ] } ``` However that is a bit redundant for most cases. This PR ensures that the tags and service meta of the parent service get copied to the sidecar service. If there are any tags or service meta set in the sidecar service definition then this copying does not take place. After the changes, the query will now return the expected results. A second change was made to prepared queries in this PR which is to allow filtering on ServiceMeta just like we allow for filtering on NodeMeta.
2019-02-04 14:36:51 +00:00
func TestAgent_loadServices_sidecarInheritMeta(t *testing.T) {
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
t.Run("normal", func(t *testing.T) {
t.Parallel()
testAgent_loadServices_sidecarInheritMeta(t, "")
})
t.Run("service manager", func(t *testing.T) {
t.Parallel()
testAgent_loadServices_sidecarInheritMeta(t, "enable_central_service_config = true")
})
}
func testAgent_loadServices_sidecarInheritMeta(t *testing.T, extraHCL string) {
t.Helper()
Improve Connect with Prepared Queries (#5291) Given a query like: ``` { "Name": "tagged-connect-query", "Service": { "Service": "foo", "Tags": ["tag"], "Connect": true } } ``` And a Consul configuration like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": {} }, "tags": ["tag"] ] } ``` If you executed the query it would always turn up with 0 results. This was because the sidecar service was being created without any tags. You could instead make your config look like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": { "tags": ["tag"] } }, "tags": ["tag"] ] } ``` However that is a bit redundant for most cases. This PR ensures that the tags and service meta of the parent service get copied to the sidecar service. If there are any tags or service meta set in the sidecar service definition then this copying does not take place. After the changes, the query will now return the expected results. A second change was made to prepared queries in this PR which is to allow filtering on ServiceMeta just like we allow for filtering on NodeMeta.
2019-02-04 14:36:51 +00:00
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
Improve Connect with Prepared Queries (#5291) Given a query like: ``` { "Name": "tagged-connect-query", "Service": { "Service": "foo", "Tags": ["tag"], "Connect": true } } ``` And a Consul configuration like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": {} }, "tags": ["tag"] ] } ``` If you executed the query it would always turn up with 0 results. This was because the sidecar service was being created without any tags. You could instead make your config look like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": { "tags": ["tag"] } }, "tags": ["tag"] ] } ``` However that is a bit redundant for most cases. This PR ensures that the tags and service meta of the parent service get copied to the sidecar service. If there are any tags or service meta set in the sidecar service definition then this copying does not take place. After the changes, the query will now return the expected results. A second change was made to prepared queries in this PR which is to allow filtering on ServiceMeta just like we allow for filtering on NodeMeta.
2019-02-04 14:36:51 +00:00
service = {
id = "rabbitmq"
name = "rabbitmq"
port = 5672
tags = ["a", "b"],
meta = {
environment = "prod"
}
connect = {
sidecar_service {
}
}
}
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
`+extraHCL)
Improve Connect with Prepared Queries (#5291) Given a query like: ``` { "Name": "tagged-connect-query", "Service": { "Service": "foo", "Tags": ["tag"], "Connect": true } } ``` And a Consul configuration like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": {} }, "tags": ["tag"] ] } ``` If you executed the query it would always turn up with 0 results. This was because the sidecar service was being created without any tags. You could instead make your config look like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": { "tags": ["tag"] } }, "tags": ["tag"] ] } ``` However that is a bit redundant for most cases. This PR ensures that the tags and service meta of the parent service get copied to the sidecar service. If there are any tags or service meta set in the sidecar service definition then this copying does not take place. After the changes, the query will now return the expected results. A second change was made to prepared queries in this PR which is to allow filtering on ServiceMeta just like we allow for filtering on NodeMeta.
2019-02-04 14:36:51 +00:00
defer a.Shutdown()
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
svc := requireServiceExists(t, a, "rabbitmq")
Improve Connect with Prepared Queries (#5291) Given a query like: ``` { "Name": "tagged-connect-query", "Service": { "Service": "foo", "Tags": ["tag"], "Connect": true } } ``` And a Consul configuration like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": {} }, "tags": ["tag"] ] } ``` If you executed the query it would always turn up with 0 results. This was because the sidecar service was being created without any tags. You could instead make your config look like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": { "tags": ["tag"] } }, "tags": ["tag"] ] } ``` However that is a bit redundant for most cases. This PR ensures that the tags and service meta of the parent service get copied to the sidecar service. If there are any tags or service meta set in the sidecar service definition then this copying does not take place. After the changes, the query will now return the expected results. A second change was made to prepared queries in this PR which is to allow filtering on ServiceMeta just like we allow for filtering on NodeMeta.
2019-02-04 14:36:51 +00:00
require.Len(t, svc.Tags, 2)
require.Len(t, svc.Meta, 1)
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
sidecar := requireServiceExists(t, a, "rabbitmq-sidecar-proxy")
Improve Connect with Prepared Queries (#5291) Given a query like: ``` { "Name": "tagged-connect-query", "Service": { "Service": "foo", "Tags": ["tag"], "Connect": true } } ``` And a Consul configuration like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": {} }, "tags": ["tag"] ] } ``` If you executed the query it would always turn up with 0 results. This was because the sidecar service was being created without any tags. You could instead make your config look like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": { "tags": ["tag"] } }, "tags": ["tag"] ] } ``` However that is a bit redundant for most cases. This PR ensures that the tags and service meta of the parent service get copied to the sidecar service. If there are any tags or service meta set in the sidecar service definition then this copying does not take place. After the changes, the query will now return the expected results. A second change was made to prepared queries in this PR which is to allow filtering on ServiceMeta just like we allow for filtering on NodeMeta.
2019-02-04 14:36:51 +00:00
require.ElementsMatch(t, svc.Tags, sidecar.Tags)
require.Len(t, sidecar.Meta, 1)
meta, ok := sidecar.Meta["environment"]
require.True(t, ok, "missing sidecar service meta")
require.Equal(t, "prod", meta)
}
func TestAgent_loadServices_sidecarOverrideMeta(t *testing.T) {
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
t.Run("normal", func(t *testing.T) {
t.Parallel()
testAgent_loadServices_sidecarOverrideMeta(t, "")
})
t.Run("service manager", func(t *testing.T) {
t.Parallel()
testAgent_loadServices_sidecarOverrideMeta(t, "enable_central_service_config = true")
})
}
func testAgent_loadServices_sidecarOverrideMeta(t *testing.T, extraHCL string) {
t.Helper()
Improve Connect with Prepared Queries (#5291) Given a query like: ``` { "Name": "tagged-connect-query", "Service": { "Service": "foo", "Tags": ["tag"], "Connect": true } } ``` And a Consul configuration like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": {} }, "tags": ["tag"] ] } ``` If you executed the query it would always turn up with 0 results. This was because the sidecar service was being created without any tags. You could instead make your config look like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": { "tags": ["tag"] } }, "tags": ["tag"] ] } ``` However that is a bit redundant for most cases. This PR ensures that the tags and service meta of the parent service get copied to the sidecar service. If there are any tags or service meta set in the sidecar service definition then this copying does not take place. After the changes, the query will now return the expected results. A second change was made to prepared queries in this PR which is to allow filtering on ServiceMeta just like we allow for filtering on NodeMeta.
2019-02-04 14:36:51 +00:00
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
Improve Connect with Prepared Queries (#5291) Given a query like: ``` { "Name": "tagged-connect-query", "Service": { "Service": "foo", "Tags": ["tag"], "Connect": true } } ``` And a Consul configuration like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": {} }, "tags": ["tag"] ] } ``` If you executed the query it would always turn up with 0 results. This was because the sidecar service was being created without any tags. You could instead make your config look like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": { "tags": ["tag"] } }, "tags": ["tag"] ] } ``` However that is a bit redundant for most cases. This PR ensures that the tags and service meta of the parent service get copied to the sidecar service. If there are any tags or service meta set in the sidecar service definition then this copying does not take place. After the changes, the query will now return the expected results. A second change was made to prepared queries in this PR which is to allow filtering on ServiceMeta just like we allow for filtering on NodeMeta.
2019-02-04 14:36:51 +00:00
service = {
id = "rabbitmq"
name = "rabbitmq"
port = 5672
tags = ["a", "b"],
meta = {
environment = "prod"
}
connect = {
sidecar_service {
tags = ["foo"],
meta = {
environment = "qa"
}
}
}
}
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
`+extraHCL)
Improve Connect with Prepared Queries (#5291) Given a query like: ``` { "Name": "tagged-connect-query", "Service": { "Service": "foo", "Tags": ["tag"], "Connect": true } } ``` And a Consul configuration like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": {} }, "tags": ["tag"] ] } ``` If you executed the query it would always turn up with 0 results. This was because the sidecar service was being created without any tags. You could instead make your config look like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": { "tags": ["tag"] } }, "tags": ["tag"] ] } ``` However that is a bit redundant for most cases. This PR ensures that the tags and service meta of the parent service get copied to the sidecar service. If there are any tags or service meta set in the sidecar service definition then this copying does not take place. After the changes, the query will now return the expected results. A second change was made to prepared queries in this PR which is to allow filtering on ServiceMeta just like we allow for filtering on NodeMeta.
2019-02-04 14:36:51 +00:00
defer a.Shutdown()
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
svc := requireServiceExists(t, a, "rabbitmq")
Improve Connect with Prepared Queries (#5291) Given a query like: ``` { "Name": "tagged-connect-query", "Service": { "Service": "foo", "Tags": ["tag"], "Connect": true } } ``` And a Consul configuration like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": {} }, "tags": ["tag"] ] } ``` If you executed the query it would always turn up with 0 results. This was because the sidecar service was being created without any tags. You could instead make your config look like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": { "tags": ["tag"] } }, "tags": ["tag"] ] } ``` However that is a bit redundant for most cases. This PR ensures that the tags and service meta of the parent service get copied to the sidecar service. If there are any tags or service meta set in the sidecar service definition then this copying does not take place. After the changes, the query will now return the expected results. A second change was made to prepared queries in this PR which is to allow filtering on ServiceMeta just like we allow for filtering on NodeMeta.
2019-02-04 14:36:51 +00:00
require.Len(t, svc.Tags, 2)
require.Len(t, svc.Meta, 1)
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
sidecar := requireServiceExists(t, a, "rabbitmq-sidecar-proxy")
Improve Connect with Prepared Queries (#5291) Given a query like: ``` { "Name": "tagged-connect-query", "Service": { "Service": "foo", "Tags": ["tag"], "Connect": true } } ``` And a Consul configuration like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": {} }, "tags": ["tag"] ] } ``` If you executed the query it would always turn up with 0 results. This was because the sidecar service was being created without any tags. You could instead make your config look like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": { "tags": ["tag"] } }, "tags": ["tag"] ] } ``` However that is a bit redundant for most cases. This PR ensures that the tags and service meta of the parent service get copied to the sidecar service. If there are any tags or service meta set in the sidecar service definition then this copying does not take place. After the changes, the query will now return the expected results. A second change was made to prepared queries in this PR which is to allow filtering on ServiceMeta just like we allow for filtering on NodeMeta.
2019-02-04 14:36:51 +00:00
require.Len(t, sidecar.Tags, 1)
require.Equal(t, "foo", sidecar.Tags[0])
require.Len(t, sidecar.Meta, 1)
meta, ok := sidecar.Meta["environment"]
require.True(t, ok, "missing sidecar service meta")
require.Equal(t, "qa", meta)
}
agent: test service and check unloading
2015-01-08 06:26:40 +00:00
func TestAgent_unloadServices(t *testing.T) {
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
t.Run("normal", func(t *testing.T) {
t.Parallel()
testAgent_unloadServices(t, "")
})
t.Run("service manager", func(t *testing.T) {
t.Parallel()
testAgent_unloadServices(t, "enable_central_service_config = true")
})
}
func testAgent_unloadServices(t *testing.T, extraHCL string) {
t.Helper()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, extraHCL)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: test service and check unloading
2015-01-08 06:26:40 +00:00
svc := &structs.NodeService{
ID: "redis",
Service: "redis",
Tags: []string{"foo"},
Port: 8000,
}
// Register the service
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddService(svc, nil, false, "", ConfigSourceLocal); err != nil {
agent: test service and check unloading
2015-01-08 06:26:40 +00:00
t.Fatalf("err: %v", err)
}
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireServiceExists(t, a, svc.ID)
agent: test service and check unloading
2015-01-08 06:26:40 +00:00
// Unload all services
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
if err := a.unloadServices(); err != nil {
agent: test service and check unloading
2015-01-08 06:26:40 +00:00
t.Fatalf("err: %s", err)
}
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if len(a.State.Services(structs.WildcardEnterpriseMeta())) != 0 {
Cleans up version 8 ACLs in the agent and the docs. (#3248) * Moves magic check and service constants into shared structs package. * Removes the "consul" service from local state. Since this service is added by the leader, it doesn't really make sense to also keep it in local state (which requires special ACLs to configure), and requires a bunch of special cases in the local state logic. This requires fewer special cases and makes ACL bootstrapping cleaner. * Makes coordinate update ACL log message a warning, similar to other AE warnings. * Adds much more detailed examples for bootstrapping ACLs. This can hopefully replace https://gist.github.com/slackpad/d89ce0e1cc0802c3c4f2d84932fa3234.
2017-07-14 05:33:47 +00:00
t.Fatalf("should have unloaded services")
agent: test service and check unloading
2015-01-08 06:26:40 +00:00
}
}
agent: test agent service maintenance mode
2015-01-15 08:25:36 +00:00
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
func TestAgent_Service_MaintenanceMode(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: test agent service maintenance mode
2015-01-15 08:25:36 +00:00
svc := &structs.NodeService{
ID: "redis",
Service: "redis",
Tags: []string{"foo"},
Port: 8000,
}
// Register the service
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddService(svc, nil, false, "", ConfigSourceLocal); err != nil {
agent: test agent service maintenance mode
2015-01-15 08:25:36 +00:00
t.Fatalf("err: %v", err)
}
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
sid := structs.NewServiceID("redis", nil)
agent: test agent service maintenance mode
2015-01-15 08:25:36 +00:00
// Enter maintenance mode for the service
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.EnableServiceMaintenance(sid, "broken", "mytoken"); err != nil {
agent: test agent service maintenance mode
2015-01-15 08:25:36 +00:00
t.Fatalf("err: %s", err)
}
// Make sure the critical health check was added
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
checkID := serviceMaintCheckID(sid)
check := a.State.Check(checkID)
if check == nil {
agent: test node maintenance mode
2015-01-15 19:26:14 +00:00
t.Fatalf("should have registered critical maintenance check")
agent: test agent service maintenance mode
2015-01-15 08:25:36 +00:00
}
agent: testing node/service maintenance using tokens
2015-09-10 19:08:08 +00:00
// Check that the token was used to register the check
local state: tests compile
2017-08-28 12:17:13 +00:00
if token := a.State.CheckToken(checkID); token != "mytoken" {
agent: testing node/service maintenance using tokens
2015-09-10 19:08:08 +00:00
t.Fatalf("expected 'mytoken', got: '%s'", token)
}
agent: support passing ?reason= for custom notes field values on maintenance checks
2015-01-21 20:21:57 +00:00
// Ensure the reason was set in notes
if check.Notes != "broken" {
t.Fatalf("bad: %#v", check)
}
agent: test agent service maintenance mode
2015-01-15 08:25:36 +00:00
// Leave maintenance mode
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.DisableServiceMaintenance(sid); err != nil {
agent: test agent service maintenance mode
2015-01-15 08:25:36 +00:00
t.Fatalf("err: %s", err)
}
// Ensure the check was deregistered
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if found := a.State.Check(checkID); found != nil {
agent: test node maintenance mode
2015-01-15 19:26:14 +00:00
t.Fatalf("should have deregistered maintenance check")
}
agent: support passing ?reason= for custom notes field values on maintenance checks
2015-01-21 20:21:57 +00:00
// Enter service maintenance mode without providing a reason
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.EnableServiceMaintenance(sid, "", ""); err != nil {
agent: support passing ?reason= for custom notes field values on maintenance checks
2015-01-21 20:21:57 +00:00
t.Fatalf("err: %s", err)
}
// Ensure the check was registered with the default notes
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
check = a.State.Check(checkID)
if check == nil {
agent: support passing ?reason= for custom notes field values on maintenance checks
2015-01-21 20:21:57 +00:00
t.Fatalf("should have registered critical check")
}
agent: use const for default maintenance reason strings
2015-01-21 22:45:09 +00:00
if check.Notes != defaultServiceMaintReason {
agent: support passing ?reason= for custom notes field values on maintenance checks
2015-01-21 20:21:57 +00:00
t.Fatalf("bad: %#v", check)
}
agent: test node maintenance mode
2015-01-15 19:26:14 +00:00
}
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
func TestAgent_Service_Reap(t *testing.T) {
agent: make timing sensitive tests more robust * make timing less aggressive * mark timing tests as non-parallel
2017-07-04 10:44:24 +00:00
// t.Parallel() // timing test. no parallel
Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.
2020-06-10 20:47:35 +00:00
a := StartTestAgent(t, TestAgent{Overrides: `
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
check_reap_interval = "50ms"
check_deregister_interval_min = "0s"
Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.
2020-06-10 20:47:35 +00:00
`})
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
Fixed flaky tests (#4626)
2018-09-04 11:31:51 +00:00
testrpc.WaitForTestAgent(t, a.RPC, "dc1")
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
svc := &structs.NodeService{
ID: "redis",
Service: "redis",
Tags: []string{"foo"},
Port: 8000,
}
agent: move structs into consul/structs pkg * CheckDefinition * ServiceDefinition * CheckType
2017-06-15 16:46:06 +00:00
chkTypes := []*structs.CheckType{
Enable gofmt simplify Code changes done automatically with 'gofmt -s -w'
2020-06-16 17:19:31 +00:00
{
agent: only use TestAgent when appropriate (#5502)
2019-03-18 16:06:16 +00:00
Status: api.HealthPassing,
TTL: 25 * time.Millisecond,
agent: make timing sensitive tests more robust * make timing less aggressive * mark timing tests as non-parallel
2017-07-04 10:44:24 +00:00
DeregisterCriticalServiceAfter: 200 * time.Millisecond,
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
},
}
// Register the service.
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddService(svc, chkTypes, false, "", ConfigSourceLocal); err != nil {
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
t.Fatalf("err: %v", err)
}
// Make sure it's there and there's no critical check yet.
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireServiceExists(t, a, "redis")
require.Len(t, a.State.CriticalCheckStates(structs.WildcardEnterpriseMeta()), 0, "should not have critical checks")
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
agent: make timing sensitive tests more robust * make timing less aggressive * mark timing tests as non-parallel
2017-07-04 10:44:24 +00:00
// Wait for the check TTL to fail but before the check is reaped.
time.Sleep(100 * time.Millisecond)
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireServiceExists(t, a, "redis")
require.Len(t, a.State.CriticalCheckStates(nil), 1, "should have 1 critical check")
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
// Pass the TTL.
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.updateTTLCheck(structs.NewCheckID("service:redis", nil), api.HealthPassing, "foo"); err != nil {
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
t.Fatalf("err: %v", err)
}
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireServiceExists(t, a, "redis")
require.Len(t, a.State.CriticalCheckStates(structs.WildcardEnterpriseMeta()), 0, "should not have critical checks")
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
// Wait for the check TTL to fail again.
agent: make timing sensitive tests more robust * make timing less aggressive * mark timing tests as non-parallel
2017-07-04 10:44:24 +00:00
time.Sleep(100 * time.Millisecond)
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireServiceExists(t, a, "redis")
require.Len(t, a.State.CriticalCheckStates(structs.WildcardEnterpriseMeta()), 1, "should have 1 critical check")
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
// Wait for the reap.
agent: make timing sensitive tests more robust * make timing less aggressive * mark timing tests as non-parallel
2017-07-04 10:44:24 +00:00
time.Sleep(400 * time.Millisecond)
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireServiceMissing(t, a, "redis")
require.Len(t, a.State.CriticalCheckStates(structs.WildcardEnterpriseMeta()), 0, "should not have critical checks")
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
}
func TestAgent_Service_NoReap(t *testing.T) {
agent: make timing sensitive tests more robust * make timing less aggressive * mark timing tests as non-parallel
2017-07-04 10:44:24 +00:00
// t.Parallel() // timing test. no parallel
Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.
2020-06-10 20:47:35 +00:00
a := StartTestAgent(t, TestAgent{Overrides: `
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
check_reap_interval = "50ms"
check_deregister_interval_min = "0s"
Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.
2020-06-10 20:47:35 +00:00
`})
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
svc := &structs.NodeService{
ID: "redis",
Service: "redis",
Tags: []string{"foo"},
Port: 8000,
}
agent: move structs into consul/structs pkg * CheckDefinition * ServiceDefinition * CheckType
2017-06-15 16:46:06 +00:00
chkTypes := []*structs.CheckType{
Enable gofmt simplify Code changes done automatically with 'gofmt -s -w'
2020-06-16 17:19:31 +00:00
{
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
Status: api.HealthPassing,
agent: make timing sensitive tests more robust * make timing less aggressive * mark timing tests as non-parallel
2017-07-04 10:44:24 +00:00
TTL: 25 * time.Millisecond,
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
},
}
// Register the service.
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddService(svc, chkTypes, false, "", ConfigSourceLocal); err != nil {
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
t.Fatalf("err: %v", err)
}
// Make sure it's there and there's no critical check yet.
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireServiceExists(t, a, "redis")
require.Len(t, a.State.CriticalCheckStates(structs.WildcardEnterpriseMeta()), 0)
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
// Wait for the check TTL to fail.
agent: make Reap test timing less aggressive
2017-07-05 09:24:03 +00:00
time.Sleep(200 * time.Millisecond)
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireServiceExists(t, a, "redis")
require.Len(t, a.State.CriticalCheckStates(structs.WildcardEnterpriseMeta()), 1)
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
// Wait a while and make sure it doesn't reap.
agent: make timing sensitive tests more robust * make timing less aggressive * mark timing tests as non-parallel
2017-07-04 10:44:24 +00:00
time.Sleep(200 * time.Millisecond)
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireServiceExists(t, a, "redis")
require.Len(t, a.State.CriticalCheckStates(structs.WildcardEnterpriseMeta()), 1)
Adds ability to deregister a service based on critical check state longer than a timeout.
2016-08-16 07:05:55 +00:00
}
Prevent status flap when re-registering a check (#4904) Fixes point `#2` of: https://github.com/hashicorp/consul/issues/4903 When registering a service each healthcheck status is saved and restored (https://github.com/hashicorp/consul/blob/master/agent/agent.go#L1914) to avoid unnecessary flaps in health state. This change extends this feature to single check registration by moving this protection in `AddCheck()` so that both `PUT /v1/agent/service/register` and `PUT /v1/agent/check/register` behave in the same idempotent way. #### Steps to reproduce 1. Register a check : ``` curl -X PUT \ http://127.0.0.1:8500/v1/agent/check/register \ -H 'Content-Type: application/json' \ -d '{ "Name": "my_check", "ServiceID": "srv", "Interval": "10s", "Args": ["true"] }' ``` 2. The check will initialize and change to `passing` 3. Run the same request again 4. The check status will quickly go from `critical` to `passing` (the delay for this transission is determined by https://github.com/hashicorp/consul/blob/master/agent/checks/check.go#L95)
2019-01-07 18:53:03 +00:00
func TestAgent_AddService_restoresSnapshot(t *testing.T) {
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
t.Run("normal", func(t *testing.T) {
t.Parallel()
testAgent_AddService_restoresSnapshot(t, "")
})
t.Run("service manager", func(t *testing.T) {
t.Parallel()
testAgent_AddService_restoresSnapshot(t, "enable_central_service_config = true")
})
}
func testAgent_AddService_restoresSnapshot(t *testing.T, extraHCL string) {
t.Helper()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, extraHCL)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: restore check status when re-registering (updating) services
2015-05-06 19:28:42 +00:00
// First register a service
svc := &structs.NodeService{
ID: "redis",
Service: "redis",
Tags: []string{"foo"},
Port: 8000,
}
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
require.NoError(t, a.AddService(svc, nil, false, "", ConfigSourceLocal))
agent: restore check status when re-registering (updating) services
2015-05-06 19:28:42 +00:00
// Register a check
check1 := &structs.HealthCheck{
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
Node: a.Config.NodeName,
agent: restore check status when re-registering (updating) services
2015-05-06 19:28:42 +00:00
CheckID: "service:redis",
Name: "redischeck",
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
Status: api.HealthPassing,
agent: restore check status when re-registering (updating) services
2015-05-06 19:28:42 +00:00
ServiceID: "redis",
ServiceName: "redis",
}
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
require.NoError(t, a.AddCheck(check1, nil, false, "", ConfigSourceLocal))
agent: restore check status when re-registering (updating) services
2015-05-06 19:28:42 +00:00
// Re-registering the service preserves the state of the check
Enable gofmt simplify Code changes done automatically with 'gofmt -s -w'
2020-06-16 17:19:31 +00:00
chkTypes := []*structs.CheckType{{TTL: 30 * time.Second}}
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
require.NoError(t, a.AddService(svc, chkTypes, false, "", ConfigSourceLocal))
check := requireCheckExists(t, a, "service:redis")
require.Equal(t, api.HealthPassing, check.Status)
agent: restore check status when re-registering (updating) services
2015-05-06 19:28:42 +00:00
}
Prevent status flap when re-registering a check (#4904) Fixes point `#2` of: https://github.com/hashicorp/consul/issues/4903 When registering a service each healthcheck status is saved and restored (https://github.com/hashicorp/consul/blob/master/agent/agent.go#L1914) to avoid unnecessary flaps in health state. This change extends this feature to single check registration by moving this protection in `AddCheck()` so that both `PUT /v1/agent/service/register` and `PUT /v1/agent/check/register` behave in the same idempotent way. #### Steps to reproduce 1. Register a check : ``` curl -X PUT \ http://127.0.0.1:8500/v1/agent/check/register \ -H 'Content-Type: application/json' \ -d '{ "Name": "my_check", "ServiceID": "srv", "Interval": "10s", "Args": ["true"] }' ``` 2. The check will initialize and change to `passing` 3. Run the same request again 4. The check status will quickly go from `critical` to `passing` (the delay for this transission is determined by https://github.com/hashicorp/consul/blob/master/agent/checks/check.go#L95)
2019-01-07 18:53:03 +00:00
func TestAgent_AddCheck_restoresSnapshot(t *testing.T) {
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
Prevent status flap when re-registering a check (#4904) Fixes point `#2` of: https://github.com/hashicorp/consul/issues/4903 When registering a service each healthcheck status is saved and restored (https://github.com/hashicorp/consul/blob/master/agent/agent.go#L1914) to avoid unnecessary flaps in health state. This change extends this feature to single check registration by moving this protection in `AddCheck()` so that both `PUT /v1/agent/service/register` and `PUT /v1/agent/check/register` behave in the same idempotent way. #### Steps to reproduce 1. Register a check : ``` curl -X PUT \ http://127.0.0.1:8500/v1/agent/check/register \ -H 'Content-Type: application/json' \ -d '{ "Name": "my_check", "ServiceID": "srv", "Interval": "10s", "Args": ["true"] }' ``` 2. The check will initialize and change to `passing` 3. Run the same request again 4. The check status will quickly go from `critical` to `passing` (the delay for this transission is determined by https://github.com/hashicorp/consul/blob/master/agent/checks/check.go#L95)
2019-01-07 18:53:03 +00:00
defer a.Shutdown()
// First register a service
svc := &structs.NodeService{
ID: "redis",
Service: "redis",
Tags: []string{"foo"},
Port: 8000,
}
if err := a.AddService(svc, nil, false, "", ConfigSourceLocal); err != nil {
t.Fatalf("err: %v", err)
}
// Register a check
check1 := &structs.HealthCheck{
Node: a.Config.NodeName,
CheckID: "service:redis",
Name: "redischeck",
Status: api.HealthPassing,
ServiceID: "redis",
ServiceName: "redis",
}
if err := a.AddCheck(check1, nil, false, "", ConfigSourceLocal); err != nil {
t.Fatalf("err: %s", err)
}
// Re-registering the check preserves its state
check1.Status = ""
if err := a.AddCheck(check1, &structs.CheckType{TTL: 30 * time.Second}, false, "", ConfigSourceLocal); err != nil {
t.Fatalf("err: %s", err)
}
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
check := requireCheckExists(t, a, "service:redis")
Prevent status flap when re-registering a check (#4904) Fixes point `#2` of: https://github.com/hashicorp/consul/issues/4903 When registering a service each healthcheck status is saved and restored (https://github.com/hashicorp/consul/blob/master/agent/agent.go#L1914) to avoid unnecessary flaps in health state. This change extends this feature to single check registration by moving this protection in `AddCheck()` so that both `PUT /v1/agent/service/register` and `PUT /v1/agent/check/register` behave in the same idempotent way. #### Steps to reproduce 1. Register a check : ``` curl -X PUT \ http://127.0.0.1:8500/v1/agent/check/register \ -H 'Content-Type: application/json' \ -d '{ "Name": "my_check", "ServiceID": "srv", "Interval": "10s", "Args": ["true"] }' ``` 2. The check will initialize and change to `passing` 3. Run the same request again 4. The check status will quickly go from `critical` to `passing` (the delay for this transission is determined by https://github.com/hashicorp/consul/blob/master/agent/checks/check.go#L95)
2019-01-07 18:53:03 +00:00
if check.Status != api.HealthPassing {
t.Fatalf("bad: %s", check.Status)
}
}
agent: test node maintenance mode
2015-01-15 19:26:14 +00:00
func TestAgent_NodeMaintenanceMode(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: test node maintenance mode
2015-01-15 19:26:14 +00:00
// Enter maintenance mode for the node
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
a.EnableNodeMaintenance("broken", "mytoken")
agent: test node maintenance mode
2015-01-15 19:26:14 +00:00
// Make sure the critical health check was added
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
check := requireCheckExists(t, a, structs.NodeMaint)
agent: test node maintenance mode
2015-01-15 19:26:14 +00:00
agent: testing node/service maintenance using tokens
2015-09-10 19:08:08 +00:00
// Check that the token was used to register the check
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if token := a.State.CheckToken(structs.NodeMaintCheckID); token != "mytoken" {
agent: testing node/service maintenance using tokens
2015-09-10 19:08:08 +00:00
t.Fatalf("expected 'mytoken', got: '%s'", token)
}
agent: support passing ?reason= for custom notes field values on maintenance checks
2015-01-21 20:21:57 +00:00
// Ensure the reason was set in notes
if check.Notes != "broken" {
t.Fatalf("bad: %#v", check)
}
agent: test node maintenance mode
2015-01-15 19:26:14 +00:00
// Leave maintenance mode
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
a.DisableNodeMaintenance()
agent: test node maintenance mode
2015-01-15 19:26:14 +00:00
// Ensure the check was deregistered
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
requireCheckMissing(t, a, structs.NodeMaint)
agent: support passing ?reason= for custom notes field values on maintenance checks
2015-01-21 20:21:57 +00:00
// Enter maintenance mode without passing a reason
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
a.EnableNodeMaintenance("", "")
agent: support passing ?reason= for custom notes field values on maintenance checks
2015-01-21 20:21:57 +00:00
// Make sure the check was registered with the default note
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
check = requireCheckExists(t, a, structs.NodeMaint)
agent: use const for default maintenance reason strings
2015-01-21 22:45:09 +00:00
if check.Notes != defaultNodeMaintReason {
agent: support passing ?reason= for custom notes field values on maintenance checks
2015-01-21 20:21:57 +00:00
t.Fatalf("bad: %#v", check)
}
agent: test agent service maintenance mode
2015-01-15 08:25:36 +00:00
}
agent: Snapshot and restore health state on reload. Fixes #693
2015-02-17 20:00:04 +00:00
func TestAgent_checkStateSnapshot(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: Snapshot and restore health state on reload. Fixes #693
2015-02-17 20:00:04 +00:00
// First register a service
svc := &structs.NodeService{
ID: "redis",
Service: "redis",
Tags: []string{"foo"},
Port: 8000,
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddService(svc, nil, false, "", ConfigSourceLocal); err != nil {
agent: Snapshot and restore health state on reload. Fixes #693
2015-02-17 20:00:04 +00:00
t.Fatalf("err: %v", err)
}
// Register a check
check1 := &structs.HealthCheck{
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
Node: a.Config.NodeName,
agent: Snapshot and restore health state on reload. Fixes #693
2015-02-17 20:00:04 +00:00
CheckID: "service:redis",
Name: "redischeck",
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
Status: api.HealthPassing,
agent: Snapshot and restore health state on reload. Fixes #693
2015-02-17 20:00:04 +00:00
ServiceID: "redis",
ServiceName: "redis",
}
[Security] Add finer control over script checks (#4715) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording
2018-10-11 12:22:11 +00:00
if err := a.AddCheck(check1, nil, true, "", ConfigSourceLocal); err != nil {
agent: Snapshot and restore health state on reload. Fixes #693
2015-02-17 20:00:04 +00:00
t.Fatalf("err: %s", err)
}
// Snapshot the state
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
snap := a.snapshotCheckState()
agent: Snapshot and restore health state on reload. Fixes #693
2015-02-17 20:00:04 +00:00
// Unload all of the checks
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
if err := a.unloadChecks(); err != nil {
agent: Snapshot and restore health state on reload. Fixes #693
2015-02-17 20:00:04 +00:00
t.Fatalf("err: %s", err)
}
agent: avoid reverting any check updates that occur while a service is being added or the config is reloaded (#6144)
2019-07-17 19:06:50 +00:00
// Reload the checks and restore the snapshot.
if err := a.loadChecks(a.Config, snap); err != nil {
agent: Snapshot and restore health state on reload. Fixes #693
2015-02-17 20:00:04 +00:00
t.Fatalf("err: %s", err)
}
// Search for the check
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
out := requireCheckExists(t, a, check1.CheckID)
agent: Snapshot and restore health state on reload. Fixes #693
2015-02-17 20:00:04 +00:00
// Make sure state was restored
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
if out.Status != api.HealthPassing {
agent: Snapshot and restore health state on reload. Fixes #693
2015-02-17 20:00:04 +00:00
t.Fatalf("should have restored check state")
}
}
agent: warn and purge checks which cannot be restored from agent state
2015-03-11 23:13:19 +00:00
func TestAgent_loadChecks_checkFails(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: warn and purge checks which cannot be restored from agent state
2015-03-11 23:13:19 +00:00
// Persist a health check with an invalid service ID
check := &structs.HealthCheck{
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
Node: a.Config.NodeName,
agent: warn and purge checks which cannot be restored from agent state
2015-03-11 23:13:19 +00:00
CheckID: "service:redis",
Name: "redischeck",
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
Status: api.HealthPassing,
agent: warn and purge checks which cannot be restored from agent state
2015-03-11 23:13:19 +00:00
ServiceID: "nope",
}
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
if err := a.persistCheck(check, nil, ConfigSourceLocal); err != nil {
agent: warn and purge checks which cannot be restored from agent state
2015-03-11 23:13:19 +00:00
t.Fatalf("err: %s", err)
}
// Check to make sure the check was persisted
Float a type balloon. Some strings are square pegs in round holes. This experiment was brought about because of variable naming confusion where name and checkIDs were interchanged. Gave CheckID an Qualified Type Name and chased downstream changes.
2016-06-06 08:53:30 +00:00
checkHash := checkIDHash(check.CheckID)
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
checkPath := filepath.Join(a.Config.DataDir, checksDir, checkHash)
agent: warn and purge checks which cannot be restored from agent state
2015-03-11 23:13:19 +00:00
if _, err := os.Stat(checkPath); err != nil {
t.Fatalf("err: %s", err)
}
// Try loading the checks from the persisted files
agent: avoid reverting any check updates that occur while a service is being added or the config is reloaded (#6144)
2019-07-17 19:06:50 +00:00
if err := a.loadChecks(a.Config, nil); err != nil {
agent: warn and purge checks which cannot be restored from agent state
2015-03-11 23:13:19 +00:00
t.Fatalf("err: %s", err)
}
// Ensure the erroneous check was purged
if _, err := os.Stat(checkPath); err == nil {
t.Fatalf("should have purged check")
}
}
agent: testing state persistence, recovery, and expiration
2015-06-05 23:45:05 +00:00
agent: check state is purged if expired
2015-06-05 23:59:41 +00:00
func TestAgent_persistCheckState(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: testing state persistence, recovery, and expiration
2015-06-05 23:45:05 +00:00
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
cid := structs.NewCheckID("check1", nil)
agent: testing state persistence, recovery, and expiration
2015-06-05 23:45:05 +00:00
// Create the TTL check to persist
Decouple the code that executes checks from the agent
2017-10-25 09:18:07 +00:00
check := &checks.CheckTTL{
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
CheckID: cid,
agent: testing state persistence, recovery, and expiration
2015-06-05 23:45:05 +00:00
TTL: 10 * time.Minute,
}
// Persist some check state for the check
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
err := a.persistCheckState(check, api.HealthCritical, "nope")
agent: testing state persistence, recovery, and expiration
2015-06-05 23:45:05 +00:00
if err != nil {
t.Fatalf("err: %s", err)
}
// Check the persisted file exists and has the content
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
file := filepath.Join(a.Config.DataDir, checkStateDir, cid.StringHash())
agent: testing state persistence, recovery, and expiration
2015-06-05 23:45:05 +00:00
buf, err := ioutil.ReadFile(file)
if err != nil {
t.Fatalf("err: %s", err)
}
// Decode the state
var p persistedCheckState
if err := json.Unmarshal(buf, &p); err != nil {
t.Fatalf("err: %s", err)
}
// Check the fields
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if p.CheckID != cid.ID {
agent: testing state persistence, recovery, and expiration
2015-06-05 23:45:05 +00:00
t.Fatalf("bad: %#v", p)
}
if p.Output != "nope" {
t.Fatalf("bad: %#v", p)
}
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
if p.Status != api.HealthCritical {
agent: testing state persistence, recovery, and expiration
2015-06-05 23:45:05 +00:00
t.Fatalf("bad: %#v", p)
}
// Check the expiration time was set
if p.Expires < time.Now().Unix() {
t.Fatalf("bad: %#v", p)
}
}
agent: use persist/load/purge convention for function names
2015-06-08 16:35:10 +00:00
func TestAgent_loadCheckState(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: testing state persistence, recovery, and expiration
2015-06-05 23:45:05 +00:00
// Create a check whose state will expire immediately
Decouple the code that executes checks from the agent
2017-10-25 09:18:07 +00:00
check := &checks.CheckTTL{
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
CheckID: structs.NewCheckID("check1", nil),
agent: testing state persistence, recovery, and expiration
2015-06-05 23:45:05 +00:00
TTL: 0,
}
// Persist the check state
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
err := a.persistCheckState(check, api.HealthPassing, "yup")
agent: testing state persistence, recovery, and expiration
2015-06-05 23:45:05 +00:00
if err != nil {
t.Fatalf("err: %s", err)
}
agent: use persist/load/purge convention for function names
2015-06-08 16:35:10 +00:00
// Try to load the state
agent: testing state persistence, recovery, and expiration
2015-06-05 23:45:05 +00:00
health := &structs.HealthCheck{
CheckID: "check1",
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
Status: api.HealthCritical,
agent: testing state persistence, recovery, and expiration
2015-06-05 23:45:05 +00:00
}
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
if err := a.loadCheckState(health); err != nil {
agent: testing state persistence, recovery, and expiration
2015-06-05 23:45:05 +00:00
t.Fatalf("err: %s", err)
}
// Should not have restored the status due to expiration
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
if health.Status != api.HealthCritical {
agent: testing state persistence, recovery, and expiration
2015-06-05 23:45:05 +00:00
t.Fatalf("bad: %#v", health)
}
if health.Output != "" {
t.Fatalf("bad: %#v", health)
}
agent: check state is purged if expired
2015-06-05 23:59:41 +00:00
// Should have purged the state
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
file := filepath.Join(a.Config.DataDir, checksDir, stringHash("check1"))
agent: check state is purged if expired
2015-06-05 23:59:41 +00:00
if _, err := os.Stat(file); !os.IsNotExist(err) {
t.Fatalf("should have purged state")
}
agent: testing state persistence, recovery, and expiration
2015-06-05 23:45:05 +00:00
// Set a TTL which will not expire before we check it
check.TTL = time.Minute
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
err = a.persistCheckState(check, api.HealthPassing, "yup")
agent: testing state persistence, recovery, and expiration
2015-06-05 23:45:05 +00:00
if err != nil {
t.Fatalf("err: %s", err)
}
agent: use persist/load/purge convention for function names
2015-06-08 16:35:10 +00:00
// Try to load
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
if err := a.loadCheckState(health); err != nil {
agent: testing state persistence, recovery, and expiration
2015-06-05 23:45:05 +00:00
t.Fatalf("err: %s", err)
}
// Should have restored
Remove duplicate constants This patch removes duplicate internal copies of constants in the structs package which are also defined in the api package. The api.KVOp type with all its values for the TXN endpoint and the api.HealthXXX constants are now used throughout the codebase. This resulted in some circular dependencies in the testutil package which have been resolved by copying code and constants and moving the WaitForLeader function into a separate testrpc package.
2017-04-19 23:00:11 +00:00
if health.Status != api.HealthPassing {
agent: testing state persistence, recovery, and expiration
2015-06-05 23:45:05 +00:00
t.Fatalf("bad: %#v", health)
}
if health.Output != "yup" {
t.Fatalf("bad: %#v", health)
}
}
agent: purge check state when checks are deregistered
2015-06-05 23:57:14 +00:00
func TestAgent_purgeCheckState(t *testing.T) {
test: run agent tests in parallel This brings down the test run from 108 sec to 15 sec. There is an occasional port conflict because of the nature the next port is chosen. So far it seems rare enough to live with it.
2017-05-21 07:54:40 +00:00
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
defer a.Shutdown()
agent: purge check state when checks are deregistered
2015-06-05 23:57:14 +00:00
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
cid := structs.NewCheckID("check1", nil)
agent: purge check state when checks are deregistered
2015-06-05 23:57:14 +00:00
// No error if the state does not exist
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.purgeCheckState(cid); err != nil {
agent: purge check state when checks are deregistered
2015-06-05 23:57:14 +00:00
t.Fatalf("err: %s", err)
}
// Persist some state to the data dir
Decouple the code that executes checks from the agent
2017-10-25 09:18:07 +00:00
check := &checks.CheckTTL{
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
CheckID: cid,
agent: purge check state when checks are deregistered
2015-06-05 23:57:14 +00:00
TTL: time.Minute,
}
agent: refactor tests for TestAgent Refactored tests that use * makeAgentXXX * makeDNSServerXXX * makeHTTPServerXXX
2017-05-21 07:11:09 +00:00
err := a.persistCheckState(check, api.HealthPassing, "yup")
agent: purge check state when checks are deregistered
2015-06-05 23:57:14 +00:00
if err != nil {
t.Fatalf("err: %s", err)
}
// Purge the check state
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
if err := a.purgeCheckState(cid); err != nil {
agent: purge check state when checks are deregistered
2015-06-05 23:57:14 +00:00
t.Fatalf("err: %s", err)
}
// Removed the file
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
file := filepath.Join(a.Config.DataDir, checkStateDir, cid.StringHash())
agent: purge check state when checks are deregistered
2015-06-05 23:57:14 +00:00
if _, err := os.Stat(file); !os.IsNotExist(err) {
t.Fatalf("should have removed file")
}
}
Makes the default protocol 2 and lets 3 interoperate with 2.
2015-10-16 02:28:31 +00:00
func TestAgent_GetCoordinate(t *testing.T) {
Update TestAgent_GetCoordinate The old test case was a very specific regresion test for a case that is no longer possible. Replaced with a new test that checks the default coordinate is returned.
2020-06-24 16:56:46 +00:00
a := NewTestAgent(t, ``)
defer a.Shutdown()
Makes the default protocol 2 and lets 3 interoperate with 2.
2015-10-16 02:28:31 +00:00
Update TestAgent_GetCoordinate The old test case was a very specific regresion test for a case that is no longer possible. Replaced with a new test that checks the default coordinate is returned.
2020-06-24 16:56:46 +00:00
coords, err := a.GetLANCoordinate()
require.NoError(t, err)
expected := lib.CoordinateSet{
"": &coordinate.Coordinate{
Error: 1.5,
Height: 1e-05,
Vec: []float64{0, 0, 0, 0, 0, 0, 0, 0},
},
Makes the default protocol 2 and lets 3 interoperate with 2.
2015-10-16 02:28:31 +00:00
}
Update TestAgent_GetCoordinate The old test case was a very specific regresion test for a case that is no longer possible. Replaced with a new test that checks the default coordinate is returned.
2020-06-24 16:56:46 +00:00
require.Equal(t, expected, coords)
Makes the default protocol 2 and lets 3 interoperate with 2.
2015-10-16 02:28:31 +00:00
}
Fix watch error when http & https are disabled (#3493) Remove an error in watch reloading that happens when http and https are both disabled, and use an https address for running watches if no http addresses are present. Fixes #3425.
2017-09-26 20:47:27 +00:00
func TestAgent_reloadWatches(t *testing.T) {
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
Fix watch error when http & https are disabled (#3493) Remove an error in watch reloading that happens when http and https are both disabled, and use an https address for running watches if no http addresses are present. Fixes #3425.
2017-09-26 20:47:27 +00:00
defer a.Shutdown()
// Normal watch with http addr set, should succeed
newConf := *a.config
newConf.Watches = []map[string]interface{}{
{
Clean up subprocess handling and make shell use optional (#3509) * Clean up handling of subprocesses and make using a shell optional * Update docs for subprocess changes * Fix tests for new subprocess behavior * More cleanup of subprocesses * Minor adjustments and cleanup for subprocess logic * Makes the watch handler reload test use the new path. * Adds check tests for new args path, and updates existing tests to use new path. * Adds support for script args in Docker checks. * Fixes the sanitize unit test. * Adds panic for unknown watch type, and reverts back to Run(). * Adds shell option back to consul lock command. * Adds shell option back to consul exec command. * Adds shell back into consul watch command. * Refactors signal forwarding and makes Windows-friendly. * Adds a clarifying comment. * Changes error wording to a warning. * Scopes signals to interrupt and kill. This avoids us trying to send SIGCHILD to the dead process. * Adds an error for shell=false for consul exec. * Adds notes about the deprecated script and handler fields. * De-nests an if statement.
2017-10-04 23:48:00 +00:00
"type": "key",
"key": "asdf",
"args": []interface{}{"ls"},
Fix watch error when http & https are disabled (#3493) Remove an error in watch reloading that happens when http and https are both disabled, and use an https address for running watches if no http addresses are present. Fixes #3425.
2017-09-26 20:47:27 +00:00
},
}
if err := a.reloadWatches(&newConf); err != nil {
t.Fatalf("bad: %s", err)
}
Don't allow connect watches in agent/cli yet
2018-04-26 17:06:26 +00:00
// Should fail to reload with connect watches
newConf.Watches = []map[string]interface{}{
{
"type": "connect_roots",
"key": "asdf",
"args": []interface{}{"ls"},
},
}
if err := a.reloadWatches(&newConf); err == nil || !strings.Contains(err.Error(), "not allowed in agent config") {
t.Fatalf("bad: %s", err)
}
Fix watch error when http & https are disabled (#3493) Remove an error in watch reloading that happens when http and https are both disabled, and use an https address for running watches if no http addresses are present. Fixes #3425.
2017-09-26 20:47:27 +00:00
// Should still succeed with only HTTPS addresses
newConf.HTTPSAddrs = newConf.HTTPAddrs
newConf.HTTPAddrs = make([]net.Addr, 0)
newConf.Watches = []map[string]interface{}{
{
Clean up subprocess handling and make shell use optional (#3509) * Clean up handling of subprocesses and make using a shell optional * Update docs for subprocess changes * Fix tests for new subprocess behavior * More cleanup of subprocesses * Minor adjustments and cleanup for subprocess logic * Makes the watch handler reload test use the new path. * Adds check tests for new args path, and updates existing tests to use new path. * Adds support for script args in Docker checks. * Fixes the sanitize unit test. * Adds panic for unknown watch type, and reverts back to Run(). * Adds shell option back to consul lock command. * Adds shell option back to consul exec command. * Adds shell back into consul watch command. * Refactors signal forwarding and makes Windows-friendly. * Adds a clarifying comment. * Changes error wording to a warning. * Scopes signals to interrupt and kill. This avoids us trying to send SIGCHILD to the dead process. * Adds an error for shell=false for consul exec. * Adds notes about the deprecated script and handler fields. * De-nests an if statement.
2017-10-04 23:48:00 +00:00
"type": "key",
"key": "asdf",
"args": []interface{}{"ls"},
Fix watch error when http & https are disabled (#3493) Remove an error in watch reloading that happens when http and https are both disabled, and use an https address for running watches if no http addresses are present. Fixes #3425.
2017-09-26 20:47:27 +00:00
},
}
if err := a.reloadWatches(&newConf); err != nil {
t.Fatalf("bad: %s", err)
}
// Should fail to reload with no http or https addrs
newConf.HTTPSAddrs = make([]net.Addr, 0)
newConf.Watches = []map[string]interface{}{
{
Clean up subprocess handling and make shell use optional (#3509) * Clean up handling of subprocesses and make using a shell optional * Update docs for subprocess changes * Fix tests for new subprocess behavior * More cleanup of subprocesses * Minor adjustments and cleanup for subprocess logic * Makes the watch handler reload test use the new path. * Adds check tests for new args path, and updates existing tests to use new path. * Adds support for script args in Docker checks. * Fixes the sanitize unit test. * Adds panic for unknown watch type, and reverts back to Run(). * Adds shell option back to consul lock command. * Adds shell option back to consul exec command. * Adds shell back into consul watch command. * Refactors signal forwarding and makes Windows-friendly. * Adds a clarifying comment. * Changes error wording to a warning. * Scopes signals to interrupt and kill. This avoids us trying to send SIGCHILD to the dead process. * Adds an error for shell=false for consul exec. * Adds notes about the deprecated script and handler fields. * De-nests an if statement.
2017-10-04 23:48:00 +00:00
"type": "key",
"key": "asdf",
"args": []interface{}{"ls"},
Fix watch error when http & https are disabled (#3493) Remove an error in watch reloading that happens when http and https are both disabled, and use an https address for running watches if no http addresses are present. Fixes #3425.
2017-09-26 20:47:27 +00:00
},
}
if err := a.reloadWatches(&newConf); err == nil || !strings.Contains(err.Error(), "watch plans require an HTTP or HTTPS endpoint") {
t.Fatalf("bad: %s", err)
}
}
Update unit tests to reflect change to func signature
2018-05-31 21:20:16 +00:00
func TestAgent_reloadWatchesHTTPS(t *testing.T) {
t.Parallel()
Remove t.Name() from TestAgent.Name And re-add the name to the logger so that log messages from different agents in a single can be identified.
2020-03-30 20:05:27 +00:00
a := TestAgent{UseTLS: true}
if err := a.Start(t); err != nil {
refactor & add better retry logic to NewTestAgent (#6363) Fixes #6361
2019-09-03 22:05:51 +00:00
t.Fatal(err)
}
Update unit tests to reflect change to func signature
2018-05-31 21:20:16 +00:00
defer a.Shutdown()
// Normal watch with http addr set, should succeed
newConf := *a.config
newConf.Watches = []map[string]interface{}{
{
"type": "key",
"key": "asdf",
"args": []interface{}{"ls"},
},
}
if err := a.reloadWatches(&newConf); err != nil {
t.Fatalf("bad: %s", err)
}
}
Added connect proxy config and local agent state setup on boot.
2018-04-16 15:00:20 +00:00
ACL Token Persistence and Reloading (#5328) This PR adds two features which will be useful for operators when ACLs are in use. 1. Tokens set in configuration files are now reloadable. 2. If `acl.enable_token_persistence` is set to `true` in the configuration, tokens set via the `v1/agent/token` endpoint are now persisted to disk and loaded when the agent starts (or during configuration reload) Note that token persistence is opt-in so our users who do not want tokens on the local disk will see no change. Some other secondary changes: * Refactored a bunch of places where the replication token is retrieved from the token store. This token isn't just for replicating ACLs and now it is named accordingly. * Allowed better paths in the `v1/agent/token/` API. Instead of paths like: `v1/agent/token/acl_replication_token` the path can now be just `v1/agent/token/replication`. The old paths remain to be valid. * Added a couple new API functions to set tokens via the new paths. Deprecated the old ones and pointed to the new names. The names are also generally better and don't imply that what you are setting is for ACLs but rather are setting ACL tokens. There is a minor semantic difference there especially for the replication token as again, its no longer used only for ACL token/policy replication. The new functions will detect 404s and fallback to using the older token paths when talking to pre-1.4.3 agents. * Docs updated to reflect the API additions and to show using the new endpoints. * Updated the ACL CLI set-agent-tokens command to use the non-deprecated APIs.
2019-02-27 19:28:31 +00:00
func TestAgent_loadTokens(t *testing.T) {
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, `
ACL Token Persistence and Reloading (#5328) This PR adds two features which will be useful for operators when ACLs are in use. 1. Tokens set in configuration files are now reloadable. 2. If `acl.enable_token_persistence` is set to `true` in the configuration, tokens set via the `v1/agent/token` endpoint are now persisted to disk and loaded when the agent starts (or during configuration reload) Note that token persistence is opt-in so our users who do not want tokens on the local disk will see no change. Some other secondary changes: * Refactored a bunch of places where the replication token is retrieved from the token store. This token isn't just for replicating ACLs and now it is named accordingly. * Allowed better paths in the `v1/agent/token/` API. Instead of paths like: `v1/agent/token/acl_replication_token` the path can now be just `v1/agent/token/replication`. The old paths remain to be valid. * Added a couple new API functions to set tokens via the new paths. Deprecated the old ones and pointed to the new names. The names are also generally better and don't imply that what you are setting is for ACLs but rather are setting ACL tokens. There is a minor semantic difference there especially for the replication token as again, its no longer used only for ACL token/policy replication. The new functions will detect 404s and fallback to using the older token paths when talking to pre-1.4.3 agents. * Docs updated to reflect the API additions and to show using the new endpoints. * Updated the ACL CLI set-agent-tokens command to use the non-deprecated APIs.
2019-02-27 19:28:31 +00:00
acl = {
enabled = true
tokens = {
agent = "alfa"
agent_master = "bravo",
default = "charlie"
replication = "delta"
}
}
`)
defer a.Shutdown()
require := require.New(t)
tokensFullPath := filepath.Join(a.config.DataDir, tokensPath)
t.Run("original-configuration", func(t *testing.T) {
require.Equal("alfa", a.tokens.AgentToken())
require.Equal("bravo", a.tokens.AgentMasterToken())
require.Equal("charlie", a.tokens.UserToken())
require.Equal("delta", a.tokens.ReplicationToken())
})
t.Run("updated-configuration", func(t *testing.T) {
cfg := &config.RuntimeConfig{
ACLToken: "echo",
ACLAgentToken: "foxtrot",
ACLAgentMasterToken: "golf",
ACLReplicationToken: "hotel",
}
// ensures no error for missing persisted tokens file
require.NoError(a.loadTokens(cfg))
require.Equal("echo", a.tokens.UserToken())
require.Equal("foxtrot", a.tokens.AgentToken())
require.Equal("golf", a.tokens.AgentMasterToken())
require.Equal("hotel", a.tokens.ReplicationToken())
})
t.Run("persisted-tokens", func(t *testing.T) {
cfg := &config.RuntimeConfig{
ACLToken: "echo",
ACLAgentToken: "foxtrot",
ACLAgentMasterToken: "golf",
ACLReplicationToken: "hotel",
}
tokens := `{
"agent" : "india",
"agent_master" : "juliett",
"default": "kilo",
"replication" : "lima"
}`
require.NoError(ioutil.WriteFile(tokensFullPath, []byte(tokens), 0600))
require.NoError(a.loadTokens(cfg))
// no updates since token persistence is not enabled
require.Equal("echo", a.tokens.UserToken())
require.Equal("foxtrot", a.tokens.AgentToken())
require.Equal("golf", a.tokens.AgentMasterToken())
require.Equal("hotel", a.tokens.ReplicationToken())
a.config.ACLEnableTokenPersistence = true
require.NoError(a.loadTokens(cfg))
require.Equal("india", a.tokens.AgentToken())
require.Equal("juliett", a.tokens.AgentMasterToken())
require.Equal("kilo", a.tokens.UserToken())
require.Equal("lima", a.tokens.ReplicationToken())
})
t.Run("persisted-tokens-override", func(t *testing.T) {
tokens := `{
"agent" : "mike",
"agent_master" : "november",
"default": "oscar",
"replication" : "papa"
}`
cfg := &config.RuntimeConfig{
ACLToken: "quebec",
ACLAgentToken: "romeo",
ACLAgentMasterToken: "sierra",
ACLReplicationToken: "tango",
}
require.NoError(ioutil.WriteFile(tokensFullPath, []byte(tokens), 0600))
require.NoError(a.loadTokens(cfg))
require.Equal("mike", a.tokens.AgentToken())
require.Equal("november", a.tokens.AgentMasterToken())
require.Equal("oscar", a.tokens.UserToken())
require.Equal("papa", a.tokens.ReplicationToken())
})
t.Run("partial-persisted", func(t *testing.T) {
tokens := `{
"agent" : "uniform",
"agent_master" : "victor"
}`
cfg := &config.RuntimeConfig{
ACLToken: "whiskey",
ACLAgentToken: "xray",
ACLAgentMasterToken: "yankee",
ACLReplicationToken: "zulu",
}
require.NoError(ioutil.WriteFile(tokensFullPath, []byte(tokens), 0600))
require.NoError(a.loadTokens(cfg))
require.Equal("uniform", a.tokens.AgentToken())
require.Equal("victor", a.tokens.AgentMasterToken())
require.Equal("whiskey", a.tokens.UserToken())
require.Equal("zulu", a.tokens.ReplicationToken())
})
t.Run("persistence-error-not-json", func(t *testing.T) {
cfg := &config.RuntimeConfig{
ACLToken: "one",
ACLAgentToken: "two",
ACLAgentMasterToken: "three",
ACLReplicationToken: "four",
}
require.NoError(ioutil.WriteFile(tokensFullPath, []byte{0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08}, 0600))
err := a.loadTokens(cfg)
require.Error(err)
require.Equal("one", a.tokens.UserToken())
require.Equal("two", a.tokens.AgentToken())
require.Equal("three", a.tokens.AgentMasterToken())
require.Equal("four", a.tokens.ReplicationToken())
})
t.Run("persistence-error-wrong-top-level", func(t *testing.T) {
cfg := &config.RuntimeConfig{
ACLToken: "alfa",
ACLAgentToken: "bravo",
ACLAgentMasterToken: "charlie",
ACLReplicationToken: "foxtrot",
}
require.NoError(ioutil.WriteFile(tokensFullPath, []byte("[1,2,3]"), 0600))
err := a.loadTokens(cfg)
require.Error(err)
require.Equal("alfa", a.tokens.UserToken())
require.Equal("bravo", a.tokens.AgentToken())
require.Equal("charlie", a.tokens.AgentMasterToken())
require.Equal("foxtrot", a.tokens.ReplicationToken())
})
}
agent: enable reloading of tls config (#5419) This PR introduces reloading tls configuration. Consul will now be able to reload the TLS configuration which previously required a restart. It is not yet possible to turn TLS ON or OFF with these changes. Only when TLS is already turned on, the configuration can be reloaded. Most importantly the certificates and CAs.
2019-03-13 09:29:06 +00:00
agent: show warning when enable_script_checks is enabled without safty net (#7437) In order to enforce a bit security on Consul agents, add a new method in agent to highlight possible security issues. This does not return an error for now, but might in the future. For now, it detects issues such as: https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ This would display this kind of messages: ``` 2020-03-11T18:27:49.873+0100 [ERROR] agent: [SECURITY] issue: error="using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/" ```
2020-04-02 07:59:23 +00:00
func TestAgent_SecurityChecks(t *testing.T) {
t.Parallel()
hcl := `
enable_script_checks = true
`
a := &TestAgent{Name: t.Name(), HCL: hcl}
defer a.Shutdown()
data := make([]byte, 0, 8192)
bytesBuffer := bytes.NewBuffer(data)
a.LogOutput = bytesBuffer
assert.NoError(t, a.Start(t))
assert.Contains(t, bytesBuffer.String(), "using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS")
}
agent: enable reloading of tls config (#5419) This PR introduces reloading tls configuration. Consul will now be able to reload the TLS configuration which previously required a restart. It is not yet possible to turn TLS ON or OFF with these changes. Only when TLS is already turned on, the configuration can be reloaded. Most importantly the certificates and CAs.
2019-03-13 09:29:06 +00:00
func TestAgent_ReloadConfigOutgoingRPCConfig(t *testing.T) {
t.Parallel()
dataDir := testutil.TempDir(t, "agent") // we manage the data dir
defer os.RemoveAll(dataDir)
hcl := `
data_dir = "` + dataDir + `"
verify_outgoing = true
ca_file = "../test/ca/root.cer"
cert_file = "../test/key/ourdomain.cer"
key_file = "../test/key/ourdomain.key"
verify_server_hostname = false
`
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, hcl)
agent: only use TestAgent when appropriate (#5502)
2019-03-18 16:06:16 +00:00
defer a.Shutdown()
agent: enable reloading of tls config (#5419) This PR introduces reloading tls configuration. Consul will now be able to reload the TLS configuration which previously required a restart. It is not yet possible to turn TLS ON or OFF with these changes. Only when TLS is already turned on, the configuration can be reloaded. Most importantly the certificates and CAs.
2019-03-13 09:29:06 +00:00
tlsConf := a.tlsConfigurator.OutgoingRPCConfig()
require.True(t, tlsConf.InsecureSkipVerify)
require.Len(t, tlsConf.ClientCAs.Subjects(), 1)
require.Len(t, tlsConf.RootCAs.Subjects(), 1)
hcl = `
data_dir = "` + dataDir + `"
verify_outgoing = true
ca_path = "../test/ca_path"
cert_file = "../test/key/ourdomain.cer"
key_file = "../test/key/ourdomain.key"
verify_server_hostname = true
`
Allow users to configure either unstructured or JSON logging (#7130) * hclog Allow users to choose between unstructured and JSON logging
2020-01-28 23:50:41 +00:00
c := TestConfig(testutil.Logger(t), config.Source{Name: t.Name(), Format: "hcl", Data: hcl})
Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.
2020-06-10 20:47:35 +00:00
require.NoError(t, a.reloadConfigInternal(c))
agent: enable reloading of tls config (#5419) This PR introduces reloading tls configuration. Consul will now be able to reload the TLS configuration which previously required a restart. It is not yet possible to turn TLS ON or OFF with these changes. Only when TLS is already turned on, the configuration can be reloaded. Most importantly the certificates and CAs.
2019-03-13 09:29:06 +00:00
tlsConf = a.tlsConfigurator.OutgoingRPCConfig()
require.False(t, tlsConf.InsecureSkipVerify)
require.Len(t, tlsConf.RootCAs.Subjects(), 2)
require.Len(t, tlsConf.ClientCAs.Subjects(), 2)
}
agent: configuration reload preserves check's statuses for services (#7345) This fixes issue #7318 Between versions 1.5.2 and 1.5.3, a regression has been introduced regarding health of services. A patch #6144 had been issued for HealthChecks of nodes, but not for healthchecks of services. What happened when a reload was: 1. save all healthcheck statuses 2. cleanup everything 3. add new services with healthchecks In step 3, the state of healthchecks was taken into account locally, so at step 3, but since we cleaned up at step 2, state was lost. This PR introduces the snap parameter, so step 3 can use information from step 1
2020-03-09 11:59:41 +00:00
func TestAgent_ReloadConfigAndKeepChecksStatus(t *testing.T) {
t.Parallel()
dataDir := testutil.TempDir(t, "agent") // we manage the data dir
defer os.RemoveAll(dataDir)
hcl := `data_dir = "` + dataDir + `"
enable_local_script_checks=true
services=[{
name="webserver1",
Fix flakey health check reload test (#7490) This test would occasionally fail because we checked for a status of "critical" initially. This races with the actual healthcheck being run and declared passing. We instead use a ttl health check so that we don't rely on timing at all.
2020-03-25 14:09:13 +00:00
check{id="check1", ttl="30s"}
}]`
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, hcl)
agent: configuration reload preserves check's statuses for services (#7345) This fixes issue #7318 Between versions 1.5.2 and 1.5.3, a regression has been introduced regarding health of services. A patch #6144 had been issued for HealthChecks of nodes, but not for healthchecks of services. What happened when a reload was: 1. save all healthcheck statuses 2. cleanup everything 3. add new services with healthchecks In step 3, the state of healthchecks was taken into account locally, so at step 3, but since we cleaned up at step 2, state was lost. This PR introduces the snap parameter, so step 3 can use information from step 1
2020-03-09 11:59:41 +00:00
defer a.Shutdown()
Fix flakey health check reload test (#7490) This test would occasionally fail because we checked for a status of "critical" initially. This races with the actual healthcheck being run and declared passing. We instead use a ttl health check so that we don't rely on timing at all.
2020-03-25 14:09:13 +00:00
require.NoError(t, a.updateTTLCheck(structs.NewCheckID("check1", nil), api.HealthPassing, "testing agent reload"))
agent: configuration reload preserves check's statuses for services (#7345) This fixes issue #7318 Between versions 1.5.2 and 1.5.3, a regression has been introduced regarding health of services. A patch #6144 had been issued for HealthChecks of nodes, but not for healthchecks of services. What happened when a reload was: 1. save all healthcheck statuses 2. cleanup everything 3. add new services with healthchecks In step 3, the state of healthchecks was taken into account locally, so at step 3, but since we cleaned up at step 2, state was lost. This PR introduces the snap parameter, so step 3 can use information from step 1
2020-03-09 11:59:41 +00:00
Fix flakey health check reload test (#7490) This test would occasionally fail because we checked for a status of "critical" initially. This races with the actual healthcheck being run and declared passing. We instead use a ttl health check so that we don't rely on timing at all.
2020-03-25 14:09:13 +00:00
// Make sure check is passing before we reload.
gotChecks := a.State.Checks(nil)
require.Equal(t, 1, len(gotChecks), "Should have a check registered, but had %#v", gotChecks)
for id, check := range gotChecks {
agent: configuration reload preserves check's statuses for services (#7345) This fixes issue #7318 Between versions 1.5.2 and 1.5.3, a regression has been introduced regarding health of services. A patch #6144 had been issued for HealthChecks of nodes, but not for healthchecks of services. What happened when a reload was: 1. save all healthcheck statuses 2. cleanup everything 3. add new services with healthchecks In step 3, the state of healthchecks was taken into account locally, so at step 3, but since we cleaned up at step 2, state was lost. This PR introduces the snap parameter, so step 3 can use information from step 1
2020-03-09 11:59:41 +00:00
require.Equal(t, "passing", check.Status, "check %q is wrong", id)
}
Fix flakey health check reload test (#7490) This test would occasionally fail because we checked for a status of "critical" initially. This races with the actual healthcheck being run and declared passing. We instead use a ttl health check so that we don't rely on timing at all.
2020-03-25 14:09:13 +00:00
c := TestConfig(testutil.Logger(t), config.Source{Name: t.Name(), Format: "hcl", Data: hcl})
Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.
2020-06-10 20:47:35 +00:00
require.NoError(t, a.reloadConfigInternal(c))
Fix flakey health check reload test (#7490) This test would occasionally fail because we checked for a status of "critical" initially. This races with the actual healthcheck being run and declared passing. We instead use a ttl health check so that we don't rely on timing at all.
2020-03-25 14:09:13 +00:00
// After reload, should be passing directly (no critical state)
agent: configuration reload preserves check's statuses for services (#7345) This fixes issue #7318 Between versions 1.5.2 and 1.5.3, a regression has been introduced regarding health of services. A patch #6144 had been issued for HealthChecks of nodes, but not for healthchecks of services. What happened when a reload was: 1. save all healthcheck statuses 2. cleanup everything 3. add new services with healthchecks In step 3, the state of healthchecks was taken into account locally, so at step 3, but since we cleaned up at step 2, state was lost. This PR introduces the snap parameter, so step 3 can use information from step 1
2020-03-09 11:59:41 +00:00
for id, check := range a.State.Checks(nil) {
require.Equal(t, "passing", check.Status, "check %q is wrong", id)
}
}
agent: enable reloading of tls config (#5419) This PR introduces reloading tls configuration. Consul will now be able to reload the TLS configuration which previously required a restart. It is not yet possible to turn TLS ON or OFF with these changes. Only when TLS is already turned on, the configuration can be reloaded. Most importantly the certificates and CAs.
2019-03-13 09:29:06 +00:00
func TestAgent_ReloadConfigIncomingRPCConfig(t *testing.T) {
t.Parallel()
dataDir := testutil.TempDir(t, "agent") // we manage the data dir
defer os.RemoveAll(dataDir)
hcl := `
data_dir = "` + dataDir + `"
verify_outgoing = true
ca_file = "../test/ca/root.cer"
cert_file = "../test/key/ourdomain.cer"
key_file = "../test/key/ourdomain.key"
verify_server_hostname = false
`
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, hcl)
agent: only use TestAgent when appropriate (#5502)
2019-03-18 16:06:16 +00:00
defer a.Shutdown()
agent: enable reloading of tls config (#5419) This PR introduces reloading tls configuration. Consul will now be able to reload the TLS configuration which previously required a restart. It is not yet possible to turn TLS ON or OFF with these changes. Only when TLS is already turned on, the configuration can be reloaded. Most importantly the certificates and CAs.
2019-03-13 09:29:06 +00:00
tlsConf := a.tlsConfigurator.IncomingRPCConfig()
require.NotNil(t, tlsConf.GetConfigForClient)
agent: only use TestAgent when appropriate (#5502)
2019-03-18 16:06:16 +00:00
tlsConf, err := tlsConf.GetConfigForClient(nil)
agent: enable reloading of tls config (#5419) This PR introduces reloading tls configuration. Consul will now be able to reload the TLS configuration which previously required a restart. It is not yet possible to turn TLS ON or OFF with these changes. Only when TLS is already turned on, the configuration can be reloaded. Most importantly the certificates and CAs.
2019-03-13 09:29:06 +00:00
require.NoError(t, err)
require.NotNil(t, tlsConf)
require.True(t, tlsConf.InsecureSkipVerify)
require.Len(t, tlsConf.ClientCAs.Subjects(), 1)
require.Len(t, tlsConf.RootCAs.Subjects(), 1)
hcl = `
data_dir = "` + dataDir + `"
verify_outgoing = true
ca_path = "../test/ca_path"
cert_file = "../test/key/ourdomain.cer"
key_file = "../test/key/ourdomain.key"
verify_server_hostname = true
`
Allow users to configure either unstructured or JSON logging (#7130) * hclog Allow users to choose between unstructured and JSON logging
2020-01-28 23:50:41 +00:00
c := TestConfig(testutil.Logger(t), config.Source{Name: t.Name(), Format: "hcl", Data: hcl})
Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.
2020-06-10 20:47:35 +00:00
require.NoError(t, a.reloadConfigInternal(c))
agent: enable reloading of tls config (#5419) This PR introduces reloading tls configuration. Consul will now be able to reload the TLS configuration which previously required a restart. It is not yet possible to turn TLS ON or OFF with these changes. Only when TLS is already turned on, the configuration can be reloaded. Most importantly the certificates and CAs.
2019-03-13 09:29:06 +00:00
tlsConf, err = tlsConf.GetConfigForClient(nil)
require.NoError(t, err)
require.False(t, tlsConf.InsecureSkipVerify)
require.Len(t, tlsConf.ClientCAs.Subjects(), 2)
require.Len(t, tlsConf.RootCAs.Subjects(), 2)
}
func TestAgent_ReloadConfigTLSConfigFailure(t *testing.T) {
t.Parallel()
dataDir := testutil.TempDir(t, "agent") // we manage the data dir
defer os.RemoveAll(dataDir)
hcl := `
data_dir = "` + dataDir + `"
verify_outgoing = true
ca_file = "../test/ca/root.cer"
cert_file = "../test/key/ourdomain.cer"
key_file = "../test/key/ourdomain.key"
verify_server_hostname = false
`
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, hcl)
agent: only use TestAgent when appropriate (#5502)
2019-03-18 16:06:16 +00:00
defer a.Shutdown()
agent: enable reloading of tls config (#5419) This PR introduces reloading tls configuration. Consul will now be able to reload the TLS configuration which previously required a restart. It is not yet possible to turn TLS ON or OFF with these changes. Only when TLS is already turned on, the configuration can be reloaded. Most importantly the certificates and CAs.
2019-03-13 09:29:06 +00:00
tlsConf := a.tlsConfigurator.IncomingRPCConfig()
hcl = `
data_dir = "` + dataDir + `"
verify_incoming = true
`
Allow users to configure either unstructured or JSON logging (#7130) * hclog Allow users to choose between unstructured and JSON logging
2020-01-28 23:50:41 +00:00
c := TestConfig(testutil.Logger(t), config.Source{Name: t.Name(), Format: "hcl", Data: hcl})
Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.
2020-06-10 20:47:35 +00:00
require.Error(t, a.reloadConfigInternal(c))
agent: only use TestAgent when appropriate (#5502)
2019-03-18 16:06:16 +00:00
tlsConf, err := tlsConf.GetConfigForClient(nil)
agent: enable reloading of tls config (#5419) This PR introduces reloading tls configuration. Consul will now be able to reload the TLS configuration which previously required a restart. It is not yet possible to turn TLS ON or OFF with these changes. Only when TLS is already turned on, the configuration can be reloaded. Most importantly the certificates and CAs.
2019-03-13 09:29:06 +00:00
require.NoError(t, err)
require.Equal(t, tls.NoClientCert, tlsConf.ClientAuth)
require.Len(t, tlsConf.ClientCAs.Subjects(), 1)
require.Len(t, tlsConf.RootCAs.Subjects(), 1)
}
tls: auto_encrypt enables automatic RPC cert provisioning for consul clients (#5597)
2019-06-27 20:22:07 +00:00
Allow raft TrailingLogs to be configured. (#6186) This fixes pathological cases where the write throughput and snapshot size are both so large that more than 10k log entries are written in the time it takes to restore the snapshot from disk. In this case followers that restart can never catch up with leader replication again and enter a loop of constantly downloading a full snapshot and restoring it only to find that snapshot is already out of date and the leader has truncated its logs so a new snapshot is sent etc. In general if you need to adjust this, you are probably abusing Consul for purposes outside its design envelope and should reconsider your usage to reduce data size and/or write volume.
2019-07-23 14:19:57 +00:00
func TestAgent_consulConfig_AutoEncryptAllowTLS(t *testing.T) {
tls: auto_encrypt enables automatic RPC cert provisioning for consul clients (#5597)
2019-06-27 20:22:07 +00:00
t.Parallel()
dataDir := testutil.TempDir(t, "agent") // we manage the data dir
defer os.RemoveAll(dataDir)
hcl := `
data_dir = "` + dataDir + `"
verify_incoming = true
ca_file = "../test/ca/root.cer"
cert_file = "../test/key/ourdomain.cer"
key_file = "../test/key/ourdomain.key"
auto_encrypt { allow_tls = true }
`
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, hcl)
tls: auto_encrypt enables automatic RPC cert provisioning for consul clients (#5597)
2019-06-27 20:22:07 +00:00
defer a.Shutdown()
require.True(t, a.consulConfig().AutoEncryptAllowTLS)
}
Allow raft TrailingLogs to be configured. (#6186) This fixes pathological cases where the write throughput and snapshot size are both so large that more than 10k log entries are written in the time it takes to restore the snapshot from disk. In this case followers that restart can never catch up with leader replication again and enter a loop of constantly downloading a full snapshot and restoring it only to find that snapshot is already out of date and the leader has truncated its logs so a new snapshot is sent etc. In general if you need to adjust this, you are probably abusing Consul for purposes outside its design envelope and should reconsider your usage to reduce data size and/or write volume.
2019-07-23 14:19:57 +00:00
func TestAgent_consulConfig_RaftTrailingLogs(t *testing.T) {
t.Parallel()
hcl := `
raft_trailing_logs = 812345
`
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, hcl)
Allow raft TrailingLogs to be configured. (#6186) This fixes pathological cases where the write throughput and snapshot size are both so large that more than 10k log entries are written in the time it takes to restore the snapshot from disk. In this case followers that restart can never catch up with leader replication again and enter a loop of constantly downloading a full snapshot and restoring it only to find that snapshot is already out of date and the leader has truncated its logs so a new snapshot is sent etc. In general if you need to adjust this, you are probably abusing Consul for purposes outside its design envelope and should reconsider your usage to reduce data size and/or write volume.
2019-07-23 14:19:57 +00:00
defer a.Shutdown()
require.Equal(t, uint64(812345), a.consulConfig().RaftConfig.TrailingLogs)
}
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
Expose HTTP-based paths through Connect proxy (#6446) Fixes: #5396 This PR adds a proxy configuration stanza called expose. These flags register listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only listening on the loopback interface, while still accepting traffic from non Connect-enabled services. Under expose there is a boolean checks flag that would automatically expose all registered HTTP and gRPC check paths. This stanza also accepts a paths list to expose individual paths. The primary use case for this functionality would be to expose paths for third parties like Prometheus or the kubelet. Listeners for requests to exposed paths are be configured dynamically at run time. Any time a proxy, or check can be registered, a listener can also be created. In this initial implementation requests to these paths are not authenticated/encrypted.
2019-09-26 02:55:52 +00:00
func TestAgent_grpcInjectAddr(t *testing.T) {
tt := []struct {
name string
grpc string
ip string
port int
want string
}{
{
name: "localhost web svc",
grpc: "localhost:8080/web",
ip: "192.168.0.0",
port: 9090,
want: "192.168.0.0:9090/web",
},
{
name: "localhost no svc",
grpc: "localhost:8080",
ip: "192.168.0.0",
port: 9090,
want: "192.168.0.0:9090",
},
{
name: "ipv4 web svc",
grpc: "127.0.0.1:8080/web",
ip: "192.168.0.0",
port: 9090,
want: "192.168.0.0:9090/web",
},
{
name: "ipv4 no svc",
grpc: "127.0.0.1:8080",
ip: "192.168.0.0",
port: 9090,
want: "192.168.0.0:9090",
},
{
name: "ipv6 no svc",
grpc: "2001:db8:1f70::999:de8:7648:6e8:5000",
ip: "192.168.0.0",
port: 9090,
want: "192.168.0.0:9090",
},
{
name: "ipv6 web svc",
grpc: "2001:db8:1f70::999:de8:7648:6e8:5000/web",
ip: "192.168.0.0",
port: 9090,
want: "192.168.0.0:9090/web",
},
{
name: "zone ipv6 web svc",
grpc: "::FFFF:C0A8:1%1:5000/web",
ip: "192.168.0.0",
port: 9090,
want: "192.168.0.0:9090/web",
},
{
name: "ipv6 literal web svc",
grpc: "::FFFF:192.168.0.1:5000/web",
ip: "192.168.0.0",
port: 9090,
want: "192.168.0.0:9090/web",
},
{
name: "ipv6 injected into ipv6 url",
grpc: "2001:db8:1f70::999:de8:7648:6e8:5000",
ip: "::FFFF:C0A8:1",
port: 9090,
want: "::FFFF:C0A8:1:9090",
},
{
name: "ipv6 injected into ipv6 url with svc",
grpc: "2001:db8:1f70::999:de8:7648:6e8:5000/web",
ip: "::FFFF:C0A8:1",
port: 9090,
want: "::FFFF:C0A8:1:9090/web",
},
{
name: "ipv6 injected into ipv6 url with special",
grpc: "2001:db8:1f70::999:de8:7648:6e8:5000/service-$name:with@special:Chars",
ip: "::FFFF:C0A8:1",
port: 9090,
want: "::FFFF:C0A8:1:9090/service-$name:with@special:Chars",
},
}
for _, tt := range tt {
t.Run(tt.name, func(t *testing.T) {
got := grpcInjectAddr(tt.grpc, tt.ip, tt.port)
if got != tt.want {
t.Errorf("httpInjectAddr() got = %v, want %v", got, tt.want)
}
})
}
}
func TestAgent_httpInjectAddr(t *testing.T) {
tt := []struct {
name string
url string
ip string
port int
want string
}{
{
name: "localhost health",
url: "http://localhost:8080/health",
ip: "192.168.0.0",
port: 9090,
want: "http://192.168.0.0:9090/health",
},
{
name: "https localhost health",
url: "https://localhost:8080/health",
ip: "192.168.0.0",
port: 9090,
want: "https://192.168.0.0:9090/health",
},
{
name: "https ipv4 health",
url: "https://127.0.0.1:8080/health",
ip: "192.168.0.0",
port: 9090,
want: "https://192.168.0.0:9090/health",
},
{
name: "https ipv4 without path",
url: "https://127.0.0.1:8080",
ip: "192.168.0.0",
port: 9090,
want: "https://192.168.0.0:9090",
},
{
name: "https ipv6 health",
url: "https://[2001:db8:1f70::999:de8:7648:6e8]:5000/health",
ip: "192.168.0.0",
port: 9090,
want: "https://192.168.0.0:9090/health",
},
{
name: "https ipv6 with zone",
url: "https://[::FFFF:C0A8:1%1]:5000/health",
ip: "192.168.0.0",
port: 9090,
want: "https://192.168.0.0:9090/health",
},
{
name: "https ipv6 literal",
url: "https://[::FFFF:192.168.0.1]:5000/health",
ip: "192.168.0.0",
port: 9090,
want: "https://192.168.0.0:9090/health",
},
{
name: "https ipv6 without path",
url: "https://[2001:db8:1f70::999:de8:7648:6e8]:5000",
ip: "192.168.0.0",
port: 9090,
want: "https://192.168.0.0:9090",
},
{
name: "ipv6 injected into ipv6 url",
url: "https://[2001:db8:1f70::999:de8:7648:6e8]:5000",
ip: "::FFFF:C0A8:1",
port: 9090,
want: "https://[::FFFF:C0A8:1]:9090",
},
{
name: "ipv6 with brackets injected into ipv6 url",
url: "https://[2001:db8:1f70::999:de8:7648:6e8]:5000",
ip: "[::FFFF:C0A8:1]",
port: 9090,
want: "https://[::FFFF:C0A8:1]:9090",
},
{
name: "short domain health",
url: "http://i.co:8080/health",
ip: "192.168.0.0",
port: 9090,
want: "http://192.168.0.0:9090/health",
},
{
name: "nested url in query",
url: "http://my.corp.com:8080/health?from=http://google.com:8080",
ip: "192.168.0.0",
port: 9090,
want: "http://192.168.0.0:9090/health?from=http://google.com:8080",
},
}
for _, tt := range tt {
t.Run(tt.name, func(t *testing.T) {
got := httpInjectAddr(tt.url, tt.ip, tt.port)
if got != tt.want {
t.Errorf("httpInjectAddr() got = %v, want %v", got, tt.want)
}
})
}
}
agent: tolerate more failure scenarios during service registration with central config enabled (#6472) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 15:04:48 +00:00
func TestDefaultIfEmpty(t *testing.T) {
require.Equal(t, "", defaultIfEmpty("", ""))
require.Equal(t, "foo", defaultIfEmpty("", "foo"))
require.Equal(t, "bar", defaultIfEmpty("bar", "foo"))
require.Equal(t, "bar", defaultIfEmpty("bar", ""))
}
func TestConfigSourceFromName(t *testing.T) {
cases := []struct {
in string
expect configSource
bad bool
}{
{in: "local", expect: ConfigSourceLocal},
{in: "remote", expect: ConfigSourceRemote},
{in: "", expect: ConfigSourceLocal},
{in: "LOCAL", bad: true},
{in: "REMOTE", bad: true},
{in: "garbage", bad: true},
{in: " ", bad: true},
}
for _, tc := range cases {
tc := tc
t.Run(tc.in, func(t *testing.T) {
got, ok := ConfigSourceFromName(tc.in)
if tc.bad {
require.False(t, ok)
require.Empty(t, got)
} else {
require.True(t, ok)
require.Equal(t, tc.expect, got)
}
})
}
}
Expose HTTP-based paths through Connect proxy (#6446) Fixes: #5396 This PR adds a proxy configuration stanza called expose. These flags register listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only listening on the loopback interface, while still accepting traffic from non Connect-enabled services. Under expose there is a boolean checks flag that would automatically expose all registered HTTP and gRPC check paths. This stanza also accepts a paths list to expose individual paths. The primary use case for this functionality would be to expose paths for third parties like Prometheus or the kubelet. Listeners for requests to exposed paths are be configured dynamically at run time. Any time a proxy, or check can be registered, a listener can also be created. In this initial implementation requests to these paths are not authenticated/encrypted.
2019-09-26 02:55:52 +00:00
func TestAgent_RerouteExistingHTTPChecks(t *testing.T) {
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
Expose HTTP-based paths through Connect proxy (#6446) Fixes: #5396 This PR adds a proxy configuration stanza called expose. These flags register listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only listening on the loopback interface, while still accepting traffic from non Connect-enabled services. Under expose there is a boolean checks flag that would automatically expose all registered HTTP and gRPC check paths. This stanza also accepts a paths list to expose individual paths. The primary use case for this functionality would be to expose paths for third parties like Prometheus or the kubelet. Listeners for requests to exposed paths are be configured dynamically at run time. Any time a proxy, or check can be registered, a listener can also be created. In this initial implementation requests to these paths are not authenticated/encrypted.
2019-09-26 02:55:52 +00:00
defer a.Shutdown()
testrpc.WaitForTestAgent(t, a.RPC, "dc1")
// Register a service without a ProxyAddr
svc := &structs.NodeService{
ID: "web",
Service: "web",
Address: "localhost",
Port: 8080,
}
chks := []*structs.CheckType{
{
CheckID: "http",
HTTP: "http://localhost:8080/mypath?query",
Interval: 20 * time.Millisecond,
TLSSkipVerify: true,
},
{
CheckID: "grpc",
GRPC: "localhost:8080/myservice",
Interval: 20 * time.Millisecond,
TLSSkipVerify: true,
},
}
if err := a.AddService(svc, chks, false, "", ConfigSourceLocal); err != nil {
t.Fatalf("failed to add svc: %v", err)
}
// Register a proxy and expose HTTP checks
// This should trigger setting ProxyHTTP and ProxyGRPC in the checks
proxy := &structs.NodeService{
Kind: "connect-proxy",
ID: "web-proxy",
Service: "web-proxy",
Address: "localhost",
Port: 21500,
Proxy: structs.ConnectProxyConfig{
DestinationServiceName: "web",
DestinationServiceID: "web",
LocalServiceAddress: "localhost",
LocalServicePort: 8080,
MeshGateway: structs.MeshGatewayConfig{},
Expose: structs.ExposeConfig{
Checks: true,
},
},
}
if err := a.AddService(proxy, nil, false, "", ConfigSourceLocal); err != nil {
t.Fatalf("failed to add svc: %v", err)
}
retry.Run(t, func(r *retry.R) {
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
chks := a.ServiceHTTPBasedChecks(structs.NewServiceID("web", nil))
Expose HTTP-based paths through Connect proxy (#6446) Fixes: #5396 This PR adds a proxy configuration stanza called expose. These flags register listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only listening on the loopback interface, while still accepting traffic from non Connect-enabled services. Under expose there is a boolean checks flag that would automatically expose all registered HTTP and gRPC check paths. This stanza also accepts a paths list to expose individual paths. The primary use case for this functionality would be to expose paths for third parties like Prometheus or the kubelet. Listeners for requests to exposed paths are be configured dynamically at run time. Any time a proxy, or check can be registered, a listener can also be created. In this initial implementation requests to these paths are not authenticated/encrypted.
2019-09-26 02:55:52 +00:00
got := chks[0].ProxyHTTP
if got == "" {
r.Fatal("proxyHTTP addr not set in check")
}
want := "http://localhost:21500/mypath?query"
if got != want {
r.Fatalf("unexpected proxy addr in check, want: %s, got: %s", want, got)
}
})
retry.Run(t, func(r *retry.R) {
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
chks := a.ServiceHTTPBasedChecks(structs.NewServiceID("web", nil))
Expose HTTP-based paths through Connect proxy (#6446) Fixes: #5396 This PR adds a proxy configuration stanza called expose. These flags register listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only listening on the loopback interface, while still accepting traffic from non Connect-enabled services. Under expose there is a boolean checks flag that would automatically expose all registered HTTP and gRPC check paths. This stanza also accepts a paths list to expose individual paths. The primary use case for this functionality would be to expose paths for third parties like Prometheus or the kubelet. Listeners for requests to exposed paths are be configured dynamically at run time. Any time a proxy, or check can be registered, a listener can also be created. In this initial implementation requests to these paths are not authenticated/encrypted.
2019-09-26 02:55:52 +00:00
// Will be at a later index than HTTP check because of the fetching order in ServiceHTTPBasedChecks
got := chks[1].ProxyGRPC
if got == "" {
r.Fatal("ProxyGRPC addr not set in check")
}
// Node that this relies on listener ports auto-incrementing in a.listenerPortLocked
want := "localhost:21501/myservice"
if got != want {
r.Fatalf("unexpected proxy addr in check, want: %s, got: %s", want, got)
}
})
// Re-register a proxy and disable exposing HTTP checks
// This should trigger resetting ProxyHTTP and ProxyGRPC to empty strings
proxy = &structs.NodeService{
Kind: "connect-proxy",
ID: "web-proxy",
Service: "web-proxy",
Address: "localhost",
Port: 21500,
Proxy: structs.ConnectProxyConfig{
DestinationServiceName: "web",
DestinationServiceID: "web",
LocalServiceAddress: "localhost",
LocalServicePort: 8080,
MeshGateway: structs.MeshGatewayConfig{},
Expose: structs.ExposeConfig{
Checks: false,
},
},
}
if err := a.AddService(proxy, nil, false, "", ConfigSourceLocal); err != nil {
t.Fatalf("failed to add svc: %v", err)
}
retry.Run(t, func(r *retry.R) {
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
chks := a.ServiceHTTPBasedChecks(structs.NewServiceID("web", nil))
Expose HTTP-based paths through Connect proxy (#6446) Fixes: #5396 This PR adds a proxy configuration stanza called expose. These flags register listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only listening on the loopback interface, while still accepting traffic from non Connect-enabled services. Under expose there is a boolean checks flag that would automatically expose all registered HTTP and gRPC check paths. This stanza also accepts a paths list to expose individual paths. The primary use case for this functionality would be to expose paths for third parties like Prometheus or the kubelet. Listeners for requests to exposed paths are be configured dynamically at run time. Any time a proxy, or check can be registered, a listener can also be created. In this initial implementation requests to these paths are not authenticated/encrypted.
2019-09-26 02:55:52 +00:00
got := chks[0].ProxyHTTP
if got != "" {
r.Fatal("ProxyHTTP addr was not reset")
}
})
retry.Run(t, func(r *retry.R) {
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
chks := a.ServiceHTTPBasedChecks(structs.NewServiceID("web", nil))
Expose HTTP-based paths through Connect proxy (#6446) Fixes: #5396 This PR adds a proxy configuration stanza called expose. These flags register listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only listening on the loopback interface, while still accepting traffic from non Connect-enabled services. Under expose there is a boolean checks flag that would automatically expose all registered HTTP and gRPC check paths. This stanza also accepts a paths list to expose individual paths. The primary use case for this functionality would be to expose paths for third parties like Prometheus or the kubelet. Listeners for requests to exposed paths are be configured dynamically at run time. Any time a proxy, or check can be registered, a listener can also be created. In this initial implementation requests to these paths are not authenticated/encrypted.
2019-09-26 02:55:52 +00:00
// Will be at a later index than HTTP check because of the fetching order in ServiceHTTPBasedChecks
got := chks[1].ProxyGRPC
if got != "" {
r.Fatal("ProxyGRPC addr was not reset")
}
})
}
func TestAgent_RerouteNewHTTPChecks(t *testing.T) {
t.Parallel()
Remove name from NewTestAgent Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 19:59:56 +00:00
a := NewTestAgent(t, "")
Expose HTTP-based paths through Connect proxy (#6446) Fixes: #5396 This PR adds a proxy configuration stanza called expose. These flags register listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only listening on the loopback interface, while still accepting traffic from non Connect-enabled services. Under expose there is a boolean checks flag that would automatically expose all registered HTTP and gRPC check paths. This stanza also accepts a paths list to expose individual paths. The primary use case for this functionality would be to expose paths for third parties like Prometheus or the kubelet. Listeners for requests to exposed paths are be configured dynamically at run time. Any time a proxy, or check can be registered, a listener can also be created. In this initial implementation requests to these paths are not authenticated/encrypted.
2019-09-26 02:55:52 +00:00
defer a.Shutdown()
testrpc.WaitForTestAgent(t, a.RPC, "dc1")
// Register a service without a ProxyAddr
svc := &structs.NodeService{
ID: "web",
Service: "web",
Address: "localhost",
Port: 8080,
}
if err := a.AddService(svc, nil, false, "", ConfigSourceLocal); err != nil {
t.Fatalf("failed to add svc: %v", err)
}
// Register a proxy and expose HTTP checks
proxy := &structs.NodeService{
Kind: "connect-proxy",
ID: "web-proxy",
Service: "web-proxy",
Address: "localhost",
Port: 21500,
Proxy: structs.ConnectProxyConfig{
DestinationServiceName: "web",
DestinationServiceID: "web",
LocalServiceAddress: "localhost",
LocalServicePort: 8080,
MeshGateway: structs.MeshGatewayConfig{},
Expose: structs.ExposeConfig{
Checks: true,
},
},
}
if err := a.AddService(proxy, nil, false, "", ConfigSourceLocal); err != nil {
t.Fatalf("failed to add svc: %v", err)
}
checks := []*structs.HealthCheck{
{
CheckID: "http",
Name: "http",
ServiceID: "web",
Status: api.HealthCritical,
},
{
CheckID: "grpc",
Name: "grpc",
ServiceID: "web",
Status: api.HealthCritical,
},
}
chkTypes := []*structs.CheckType{
{
CheckID: "http",
HTTP: "http://localhost:8080/mypath?query",
Interval: 20 * time.Millisecond,
TLSSkipVerify: true,
},
{
CheckID: "grpc",
GRPC: "localhost:8080/myservice",
Interval: 20 * time.Millisecond,
TLSSkipVerify: true,
},
}
// ProxyGRPC and ProxyHTTP should be set when creating check
// since proxy.expose.checks is enabled on the proxy
if err := a.AddCheck(checks[0], chkTypes[0], false, "", ConfigSourceLocal); err != nil {
t.Fatalf("failed to add check: %v", err)
}
if err := a.AddCheck(checks[1], chkTypes[1], false, "", ConfigSourceLocal); err != nil {
t.Fatalf("failed to add check: %v", err)
}
retry.Run(t, func(r *retry.R) {
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
chks := a.ServiceHTTPBasedChecks(structs.NewServiceID("web", nil))
Expose HTTP-based paths through Connect proxy (#6446) Fixes: #5396 This PR adds a proxy configuration stanza called expose. These flags register listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only listening on the loopback interface, while still accepting traffic from non Connect-enabled services. Under expose there is a boolean checks flag that would automatically expose all registered HTTP and gRPC check paths. This stanza also accepts a paths list to expose individual paths. The primary use case for this functionality would be to expose paths for third parties like Prometheus or the kubelet. Listeners for requests to exposed paths are be configured dynamically at run time. Any time a proxy, or check can be registered, a listener can also be created. In this initial implementation requests to these paths are not authenticated/encrypted.
2019-09-26 02:55:52 +00:00
got := chks[0].ProxyHTTP
if got == "" {
r.Fatal("ProxyHTTP addr not set in check")
}
want := "http://localhost:21500/mypath?query"
if got != want {
r.Fatalf("unexpected proxy addr in http check, want: %s, got: %s", want, got)
}
})
retry.Run(t, func(r *retry.R) {
Sync of OSS changes to support namespaces (#6909)
2019-12-10 02:26:41 +00:00
chks := a.ServiceHTTPBasedChecks(structs.NewServiceID("web", nil))
Expose HTTP-based paths through Connect proxy (#6446) Fixes: #5396 This PR adds a proxy configuration stanza called expose. These flags register listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only listening on the loopback interface, while still accepting traffic from non Connect-enabled services. Under expose there is a boolean checks flag that would automatically expose all registered HTTP and gRPC check paths. This stanza also accepts a paths list to expose individual paths. The primary use case for this functionality would be to expose paths for third parties like Prometheus or the kubelet. Listeners for requests to exposed paths are be configured dynamically at run time. Any time a proxy, or check can be registered, a listener can also be created. In this initial implementation requests to these paths are not authenticated/encrypted.
2019-09-26 02:55:52 +00:00
// Will be at a later index than HTTP check because of the fetching order in ServiceHTTPBasedChecks
got := chks[1].ProxyGRPC
if got == "" {
r.Fatal("ProxyGRPC addr not set in check")
}
want := "localhost:21501/myservice"
if got != want {
r.Fatalf("unexpected proxy addr in grpc check, want: %s, got: %s", want, got)
}
})
}
agent: cache notifications work after error if the underlying RPC returns index=1 (#6547) Fixes #6521 Ensure that initial failures to fetch an agent cache entry using the notify API where the underlying RPC returns a synthetic index of 1 correctly recovers when those RPCs resume working. The bug in the Cache.notifyBlockingQuery used to incorrectly "fix" the index for the next query from 0 to 1 for all queries, when it should have not done so for queries that errored. Also fixed some things that made debugging difficult: - config entry read/list endpoints send back QueryMeta headers - xds event loops don't swallow the cache notification errors
2019-09-26 15:42:17 +00:00
func TestAgentCache_serviceInConfigFile_initialFetchErrors_Issue6521(t *testing.T) {
t.Parallel()
// Ensure that initial failures to fetch the discovery chain via the agent
// cache using the notify API for a service with no config entries
// correctly recovers when those RPCs resume working. The key here is that
// the lack of config entries guarantees that the RPC will come back with a
// synthetic index of 1.
//
// The bug in the Cache.notifyBlockingQuery used to incorrectly "fix" the
// index for the next query from 0 to 1 for all queries, when it should
// have not done so for queries that errored.
Rename NewTestAgentWithFields to StartTestAgent This function now only starts the agent. Using: git grep -l 'StartTestAgent(t, true,' | \ xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'
2020-03-31 20:24:39 +00:00
a1 := StartTestAgent(t, TestAgent{Name: "Agent1"})
agent: cache notifications work after error if the underlying RPC returns index=1 (#6547) Fixes #6521 Ensure that initial failures to fetch an agent cache entry using the notify API where the underlying RPC returns a synthetic index of 1 correctly recovers when those RPCs resume working. The bug in the Cache.notifyBlockingQuery used to incorrectly "fix" the index for the next query from 0 to 1 for all queries, when it should have not done so for queries that errored. Also fixed some things that made debugging difficult: - config entry read/list endpoints send back QueryMeta headers - xds event loops don't swallow the cache notification errors
2019-09-26 15:42:17 +00:00
defer a1.Shutdown()
testrpc.WaitForLeader(t, a1.RPC, "dc1")
Rename NewTestAgentWithFields to StartTestAgent This function now only starts the agent. Using: git grep -l 'StartTestAgent(t, true,' | \ xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'
2020-03-31 20:24:39 +00:00
a2 := StartTestAgent(t, TestAgent{Name: "Agent2", HCL: `
agent: cache notifications work after error if the underlying RPC returns index=1 (#6547) Fixes #6521 Ensure that initial failures to fetch an agent cache entry using the notify API where the underlying RPC returns a synthetic index of 1 correctly recovers when those RPCs resume working. The bug in the Cache.notifyBlockingQuery used to incorrectly "fix" the index for the next query from 0 to 1 for all queries, when it should have not done so for queries that errored. Also fixed some things that made debugging difficult: - config entry read/list endpoints send back QueryMeta headers - xds event loops don't swallow the cache notification errors
2019-09-26 15:42:17 +00:00
server = false
bootstrap = false
services {
name = "echo-client"
port = 8080
connect {
sidecar_service {
proxy {
upstreams {
destination_name = "echo"
local_bind_port = 9191
}
}
}
}
}
services {
name = "echo"
port = 9090
connect {
sidecar_service {}
}
}
Convert the remaining calls to NewTestAgentWithFields After removing the t.Name() parameter with sed, convert the last few tests which use a custom name to call NewTestAgentWithFields instead.
2020-03-31 20:12:33 +00:00
`})
agent: cache notifications work after error if the underlying RPC returns index=1 (#6547) Fixes #6521 Ensure that initial failures to fetch an agent cache entry using the notify API where the underlying RPC returns a synthetic index of 1 correctly recovers when those RPCs resume working. The bug in the Cache.notifyBlockingQuery used to incorrectly "fix" the index for the next query from 0 to 1 for all queries, when it should have not done so for queries that errored. Also fixed some things that made debugging difficult: - config entry read/list endpoints send back QueryMeta headers - xds event loops don't swallow the cache notification errors
2019-09-26 15:42:17 +00:00
defer a2.Shutdown()
// Starting a client agent disconnected from a server with services.
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
ch := make(chan cache.UpdateEvent, 1)
require.NoError(t, a2.cache.Notify(ctx, cachetype.CompiledDiscoveryChainName, &structs.DiscoveryChainRequest{
Datacenter: "dc1",
Name: "echo",
EvaluateInDatacenter: "dc1",
EvaluateInNamespace: "default",
}, "foo", ch))
{ // The first event is an error because we are not joined yet.
evt := <-ch
require.Equal(t, "foo", evt.CorrelationID)
require.Nil(t, evt.Result)
require.Error(t, evt.Err)
require.Equal(t, evt.Err, structs.ErrNoServers)
}
t.Logf("joining client to server")
// Now connect to server
_, err := a1.JoinLAN([]string{
fmt.Sprintf("127.0.0.1:%d", a2.Config.SerfPortLAN),
})
require.NoError(t, err)
t.Logf("joined client to server")
deadlineCh := time.After(10 * time.Second)
start := time.Now()
LOOP:
for {
select {
case evt := <-ch:
// We may receive several notifications of an error until we get the
// first successful reply.
require.Equal(t, "foo", evt.CorrelationID)
if evt.Err != nil {
break LOOP
}
require.NoError(t, evt.Err)
require.NotNil(t, evt.Result)
t.Logf("took %s to get first success", time.Since(start))
case <-deadlineCh:
t.Fatal("did not get notified successfully")
}
}
}
wan federation via mesh gateways (#6884) This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch: There are several distinct chunks of code that are affected: * new flags and config options for the server * retry join WAN is slightly different * retry join code is shared to discover primary mesh gateways from secondary datacenters * because retry join logic runs in the *agent* and the results of that operation for primary mesh gateways are needed in the *server* there are some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur at multiple layers of abstraction just to pass the data down to the right layer. * new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers * the function signature for RPC dialing picked up a new required field (the node name of the destination) * several new RPCs for manipulating a FederationState object: `FederationState:{Apply,Get,List,ListMeshGateways}` * 3 read-only internal APIs for debugging use to invoke those RPCs from curl * raft and fsm changes to persist these FederationStates * replication for FederationStates as they are canonically stored in the Primary and replicated to the Secondaries. * a special derivative of anti-entropy that runs in secondaries to snapshot their local mesh gateway `CheckServiceNodes` and sync them into their upstream FederationState in the primary (this works in conjunction with the replication to distribute addresses for all mesh gateways in all DCs to all other DCs) * a "gateway locator" convenience object to make use of this data to choose the addresses of gateways to use for any given RPC or gossip operation to a remote DC. This gets data from the "retry join" logic in the agent and also directly calls into the FSM. * RPC (`:8300`) on the server sniffs the first byte of a new connection to determine if it's actually doing native TLS. If so it checks the ALPN header for protocol determination (just like how the existing system uses the type-byte marker). * 2 new kinds of protocols are exclusively decoded via this native TLS mechanism: one for ferrying "packet" operations (udp-like) from the gossip layer and one for "stream" operations (tcp-like). The packet operations re-use sockets (using length-prefixing) to cut down on TLS re-negotiation overhead. * the server instances specially wrap the `memberlist.NetTransport` when running with gateway federation enabled (in a `wanfed.Transport`). The general gist is that if it tries to dial a node in the SAME datacenter (deduced by looking at the suffix of the node name) there is no change. If dialing a DIFFERENT datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh gateways to eventually end up in a server's :8300 port. * a new flag when launching a mesh gateway via `consul connect envoy` to indicate that the servers are to be exposed. This sets a special service meta when registering the gateway into the catalog. * `proxycfg/xds` notice this metadata blob to activate additional watches for the FederationState objects as well as the location of all of the consul servers in that datacenter. * `xds:` if the extra metadata is in place additional clusters are defined in a DC to bulk sink all traffic to another DC's gateways. For the current datacenter we listen on a wildcard name (`server.<dc>.consul`) that load balances all servers as well as one mini-cluster per node (`<node>.server.<dc>.consul`) * the `consul tls cert create` command got a new flag (`-node`) to help create an additional SAN in certs that can be used with this flavor of federation.
2020-03-09 20:59:02 +00:00
// This is a mirror of a similar test in agent/consul/server_test.go
func TestAgent_JoinWAN_viaMeshGateway(t *testing.T) {
t.Parallel()
gwPort := freeport.MustTake(1)
defer freeport.Return(gwPort)
gwAddr := ipaddr.FormatAddressPort("127.0.0.1", gwPort[0])
// Due to some ordering, we'll have to manually configure these ports in
// advance.
secondaryRPCPorts := freeport.MustTake(2)
defer freeport.Return(secondaryRPCPorts)
Rename NewTestAgentWithFields to StartTestAgent This function now only starts the agent. Using: git grep -l 'StartTestAgent(t, true,' | \ xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'
2020-03-31 20:24:39 +00:00
a1 := StartTestAgent(t, TestAgent{Name: "bob", HCL: `
wan federation via mesh gateways (#6884) This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch: There are several distinct chunks of code that are affected: * new flags and config options for the server * retry join WAN is slightly different * retry join code is shared to discover primary mesh gateways from secondary datacenters * because retry join logic runs in the *agent* and the results of that operation for primary mesh gateways are needed in the *server* there are some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur at multiple layers of abstraction just to pass the data down to the right layer. * new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers * the function signature for RPC dialing picked up a new required field (the node name of the destination) * several new RPCs for manipulating a FederationState object: `FederationState:{Apply,Get,List,ListMeshGateways}` * 3 read-only internal APIs for debugging use to invoke those RPCs from curl * raft and fsm changes to persist these FederationStates * replication for FederationStates as they are canonically stored in the Primary and replicated to the Secondaries. * a special derivative of anti-entropy that runs in secondaries to snapshot their local mesh gateway `CheckServiceNodes` and sync them into their upstream FederationState in the primary (this works in conjunction with the replication to distribute addresses for all mesh gateways in all DCs to all other DCs) * a "gateway locator" convenience object to make use of this data to choose the addresses of gateways to use for any given RPC or gossip operation to a remote DC. This gets data from the "retry join" logic in the agent and also directly calls into the FSM. * RPC (`:8300`) on the server sniffs the first byte of a new connection to determine if it's actually doing native TLS. If so it checks the ALPN header for protocol determination (just like how the existing system uses the type-byte marker). * 2 new kinds of protocols are exclusively decoded via this native TLS mechanism: one for ferrying "packet" operations (udp-like) from the gossip layer and one for "stream" operations (tcp-like). The packet operations re-use sockets (using length-prefixing) to cut down on TLS re-negotiation overhead. * the server instances specially wrap the `memberlist.NetTransport` when running with gateway federation enabled (in a `wanfed.Transport`). The general gist is that if it tries to dial a node in the SAME datacenter (deduced by looking at the suffix of the node name) there is no change. If dialing a DIFFERENT datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh gateways to eventually end up in a server's :8300 port. * a new flag when launching a mesh gateway via `consul connect envoy` to indicate that the servers are to be exposed. This sets a special service meta when registering the gateway into the catalog. * `proxycfg/xds` notice this metadata blob to activate additional watches for the FederationState objects as well as the location of all of the consul servers in that datacenter. * `xds:` if the extra metadata is in place additional clusters are defined in a DC to bulk sink all traffic to another DC's gateways. For the current datacenter we listen on a wildcard name (`server.<dc>.consul`) that load balances all servers as well as one mini-cluster per node (`<node>.server.<dc>.consul`) * the `consul tls cert create` command got a new flag (`-node`) to help create an additional SAN in certs that can be used with this flavor of federation.
2020-03-09 20:59:02 +00:00
domain = "consul"
node_name = "bob"
datacenter = "dc1"
primary_datacenter = "dc1"
# tls
ca_file = "../test/hostname/CertAuth.crt"
cert_file = "../test/hostname/Bob.crt"
key_file = "../test/hostname/Bob.key"
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
# wanfed
connect {
enabled = true
enable_mesh_gateway_wan_federation = true
}
Convert the remaining calls to NewTestAgentWithFields After removing the t.Name() parameter with sed, convert the last few tests which use a custom name to call NewTestAgentWithFields instead.
2020-03-31 20:12:33 +00:00
`})
wan federation via mesh gateways (#6884) This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch: There are several distinct chunks of code that are affected: * new flags and config options for the server * retry join WAN is slightly different * retry join code is shared to discover primary mesh gateways from secondary datacenters * because retry join logic runs in the *agent* and the results of that operation for primary mesh gateways are needed in the *server* there are some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur at multiple layers of abstraction just to pass the data down to the right layer. * new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers * the function signature for RPC dialing picked up a new required field (the node name of the destination) * several new RPCs for manipulating a FederationState object: `FederationState:{Apply,Get,List,ListMeshGateways}` * 3 read-only internal APIs for debugging use to invoke those RPCs from curl * raft and fsm changes to persist these FederationStates * replication for FederationStates as they are canonically stored in the Primary and replicated to the Secondaries. * a special derivative of anti-entropy that runs in secondaries to snapshot their local mesh gateway `CheckServiceNodes` and sync them into their upstream FederationState in the primary (this works in conjunction with the replication to distribute addresses for all mesh gateways in all DCs to all other DCs) * a "gateway locator" convenience object to make use of this data to choose the addresses of gateways to use for any given RPC or gossip operation to a remote DC. This gets data from the "retry join" logic in the agent and also directly calls into the FSM. * RPC (`:8300`) on the server sniffs the first byte of a new connection to determine if it's actually doing native TLS. If so it checks the ALPN header for protocol determination (just like how the existing system uses the type-byte marker). * 2 new kinds of protocols are exclusively decoded via this native TLS mechanism: one for ferrying "packet" operations (udp-like) from the gossip layer and one for "stream" operations (tcp-like). The packet operations re-use sockets (using length-prefixing) to cut down on TLS re-negotiation overhead. * the server instances specially wrap the `memberlist.NetTransport` when running with gateway federation enabled (in a `wanfed.Transport`). The general gist is that if it tries to dial a node in the SAME datacenter (deduced by looking at the suffix of the node name) there is no change. If dialing a DIFFERENT datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh gateways to eventually end up in a server's :8300 port. * a new flag when launching a mesh gateway via `consul connect envoy` to indicate that the servers are to be exposed. This sets a special service meta when registering the gateway into the catalog. * `proxycfg/xds` notice this metadata blob to activate additional watches for the FederationState objects as well as the location of all of the consul servers in that datacenter. * `xds:` if the extra metadata is in place additional clusters are defined in a DC to bulk sink all traffic to another DC's gateways. For the current datacenter we listen on a wildcard name (`server.<dc>.consul`) that load balances all servers as well as one mini-cluster per node (`<node>.server.<dc>.consul`) * the `consul tls cert create` command got a new flag (`-node`) to help create an additional SAN in certs that can be used with this flavor of federation.
2020-03-09 20:59:02 +00:00
defer a1.Shutdown()
testrpc.WaitForTestAgent(t, a1.RPC, "dc1")
// We'll use the same gateway for all datacenters since it doesn't care.
var (
rpcAddr1 = ipaddr.FormatAddressPort("127.0.0.1", a1.Config.ServerPort)
rpcAddr2 = ipaddr.FormatAddressPort("127.0.0.1", secondaryRPCPorts[0])
rpcAddr3 = ipaddr.FormatAddressPort("127.0.0.1", secondaryRPCPorts[1])
)
var p tcpproxy.Proxy
p.AddSNIRoute(gwAddr, "bob.server.dc1.consul", tcpproxy.To(rpcAddr1))
p.AddSNIRoute(gwAddr, "server.dc1.consul", tcpproxy.To(rpcAddr1))
p.AddSNIRoute(gwAddr, "betty.server.dc2.consul", tcpproxy.To(rpcAddr2))
p.AddSNIRoute(gwAddr, "server.dc2.consul", tcpproxy.To(rpcAddr2))
p.AddSNIRoute(gwAddr, "bonnie.server.dc3.consul", tcpproxy.To(rpcAddr3))
p.AddSNIRoute(gwAddr, "server.dc3.consul", tcpproxy.To(rpcAddr3))
p.AddStopACMESearch(gwAddr)
require.NoError(t, p.Start())
defer func() {
p.Close()
p.Wait()
}()
t.Logf("routing %s => %s", "{bob.,}server.dc1.consul", rpcAddr1)
t.Logf("routing %s => %s", "{betty.,}server.dc2.consul", rpcAddr2)
t.Logf("routing %s => %s", "{bonnie.,}server.dc3.consul", rpcAddr3)
// Register this into the agent in dc1.
{
args := &structs.ServiceDefinition{
Kind: structs.ServiceKindMeshGateway,
ID: "mesh-gateway",
Name: "mesh-gateway",
Meta: map[string]string{structs.MetaWANFederationKey: "1"},
Port: gwPort[0],
}
req, err := http.NewRequest("PUT", "/v1/agent/service/register", jsonReader(args))
require.NoError(t, err)
obj, err := a1.srv.AgentRegisterService(nil, req)
require.NoError(t, err)
require.Nil(t, obj)
}
waitForFederationState := func(t *testing.T, a *TestAgent, dc string) {
retry.Run(t, func(r *retry.R) {
req, err := http.NewRequest("GET", "/v1/internal/federation-state/"+dc, nil)
require.NoError(r, err)
resp := httptest.NewRecorder()
obj, err := a.srv.FederationStateGet(resp, req)
require.NoError(r, err)
require.NotNil(r, obj)
out, ok := obj.(structs.FederationStateResponse)
require.True(r, ok)
require.NotNil(r, out.State)
require.Len(r, out.State.MeshGateways, 1)
})
}
// Wait until at least catalog AE and federation state AE fire.
waitForFederationState(t, a1, "dc1")
retry.Run(t, func(r *retry.R) {
require.NotEmpty(r, a1.PickRandomMeshGatewaySuitableForDialing("dc1"))
})
Rename NewTestAgentWithFields to StartTestAgent This function now only starts the agent. Using: git grep -l 'StartTestAgent(t, true,' | \ xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'
2020-03-31 20:24:39 +00:00
a2 := StartTestAgent(t, TestAgent{Name: "betty", HCL: `
wan federation via mesh gateways (#6884) This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch: There are several distinct chunks of code that are affected: * new flags and config options for the server * retry join WAN is slightly different * retry join code is shared to discover primary mesh gateways from secondary datacenters * because retry join logic runs in the *agent* and the results of that operation for primary mesh gateways are needed in the *server* there are some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur at multiple layers of abstraction just to pass the data down to the right layer. * new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers * the function signature for RPC dialing picked up a new required field (the node name of the destination) * several new RPCs for manipulating a FederationState object: `FederationState:{Apply,Get,List,ListMeshGateways}` * 3 read-only internal APIs for debugging use to invoke those RPCs from curl * raft and fsm changes to persist these FederationStates * replication for FederationStates as they are canonically stored in the Primary and replicated to the Secondaries. * a special derivative of anti-entropy that runs in secondaries to snapshot their local mesh gateway `CheckServiceNodes` and sync them into their upstream FederationState in the primary (this works in conjunction with the replication to distribute addresses for all mesh gateways in all DCs to all other DCs) * a "gateway locator" convenience object to make use of this data to choose the addresses of gateways to use for any given RPC or gossip operation to a remote DC. This gets data from the "retry join" logic in the agent and also directly calls into the FSM. * RPC (`:8300`) on the server sniffs the first byte of a new connection to determine if it's actually doing native TLS. If so it checks the ALPN header for protocol determination (just like how the existing system uses the type-byte marker). * 2 new kinds of protocols are exclusively decoded via this native TLS mechanism: one for ferrying "packet" operations (udp-like) from the gossip layer and one for "stream" operations (tcp-like). The packet operations re-use sockets (using length-prefixing) to cut down on TLS re-negotiation overhead. * the server instances specially wrap the `memberlist.NetTransport` when running with gateway federation enabled (in a `wanfed.Transport`). The general gist is that if it tries to dial a node in the SAME datacenter (deduced by looking at the suffix of the node name) there is no change. If dialing a DIFFERENT datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh gateways to eventually end up in a server's :8300 port. * a new flag when launching a mesh gateway via `consul connect envoy` to indicate that the servers are to be exposed. This sets a special service meta when registering the gateway into the catalog. * `proxycfg/xds` notice this metadata blob to activate additional watches for the FederationState objects as well as the location of all of the consul servers in that datacenter. * `xds:` if the extra metadata is in place additional clusters are defined in a DC to bulk sink all traffic to another DC's gateways. For the current datacenter we listen on a wildcard name (`server.<dc>.consul`) that load balances all servers as well as one mini-cluster per node (`<node>.server.<dc>.consul`) * the `consul tls cert create` command got a new flag (`-node`) to help create an additional SAN in certs that can be used with this flavor of federation.
2020-03-09 20:59:02 +00:00
domain = "consul"
node_name = "betty"
datacenter = "dc2"
primary_datacenter = "dc1"
# tls
ca_file = "../test/hostname/CertAuth.crt"
cert_file = "../test/hostname/Betty.crt"
key_file = "../test/hostname/Betty.key"
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
ports {
Convert the remaining calls to NewTestAgentWithFields After removing the t.Name() parameter with sed, convert the last few tests which use a custom name to call NewTestAgentWithFields instead.
2020-03-31 20:12:33 +00:00
server = ` + strconv.Itoa(secondaryRPCPorts[0]) + `
wan federation via mesh gateways (#6884) This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch: There are several distinct chunks of code that are affected: * new flags and config options for the server * retry join WAN is slightly different * retry join code is shared to discover primary mesh gateways from secondary datacenters * because retry join logic runs in the *agent* and the results of that operation for primary mesh gateways are needed in the *server* there are some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur at multiple layers of abstraction just to pass the data down to the right layer. * new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers * the function signature for RPC dialing picked up a new required field (the node name of the destination) * several new RPCs for manipulating a FederationState object: `FederationState:{Apply,Get,List,ListMeshGateways}` * 3 read-only internal APIs for debugging use to invoke those RPCs from curl * raft and fsm changes to persist these FederationStates * replication for FederationStates as they are canonically stored in the Primary and replicated to the Secondaries. * a special derivative of anti-entropy that runs in secondaries to snapshot their local mesh gateway `CheckServiceNodes` and sync them into their upstream FederationState in the primary (this works in conjunction with the replication to distribute addresses for all mesh gateways in all DCs to all other DCs) * a "gateway locator" convenience object to make use of this data to choose the addresses of gateways to use for any given RPC or gossip operation to a remote DC. This gets data from the "retry join" logic in the agent and also directly calls into the FSM. * RPC (`:8300`) on the server sniffs the first byte of a new connection to determine if it's actually doing native TLS. If so it checks the ALPN header for protocol determination (just like how the existing system uses the type-byte marker). * 2 new kinds of protocols are exclusively decoded via this native TLS mechanism: one for ferrying "packet" operations (udp-like) from the gossip layer and one for "stream" operations (tcp-like). The packet operations re-use sockets (using length-prefixing) to cut down on TLS re-negotiation overhead. * the server instances specially wrap the `memberlist.NetTransport` when running with gateway federation enabled (in a `wanfed.Transport`). The general gist is that if it tries to dial a node in the SAME datacenter (deduced by looking at the suffix of the node name) there is no change. If dialing a DIFFERENT datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh gateways to eventually end up in a server's :8300 port. * a new flag when launching a mesh gateway via `consul connect envoy` to indicate that the servers are to be exposed. This sets a special service meta when registering the gateway into the catalog. * `proxycfg/xds` notice this metadata blob to activate additional watches for the FederationState objects as well as the location of all of the consul servers in that datacenter. * `xds:` if the extra metadata is in place additional clusters are defined in a DC to bulk sink all traffic to another DC's gateways. For the current datacenter we listen on a wildcard name (`server.<dc>.consul`) that load balances all servers as well as one mini-cluster per node (`<node>.server.<dc>.consul`) * the `consul tls cert create` command got a new flag (`-node`) to help create an additional SAN in certs that can be used with this flavor of federation.
2020-03-09 20:59:02 +00:00
}
# wanfed
Convert the remaining calls to NewTestAgentWithFields After removing the t.Name() parameter with sed, convert the last few tests which use a custom name to call NewTestAgentWithFields instead.
2020-03-31 20:12:33 +00:00
primary_gateways = ["` + gwAddr + `"]
wan federation via mesh gateways (#6884) This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch: There are several distinct chunks of code that are affected: * new flags and config options for the server * retry join WAN is slightly different * retry join code is shared to discover primary mesh gateways from secondary datacenters * because retry join logic runs in the *agent* and the results of that operation for primary mesh gateways are needed in the *server* there are some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur at multiple layers of abstraction just to pass the data down to the right layer. * new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers * the function signature for RPC dialing picked up a new required field (the node name of the destination) * several new RPCs for manipulating a FederationState object: `FederationState:{Apply,Get,List,ListMeshGateways}` * 3 read-only internal APIs for debugging use to invoke those RPCs from curl * raft and fsm changes to persist these FederationStates * replication for FederationStates as they are canonically stored in the Primary and replicated to the Secondaries. * a special derivative of anti-entropy that runs in secondaries to snapshot their local mesh gateway `CheckServiceNodes` and sync them into their upstream FederationState in the primary (this works in conjunction with the replication to distribute addresses for all mesh gateways in all DCs to all other DCs) * a "gateway locator" convenience object to make use of this data to choose the addresses of gateways to use for any given RPC or gossip operation to a remote DC. This gets data from the "retry join" logic in the agent and also directly calls into the FSM. * RPC (`:8300`) on the server sniffs the first byte of a new connection to determine if it's actually doing native TLS. If so it checks the ALPN header for protocol determination (just like how the existing system uses the type-byte marker). * 2 new kinds of protocols are exclusively decoded via this native TLS mechanism: one for ferrying "packet" operations (udp-like) from the gossip layer and one for "stream" operations (tcp-like). The packet operations re-use sockets (using length-prefixing) to cut down on TLS re-negotiation overhead. * the server instances specially wrap the `memberlist.NetTransport` when running with gateway federation enabled (in a `wanfed.Transport`). The general gist is that if it tries to dial a node in the SAME datacenter (deduced by looking at the suffix of the node name) there is no change. If dialing a DIFFERENT datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh gateways to eventually end up in a server's :8300 port. * a new flag when launching a mesh gateway via `consul connect envoy` to indicate that the servers are to be exposed. This sets a special service meta when registering the gateway into the catalog. * `proxycfg/xds` notice this metadata blob to activate additional watches for the FederationState objects as well as the location of all of the consul servers in that datacenter. * `xds:` if the extra metadata is in place additional clusters are defined in a DC to bulk sink all traffic to another DC's gateways. For the current datacenter we listen on a wildcard name (`server.<dc>.consul`) that load balances all servers as well as one mini-cluster per node (`<node>.server.<dc>.consul`) * the `consul tls cert create` command got a new flag (`-node`) to help create an additional SAN in certs that can be used with this flavor of federation.
2020-03-09 20:59:02 +00:00
connect {
enabled = true
enable_mesh_gateway_wan_federation = true
}
Convert the remaining calls to NewTestAgentWithFields After removing the t.Name() parameter with sed, convert the last few tests which use a custom name to call NewTestAgentWithFields instead.
2020-03-31 20:12:33 +00:00
`})
wan federation via mesh gateways (#6884) This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch: There are several distinct chunks of code that are affected: * new flags and config options for the server * retry join WAN is slightly different * retry join code is shared to discover primary mesh gateways from secondary datacenters * because retry join logic runs in the *agent* and the results of that operation for primary mesh gateways are needed in the *server* there are some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur at multiple layers of abstraction just to pass the data down to the right layer. * new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers * the function signature for RPC dialing picked up a new required field (the node name of the destination) * several new RPCs for manipulating a FederationState object: `FederationState:{Apply,Get,List,ListMeshGateways}` * 3 read-only internal APIs for debugging use to invoke those RPCs from curl * raft and fsm changes to persist these FederationStates * replication for FederationStates as they are canonically stored in the Primary and replicated to the Secondaries. * a special derivative of anti-entropy that runs in secondaries to snapshot their local mesh gateway `CheckServiceNodes` and sync them into their upstream FederationState in the primary (this works in conjunction with the replication to distribute addresses for all mesh gateways in all DCs to all other DCs) * a "gateway locator" convenience object to make use of this data to choose the addresses of gateways to use for any given RPC or gossip operation to a remote DC. This gets data from the "retry join" logic in the agent and also directly calls into the FSM. * RPC (`:8300`) on the server sniffs the first byte of a new connection to determine if it's actually doing native TLS. If so it checks the ALPN header for protocol determination (just like how the existing system uses the type-byte marker). * 2 new kinds of protocols are exclusively decoded via this native TLS mechanism: one for ferrying "packet" operations (udp-like) from the gossip layer and one for "stream" operations (tcp-like). The packet operations re-use sockets (using length-prefixing) to cut down on TLS re-negotiation overhead. * the server instances specially wrap the `memberlist.NetTransport` when running with gateway federation enabled (in a `wanfed.Transport`). The general gist is that if it tries to dial a node in the SAME datacenter (deduced by looking at the suffix of the node name) there is no change. If dialing a DIFFERENT datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh gateways to eventually end up in a server's :8300 port. * a new flag when launching a mesh gateway via `consul connect envoy` to indicate that the servers are to be exposed. This sets a special service meta when registering the gateway into the catalog. * `proxycfg/xds` notice this metadata blob to activate additional watches for the FederationState objects as well as the location of all of the consul servers in that datacenter. * `xds:` if the extra metadata is in place additional clusters are defined in a DC to bulk sink all traffic to another DC's gateways. For the current datacenter we listen on a wildcard name (`server.<dc>.consul`) that load balances all servers as well as one mini-cluster per node (`<node>.server.<dc>.consul`) * the `consul tls cert create` command got a new flag (`-node`) to help create an additional SAN in certs that can be used with this flavor of federation.
2020-03-09 20:59:02 +00:00
defer a2.Shutdown()
testrpc.WaitForTestAgent(t, a2.RPC, "dc2")
Rename NewTestAgentWithFields to StartTestAgent This function now only starts the agent. Using: git grep -l 'StartTestAgent(t, true,' | \ xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'
2020-03-31 20:24:39 +00:00
a3 := StartTestAgent(t, TestAgent{Name: "bonnie", HCL: `
wan federation via mesh gateways (#6884) This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch: There are several distinct chunks of code that are affected: * new flags and config options for the server * retry join WAN is slightly different * retry join code is shared to discover primary mesh gateways from secondary datacenters * because retry join logic runs in the *agent* and the results of that operation for primary mesh gateways are needed in the *server* there are some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur at multiple layers of abstraction just to pass the data down to the right layer. * new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers * the function signature for RPC dialing picked up a new required field (the node name of the destination) * several new RPCs for manipulating a FederationState object: `FederationState:{Apply,Get,List,ListMeshGateways}` * 3 read-only internal APIs for debugging use to invoke those RPCs from curl * raft and fsm changes to persist these FederationStates * replication for FederationStates as they are canonically stored in the Primary and replicated to the Secondaries. * a special derivative of anti-entropy that runs in secondaries to snapshot their local mesh gateway `CheckServiceNodes` and sync them into their upstream FederationState in the primary (this works in conjunction with the replication to distribute addresses for all mesh gateways in all DCs to all other DCs) * a "gateway locator" convenience object to make use of this data to choose the addresses of gateways to use for any given RPC or gossip operation to a remote DC. This gets data from the "retry join" logic in the agent and also directly calls into the FSM. * RPC (`:8300`) on the server sniffs the first byte of a new connection to determine if it's actually doing native TLS. If so it checks the ALPN header for protocol determination (just like how the existing system uses the type-byte marker). * 2 new kinds of protocols are exclusively decoded via this native TLS mechanism: one for ferrying "packet" operations (udp-like) from the gossip layer and one for "stream" operations (tcp-like). The packet operations re-use sockets (using length-prefixing) to cut down on TLS re-negotiation overhead. * the server instances specially wrap the `memberlist.NetTransport` when running with gateway federation enabled (in a `wanfed.Transport`). The general gist is that if it tries to dial a node in the SAME datacenter (deduced by looking at the suffix of the node name) there is no change. If dialing a DIFFERENT datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh gateways to eventually end up in a server's :8300 port. * a new flag when launching a mesh gateway via `consul connect envoy` to indicate that the servers are to be exposed. This sets a special service meta when registering the gateway into the catalog. * `proxycfg/xds` notice this metadata blob to activate additional watches for the FederationState objects as well as the location of all of the consul servers in that datacenter. * `xds:` if the extra metadata is in place additional clusters are defined in a DC to bulk sink all traffic to another DC's gateways. For the current datacenter we listen on a wildcard name (`server.<dc>.consul`) that load balances all servers as well as one mini-cluster per node (`<node>.server.<dc>.consul`) * the `consul tls cert create` command got a new flag (`-node`) to help create an additional SAN in certs that can be used with this flavor of federation.
2020-03-09 20:59:02 +00:00
domain = "consul"
node_name = "bonnie"
datacenter = "dc3"
primary_datacenter = "dc1"
# tls
ca_file = "../test/hostname/CertAuth.crt"
cert_file = "../test/hostname/Bonnie.crt"
key_file = "../test/hostname/Bonnie.key"
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
ports {
Convert the remaining calls to NewTestAgentWithFields After removing the t.Name() parameter with sed, convert the last few tests which use a custom name to call NewTestAgentWithFields instead.
2020-03-31 20:12:33 +00:00
server = ` + strconv.Itoa(secondaryRPCPorts[1]) + `
wan federation via mesh gateways (#6884) This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch: There are several distinct chunks of code that are affected: * new flags and config options for the server * retry join WAN is slightly different * retry join code is shared to discover primary mesh gateways from secondary datacenters * because retry join logic runs in the *agent* and the results of that operation for primary mesh gateways are needed in the *server* there are some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur at multiple layers of abstraction just to pass the data down to the right layer. * new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers * the function signature for RPC dialing picked up a new required field (the node name of the destination) * several new RPCs for manipulating a FederationState object: `FederationState:{Apply,Get,List,ListMeshGateways}` * 3 read-only internal APIs for debugging use to invoke those RPCs from curl * raft and fsm changes to persist these FederationStates * replication for FederationStates as they are canonically stored in the Primary and replicated to the Secondaries. * a special derivative of anti-entropy that runs in secondaries to snapshot their local mesh gateway `CheckServiceNodes` and sync them into their upstream FederationState in the primary (this works in conjunction with the replication to distribute addresses for all mesh gateways in all DCs to all other DCs) * a "gateway locator" convenience object to make use of this data to choose the addresses of gateways to use for any given RPC or gossip operation to a remote DC. This gets data from the "retry join" logic in the agent and also directly calls into the FSM. * RPC (`:8300`) on the server sniffs the first byte of a new connection to determine if it's actually doing native TLS. If so it checks the ALPN header for protocol determination (just like how the existing system uses the type-byte marker). * 2 new kinds of protocols are exclusively decoded via this native TLS mechanism: one for ferrying "packet" operations (udp-like) from the gossip layer and one for "stream" operations (tcp-like). The packet operations re-use sockets (using length-prefixing) to cut down on TLS re-negotiation overhead. * the server instances specially wrap the `memberlist.NetTransport` when running with gateway federation enabled (in a `wanfed.Transport`). The general gist is that if it tries to dial a node in the SAME datacenter (deduced by looking at the suffix of the node name) there is no change. If dialing a DIFFERENT datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh gateways to eventually end up in a server's :8300 port. * a new flag when launching a mesh gateway via `consul connect envoy` to indicate that the servers are to be exposed. This sets a special service meta when registering the gateway into the catalog. * `proxycfg/xds` notice this metadata blob to activate additional watches for the FederationState objects as well as the location of all of the consul servers in that datacenter. * `xds:` if the extra metadata is in place additional clusters are defined in a DC to bulk sink all traffic to another DC's gateways. For the current datacenter we listen on a wildcard name (`server.<dc>.consul`) that load balances all servers as well as one mini-cluster per node (`<node>.server.<dc>.consul`) * the `consul tls cert create` command got a new flag (`-node`) to help create an additional SAN in certs that can be used with this flavor of federation.
2020-03-09 20:59:02 +00:00
}
# wanfed
Convert the remaining calls to NewTestAgentWithFields After removing the t.Name() parameter with sed, convert the last few tests which use a custom name to call NewTestAgentWithFields instead.
2020-03-31 20:12:33 +00:00
primary_gateways = ["` + gwAddr + `"]
wan federation via mesh gateways (#6884) This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch: There are several distinct chunks of code that are affected: * new flags and config options for the server * retry join WAN is slightly different * retry join code is shared to discover primary mesh gateways from secondary datacenters * because retry join logic runs in the *agent* and the results of that operation for primary mesh gateways are needed in the *server* there are some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur at multiple layers of abstraction just to pass the data down to the right layer. * new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers * the function signature for RPC dialing picked up a new required field (the node name of the destination) * several new RPCs for manipulating a FederationState object: `FederationState:{Apply,Get,List,ListMeshGateways}` * 3 read-only internal APIs for debugging use to invoke those RPCs from curl * raft and fsm changes to persist these FederationStates * replication for FederationStates as they are canonically stored in the Primary and replicated to the Secondaries. * a special derivative of anti-entropy that runs in secondaries to snapshot their local mesh gateway `CheckServiceNodes` and sync them into their upstream FederationState in the primary (this works in conjunction with the replication to distribute addresses for all mesh gateways in all DCs to all other DCs) * a "gateway locator" convenience object to make use of this data to choose the addresses of gateways to use for any given RPC or gossip operation to a remote DC. This gets data from the "retry join" logic in the agent and also directly calls into the FSM. * RPC (`:8300`) on the server sniffs the first byte of a new connection to determine if it's actually doing native TLS. If so it checks the ALPN header for protocol determination (just like how the existing system uses the type-byte marker). * 2 new kinds of protocols are exclusively decoded via this native TLS mechanism: one for ferrying "packet" operations (udp-like) from the gossip layer and one for "stream" operations (tcp-like). The packet operations re-use sockets (using length-prefixing) to cut down on TLS re-negotiation overhead. * the server instances specially wrap the `memberlist.NetTransport` when running with gateway federation enabled (in a `wanfed.Transport`). The general gist is that if it tries to dial a node in the SAME datacenter (deduced by looking at the suffix of the node name) there is no change. If dialing a DIFFERENT datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh gateways to eventually end up in a server's :8300 port. * a new flag when launching a mesh gateway via `consul connect envoy` to indicate that the servers are to be exposed. This sets a special service meta when registering the gateway into the catalog. * `proxycfg/xds` notice this metadata blob to activate additional watches for the FederationState objects as well as the location of all of the consul servers in that datacenter. * `xds:` if the extra metadata is in place additional clusters are defined in a DC to bulk sink all traffic to another DC's gateways. For the current datacenter we listen on a wildcard name (`server.<dc>.consul`) that load balances all servers as well as one mini-cluster per node (`<node>.server.<dc>.consul`) * the `consul tls cert create` command got a new flag (`-node`) to help create an additional SAN in certs that can be used with this flavor of federation.
2020-03-09 20:59:02 +00:00
connect {
enabled = true
enable_mesh_gateway_wan_federation = true
}
Convert the remaining calls to NewTestAgentWithFields After removing the t.Name() parameter with sed, convert the last few tests which use a custom name to call NewTestAgentWithFields instead.
2020-03-31 20:12:33 +00:00
`})
wan federation via mesh gateways (#6884) This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch: There are several distinct chunks of code that are affected: * new flags and config options for the server * retry join WAN is slightly different * retry join code is shared to discover primary mesh gateways from secondary datacenters * because retry join logic runs in the *agent* and the results of that operation for primary mesh gateways are needed in the *server* there are some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur at multiple layers of abstraction just to pass the data down to the right layer. * new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers * the function signature for RPC dialing picked up a new required field (the node name of the destination) * several new RPCs for manipulating a FederationState object: `FederationState:{Apply,Get,List,ListMeshGateways}` * 3 read-only internal APIs for debugging use to invoke those RPCs from curl * raft and fsm changes to persist these FederationStates * replication for FederationStates as they are canonically stored in the Primary and replicated to the Secondaries. * a special derivative of anti-entropy that runs in secondaries to snapshot their local mesh gateway `CheckServiceNodes` and sync them into their upstream FederationState in the primary (this works in conjunction with the replication to distribute addresses for all mesh gateways in all DCs to all other DCs) * a "gateway locator" convenience object to make use of this data to choose the addresses of gateways to use for any given RPC or gossip operation to a remote DC. This gets data from the "retry join" logic in the agent and also directly calls into the FSM. * RPC (`:8300`) on the server sniffs the first byte of a new connection to determine if it's actually doing native TLS. If so it checks the ALPN header for protocol determination (just like how the existing system uses the type-byte marker). * 2 new kinds of protocols are exclusively decoded via this native TLS mechanism: one for ferrying "packet" operations (udp-like) from the gossip layer and one for "stream" operations (tcp-like). The packet operations re-use sockets (using length-prefixing) to cut down on TLS re-negotiation overhead. * the server instances specially wrap the `memberlist.NetTransport` when running with gateway federation enabled (in a `wanfed.Transport`). The general gist is that if it tries to dial a node in the SAME datacenter (deduced by looking at the suffix of the node name) there is no change. If dialing a DIFFERENT datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh gateways to eventually end up in a server's :8300 port. * a new flag when launching a mesh gateway via `consul connect envoy` to indicate that the servers are to be exposed. This sets a special service meta when registering the gateway into the catalog. * `proxycfg/xds` notice this metadata blob to activate additional watches for the FederationState objects as well as the location of all of the consul servers in that datacenter. * `xds:` if the extra metadata is in place additional clusters are defined in a DC to bulk sink all traffic to another DC's gateways. For the current datacenter we listen on a wildcard name (`server.<dc>.consul`) that load balances all servers as well as one mini-cluster per node (`<node>.server.<dc>.consul`) * the `consul tls cert create` command got a new flag (`-node`) to help create an additional SAN in certs that can be used with this flavor of federation.
2020-03-09 20:59:02 +00:00
defer a3.Shutdown()
testrpc.WaitForTestAgent(t, a3.RPC, "dc3")
// The primary_gateways config setting should cause automatic mesh join.
// Assert that the secondaries have joined the primary.
findPrimary := func(r *retry.R, a *TestAgent) *serf.Member {
var primary *serf.Member
for _, m := range a.WANMembers() {
if m.Tags["dc"] == "dc1" {
require.Nil(r, primary, "already found one node in dc1")
primary = &m
}
}
require.NotNil(r, primary)
return primary
}
retry.Run(t, func(r *retry.R) {
p2, p3 := findPrimary(r, a2), findPrimary(r, a3)
require.Equal(r, "bob.dc1", p2.Name)
require.Equal(r, "bob.dc1", p3.Name)
})
testrpc.WaitForLeader(t, a2.RPC, "dc2")
testrpc.WaitForLeader(t, a3.RPC, "dc3")
// Now we can register this into the catalog in dc2 and dc3.
{
args := &structs.ServiceDefinition{
Kind: structs.ServiceKindMeshGateway,
ID: "mesh-gateway",
Name: "mesh-gateway",
Meta: map[string]string{structs.MetaWANFederationKey: "1"},
Port: gwPort[0],
}
req, err := http.NewRequest("PUT", "/v1/agent/service/register", jsonReader(args))
require.NoError(t, err)
obj, err := a2.srv.AgentRegisterService(nil, req)
require.NoError(t, err)
require.Nil(t, obj)
}
{
args := &structs.ServiceDefinition{
Kind: structs.ServiceKindMeshGateway,
ID: "mesh-gateway",
Name: "mesh-gateway",
Meta: map[string]string{structs.MetaWANFederationKey: "1"},
Port: gwPort[0],
}
req, err := http.NewRequest("PUT", "/v1/agent/service/register", jsonReader(args))
require.NoError(t, err)
obj, err := a3.srv.AgentRegisterService(nil, req)
require.NoError(t, err)
require.Nil(t, obj)
}
// Wait until federation state replication functions
waitForFederationState(t, a1, "dc1")
waitForFederationState(t, a1, "dc2")
waitForFederationState(t, a1, "dc3")
waitForFederationState(t, a2, "dc1")
waitForFederationState(t, a2, "dc2")
waitForFederationState(t, a2, "dc3")
waitForFederationState(t, a3, "dc1")
waitForFederationState(t, a3, "dc2")
waitForFederationState(t, a3, "dc3")
retry.Run(t, func(r *retry.R) {
require.NotEmpty(r, a1.PickRandomMeshGatewaySuitableForDialing("dc1"))
require.NotEmpty(r, a1.PickRandomMeshGatewaySuitableForDialing("dc2"))
require.NotEmpty(r, a1.PickRandomMeshGatewaySuitableForDialing("dc3"))
require.NotEmpty(r, a2.PickRandomMeshGatewaySuitableForDialing("dc1"))
require.NotEmpty(r, a2.PickRandomMeshGatewaySuitableForDialing("dc2"))
require.NotEmpty(r, a2.PickRandomMeshGatewaySuitableForDialing("dc3"))
require.NotEmpty(r, a3.PickRandomMeshGatewaySuitableForDialing("dc1"))
require.NotEmpty(r, a3.PickRandomMeshGatewaySuitableForDialing("dc2"))
require.NotEmpty(r, a3.PickRandomMeshGatewaySuitableForDialing("dc3"))
})
retry.Run(t, func(r *retry.R) {
if got, want := len(a1.WANMembers()), 3; got != want {
r.Fatalf("got %d WAN members want at least %d", got, want)
}
if got, want := len(a2.WANMembers()), 3; got != want {
r.Fatalf("got %d WAN members want at least %d", got, want)
}
if got, want := len(a3.WANMembers()), 3; got != want {
r.Fatalf("got %d WAN members want at least %d", got, want)
}
})
// Ensure we can do some trivial RPC in all directions.
agents := map[string]*TestAgent{"dc1": a1, "dc2": a2, "dc3": a3}
names := map[string]string{"dc1": "bob", "dc2": "betty", "dc3": "bonnie"}
for _, srcDC := range []string{"dc1", "dc2", "dc3"} {
a := agents[srcDC]
for _, dstDC := range []string{"dc1", "dc2", "dc3"} {
if srcDC == dstDC {
continue
}
t.Run(srcDC+" to "+dstDC, func(t *testing.T) {
req, err := http.NewRequest("GET", "/v1/catalog/nodes?dc="+dstDC, nil)
require.NoError(t, err)
resp := httptest.NewRecorder()
obj, err := a.srv.CatalogNodes(resp, req)
require.NoError(t, err)
require.NotNil(t, obj)
nodes, ok := obj.(structs.Nodes)
require.True(t, ok)
require.Len(t, nodes, 1)
node := nodes[0]
require.Equal(t, dstDC, node.Datacenter)
require.Equal(t, names[dstDC], node.Node)
})
}
}
}
Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.
2020-06-10 20:47:35 +00:00
func TestAutoConfig_Integration(t *testing.T) {
// eventually this test should really live with integration tests
// the goal here is to have one test server and another test client
// spin up both agents and allow the server to authorize the auto config
Agent Auto Config: Implement Certificate Generation (#8360) Most of the groundwork was laid in previous PRs between adding the cert-monitor package to extracting the logic of signing certificates out of the connect_ca_endpoint.go code and into a method on the server. This also refactors the auto-config package a bit to split things out into multiple files.
2020-07-28 19:31:48 +00:00
// request and then see the client joined. Finally we force a CA roots
// update and wait to see that the agents TLS certificate gets updated.
Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.
2020-06-10 20:47:35 +00:00
cfgDir := testutil.TempDir(t, "auto-config")
// write some test TLS certificates out to the cfg dir
cert, key, cacert, err := testTLSCertificates("server.dc1.consul")
require.NoError(t, err)
certFile := filepath.Join(cfgDir, "cert.pem")
caFile := filepath.Join(cfgDir, "cacert.pem")
keyFile := filepath.Join(cfgDir, "key.pem")
require.NoError(t, ioutil.WriteFile(certFile, []byte(cert), 0600))
require.NoError(t, ioutil.WriteFile(caFile, []byte(cacert), 0600))
require.NoError(t, ioutil.WriteFile(keyFile, []byte(key), 0600))
// generate a gossip key
gossipKey := make([]byte, 32)
n, err := rand.Read(gossipKey)
require.NoError(t, err)
require.Equal(t, 32, n)
gossipKeyEncoded := base64.StdEncoding.EncodeToString(gossipKey)
// generate the JWT signing keys
pub, priv, err := oidcauthtest.GenerateKey()
require.NoError(t, err)
hclConfig := TestACLConfigWithParams(nil) + `
encrypt = "` + gossipKeyEncoded + `"
encrypt_verify_incoming = true
encrypt_verify_outgoing = true
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
ca_file = "` + caFile + `"
cert_file = "` + certFile + `"
key_file = "` + keyFile + `"
connect { enabled = true }
auto_config {
Change auto config authorizer to allow for future extension The envisioned changes would allow extra settings to enable dynamically defined auth methods to be used instead of or in addition to the statically defined one in the configuration.
2020-06-18 15:16:57 +00:00
authorization {
Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.
2020-06-10 20:47:35 +00:00
enabled = true
Change auto config authorizer to allow for future extension The envisioned changes would allow extra settings to enable dynamically defined auth methods to be used instead of or in addition to the statically defined one in the configuration.
2020-06-18 15:16:57 +00:00
static {
claim_mappings = {
consul_node_name = "node"
}
claim_assertions = [
"value.node == \"${node}\""
]
bound_issuer = "consul"
bound_audiences = [
"consul"
]
jwt_validation_pub_keys = ["` + strings.ReplaceAll(pub, "\n", "\\n") + `"]
Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.
2020-06-10 20:47:35 +00:00
}
}
}
`
srv := StartTestAgent(t, TestAgent{Name: "TestAgent-Server", HCL: hclConfig})
defer srv.Shutdown()
testrpc.WaitForTestAgent(t, srv.RPC, "dc1", testrpc.WithToken(TestDefaultMasterToken))
// sign a JWT token
now := time.Now()
token, err := oidcauthtest.SignJWT(priv, jwt.Claims{
Subject: "consul",
Issuer: "consul",
Audience: jwt.Audience{"consul"},
NotBefore: jwt.NewNumericDate(now.Add(-1 * time.Second)),
Expiry: jwt.NewNumericDate(now.Add(5 * time.Minute)),
}, map[string]interface{}{
"consul_node_name": "test-client",
})
require.NoError(t, err)
Ensure certificates retrieved through the cache get persisted with auto-config (#8409)
2020-07-30 15:37:18 +00:00
client := StartTestAgent(t, TestAgent{Name: "test-client",
Overrides: `
connect {
test_ca_leaf_root_change_spread = "1ns"
}
`,
HCL: `
bootstrap = false
server = false
ca_file = "` + caFile + `"
verify_outgoing = true
verify_server_hostname = true
node_name = "test-client"
ports {
server = ` + strconv.Itoa(srv.Config.RPCBindAddr.Port) + `
}
auto_config {
enabled = true
intro_token = "` + token + `"
server_addresses = ["` + srv.Config.RPCBindAddr.String() + `"]
}`,
})
Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.
2020-06-10 20:47:35 +00:00
defer client.Shutdown()
Agent Auto Config: Implement Certificate Generation (#8360) Most of the groundwork was laid in previous PRs between adding the cert-monitor package to extracting the logic of signing certificates out of the connect_ca_endpoint.go code and into a method on the server. This also refactors the auto-config package a bit to split things out into multiple files.
2020-07-28 19:31:48 +00:00
retry.Run(t, func(r *retry.R) {
require.NotNil(r, client.Agent.tlsConfigurator.Cert())
})
Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.
2020-06-10 20:47:35 +00:00
// when this is successful we managed to get the gossip key and serf addresses to bind to
// and then connect. Additionally we would have to have certificates or else the
// verify_incoming config on the server would not let it work.
testrpc.WaitForTestAgent(t, client.RPC, "dc1", testrpc.WithToken(TestDefaultMasterToken))
Agent Auto Config: Implement Certificate Generation (#8360) Most of the groundwork was laid in previous PRs between adding the cert-monitor package to extracting the logic of signing certificates out of the connect_ca_endpoint.go code and into a method on the server. This also refactors the auto-config package a bit to split things out into multiple files.
2020-07-28 19:31:48 +00:00
// grab the existing cert
cert1 := client.Agent.tlsConfigurator.Cert()
require.NotNil(t, cert1)
// force a roots rotation by updating the CA config
t.Logf("Forcing roots rotation on the server")
ca := connect.TestCA(t, nil)
req := &structs.CARequest{
Datacenter: "dc1",
WriteRequest: structs.WriteRequest{Token: TestDefaultMasterToken},
Config: &structs.CAConfiguration{
Provider: "consul",
Config: map[string]interface{}{
"LeafCertTTL": "1h",
"PrivateKey": ca.SigningKey,
"RootCert": ca.RootCert,
"RotationPeriod": "6h",
"IntermediateCertTTL": "3h",
},
},
}
var reply interface{}
require.NoError(t, srv.RPC("ConnectCA.ConfigurationSet", &req, &reply))
// ensure that a new cert gets generated and pushed into the TLS configurator
retry.Run(t, func(r *retry.R) {
require.NotEqual(r, cert1, client.Agent.tlsConfigurator.Cert())
Ensure certificates retrieved through the cache get persisted with auto-config (#8409)
2020-07-30 15:37:18 +00:00
// check that the on disk certs match expectations
data, err := ioutil.ReadFile(filepath.Join(client.DataDir, "auto-config.json"))
require.NoError(r, err)
rdr := strings.NewReader(string(data))
var resp pbautoconf.AutoConfigResponse
pbUnmarshaler := &jsonpb.Unmarshaler{
AllowUnknownFields: false,
}
require.NoError(r, pbUnmarshaler.Unmarshal(rdr, &resp), "data: %s", data)
actual, err := tls.X509KeyPair([]byte(resp.Certificate.CertPEM), []byte(resp.Certificate.PrivateKeyPEM))
require.NoError(r, err)
require.Equal(r, client.Agent.tlsConfigurator.Cert(), &actual)
Agent Auto Config: Implement Certificate Generation (#8360) Most of the groundwork was laid in previous PRs between adding the cert-monitor package to extracting the logic of signing certificates out of the connect_ca_endpoint.go code and into a method on the server. This also refactors the auto-config package a bit to split things out into multiple files.
2020-07-28 19:31:48 +00:00
})
Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.
2020-06-10 20:47:35 +00:00
// spot check that we now have an ACL token
require.NotEmpty(t, client.tokens.AgentToken())
}
Add an AutoEncrypt “integration” test Also fix a bug where Consul could segfault if TLS was enabled but no client certificate was provided. How no one has reported this as a problem I am not sure.
2020-06-30 17:52:57 +00:00
func TestAgent_AutoEncrypt(t *testing.T) {
// eventually this test should really live with integration tests
// the goal here is to have one test server and another test client
// spin up both agents and allow the server to authorize the auto encrypt
// request and then see the client get a TLS certificate
cfgDir := testutil.TempDir(t, "auto-encrypt")
// write some test TLS certificates out to the cfg dir
cert, key, cacert, err := testTLSCertificates("server.dc1.consul")
require.NoError(t, err)
certFile := filepath.Join(cfgDir, "cert.pem")
caFile := filepath.Join(cfgDir, "cacert.pem")
keyFile := filepath.Join(cfgDir, "key.pem")
require.NoError(t, ioutil.WriteFile(certFile, []byte(cert), 0600))
require.NoError(t, ioutil.WriteFile(caFile, []byte(cacert), 0600))
require.NoError(t, ioutil.WriteFile(keyFile, []byte(key), 0600))
hclConfig := TestACLConfigWithParams(nil) + `
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
ca_file = "` + caFile + `"
cert_file = "` + certFile + `"
key_file = "` + keyFile + `"
connect { enabled = true }
auto_encrypt { allow_tls = true }
`
srv := StartTestAgent(t, TestAgent{Name: "test-server", HCL: hclConfig})
defer srv.Shutdown()
testrpc.WaitForTestAgent(t, srv.RPC, "dc1", testrpc.WithToken(TestDefaultMasterToken))
client := StartTestAgent(t, TestAgent{Name: "test-client", HCL: TestACLConfigWithParams(nil) + `
bootstrap = false
server = false
ca_file = "` + caFile + `"
verify_outgoing = true
verify_server_hostname = true
node_name = "test-client"
auto_encrypt {
tls = true
}
ports {
server = ` + strconv.Itoa(srv.Config.RPCBindAddr.Port) + `
}
retry_join = ["` + srv.Config.SerfBindAddrLAN.String() + `"]`,
UseTLS: true,
})
defer client.Shutdown()
// when this is successful we managed to get a TLS certificate and are using it for
// encrypted RPC connections.
testrpc.WaitForTestAgent(t, client.RPC, "dc1", testrpc.WithToken(TestDefaultMasterToken))
// now we need to validate that our certificate has the correct CN
aeCert := client.tlsConfigurator.Cert()
require.NotNil(t, aeCert)
id := connect.SpiffeIDAgent{
Host: connect.TestClusterID + ".consul",
Datacenter: "dc1",
Agent: "test-client",
}
expectedCN := connect.AgentCN("test-client", connect.TestClusterID)
x509Cert, err := x509.ParseCertificate(aeCert.Certificate[0])
require.NoError(t, err)
require.Equal(t, expectedCN, x509Cert.Subject.CommonName)
require.Len(t, x509Cert.URIs, 1)
require.Equal(t, id.URI(), x509Cert.URIs[0])
}