You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
consul/api/api_test.go

1003 lines
24 KiB

package api
import (
crand "crypto/rand"
"crypto/tls"
"fmt"
"io/ioutil"
"net"
"net/http"
"os"
"path/filepath"
"reflect"
"runtime"
"strings"
"testing"
"time"
"github.com/hashicorp/consul/sdk/testutil"
"github.com/hashicorp/consul/sdk/testutil/retry"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
type configCallback func(c *Config)
func makeClient(t *testing.T) (*Client, *testutil.TestServer) {
return makeClientWithConfig(t, nil, nil)
}
func makeClientWithoutConnect(t *testing.T) (*Client, *testutil.TestServer) {
return makeClientWithConfig(t, nil, func(serverConfig *testutil.TestServerConfig) {
serverConfig.Connect = nil
})
}
func makeACLClient(t *testing.T) (*Client, *testutil.TestServer) {
return makeClientWithConfig(t, func(clientConfig *Config) {
clientConfig.Token = "root"
}, func(serverConfig *testutil.TestServerConfig) {
New ACLs (#4791) This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week. Description At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers. On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though. Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though. All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management. Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are: A server running the new system must still support other clients using the legacy system. A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system. The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode. So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.
6 years ago
serverConfig.PrimaryDatacenter = "dc1"
serverConfig.ACL.Tokens.Master = "root"
serverConfig.ACL.Tokens.Agent = "root"
New ACLs (#4791) This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week. Description At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers. On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though. Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though. All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management. Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are: A server running the new system must still support other clients using the legacy system. A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system. The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode. So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.
6 years ago
serverConfig.ACL.Enabled = true
serverConfig.ACL.DefaultPolicy = "deny"
})
}
func makeClientWithConfig(
t *testing.T,
cb1 configCallback,
cb2 testutil.ServerConfigCallback) (*Client, *testutil.TestServer) {
// Make client config
conf := DefaultConfig()
if cb1 != nil {
cb1(conf)
}
// Create server
var server *testutil.TestServer
var err error
retry.RunWith(retry.ThreeTimes(), t, func(r *retry.R) {
server, err = testutil.NewTestServerConfigT(t, cb2)
if err != nil {
testutil: NewLogBuffer - buffer logs until a test fails Replaces #7559 Running tests in parallel, with background goroutines, results in test output not being associated with the correct test. `go test` does not make any guarantees about output from goroutines being attributed to the correct test case. Attaching log output from background goroutines also cause data races. If the goroutine outlives the test, it will race with the test being marked done. Previously this was noticed as a panic when logging, but with the race detector enabled it is shown as a data race. The previous solution did not address the problem of correct test attribution because test output could still be hidden when it was associated with a test that did not fail. You would have to look at all of the log output to find the relevant lines. It also made debugging test failures more difficult because each log line was very long. This commit attempts a new approach. Instead of printing all the logs, only print when a test fails. This should work well when there are a small number of failures, but may not work well when there are many test failures at the same time. In those cases the failures are unlikely a result of a specific test, and the log output is likely less useful. All of the logs are printed from the test goroutine, so they should be associated with the correct test. Also removes some test helpers that were not used, or only had a single caller. Packages which expose many functions with similar names can be difficult to use correctly. Related: https://github.com/golang/go/issues/38458 (may be fixed in go1.15) https://github.com/golang/go/issues/38382#issuecomment-612940030
5 years ago
r.Fatalf("Failed to start server: %v", err.Error())
}
})
if server.Config.Bootstrap {
server.WaitForLeader(t)
8 years ago
}
conf.Address = server.HTTPAddr
// Create client
client, err := NewClient(conf)
if err != nil {
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
7 years ago
server.Stop()
t.Fatalf("err: %v", err)
}
return client, server
}
func testKey() string {
buf := make([]byte, 16)
if _, err := crand.Read(buf); err != nil {
panic(fmt.Errorf("Failed to read random bytes: %v", err))
}
return fmt.Sprintf("%08x-%04x-%04x-%04x-%12x",
buf[0:4],
buf[4:6],
buf[6:8],
buf[8:10],
buf[10:16])
}
func testNodeServiceCheckRegistrations(t *testing.T, client *Client, datacenter string) {
t.Helper()
registrations := map[string]*CatalogRegistration{
"Node foo": {
Datacenter: datacenter,
Node: "foo",
ID: "e0155642-135d-4739-9853-a1ee6c9f945b",
Address: "127.0.0.2",
TaggedAddresses: map[string]string{
"lan": "127.0.0.2",
"wan": "198.18.0.2",
},
NodeMeta: map[string]string{
"env": "production",
"os": "linux",
},
Checks: HealthChecks{
&HealthCheck{
Node: "foo",
CheckID: "foo:alive",
Name: "foo-liveness",
Status: HealthPassing,
Notes: "foo is alive and well",
},
&HealthCheck{
Node: "foo",
CheckID: "foo:ssh",
Name: "foo-remote-ssh",
Status: HealthPassing,
Notes: "foo has ssh access",
},
},
},
"Service redis v1 on foo": {
Datacenter: datacenter,
Node: "foo",
SkipNodeUpdate: true,
Service: &AgentService{
Kind: ServiceKindTypical,
ID: "redisV1",
Service: "redis",
Tags: []string{"v1"},
Meta: map[string]string{"version": "1"},
Port: 1234,
Address: "198.18.1.2",
},
Checks: HealthChecks{
&HealthCheck{
Node: "foo",
CheckID: "foo:redisV1",
Name: "redis-liveness",
Status: HealthPassing,
Notes: "redis v1 is alive and well",
ServiceID: "redisV1",
ServiceName: "redis",
},
},
},
"Service redis v2 on foo": {
Datacenter: datacenter,
Node: "foo",
SkipNodeUpdate: true,
Service: &AgentService{
Kind: ServiceKindTypical,
ID: "redisV2",
Service: "redis",
Tags: []string{"v2"},
Meta: map[string]string{"version": "2"},
Port: 1235,
Address: "198.18.1.2",
},
Checks: HealthChecks{
&HealthCheck{
Node: "foo",
CheckID: "foo:redisV2",
Name: "redis-v2-liveness",
Status: HealthPassing,
Notes: "redis v2 is alive and well",
ServiceID: "redisV2",
ServiceName: "redis",
},
},
},
"Node bar": {
Datacenter: datacenter,
Node: "bar",
ID: "c6e7a976-8f4f-44b5-bdd3-631be7e8ecac",
Address: "127.0.0.3",
TaggedAddresses: map[string]string{
"lan": "127.0.0.3",
"wan": "198.18.0.3",
},
NodeMeta: map[string]string{
"env": "production",
"os": "windows",
},
Checks: HealthChecks{
&HealthCheck{
Node: "bar",
CheckID: "bar:alive",
Name: "bar-liveness",
Status: HealthPassing,
Notes: "bar is alive and well",
},
},
},
"Service redis v1 on bar": {
Datacenter: datacenter,
Node: "bar",
SkipNodeUpdate: true,
Service: &AgentService{
Kind: ServiceKindTypical,
ID: "redisV1",
Service: "redis",
Tags: []string{"v1"},
Meta: map[string]string{"version": "1"},
Port: 1234,
Address: "198.18.1.3",
},
Checks: HealthChecks{
&HealthCheck{
Node: "bar",
CheckID: "bar:redisV1",
Name: "redis-liveness",
Status: HealthPassing,
Notes: "redis v1 is alive and well",
ServiceID: "redisV1",
ServiceName: "redis",
},
},
},
"Service web v1 on bar": {
Datacenter: datacenter,
Node: "bar",
SkipNodeUpdate: true,
Service: &AgentService{
Kind: ServiceKindTypical,
ID: "webV1",
Service: "web",
Tags: []string{"v1", "connect"},
Meta: map[string]string{"version": "1", "connect": "enabled"},
Port: 443,
Address: "198.18.1.4",
Connect: &AgentServiceConnect{Native: true},
},
Checks: HealthChecks{
&HealthCheck{
Node: "bar",
CheckID: "bar:web:v1",
Name: "web-v1-liveness",
Status: HealthPassing,
Notes: "web connect v1 is alive and well",
ServiceID: "webV1",
ServiceName: "web",
},
},
},
"Node baz": {
Datacenter: datacenter,
Node: "baz",
ID: "12f96b27-a7b0-47bd-add7-044a2bfc7bfb",
Address: "127.0.0.4",
TaggedAddresses: map[string]string{
"lan": "127.0.0.4",
},
NodeMeta: map[string]string{
"env": "qa",
"os": "linux",
},
Checks: HealthChecks{
&HealthCheck{
Node: "baz",
CheckID: "baz:alive",
Name: "baz-liveness",
Status: HealthPassing,
Notes: "baz is alive and well",
},
&HealthCheck{
Node: "baz",
CheckID: "baz:ssh",
Name: "baz-remote-ssh",
Status: HealthPassing,
Notes: "baz has ssh access",
},
},
},
"Service web v1 on baz": {
Datacenter: datacenter,
Node: "baz",
SkipNodeUpdate: true,
Service: &AgentService{
Kind: ServiceKindTypical,
ID: "webV1",
Service: "web",
Tags: []string{"v1", "connect"},
Meta: map[string]string{"version": "1", "connect": "enabled"},
Port: 443,
Address: "198.18.1.4",
Connect: &AgentServiceConnect{Native: true},
},
Checks: HealthChecks{
&HealthCheck{
Node: "baz",
CheckID: "baz:web:v1",
Name: "web-v1-liveness",
Status: HealthPassing,
Notes: "web connect v1 is alive and well",
ServiceID: "webV1",
ServiceName: "web",
},
},
},
"Service web v2 on baz": {
Datacenter: datacenter,
Node: "baz",
SkipNodeUpdate: true,
Service: &AgentService{
Kind: ServiceKindTypical,
ID: "webV2",
Service: "web",
Tags: []string{"v2", "connect"},
Meta: map[string]string{"version": "2", "connect": "enabled"},
Port: 8443,
Address: "198.18.1.4",
Connect: &AgentServiceConnect{Native: true},
},
Checks: HealthChecks{
&HealthCheck{
Node: "baz",
CheckID: "baz:web:v2",
Name: "web-v2-liveness",
Status: HealthPassing,
Notes: "web connect v2 is alive and well",
ServiceID: "webV2",
ServiceName: "web",
},
},
},
"Service critical on baz": {
Datacenter: datacenter,
Node: "baz",
SkipNodeUpdate: true,
Service: &AgentService{
Kind: ServiceKindTypical,
ID: "criticalV2",
Service: "critical",
Tags: []string{"v2"},
Meta: map[string]string{"version": "2"},
Port: 8080,
Address: "198.18.1.4",
},
Checks: HealthChecks{
&HealthCheck{
Node: "baz",
CheckID: "baz:critical:v2",
Name: "critical-v2-liveness",
Status: HealthCritical,
Notes: "critical v2 is in the critical state",
ServiceID: "criticalV2",
ServiceName: "critical",
},
},
},
"Service warning on baz": {
Datacenter: datacenter,
Node: "baz",
SkipNodeUpdate: true,
Service: &AgentService{
Kind: ServiceKindTypical,
ID: "warningV2",
Service: "warning",
Tags: []string{"v2"},
Meta: map[string]string{"version": "2"},
Port: 8081,
Address: "198.18.1.4",
},
Checks: HealthChecks{
&HealthCheck{
Node: "baz",
CheckID: "baz:warning:v2",
Name: "warning-v2-liveness",
Status: HealthWarning,
Notes: "warning v2 is in the warning state",
ServiceID: "warningV2",
ServiceName: "warning",
},
},
},
}
catalog := client.Catalog()
for name, reg := range registrations {
_, err := catalog.Register(reg, nil)
require.NoError(t, err, "Failed catalog registration for %q: %v", name, err)
}
}
func TestAPI_DefaultConfig_env(t *testing.T) {
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
7 years ago
// t.Parallel() // DO NOT ENABLE !!!
// do not enable t.Parallel for this test since it modifies global state
// (environment) which has non-deterministic effects on the other tests
// which derive their default configuration from the environment
addr := "1.2.3.4:5678"
token := "abcd1234"
auth := "username:password"
os.Setenv(HTTPAddrEnvName, addr)
defer os.Setenv(HTTPAddrEnvName, "")
os.Setenv(HTTPTokenEnvName, token)
defer os.Setenv(HTTPTokenEnvName, "")
os.Setenv(HTTPAuthEnvName, auth)
defer os.Setenv(HTTPAuthEnvName, "")
os.Setenv(HTTPSSLEnvName, "1")
defer os.Setenv(HTTPSSLEnvName, "")
os.Setenv(HTTPCAFile, "ca.pem")
defer os.Setenv(HTTPCAFile, "")
os.Setenv(HTTPCAPath, "certs/")
defer os.Setenv(HTTPCAPath, "")
os.Setenv(HTTPClientCert, "client.crt")
defer os.Setenv(HTTPClientCert, "")
os.Setenv(HTTPClientKey, "client.key")
defer os.Setenv(HTTPClientKey, "")
os.Setenv(HTTPTLSServerName, "consul.test")
defer os.Setenv(HTTPTLSServerName, "")
os.Setenv(HTTPSSLVerifyEnvName, "0")
defer os.Setenv(HTTPSSLVerifyEnvName, "")
for i, config := range []*Config{
DefaultConfig(),
DefaultConfigWithLogger(testutil.Logger(t)),
DefaultNonPooledConfig(),
} {
if config.Address != addr {
t.Errorf("expected %q to be %q", config.Address, addr)
}
if config.Token != token {
t.Errorf("expected %q to be %q", config.Token, token)
}
if config.HttpAuth == nil {
t.Fatalf("expected HttpAuth to be enabled")
}
if config.HttpAuth.Username != "username" {
t.Errorf("expected %q to be %q", config.HttpAuth.Username, "username")
}
if config.HttpAuth.Password != "password" {
t.Errorf("expected %q to be %q", config.HttpAuth.Password, "password")
}
if config.Scheme != "https" {
t.Errorf("expected %q to be %q", config.Scheme, "https")
}
if config.TLSConfig.CAFile != "ca.pem" {
t.Errorf("expected %q to be %q", config.TLSConfig.CAFile, "ca.pem")
}
if config.TLSConfig.CAPath != "certs/" {
t.Errorf("expected %q to be %q", config.TLSConfig.CAPath, "certs/")
}
if config.TLSConfig.CertFile != "client.crt" {
t.Errorf("expected %q to be %q", config.TLSConfig.CertFile, "client.crt")
}
if config.TLSConfig.KeyFile != "client.key" {
t.Errorf("expected %q to be %q", config.TLSConfig.KeyFile, "client.key")
}
if config.TLSConfig.Address != "consul.test" {
t.Errorf("expected %q to be %q", config.TLSConfig.Address, "consul.test")
}
if !config.TLSConfig.InsecureSkipVerify {
t.Errorf("expected SSL verification to be off")
}
// Use keep alives as a check for whether pooling is on or off.
if pooled := i != 2; pooled {
if config.Transport.DisableKeepAlives != false {
t.Errorf("expected keep alives to be enabled")
}
} else {
if config.Transport.DisableKeepAlives != true {
t.Errorf("expected keep alives to be disabled")
}
}
}
}
func TestAPI_SetupTLSConfig(t *testing.T) {
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
7 years ago
t.Parallel()
// A default config should result in a clean default client config.
tlsConfig := &TLSConfig{}
cc, err := SetupTLSConfig(tlsConfig)
if err != nil {
t.Fatalf("err: %v", err)
}
expected := &tls.Config{RootCAs: cc.RootCAs}
if !reflect.DeepEqual(cc, expected) {
t.Fatalf("bad: \n%v, \n%v", cc, expected)
}
// Try some address variations with and without ports.
tlsConfig.Address = "127.0.0.1"
cc, err = SetupTLSConfig(tlsConfig)
if err != nil {
t.Fatalf("err: %v", err)
}
expected.ServerName = "127.0.0.1"
if !reflect.DeepEqual(cc, expected) {
t.Fatalf("bad: %v", cc)
}
tlsConfig.Address = "127.0.0.1:80"
cc, err = SetupTLSConfig(tlsConfig)
if err != nil {
t.Fatalf("err: %v", err)
}
expected.ServerName = "127.0.0.1"
if !reflect.DeepEqual(cc, expected) {
t.Fatalf("bad: %v", cc)
}
tlsConfig.Address = "demo.consul.io:80"
cc, err = SetupTLSConfig(tlsConfig)
if err != nil {
t.Fatalf("err: %v", err)
}
expected.ServerName = "demo.consul.io"
if !reflect.DeepEqual(cc, expected) {
t.Fatalf("bad: %v", cc)
}
tlsConfig.Address = "[2001:db8:a0b:12f0::1]"
cc, err = SetupTLSConfig(tlsConfig)
if err != nil {
t.Fatalf("err: %v", err)
}
expected.ServerName = "[2001:db8:a0b:12f0::1]"
if !reflect.DeepEqual(cc, expected) {
t.Fatalf("bad: %v", cc)
}
tlsConfig.Address = "[2001:db8:a0b:12f0::1]:80"
cc, err = SetupTLSConfig(tlsConfig)
if err != nil {
t.Fatalf("err: %v", err)
}
expected.ServerName = "2001:db8:a0b:12f0::1"
if !reflect.DeepEqual(cc, expected) {
t.Fatalf("bad: %v", cc)
}
// Skip verification.
tlsConfig.InsecureSkipVerify = true
cc, err = SetupTLSConfig(tlsConfig)
if err != nil {
t.Fatalf("err: %v", err)
}
expected.InsecureSkipVerify = true
if !reflect.DeepEqual(cc, expected) {
t.Fatalf("bad: %v", cc)
}
// Make a new config that hits all the file parsers.
tlsConfig = &TLSConfig{
CertFile: "../test/hostname/Alice.crt",
KeyFile: "../test/hostname/Alice.key",
CAFile: "../test/hostname/CertAuth.crt",
}
cc, err = SetupTLSConfig(tlsConfig)
if err != nil {
t.Fatalf("err: %v", err)
}
if len(cc.Certificates) != 1 {
t.Fatalf("missing certificate: %v", cc.Certificates)
}
if cc.RootCAs == nil {
t.Fatalf("didn't load root CAs")
}
// Use a directory to load the certs instead
cc, err = SetupTLSConfig(&TLSConfig{
CAPath: "../test/ca_path",
})
if err != nil {
t.Fatalf("err: %v", err)
}
if len(cc.RootCAs.Subjects()) != 2 {
t.Fatalf("didn't load root CAs")
}
// Load certs in-memory
certPEM, err := ioutil.ReadFile("../test/hostname/Alice.crt")
if err != nil {
t.Fatalf("err: %v", err)
}
keyPEM, err := ioutil.ReadFile("../test/hostname/Alice.key")
if err != nil {
t.Fatalf("err: %v", err)
}
caPEM, err := ioutil.ReadFile("../test/hostname/CertAuth.crt")
if err != nil {
t.Fatalf("err: %v", err)
}
// Setup config with in-memory certs
cc, err = SetupTLSConfig(&TLSConfig{
CertPEM: certPEM,
KeyPEM: keyPEM,
CAPem: caPEM,
})
if err != nil {
t.Fatalf("err: %v", err)
}
if len(cc.Certificates) != 1 {
t.Fatalf("missing certificate: %v", cc.Certificates)
}
if cc.RootCAs == nil {
t.Fatalf("didn't load root CAs")
}
}
func TestAPI_ClientTLSOptions(t *testing.T) {
t.Parallel()
// Start a server that verifies incoming HTTPS connections
_, srvVerify := makeClientWithConfig(t, nil, func(conf *testutil.TestServerConfig) {
conf.CAFile = "../test/client_certs/rootca.crt"
conf.CertFile = "../test/client_certs/server.crt"
conf.KeyFile = "../test/client_certs/server.key"
conf.VerifyIncomingHTTPS = true
})
defer srvVerify.Stop()
// Start a server without VerifyIncomingHTTPS
_, srvNoVerify := makeClientWithConfig(t, nil, func(conf *testutil.TestServerConfig) {
conf.CAFile = "../test/client_certs/rootca.crt"
conf.CertFile = "../test/client_certs/server.crt"
conf.KeyFile = "../test/client_certs/server.key"
conf.VerifyIncomingHTTPS = false
})
defer srvNoVerify.Stop()
// Client without a cert
t.Run("client without cert, validation", func(t *testing.T) {
client, err := NewClient(&Config{
Address: srvVerify.HTTPSAddr,
Scheme: "https",
TLSConfig: TLSConfig{
Address: "consul.test",
CAFile: "../test/client_certs/rootca.crt",
},
})
if err != nil {
t.Fatal(err)
}
// Should fail
_, err = client.Agent().Self()
if err == nil || !strings.Contains(err.Error(), "bad certificate") {
t.Fatal(err)
}
})
// Client with a valid cert
t.Run("client with cert, validation", func(t *testing.T) {
client, err := NewClient(&Config{
Address: srvVerify.HTTPSAddr,
Scheme: "https",
TLSConfig: TLSConfig{
Address: "consul.test",
CAFile: "../test/client_certs/rootca.crt",
CertFile: "../test/client_certs/client.crt",
KeyFile: "../test/client_certs/client.key",
},
})
if err != nil {
t.Fatal(err)
}
// Should succeed
_, err = client.Agent().Self()
if err != nil {
t.Fatal(err)
}
})
// Client without a cert
t.Run("client without cert, no validation", func(t *testing.T) {
client, err := NewClient(&Config{
Address: srvNoVerify.HTTPSAddr,
Scheme: "https",
TLSConfig: TLSConfig{
Address: "consul.test",
CAFile: "../test/client_certs/rootca.crt",
},
})
if err != nil {
t.Fatal(err)
}
// Should succeed
_, err = client.Agent().Self()
if err != nil {
t.Fatal(err)
}
})
// Client with a valid cert
t.Run("client with cert, no validation", func(t *testing.T) {
client, err := NewClient(&Config{
Address: srvNoVerify.HTTPSAddr,
Scheme: "https",
TLSConfig: TLSConfig{
Address: "consul.test",
CAFile: "../test/client_certs/rootca.crt",
CertFile: "../test/client_certs/client.crt",
KeyFile: "../test/client_certs/client.key",
},
})
if err != nil {
t.Fatal(err)
}
// Should succeed
_, err = client.Agent().Self()
if err != nil {
t.Fatal(err)
}
})
}
func TestAPI_SetQueryOptions(t *testing.T) {
t.Parallel()
c, s := makeClient(t)
defer s.Stop()
assert := assert.New(t)
r := c.newRequest("GET", "/v1/kv/foo")
q := &QueryOptions{
Namespace: "operator",
Datacenter: "foo",
AllowStale: true,
RequireConsistent: true,
WaitIndex: 1000,
WaitTime: 100 * time.Second,
Token: "12345",
Near: "nodex",
LocalOnly: true,
}
r.setQueryOptions(q)
if r.params.Get("ns") != "operator" {
t.Fatalf("bad: %v", r.params)
}
if r.params.Get("dc") != "foo" {
t.Fatalf("bad: %v", r.params)
}
if _, ok := r.params["stale"]; !ok {
t.Fatalf("bad: %v", r.params)
}
if _, ok := r.params["consistent"]; !ok {
t.Fatalf("bad: %v", r.params)
}
if r.params.Get("index") != "1000" {
t.Fatalf("bad: %v", r.params)
}
if r.params.Get("wait") != "100000ms" {
t.Fatalf("bad: %v", r.params)
}
if r.header.Get("X-Consul-Token") != "12345" {
t.Fatalf("bad: %v", r.header)
}
if r.params.Get("near") != "nodex" {
t.Fatalf("bad: %v", r.params)
}
if r.params.Get("local-only") != "true" {
t.Fatalf("bad: %v", r.params)
}
assert.Equal("", r.header.Get("Cache-Control"))
r = c.newRequest("GET", "/v1/kv/foo")
q = &QueryOptions{
UseCache: true,
MaxAge: 30 * time.Second,
StaleIfError: 345678 * time.Millisecond, // Fractional seconds should be rounded
}
r.setQueryOptions(q)
_, ok := r.params["cached"]
assert.True(ok)
assert.Equal("max-age=30, stale-if-error=346", r.header.Get("Cache-Control"))
}
func TestAPI_SetWriteOptions(t *testing.T) {
t.Parallel()
c, s := makeClient(t)
defer s.Stop()
r := c.newRequest("GET", "/v1/kv/foo")
q := &WriteOptions{
Namespace: "operator",
Datacenter: "foo",
Token: "23456",
}
r.setWriteOptions(q)
if r.params.Get("ns") != "operator" {
t.Fatalf("bad: %v", r.params)
}
if r.params.Get("dc") != "foo" {
t.Fatalf("bad: %v", r.params)
}
if r.header.Get("X-Consul-Token") != "23456" {
t.Fatalf("bad: %v", r.header)
}
}
func TestAPI_RequestToHTTP(t *testing.T) {
t.Parallel()
c, s := makeClient(t)
defer s.Stop()
r := c.newRequest("DELETE", "/v1/kv/foo")
q := &QueryOptions{
Datacenter: "foo",
}
r.setQueryOptions(q)
req, err := r.toHTTP()
if err != nil {
t.Fatalf("err: %v", err)
}
if req.Method != "DELETE" {
t.Fatalf("bad: %v", req)
}
if req.URL.RequestURI() != "/v1/kv/foo?dc=foo" {
t.Fatalf("bad: %v", req)
}
}
func TestAPI_ParseQueryMeta(t *testing.T) {
t.Parallel()
resp := &http.Response{
Header: make(map[string][]string),
}
resp.Header.Set("X-Consul-Index", "12345")
resp.Header.Set("X-Consul-LastContact", "80")
resp.Header.Set("X-Consul-KnownLeader", "true")
resp.Header.Set("X-Consul-Translate-Addresses", "true")
qm := &QueryMeta{}
if err := parseQueryMeta(resp, qm); err != nil {
t.Fatalf("err: %v", err)
}
if qm.LastIndex != 12345 {
t.Fatalf("Bad: %v", qm)
}
if qm.LastContact != 80*time.Millisecond {
t.Fatalf("Bad: %v", qm)
}
if !qm.KnownLeader {
t.Fatalf("Bad: %v", qm)
}
if !qm.AddressTranslationEnabled {
t.Fatalf("Bad: %v", qm)
}
}
func TestAPI_UnixSocket(t *testing.T) {
t.Parallel()
if runtime.GOOS == "windows" {
t.SkipNow()
}
tempDir := testutil.TempDir(t, "consul")
socket := filepath.Join(tempDir, "test.sock")
c, s := makeClientWithConfig(t, func(c *Config) {
c.Address = "unix://" + socket
}, func(c *testutil.TestServerConfig) {
c.Addresses = &testutil.TestAddressConfig{
HTTP: "unix://" + socket,
}
})
defer s.Stop()
agent := c.Agent()
info, err := agent.Self()
if err != nil {
t.Fatalf("err: %s", err)
}
if info["Config"]["NodeName"].(string) == "" {
t.Fatalf("bad: %v", info)
}
}
func TestAPI_durToMsec(t *testing.T) {
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
7 years ago
t.Parallel()
if ms := durToMsec(0); ms != "0ms" {
t.Fatalf("bad: %s", ms)
}
if ms := durToMsec(time.Millisecond); ms != "1ms" {
t.Fatalf("bad: %s", ms)
}
if ms := durToMsec(time.Microsecond); ms != "1ms" {
t.Fatalf("bad: %s", ms)
}
if ms := durToMsec(5 * time.Millisecond); ms != "5ms" {
t.Fatalf("bad: %s", ms)
}
}
func TestAPI_IsRetryableError(t *testing.T) {
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
7 years ago
t.Parallel()
if IsRetryableError(nil) {
t.Fatal("should not be a retryable error")
}
if IsRetryableError(fmt.Errorf("not the error you are looking for")) {
t.Fatal("should not be a retryable error")
}
if !IsRetryableError(fmt.Errorf(serverError)) {
t.Fatal("should be a retryable error")
}
if !IsRetryableError(&net.OpError{Err: fmt.Errorf("network conn error")}) {
t.Fatal("should be a retryable error")
}
}
func TestAPI_GenerateEnv(t *testing.T) {
t.Parallel()
c := &Config{
Address: "127.0.0.1:8500",
Token: "test",
TokenFile: "test.file",
Scheme: "http",
TLSConfig: TLSConfig{
CAFile: "",
CAPath: "",
CertFile: "",
KeyFile: "",
Address: "",
InsecureSkipVerify: true,
},
}
expected := []string{
"CONSUL_HTTP_ADDR=127.0.0.1:8500",
"CONSUL_HTTP_TOKEN=test",
"CONSUL_HTTP_TOKEN_FILE=test.file",
"CONSUL_HTTP_SSL=false",
"CONSUL_CACERT=",
"CONSUL_CAPATH=",
"CONSUL_CLIENT_CERT=",
"CONSUL_CLIENT_KEY=",
"CONSUL_TLS_SERVER_NAME=",
"CONSUL_HTTP_SSL_VERIFY=false",
"CONSUL_HTTP_AUTH=",
}
require.Equal(t, expected, c.GenerateEnv())
}
func TestAPI_GenerateEnvHTTPS(t *testing.T) {
t.Parallel()
c := &Config{
Address: "127.0.0.1:8500",
Token: "test",
TokenFile: "test.file",
Scheme: "https",
TLSConfig: TLSConfig{
CAFile: "/var/consul/ca.crt",
CAPath: "/var/consul/ca.dir",
CertFile: "/var/consul/server.crt",
KeyFile: "/var/consul/ssl/server.key",
Address: "127.0.0.1:8500",
InsecureSkipVerify: false,
},
HttpAuth: &HttpBasicAuth{
Username: "user",
Password: "password",
},
}
expected := []string{
"CONSUL_HTTP_ADDR=127.0.0.1:8500",
"CONSUL_HTTP_TOKEN=test",
"CONSUL_HTTP_TOKEN_FILE=test.file",
"CONSUL_HTTP_SSL=true",
"CONSUL_CACERT=/var/consul/ca.crt",
"CONSUL_CAPATH=/var/consul/ca.dir",
"CONSUL_CLIENT_CERT=/var/consul/server.crt",
"CONSUL_CLIENT_KEY=/var/consul/ssl/server.key",
"CONSUL_TLS_SERVER_NAME=127.0.0.1:8500",
"CONSUL_HTTP_SSL_VERIFY=true",
"CONSUL_HTTP_AUTH=user:password",
}
require.Equal(t, expected, c.GenerateEnv())
}