496 Commits (262897469260cc6319ea9369518e20ac2fed3178)

Author	SHA1	Message	Date
Daniel Nephin	9f03e23e44	Merge pull request #9881 from hashicorp/dnephin/state-index-service-check-nodes ... state: convert services.node and checks.node indexes	4 years ago
Daniel Nephin	bd6332ae25	Merge pull request #9863 from hashicorp/dnephin/config-entry-kind-name ... state: move ConfigEntryKindName	4 years ago
Christopher Broglie	f0307c73e5	Add support for configuring TLS ServerName for health checks ... Some TLS servers require SNI, but the Golang HTTP client doesn't include it in the ClientHello when connecting to an IP address. This change adds a new TLSServerName field to health check definitions to optionally set it. This fixes #9473.	4 years ago
Daniel Nephin	34eb6c01ff	state: convert services.node and checks.node indexes ... Using NodeIdentity to share the indexes with both.	4 years ago
freddygv	4bdbcff9c0	Fixup upstream test	4 years ago
freddygv	6090cfcf68	PR comments	4 years ago
freddygv	7df846aa24	Pass MeshGateway config in service config request ... ResolveServiceConfig is called by service manager before the proxy registration is in the catalog. Therefore we should pass proxy registration flags in the request rather than trying to fetch them from the state store (where they may not exist yet).	4 years ago
freddygv	8b46d8dcbb	Restore old Envoy prefix on escape hatches ... This is done because after removing ID and NodeName from ServiceConfigRequest we will no longer know whether a request coming in is for a Consul client earlier than v1.10.	4 years ago
freddygv	93c3c1780d	Only lowercase the protocol when normalizing	4 years ago
freddygv	41b2ba1e58	Add omitempty across the board for UpstreamConfig	4 years ago
freddygv	756ab4c546	Fixup protobufs and tests	4 years ago
freddygv	df1f3995f8	Update service manager to store centrally configured upstreams	4 years ago
freddygv	6fd30d0384	Add TransparentProxy opt to proxy definition	4 years ago
freddygv	306ef7d252	Restore old escape hatch alias	4 years ago
freddygv	e3dc2a49df	Turn Limits and PassiveHealthChecks into pointers	4 years ago
freddygv	acec711a6a	Update server-side config resolution and client-side merging	4 years ago
freddygv	1710ec87d2	finish moving UpstreamConfig and related fields to structs pkg	4 years ago
Daniel Nephin	b06b3dd8f8	state: move ConfigEntryKindName ... Previously this type was defined in structs, but unlike the other types in structs this type is not used by RPC requests. By moving it to state we can better indicate that this is not an API type, but part of the state implementation.	4 years ago
Daniel Nephin	71b0f0a7a6	structs: remove EnterpriseMeta.GetNamespace ... I added this recently without realizing that the method already existed and was named NamespaceOrEmpty. Replace all calls to GetNamespace with NamespaceOrEmpty or NamespaceOrDefault as appropriate.	4 years ago
freddygv	87cde19b4c	Create new types for service-defaults upstream cfg	4 years ago
Daniel Nephin	5c8a6311b6	Merge pull request #9703 from pierresouchay/streaming_tags_and_case_insensitive ... Streaming filter tags + case insensitive lookups for Service Names	4 years ago
John Cowen	5892e75452	ui: Remove any trailing fullstop/period DNS characters from Gateways UI API (#9752) ... Previous to this commit, the API response would include Gateway Addresses in the form `domain.name.:8080`, which due to the addition of the port is probably not the expected response. This commit rightTrims any `.` characters from the end of the domain before formatting the address to include the port resulting in `domain.name:8080`	4 years ago
Mark Anderson	b9d22f48cd	Add fields to the /acl/auth-methods endpoint. (#9741) ... * A GET of the /acl/auth-method/:name endpoint returns the fields MaxTokenTTL and TokenLocality, while a LIST (/acl/auth-methods) does not. The list command returns a filtered subset of the full set. This is somewhat deliberate, so that secrets aren't shown, but the TTL and Locality fields aren't (IMO) security critical, and it is useful for the front end to be able to show them. For consistency these changes mirror the 'omit empty' and string representation choices made for the GET call. This includes changes to the gRPC and API code in the client. The new output looks similar to this curl 'http://localhost:8500/v1/acl/auth-methods' | jq '.' { "MaxTokenTTL": "8m20s", "Name": "minikube-ttl-local2", "Type": "kubernetes", "Description": "minikube auth method", "TokenLocality": "local", "CreateIndex": 530, "ModifyIndex": 530, "Namespace": "default" } ] Signed-off-by: Mark Anderson <manderson@hashicorp.com> * Add changelog Signed-off-by: Mark Anderson <manderson@hashicorp.com>	4 years ago
Daniel Nephin	d1772ae305	structs: rename EnterpriseMeta constructor ... To match the Go convention.	4 years ago
Pierre Souchay	6f91085869	Use lower case for serviceName computation of cache keys	4 years ago
R.B. Boyer	03790a1f91	server: add OSS stubs supporting validation of source namespaces in service-intentions config entries (#9527)	4 years ago
Daniel Nephin	29ce5ec575	structs: fix caching of ServiceSpecificRequest when ingress=true ... The field was not being included in the cache info key. This would result in a DNS request for web.service.consul returning the same result as web.ingress.consul, when those results should not be the same.	4 years ago
kevinkengne	2e7e78999d	add completeness test for types with CacheInfo method (#9480) ... include all fields when fuzzing in tests split tests by struct type Ensure the new value for the field is different fuzzer.Fuzz could produce the same value again in some cases. Use a custom fuzz function for QueryOptions. That type is an embedded struct in the request types but only one of the fields is important to include in the cache key. Move enterpriseMetaField to an oss file so that we can change it in enterprise.	4 years ago
Daniel Nephin	2187808803	structs: add tests for String() methods ... To show that printing one of these IDs works properly now that the String() method receiver is no longer a pointer.	4 years ago
Daniel Nephin	d113f0e690	structs: Fix printing of IDs ... These types are used as values (not pointers) in other structs. Using a pointer receiver causes problems when the value is printed. fmt will not call the String method if it is passed a value and the String method has a pointer receiver. By using a value receiver the correct string is printed. Also remove some unused methods.	4 years ago
Matt Keeler	85e5da53d5	Special case the error returned when we have a Raft leader but are not tracking it in the ServerLookup (#9487) ... This can happen when one other node in the cluster such as a client is unable to communicate with the leader server and sees it as failed. When that happens its failing status eventually gets propagated to the other servers in the cluster and eventually this can result in RPCs returning “No cluster leader” error. That error is misleading and unhelpful for determing the root cause of the issue as its not raft stability but rather and client -> server networking issue. Therefore this commit will add a new error that will be returned in that case to differentiate between the two cases.	4 years ago
R.B. Boyer	d921690bfe	acl: global tokens created by auth methods now correctly replicate to secondary datacenters (#9351) ... Previously the tokens would fail to insert into the secondary's state store because the AuthMethod field of the ACLToken did not point to a known auth method from the primary.	4 years ago
Kyle Havlovitz	c4eff420be	Merge pull request #9009 from hashicorp/update-secondary-ca ... connect: Fix an issue with updating CA config in a secondary datacenter	4 years ago
Kyle Havlovitz	0bfda4481f	Add CA server delegate interface for testing	4 years ago
Kyle Havlovitz	9be7c6401c	connect: update some function comments in CA manager	4 years ago
Matt Keeler	66fd23d67f	Refactor to call non-voting servers read replicas (#9191) ... Co-authored-by: Kit Patella <kit@jepsen.io>	4 years ago
R.B. Boyer	c7233ba871	server: remove config entry CAS in legacy intention API bridge code (#9151) ... Change so line-item intention edits via the API are handled via the state store instead of via CAS operations. Fixes #9143	4 years ago
R.B. Boyer	c003871c54	server: break up Intention.Apply monolithic method (#9007) ... The Intention.Apply RPC is quite large, so this PR attempts to break it down into smaller functions and dissolves the pre-config-entry approach to the breakdown as it only confused things.	4 years ago
Matt Keeler	c048e86bb2	Switch to using the external autopilot module	4 years ago
Daniel Nephin	b532e092dc	structs: add a namespace test for CheckServiceNode.CanRead	4 years ago
Daniel Nephin	c42fe5ae43	subscribe: set the request namespace	4 years ago
Daniel Nephin	a5dd2001cf	stream: remove Event.Key ... Makes Payload a type with FilterByKey so that Payloads can implement filtering by key. With this approach we don't need to expose a Namespace field on Event, and we don't need to invest micro formats or require a bunch of code to be aware of exactly how the key field is encoded.	4 years ago
R.B. Boyer	58387fef0a	server: config entry replication now correctly uses namespaces in comparisons (#9024) ... Previously config entries sharing a kind & name but in different namespaces could occasionally cause "stuck states" in replication because the namespace fields were ignored during the differential comparison phase. Example: Two config entries written to the primary: kind=A,name=web,namespace=bar kind=A,name=web,namespace=foo Under the covers these both get saved to memdb, so they are sorted by all 3 components (kind,name,namespace) during natural iteration. This means that before the replication code does it's own incomplete sort, the underlying data IS sorted by namespace ascending (bar comes before foo). After one pass of replication the primary and secondary datacenters have the same set of config entries present. If "kind=A,name=web,namespace=bar" were to be deleted, then things get weird. Before replication the two sides look like: primary: [ kind=A,name=web,namespace=foo ] secondary: [ kind=A,name=web,namespace=bar kind=A,name=web,namespace=foo ] The differential comparison phase walks these two lists in sorted order and first compares "kind=A,name=web,namespace=foo" vs "kind=A,name=web,namespace=bar" and falsely determines they are the SAME and are thus cause an update of "kind=A,name=web,namespace=foo". Then it compares "<nothing>" with "kind=A,name=web,namespace=foo" and falsely determines that the latter should be DELETED. During reconciliation the deletes are processed before updates, and so for a brief moment in the secondary "kind=A,name=web,namespace=foo" is erroneously deleted and then immediately restored. Unfortunately after this replication phase the final state is identical to the initial state, so when it loops around again (rate limited) it repeats the same set of operations indefinitely.	4 years ago
Freddy	9c04cbc40f	Add HasExact to topology endpoint (#9010)	4 years ago
Daniel Nephin	8b601fdcac	Merge pull request #8771 from amenzhinsky/fix-grpc-use-tls-mapping ... Fix GRPCUseTLS flag HTTP API mapping	4 years ago
s-christoff	9bb348c6c7	Enhance the output of consul snapshot inspect (#8787)	4 years ago
Freddy	13df5d5bf8	Add protocol to the topology endpoint response (#8868)	4 years ago
Daniel Nephin	da6400192b	Merge pull request #8768 from hashicorp/streaming/add-subscribe-service ... subscribe: add subscribe service for streaming change events	4 years ago
Freddy	da91e999f6	Return intention info in svc topology endpoint (#8853)	4 years ago
Daniel Nephin	21c21191f4	structs: add CheckServiceNode.CanRead ... And use it from the subscribe endpoint.	4 years ago
R.B. Boyer	1b413b0444	connect: support defining intentions using layer 7 criteria (#8839) ... Extend Consul’s intentions model to allow for request-based access control enforcement for HTTP-like protocols in addition to the existing connection-based enforcement for unspecified protocols (e.g. tcp).	4 years ago
R.B. Boyer	a2a8e9c783	connect: intentions are now managed as a new config entry kind "service-intentions" (#8834) ... - Upgrade the ConfigEntry.ListAll RPC to be kind-aware so that older copies of consul will not see new config entries it doesn't understand replicate down. - Add shim conversion code so that the old API/CLI method of interacting with intentions will continue to work so long as none of these are edited via config entry endpoints. Almost all of the read-only APIs will continue to function indefinitely. - Add new APIs that operate on individual intentions without IDs so that the UI doesn't need to implement CAS operations. - Add a new serf feature flag indicating support for intentions-as-config-entries. - The old line-item intentions way of interacting with the state store will transparently flip between the legacy memdb table and the config entry representations so that readers will never see a hiccup during migration where the results are incomplete. It uses a piece of system metadata to control the flip. - The primary datacenter will begin migrating intentions into config entries on startup once all servers in the datacenter are on a version of Consul with the intentions-as-config-entries feature flag. When it is complete the old state store representations will be cleared. We also record a piece of system metadata indicating this has occurred. We use this metadata to skip ALL of this code the next time the leader starts up. - The secondary datacenters continue to run the old intentions replicator until all servers in the secondary DC and primary DC support intentions-as-config-entries (via serf flag). Once this condition it met the old intentions replicator ceases. - The secondary datacenters replicate the new config entries as they are migrated in the primary. When they detect that the primary has zeroed it's old state store table it waits until all config entries up to that point are replicated and then zeroes its own copy of the old state store table. We also record a piece of system metadata indicating this has occurred. We use this metadata to skip ALL of this code the next time the leader starts up.	4 years ago
R.B. Boyer	4998a08c56	server: create new memdb table for storing system metadata (#8703) ... This adds a new very tiny memdb table and corresponding raft operation for updating a very small effective map[string]string collection of "system metadata". This can persistently record a fact about the Consul state machine itself. The first use of this feature will come in a later PR.	4 years ago
freddygv	413a894a1a	Do not evaluate discovery chain for topology upstreams	4 years ago
freddygv	cf7b7fcdd6	Single DB txn for ServiceTopology and other PR comments	4 years ago
freddygv	f906b94351	Add func to combine up+downstream queries	4 years ago
freddygv	b012d8374e	support querying upstreams/downstreams from registrations	4 years ago
Aliaksandr Mianzhynski	74cfba7065	Fix GRPCUseTLS flag HTTP API mapping	4 years ago
freddygv	9fa1b13df9	Resolve conflicts	4 years ago
R.B. Boyer	0fb088aac3	agent: make the json/hcl decoding of ConnectProxyConfig fully work with CamelCase and snake_case (#8741) ... Fixes #7418	4 years ago
Paul Banks	7d58901ae8	Fix bad int -> string conversions caught by go vet changes in 1.15 (#8739)	4 years ago
Kyle Havlovitz	35bb09f85c	Merge pull request #8646 from hashicorp/common-intermediate-ttl ... Move IntermediateCertTTL to common CA config	4 years ago
freddygv	7fd518ff1d	Merge master	4 years ago
freddygv	7b9d1b41d5	Resolve conflicts against master	4 years ago
freddygv	768dbaa68d	Add session flag to cookie config	4 years ago
freddygv	eab90ea9fa	Revert EnvoyConfig nesting	4 years ago
Kyle Havlovitz	2f7210bde2	Move IntermediateCertTTL to common CA config	4 years ago
freddygv	cd4cf5161f	Update resolver defaulting	4 years ago
freddygv	eaa250cc80	Ensure resolver node with LB isn't considered default	4 years ago
freddygv	ef877449ce	Move valid policies to pkg level	4 years ago
freddygv	f81fe6a1a1	Remove LB infix and move injection to xds	4 years ago
R.B. Boyer	119e945c3e	connect: all config entries pick up a meta field (#8596) ... Fixes #8595	4 years ago
R.B. Boyer	d0f74cd1e8	connect: fix bug in preventing some namespaced config entry modifications (#8601) ... Whenever an upsert/deletion of a config entry happens, within the open state store transaction we speculatively test compile all discovery chains that may be affected by the pending modification to verify that the write would not create an erroneous scenario (such as splitting traffic to a subset that did not exist). If a single discovery chain evaluation references two config entries with the same kind and name in different namespaces then sometimes the upsert/deletion would be falsely rejected. It does not appear as though this bug would've let invalid writes through to the state store so the correction does not require a cleanup phase.	4 years ago
freddygv	63f79e5f9b	Restructure structs and other PR comments	4 years ago
freddygv	0236e169bb	Add documentation for resolver LB cfg	4 years ago
freddygv	81115b6eaa	Compile down LB policy to disco chain nodes	4 years ago
freddygv	ff56a64b08	Add LB policy to service-resolver	4 years ago
Jack	9e1c6727f9	Add http2 and grpc support to ingress gateways (#8458)	4 years ago
R.B. Boyer	74d5df7c7a	xds: use envoy's rbac filter to handle intentions entirely within envoy (#8569)	4 years ago
Daniel Nephin	07ad662131	Fix panic when decoding 'Connect: null' ... Surprisingly the json Unmarshal updates the aux pointer to a nil.	4 years ago
Hans Hasselberg	a932aafc91	add primary keys to list keyring (#8522) ... During gossip encryption key rotation it would be nice to be able to see if all nodes are using the same key. This PR adds another field to the json response from `GET v1/operator/keyring` which lists the primary keys in use per dc. That way an operator can tell when a key was successfully setup as primary key. Based on https://github.com/hashicorp/serf/pull/611 to add primary key to list keyring output: ```json [ { "WAN": true, "Datacenter": "dc2", "Segment": "", "Keys": { "0OuM4oC3Os18OblWiBbZUaHA7Hk+tNs/6nhNYtaNduM=": 6, "SINm887hKTzmMWeBNKTJReaTLX3mBEJKriDyt88Ad+g=": 6 }, "PrimaryKeys": { "SINm887hKTzmMWeBNKTJReaTLX3mBEJKriDyt88Ad+g=": 6 }, "NumNodes": 6 }, { "WAN": false, "Datacenter": "dc2", "Segment": "", "Keys": { "0OuM4oC3Os18OblWiBbZUaHA7Hk+tNs/6nhNYtaNduM=": 8, "SINm887hKTzmMWeBNKTJReaTLX3mBEJKriDyt88Ad+g=": 8 }, "PrimaryKeys": { "SINm887hKTzmMWeBNKTJReaTLX3mBEJKriDyt88Ad+g=": 8 }, "NumNodes": 8 }, { "WAN": false, "Datacenter": "dc1", "Segment": "", "Keys": { "0OuM4oC3Os18OblWiBbZUaHA7Hk+tNs/6nhNYtaNduM=": 3, "SINm887hKTzmMWeBNKTJReaTLX3mBEJKriDyt88Ad+g=": 8 }, "PrimaryKeys": { "SINm887hKTzmMWeBNKTJReaTLX3mBEJKriDyt88Ad+g=": 8 }, "NumNodes": 8 } ] ``` I intentionally did not change the CLI output because I didn't find a good way of displaying this information. There are a couple of options that we could implement later: * add a flag to show the primary keys * add a flag to show json output Fixes #3393.	4 years ago
R.B. Boyer	e3cd4a8539	connect: use stronger validation that ingress gateways have compatible protocols defined for their upstreams (#8470) ... Fixes #8466 Since Consul 1.8.0 there was a bug in how ingress gateway protocol compatibility was enforced. At the point in time that an ingress-gateway config entry was modified the discovery chain for each upstream was checked to ensure the ingress gateway protocol matched. Unfortunately future modifications of other config entries were not validated against existing ingress-gateway definitions, such as: 1. create tcp ingress-gateway pointing to 'api' (ok) 2. create service-defaults for 'api' setting protocol=http (worked, but not ok) 3. create service-splitter or service-router for 'api' (worked, but caused an agent panic) If you were to do these in a different order, it would fail without a crash: 1. create service-defaults for 'api' setting protocol=http (ok) 2. create service-splitter or service-router for 'api' (ok) 3. create tcp ingress-gateway pointing to 'api' (fail with message about protocol mismatch) This PR introduces the missing validation. The two new behaviors are: 1. create tcp ingress-gateway pointing to 'api' (ok) 2. (NEW) create service-defaults for 'api' setting protocol=http ("ok" for back compat) 3. (NEW) create service-splitter or service-router for 'api' (fail with message about protocol mismatch) In consideration for any existing users that may be inadvertently be falling into item (2) above, that is now officiall a valid configuration to be in. For anyone falling into item (3) above while you cannot use the API to manufacture that scenario anymore, anyone that has old (now bad) data will still be able to have the agent use them just enough to generate a new agent/proxycfg error message rather than a panic. Unfortunately we just don't have enough information to properly fix the config entries.	4 years ago
Kyle Havlovitz	c39a275666	fsm: Fix snapshot bug with restoring node/service/check indexes	4 years ago
freddygv	aa6c59dbfc	end to end changes to pass gatewayservices to /ui/services/	4 years ago
Matt Keeler	2ee9fe0a4d	Move generation of the CA Configuration from the agent code into a method on the RuntimeConfig (#8363) ... This allows this to be reused elsewhere.	4 years ago
Matt Keeler	2713c0e682	Refactor the agentpb package (#8362) ... First move the whole thing to the top-level proto package name. Secondly change some things around internally to have sub-packages.	4 years ago
Daniel Nephin	5d36f98710	agent/consul: Add support for NotModified to two endpoints ... A query made with AllowNotModifiedResponse and a MinIndex, where the result has the same Index as MinIndex, will return an empty response with QueryMeta.NotModified set to true. Co-authored-by: Pierre Souchay <pierresouchay@users.noreply.github.com>	4 years ago
Yury Evtikhov	10361dd210	DNS: add IsErrQueryNotFound function for easier error evaluation	4 years ago
Yury Evtikhov	8d18422f19	DNS: fix agent returning SERVFAIL where NXDOMAIN should be returned	4 years ago
R.B. Boyer	462f0f37ed	connect: various changes to make namespaces for intentions work more like for other subsystems (#8194) ... Highlights: - add new endpoint to query for intentions by exact match - using this endpoint from the CLI instead of the dump+filter approach - enforcing that OSS can only read/write intentions with a SourceNS or DestinationNS field of "default". - preexisting OSS intentions with now-invalid namespace fields will delete those intentions on initial election or for wildcard namespaces an attempt will be made to downgrade them to "default" unless one exists. - also allow the '-namespace' CLI arg on all of the intention subcommands - update lots of docs	4 years ago
Freddy	10d6e9c458	Split up unused key validation for oss/ent (#8189) ... Split up unused key validation in config entry decode for oss/ent. This is needed so that we can return an informative error in OSS if namespaces are provided.	4 years ago
Chris Piraino	df48db0abd	Merge pull request #7932 from hashicorp/ingress/internal-ui-endpoint-multiple-ports ... Update gateway-services-nodes API endpoint to allow multiple addresses	4 years ago
Chris Piraino	f213d3592a	remove obsolete comments about test parallelization	4 years ago
Chris Piraino	b3db907bdf	Update gateway-services-nodes API endpoint to allow multiple addresses ... Previously, we were only returning a single ListenerPort for a single service. However, we actually allow a single service to be serviced over multiple ports, as well as allow users to define what hostnames they expect their services to be contacted over. When no hosts are defined, we return the default ingress domain for any configured DNS domain. To show this in the UI, we modify the gateway-services-nodes API to return a GatewayConfig.Addresses field, which is a list of addresses over which the specific service can be contacted.	4 years ago
freddygv	c791fbc79c	Update namespaces subject-verb agreement	4 years ago
freddygv	044d027ff8	Remove break	4 years ago
freddygv	70810b0602	Let users know namespaces are ent only in config entry decode	4 years ago
Daniel Nephin	068b43df90	Enable gofmt simplify ... Code changes done automatically with 'gofmt -s -w'	5 years ago
Matt Keeler	d3881dd754	ACL Node Identities (#7970) ... A Node Identity is very similar to a service identity. Its main targeted use is to allow creating tokens for use by Consul agents that will grant the necessary permissions for all the typical agent operations (node registration, coordinate updates, anti-entropy). Half of this commit is for golden file based tests of the acl token and role cli output. Another big updates was to refactor many of the tests in agent/consul/acl_endpoint_test.go to use the same style of tests and the same helpers. Besides being less boiler plate in the tests it also uses a common way of starting a test server with ACLs that should operate without any warnings regarding deprecated non-uuid master tokens etc.	5 years ago
freddygv	cd927eed5e	Remove unused method and fixup docs ref	5 years ago