consul

Commit Graph

Author	SHA1	Message	Date
hc-github-team-consul-core	8d9fdc3a59	Backport of V1 Compat Exported Services Controller Optimizations into release/1.18.x (#20518 ) * V1 Compat Exported Services Controller Optimizations (#20517) V1 compat exported services controller optimizations * Don't start the v2 exported services controller in v1 mode. * Use the controller cache. * Trigger the V1 Compat exported-services Controller when V1 Config Entries are Updated (#20456) * Trigger the v1 compat exported-services controller when the v1 config entry is modified. * Hook up exported-services config entries to the event publisher. * Add tests to the v2 exported services shim. * Use the local materializer trigger updates on the v1 compat exported services controller when exported-services config entries are modified. * stop sleeping when context is cancelled --------- Co-authored-by: Eric Haberkorn <erichaberkorn@gmail.com>	10 months ago
Matt Keeler	326c0ecfbe	In-Memory gRPC (#19942 ) * Implement In-Process gRPC for use by controller caching/indexing This replaces the pipe base listener implementation we were previously using. The new style CAN avoid cloning resources which our controller caching/indexing is taking advantage of to not duplicate resource objects in memory. To maintain safety for controllers and for them to be able to modify data they get back from the cache and the resource service, the client they are presented in their runtime will be wrapped with an autogenerated client which clones request and response messages as they pass through the client. Another sizable change in this PR is to consolidate how server specific gRPC services get registered and managed. Before this was in a bunch of different methods and it was difficult to track down how gRPC services were registered. Now its all in one place. * Fix race in tests * Ensure the resource service is registered to the multiplexed handler for forwarding from client agents * Expose peer streaming on the internal handler	10 months ago
Semir Patel	219283d6ae	unhack: fix broken `make lint` on macbooks (#19611 ) Fix broken `make lint` on macbooks	1 year ago
Chris S. Kim	92ce814693	Remove old build tags (#19128 )	1 year ago
Dhia Ayachi	b1688ad856	Run copyright after running deep-copy as part of the Makefile/CI (#18741 ) * execute copyright headers after performing deep-copy generation. * fix copyright install * Apply suggestions from code review Co-authored-by: Semir Patel <semir.patel@hashicorp.com> * Apply suggestions from code review Co-authored-by: Semir Patel <semir.patel@hashicorp.com> * rename steps to match codegen naming * remove copywrite install category --------- Co-authored-by: Semir Patel <semir.patel@hashicorp.com>	1 year ago
Ashesh Vidyut	13eefbb840	Net 3181 consul gh issue 15709 allow log file naming like nomad - fix bug (#18631 ) * fixes file name for consul * added log file * added tests for rename method * append instead of trunc * fix file truncate issue * added changelog * fix for build destros ci * removed changelog * solaris	1 year ago
Ashesh Vidyut	d45c3c2755	NET-3181 - Allow log file naming like Nomad (#18617 ) * fixes file name for consul * added log file * added tests for rename method	1 year ago
hashicorp-copywrite[bot]	5fb9df1640	[COMPLIANCE] License changes (#18443 ) * Adding explicit MPL license for sub-package This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository. * Adding explicit MPL license for sub-package This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository. * Updating the license from MPL to Business Source License Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at <Blog URL>, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl. * add missing license headers * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 --------- Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>	1 year ago
Dan Upton	b9c485dcb8	Controller Supervision (#17016 )	2 years ago
Ronald	b64674623e	Copyright headers for missing files/folders (#16708 ) * copyright headers for agent folder	2 years ago
Nathan Coleman	e0f4f6c152	Run config entry controller routines on leader (#16054 )	2 years ago
Kyle Schochenmaier	bf0f61a878	removes ioutil usage everywhere which was deprecated in go1.16 (#15297 ) * update go version to 1.18 for api and sdk, go mod tidy * removes ioutil usage everywhere which was deprecated in go1.16 in favour of io and os packages. Also introduces a lint rule which forbids use of ioutil going forward. Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>	2 years ago
Chris S. Kim	bde57c0dd0	Regenerate files according to 1.19.2 formatter	2 years ago
Dan Upton	1c2c975b0b	xDS Load Balancing (#14397 ) Prior to #13244, connect proxies and gateways could only be configured by an xDS session served by the local client agent. In an upcoming release, it will be possible to deploy a Consul service mesh without client agents. In this model, xDS sessions will be handled by the servers themselves, which necessitates load-balancing to prevent a single server from receiving a disproportionate amount of load and becoming overwhelmed. This introduces a simple form of load-balancing where Consul will attempt to achieve an even spread of load (xDS sessions) between all healthy servers. It does so by implementing a concurrent session limiter (limiter.SessionLimiter) and adjusting the limit according to autopilot state and proxy service registrations in the catalog. If a server is already over capacity (i.e. the session limit is lowered), Consul will begin draining sessions to rebalance the load. This will result in the client receiving a `RESOURCE_EXHAUSTED` status code. It is the client's responsibility to observe this response and reconnect to a different server. Users of the gRPC client connection brokered by the consul-server-connection-manager library will get this for free. The rate at which Consul will drain sessions to rebalance load is scaled dynamically based on the number of proxies in the catalog.	2 years ago
alex	927cee692b	peering: emit exported services count metric (#13811 ) Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
Riddhi Shah	d8d8c8603e	Add support for merge-central-config query param (#13001 ) Adds a new query param merge-central-config for use with the below endpoints: /catalog/service/:service /catalog/connect/:service /health/service/:service /health/connect/:service If set on the request, the response will include a fully resolved service definition which is merged with the proxy-defaults/global and service-defaults/:service config entries (on-demand style). This is useful to view the full service definition for a mesh service (connect-proxy kind or gateway kind) which might not be merged before being written into the catalog (example: in case of services in the agentless model).	3 years ago
R.B. Boyer	f507f62f3c	peering: initial sync (#12842 ) - Add endpoints related to peering: read, list, generate token, initiate peering - Update node/service/check table indexing to account for peers - Foundational changes for pushing service updates to a peer - Plumb peer name through Health.ServiceNodes path see: ENT-1765, ENT-1280, ENT-1283, ENT-1283, ENT-1756, ENT-1739, ENT-1750, ENT-1679, ENT-1709, ENT-1704, ENT-1690, ENT-1689, ENT-1702, ENT-1701, ENT-1683, ENT-1663, ENT-1650, ENT-1678, ENT-1628, ENT-1658, ENT-1640, ENT-1637, ENT-1597, ENT-1634, ENT-1613, ENT-1616, ENT-1617, ENT-1591, ENT-1588, ENT-1596, ENT-1572, ENT-1555 Co-authored-by: R.B. Boyer <rb@hashicorp.com> Co-authored-by: freddygv <freddy@hashicorp.com> Co-authored-by: Chris S. Kim <ckim@hashicorp.com> Co-authored-by: Evan Culver <eculver@hashicorp.com> Co-authored-by: Nitya Dhanushkodi <nitya@hashicorp.com>	3 years ago
Boris Korzun	319a0a08cd	Fix incorrect severity syslog messages (#12079 ) Co-authored-by: Evan Culver <eculver@hashicorp.com>	3 years ago
R.B. Boyer	b60d89e7ef	bulk rewrite using this script set -euo pipefail unset CDPATH cd "$(dirname "$0")" for f in $(git grep '\brequire := require\.New(' \| cut -d':' -f1 \| sort -u); do echo "=== require: $f ===" sed -i '/require := require.New(t)/d' $f # require.XXX(blah) but not require.XXX(tblah) or require.XXX(rblah) sed -i 's/\brequire\.$[a-zA-Z0-9_]$($[^tr]$/require.\1(t,\2/g' $f # require.XXX(tblah) but not require.XXX(t, blah) sed -i 's/\brequire\.$[a-zA-Z0-9_]$($t[^,]$/require.\1(t,\2/g' $f # require.XXX(rblah) but not require.XXX(r, blah) sed -i 's/\brequire\.$[a-zA-Z0-9_]$($r[^,]$/require.\1(t,\2/g' $f gofmt -s -w $f done for f in $(git grep '\bassert := assert\.New(' \| cut -d':' -f1 \| sort -u); do echo "=== assert: $f ===" sed -i '/assert := assert.New(t)/d' $f # assert.XXX(blah) but not assert.XXX(tblah) or assert.XXX(rblah) sed -i 's/\bassert\.$[a-zA-Z0-9_]$($[^tr]$/assert.\1(t,\2/g' $f # assert.XXX(tblah) but not assert.XXX(t, blah) sed -i 's/\bassert\.$[a-zA-Z0-9_]$($t[^,]$/assert.\1(t,\2/g' $f # assert.XXX(rblah) but not assert.XXX(r, blah) sed -i 's/\bassert\.$[a-zA-Z0-9_]$($r[^,]$/assert.\1(t,\2/g' $f gofmt -s -w $f done	3 years ago
Chris S. Kim	08af4f7ffc	Fix Windows logging to files (#11960 )	3 years ago
R.B. Boyer	fc9b1a277d	sync changes to oss files made in enterprise (#10670 )	3 years ago
Dhia Ayachi	c8ba2d40fd	improve monitor performance (#10368 ) * remove flush for each write to http response in the agent monitor endpoint * fix race condition when we stop and start monitor multiple times, the doneCh is closed and never recover. * start log reading goroutine before adding the sink to avoid filling the log channel before getting a chance of reading from it * flush every 500ms to optimize log writing in the http server side. * add changelog file * add issue url to changelog * fix changelog url * Update changelog Co-authored-by: Daniel Nephin <dnephin@hashicorp.com> * use ticker to flush and avoid race condition when flushing in a different goroutine * stop the ticker when done Co-authored-by: Daniel Nephin <dnephin@hashicorp.com> * Revert "fix race condition when we stop and start monitor multiple times, the doneCh is closed and never recover." This reverts commit `1eeddf7a` * wait for log consumer loop to start before registering the sink Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>	3 years ago
R.B. Boyer	a2460eea24	grpc: move gRPC INFO logs to be emitted as TRACE logs from Consul (#10395 ) Fixes #10183	3 years ago
R.B. Boyer	71d45a3460	Support Incremental xDS mode (#9855 ) This adds support for the Incremental xDS protocol when using xDS v3. This is best reviewed commit-by-commit and will not be squashed when merged. Union of all commit messages follows to give an overarching summary: xds: exclusively support incremental xDS when using xDS v3 Attempts to use SoTW via v3 will fail, much like attempts to use incremental via v2 will fail. Work around a strange older envoy behavior involving empty CDS responses over incremental xDS. xds: various cleanups and refactors that don't strictly concern the addition of incremental xDS support Dissolve the connectionInfo struct in favor of per-connection ResourceGenerators instead. Do a better job of ensuring the xds code uses a well configured logger that accurately describes the connected client. xds: pull out checkStreamACLs method in advance of a later commit xds: rewrite SoTW xDS protocol tests to use protobufs rather than hand-rolled json strings In the test we very lightly reuse some of the more boring protobuf construction helper code that is also technically under test. The important thing of the protocol tests is testing the protocol. The actual inputs and outputs are largely already handled by the xds golden output tests now so these protocol tests don't have to do double-duty. This also updates the SoTW protocol test to exclusively use xDS v2 which is the only variant of SoTW that will be supported in Consul 1.10. xds: default xds.Server.AuthCheckFrequency at use-time instead of construction-time	4 years ago
freddygv	5ba14ad41d	Add trace logs to proxycfg state runner and xds srv	4 years ago
Daniel Nephin	b9e60c0775	testing: skip slow tests with -short Add a skip condition to all tests slower than 100ms. This change was made using `gotestsum tool slowest` with data from the last 3 CI runs of master. See https://github.com/gotestyourself/gotestsum#finding-and-skipping-slow-tests With this change: ``` $ time go test -count=1 -short ./agent ok github.com/hashicorp/consul/agent 0.743s real 0m4.791s $ time go test -count=1 -short ./agent/consul ok github.com/hashicorp/consul/agent/consul 4.229s real 0m8.769s ```	4 years ago
Daniel Nephin	752e3ec0b1	logging: improve tests Standardize naming Use stricter assertions and reduce boilerplate to make the intent of the tests more obvious. Also explicitly sort the filenames so that the correct files are pruned, and so that the tests can not flake.	4 years ago
Daniel Nephin	2dcc6966fe	logging: call pruneFiles on startup To ensure that files are pruned before a new one is created. Also clean up the logic in pruneFiles	4 years ago
Daniel Nephin	b847c035d0	logging: remove unnecessary package vars	4 years ago
Paul Banks	332477f24c	Add /v1/internal/ui/metrics-proxy API endpoint that proxies to a configured metrics provider backend.	4 years ago
Paul Banks	e4db845246	Refactor uiserver to separate package, cleaner Reloading	4 years ago
Chris Piraino	3feae7f77b	Report node/service usage metrics from every server Using the newly provided state store methods, we periodically emit usage metrics from the servers. We decided to emit these metrics from all servers, not just the leader, because that means we do not have to care about leader election flapping causing metrics turbulence, and it seems reasonable for each server to emit its own view of the state, even if they should always converge rapidly.	4 years ago
Daniel Nephin	edddc95e0f	logging: Remove t.Parallel from tests The tests all run fast enough that we do not get any advantage from using Parallel. The one test that was slow used a long sleep. Changing the sleep to a few milliseconds speeds up the test considerably.	4 years ago
Kyle Havlovitz	97f1f341d6	Automatically renew the token used by the Vault CA provider	4 years ago
Daniel Nephin	f2373a5575	logging: move init of grpclog This line initializes global state. Moving it out of the constructor and closer to where logging is setup helps keep related things together.	4 years ago
Daniel Nephin	33c401a16e	logging: Setup accept io.Writer instead of []io.Writer Also accept a non-pointer Config, since the config is not modified	4 years ago
Daniel Nephin	5d4df54296	agent: extract dependency creation from New With this change, Agent.New() accepts many of the dependencies instead of creating them in New. Accepting fully constructed dependencies from a constructor makes the type easier to test, and easier to change. There are still a number of dependencies created in Start() which can be addressed in a follow up.	4 years ago
Daniel Nephin	d68edcecf4	testing: Remove all the defer os.Removeall Now that testutil uses t.Cleanup to remove the directory the caller no longer has to manage the removal	4 years ago
Daniel Nephin	0420d91cdd	Remove LogOutput from Agent Now that it is no longer used, we can remove this unnecessary field. This is a pre-step in cleanup up RuntimeConfig->Consul.Config, which is a pre-step to adding a gRPCHandler component to Server for streaming. Removing this field also allows us to remove one of the return values from logging.Setup.	4 years ago
Matt Keeler	3dbbd2d37d	Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.	4 years ago
freddygv	15c74d6943	Move GatewayServices out of Internal	5 years ago
Jono Sosulska	9b363e9f23	Fix spelling of deregister (#7804 )	5 years ago
Hans Hasselberg	51549bd232	rpc: oss changes for network area connection pooling (#7735 )	5 years ago
freddygv	24207226ca	Add proxycfg state management for terminating-gateways	5 years ago
Deepjyoti Mondal	0b3af0e070	Relates to #7447 This PR will allow consul to throw log error if provided log directory has permission issues for writing or is non existent. Signed-off-by: Deepjyoti Mondal djmdeveloper060796@gmail.com	5 years ago
R.B. Boyer	6adad71125	wan federation via mesh gateways (#6884 ) This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch: There are several distinct chunks of code that are affected: * new flags and config options for the server * retry join WAN is slightly different * retry join code is shared to discover primary mesh gateways from secondary datacenters * because retry join logic runs in the agent and the results of that operation for primary mesh gateways are needed in the server there are some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur at multiple layers of abstraction just to pass the data down to the right layer. * new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers * the function signature for RPC dialing picked up a new required field (the node name of the destination) * several new RPCs for manipulating a FederationState object: `FederationState:{Apply,Get,List,ListMeshGateways}` * 3 read-only internal APIs for debugging use to invoke those RPCs from curl * raft and fsm changes to persist these FederationStates * replication for FederationStates as they are canonically stored in the Primary and replicated to the Secondaries. * a special derivative of anti-entropy that runs in secondaries to snapshot their local mesh gateway `CheckServiceNodes` and sync them into their upstream FederationState in the primary (this works in conjunction with the replication to distribute addresses for all mesh gateways in all DCs to all other DCs) * a "gateway locator" convenience object to make use of this data to choose the addresses of gateways to use for any given RPC or gossip operation to a remote DC. This gets data from the "retry join" logic in the agent and also directly calls into the FSM. * RPC (`:8300`) on the server sniffs the first byte of a new connection to determine if it's actually doing native TLS. If so it checks the ALPN header for protocol determination (just like how the existing system uses the type-byte marker). * 2 new kinds of protocols are exclusively decoded via this native TLS mechanism: one for ferrying "packet" operations (udp-like) from the gossip layer and one for "stream" operations (tcp-like). The packet operations re-use sockets (using length-prefixing) to cut down on TLS re-negotiation overhead. * the server instances specially wrap the `memberlist.NetTransport` when running with gateway federation enabled (in a `wanfed.Transport`). The general gist is that if it tries to dial a node in the SAME datacenter (deduced by looking at the suffix of the node name) there is no change. If dialing a DIFFERENT datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh gateways to eventually end up in a server's :8300 port. * a new flag when launching a mesh gateway via `consul connect envoy` to indicate that the servers are to be exposed. This sets a special service meta when registering the gateway into the catalog. * `proxycfg/xds` notice this metadata blob to activate additional watches for the FederationState objects as well as the location of all of the consul servers in that datacenter. * `xds:` if the extra metadata is in place additional clusters are defined in a DC to bulk sink all traffic to another DC's gateways. For the current datacenter we listen on a wildcard name (`server.<dc>.consul`) that load balances all servers as well as one mini-cluster per node (`<node>.server.<dc>.consul`) * the `consul tls cert create` command got a new flag (`-node`) to help create an additional SAN in certs that can be used with this flavor of federation.	5 years ago
R.B. Boyer	0ecb4538c1	agent: differentiate wan vs lan loggers in memberlist and serf (#7205 ) This should be a helpful change until memberlist and serf can be properly switched to native hclog.	5 years ago
Chris Piraino	401221de58	Allow users to configure either unstructured or JSON logging (#7130 ) * hclog Allow users to choose between unstructured and JSON logging	5 years ago

50 Commits (CC-7146/hcp-link-item-in-the-nav-bar)