github
/
consul
mirror of https://github.com/hashicorp/consul
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
21458 Commits
1702 Branches
457 Tags
697 MiB
 
 
 
 
 
 
Tree: 2d19cd5810
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '2d19cd5810'
${ noResults }
consul/website/content/docs/connect/proxies/envoy.mdx

1214 lines
54 KiB
Raw Blame History

---
layout: docs
page_title: Envoy Proxy Configuration | Service Mesh
description: >-
Consul supports Envoy proxies to direct traffic throughout the service mesh. Learn about Consul versions and their Envoy support, and use the reference guide to review options for bootstrap configuration, dynamic configuration, and advanced topics like escape hatch overrides.
---
# Envoy Proxy Configuration for Service Mesh
Consul service mesh has first class support for using
[Envoy](https://www.envoyproxy.io) as a proxy. Consul configures Envoy by
optionally exposing a gRPC service on the local agent that serves [Envoy's xDS
configuration
API](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-docs/xds_protocol).
Consul can configure Envoy sidecars to proxy traffic over the following protocols:
| Protocol | Service network support |
| ----------------------- | ----------------------- |
| HTTP/1.1 | L7 |
| HTTP2 | L7 |
| gRPC | L7 |
| All TCP-based protocols | L4 |
On Consul 1.5.0 and older, Envoy proxies can only proxy TCP traffic at L4.
You can configure some [L7 features](/consul/docs/connect/manage-traffic) in [configuration entries](/consul/docs/agent/config-entries). You can add [custom Envoy configurations](#advanced-configuration) to the [proxy service definition](/consul/docs/connect/proxies/proxy-config-reference), which enables you to leverage Envoy features that are not exposed through configuration entries. You can also use the [Consul Envoy extensions](/consul/docs/connect/proxies/envoy-extensions) to implement Envoy features.
~> **Note:** When using Envoy with Consul and not using the [`consul connect envoy` command](/consul/commands/connect/envoy)
Envoy must be run with the `--max-obj-name-len` option set to `256` or greater for Envoy versions prior to 1.11.0.
## Supported Versions
The following matrix describes Envoy compatibility for the
currently supported major Consul releases:
- The latest (N) release of Consul community edition (CE) and Enterprise
- The 2 preceding major releases (N-1 and N-2) of Consul Enterprise
- The 2 latest [Consul Enterprise LTS](/consul/docs/enterprise/long-term-support) major releases
For previous Consul version compatibility,
refer to the previous release's version of this page.
### Envoy and Consul Client Agent
Every major Consul release initially supports **four major Envoy releases**.
Standard major Consul releases do not expand that support in minor releases.
However, [Consul Enterprise Long Term Support (LTS)](/consul/docs/enterprise/long-term-support)
releases do expand their Envoy version compatibility window in minor releases
to ensure compatibility with a maintained Envoy version.
Every major Consul release maintains and tests compatibility with specific Envoy
patch releases to ensure users can benefit from bug and security fixes in Envoy.
#### Standard releases
Unless otherwise noted, rows in the following compatibility table
apply to both Consul Enterprise and Consul community edition (CE).
| Consul Version | Compatible Envoy Versions |
| -------------- | -------------------------------------- |
| 1.20.x CE | 1.31.x, 1.30.x, 1.29.x, 1.28.x |
| 1.19.x CE | 1.29.x, 1.28.x, 1.27.x, 1.26.x |
| 1.18.x CE | 1.28.x, 1.27.x, 1.26.x, 1.25.x |
| 1.17.x | 1.27.x, 1.26.x, 1.25.x, 1.24.x |
| 1.16.x | 1.26.x, 1.25.x, 1.24.x, 1.23.x |
#### Enterprise Long Term Support releases
Active Consul Enterprise
[Long Term Support (LTS)](/consul/docs/enterprise/long-term-support)
releases expand their Envoy version compatibility window
until the LTS release reaches its end of maintenance.
| Consul Version | Compatible Envoy Versions |
| -------------- | -----------------------------------------------------------------------------------|
| 1.18.x Ent | 1.29.x, 1.28.x, 1.27.x, 1.26.x, 1.25.x |
| 1.15.x Ent | 1.29.x, 1.28.x, 1.27.x, 1.26.x, 1.25.x, 1.24.x, 1.23.x, 1.22.x |
### Envoy and Consul Dataplane
The Consul dataplane component was introduced in Consul v1.14
as a way to manage Envoy proxies without the use of Consul clients.
Each major version of Consul is released with a new major version of Consul dataplane,
which packages both Envoy and the `consul-dataplane` binary in a single container image.
To enable seamless upgrades, each major version of Consul also supports
the previous and next Consul dataplane versions.
Compared to standard Consul releases, Consul Enterprise
[Long Term Support (LTS)](/consul/docs/enterprise/long-term-support)
releases have the following differences with Consul dataplane compatibility:
- [Expanded compatibility window](#enterprise-long-term-support-releases):
Active Consul Enterprise LTS releases expand their Consul dataplane
version compatibility window until the LTS release reaches its
end of maintenance.
- [Maintained Envoy version](#consul-dataplane-releases-that-span-envoy-major-versions):
Major versions of Consul dataplane aligned with a Consul Enterprise LTS version
may contain minor version updates that use a new major version of Envoy.
These minor version updates are necessary to ensure maintained versions
of Consul dataplane use a maintained version of Envoy.
#### Standard releases
Unless otherwise noted, rows in the following compatibility table
apply to both Consul Enterprise and Consul community edition (CE).
| Consul Version | Default `consul-dataplane` Version | Other compatible `consul-dataplane` Versions |
| -------------- | -------------------------------------|----------------------------------------------|
| 1.19.x CE | 1.5.x (Envoy 1.29.x) | 1.4.x (Envoy 1.28.x) |
| 1.18.x CE | 1.4.x (Envoy 1.28.x) | 1.3.x (Envoy 1.27.x) |
| 1.17.x | 1.3.x (Envoy 1.27.x) | 1.4.x (Envoy 1.28.x), 1.2.x (Envoy 1.26.x) |
| 1.16.x | 1.2.x (Envoy 1.26.x) | 1.3.x (Envoy 1.27.x), 1.1.x (Envoy 1.25.x) |
#### Enterprise Long Term Support releases
Active Consul Enterprise
[Long Term Support (LTS)](/consul/docs/enterprise/long-term-support)
releases expand their Envoy version compatibility window
until the LTS release reaches its end of maintenance.
| Consul Version | Default `consul-dataplane` Version | Other compatible `consul-dataplane` Versions |
| -------------- | -------------------------------------|----------------------------------------------|
| 1.19.x Ent | 1.5.x (Envoy 1.29.x) | 1.4.x (Envoy 1.28.x) |
| 1.18.x Ent | 1.4.x (Envoy 1.28.x) | 1.3.x (Envoy 1.27.x) |
| 1.15.x Ent | 1.1.x (Envoy 1.26.x) | 1.4.x (Envoy 1.28.x) - 1.0.x (Envoy 1.24.x) |
#### Consul dataplane releases that span Envoy major versions
Major versions of Consul dataplane aligned with a Consul Enterprise LTS version
may contain minor version updates that use a new major version of Envoy.
These minor version updates are necessary to ensure maintained versions
of Consul dataplane use a maintained version of Envoy.
| `consul-dataplane` Version Range | Associated Consul Enterprise LTS version | Contained Envoy Binary Version |
| -------------------------------- | ---------------------------------------- | ------------------------------ |
| 1.5.0 - 1.5.latest | 1.18.x Ent | Envoy 1.29.x |
| 1.4.0 - 1.4.latest | 1.18.x Ent | Envoy 1.28.x |
| 1.1.9 - 1.1.latest | 1.15.x Ent | Envoy 1.26.x |
| 1.1.0 - 1.1.8 | 1.15.x Ent | Envoy 1.25.x |
## Getting Started
To get started with Envoy and see a working example you can follow the [Using
Envoy with Consul service mesh](/consul/tutorials/developer-mesh/service-mesh-with-envoy-proxy?utm_source=docs) tutorial.
## Configuration
Envoy proxies require two types of configuration: an initial _bootstrap
configuration_ and a _dynamic configuration_ that is discovered from a "management
server", in this case Consul.
The bootstrap configuration at a minimum needs to configure the proxy with an
identity (node id) and the location of its local Consul agent from which it
discovers all of its dynamic configuration. See [Bootstrap
Configuration](#bootstrap-configuration) for more details.
The dynamic configuration Consul service mesh provides to each Envoy instance includes:
- TLS certificates and keys to enable mutual authentication and keep certificates
rotating.
- [Intentions] to enforce service-to-service authorization rules.
- Service-discovery results for upstreams to enable each sidecar proxy to load-balance
outgoing connections.
- L7 configuration including timeouts and protocol-specific options.
- Configuration to [expose specific HTTP paths](/consul/docs/connect/proxies/proxy-config-reference#expose-paths-configuration-reference).
For more information on the parts of the Envoy proxy runtime configuration
that are currently controllable via Consul service mesh, refer to [Dynamic
Configuration](#dynamic-configuration).
We plan to enable more and more of Envoy's features through
Consul service mesh's first-class configuration over time, however some advanced users will
need additional control to configure Envoy in specific ways. To enable this, we
provide several ["escape hatch"](#advanced-configuration) options that allow
users to provide low-level raw Envoy config syntax for some sub-components in each
Envoy instance. This allows operators to have full control over and
responsibility for correctly configuring Envoy and ensuring version support etc.
## Intention Enforcement
[Intentions](/consul/docs/connect/intentions) are enforced using Envoy's RBAC filters. Depending on the
configured [protocol](/consul/docs/connect/config-entries/service-defaults#protocol) of the proxied service, intentions are either enforced
per-connection (L4) using a network filter, or per-request (L7) using an HTTP
filter.
-> **Note:** Prior to Consul 1.9.0 intentions were exclusively enforced
per-connection (L4) using an `ext_authz` network filter.
## Fetching Certificates
Envoy will use the [`CONSUL_HTTP_TOKEN`](/consul/commands#consul_http_token) and [`CONSUL_HTTP_ADDR`](/consul/commands#consul_http_addr) environment variables to contact Consul to fetch certificates if the following conditions are met:
- The `CONSUL_HTTP_TOKEN` environment variable contains a Consul ACL token.
- The Consul ACL token has the necessary permissions to read configuration for that service.
If TLS is enabled on Consul, you will also need to add the following environment variables _prior_ to starting Envoy:
- [`CONSUL_CACERT`](/consul/commands#consul_cacert)
- [`CONSUL_CLIENT_CERT`](/consul/commands#consul_client_cert)
- [`CONSUL_CLIENT_KEY`](/consul/commands#consul_client_key)
- [`CONSUL_HTTP_SSL`](/consul/commands#consul_http_ssl)
## Bootstrap Configuration
Envoy requires an initial bootstrap configuration file. You can either create the file manually using the Consul command line or configure Consul Dataplane to generate the file.
### Generate the bootstrap file on the Consul CLI
Connect to a local Consul client agent and run the [`consul connect envoy` command](/consul/commands/connect/envoy) to create the Envoy bootstrap configuration. The command either outputs the bootstrap configuration directly to stdout or generates the configuration and issues an `exec` command to the Envoy binary as a convenience wrapper. For more information about using `exec` to bootstrap Envoy, refer to [Exec Security Details](/consul/commands/connect/envoy#exec-security-details).
If you experience issues when bootstrapping Envoy proxies from the CLI, use the
`-enable-config-gen-logging` flag to enable debug message logging. These logs can
help you troubleshoot issues that occur during the bootstrapping process.
For more information about available flags and parameters, refer to the
[`consul connect envoy CLI` reference](/consul/commands/connect/envoy).
### Generate the bootstrap file from Consul Dataplane
Consul Dataplane automatically configures and manages an Envoy process. Consul Dataplane generates the Envoy bootstrap configuration file prior to starting Envoy. To configure how Consul Dataplane starts Envoy, refer to the [Consul Dataplane CLI reference](/consul/docs/connect/dataplane/consul-dataplane).
### Control bootstrap configuration from proxy configuration
Consul service mesh can control some parts of the bootstrap configuration by specifying Envoy proxy configuration options.
Add the following configuration items to the [global `proxy-defaults` configuration
entry](/consul/docs/connect/config-entries/proxy-defaults) or override them directly in the `proxy.config`
field of a [proxy service definition](/consul/docs/proxies/proxy-config-reference). When
connected to a Consul client agent, you can place the configuration in the `proxy.config` field of
the [`sidecar_service`](/consul/docs/connect/proxies/deploy-sidecar-services) block.
- `envoy_statsd_url` - A URL in the form `udp://ip:port` identifying a UDP
StatsD listener that Envoy should deliver metrics to. For example, this may be
`udp://127.0.0.1:8125` if every host has a local StatsD listener. In this case
users can configure this property once in the [global `proxy-defaults`
configuration entry](/consul/docs/connect/config-entries/proxy-defaults) for convenience. Currently, TCP is not supported.
~> **Note:** currently the url **must use an ip address** not a dns name due
to the way Envoy is setup for StatsD.
Expansion of the environment variable `HOST_IP` is supported, e.g.
`udp://${HOST_IP}:8125`.
Users can also specify the whole parameter in the form `$ENV_VAR_NAME`, which
will cause the `consul connect envoy` command to resolve the actual URL from
the named environment variable when it runs. This, for example, allows each
pod in a Kubernetes cluster to learn of a pod-specific IP address for StatsD
when the Envoy instance is bootstrapped while still allowing global
configuration of all proxies to use StatsD in the [global `proxy-defaults`
configuration entry](/consul/docs/connect/config-entries/proxy-defaults). The env
variable must contain a full valid URL value as specified above and nothing else.
- `envoy_dogstatsd_url` - The same as `envoy_statsd_url` with the following
differences in behavior:
- Envoy will use dogstatsd tags instead of statsd dot-separated metric names.
- As well as `udp://`, a `unix://` URL may be specified if your agent can
listen on a unix socket (e.g. the dogstatsd agent).
- `envoy_prometheus_bind_addr` - Specifies that the proxy should expose a Prometheus
metrics endpoint to the _public_ network. It must be supplied in the form
`ip:port` and port and the ip/port combination must be free within the network
namespace the proxy runs. Typically the IP would be `0.0.0.0` to bind to all
available interfaces or a pod IP address.
-> **Note:** Envoy versions prior to 1.10 do not export timing histograms
using the internal Prometheus endpoint.
- `envoy_stats_bind_addr` - Specifies that the proxy should expose the /stats prefix
to the _public_ network. It must be supplied in the form `ip:port` and
the ip/port combination must be free within the network namespace the proxy runs.
Typically the IP would be `0.0.0.0` to bind to all available interfaces or a pod IP address.
- `envoy_stats_tags` - Specifies one or more static tags that will be added to
all metrics produced by the proxy.
- `envoy_stats_flush_interval` - Configures Envoy's
[`stats_flush_interval`](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/bootstrap/v3/bootstrap.proto#envoy-v3-api-field-config-bootstrap-v3-bootstrap-stats-flush-interval).
- `envoy_telemetry_collector_bind_socket_dir` - Specifies the directory where Envoy creates a Unix socket.
Envoy sends metrics to the socket where a Consul telemetry collector can collect them.
The socket is not configured by default.
Enabling this sets Envoy's [`stats_flush_interval`](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/bootstrap/v3/bootstrap.proto#envoy-v3-api-field-config-bootstrap-v3-bootstrap-stats-flush-interval) to one minute if `envoy_stats_flush_interval` is unset and if no other stats sinks are configured, like `envoy_dogstats_url`, for instance.
The [Advanced Configuration](#advanced-configuration) section describes additional configurations that allow incremental or complete control over the bootstrap configuration generated.
### Bootstrap Envoy on Windows VMs
> Complete the [Connect Services on Windows Workloads to Consul Service Mesh tutorial](/consul/tutorials/developer-mesh/consul-windows-workloads) to learn how to deploy Consul and use its service mesh on Windows VMs.
If you are running Consul on a Windows VM, attempting to bootstrap Envoy with the `consul connect envoy` command returns the following output:
```shell-session hideClipboard
Directly running Envoy is only supported on linux and macOS since envoy itself doesn't build on other platforms currently.
Use the -bootstrap option to generate the JSON to use when running envoy on a supported OS or via a container or VM.
```
To bootstrap Envoy on Windows VMs, you must generate the bootstrap configuration as a .json file and then manually edit it to add both your ACL token and a valid access log path.
To generate the bootstrap configuration file, add the `-bootstrap` option to the command and then save the output to a file:
```shell-session
$ consul connect envoy -bootstrap > bootstrap.json
```
Then, open `bootstrap.json` and update the following sections with your ACL token and log path.
<CodeBlockConfig filename="bootstrap.json" hideClipboard lineNumbers highlight="2,19">
```json
{
"admin": {
"access_log": [
{
"name": "envoy.access_loggers.file",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
"path": "/dev/null"
}
}
],
"address": {
"socket_address": {
"address": "127.0.0.1",
"port_value": 19000
}
}
},
## ...
"dynamic_resources": {
## ...
"ads_config": {
## ...
"grpc_services": {
"initial_metadata": [
{
"key": "x-consul-token",
"value": "<ACL-Token>"
}
],
## ...
}
}
}
}
```
</CodeBlockConfig>
To complete the bootstrap process, start Envoy and include the path to `bootstrap.json`:
```shell-session
$ envoy -c bootstrap.json
```
~> **Security Note**: The bootstrap JSON contains the ACL token and should be handled as a secret. Because this token authorizes the identity of any service it has `service:write` permissions for, it can be used to access upstream services.
## Dynamic Configuration
Consul automatically generates Envoy's dynamic configuration based on its
knowledge of the cluster. Users may specify default configuration options for
a service through the available fields in the [`service-defaults` configuration
entry](/consul/docs/connect/config-entries/service-defaults). Consul will use this
information to configure appropriate proxy settings for that service's proxies
and also for the upstream listeners used by the service.
One example is how users can define a service's protocol in the `Protocol` field of [`service-defaults` configuration
entry](/consul/docs/connect/config-entries/service-defaults). Agents with
[`enable_central_service_config`](/consul/docs/agent/config/config-files#enable_central_service_config)
set to true will automatically discover the protocol when configuring a proxy
for a service. The proxy will discover the main protocol of the service it
represents and use this to configure its main public listener. It will also
discover the protocols defined for any of its upstream services and
automatically configure its upstream listeners appropriately too as below.
This automated discovery results in Consul auto-populating the `proxy.config`
and `proxy.upstreams[*].config` fields of the [proxy service
definition](/consul/docs/connect/proxies/proxy-config-reference) that is
actually registered.
To learn about other options that can be configured centrally see the
[Configuration Entries](/consul/docs/agent/config-entries) docs.
### Proxy Config Options
These fields may also be overridden explicitly in `proxy.config` of the [proxy service
definition](/consul/docs/connect/proxies/proxy-config-reference), or defined in
the [global `proxy-defaults` configuration
entry](/consul/docs/connect/config-entries/proxy-defaults) to act as
defaults that are inherited by all services.
- `protocol` - The protocol the service speaks. Consul service mesh's Envoy integration
currently supports the following `protocol` values:
- `tcp` - Unless otherwise specified this is the default, which causes Envoy
to proxy at L4. This provides all the security benefits of the service mesh's mTLS
and works for any TCP-based protocol. Load-balancing and metrics are
available at the connection level.
- `http` - This specifies that the service speaks HTTP/1.x. Envoy will setup an
`http_connection_manager` and will be able to load-balance requests
individually to available upstream services. Envoy will also emit L7 metrics
such as request rates broken down by HTTP response code family (2xx, 4xx, 5xx,
etc).
- `http2` - This specifies that the service speaks http2 (specifically h2c since
Envoy will still only connect to the local service instance via plain TCP not
TLS). This behaves much like `http` with L7 load-balancing and metrics but has
additional settings that correctly enable end-to-end http2.
- `grpc` - gRPC is a common RPC protocol based on http2. In addition to the
http2 support above, Envoy listeners will be configured with a
[gRPC bridge
filter](https://www.envoyproxy.io/docs/envoy/v1.17.2/configuration/http/http_filters/grpc_http1_bridge_filter)
that translates HTTP/1.1 calls into gRPC, and instruments
metrics with `gRPC-status` trailer codes.
~> **Note:** The protocol of a service should ideally be configured via the
[`protocol`](/consul/docs/connect/config-entries/service-defaults#protocol)
field of a
[`service-defaults`](/consul/docs/connect/config-entries/service-defaults)
config entry for the service. Configuring it in a
proxy config will not fully enable some [L7
features](/consul/docs/connect/manage-traffic).
It is supported here for backwards compatibility with Consul versions prior to 1.6.0.
- `bind_address` - Override the address Envoy's public listener binds to. By
default Envoy will bind to the service address or 0.0.0.0 if there is not explicit address on the service registration.
- `bind_port` - Override the port Envoy's public listener binds to. By default
Envoy will bind to the service port.
- `local_connect_timeout_ms` - The number of milliseconds allowed to make
connections to the local application instance before timing out. Defaults to 5000
(5 seconds).
- `local_request_timeout_ms` - In milliseconds, the request timeout for HTTP requests
to the local application instance. Applies to HTTP based protocols only. If not
specified, inherits the Envoy default for route timeouts (15s). A value of 0 will
disable request timeouts.
- `local_idle_timeout_ms` - In milliseconds, the idle timeout for HTTP requests
to the local application instance. Applies to HTTP based protocols only. If not
specified, inherits the Envoy default for route idle timeouts (15s). A value of 0
disables request timeouts.
- `max_inbound_connections` - The maximum number of concurrent inbound connections
to the local application instance. If not specified, inherits the Envoy default (1024).
- `balance_inbound_connections` - The strategy used for balancing inbound connections
across Envoy worker threads. Consul service mesh Envoy integration supports the
following `balance_inbound_connections` values:
- `""` - Empty string (default). No connection balancing strategy is used. Consul does not balance inbound connections.
- `exact_balance` - Inbound connections to the service use the
[Envoy Exact Balance Strategy.](https://cloudnative.to/envoy/api-v3/config/listener/v3/listener.proto.html#config-listener-v3-listener-connectionbalanceconfig-exactbalance)
- `xds_fetch_timeout_ms` - In milliseconds, the amount of time for Envoy to wait for EDS and RDS configuration before timing out. If not specified, this field uses Envoy's default value of `15000`, or 15 seconds. When an Envoy instance is configured with a large number of upstreams that take a significant amount of time to populate with data, setting this field to a higher value may prevent temporary disruption caused by unexpected timeouts.
### Proxy Upstream Config Options
The following configuration items may be overridden directly in the
`proxy.upstreams[].config` field of a [proxy service
definition](/consul/docs/connect/proxies/proxy-config-reference) or
[`sidecar_service`](/consul/docs/connect/proxies/deploy-sidecar-services) block.
- `protocol` - Same as above in main config but affects the listener setup for
the upstream.
~> **Note:** The protocol of a service should ideally be configured via the
[`protocol`](/consul/docs/connect/config-entries/service-defaults#protocol)
field of a
[`service-defaults`](/consul/docs/connect/config-entries/service-defaults)
config entry for the upstream destination service. Configuring it in a
proxy upstream config will not fully enable some [L7
features](/consul/docs/connect/manage-traffic).
It is supported here for backwards compatibility with Consul versions prior to 1.6.0.
- `connect_timeout_ms` - The number of milliseconds to allow when making upstream
connections before timing out. Defaults to 5000
(5 seconds).
~> **Note:** The connection timeout for a service should ideally be
configured via the
[`connect_timeout`](/consul/docs/connect/config-entries/service-resolver#connecttimeout)
field of a
[`service-resolver`](/consul/docs/connect/config-entries/service-resolver)
config entry for the upstream destination service. Configuring it in a
proxy upstream config will override any values defined in config entries.
It is supported here for backwards compatibility with Consul versions prior to 1.6.0.
- `limits` - A set of limits to apply when connecting to the upstream service.
These limits are applied on a per-service-instance basis. The following
limits are respected:
- `max_connections` - The maximum number of connections a service instance
will be allowed to establish against the given upstream. Use this to limit
HTTP/1.1 traffic, since HTTP/1.1 has a request per connection.
- `max_pending_requests` - The maximum number of requests that will be queued
while waiting for a connection to be established. For this configuration to
be respected, a L7 protocol must be defined in the `protocol` field.
- `max_concurrent_requests` - The maximum number of concurrent requests that
will be allowed at a single point in time. Use this to limit HTTP/2 traffic,
since HTTP/2 has many requests per connection. For this configuration to be
respected, a L7 protocol must be defined in the `protocol` field.
- `passive_health_check` - Passive health checks remove hosts from the upstream
cluster that are unreachable or that return errors.
- `interval` - The time in nanosecond between checks. Each check will cause
hosts which have exceeded `max_failures` to be removed from the load
balancer, and any hosts which have passed their ejection time to be
returned to the load balancer. If not specified, it uses the default value.
For example, 10s for Envoy proxy.
- `max_failures` - The number of consecutive failures that cause a host to be
removed from the upstream cluster. If not specified, Consul uses the proxy's
default value. For example, `5` for Envoy proxy.
- `enforcing_consecutive_5xx` - A percentage representing the chance that a
host will be actually ejected when the proxy detects an outlier status
through consecutive errors in the 500 code range. If not specified, Consul
uses the proxy's default value. For example, `100` for Envoy proxy.
- `max_ejection_percent` - The maximum percentage of hosts that can be ejected
from a upstream cluster due to passive health check failures. If not specified,
inherits Envoy's default of 10% or at least one host.
- `base_ejection_time` - The base time that a host is ejected for. The real
time is equal to the base time multiplied by the number of times the host has
been ejected and is capped by max_ejection_time (Default 300s). If not
specified, inherits Envoy's default value of 30s.
- `balance_outbound_connections` - Specifies the strategy for balancing outbound connections
across Envoy worker threads. Consul service mesh Envoy integration supports the
following `balance_outbound_connections` values:
- `""` - Empty string (default). No connection balancing strategy is used. Consul does not balance outbound connections.
- `exact_balance` - Outbound connections from the upstream use the
[Envoy Exact Balance Strategy.](https://cloudnative.to/envoy/api-v3/config/listener/v3/listener.proto.html#config-listener-v3-listener-connectionbalanceconfig-exactbalance)
### Gateway Options
These fields may also be overridden explicitly in the [proxy service
definition](/consul/docs/connect/proxies/proxy-config-reference), or defined in
the [global `proxy-defaults` configuration
entry](/consul/docs/connect/config-entries/proxy-defaults) to act as
defaults that are inherited by all services.
Prior to 1.8.0 these settings were specific to Mesh Gateways. The deprecated
names such as `envoy_mesh_gateway_bind_addresses` and `envoy_mesh_gateway_no_default_bind`
will continue to be supported.
- `connect_timeout_ms` - The number of milliseconds to allow when making upstream
connections before timing out. Defaults to 5000 (5 seconds). If the upstream
service has the configuration option
[`connect_timeout_ms`](/consul/docs/connect/config-entries/service-resolver#connecttimeout)
set for the `service-resolver`, that timeout value will take precedence over
this gateway option.
- `envoy_gateway_bind_tagged_addresses` - Indicates that the gateway
services tagged addresses should be bound to listeners in addition to the
default listener address.
- `envoy_gateway_bind_addresses` - A map of additional addresses to be bound.
This map's keys are the name of the listeners to be created and the values are
a map with two keys, address and port, that combined make the address to bind the
listener to. These are bound in addition to the default address.
- `envoy_gateway_no_default_bind` - Prevents binding to the default address
of the gateway service. This should be used with one of the other options
to configure the gateway's bind addresses.
- `envoy_dns_discovery_type` - Determines how Envoy will resolve hostnames. Defaults to `LOGICAL_DNS`.
Must be one of `STRICT_DNS` or `LOGICAL_DNS`. Details for each type are available in
the [Envoy documentation](https://www.envoyproxy.io/docs/envoy/v1.17.2/intro/arch_overview/upstream/service_discovery).
This option applies to terminating gateways that route to services
addressed by a hostname, such as a managed database. It also applies to mesh gateways,
such as when gateways in other Consul datacenters are behind a load balancer that is addressed by a hostname.
- `envoy_gateway_remote_tcp_enable_keepalive` - Enables TCP keepalive settings on remote
upstream connections for mesh and terminating gateways. Defaults to `false`. Must be one
of `true` or `false`. Details for this feature are available in the
[Envoy documentation](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/address.proto#envoy-v3-api-msg-config-core-v3-tcpkeepalive).
- `envoy_gateway_remote_tcp_keepalive_time` - The number of seconds a connection needs to
be idle before keep-alive probes start being sent. For more information, see the
[Envoy documentation](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/address.proto#envoy-v3-api-msg-config-core-v3-tcpkeepalive).
This option only applies to remote upstream connections for mesh and terminating gateways.
- `envoy_gateway_remote_tcp_keepalive_interval` - The number of seconds between keep-alive probes.
For more information, see the
[Envoy documentation](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/address.proto#envoy-v3-api-msg-config-core-v3-tcpkeepalive).
This option only applies to remote upstream connections for mesh and terminating gateways.
- `envoy_gateway_remote_tcp_keepalive_probes` - Maximum number of keepalive probes to send without
response before deciding the connection is dead. For more information, see the
[Envoy documentation](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/address.proto#envoy-v3-api-msg-config-core-v3-tcpkeepalive).
This option only applies to remote upstream connections for mesh and terminating gateways.
## Advanced Configuration
To support more flexibility when configuring Envoy, several "lower-level" options exist
that require knowledge of Envoy's configuration format.
Many options allow configuring a subsection of either the bootstrap or
dynamic configuration using your own custom protobuf config.
We separate these into two sets, [Advanced Bootstrap
Options](#advanced-bootstrap-options) and [Escape Hatch
Overrides](#escape-hatch-overrides). Both require writing Envoy config in the
protobuf JSON encoding. Advanced options cover smaller chunks that might
commonly need to be set for tasks like configuring tracing. In contrast, escape hatches
give almost complete control over the proxy setup, but require operators to
manually code the entire configuration in protobuf JSON.
~> **Advanced Topic!** This section covers options that allow users to take almost
complete control of Envoy's configuration. We provide these options so users can
experiment or take advantage of features not yet fully supported in Consul service mesh. We
plan to retain this ability in the future, but it should still be considered
experimental because it requires in-depth knowledge of Envoy's configuration format.
Users should consider Envoy version compatibility when using these features because they can configure Envoy in ways that
are outside of Consul's control. Incorrect configuration could prevent all
proxies in your mesh from functioning correctly, or bypass the security
guarantees Consul service mesh is designed to enforce.
### Configuration Formatting
All configurations are specified as strings containing the serialized proto3 JSON encoding
of the specified Envoy configuration type. They are full JSON types except where
noted.
The JSON supplied may describe a protobuf `types.Any` message with an `@type`
field set to the appropriate type (for example
`type.googleapis.com/envoy.config.listener.v3.Listener`).
For example, given a tracing config:
<CodeBlockConfig heading="Example envoy_tracing_json configuration">
```json
{
"http": {
"name": "envoy.tracers.zipkin",
"typedConfig": {
"@type": "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
"collector_cluster": "zipkin",
"collector_endpoint_version": "HTTP_JSON",
"collector_endpoint": "/api/v1/spans",
"shared_span_context": false
}
}
}
```
</CodeBlockConfig>
JSON escape the value of `tracing` into a string, for example using [https://codebeautify.org/json-escape-unescape](https://codebeautify.org/json-escape-unescape),
or using [jq](https://stedolan.github.io/jq/).
```shell
$ cat <<EOF | jq '. | @json'
{
"http": {
"name": "envoy.tracers.zipkin",
"typedConfig": {
"@type": "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
"collector_cluster": "zipkin",
"collector_endpoint_version": "HTTP_JSON",
"collector_endpoint": "/api/v1/spans",
"shared_span_context": false
}
}
}
EOF
"{\"http\":{\"name\":\"envoy.tracers.zipkin\",\"typedConfig\":{\"@type\":\"type.googleapis.com/envoy.config.trace.v3.ZipkinConfig\",\"collector_cluster\":\"zipkin\",\"collector_endpoint_version\":\"HTTP_JSON\",\"collector_endpoint\":\"/api/v1/spans\",\"shared_span_context\":false}}}"
```
Then use that as the value for `envoy_tracing_json`:
```json
{
"kind": "proxy-defaults",
"name": "global",
"config": {
"envoy_tracing_json": "{\"http\":{\"name\":\"envoy.tracers.zipkin\",\"typedConfig\":{\"@type\":\"type.googleapis.com/envoy.config.trace.v3.ZipkinConfig\",\"collector_cluster\":\"zipkin\",\"collector_endpoint_version\":\"HTTP_JSON\",\"collector_endpoint\":\"/api/v1/spans\",\"shared_span_context\":false}}}\n"
}
}
```
If using HCL, this escaping is done automatically:
```hcl
Kind = "proxy-defaults"
Name = "global"
Config {
envoy_tracing_json = <<EOF
{
"http": {
"name": "envoy.tracers.zipkin",
"typedConfig": {
"@type": "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
"collector_cluster": "zipkin",
"collector_endpoint_version": "HTTP_JSON",
"collector_endpoint": "/api/v1/spans",
"shared_span_context": false
}
}
}
EOF
}
```
### Advanced Bootstrap Options
Users may add the following configuration items to the [global `proxy-defaults`
configuration
entry](/consul/docs/connect/config-entries/proxy-defaults) or
override them directly in the `proxy.config` field of a [proxy service
definition](/consul/docs/connect/proxies/proxy-config-reference) or
[`sidecar_service`](/consul/docs/connect/proxies/deploy-sidecar-services) block.
- `envoy_extra_static_clusters_json` - Specifies one or more [Envoy clusters](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/cluster/v3/cluster.proto)
that will be appended to the array of [static
clusters](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/bootstrap/v3/bootstrap.proto#envoy-v3-api-field-config-bootstrap-v3-bootstrap-staticresources-clusters)
in the bootstrap config. This enables you to add custom clusters for tracing sinks,
for example. In order to configure a single cluster, specify a single JSON object with the cluster details. For multiple
clusters, specify objects in a comma-separated list with no trailing comma. The
cluster objects will be interpolated directly into a JSON array.
<CodeBlockConfig heading="Example envoy_extra_static_clusters_json">
```json
{
"name": "local-service-cluster",
"load_assignment": {
"cluster_name": "local-service-cluster",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "127.0.0.1",
"port_value": 32769
}
}
}
}
]
}
]
}
}
```
</CodeBlockConfig>
- `envoy_extra_static_listeners_json` - Similar to
`envoy_extra_static_clusters_json` but appends one or more [Envoy listeners](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/listener/v3/listener.proto) to the array of [static
listener](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/bootstrap/v3/bootstrap.proto#envoy-v3-api-field-config-bootstrap-v3-bootstrap-staticresources-listeners) definitions.
Can be used to setup limited access that bypasses the service mesh's mTLS or
authorization for health checks or metrics.
<CodeBlockConfig heading="Example envoy_extra_static_listeners_json">
```json
{
"name": "test_envoy_mtls_bypass_listener",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": 20201
}
},
"filter_chains": [
{
"filters": [
{
"name": "envoy.filters.network.http_connection_manager",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"http_filters": [
{
"name": "envoy.filters.http.router"
}
],
"route_config": {
"name": "self_admin_route",
"virtual_hosts": [
{
"name": "self_admin",
"domains": [
"*"
],
"routes": [
{
"match": {
"path": "/"
},
"route": {
"cluster": "local-service-cluster"
}
}
]
}
]
},
"stat_prefix": "envoy_mtls_bypass",
"tracing": {
"random_sampling": {}
}
}
}
]
}
]
}
```
</CodeBlockConfig>
- `envoy_extra_stats_sinks_json` - Similar to `envoy_extra_static_clusters_json`
but for [stats sinks](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/bootstrap/v3/bootstrap.proto#envoy-v3-api-field-config-bootstrap-v3-bootstrap-stats-sinks).
These are appended to any sinks defined by use of the
higher-level [`envoy_statsd_url`](#envoy_statsd_url) or
[`envoy_dogstatsd_url`](#envoy_dogstatsd_url) config options.
<CodeBlockConfig heading="Example envoy_extra_stats_sinks_json">
```json
{
"name": "envoy.stat_sinks.dog_statsd",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.metrics.v3.DogStatsdSink",
"address": {
"socket_address": {
"protocol": "UDP",
"port_value": 8125,
"address": "172.31.20.6"
}
}
}
}
```
</CodeBlockConfig>
- `envoy_stats_config_json` - The entire [stats
config](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/bootstrap/v3/bootstrap.proto#envoy-v3-api-field-config-bootstrap-v3-bootstrap-stats-config).
If provided this will override the higher-level
[`envoy_stats_tags`](#envoy_stats_tags). It allows full control over dynamic
tag replacements etc.
<CodeBlockConfig heading="Example envoy_stats_config_json">
```json
{
"stats_matcher": {
"reject_all": true
},
"stats_tags": [
{
"tag_name": "envoy.http_user_agent",
"regex": "^http(?=\\.).*?\\.user_agent\\.((.+?)\\.)\\w+?$"
}
],
"use_all_default_tags": false
}
```
</CodeBlockConfig>
- `envoy_tracing_json` - The entire [tracing
config](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/bootstrap/v3/bootstrap.proto#envoy-v3-api-field-config-bootstrap-v3-bootstrap-tracing).
Most tracing providers will also require adding static clusters to define the
endpoints to send tracing data to.
<CodeBlockConfig heading="Example envoy_tracing_json">
```json
{
"http": {
"name": "envoy.tracers.zipkin",
"typedConfig": {
"@type": "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
"collector_cluster": "zipkin",
"collector_endpoint_version": "HTTP_JSON",
"collector_endpoint": "/api/v1/spans",
"shared_span_context": false
}
}
}
```
</CodeBlockConfig>
### Escape-Hatch Overrides
Users may add the following configuration items to the [global `proxy-defaults`
configuration
entry](/consul/docs/connect/config-entries/proxy-defaults) or
override them directly in the `proxy.config` field of a [proxy service
definition](/consul/docs/connect/proxies/proxy-config-reference) or
[`sidecar_service`](/consul/docs/connect/proxies/deploy-sidecar-services) block.
- `envoy_bootstrap_json_tpl` - Specifies a template in Go template syntax that
is used in place of [the default
template](https://github.com/hashicorp/consul/blob/71d45a34601423abdfc0a64d44c6a55cf88fa2fc/command/connect/envoy/bootstrap_tpl.go#L129)
when generating bootstrap via [`consul connect envoy`
command](/consul/commands/connect/envoy). The variables that are available
to be interpolated are [documented
here](https://github.com/hashicorp/consul/blob/71d45a34601423abdfc0a64d44c6a55cf88fa2fc/command/connect/envoy/bootstrap_tpl.go#L5).
This offers complete control of the proxy's bootstrap although major
deviations from the default template may break Consul's ability to correctly
manage the proxy or enforce its security model.
- `envoy_public_listener_json` - Specifies a complete [Envoy listener](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/listener/v3/listener.proto)
to be delivered in place of the main public listener that the proxy used to
accept inbound connections. This will be used verbatim with the following
exceptions:
- Every `FilterChain` added to the listener will have its `TlsContext`
overridden by the Connect TLS certificates and validation context. This
means there is no way to override the service mesh's mutual TLS for the public
listener.
- Every `FilterChain` will have the `envoy.filters.{network|http}.rbac` filter
prepended to the filters array to ensure that all inbound connections are
authorized by the service mesh. Before Consul 1.9.0 `envoy.ext_authz` was inserted instead.
<CodeTabs heading="Example envoy_public_listener_json" tabs={[ "HTTP", "TCP" ]}>
```json
{
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "public_listener",
"address": {
"socket_address": {
"address": "127.0.0.1",
"port_value": 21002
}
},
"filter_chains": [
{
"filters": [
{
"name": "envoy.filters.network.http_connection_manager",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"stat_prefix": "ingress_http",
"http_filters": [
{
"name": "envoy.filters.http.router"
}
],
"route_config": {
"name": "local_route",
"virtual_hosts": [
{
"name": "local_service",
"domains": ["*"],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"cluster": "local-service-cluster",
}
}
]
}
]
}
}
}
]
}
],
"traffic_direction": "INBOUND"
}
```
```json
{
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "public_listener",
"address": {
"socket_address": {
"address": "127.0.0.1",
"port_value": 21002
}
},
"filter_chains": [
{
"filters": [
{
"name": "envoy.filters.network.tcp_proxy",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
"stat_prefix": "ingress_tcp",
"cluster": "local-service-cluster"
}
}
]
}
],
"traffic_direction": "INBOUND"
}
```
</CodeTabs>
- `envoy_listener_tracing_json` - Specifies a [tracing
configuration](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#envoy-v3-api-msg-extensions-filters-network-http-connection-manager-v3-httpconnectionmanager-tracing)
to be inserted in the proxy's public and upstreams listeners.
<CodeBlockConfig heading="Example envoy_listener_tracing_json">
```json
{
"@type" : "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager.Tracing",
"provider" : {
"name" : "envoy.tracers.zipkin",
"typed_config" : {
"@type" : "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
"collector_cluster" : "otelcolector",
"collector_endpoint" : "/api/v2/spans",
"collector_endpoint_version" : "HTTP_JSON",
"shared_span_context" : false
}
},
"custom_tags" : [
{
"tag" : "custom_header",
"request_header" : {
"name" : "x-custom-traceid",
"default_value" : ""
}
},
{
"tag" : "alloc_id",
"environment" : {
"name" : "NOMAD_ALLOC_ID"
}
}
]
}
```
</CodeBlockConfig>
- `envoy_local_cluster_json` - Specifies a complete [Envoy cluster](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/cluster/v3/cluster.proto)
to be delivered in place of the local application cluster. This allows
customization of timeouts, rate limits, load balancing strategy etc.
<CodeBlockConfig heading="Example envoy_local_cluster_json">
```json
{
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "local_app",
"type": "STATIC",
"connect_timeout": "5s",
"circuit_breakers": {
"thresholds": [
{
"priority": "DEFAULT",
"max_connections": 2048
}
]
},
"load_assignment": {
"cluster_name": "local_app",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "127.0.0.1",
"port_value": 8080
}
}
}
}
]
}
]
}
}
```
</CodeBlockConfig>
The following configuration items may be overridden directly in the
`proxy.upstreams[].config` field of a [proxy service
definition](/consul/docs/connect/proxies/proxy-config-reference) or
[`sidecar_service`](/consul/docs/connect/deploy/deploy-sidecar-services) block.
~> **Note:** - When a
[`service-router`](/consul/docs/connect/config-entries/service-router),
[`service-splitter`](/consul/docs/connect/config-entries/service-splitter), or
[`service-resolver`](/consul/docs/connect/config-entries/service-resolver) config
entry exists for a service the below escape hatches are ignored and will log a
warning.
- `envoy_listener_json` - Specifies a complete [Listener](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/listener/v3/listener.proto)
to be delivered in place of the upstream listener that the proxy exposes to
the application for outbound connections. This will be used verbatim with the
following exceptions:
- Every `FilterChain` added to the listener will have its `TlsContext`
overridden by the service mesh TLS certificates and validation context. This
means there is no way to override the service mesh's mutual TLS for the public
listener.
<CodeTabs heading="Example upstream envoy_listener_json">
```json
{
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "example-service",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": 14000
}
},
"filter_chains": [
{
"filters": [
{
"name": "envoy.filters.network.http_connection_manager",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"access_log": [
{
"name": "envoy.access_loggers.file",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
"path": "/var/log/envoy-access/example-service.log"
}
}
],
"http_filters": [
{
"name": "envoy.filters.http.router"
}
],
"route_config": {
"name": "example-service",
"virtual_hosts": [
{
"name": "example-service",
"domains": [
"*"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"cluster": "example-service",
"timeout": "90s",
"retry_policy": {
"retry_on": "5xx,connect-failure",
"num_retries": 2,
"per_try_timeout": "60s"
}
}
}
]
}
]
},
"stat_prefix": "example-service",
"tracing": {
"random_sampling": {}
}
}
}
]
}
],
"traffic_direction": "OUTBOUND"
}
```
</CodeTabs>
- `envoy_cluster_json` - Specifies a complete [Envoy cluster](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/cluster/v3/cluster.proto)
to be delivered in place of the discovered upstream cluster. This allows
customization of timeouts, circuit breaking, rate limits, load balancing
strategy etc.
<CodeTabs heading="Example upstream envoy_cluster_json">
```json
{
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "example-service",
"type": "EDS",
"eds_cluster_config": {
"eds_config": {
"ads": {}
}
},
"connect_timeout": "90s",
"lb_policy": "ROUND_ROBIN",
"circuit_breakers": {
"thresholds": [
{
"priority": "DEFAULT",
"max_connections": 1024,
"max_pending_requests": 1024,
"max_requests": 1024,
"max_retries": 3
}
]
}
}
```
</CodeTabs>