mirror of https://github.com/hashicorp/consul
368 lines
17 KiB
Markdown
368 lines
17 KiB
Markdown
---
|
|
layout: commands
|
|
page_title: 'Commands: Connect Envoy'
|
|
sidebar_title: envoy
|
|
description: The connect envoy subcommand is used to generate a bootstrap configuration for Envoy.
|
|
---
|
|
|
|
# Consul Connect Envoy
|
|
|
|
Command: `consul connect envoy`
|
|
|
|
The connect Envoy command is used to generate a bootstrap configuration for
|
|
[Envoy proxy](https://envoyproxy.io) for use with [Consul
|
|
service mesh](/consul/docs/connect/).
|
|
|
|
Refer to the [examples](#examples) for guidance on common use cases,
|
|
such as [launching a service instance's sidecar proxy
|
|
when ACLs are enabled](#sidecar-proxy-with-acls-enabled).
|
|
|
|
The default behavior is to generate the necessary bootstrap configuration for
|
|
Envoy based on the environment variables and options provided and by talking to
|
|
the local Consul agent. It `exec`s an external Envoy binary with that
|
|
configuration leaving the Envoy process running in the foreground. An error is
|
|
returned on operating systems other than linux or macOS since Envoy does not
|
|
build for other platforms currently.
|
|
|
|
If the `-bootstrap` option is specified, the bootstrap config is generated in
|
|
the same way and then printed to stdout. This allows it to be redirected to a
|
|
file and used with `envoy --config-path bootstrap.json`. This works on all operating
|
|
systems allowing configuration to be generated on a host that Envoy doesn't
|
|
build on but then used in a virtualized environment that can run Envoy.
|
|
|
|
## Usage
|
|
|
|
Usage: `consul connect envoy [options] [-- pass-through options]`
|
|
|
|
#### Envoy Options for both Sidecars and Gateways
|
|
|
|
- `-proxy-id` - The [proxy service](/consul/docs/connect/registration/service-registration) ID.
|
|
This service ID must already be registered with the local agent unless a gateway is being
|
|
registered with the `-register` flag. As of Consul 1.8.0, this can also be
|
|
specified via the `CONNECT_PROXY_ID` environment variable.
|
|
|
|
- `-envoy-binary` - The full path to a specific Envoy binary to exec. By
|
|
default the current `$PATH` is searched for `envoy`.
|
|
|
|
- `-admin-bind` - The `host:port` to bind Envoy's admin HTTP API. Default is
|
|
`localhost:19000`. Envoy requires that this be enabled. The host part must be
|
|
resolvable DNS name or IP address.
|
|
|
|
- `-bootstrap` - If present, the command will simply output the generated
|
|
bootstrap config to stdout in JSON protobuf form. This can be directed to a
|
|
file and used to start Envoy with `envoy --config-path bootstrap.json`.
|
|
|
|
~> **Security Note:** If ACLs are enabled the bootstrap JSON will contain the
|
|
ACL token from `-token` or the environment and so should be handled as a secret.
|
|
This token grants the identity of any service it has `service:write` permission
|
|
for and so can be used to access any upstream service that that service is
|
|
allowed to access by [service mesh intentions](/consul/docs/connect/intentions).
|
|
|
|
- `-envoy-version` - The version of envoy that is being started. Default is
|
|
`1.23.1`. This is required so that the correct configuration can be generated.
|
|
|
|
- `-grpc-ca-file` - Path to a CA file to use for TLS when communicating with the
|
|
Consul agent through xDS. This can also be specified via the CONSUL_GRPC_CACERT
|
|
environment variable.
|
|
|
|
- `-grpc-ca-path` - Path to a directory of CA certificates to use for TLS when
|
|
communicating with the Consul agent through xDS. This can also be specified via
|
|
the CONSUL_GRPC_CAPATH environment variable.
|
|
|
|
- `-no-central-config` - By default the proxy's bootstrap configuration can be
|
|
customized centrally. This requires that the command run on the same agent
|
|
as the proxy will and that the agent is reachable when the command is run.
|
|
In cases where either assumption is violated this flag will prevent the
|
|
command attempting to resolve config from the local agent.
|
|
|
|
- `-envoy-ready-bind-address` - By default the proxy does not have a readiness probe
|
|
configured on it. This flag in conjunction with the `envoy-ready-bind-port` flag
|
|
configures where the envoy readiness probe is configured on the proxy. A `/ready` HTTP
|
|
endpoint will be instantiated at the specified address and port. When configured, Consul
|
|
uses `/ready` HTTP endpoints to check proxy health.
|
|
|
|
- `-envoy-ready-bind-port` - By default the proxy does not have a readiness probe
|
|
configured on it. This flag in conjunction with the `envoy-ready-bind-address` flag
|
|
configures where the envoy readiness probe is configured on the proxy. A `/ready` HTTP
|
|
endpoint will be instantiated at the specified address and port. When configured, Consul
|
|
uses `/ready` HTTP endpoints to check proxy health.
|
|
|
|
- `-prometheus-backend-port` - Sets the backend port for the "prometheus_backend"
|
|
cluster that `envoy_prometheus_bind_addr` will point to. Without this flag,
|
|
`envoy_prometheus_bind_addr` would point to the "self_admin" cluster where Envoy metrics
|
|
are exposed. The metrics merging feature in consul-k8s uses this to point to the
|
|
merged metrics endpoint combining Envoy and service metrics.
|
|
Only applicable when `envoy_prometheus_bind_addr` is set in proxy config.
|
|
|
|
- `-prometheus-scrape-path` - Sets the path where Envoy will expose metrics on the
|
|
`envoy_prometheus_bind_addr` listener. Default is `/metrics`. For example, if `envoy_prometheus_bind_addr`
|
|
is `0.0.0.0:20200`, and this flag is set to `/scrape-metrics`, prometheus metrics would
|
|
be scrapable at `0.0.0.0:20200/scrape-metrics`.
|
|
Only applicable when `envoy_prometheus_bind_addr` is set in proxy config.
|
|
|
|
- `-prometheus-ca-file` - Path to a CA file for Envoy to use when serving TLS on
|
|
the Prometheus metrics endpoint. Only applicable when `envoy_prometheus_bind_addr`
|
|
is set in proxy config.
|
|
|
|
- `-prometheus-ca-path` - Path to a directory of CA certificates for Envoy to use when
|
|
serving the Prometheus metrics endpoint. Only applicable when `envoy_prometheus_bind_addr`
|
|
is set in proxy config.
|
|
|
|
- `-prometheus-cert-file` - Path to a certificate file for Envoy to use when serving
|
|
TLS on the Prometheus metrics endpoint. Only applicable when `envoy_prometheus_bind_addr`
|
|
is set in proxy config.
|
|
|
|
- `-prometheus-key-file` - Path to a private key file for Envoy to use when serving
|
|
TLS on the Prometheus metrics endpoint. Only applicable when `envoy_prometheus_bind_addr`
|
|
is set in proxy config.
|
|
|
|
- `-ignore-envoy-compatibility` - If set to `true`, this flag ignores the Envoy version
|
|
compatibility check. We recommend setting this flag to `false` to ensure
|
|
compatibility with Envoy and prevent potential issues. Default is `false`.
|
|
|
|
- `-- [pass-through options]` - Any options given after a double dash are passed
|
|
directly through to the `envoy` invocation. See [Envoy's
|
|
documentation](https://www.envoyproxy.io/docs) for more details. The command
|
|
always specifies `--config-file` and `--v2-config-only` and by default passes
|
|
`--disable-hot-restart` see [hot restart](#envoy-hot-restart).
|
|
|
|
- `-enable-config-gen-logging` - This flag enables debug message logging when generating the Envoy bootstrap configuration to help troubleshoot issues. Logs output to `stderr`. For more information about generating the Envoy bootstrap configuration, refer to [Envoy proxy configuration](/consul/docs/connect/proxies/envoy#bootstrap-configuration).
|
|
|
|
#### Envoy Sidecar Proxy Options
|
|
|
|
- `-sidecar-for` - The _ID_ (not name if they differ) of the service instance
|
|
this proxy will represent. The target service doesn't need to exist on the
|
|
local agent yet but a [sidecar proxy
|
|
registration](/consul/docs/connect/registration/service-registration) with
|
|
`proxy.destination_service_id` equal to the passed value must be present. If
|
|
multiple proxy registrations targeting the same local service instance are
|
|
present the command will error and `-proxy-id` should be used instead.
|
|
As of Consul 1.8.0, this can also be specified via the `CONNECT_SIDECAR_FOR`
|
|
environment variable.
|
|
|
|
-> **Note:** If ACLs are enabled, a token granting `service:write` for the
|
|
_target_ service (configured in `proxy.destination_service_name`) must be
|
|
passed using the `-token` option or `CONSUL_HTTP_TOKEN` environment variable.
|
|
This token authorizes the proxy to obtain TLS certificates representing the
|
|
target service.
|
|
|
|
#### Envoy Gateway Options
|
|
|
|
- `-gateway` - Flag to indicate that Envoy should be configured as a Gateway.
|
|
Must be one of: `terminating`, `ingress`, or `mesh`.
|
|
If multiple gateways are managed by the same local agent then
|
|
`-proxy-id` should be used as well to specify the instance this represents.
|
|
|
|
- `-register` - Indicates that the gateway service should be registered
|
|
with the local agent instead of expecting it to already exist. This flag
|
|
is unused for traditional sidecar proxies.
|
|
|
|
- `-address` - The address to advertise for services within the local datacenter
|
|
to use to reach the gateway instance. This flag is used in combination with
|
|
`-register`. This takes the form of `<ip address>:<port>` but also supports go-sockaddr
|
|
templates.
|
|
|
|
If Envoy is configured as a terminating or mesh gateway, traffic from services
|
|
within the mesh will be received at the specified IP and port.
|
|
|
|
If Envoy is configured as an ingress gateway, Consul instantiates a `/ready` HTTP endpoint at the specified IP and port. Consul uses `/ready` HTTP endpoints
|
|
to check gateway health. Ingress gateways also use the specified IP when instantiating user-defined listeners configured in the
|
|
[ingress gateway configuration entry](/consul/docs/connect/config-entries/ingress-gateway).
|
|
|
|
~> **Note**: Ensure that user-defined ingress gateway listeners use a
|
|
different port than the port specified in `-address` so that they do not
|
|
conflict with the health check endpoint.
|
|
|
|
- `-admin-access-log-path` -
|
|
**Deprecated in Consul 1.15.0 in favor of [`proxy-defaults` access logs](/consul/docs/connect/config-entries/proxy-defaults#accesslogs).**
|
|
The path to write the access log for the administration
|
|
server. If no access log is desired specify `/dev/null`. By default it will
|
|
use `/dev/null`.
|
|
|
|
- `-bind-address` - The bind address to use instead of the default binding rules.
|
|
Specified as `<name>=<ip>:<port>` pairs. This flag may be used multiple times
|
|
to add multiple bind addresses.
|
|
|
|
- `-expose-servers` - Expose the servers for WAN federation via this mesh
|
|
gateway.
|
|
|
|
- `-wan-address` - The address to advertise for services within remote datacenters
|
|
to use to reach the gateway instance. This flag is used in combination with
|
|
`-register`. This takes the form of `<ip address>:<port>` but also supports go-sockaddr
|
|
templates.
|
|
|
|
- `-service` - The name of the gateway service to register. This flag is used
|
|
in combination with `-register`.
|
|
|
|
- `-deregister-after-critical` - The amount of time the gateway services health check can
|
|
be failing before being deregistered. This flag is used in combination with `-register`
|
|
|
|
#### Enterprise Options
|
|
|
|
@include 'http_api_partition_options.mdx'
|
|
|
|
@include 'http_api_namespace_options.mdx'
|
|
|
|
#### API Options
|
|
|
|
The standard API options are used to connect to the local agent to discover the
|
|
proxy configuration needed.
|
|
|
|
- `-grpc-addr=<addr>` - Address of the Consul agent with `grpc` port. This can
|
|
be an IP address or DNS address, but it must include the port. This can also
|
|
be specified via the CONSUL_GRPC_ADDR environment variable. In Consul 1.3 and
|
|
later, the default value is 127.0.0.1:8502, and https can optionally
|
|
be used instead. The scheme can also be set to HTTPS by setting the
|
|
environment variable CONSUL_HTTP_SSL=true. This may be a unix domain socket
|
|
using `unix:///path/to/socket` if the [agent is configured to
|
|
listen](/consul/docs/agent/config/config-files#addresses) that way.
|
|
|
|
-> **Note:** gRPC uses the same TLS
|
|
settings as the HTTPS API. If HTTPS is enabled then gRPC will require HTTPS
|
|
as well.
|
|
|
|
@include 'http_api_options_client.mdx'
|
|
|
|
## Examples
|
|
|
|
Assume a local service instance is registered on the local agent with a
|
|
sidecar proxy (using the [sidecar service
|
|
registration](/consul/docs/connect/registration/service-registration) helper) as below.
|
|
|
|
```hcl
|
|
service {
|
|
name = "web"
|
|
port = 8080
|
|
connect { sidecar_service {} }
|
|
}
|
|
```
|
|
|
|
### Basic sidecar proxy
|
|
|
|
The sidecar Envoy process can be started with.
|
|
|
|
```shell-session
|
|
$ consul connect envoy -sidecar-for web
|
|
```
|
|
|
|
This example assumes that the correct [environment variables](#api-options) are
|
|
used to set the local agent connection information and ACL token, or that the
|
|
agent is using all-default configuration.
|
|
|
|
### Sidecar proxy with ACLs enabled
|
|
|
|
In secure deployments, Consul's ACL system is enabled with a default `deny` policy.
|
|
To access Consul API resources, an API request must present a Consul ACL token
|
|
with the necessary privileges. If ACLs are enabled, you must provide an ACL token to the sidecar proxy. The token must grant the proxy privileges to register itself and the service it fronts and to access all potential upstreams of that service.
|
|
We recommend [using a service identity](/consul/commands/acl/token/create#create-a-token-for-a-service)
|
|
to directly create a token with the privileges necessary for a service and its sidecar proxy,
|
|
rather than creating a unique ACL policy for every service.
|
|
|
|
After creating a token using a service identity, provide the token when
|
|
launching the Envoy sidecar proxy instance with the `consul connect envoy` command.
|
|
You can provide the token through an environment variable or CLI flag.
|
|
|
|
Environment variables:
|
|
- [`CONSUL_HTTP_TOKEN`](/consul/commands#consul_http_token) or
|
|
- [`CONSUL_HTTP_TOKEN_FILE`](/consul/commands#consul_http_token_file)
|
|
|
|
CLI flags:
|
|
- [`-token`](/consul/commands/connect/envoy#token) or
|
|
- [`-token-file`](/consul/commands/connect/envoy#token-file)
|
|
|
|
```shell-session
|
|
$ export CONSUL_HTTP_TOKEN="<secret-id-of-token-with-service-identity-for-web>"
|
|
$ consul connect envoy -sidecar-for web
|
|
```
|
|
|
|
### Additional Envoy Arguments
|
|
|
|
To pass additional arguments directly to Envoy, for example output logging
|
|
level, you can use:
|
|
|
|
```shell-session
|
|
$ consul connect envoy -sidecar-for web -- --log-level debug
|
|
```
|
|
|
|
### Multiple Proxy Instances
|
|
|
|
To run multiple different proxy instances on the same host, you will
|
|
need to use `-admin-bind` on all but one to ensure they don't attempt to bind to
|
|
the same port as in the following example.
|
|
|
|
```shell-session
|
|
$ consul connect envoy -sidecar-for db -admin-bind localhost:19001
|
|
```
|
|
|
|
### Mesh Gateways
|
|
|
|
The mesh gateway Envoy process can be auto-registered and started with the following command.
|
|
|
|
```shell-session
|
|
$ consul connect envoy -gateway=mesh -register \
|
|
-address '{{ GetInterfaceIP "eth0" }}:8443' \
|
|
-wan-address '{{ GetInterfaceIP "eth1" }}:8443'
|
|
```
|
|
|
|
### Terminating Gateways
|
|
|
|
The terminating gateway Envoy process can be auto-registered and started with the following command.
|
|
|
|
```shell-session
|
|
$ consul connect envoy -gateway=terminating -register -service my-gateway \
|
|
-address '{{ GetInterfaceIP "eth0" }}:8443'
|
|
```
|
|
|
|
### Ingress Gateways
|
|
|
|
The ingress gateway Envoy process can be auto-registered and started with the following command.
|
|
|
|
```shell-session
|
|
$ consul connect envoy -gateway=ingress -register -service ingress-service \
|
|
-address '{{ GetInterfaceIP "eth0" }}:8888'
|
|
```
|
|
|
|
## Exec Security Details
|
|
|
|
The command needs to pass the bootstrap config through to Envoy. Envoy currently
|
|
only supports passing this as a file path or passing a whole string on the
|
|
command line with `--config-yaml`. Since the bootstrap needs to contain the ACL
|
|
token to authorize the proxy, this secret needs careful handling.
|
|
|
|
Passing a secret via command option is unacceptable as on many unix systems
|
|
these are readable to any user on the host for example via `/proc` or via a
|
|
setuid process like `ps`.
|
|
|
|
Creating a temporary file is more secure in that it can only be read by the
|
|
current user but risks leaving secret material on disk for an unbounded length
|
|
of time and in a location that is opaque to the operator.
|
|
|
|
To work around these issues, the command currently creates a temporary file and
|
|
immediately unlinks it so it can't be read by any other process that doesn't
|
|
already have the file descriptor. It then writes the bootstrap JSON, and unsets
|
|
the CLOEXEC bit on the file handle so that it remains available to the Envoy
|
|
process after exec. Finally it `exec`s Envoy with `--config-file /dev/fd/X`
|
|
where `X` is the the file descriptor number of the temp file.
|
|
|
|
This ensures that Envoy can read the file without any other normal user process
|
|
being able to (assuming they don't have privileged access to /proc). Once the
|
|
Envoy process stops, there is no longer any reference to the file to clean up.
|
|
|
|
## Envoy Hot Restart
|
|
|
|
Envoy supports hot restart which requires simple external coordination. By
|
|
default, this command will add `--disable-hot-restart` when it runs Envoy.
|
|
|
|
The reason for this default behavior is to make it easy to test and run local
|
|
demonstrations with multiple Envoy instances outside of cgroups or network
|
|
namespaces.
|
|
|
|
To use hot restart, Envoy needs to be started with either the `--restart-epoch`
|
|
option. If this command detects that option in the pass-through flags it will
|
|
_not_ add `--disable-hot-restart` allowing hot restart to work normally.
|
|
|
|
The only difference to note over running Envoy directly is that
|
|
`--restart-epoch` must be explicitly set to `0` for the initial launch of the
|
|
Envoy instance to avoid disabling hot restart entirely. The official
|
|
`hot-restarter.py` always sets this option so should work as recommended.
|