Any proxy can be extended to support Connect. Consul ships with a built-in
This topic describes the process and API endpoints you can use to extend proxies for integration with Consul.
proxy for a good development and out of the box experience, but production
users will require other proxy solutions.
## Overview
A proxy must serve one or both of the following two roles: it must accept
You can extend any proxy to support Connect. Consul ships with a built-in
inbound connections or establish outbound connections identified as a
proxy suitable for an out-of-the-box development experience, but you may require a more robust proxy solution for production environments.
particular service. One or both of these may be implemented depending on the
case, although generally both must be supported for full sidecar functionality.
The proxy you integrate must be able to accept inbound connections and/or establish outbound connections identified as a particular service. In some cases, either ability may be acceptable, but both are generally required to support for full sidecar functionality.
There are also two different levels of compatibility as a sidecar: L4 or L7.
Sidecar proxies may support L4 or L7 network functionality. L4 integration is simpler and adequate for securing all traffic. L4 treats all traffic as TCP, however, so advanced routing or metrics features are not supported.
L4 integration is simpler and adequate to secure all traffic but treats all
traffic as TCP so no advanced routing or metrics features can be supported.
Full L7 support is built on top of L4 support. An L7 proxy integration supports most or all of the L7 traffic routing features in Connect by dynamically configuring routing, retries, and other L7 features. The built-in proxy only supports L4, while Envoy supports the full L7 feature set.
Full L7 support is built on top of L4 support and includes supporting most or
all of the L7 traffic routing features in Connect by dynamically configuring
Areas where the integration approach differs between L4 and L7 are identified in this topic.
routing, retries and more L7 features. Currently The built-in proxy only
supports L4 while Envoy supports the full L7 feature set.
Places where the integration approach diverges for L4/L7 support is indicated
below.
## Accepting Inbound Connections
## Accepting Inbound Connections
For inbound connections, the proxy must accept TLS connections on some port.
The proxy must accept TLS connections on some port to accept inbound connections.
The certificate served should be obtained from the
[`/v1/agent/connect/ca/leaf/`] API endpoint. The client certificate should be
validated against the root certificates provided by the
[`/v1/agent/connect/ca/roots`] endpoint. After validating the client
certificate from the caller, depending upon the [protocol] of the proxied
service service the proxy must either authorize the entire connection (L4) or
each request (L7).
Connection authorization can be performed one of two ways:
1. The first is by calling the
[`/v1/agent/connect/authorize`](/api/agent/connect) endpoint. The authorize
endpoint is expected to be called in the connection path, so if the local
Consul agent is down or unresponsive it will impact the success rate of new
connections. The agent uses locally cached data to authorize the connection
and typically responds in microseconds. Therefore, the impact to the TLS
handshake is typically microseconds.
~> **Note:** This endpoint is only suited for networking layer 4 (e.g. TCP)
integration. The endpoint will always treat intentions with Permissions
defined (i.e., layer 7 criteria) as deny intentions during evaluation.
2. Alternatively, proxies may list intentions that match the destination by
querying the [intention match
API](/api/connect/intentions#list-matching-intentions) endpoint, and
represent them in the native configuration of the proxy itself (such as RBAC
for Envoy). For performance and reliability reasons this is the desirable
method for implementing intention enforcement. The cached intentions should
be consulted for each incoming connection (L4) or request (L7) to determine
if the should be accepted or rejected.
All of these API endpoints operate on agent-local data that is updated in the
background. The leaf, roots, and intentions should be updated in the background
by the proxy.
The leaf cert, root cert, and intentions endpoints support [blocking
### Obtaining and validating client certificates
queries](/api/features/blocking), which should be used to get near-immediate
updates for root key rotations, new leaf certs before expiry, and intention
changes.
Although Consul follows the SPIFFE spec for certificates, some currently
Call the [`/v1/agent/connect/ca/leaf/`] API endpoint to obtain the client certificate, e.g.:
supported CA providers don't allow strict adherence. For example, CA
certificates may not have the correct trust-domain SPIFFE URI SAN for the
```shell-session
cluster. If SPIFFE validation is performed in the proxy, be aware that it
should be possible to opt out, otherwise certain CA providers supported by
After validating the client certificate from the caller, the proxy can authorize the entire connection (L4) or each request (L7). Depending upon the [protocol] of the proxied service, authorization is performed either on a per-connection (L4) or per-request (L7) basis. Authentication is based on "service identity" (TLS), and is implemented at the
transport layer.
Authentication is based on "service identity" (TLS), and is implemented at the
-> **Note:** Some features, such as (local) rate limiting or max connections, are expected to be proxy-level configurations enforced separately when authorization calls are made. Proxies can enforce the configurations based on information about request rates and other states that should already be availabe.
transport layer. Depending upon the [protocol] of the proxied service,
authorization is performed either on a per-connection (L4) or per-request (L7)
basis.
-> **Note:** Features like (local) rate limiting or max connections are
The proxy can authorize the connection by either calling the [`/v1/agent/connect/authorize`](/api/agent/connect) API endpoint or by querying the [intention match API](/api/connect/intentions#list-matching-intentions) endpoint.
configurations that we expect to push into proxies and have them enforce
separately to the AuthZ call based on the state they already have about request
rates etc.
#### Persistent TCP Connections and Intentions
The [`/v1/agent/connect/authorize`](/api/agent/connect) endpoint should be called in the connection path.
If the local Consul agent is down or unresponsive, the success rate of new connections will be compromised.
The agent uses locally-cached data to authorize the connection and typically responds in microseconds. As a result, the TLS handshake typically spans microseconds.
For a proxied service configured with a [protocol] of TCP, potentially
~> **Note:** This endpoint is only suitable for L4 (e.g., TCP) integration. The endpoint always treats intentions with `Permissions` defined (i.e., L7 criteria) as `deny` intentions during evaluation.
long-lived TCP connections will be authorized only when they are established.
Since many services (e.g. databases) typically use persistent connection pools,
a change in intentions that newly denies access currently does not terminate
existing connections in violation of the updated intention. In this case it may
appear as if the intention is not being enforced.
Consul eventually may support a mechanism for tracking specific connections in
The proxy can query the [intention match API](/api/connect/intentions#list-matching-intentions) endpoint, which lists intentions that match the proxy destination. The matches are represented in the native configuration of the proxy, such as RBAC for Envoy.
the agent and then allow the agent to tell the proxy to close those connections
when their authorization state changes, but for now that is not on the roadmap.
It is recommended therefore to do one of the following:
For performance and reliability reasons, querying the intention match API endpoint is the recommended method for implementing intention enforcement. The cached intentions should be consulted for each incoming connection (L4) or request (L7) to determine if the should be accepted or rejected.
1. Have connections terminate after a configurable maximum lifetime of say
#### Persistent TCP connections and intentions
several hours. This balances the overhead of establishing new connections
while keeping an upper bound on how long after Intention changes existing
connections remain open.
2. Periodically re-authorize every open connection. The AuthZ call itself is
not expensive and should be a local, in-memory operation so authorizing
thousands of open connections once every minute or so is likely to be
negligible overhead, but enforces a tighter upper bound on how long it takes
to enforce Intention changes without affecting protocol efficiency of
persistent connections.
#### Certificate Serial in AuthZ
For a proxied service configured with the TCP [protocol], potentially long-lived TCP connections will only be authorized when the connections are established. But because many services, such as databases, typically use persistent connection pools, changing intentions to deny access does not terminate existing connections. This behavior violates the updated intention. In these cases, it may appear as if the intention is not being enforced.
Intentions currently utilize TLS' URI Subject Alternative Name (SAN) for
Implement one of the following strategies to close connections:
enforcement. In the future, Consul will support revoking specific certificates
by serial number. The AuthZ API in the Go SDK has a field to pass the serial
1. **Configure connections to terminate after a maximum lifetime**, e.g., several hours. This balances the overhead of establishing new connections with determining how long existing connections remain open after an intention changes.
number ([`consul/connect/tls.go`]). Proxies may provide this value during
authorization.
1. **Periodically re-authorize every open connection**. The authorization call is inexpensive and should be a local, in-memory operation. Periodically authorizing thousands of open connections (e.g., once every minute) is likely to be negligible overhead, but doing so enforces a tighter upper boundary on how long it takes to enforce intention changes without affecting the protocol efficiency of persistent connections.
#### Certificate serial in authorization
Intentions currently use TLS URI Subject Alternative Name (SAN) for enforcement. The `AuthZ` API in the Go SDK contains a field for passing the serial number ([`consul/connect/tls.go`]). Proxies may provide this value during authorization.
### Updating data
The API endpoints described in this section operate on agent-local data that is updated in the
background. The leaf, roots, and intentions should be updated in the background
by the proxy.
The leaf cert, root cert, and intentions endpoints support [blocking
queries](/api/features/blocking), which should be used to get near-immediate
updates for root key rotations, new leaf certs before expiry, and intention
changes.
### SPIFFE certificates
Although Consul follows the SPIFFE spec for certificates, some CA providers do not allow strict adherence. For example, CA certificates may not have the correct trust-domain SPIFFE URI SAN for the
cluster. If SPIFFE validation is performed in the proxy, be aware that it
should be possible to opt out, otherwise certain CA providers supported by
Consul will not be compatible with the use of that proxy. Neither
Envoy nor the built-in proxy currently validate the SPIFFE URI of the chain beyond the
leaf certificate.
## Establishing Outbound Connections
## Establishing Outbound Connections
For outbound connections, the proxy should communicate to a Connect-capable
For outbound connections, the proxy should communicate with a Connect-capable
endpoint for a service and provide a client certificate from the
endpoint for a service and provide a client certificate from the
[`/v1/agent/connect/ca/leaf/`] API endpoint. The certificate served by the
[`/v1/agent/connect/ca/leaf/`] API endpoint. The proxy can call the [`/v1/agent/connect/ca/roots`] endpoint to verify the certificate served from the remote endpoint against the root certificates.
remote endpoint may be verified against the root certificates from the
[`/v1/agent/connect/ca/roots`] endpoint.
## Configuration Discovery
## Configuration Discovery
Any proxy can discover proxy configuration registered with a local service
The [`/v1/agent/service/:service_id`](/api/agent/service#get-service-configuration)
instance using the
API endpoint enables any proxy to discover proxy configurations registered with a local service. This endpoint supports hash-based blocking, which enables long-polling for changes
This endpoint supports hash-based blocking, enabling long-polling for changes
to the registration/configuration. Any changes to the registration/config will
to the registration/configuration. Any changes to the registration/config will
result in the new config being returned immediately. An example implementation
result in the new config being returned immediately.
may be found in our [built-in proxy](/docs/connect/proxies/built-in) which
utilizes our Go SDK, and uses the HTTP "pull" API (via our `watch` package):
Refer to the [built-in proxy](/docs/connect/proxies/built-in) for an example implementation. Using the Go SDK, the proxy calls the HTTP "pull" API via the `watch` package: [`consul/connect/proxy/config.go`].
[`consul/connect/proxy/config.go`].
The [discovery chain] for each upstream service should be fetched from the
The [discovery chain] for each upstream service should be fetched from the
API endpoint. This will return a compiled graph of configurations needed by
API endpoint. This will return a compiled graph of configurations a sidecar needs for a particular upstream service.
sidecars for a particular upstream service. If you are only implementing L4
support in your proxy, set the
If you are only implementing L4 support in your proxy, set the
[`OverrideProtocol`](/api/discovery-chain#overrideprotocol) value to "tcp" when
[`OverrideProtocol`](/api/discovery-chain#overrideprotocol) value to `tcp` when
fetching the discovery chain so that L7 features such as HTTP routing rules are
fetching the discovery chain so that L7 features, such as HTTP routing rules, are
not returned.
not returned.
For each [target](/docs/internals/discovery-chain#targets) in the resulting
For each [target](/docs/internals/discovery-chain#targets) in the resulting
discovery chain, a list of healthy, Connect-capable endpoints may be fetched
discovery chain, a list of healthy, Connect-capable endpoints may be fetched
from the [`/v1/health/connect/:service_id`] API endpoint per the [Service
from the [`/v1/health/connect/:service_id`] API endpoint as described in the [Service
Discovery](#service-discovery) section below.
Discovery](#service-discovery) section.
The rest of the nodes in the chain include configurations that should be
The remaining nodes in the chain include configurations that should be
translated into the nearest equivalent for things like HTTP routing, connection
translated into the nearest equivalent for features, such as HTTP routing, connection
timeouts, connection pool settings, rate limits, etc. See the full [discovery
timeouts, connection pool settings, rate limits, etc. See the full [discovery
chain] documentation and relevant [config entry](/docs/agent/config-entries)
chain] documentation and relevant [config entry](/docs/agent/config-entries)
documentation for details of supported configuration parameters.
documentation for details about supported configuration parameters.
We expect config here to evolve reasonably rapidly. While we do not intend to
make backwards incompatible API changes, there are likely to be new
configurations and features added regularly. Some proxies may not be able to
support all features or may have differing semantics with the way they support
them. We intend to find a suitable format to document the behavior differences
between proxy implementations as they mature.
### Service Discovery
### Service Discovery
Proxies can use Consul's service discovery API
Proxies can use Consul's [service discovery API]
[`/v1/health/connect/:service_id`] to return all available, Connect-capable
(`/v1/health/connect/:service_id`) to return all available, Connect-capable
endpoints for a given service. This endpoint supports a `?cached` parameter
endpoints for a given service. This endpoint supports a `cached` query parameter,
which makes use of [agent caching](/api/features/caching) and thus has
which uses [agent caching](/api/features/caching) to improve
performance benefits. The API package provides a [`UseCache`] query option to
performance. The API package provides a [`UseCache`] query option to
leverage this. In addition to performance improvements, use of the cache makes
leverage caching. In addition to performance improvements, using the cache makes
the mesh more resilient to Consul server outages - the mesh "fails static" with
the mesh more resilient to Consul server outages. This is because the mesh "fails static" with
the last known set of service instances still used rather than errors on new
the last known set of service instances still used, rather than errors on new
connections.
connections.
Proxies can decide whether to perform just-in-time queries to the API when a
Proxies can decide whether to perform just-in-time queries to the API when a
@ -201,35 +159,29 @@ the PreparedQuery API does not support blocking, so proxies choosing to
populate endpoints in memory will need to poll the endpoint at a suitable and
populate endpoints in memory will need to poll the endpoint at a suitable and
ideally configurable frequency.
ideally configurable frequency.
-> **Note:** Long-term the [`service-resolver` config
-> **Long-term support for [`service-resolver`](/docs/connect/config-entries/service-resolver) configuration
entries](/docs/connect/config-entries/service-resolver) are intended to replace
entries**. The `service-resolver` configuration will completely replace prepared queries in future versions of Consul. In some instances, however, prepared queries are still used.
Prepared Queries in Consul entirely, but for now these are still used in some
configurations.
## Sidecar instantiation
## Sidecar Instantiation
Consul does not start or manage sidecar proxies processes. Proxies running on a
Consul does not start or manage sidecar proxy processes. Proxies running on a
physical host or VM are designed to be started and run by process supervisor
physical host or VM are designed to be started and run by process supervisor
systems such as init, systemd, supervisord, etc. Or, if deployed within a
systems, such as init, systemd, supervisord, etc. If deployed within a
cluster scheduler (Kubernetes, Nomad) running as a sidecar container in the
cluster scheduler (Kubernetes, Nomad), proxies should run as a sidecar container in the
same namespace.
same namespace.
The proxy will use the [`CONSUL_HTTP_TOKEN`](/commands#consul_http_token) and
Proxies use the [`CONSUL_HTTP_TOKEN`](/commands#consul_http_token) and
[`CONSUL_HTTP_ADDR`](/commands#consul_http_addr) environment variables to
[`CONSUL_HTTP_ADDR`](/commands#consul_http_addr) environment variables to
contact Consul to fetch certificates, provided the `CONSUL_HTTP_TOKEN`
contact Consul and fetch certificates. This occurs if the `CONSUL_HTTP_TOKEN`
environment variable contains a Consul ACL that has the necessary permissions
environment variable contains a Consul ACL that has the necessary permissions
to read configuration for that service. If you use our Go [`api` package] then
to read configuration for that service. If you use the Go [`api` package], then
those environment variables will be read and the client configured for you
the environment variables will be read and the client configured for you
automatically.
automatically.
The ID of the proxy service comes from the user. See [`consul connect envoy`](/commands/connect/envoy) as an example. You may start it with the
The proxy service ID comes from the user. See [`consul connect envoy`](/commands/connect/envoy#examples) for an example. You can use the `-proxy-id` flag to specify the ID of the proxy service you have already registered with the local agent.
`-proxy-id` flag and pass the ID of the proxy service you registered elsewhere.
A nicer UX is available for end-users using the `-sidecar-for=<service>`
Alternatively, you can start the service using the `-sidecar-for=<service>` option. This option queries Consul for a proxy that is registered as a sidecar for the specified `<service>`. If exactly one service associated with the proxy is returned, the ID will be used to start the proxy. Your controller only needs to accept `-proxy-id` as an argument; the Consul CLI will resolve the
argument, which causes the command to query Consul for a proxy that is
ID for the name specified in `-sidecar-for` flag.
registered as a sidecar for the specified `<service>`. If there is exactly one
such proxy, that ID will be used to start the proxy. Your controller only needs
to accept `-proxy-id` as an argument; the Consul CLI will handle resolving the
trujillo-adam
3 years ago