mirror of https://github.com/hashicorp/consul
370 lines
14 KiB
Markdown
370 lines
14 KiB
Markdown
---
|
|
layout: docs
|
|
page_title: Layer 7 Observability with Kubernetes and Consul Connect
|
|
description: |-
|
|
Collect and visualize layer 7 metrics from services in your Kubernetes cluster
|
|
using Consul Connect, Prometheus, and Grafana.
|
|
---
|
|
|
|
A service mesh is made up of proxies deployed locally alongside each service
|
|
instance, which control network traffic between their local instance and other
|
|
services on the network. These proxies "see" all the traffic that runs through
|
|
them, and in addition to securing that traffic, they can also collect data about
|
|
it. Starting with version 1.5, Consul Connect is able to configure Envoy proxies
|
|
to collect layer 7 metrics including HTTP status codes and request latency, along
|
|
with many others, and export those to monitoring tools like Prometheus.
|
|
|
|
In this guide, you will deploy a basic metrics collection and visualization
|
|
pipeline on a Kubernetes cluster using the official Helm charts for Consul,
|
|
Prometheus, and Grafana. This pipeline will collect and display metrics from a
|
|
demo application.
|
|
|
|
-> **Tip:** While this guide shows you how to deploy a metrics pipeline on
|
|
Kubernetes, all the technologies the guide uses are platform agnostic;
|
|
Kubernetes is not necessary to collect and visualize layer 7 metrics with Consul
|
|
Connect.
|
|
|
|
Learning Objectives:
|
|
|
|
- Configure Consul Connect with metrics using Helm
|
|
- Install Prometheus and Grafana using Helm
|
|
- Install and start the demo application
|
|
- Collect metrics
|
|
|
|
## Prerequisites
|
|
|
|
If you already have a Kubernetes cluster with Helm and kubectl up and running,
|
|
you can start on the demo right away. If not, set up a Kubernetes cluster using
|
|
your favorite method that supports persistent volume claims, or install and
|
|
start [Minikube](https://kubernetes.io/docs/tasks/tools/install-minikube/). If
|
|
you do use Minikube, you may want to start it with a little bit of extra memory.
|
|
|
|
```shell-session
|
|
$ minikube start --memory 4096
|
|
```
|
|
|
|
You will also need to install
|
|
[kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl),
|
|
and both install and initialize
|
|
[Helm](https://helm.sh/docs/using_helm/#installing-helm) by following their
|
|
official instructions.
|
|
|
|
If you already had Helm installed, check that you have up
|
|
to date versions of the Grafana, Prometheus, and Consul charts. You can update
|
|
all your charts to the latest versions by running `helm repo update`.
|
|
|
|
Clone the GitHub repository that contains the configuration files you'll use
|
|
while following this guide, and change directories into it. We'll refer to this
|
|
directory as your working directory, and you'll run the rest of the commands in
|
|
this guide from inside it.
|
|
|
|
```shell-session
|
|
$ git clone https://github.com/hashicorp/consul-k8s-l7-obs-guide.git
|
|
|
|
$ cd consul-k8s-l7-obs-guide
|
|
```
|
|
|
|
## Deploy Consul Connect Using Helm
|
|
|
|
Once you have set up the prerequisites, you're ready to install Consul. Start by
|
|
cloning the official Consul Helm chart into your working directory.
|
|
|
|
```shell-session
|
|
$ git clone https://github.com/hashicorp/consul-helm.git
|
|
```
|
|
|
|
Open the file in your working directory called `consul-values.yaml`. This file
|
|
will configure the Consul Helm chart to:
|
|
|
|
- specify a name for your Consul datacenter
|
|
- enable the Consul web UI
|
|
- enable secure communication between pods with Connect
|
|
- configure the Consul settings necessary for layer 7 metrics collection
|
|
- specify that this Consul cluster should run one server
|
|
- enable metrics collection on servers and agents so that you can monitor the
|
|
Consul cluster itself
|
|
|
|
You can override many of the values in Consul's values file using annotations on
|
|
specific services. For example, later in the guide you will override the
|
|
centralized configuration of `defaultProtocol`.
|
|
|
|
```yaml
|
|
# name your datacenter
|
|
global:
|
|
datacenter: dc1
|
|
|
|
server:
|
|
# use 1 server
|
|
replicas: 1
|
|
bootstrapExpect: 1
|
|
disruptionBudget:
|
|
enabled: true
|
|
maxUnavailable: 0
|
|
|
|
client:
|
|
enabled: true
|
|
# enable grpc on your client to support consul connect
|
|
grpc: true
|
|
|
|
ui:
|
|
enabled: true
|
|
|
|
connectInject:
|
|
enabled: true
|
|
# inject an envoy sidecar into every new pod,
|
|
# except for those with annotations that prevent injection
|
|
default: true
|
|
# these settings enable L7 metrics collection and are new in 1.5
|
|
centralConfig:
|
|
enabled: true
|
|
# set the default protocol (can be overwritten with annotations)
|
|
defaultProtocol: 'http'
|
|
# tell envoy where to send metrics
|
|
proxyDefaults: |
|
|
{
|
|
"envoy_dogstatsd_url": "udp://127.0.0.1:9125"
|
|
}
|
|
```
|
|
|
|
!> **Warning:** By default, the chart will install an insecure configuration of
|
|
Consul. This provides a less complicated out-of-box experience for new users but
|
|
is not appropriate for a production setup. Make sure that your Kubernetes
|
|
cluster is properly secured to prevent unwanted access to Consul, or that you
|
|
understand and enable the
|
|
[recommended Consul security features](/docs/internals/security).
|
|
Currently, some of these features are not supported in the Helm chart and
|
|
require additional manual configuration.
|
|
|
|
Now install Consul in your Kubernetes cluster and give Kubernetes a name for
|
|
your Consul installation. The output will be a list of all the Kubernetes
|
|
resources created (abbreviated in the code snippet).
|
|
|
|
```shell-session
|
|
$ helm install -f consul-values.yaml --name l7-guide ./consul-helm
|
|
NAME: consul
|
|
LAST DEPLOYED: Wed May 1 16:02:40 2019
|
|
NAMESPACE: default
|
|
STATUS: DEPLOYED
|
|
|
|
RESOURCES:
|
|
```
|
|
|
|
Check that Consul is running in your Kubernetes cluster via the Kubernetes
|
|
dashboard or CLI. If you are using Minikube, the below command will run in your
|
|
current terminal window and automatically open the dashboard in your browser.
|
|
|
|
```shell-session
|
|
$ minikube dashboard
|
|
```
|
|
|
|
Open a new terminal tab to let the dashboard run in the current one, and change
|
|
directories back into `consul-k8s-l7-obs-guide`. Next, forward the port for the
|
|
Consul UI to localhost:8500 and navigate to it in your browser. Once you run the
|
|
below command it will continue to run in your current terminal window for as
|
|
long as it is forwarding the port.
|
|
|
|
```shell-session
|
|
$ kubectl port-forward l7-guide-consul-server-0 8500:8500
|
|
Forwarding from 127.0.0.1:8500 -> 8500
|
|
Forwarding from [::1]:8500 -> 8500
|
|
Handling connection for 8500
|
|
```
|
|
|
|
Let the consul dashboard port forwarding run and open a new terminal tab to the
|
|
`consul-k8s-l7-obs-guide` directory.
|
|
|
|
## Deploy the Metrics Pipeline
|
|
|
|
In this guide, you will use Prometheus and Grafana to collect and visualize
|
|
metrics. Consul Connect can integrate with a variety of other metrics tooling as
|
|
well.
|
|
|
|
### Deploy Prometheus with Helm
|
|
|
|
You'll follow a similar process as you did with Consul to install Prometheus via
|
|
Helm. First, open the file named `prometheus-values.yaml` that configures the
|
|
Prometheus Helm chart.
|
|
|
|
The file specifies how often Prometheus should scrape for metrics, and which
|
|
endpoints it should scrape from. By default, Prometheus scrapes all the
|
|
endpoints that Kubernetes knows about, even if those endpoints don't expose
|
|
Prometheus metrics. To prevent Prometheus from scraping these endpoints
|
|
unnecessarily, the values file includes some relabel configurations.
|
|
|
|
Install the official Prometheus Helm chart using the values in
|
|
`prometheus-values.yaml`.
|
|
|
|
```shell-session
|
|
$ helm install -f prometheus-values.yaml --name prometheus stable/prometheus
|
|
NAME: prometheus
|
|
LAST DEPLOYED: Wed May 1 16:09:48 2019
|
|
NAMESPACE: default
|
|
STATUS: DEPLOYED
|
|
|
|
RESOURCES:
|
|
```
|
|
|
|
The output above has been abbreviated; you will see all the Kubernetes resources
|
|
that the Helm chart created. Once Prometheus has come up, you should be able to
|
|
see your new services on the Minikube dashboard and in the Consul UI. This
|
|
might take a short while.
|
|
|
|
### Deploy Grafana with Helm
|
|
|
|
Installing Grafana will follow a similar process. Open and look through the file
|
|
named `grafana-values.yaml`. It configures Grafana to use Prometheus as its
|
|
datasource.
|
|
|
|
Use the official Helm chart to install Grafana with your values file.
|
|
|
|
```shell-session
|
|
$ helm install -f grafana-values.yaml --name grafana stable/grafana
|
|
NAME: grafana
|
|
LAST DEPLOYED: Wed May 1 16:57:11 2019
|
|
NAMESPACE: default
|
|
STATUS: DEPLOYED
|
|
|
|
RESOURCES:
|
|
|
|
...
|
|
|
|
NOTES:
|
|
1. Get your 'admin' user password by running:
|
|
|
|
kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
|
|
|
|
2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:
|
|
|
|
grafana.default.svc.cluster.local
|
|
|
|
Get the Grafana URL to visit by running these commands in the same shell:
|
|
|
|
export POD_NAME=$(kubectl get pods --namespace default -l "app=grafana,release=grafana" -o jsonpath="{.items[0].metadata.name}")
|
|
kubectl --namespace default port-forward $POD_NAME 3000
|
|
|
|
3. Login with the password from step 1 and the username: admin
|
|
```
|
|
|
|
Again, the above output has been abbreviated. At the bottom of your terminal
|
|
output are shell-specific instructions to access your Grafana UI and log in,
|
|
displayed as a numbered list. Accessing Grafana involves:
|
|
|
|
1. Getting the secret that serves as your Grafana password
|
|
1. Forwarding the Grafana UI to localhost:3000, which will not succeed until
|
|
Grafana is running
|
|
1. Visiting the UI and logging in
|
|
|
|
Once you have logged into the Grafana UI, hover over the dashboards icon (four
|
|
squares in the left hand menu) and then click the "manage" option. This will
|
|
take you to a page that gives you some choices about how to upload Grafana
|
|
dashboards. Click the black "Import" button on the right hand side of the
|
|
screen.
|
|
|
|
|
|
|
|
Open the file called `overview-dashboard.json` and copy the contents into the
|
|
json window of the Grafana UI. Click through the rest of the options, and you
|
|
will end up with a blank dashboard, waiting for data to display.
|
|
|
|
### Deploy a Demo Application on Kubernetes
|
|
|
|
Now that your monitoring pipeline is set up, deploy a demo application that will
|
|
generate data. We will be using Emojify, an application that recognizes faces in
|
|
an image and pastes emojis over them. The application consists of a few
|
|
different services and automatically generates traffic and HTTP error codes.
|
|
|
|
All the files defining Emojify are in the `app` directory. Open `app/cache.yml`
|
|
and take a look at the file. Most of services that make up Emojify communicate
|
|
over HTTP, but the cache service uses gRPC. In the annotations section of the
|
|
file you'll see where `consul.hashicorp.com/connect-service-protocol` specifies
|
|
gRPC, overriding the `defaultProtocol` of HTTP that we centrally configured in
|
|
Consul's value file.
|
|
|
|
At the bottom of each file defining part of the Emojify app, notice the block
|
|
defining a `prometheus-statsd` pod. These pods translate the metrics that Envoy
|
|
exposes to a format that Prometheus can scrape. They won't be necessary anymore
|
|
once Consul Connect becomes compatible with Envoy 1.10. Apply the configuration
|
|
to deploy Emojify into your cluster.
|
|
|
|
```shell-session
|
|
$ kubectl apply -f app
|
|
```
|
|
|
|
Emojify will take a little while to deploy. Once it's running you can check that
|
|
it's healthy by taking a look at your Kubernetes dashboard or Consul UI. Next,
|
|
visit the Emojify UI. This will be located at the IP address of the host where
|
|
the ingress server is running, at port 30000. If you're using Minikube you can
|
|
find the UI with the following command.
|
|
|
|
```shell-session
|
|
$ minikube service emojify-ingress --url
|
|
http://192.168.99.106:30000
|
|
```
|
|
|
|
Test the application by emojifying a picture. You can do this by pasting the
|
|
following URL into the URL bar and clicking the submit button. (We provide a
|
|
demo URL because Emojify can be picky about processing some image URLs if they
|
|
don't link directly to the actual picture.)
|
|
|
|
`https://emojify.today/pictures/1.jpg`
|
|
|
|
Now that you know the application is working, start generating automatic load so
|
|
that you will have some interesting metrics to look at.
|
|
|
|
```shell-session
|
|
$ kubectl apply -f traffic.yaml
|
|
```
|
|
|
|
## Collect Application Metrics
|
|
|
|
Envoy exposes a huge number of
|
|
[metrics](https://www.envoyproxy.io/docs/envoy/v1.10.0/operations/stats_overview),
|
|
but you will probably only want to monitor or alert on a subset of them. Which
|
|
metrics are important to monitor will depend on your application. For this
|
|
getting-started guide we have preconfigured an Emojify-specific Grafana
|
|
dashboard with a couple of basic metrics, but you should systematically consider
|
|
what others you will need to collect as you move from testing into production.
|
|
|
|
### Review Dashboard Metrics
|
|
|
|
Now that you have metrics flowing through your pipeline, navigate back to your
|
|
Grafana dashboard at `localhost:3000`. The top row of the dashboard displays
|
|
general metrics about the Emojify application as a whole, including request and
|
|
error rates. Although changes in these metrics can reflect application health
|
|
issues once you understand their baseline levels, they don't provide enough
|
|
information to diagnose specific issues.
|
|
|
|
The following rows of the dashboard report on some of the specific services that
|
|
make up the emojify application: the website, API, and cache services. The
|
|
website and API services show request count and response time, while the cache
|
|
reports on request count and methods.
|
|
|
|
## Clean up
|
|
|
|
If you've been using Minikube, you can tear down your environment by running
|
|
`minikube delete`.
|
|
|
|
If you want to get rid of the configurations files and Consul Helm chart,
|
|
recursively remove the `consul-k8s-l7-obs-guide` directory.
|
|
`
|
|
|
|
```shell-session
|
|
$ cd ..
|
|
|
|
$ rm -rf consul-k8s-l7-obs-guide
|
|
```
|
|
|
|
## Summary
|
|
|
|
In this guide, you set up layer 7 metrics collection and visualization in a
|
|
Minikube cluster using Consul Connect, Prometheus, and Grafana, all deployed via
|
|
Helm charts. Because all of these programs can run outside of Kubernetes, you
|
|
can set this pipeline up in any environment or collect metrics from workloads
|
|
running on mixed infrastructure.
|
|
|
|
To learn more about the configuration options in Consul that enable layer 7
|
|
metrics collection with or without Kubernetes, refer to [our
|
|
documentation](/docs/connect/proxies/envoy). For more information on
|
|
centrally configuring Consul, take a look at the [centralized configuration
|
|
documentation](/docs/agent/config-entries).
|