consul/website/content/docs/guides/kubernetes-observability.mdx

---
layout: docs
page_title: Layer 7 Observability with Kubernetes and Consul Connect
description: |-
  Collect and visualize layer 7 metrics from services in your Kubernetes cluster
  using Consul  Connect, Prometheus, and Grafana.
---

A service mesh is made up of proxies deployed locally alongside each service
instance, which control network traffic between their local instance and other
services on the network. These proxies "see" all the traffic that runs through
them, and in addition to securing that traffic, they can also collect data about
it. Starting with version 1.5, Consul Connect is able to configure Envoy proxies
to collect layer 7 metrics including HTTP status codes and request latency, along
with many others, and export those to monitoring tools like Prometheus.

In this guide, you will deploy a basic metrics collection and visualization
pipeline on a Kubernetes cluster using the official Helm charts for Consul,
Prometheus, and Grafana. This pipeline will collect and display metrics from a
demo application.

-> **Tip:** While this guide shows you how to deploy a metrics pipeline on
Kubernetes, all the technologies the guide uses are platform agnostic;
Kubernetes is not necessary to collect and visualize layer 7 metrics with Consul
Connect.

Learning Objectives:

- Configure Consul Connect with metrics using Helm
- Install Prometheus and Grafana using Helm
- Install and start the demo application
- Collect metrics

## Prerequisites

If you already have a Kubernetes cluster with Helm and kubectl up and running,
you can start on the demo right away. If not, set up a Kubernetes cluster using
your favorite method that supports persistent volume claims, or install and
start [Minikube](https://kubernetes.io/docs/tasks/tools/install-minikube/). If
you do use Minikube, you may want to start it with a little bit of extra memory.

```shell-session
$ minikube start --memory 4096
```

You will also need to install
[kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl),
and both install and initialize
[Helm](https://helm.sh/docs/using_helm/#installing-helm) by following their
official instructions.

If you already had Helm installed, check that you have up
to date versions of the Grafana, Prometheus, and Consul charts. You can update
all your charts to the latest versions by running `helm repo update`.

Clone the GitHub repository that contains the configuration files you'll use
while following this guide, and change directories into it. We'll refer to this
directory as your working directory, and you'll run the rest of the commands in
this guide from inside it.

```shell-session
$ git clone https://github.com/hashicorp/consul-k8s-l7-obs-guide.git

$ cd consul-k8s-l7-obs-guide
```

## Deploy Consul Connect Using Helm

Once you have set up the prerequisites, you're ready to install Consul. Start by
cloning the official Consul Helm chart into your working directory.

```shell-session
$ git clone https://github.com/hashicorp/consul-helm.git
```

Open the file in your working directory called `consul-values.yaml`. This file
will configure the Consul Helm chart to:

- specify a name for your Consul datacenter
- enable the Consul web UI
- enable secure communication between pods with Connect
- configure the Consul settings necessary for layer 7 metrics collection
- specify that this Consul cluster should run one server
- enable metrics collection on servers and agents so that you can monitor the
  Consul cluster itself

You can override many of the values in Consul's values file using annotations on
specific services. For example, later in the guide you will override the
centralized configuration of `defaultProtocol`.

```yaml
# name your datacenter
global:
  datacenter: dc1

server:
  # use 1 server
  replicas: 1
  bootstrapExpect: 1
  disruptionBudget:
    enabled: true
    maxUnavailable: 0

client:
  enabled: true
  # enable grpc on your client to support consul connect
  grpc: true

ui:
  enabled: true

connectInject:
  enabled: true
  # inject an envoy sidecar into every new pod,
  # except for those with annotations that prevent injection
  default: true
  # these settings enable L7 metrics collection and are new in 1.5
  centralConfig:
    enabled: true
    # set the default protocol (can be overwritten with annotations)
    defaultProtocol: 'http'
    # tell envoy where  to send metrics
    proxyDefaults: |
      {
      "envoy_dogstatsd_url": "udp://127.0.0.1:9125"
      }
```

!> **Warning:** By default, the chart will install an insecure configuration of
Consul. This provides a less complicated out-of-box experience for new users but
is not appropriate for a production setup. Make sure that your Kubernetes
cluster is properly secured to prevent unwanted access to Consul, or that you
understand and enable the
[recommended Consul security features](/docs/internals/security).
Currently, some of these features are not supported in the Helm chart and
require additional manual configuration.

Now install Consul in your Kubernetes cluster and give Kubernetes a name for
your Consul installation. The output will be a list of all the Kubernetes
resources created (abbreviated in the code snippet).

```shell-session
$ helm install -f consul-values.yaml --name l7-guide ./consul-helm
NAME:   consul
LAST DEPLOYED: Wed May  1 16:02:40 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
```

Check that Consul is running in your Kubernetes cluster via the Kubernetes
dashboard or CLI. If you are using Minikube, the below command will run in your
current terminal window and automatically open the dashboard in your browser.

```shell-session
$ minikube dashboard
```

Open a new terminal tab to let the dashboard run in the current one, and change
directories back into `consul-k8s-l7-obs-guide`. Next, forward the port for the
Consul UI to localhost:8500 and navigate to it in your browser. Once you run the
below command it will continue to run in your current terminal window for as
long as it is forwarding the port.

```shell-session
$ kubectl port-forward l7-guide-consul-server-0 8500:8500
Forwarding from 127.0.0.1:8500 -> 8500
Forwarding from [::1]:8500 -> 8500
Handling connection for 8500
```

Let the consul dashboard port forwarding run and open a new terminal tab to the
`consul-k8s-l7-obs-guide` directory.

## Deploy the Metrics Pipeline

In this guide, you will use Prometheus and Grafana to collect and visualize
metrics. Consul Connect can integrate with a variety of other metrics tooling as
well.

### Deploy Prometheus with Helm

You'll follow a similar process as you did with Consul to install Prometheus via
Helm. First, open the file named `prometheus-values.yaml` that configures the
Prometheus Helm chart.

The file specifies how often Prometheus should scrape for metrics, and which
endpoints it should scrape from. By default, Prometheus scrapes all the
endpoints that Kubernetes knows about, even if those endpoints don't expose
Prometheus metrics. To prevent Prometheus from scraping these endpoints
unnecessarily, the values file includes some relabel configurations.

Install the official Prometheus Helm chart using the values in
`prometheus-values.yaml`.

```shell-session
$ helm install -f prometheus-values.yaml --name prometheus stable/prometheus
NAME:   prometheus
LAST DEPLOYED: Wed May  1 16:09:48 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
```

The output above has been abbreviated; you will see all the Kubernetes resources
that the Helm chart created. Once Prometheus has come up, you should be able to
see your new services on the Minikube dashboard and in the Consul UI. This
might take a short while.

### Deploy Grafana with Helm

Installing Grafana will follow a similar process. Open and look through the file
named `grafana-values.yaml`. It configures Grafana to use Prometheus as its
datasource.

Use the official Helm chart to install Grafana with your values file.

```shell-session
$ helm install -f grafana-values.yaml --name grafana stable/grafana
NAME:   grafana
LAST DEPLOYED: Wed May  1 16:57:11 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:

...

NOTES:
1. Get your 'admin' user password by running:

   kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:

   grafana.default.svc.cluster.local

   Get the Grafana URL to visit by running these commands in the same shell:

     export POD_NAME=$(kubectl get pods --namespace default -l "app=grafana,release=grafana" -o jsonpath="{.items[0].metadata.name}")
     kubectl --namespace default port-forward $POD_NAME 3000

3. Login with the password from step 1 and the username: admin
```

Again, the above output has been abbreviated. At the bottom of your terminal
output are shell-specific instructions to access your Grafana UI and log in,
displayed as a numbered list. Accessing Grafana involves:

1. Getting the secret that serves as your Grafana password
1. Forwarding the Grafana UI to localhost:3000, which will not succeed until
   Grafana is running
1. Visiting the UI and logging in

Once you have logged into the Grafana UI, hover over the dashboards icon (four
squares in the left hand menu) and then click the "manage" option. This will
take you to a page that gives you some choices about how to upload Grafana
dashboards. Click the black "Import" button on the right hand side of the
screen.

![Add a dashboard using the Grafana GUI](/img/consul-grafana-add-dash.png)

Open the file called `overview-dashboard.json` and copy the contents into the
json window of the Grafana UI. Click through the rest of the options, and you
will end up with a blank dashboard, waiting for data to display.

### Deploy a Demo Application on Kubernetes

Now that your monitoring pipeline is set up, deploy a demo application that will
generate data. We will be using Emojify, an application that recognizes faces in
an image and pastes emojis over them. The application consists of a few
different services and automatically generates traffic and HTTP error codes.

All the files defining Emojify are in the `app` directory. Open `app/cache.yml`
and take a look at the file. Most of services that make up Emojify communicate
over HTTP, but the cache service uses gRPC. In the annotations section of the
file you'll see where `consul.hashicorp.com/connect-service-protocol` specifies
gRPC, overriding the `defaultProtocol` of HTTP that we centrally configured in
Consul's value file.

At the bottom of each file defining part of the Emojify app, notice the block
defining a `prometheus-statsd` pod. These pods translate the metrics that Envoy
exposes to a format that Prometheus can scrape. They won't be necessary anymore
once Consul Connect becomes compatible with Envoy 1.10. Apply the configuration
to deploy Emojify into your cluster.

```shell-session
$ kubectl apply -f app
```

Emojify will take a little while to deploy. Once it's running you can check that
it's healthy by taking a look at your Kubernetes dashboard or Consul UI. Next,
visit the Emojify UI. This will be located at the IP address of the host where
the ingress server is running, at port 30000. If you're using Minikube you can
find the UI with the following command.

```shell-session
$ minikube service emojify-ingress --url
http://192.168.99.106:30000
```

Test the application by emojifying a picture. You can do this by pasting the
following URL into the URL bar and clicking the submit button. (We provide a
demo URL because Emojify can be picky about processing some image URLs if they
don't link directly to the actual picture.)

`https://emojify.today/pictures/1.jpg`

Now that you know the application is working, start generating automatic load so
that you will have some interesting metrics to look at.

```shell-session
$ kubectl apply -f traffic.yaml
```

## Collect Application Metrics

Envoy exposes a huge number of
[metrics](https://www.envoyproxy.io/docs/envoy/v1.10.0/operations/stats_overview),
but you will probably only want to monitor or alert on a subset of them. Which
metrics are important to monitor will depend on your application. For this
getting-started guide we have preconfigured an Emojify-specific Grafana
dashboard with a couple of basic metrics, but you should systematically consider
what others you will need to collect as you move from testing into production.

### Review Dashboard Metrics

Now that you have metrics flowing through your pipeline, navigate back to your
Grafana dashboard at `localhost:3000`. The top row of the dashboard displays
general metrics about the Emojify application as a whole, including request and
error rates. Although changes in these metrics can reflect application health
issues once you understand their baseline levels, they don't provide enough
information to diagnose specific issues.

The following rows of the dashboard report on some of the specific services that
make up the emojify application: the website, API, and cache services. The
website and API services show request count and response time, while the cache
reports on request count and methods.

## Clean up

If you've been using Minikube, you can tear down your environment by running
`minikube delete`.

If you want to get rid of the configurations files and Consul Helm chart,
recursively remove the `consul-k8s-l7-obs-guide` directory.
`

```shell-session
$ cd ..

$ rm -rf consul-k8s-l7-obs-guide
```

## Summary

In this guide, you set up layer 7 metrics collection and visualization in a
Minikube cluster using Consul Connect, Prometheus, and Grafana, all deployed via
Helm charts. Because all of these programs can run outside of Kubernetes, you
can set this pipeline up in any environment or collect metrics from workloads
running on mixed infrastructure.

To learn more about the configuration options in Consul that enable layer 7
metrics collection with or without Kubernetes, refer to [our
documentation](/docs/connect/proxies/envoy). For more information on
centrally configuring Consul, take a look at the [centralized configuration
documentation](/docs/agent/config-entries).