consul/website/source/docs/guides/kubernetes-observability.html.md

layout	page_title	sidebar_current	description
docs	Layer 7 Observability with Kubernetes and Consul Connect	docs-guides-kubernetes-observability	Collect and visualize layer 7 metrics from services in your Kubernetes cluster using Consul Connect, Prometheus, and Grafana.

A service mesh is made up of proxies deployed locally alongside each service instance, which control network traffic between their local instance and other services on the network. These proxies "see" all the traffic that runs through them, and in addition to securing that traffic, they can also collect data about it. Starting with version 1.5, Consul Connect is able to configure Envoy proxies to collect layer 7 metrics including HTTP status codes and request latency, along with many others, and export those to monitoring tools like Prometheus.

In this guide, you will deploy a basic metrics collection and visualization pipeline on a Kubernetes cluster using the official Helm charts for Consul, Prometheus, and Grafana. This pipeline will collect and display metrics from a demo application.

-> Tip: While this guide shows you how to deploy a metrics pipeline on Kubernetes, all the technologies the guide uses are platform agnostic; Kubernetes is not necessary to collect and visualize layer 7 metrics with Consul Connect.

Learning Objectives:

• Configure Consul Connect with metrics using Helm
• Install Prometheus and Grafana using Helm
• Install and start the demo application
• Collect metrics

Prerequisites

If you already have a Kubernetes cluster with Helm and kubectl up and running, you can start on the demo right away. If not, set up a Kubernetes cluster using your favorite method that supports persistent volume claims, or install and start Minikube. If you do use Minikube, you may want to start it with a little bit of extra memory.

$ minikube start --memory 4096

You will also need to install kubectl, and both install and initialize Helm by following their official instructions.

If you already had Helm installed, check that you have up to date versions of the Grafana, Prometheus, and Consul charts. You can update all your charts to the latest versions by running helm repo update.

Clone the GitHub repository that contains the configuration files you'll use while following this guide, and change directories into it. We'll refer to this directory as your working directory, and you'll run the rest of the commands in this guide from inside it.

$ git clone https://github.com/hashicorp/consul-k8s-l7-obs-guide.git

$ cd consul-k8s-l7-obs-guide

Deploy Consul Connect Using Helm

Once you have set up the prerequisites, you're ready to install Consul. Start by cloning the official Consul Helm chart into your working directory.

$ git clone https://github.com/hashicorp/consul-helm.git

Open the file in your working directory called consul-values.yaml. This file will configure the Consul Helm chart to:

• specify a name for your Consul datacenter
• enable the Consul web UI
• enable secure communication between pods with Connect
• configure the Consul settings necessary for layer 7 metrics collection
• specify that this Consul cluster should run one server
• enable metrics collection on servers and agents so that you can monitor the Consul cluster itself

You can override many of the values in Consul's values file using annotations on specific services. For example, later in the guide you will override the centralized configuration of defaultProtocol.

# name your datacenter
global:
  datacenter: dc1

server:
  # use 1 server
  replicas: 1
  bootstrapExpect: 1
  disruptionBudget:
    enabled: true
    maxUnavailable: 0

client:
  enabled: true
  # enable grpc on your client to support consul connect
  grpc: true

ui:
  enabled: true

connectInject:
  enabled: true
  # inject an envoy sidecar into every new pod,
  # except for those with annotations that prevent injection
  default: true
  # these settings enable L7 metrics collection and are new in 1.5
  centralConfig:
    enabled: true
    # set the default protocol (can be overwritten with annotations)
    defaultProtocol: "http"
    # tell envoy where  to send metrics
    proxyDefaults: |
      {
      "envoy_dogstatsd_url": "udp://127.0.0.1:9125"
      }      

!> Warning: By default, the chart will install an insecure configuration of Consul. This provides a less complicated out-of-box experience for new users but is not appropriate for a production setup. Make sure that your Kubernetes cluster is properly secured to prevent unwanted access to Consul, or that you understand and enable the recommended Consul security features. Currently, some of these features are not supported in the Helm chart and require additional manual configuration.

Now install Consul in your Kubernetes cluster and give Kubernetes a name for your Consul installation. The output will be a list of all the Kubernetes resources created (abbreviated in the code snippet).

$ helm install -f consul-values.yaml --name l7-guide ./consul-helm
NAME:   consul
LAST DEPLOYED: Wed May  1 16:02:40 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:

Check that Consul is running in your Kubernetes cluster via the Kubernetes dashboard or CLI. If you are using Minikube, the below command will run in your current terminal window and automatically open the dashboard in your browser.

$ minikube dashboard

Open a new terminal tab to let the dashboard run in the current one, and change directories back into consul-k8s-l7-obs-guide. Next, forward the port for the Consul UI to localhost:8500 and navigate to it in your browser. Once you run the below command it will continue to run in your current terminal window for as long as it is forwarding the port.

$ kubectl port-forward l7-guide-consul-server-0 8500:8500
Forwarding from 127.0.0.1:8500 -> 8500
Forwarding from [::1]:8500 -> 8500
Handling connection for 8500

Let the consul dashboard port forwarding run and open a new terminal tab to the consul-k8s-l7-obs-guide directory.

Deploy the Metrics Pipeline

In this guide, you will use Prometheus and Grafana to collect and visualize metrics. Consul Connect can integrate with a variety of other metrics tooling as well.

Deploy Prometheus with Helm

You'll follow a similar process as you did with Consul to install Prometheus via Helm. First, open the file named prometheus-values.yaml that configures the Prometheus Helm chart.

The file specifies how often Prometheus should scrape for metrics, and which endpoints it should scrape from. By default, Prometheus scrapes all the endpoints that Kubernetes knows about, even if those endpoints don't expose Prometheus metrics. To prevent Prometheus from scraping these endpoints unnecessarily, the values file includes some relabel configurations.

Install the official Prometheus Helm chart using the values in prometheus-values.yaml.

$ helm install -f prometheus-values.yaml --name prometheus stable/prometheus
NAME:   prometheus
LAST DEPLOYED: Wed May  1 16:09:48 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:

The output above has been abbreviated; you will see all the Kubernetes resources that the Helm chart created. Once Prometheus has come up, you should be able to see your new services on the Minikube dashboard and in the Consul UI. This might take a short while.

Deploy Grafana with Helm

Installing Grafana will follow a similar process. Open and look through the file named grafana-values.yaml. It configures Grafana to use Prometheus as its datasource.

Use the official Helm chart to install Grafana with your values file.

$ helm install -f grafana-values.yaml --name grafana stable/grafana
NAME:   grafana
LAST DEPLOYED: Wed May  1 16:57:11 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:

...

NOTES:
1. Get your 'admin' user password by running:

   kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:

   grafana.default.svc.cluster.local

   Get the Grafana URL to visit by running these commands in the same shell:

     export POD_NAME=$(kubectl get pods --namespace default -l "app=grafana,release=grafana" -o jsonpath="{.items[0].metadata.name}")
     kubectl --namespace default port-forward $POD_NAME 3000

3. Login with the password from step 1 and the username: admin

Again, the above output has been abbreviated. At the bottom of your terminal output are shell-specific instructions to access your Grafana UI and log in, displayed as a numbered list. Accessing Grafana involves:

1. Getting the secret that serves as your Grafana password
2. Forwarding the Grafana UI to localhost:3000, which will not succeed until Grafana is running
3. Visiting the UI and logging in

Once you have logged into the Grafana UI, hover over the dashboards icon (four squares in the left hand menu) and then click the "manage" option. This will take you to a page that gives you some choices about how to upload Grafana dashboards. Click the black "Import" button on the right hand side of the screen.

Open the file called overview-dashboard.json and copy the contents into the json window of the Grafana UI. Click through the rest of the options, and you will end up with a blank dashboard, waiting for data to display.

Deploy a Demo Application on Kubernetes

Now that your monitoring pipeline is set up, deploy a demo application that will generate data. We will be using Emojify, an application that recognizes faces in an image and pastes emojis over them. The application consists of a few different services and automatically generates traffic and HTTP error codes.

All the files defining Emojify are in the app directory. Open app/cache.yml and take a look at the file. Most of services that make up Emojify communicate over HTTP, but the cache service uses gRPC. In the annotations section of the file you'll see where consul.hashicorp.com/connect-service-protocol specifies gRPC, overriding the defaultProtocol of HTTP that we centrally configured in Consul's value file.

At the bottom of each file defining part of the Emojify app, notice the block defining a prometheus-statsd pod. These pods translate the metrics that Envoy exposes to a format that Prometheus can scrape. They won't be necessary anymore once Consul Connect becomes compatible with Envoy 1.10. Apply the configuration to deploy Emojify into your cluster.

$ kubectl apply -f app

Emojify will take a little while to deploy. Once it's running you can check that it's healthy by taking a look at your Kubernetes dashboard or Consul UI. Next, visit the Emojify UI. This will be located at the IP address of the host where the ingress server is running, at port 30000. If you're using Minikube you can find the UI with the following command.

$ minikube service emojify-ingress --url
http://192.168.99.106:30000

Test the application by emojifying a picture. You can do this by pasting the following URL into the URL bar and clicking the submit button. (We provide a demo URL because Emojify can be picky about processing some image URLs if they don't link directly to the actual picture.)

https://emojify.today/pictures/1.jpg

Now that you know the application is working, start generating automatic load so that you will have some interesting metrics to look at.

$ kubectl apply -f traffic.yaml

Collect Application Metrics

Envoy exposes a huge number of metrics, but you will probably only want to monitor or alert on a subset of them. Which metrics are important to monitor will depend on your application. For this getting-started guide we have preconfigured an Emojify-specific Grafana dashboard with a couple of basic metrics, but you should systematically consider what others you will need to collect as you move from testing into production.

Review Dashboard Metrics

Now that you have metrics flowing through your pipeline, navigate back to your Grafana dashboard at localhost:3000. The top row of the dashboard displays general metrics about the Emojify application as a whole, including request and error rates. Although changes in these metrics can reflect application health issues once you understand their baseline levels, they don't provide enough information to diagnose specific issues.

The following rows of the dashboard report on some of the specific services that make up the emojify application: the website, API, and cache services. The website and API services show request count and response time, while the cache reports on request count and methods.

Clean up

If you've been using Minikube, you can tear down your environment by running minikube delete.

If you want to get rid of the configurations files and Consul Helm chart, recursively remove the consul-k8s-l7-obs-guide directory. `

$ cd ..

$ rm -rf consul-k8s-l7-obs-guide

Summary

In this guide, you set up layer 7 metrics collection and visualization in a Minikube cluster using Consul Connect, Prometheus, and Grafana, all deployed via Helm charts. Because all of these programs can run outside of Kubernetes, you can set this pipeline up in any environment or collect metrics from workloads running on mixed infrastructure.

To learn more about the configuration options in Consul that enable layer 7 metrics collection with or without Kubernetes, refer to our documentation. For more information on centrally configuring Consul, take a look at the centralized configuration documentation.