Twitch - Driving Towards a Modern Infrastructure
In this talk from HashiConf 2015, Tarrant Rollins discusses how Twitch uses HashiCorp Consul to overcome legacy infrastructure and solve complex problems.
WatchService discovery made easy
Service registry, integrated health checks, and DNS and HTTP interfaces enable any service to discover and be discovered by other services
The Challenge
Service load balancers aren't efficient in a dynamic world.
<%= inline_svg 'consul-connect/svgs/discovery-challenge.svg' %>
Load balancers are often used to front a service tier and provide a static IP. These load balancers add cost, increase latency, introduce single points of failure, and must be updated as services scale up/down.
The Solution
Service discovery for dynamic infrastructure.
<%= inline_svg 'consul-connect/svgs/discovery-solution.svg' %>
Instead of load balancers, connectivity in dynamic infrastructure is best solved with service discovery. Service discovery uses a registry to keep a real-time list of services, their location, and their health. Services query the registry to discover the location of upstream services and then connect directly. This allows services to scale up/down and gracefully handle failure without a load balancer intermediary.
Features
Service Registry
Consul provides a registry of all the running nodes and services, along with their current health status. This allows operators to understand the environment, and applications and automation tools to interact with dynamic infrastructure using an HTTP API.
DNS Query Interface
Consul enables service discovery using a built-in DNS server. This allows existing applications to easily integrate, as almost all applications support using DNS to resolve IP addresses. Using DNS instead of a static IP address allows services to scale up/down and route around failures easily.
$ dig web-frontend.service.consul. ANY
; <<>> DiG 9.8.3-P1 <<>> web-frontend.service.consul. ANY
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29981
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;web-frontend.service.consul. IN ANY
;; ANSWER SECTION:
web-frontend.service.consul. 0 IN A 10.0.3.83
web-frontend.service.consul. 0 IN A 10.0.1.109
HTTP API with Edge Triggers
Consul provides an HTTP API to query the service registry for nodes, services, and health check information. The API also supports blocking queries, or long-polling for any changes. This allows automation tools to react to services being registered or health status changes to change configurations or traffic routing in real time.
$ curl http://localhost:8500/v1/health/service/web?index=11&wait=30s
{
...
"Node": "10-0-1-109",
"CheckID": "service:web",
"Name": "Service 'web' check",
"Status": "critical",
"ServiceID": "web",
"ServiceName": "web",
"CreateIndex": 10,
"ModifyIndex": 20
...
}
Multi Datacenter
Consul supports multiple datacenters out of the box with no complicated configuration. Look up services in other datacenters or keep the request local. Advanced features like Prepared Queries enable automatic failover to other datacenters.
$ curl http://localhost:8500/v1/catalog/datacenters
["dc1", "dc2"]$ curl http://localhost:8500/v1/catalog/nodes?dc=dc2
[
{
"ID": "7081dcdf-fdc0-0432-f2e8-a357d36084e1",
"Node": "10-0-1-109",
"Address": "10.0.1.109",
"Datacenter": "dc2",
"TaggedAddresses": {
"lan": "10.0.1.109",
"wan": "10.0.1.109"
},
"CreateIndex": 112,
"ModifyIndex": 125
},
...
Health Checks
Pairing service discovery with health checking prevents routing requests to unhealthy hosts and enables services to easily provide circuit breakers.
Case Studies
Video
Video
Jet.com - Nomad Auto-Proxy with Consul-Template and NGINX
Justen Walker explains how Jet.com uses HashiCorp Consul and Nomad to allow hundreds of developers to have self-service access, despite relying on NGINX static configs—and with a remarkably small DevOps team.
Watch