From c5b41671d3bd2ead9076ecf0646c1efe1be477bc Mon Sep 17 00:00:00 2001 From: natemollica-dev Date: Thu, 15 Feb 2024 13:19:06 -0800 Subject: [PATCH] datadog: Verification command table dogstatsd --- .../k8s/connect/observability/datadog.mdx | 59 ++++++++++++++----- 1 file changed, 43 insertions(+), 16 deletions(-) diff --git a/website/content/docs/k8s/connect/observability/datadog.mdx b/website/content/docs/k8s/connect/observability/datadog.mdx index 30d507055a..f17d083ffa 100644 --- a/website/content/docs/k8s/connect/observability/datadog.mdx +++ b/website/content/docs/k8s/connect/observability/datadog.mdx @@ -32,6 +32,8 @@ than a Datadog agent attempting to reach Consul and scrape the `/v1/agent/metric This is accomplished by updating each server agent's configuration telemetry stanza. +### Helm Chart Configuration + @@ -124,6 +126,9 @@ This is accomplished by updating each server agent's configuration telemetry sta This integration method accomplishes metrics collection by leveraging either [Unix Domain Sockets](https://docs.datadoghq.com/developers/dogstatsd/unix_socket/?tab=kubernetes) (**UDS**) or [User Datagram Protocol](https://docs.datadoghq.com/developers/dogstatsd/?tab=kubernetes#agent) (**UDP**) transport. +Practitioners who manage their Kubernetes infrastructure and/or service-mesh should take into account the implications outlined in the table below. + +### UDS/UDP Advantages and Disadvantages @@ -137,8 +142,8 @@ This integration method accomplishes metrics collection by leveraging either [Un - No IP or DNS resolution requirement for Datadog Agent

Improved network performance

Packet error handling

Automatic container ID tagging - Requires hostPath Volume attachment + No IP or DNS resolution requirement for Datadog Agent

Improved network performance

Higher throughput capacity

Packet error handling

Automatic container ID tagging + Requires hostPath Volume attachment

Datadog Agent must run on every host you send metrics from Unix Domain Socket File @@ -155,8 +160,8 @@ This integration method accomplishes metrics collection by leveraging either [Un - Does not require hostPath Volume attachment

Similar IP:Port configuration as Virtual Machine hosts - (Hostport) Requires a networking provider that adheres to the CNI specification, such as Calico, Canal, or Flannel.

(Hostport) Requires port to be exposed on host using hostNetwork

(Hostport) Requires firewall access controls to permit access

(Hostport) Network Namespace sharing is required + Does not require hostPath Volume attachment

(KubeDNS) Does not require Hostport exposure if accessible from cluster

Similar IP:Port configuration as Virtual Machine hosts + No packet error handling

(Hostport) Requires a networking provider that adheres to the CNI specification, such as Calico, Canal, or Flannel.

(Hostport) Requires port to be exposed on host using hostNetwork

(Hostport) Requires firewall access controls to permit access

(Hostport) Network Namespace sharing is required Kubernetes Service IP:Port

or

Container host port @@ -166,9 +171,16 @@ This integration method accomplishes metrics collection by leveraging either [Un #### Verifying DogstatsD Metric Collection - | Datadog Agent Command | Pod | Container | - |-----------------------|---------------|-----------| - | `$ agent status` | datadog-agent | agent | +To confirm you're Datadog agent is receiving traffic, the `status` subcommand can be ran from the Datadog Agent expecting to receive DogstatsD traffic from Consul. + +There should be an increase in either UDP or UDS traffic packet counts from the resultant output after the configuration has been properly established. + + + | Transport | Command | Pod | Container | Example Output | + |:------------|----------------------------------------------------------------------------|---------------|-----------|------------------------------------------------------------------------------------------------| + | `UDP`/`UDS` | `agent status` | datadog-agent | agent | See below. | + | `UDP` | `netstat -nup \| grep ":8125.*ESTABLISHED"` | consul-server | consul | `udp 0 0 127.0.0.1:53874 127.0.0.1:8125 ESTABLISHED 23176/consul` | + | `UDS` | `nc -U -u -w1 /var/run/datadog/dsd.socket` _(update socket path as req'd)_ | consul-server | consul | `Bound on /tmp/nc-IjJkoG/recv.sock` | @@ -228,6 +240,27 @@ This integration method accomplishes metrics collection by leveraging either [Un ## Datadog Checks: Official Consul Integration +The Datadog Agent package includes official third-party integrations for built-in availability upon agent deployment. + +The Consul Integration Datadog checks provided some additional metric verification checks that leverage Consul's built-in feature-set, and help monitor Consul +during normal operation beyond that of Consul's available metrics. + +See the below table for an outline of the features added by the official integration. + +### Additional Integration Checks Performed + +| Consul Component | Description | API Endpoint(s) | +|------------------|-----------------------------------------------------|----------------------------------------------------------------------| +| Agent | Agent Metadata (i.e., version) | `/v1/agent/self` | +| Metrics | Prometheus formatted metrics | `/v1/agent/metrics` | +| Serf | Events and Membership Flaps | `/v1/health/service/consul` `/v1/agent/self` | +| Raft | Monitors Raft peer information and leader elections | `/v1/status/leader` `/v1/status/peers` | +| Catalog Services | Service Health Status and Node Count | `/v1/catalog/services` `/v1/health/state/any` | +| Catalog Nodes | Node Service Count and Health Status | `/v1/health/state/any` `/v1/health/service/` | +| Consul Latency | Consul LAN + WAN Coordinate Latency Calculations | `/v1/agent/self` `/v1/coordinate/nodes` `/v1/coordinate/datacenters` | + + +### Helm Chart Configuration @@ -242,15 +275,7 @@ This integration method accomplishes metrics collection by leveraging either [Un - | Consul Component | Description | API Endpoint(s) | - |------------------|-----------------------------------------------------|----------------------------------------------------------------------| - | Agent | Agent Metadata (i.e., version) | `/v1/agent/self` | - | Metrics | Prometheus formatted metrics | `/v1/agent/metrics` | - | Serf | Events and Membership Flaps | `/v1/health/service/consul` `/v1/agent/self` | - | Raft | Monitors Raft peer information and leader elections | `/v1/status/leader` `/v1/status/peers` | - | Catalog Services | Service Health Status and Node Count | `/v1/catalog/services` `/v1/health/state/any` | - | Catalog Nodes | Node Service Count and Health Status | `/v1/health/state/any` `/v1/health/service/` | - | Consul Latency | Consul LAN + WAN Coordinate Latency Calculations | `/v1/agent/self` `/v1/coordinate/nodes` `/v1/coordinate/datacenters` | + #### Metrics Data Collected @@ -263,6 +288,8 @@ Review the [Datadog Documentation](https://docs.datadoghq.com/integrations/consu ## Openmetrics Prometheus +### Helm Chart Configuration + ```yaml