docs: Consul DNS views on Kubernetes (#21802)

* Backport of ci: update the security-scanner gha token into release/1.20.x (#21754)

backport of commit eb9dbc93f8

Co-authored-by: dduzgun-security <deniz.duzgun@hashicorp.com>

* Backport of Initialize 1.20 Release into release/1.20.x (#21753)

* backport of commit a33e903cdf

* backport of commit 37163dc1a8

* backport of commit 38f0907c7a

* backport of commit 6ab7ec254b

* backport of commit 7ac4178186

* backport of commit 5dfebb2cf3

* backport of commit 316d68cb84

---------

Co-authored-by: Sarah Alsmiller <sarah.alsmiller@hashicorp.com>
Co-authored-by: sarahalsmiller <100602640+sarahalsmiller@users.noreply.github.com>

* Backport of Stage rc release into release/1.20.x (#21772)

backport of commit d311f2b638

Co-authored-by: Sarah Alsmiller <sarah.alsmiller@hashicorp.com>

* Backport of Upgrade ubi image to 9.4 into release/1.20.x (#21773)

* backport of commit 888e302f6e

* backport of commit 17499dc4dc

* backport of commit d933d3727d

---------

Co-authored-by: Dhia Ayachi <dhia.ayachi@gmail.com>
Co-authored-by: sarahalsmiller <100602640+sarahalsmiller@users.noreply.github.com>

* Backport of security: update alpine base image to 3.20 into release/1.20.x (#21774)

* backport of commit 4421ce1677

* Upgrade ubi image to 9.4 (#21750)

---------

Co-authored-by: Michael Zalimeni <michael.zalimeni@hashicorp.com>
Co-authored-by: Sarah Alsmiller <sarah.alsmiller@hashicorp.com>
Co-authored-by: sarahalsmiller <100602640+sarahalsmiller@users.noreply.github.com>

* Backport of fix spacing of bash scripts into release/1.20.x (#21769)

* backport of commit 1e97297215

* backport of commit b7053f5361

* backport of commit a391f2fa3c

---------

Co-authored-by: jm96441n <john.maguire@hashicorp.com>

* Backport of [NET-11150] ci: fix conditional skip and add safeguard into release/1.20.x (#21783)

backport of commit c3db6c9001

Co-authored-by: Michael Zalimeni <michael.zalimeni@hashicorp.com>

* initial commit

* Initial pages

* Edits to other pages + nav & redirects

* minor fixes

* Backport of security: update alpine base image to 3.20 into release/1.20.x (#21774)

* backport of commit 4421ce1677

* Upgrade ubi image to 9.4 (#21750)

---------

Co-authored-by: Michael Zalimeni <michael.zalimeni@hashicorp.com>
Co-authored-by: Sarah Alsmiller <sarah.alsmiller@hashicorp.com>
Co-authored-by: sarahalsmiller <100602640+sarahalsmiller@users.noreply.github.com>

* CE-679

* align with main

* Content updates

* minor edit

* Apply suggestions from code review

Co-authored-by: Aimee Ukasick <aimee.ukasick@hashicorp.com>
Co-authored-by: Blake Covarrubias <blake@covarrubi.as>

* CoreDNS config update

* small edits

* typo fix

---------

Co-authored-by: hc-github-team-consul-core <github-team-consul-core@hashicorp.com>
Co-authored-by: dduzgun-security <deniz.duzgun@hashicorp.com>
Co-authored-by: Sarah Alsmiller <sarah.alsmiller@hashicorp.com>
Co-authored-by: sarahalsmiller <100602640+sarahalsmiller@users.noreply.github.com>
Co-authored-by: Dhia Ayachi <dhia.ayachi@gmail.com>
Co-authored-by: Michael Zalimeni <michael.zalimeni@hashicorp.com>
Co-authored-by: jm96441n <john.maguire@hashicorp.com>
Co-authored-by: Aimee Ukasick <aimee.ukasick@hashicorp.com>
Co-authored-by: Blake Covarrubias <blake@covarrubi.as>

consul-k8s

@@ -0,0 +1 @@
+Subproject commit 0d85bbc3131ce8be23c57e30b213ba6056623976

version/VERSION

@@ -1 +1 @@
-1.21.0-dev
+1.21.0-dev

website/content/docs/k8s/dns/enable.mdx

@@ -5,16 +5,15 @@ description: >-
   Use a k8s ConfigMap to configure KubeDNS or CoreDNS so that you can use Consul's `<service-name>.service.consul` syntax for queries and other DNS requests. In Kubernetes, this process uses either stub-domain or proxy configuration.
 ---
 
-# Resolve Consul DNS Requests in Kubernetes
+# Resolve Consul DNS requests in Kubernetes
 
-One of the primary query interfaces to Consul is the
+This topic describes how to configure Consul DNS in
-[DNS interface](/consul/docs/services/discovery/dns-overview). You can configure Consul DNS in
 Kubernetes using a
 [stub-domain configuration](https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#configure-stub-domain-and-upstream-dns-servers)
 if using KubeDNS or a [proxy configuration](https://coredns.io/plugins/forward/) if using CoreDNS.
 
 Once configured, DNS requests in the form `<consul-service-name>.service.consul` will
-resolve for services in Consul. This will work from all Kubernetes namespaces.
+resolve for services in Consul. This works from all Kubernetes namespaces.
 
 -> **Note:** If you want requests to just `<consul-service-name>` (without the `.service.consul`) to resolve, then you'll need
 to turn on [Consul to Kubernetes Service Sync](/consul/docs/k8s/service-sync#consul-to-kubernetes).

website/content/docs/k8s/dns/views/enable.mdx

@@ -0,0 +1,102 @@
+---
+layout: docs
+page_title: Enable Consul DNS proxy for Kubernetes
+description: ->
+    Learn how to schedule a Consul DNS proxy for a Kubernetes Pod so that your services can return Consul DNS results for service discovery.
+---
+
+# Enable Consul DNS proxy for Kubernetes
+
+This page describes the process to deploy a Consul DNS proxy in a Kubernetes Pod so that Services can resolve Consul DNS requests. For more information, refer to [Consul DNS views for Kubernetes](/consul/docs/k8s/dns/views).
+
+## Prerequisites
+
+You must meet the following minimum application versions to enable the Consul DNS proxy for Kubernetes:
+
+- Consul v1.20.0 or higher
+- Either Consul on Kubernetes or the Consul Helm chart, v1.6.0 or higher
+
+## Update Helm values
+
+To enable the Consul DNS proxy, add the required [Helm values](/consul/docs/k8s/helm) to your Consul on Kubernetes deployment.
+
+```yaml
+connectInject:
+  enabled: true
+dns:
+  enabled: true
+  proxy: true
+```
+
+### ACLs
+
+We recommend you create a dedicated [ACL token with DNS permissions](/consul/docs/security/acl/tokens/create/create-a-dns-token) for the Consul DNS proxy. The Consul DNS proxy requires these ACL permissions.
+
+```hcl
+node_prefix "" {
+  policy = "read"
+}
+
+service_prefix "" {
+  policy = "read"
+}
+```
+
+You can manage ACL tokens with Consul on Kubernetes, or you can configure the DNS proxy to access a token stored in Kubernetes secret. To use a Kubernetes secret, add the following configuration to your Helm chart.
+
+```yaml
+dns:
+  proxy:
+    aclToken:
+      secretName: <Consul-DNS-Token>
+      secretKey: <Token-Value>
+```
+
+## Retrieve Consul DNS proxy's address
+
+To look up the IP address for the Consul DNS proxy in the Kubernetes Pod, run the following command.
+
+```shell-session
+$ kubectl get services –-all-namespaces --selector="app=consul,component=dns-proxy" --output jsonpath='{.spec.clusterIP}'  
+10.96.148.46  
+```
+
+Use this address when you update the ConfigMap resource.
+
+## Update Kubernetes ConfigMap
+
+Create or update a [ConfigMap object in the Kubernetes cluster](https://kubernetes.io/docs/concepts/configuration/configmap/) so that Kubernetes forwards DNS requests with the `.consul` domain to the IP address of the Consul DNS proxy.
+
+The following example of a `coredns-custom` ConfigMap configures Kubernetes to forward Consul DNS requests in the cluster to the Consul DNS Proxy running on `10.96.148.46`. This resource modifies the CoreDNS without modifications to the original `Corefile`.
+
+```yaml
+kind: ConfigMap  
+metadata:  
+  name: coredns-custom  
+  namespace: kube-system  
+data:  
+  consul.server: |  
+    consul:53 {  
+      errors  
+      cache 30  
+      forward . 10.96.148.46
+      reload  
+    }  
+```
+
+After updating the DNS configuration, perform a rolling restart of the CoreDNS.
+
+```shell-session
+kubectl -n kube-system rollout restart deployment coredns
+```
+
+For more information about using a `coredns-custom` resource, refer to the [Rewrite DNS guide in the Azure documentation](https://learn.microsoft.com/en-us/azure/aks/coredns-custom#rewrite-dns). For general information about modifying a ConfigMap, refer to [the Kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#coredns).
+
+## Next steps
+
+After you enable the Consul DNS proxy, services in the Kubernetes cluster can resolve Consul DNS addresses.
+
+- To learn more about Consul DNS for service discovery, refer to [DNS usage overview](/consul/docs/services/discovery/dns-overview).
+- If your datacenter has ACLs enabled, create a [Consul ACL token](/consul/docs/security/acl/tokens) for the Consul DNS proxy and then restart the DNS proxy.
+- To enable service discovery across admin partitions, [export services between partitions](/consul/docs/connect/config-entries/exported-services).
+- To use Consul DNS for service discovery with other runtimes, across cloud regions, or between cloud providers, [establish a cluster peering connection](/consul/docs/k8s/connect/cluster-peering/usage/establish-peering).

website/content/docs/k8s/dns/views/index.mdx

@@ -0,0 +1,48 @@
+---
+layout: docs
+page_title: Consul DNS views for Kubernetes
+description: ->
+    Kubernetes clusters can use the Consul DNS proxy to return service discovery results from the Consul catalog. Learn about how to configure your k8s cluster so that applications can resolve Consul DNS addresses without gossip communication.
+---
+
+# Consul DNS views for Kubernetes
+
+This topic describes how to schedule a dedicated Consul DNS proxy in a Kubernetes Pod so that applications in Kubernetes can resolve Consul DNS addresses. You can use the Consul DNS proxy to enable service discovery across admin partitions in Kubernetes deployments without needing to deploy Consul client agents.
+
+## Introduction
+
+Kubernetes operators typically choose networking tools such as [kube-dns](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/) or [CoreDNS](https://kubernetes.io/docs/tasks/administer-cluster/coredns/) for their service discovery operations, and choose to bypass Consul DNS entirely. These DNS options are often sufficient for service networking operations within a single Kubernetes cluster.
+
+Consul on Kubernetes supports [configuring Kubernetes to resolve Consul DNS](/consul/docs/k8s/dns). However, two common challenges result when you rely on these configurations:
+
+- Kubernetes requires Consul to use gossip communication with agents or dataplanes in order to enable Consul DNS.
+- Consul requires that admin partitions be included in the DNS address. Otherwise, DNS queries assume the `default` partition by default.
+
+The `consul-dns` proxy does not require the presence of Consul client agents or Consul dataplanes, removing gossip communication as a requirement for Consul DNS on Kubernetes. The proxy is also designed for deployment in a Kubernetes cluster with [external servers enabled](/consul/docs/k8s/deployment-configurations/servers-outside-kubernetes). When a cluster runs in a non-default admin partition and uses the proxy to query external servers, Consul automatically recognizes the admin partition that originated the request and returns service discovery results scoped to that specific admin partition.
+
+To use Consul DNS for service discovery on Kubernetes, deploy a `dns-proxy` service in each Kubernetes Pod that needs to resolve Consul DNS. Kubernetes sends all DNS requests to the Kubernetes controller first. The controller forwards requests for the `.consul` domain to the `dns-proxy` service, which then queries the Consul catalog and returns service discovery results.
+
+## Workflows
+
+The process to enable Consul DNS views for service discovery in Kubernetes deployments consists of the following steps:
+
+1. In a cluster configured to use [external Consul servers](/consul/docs/k8s/deployment-configurations/servers-outside-kubernetes), update the Helm values for your Consul on Kubernetes deployment so that `dns.proxy.enabled=true`. When you apply the updated configuration, Kubernetes deploys the Consul DNS proxy.
+1. Look up the IP address for the Consul DNS proxy in the Kubernetes cluster.
+1. Update the ConfigMap resource in the Kubernetes cluster so that it forwards requests for the `.consul` domain to the IP address of the Consul DNS proxy.
+
+For more information about the underlying concepts described in this workflow, refer to [DNS forwarding overview](/consul/docs/services/discovery/dns-forwarding).
+
+## Benefits
+
+Consul on Kubernetes currently uses [Consul dataplanes](/consul/docs/connect/dataplane) by default. These lightweight processes provide Consul access to the sidecar proxies in the service mesh, but leave Kubernetes in charge of most other service discovery and service mesh operations.
+
+- **Use Kubernetes DNS and Consul DNS in a single deployment**. The Consul DNS proxy enables any application in a Pod to resolve an address through Consul DNS without disrupting the underlying Kubernetes DNS functionality.
+- **Consul service discovery using fewer resources**. When you use the Consul DNS proxy for service discovery, you do not need to schedule Consul client agents or dataplanes as sidecars. One Kubernetes Service that uses the same resources as a single Consul dataplane provides Pods access to the Consul service catalog.
+- **Consul DNS without gossip communication**. The Consul DNS service runs on both Consul server and Consul client agents, which use [gossip communication](/consul/docs/security/encryption/gossip) to ensure that service discovery results are up-to-date. The Consul DNS proxy provides access to Consul DNS without the security overhead of agent-to-agent gossip.
+
+## Constraints and limitations
+
+If you experience issues using the Consul DNS proxy for Kubernetes, refer to the following list of technical constraints and limitations.
+
+-  You must use Kubernetes as your runtime to use the Consul DNS proxy. You cannot schedule the Consul DNS proxy in other container-based environments.
+-  To perform DNS lookups on other admin partitions, you must [export services between partitions](/consul/docs/connect/config-entries/exported-services) before you can query them.

website/content/docs/services/discovery/dns-cache.mdx

@@ -1,22 +1,33 @@
 ---
 layout: docs
-page_title: Enable dynamic DNS queries
+page_title: Scale Consul DNS
 description: ->
     You tune Consul DNS query handling to balance between current information and reducing request response time. Learn how to enable caching by modifying TTL values, how to return stale results from the DNS cache, and how to configure Consul for negative response caching.
 ---
 
-# DNS caching
+# Scale Consul DNS
 
 This page describes the process to return cached results in response to DNS lookups. Consul agents can use DNS caching to reduce response time, but might provide stale information in the process.
 
-## Introduction
+## Scaling techniques
 
-By default, Consul serves all DNS results with a `0` TTL value, which prevents any 
+By default, Consul serves all DNS results with a `0` TTL value so that it returns the most recent information. When operating at scale, this configuration may result in additional latency because servers must respond to every DNS query. There are several strategies for distributing this burden in your datacenter:
-caching. This configuration returns the most recent information because each DNS lookup 
-runs every time. However, this configuration adds latency to each lookup and can potentially 
-exhaust the query throughput of a datacenter.
 
-There are several ways you can modify to fine-tune Consul DNS lookup behavior to best suit your network's requirements.
+- [Allow Stale Reads](#stale-reads). Allows other servers besides the leader to answer the query rather than forwarding it to the leader.
+- [Configure DNS TTLs](#ttl-values). Configure DNS time-to-live (TTL) values for nodes or services so that the DNS subsystem on the container’s operating system can cache responses. Services then resolve DNS queries locally without any external requests.  
+- [Add Read Replicas](/consul/docs/enterprise/read-scale). Enterprise users can use read replicas, which are additional Consul servers that replicate cluster data without participating in the Raft quorum.
+- [Use Cache to prevent server requests](/consul/docs/agent/config/config-files#dns_use_cache). Configure the Consul client to use its agent cache to subscribe to events about a service or node. After you establish the watch, the local Consul client agent can resolve DNS queries about the service or node without querying Consul servers.
+
+The following table describes the availability of each scaling technique depending on whether you configure Consul to offload DNS requests from the cluster leader to a client agent, dataplane, or DNS proxy.
+
+| Scaling technique                   | Supported by client agents    | Supported by dataplanes       | Supported by Consul DNS Proxy |
+| :---------------------------------- | :---------------------------: | :---------------------------: | :---------------------------: |
+| Configure DNS TTLs                  | ✅                       | ✅                       | ✅                       |
+| Allow Stale Reads                   | ✅                       | ✅                       | ✅                       |
+| Add Read Replicas                   | ✅                       | ✅                       | ✅                       |
+| Use Cache to prevent server request | ✅                       | ❌                      | ❌                      |
+
+For more information about considerations for Consul deployments that operate at scale, refer to [Operating Consul at Scale](/consul/docs/architecture/scale).
 
 ## TTL values ((#ttl))
 

website/content/docs/services/discovery/dns-forwarding/index.mdx

@@ -20,12 +20,13 @@ When configured with default values, Consul exposes the DNS interface on port `8
 Instead of running Consul with an administrative or root account, you can forward appropriate queries to Consul, running on an unprivileged port, from another DNS server or using port redirect.
 
 There are two configurations for a node's DNS forwarding behavior:
+
  - **Conditional DNS forwarding**: the local DNS servers are configured to forward to Consul only queries relative to the `.consul` zone. All other queries are still served via the default DNS server in the node.
  - **Full DNS forwarding**: Consul serves all DNS queries and forwards to a remote DNS server the ones outside `.consul` domain.
 
 ### Conditional DNS forwarding
 
-We recommend the conditional DNS forwarding approach. This configuration lowers the Consul agent's resource consumption by limiting the number of DNS requests it handles. 
+We recommend the conditional DNS forwarding approach. This configuration lowers the Consul agent's resource consumption by limiting the number of DNS requests it handles.
 
 ![Consul DNS conditional forwarding - Only .consul requests are routed to Consul](/img/consul-dns-conditional-forwarding.png#light-theme-only)
 ![Consul DNS conditional forwarding - Only .consul requests are routed to Consul](/img/consul-dns-conditional-forwarding-dark.png#dark-theme-only)
@@ -43,7 +44,7 @@ This approach can be useful in scenarios where the Consul agent's node is alloca
 
 This behavior is not enabled by default. Consul standard configuration only resolves DNS records inside the `.consul` zone. To enable DNS forwarding, you need to set the [recursors](/consul/docs/agent/config/config-files#recursors) option in your Consul configuration.
 
-In this scenario, if a Consul DNS reply includes a `CNAME` record pointing outside the `.consul` top level domain, then the DNS reply only includes `CNAME` records by default. 
+In this scenario, if a Consul DNS reply includes a `CNAME` record pointing outside the `.consul` top level domain, then the DNS reply only includes `CNAME` records by default.
 
 When `recursors` is set and the upstream resolver is functioning correctly, Consul tries to resolve CNAMEs and include any records (for example, A, AAAA, PTR) for them in its DNS reply. In these scenarios, Consul is used for full DNS forwarding and is able to serve queries for all domains.
 

website/content/docs/services/discovery/dns-overview.mdx

@@ -10,6 +10,7 @@ description: >-
 This topic provides overview information about how to look up Consul nodes and services using the Consul DNS.
 
 ## Consul DNS
+
 The Consul DNS is the primary interface for discovering services registered in the Consul catalog. The DNS enables you to look up services and nodes registered with Consul using terminal commands instead of making HTTP API requests to Consul.
 
 We recommend using the DNS for service discovery in virtual machine (VM) environments because it removes the need to modify native applications so that they can consume the Consul service discovery APIs.
@@ -17,19 +18,29 @@ We recommend using the DNS for service discovery in virtual machine (VM) environ
 The DNS has several default configurations, but you can customize how the server processes lookups. Refer to [Configure DNS Behaviors](/consul/docs/services/discovery/dns-configuration) for additional information.
 
 ### DNS versus native app integration
+
 You can use DNS to reach services registered with Consul or modify your application to natively consume the Consul service discovery HTTP APIs.
 
 We recommend using the DNS because it is less invasive. You do not have to modify your application with Consul to retrieve the service’s connection information. Instead, you can use a DNS fully qualified domain (FQDN) that conforms to Consul's lookup format to retrieve the relevant information.
 
-Refer to [ Native App Integration](/consul/docs/connect/native) and its [Go package](/consul/docs/connect/native/go) for additional information.
+Refer to [Native App Integration](/consul/docs/connect/native) and its [Go package](/consul/docs/connect/native/go) for additional information.
 
 ### DNS versus upstreams
+
 If you are using Consul for service discovery and have not enabled service mesh features, then use the DNS to discover services and nodes in the Consul catalog.
 
 If you are using Consul for service mesh on VMs, you can use upstreams or DNS. We recommend using upstreams because you can query services and nodes without modifying the application code or environment variables. Refer to [Upstream Configuration Reference](/consul/docs/connect/proxies/proxy-config-reference#upstream-configuration-reference) for additional information.
 
 If you are using Consul on Kubernetes, refer to [the upstreams annotation documentation](/consul/docs/k8s/annotations-and-labels#consul-hashicorp-com-connect-service-upstreams) for additional information.
 
+## Consul DNS on Kubernetes
+
+Consul on Kubernetes supports approaches to using the Consul DNS in Kubernetes clusters.
+
+For service discovery operations, refer to [Consul DNS views on Kubernetes](/consul/docs/k8s/dns/views).
+
+For service mesh operations, refer to [Resolve Consul DNS requests in Kubernetes](/consul/docs/k8s/dns/enable).
+
 ## DNS forwarding
 
 You can configure your local DNS servers to use Consul.
@@ -37,6 +48,7 @@ You can configure your local DNS servers to use Consul.
 Refer to [DNS Forwarding](/consul/docs/services/discovery/dns-forwarding) for additional information.
 
 ## Static queries
+
 Node lookups and service lookups are the fundamental types of static queries. Depending on your use case, you may need to use different query methods and syntaxes to query the DNS for services and nodes.
 
 Consul relies on a very specific format for queries to resolve names. Note that all queries are case-sensitive.
@@ -44,4 +56,5 @@ Consul relies on a very specific format for queries to resolve names. Note that
 Refer to [Perform Static DNS Lookups](/consul/docs/services/discovery/dns-static-lookups) for details about how to perform node and service lookups.
 
 ## Prepared queries
+
 Prepared queries are configurations that enable you to register complex DNS queries. They provide lookup features that extend Consul's service discovery capabilities, such as filtering by multiple tags and automatically querying remote datacenters for services if no healthy nodes are available in the local datacenter. You can also create prepared query templates that match names using a prefix match, allowing a single template to apply to potentially many services. Refer to [Enable Dynamic DNS Queries](/consul/docs/services/discovery/dns-dynamic-lookups) for additional information.

website/data/docs-nav-data.json

@@ -1675,7 +1675,25 @@
       },
       {
         "title": "Consul DNS",
-        "path": "k8s/dns"
+        "routes": [
+          {
+            "title": "DNS proxy for service discovery",
+            "routes": [
+              {
+                "title": "Overview",
+                "path": "k8s/dns/views"
+              },
+              {
+                "title": "Enable Consul DNS proxy",
+                "path": "k8s/dns/views/enable"
+              }
+            ]
+          },
+          {
+          "title": "Enable on dataplanes",
+          "path": "k8s/dns/enable"
+          }
+        ]
       },
       {
         "title": "Upgrade",

website/redirects.js

@@ -247,4 +247,14 @@ module.exports = [
     destination: '/consul/docs/architecture/catalog#v2-catalog',
     permanent: true,
   },
+  {
+    source: '/consul/docs/k8s/dns',
+    destination: '/consul/docs/k8s/dns/enable',
+    permanent: true,
+  },
+  {
+    source: '/consul/docs/:version(v1\.(?:11|12|13|14|15|16|17|18)\.x)/k8s/dns/enable',
+    destination: '/consul/docs/:version/k8s/dns',
+    permanent: true,
+  }
 ]