mirror of https://github.com/k3s-io/k3s
Documentation for logging with Elasticsearch and Kibana
Satnam Singh
parent
19991d5315
commit
9cdc7bdef3
Binary file not shown.
|
After Width: | Height: | Size: 40 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 180 KiB |
|
|
@ -0,0 +1,222 @@
|
|||
# Cluster Level Logging with Elasticsearch and Kibana
|
||||
|
||||
On the GCE platform the default cluster level logging support targets
|
||||
[Google Cloud Logging](https://cloud.google.com/logging/docs/) as described at the [Logging](logging.md) getting
|
||||
started page. Here we describe how to set up a cluster to ingest logs into Elasticsearch and view them using Kibana as an
|
||||
alternative to Google Cloud Logging.
|
||||
|
||||
To use Elasticsearch and Kibana for cluster logging you should set the following environment variable as shown below:
|
||||
```
|
||||
KUBE_LOGGING_DESTINATION=elasticsearch
|
||||
```
|
||||
|
||||
You should also ensure that `KUBE_ENABLE_NODE_LOGGING=true` (which is the default for the GCE platform).
|
||||
|
||||
Now when you create a cluster a message will indicate that the Fluentd node-level log collectors
|
||||
will target Elasticsearch:
|
||||
|
||||
```
|
||||
$ cluster/kube-up.sh
|
||||
...
|
||||
Project: kubernetes-satnam
|
||||
Zone: us-central1-b
|
||||
... calling kube-up
|
||||
Project: kubernetes-satnam
|
||||
Zone: us-central1-b
|
||||
+++ Staging server tars to Google Storage: gs://kubernetes-staging-e6d0e81793/devel
|
||||
+++ kubernetes-server-linux-amd64.tar.gz uploaded (sha1 = 6987c098277871b6d69623141276924ab687f89d)
|
||||
+++ kubernetes-salt.tar.gz uploaded (sha1 = bdfc83ed6b60fa9e3bff9004b542cfc643464cd0)
|
||||
Looking for already existing resources
|
||||
Starting master and configuring firewalls
|
||||
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/disks/kubernetes-master-pd].
|
||||
NAME ZONE SIZE_GB TYPE STATUS
|
||||
kubernetes-master-pd us-central1-b 20 pd-ssd READY
|
||||
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/regions/us-central1/addresses/kubernetes-master-ip].
|
||||
+++ Logging using Fluentd to elasticsearch
|
||||
|
||||
```
|
||||
|
||||
The node level Fluentd collector pods and the Elasticsearech pods used to ingest cluster logs and the pod for the Kibana
|
||||
viewer should be running soon after the cluster comes to life.
|
||||
|
||||
```
|
||||
$ kubectl get pods
|
||||
NAME READY REASON RESTARTS AGE
|
||||
elasticsearch-logging-v1-78nog 1/1 Running 0 2h
|
||||
elasticsearch-logging-v1-nj2nb 1/1 Running 0 2h
|
||||
fluentd-elasticsearch-kubernetes-minion-5oq0 1/1 Running 0 2h
|
||||
fluentd-elasticsearch-kubernetes-minion-6896 1/1 Running 0 2h
|
||||
fluentd-elasticsearch-kubernetes-minion-l1ds 1/1 Running 0 2h
|
||||
fluentd-elasticsearch-kubernetes-minion-lz9j 1/1 Running 0 2h
|
||||
kibana-logging-v1-bhpo8 1/1 Running 0 2h
|
||||
kube-dns-v3-7r1l9 3/3 Running 0 2h
|
||||
monitoring-heapster-v4-yl332 1/1 Running 1 2h
|
||||
monitoring-influx-grafana-v1-o79xf 2/2 Running 0 2h
|
||||
|
||||
```
|
||||
|
||||
Here we see that for a four node cluster there is a `fluent-elasticsearch` pod running which gathers
|
||||
the Docker container logs and sends them to Elasticsearch. The Fluentd collector communicates to
|
||||
a Kubernetes service that maps requests to specific Elasticsearch pods. Similarly, Kibana can also be
|
||||
accessed via a Kubernetes service definition.
|
||||
|
||||
|
||||
```
|
||||
$ kubectl get services
|
||||
NAME LABELS SELECTOR IP(S) PORT(S)
|
||||
elasticsearch-logging k8s-app=elasticsearch-logging,kubernetes.io/cluster-service=true,kubernetes.io/name=Elasticsearch k8s-app=elasticsearch-logging 10.0.222.57 9200/TCP
|
||||
kibana-logging k8s-app=kibana-logging,kubernetes.io/cluster-service=true,kubernetes.io/name=Kibana k8s-app=kibana-logging 10.0.193.226 5601/TCP
|
||||
kube-dns k8s-app=kube-dns,kubernetes.io/cluster-service=true,kubernetes.io/name=KubeDNS k8s-app=kube-dns 10.0.0.10 53/UDP
|
||||
53/TCP
|
||||
kubernetes component=apiserver,provider=kubernetes <none> 10.0.0.1 443/TCP
|
||||
monitoring-grafana kubernetes.io/cluster-service=true,kubernetes.io/name=Grafana k8s-app=influxGrafana 10.0.167.139 80/TCP
|
||||
monitoring-heapster kubernetes.io/cluster-service=true,kubernetes.io/name=Heapster k8s-app=heapster 10.0.208.221 80/TCP
|
||||
monitoring-influxdb kubernetes.io/cluster-service=true,kubernetes.io/name=InfluxDB k8s-app=influxGrafana 10.0.188.57 8083/TCP
|
||||
```
|
||||
|
||||
By default two Elasticsearch replicas are created and one Kibana replica is created.
|
||||
|
||||
```
|
||||
$ kubectl get rc
|
||||
CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS
|
||||
elasticsearch-logging-v1 elasticsearch-logging gcr.io/google_containers/elasticsearch:1.4 k8s-app=elasticsearch-logging,version=v1 2
|
||||
kibana-logging-v1 kibana-logging gcr.io/google_containers/kibana:1.3 k8s-app=kibana-logging,version=v1 1
|
||||
kube-dns-v3 etcd gcr.io/google_containers/etcd:2.0.9 k8s-app=kube-dns,version=v3 1
|
||||
kube2sky gcr.io/google_containers/kube2sky:1.9
|
||||
skydns gcr.io/google_containers/skydns:2015-03-11-001
|
||||
monitoring-heapster-v4 heapster gcr.io/google_containers/heapster:v0.14.3 k8s-app=heapster,version=v4 1
|
||||
monitoring-influx-grafana-v1 influxdb gcr.io/google_containers/heapster_influxdb:v0.3 k8s-app=influxGrafana,version=v1 1
|
||||
grafana gcr.io/google_containers/heapster_grafana:v0.7
|
||||
```
|
||||
|
||||
The Elasticsearch and Kibana services are not directly exposed via a publicly reachable IP address. Instead,
|
||||
they can be accessed via the service proxy running at the master. The URLs for accessing Elasticsearch
|
||||
and Kibana via the service proxy can be found using the `kubectl cluster-info` command.
|
||||
|
||||
```
|
||||
$ kubectl cluster-info
|
||||
Kubernetes master is running at https://146.148.94.154
|
||||
Elasticsearch is running at https://146.148.94.154/api/v1/proxy/namespaces/default/services/elasticsearch-logging
|
||||
Kibana is running at https://146.148.94.154/api/v1/proxy/namespaces/default/services/kibana-logging
|
||||
KubeDNS is running at https://146.148.94.154/api/v1/proxy/namespaces/default/services/kube-dns
|
||||
Grafana is running at https://146.148.94.154/api/v1/proxy/namespaces/default/services/monitoring-grafana
|
||||
Heapster is running at https://146.148.94.154/api/v1/proxy/namespaces/default/services/monitoring-heapster
|
||||
InfluxDB is running at https://146.148.94.154/api/v1/proxy/namespaces/default/services/monitoring-influxdb
|
||||
|
||||
```
|
||||
|
||||
Before accessing the logs ingested into Elasticsearch using a browser and the service proxy URL we need to find out
|
||||
the `admin` password for the cluster using `kubectl config view`.
|
||||
|
||||
```
|
||||
$ kubectl config view
|
||||
...
|
||||
- name: kubernetes-satnam_kubernetes-basic-auth
|
||||
user:
|
||||
password: 7GlspJ9Q43OnGIJO
|
||||
username: admin
|
||||
...
|
||||
```
|
||||
|
||||
The first time you try to access the cluster from a browser a dialog box appears asking for the username and password.
|
||||
Use the username `admin` and provide the basic auth password reported by `kubectl config view` for the
|
||||
cluster you are trying to connect to. Connecting to the Elasticsearch URL should then give the
|
||||
status page for Elasticsearch.
|
||||
|
||||
|
||||
|
||||
You can now type Elasticsearch queries directly into the browser. Alternatively you can query Elasticsearch
|
||||
from your local machine using `curl` but first you need to know what your bearer token is:
|
||||
```
|
||||
$ kubectl config view --minify
|
||||
apiVersion: v1
|
||||
clusters:
|
||||
- cluster:
|
||||
certificate-authority-data: REDACTED
|
||||
server: https://146.148.94.154
|
||||
name: kubernetes-satnam_kubernetes
|
||||
contexts:
|
||||
- context:
|
||||
cluster: kubernetes-satnam_kubernetes
|
||||
user: kubernetes-satnam_kubernetes
|
||||
name: kubernetes-satnam_kubernetes
|
||||
current-context: kubernetes-satnam_kubernetes
|
||||
kind: Config
|
||||
preferences: {}
|
||||
users:
|
||||
- name: kubernetes-satnam_kubernetes
|
||||
user:
|
||||
client-certificate-data: REDACTED
|
||||
client-key-data: REDACTED
|
||||
token: JsUe2Z3cXqa17UQqQ8qWGGf4nOSLwSnp
|
||||
```
|
||||
|
||||
Now you can can issue requests to Elasticsearch:
|
||||
```
|
||||
$ curl --header "Authorization: Bearer JsUe2Z3cXqa17UQqQ8qWGGf4nOSLwSnp" --insecure https://146.148.94.154/api/v1/proxy/namespaces/default/services/elasticsearch-logging/
|
||||
{
|
||||
"status" : 200,
|
||||
"name" : "Vance Astrovik",
|
||||
"cluster_name" : "kubernetes-logging",
|
||||
"version" : {
|
||||
"number" : "1.5.2",
|
||||
"build_hash" : "62ff9868b4c8a0c45860bebb259e21980778ab1c",
|
||||
"build_timestamp" : "2015-04-27T09:21:06Z",
|
||||
"build_snapshot" : false,
|
||||
"lucene_version" : "4.10.4"
|
||||
},
|
||||
"tagline" : "You Know, for Search"
|
||||
}
|
||||
|
||||
```
|
||||
Note that you need the trailing slash at the end of the service proxy URL. Here is an example of a search:
|
||||
```
|
||||
$ curl --header "Authorization: Bearer JsUe2Z3cXqa17UQqQ8qWGGf4nOSLwSnp" --insecure https://146.148.94.154/api/v1/proxy/namespaces/default/services/elasticsearch-logging/_search?pretty=true
|
||||
{
|
||||
"took" : 7,
|
||||
"timed_out" : false,
|
||||
"_shards" : {
|
||||
"total" : 6,
|
||||
"successful" : 6,
|
||||
"failed" : 0
|
||||
},
|
||||
"hits" : {
|
||||
"total" : 123711,
|
||||
"max_score" : 1.0,
|
||||
"hits" : [ {
|
||||
"_index" : ".kibana",
|
||||
"_type" : "config",
|
||||
"_id" : "4.0.2",
|
||||
"_score" : 1.0,
|
||||
"_source":{"buildNum":6004,"defaultIndex":"logstash-*"}
|
||||
}, {
|
||||
...
|
||||
"_index" : "logstash-2015.06.22",
|
||||
"_type" : "fluentd",
|
||||
"_id" : "AU4c_GvFZL5p_gZ8dxtx",
|
||||
"_score" : 1.0,
|
||||
"_source":{"log":"synthetic-logger-10lps-pod: 31: 2015-06-22 20:35:33.597918073+00:00\n","stream":"stdout","tag":"kubernetes.synthetic-logger-10lps-pod_default_synth-lgr","@timestamp":"2015-06-22T20:35:33+00:00"}
|
||||
}, {
|
||||
"_index" : "logstash-2015.06.22",
|
||||
"_type" : "fluentd",
|
||||
"_id" : "AU4c_GvFZL5p_gZ8dxt2",
|
||||
"_score" : 1.0,
|
||||
"_source":{"log":"synthetic-logger-10lps-pod: 36: 2015-06-22 20:35:34.108780133+00:00\n","stream":"stdout","tag":"kubernetes.synthetic-logger-10lps-pod_default_synth-lgr","@timestamp":"2015-06-22T20:35:34+00:00"}
|
||||
} ]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The Elasticsearch website contains information about [URI search queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-uri-request.html) which can be used to extract the required logs.
|
||||
|
||||
Alternatively you can view the ingested logs using Kibana. The first time you visit the Kibana URL you will be
|
||||
presented with a page that asks you to configure your view of the ingested logs. Select the option for
|
||||
timeseries values and select `@timestamp`. On the following page select the `Discover` tab and then you
|
||||
should be able to see the ingested logs. You can set the refresh interval to 5 seconds to have the logs
|
||||
regulary refreshed. Here is a typical view of ingested logs from the Kibana viewer.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
[]()
|
||||
Loading…
Reference in New Issue