k3s/cluster/addons/fluentd-elasticsearch

Elasticsearch Add-On

This add-on consists of a combination of Elasticsearch, Fluentd and Kibana. Elasticsearch is a search engine that is responsible for storing our logs and allowing for them to be queried. Fluentd sends log messages from Kubernetes to Elasticsearch, whereas Kibana is a graphical interface for viewing and querying the logs stored in Elasticsearch.

Note: this addon should not be used as-is in production. This is an example and you should treat it as such. Please see at least the Security and the Storage sections for more information.

Elasticsearch

Elasticsearch is deployed as a StatefulSet, which is like a Deployment, but allows for maintaining state on storage volumes.

Security

Elasticsearch has capabilities to enable authorization using the X-Pack plugin. For the sake of simplicity this example uses the fully open source prebuild images from elastic that do not contain the X-Pack plugin. If you need these features, please consider building the images from either the "basic" or "platinum" version. After enabling these features, follow official documentation to set up credentials in Elasticsearch and Kibana. Don't forget to propagate those credentials also to Fluentd in its configuration, using for example environment variables. You can utilize ConfigMaps and Secrets to store credentials in the Kubernetes apiserver.

Initialization

The Elasticsearch StatefulSet manifest specifies that there shall be an init container executing before Elasticsearch containers themselves, in order to ensure that the kernel state variable vm.max_map_count is at least 262144, since this is a requirement of Elasticsearch. You may remove the init container if you know that your host OS meets this requirement.

Storage

The Elasticsearch StatefulSet will use the EmptyDir volume to store data. EmptyDir is erased when the pod terminates, here it is used only for testing purposes. Important: please change the storage to persistent volume claim before actually using this StatefulSet in your setup!

Fluentd

Fluentd is deployed as a DaemonSet which spawns a pod on each node that reads logs, generated by kubelet, container runtime and containers and sends them to Elasticsearch.

Learn more in the official Kubernetes documentation.

Building

Both images are now being hosted in google cloud and are built via the cloud build product. To build these images yourself you will need to have the gcloud sdk installed and you will need to login. You can then run make in either image directory to trigger a container build.

Known problems

Since Fluentd talks to the Elasticsearch service inside the cluster, instances on masters won't work, because masters have no kube-proxy. Don't mark masters with the label mentioned in the previous paragraph or add a taint on them to avoid Fluentd pods scheduling there.

Running EFK stack in production

The configurations provided here, for Elasticsearch, Fluentd & Kibana (EFK), are a starting point and should not be considered to be production ready.

If you like to run these tools in a production environment you could use the Helm charts, provided by the Helm community, which are used by a lot of people and therefore are widely tested. You can find them all via the Helm Hub.

The source of the mentioned charts can be found here:

• Elasticsearch
• Fluentd-elasticsearch
• Kibana