Commit Graph

55 Commits (1b7dacde4bfb91002205956bb90767a9b6a1cee4)

Author SHA1 Message Date
Jeff Grafton 1c2ea28080 Update images that use ubuntu-slim base image to :0.6 2017-01-11 15:07:04 -08:00
kayrus 8435d19982 Enable kubernetes_metadata by default for ELK stack 2017-01-11 14:08:01 +01:00
Jeff Grafton 19aafd291c Always --pull in docker build to ensure recent base images 2017-01-10 16:21:05 -08:00
Mik Vyatskov 57ec7b77fd Fix fluentd-gcp image config by avoiding processing its own logs 2017-01-09 10:05:33 +01:00
Lucas Käldström 3c5b5f5963 Remove all MAINTAINER statements in the codebase as they aren't very useful and now deprecated 2016-12-17 20:34:10 +02:00
keglevich3 b68e4c0a6d enable the kubernetes-metadata-plugin
instruction to users that would like to enable the fluentd kubernetes-metadata plugin.
Relevant only if you are using daemon-set
2016-12-06 23:48:19 +02:00
Mik Vyatskov 81eeeec894 Bump fluentd-es image version 2016-11-22 12:03:46 +01:00
Mik Vyatskov b5b47620f5 Add kube-proxy logs to fluentd configs 2016-11-21 15:43:28 +01:00
Mik Vyatskov cf845eda7c Remove fixed docker daemon log time format in fluentd configs 2016-11-17 15:52:07 +01:00
Mik Vyatskov 800aafea9b Add rescheduler logs to the fluentd-elasticsearch configuration 2016-11-07 20:24:06 +01:00
Eduardo Silva 5e321af2ed cluser/addons: fluentd-elasticsearch: preload Jemalloc for td-agent
The current DockerFile build an image using td-agent package but it let
the service run with the default memory allocator provided by glibc.

In high load environments, is highly required to use a customized memory
allocator such as Jemalloc. Otherwise the service will face a high memory
fragmentation ending up in 'high memory' usage from a monitoring perspective.

td-agent package by default install Jemalloc and set the LD_PRELOAD
environment variable through it init script, but since the service is
launched through Docker the env variable needs to be set manually.

After this patch, when running td-agent container image now is possible
to see that Jemalloc is used:

  root@monotop:/proc/18810# cat maps |grep jemall
  7f251eddd000-7f251ee1b000 ... /opt/td-agent/embedded/lib/libjemalloc.so.2
  7f251ee1b000-7f251f01b000 ... /opt/td-agent/embedded/lib/libjemalloc.so.2
  7f251f01b000-7f251f01d000 ... /opt/td-agent/embedded/lib/libjemalloc.so.2
  7f251f01d000-7f251f01e000 ... /opt/td-agent/embedded/lib/libjemalloc.so.2

For a reference about the memory usage difference between malloc v/s jemalloc
please refer to the following chart:

  https://goo.gl/dVYTmw

Signed-off-by: Eduardo Silva <eduardo@treasure-data.com>
2016-10-20 10:50:58 -06:00
Jess Frazelle cde5559d09
Update `gcloud docker` commands to use `gcloud docker -- ARGS`
We can then avoid the following warning:
```
WARNING: The '--' argument must be specified between gcloud specific args on the left and DOCKER_ARGS on the right. IMPORTANT: previously, commands allowed the omission of the --, and unparsed arguments were treated as implementation args. This usage is being deprecated and will be removed in March 2017.
This will be strictly enforced in March 2017. Use 'gcloud beta docker' to see new behavior.
```

Signed-off-by: Jess Frazelle <acidburn@google.com>
2016-10-10 13:42:34 -07:00
Manuel de Brito Fontes e2c50152dd Remove kubernetes metadata plugin 2016-08-23 08:21:46 -03:00
Manuel de Brito Fontes 5753a3196c Reduce the size of fluentd images 2016-08-23 08:21:45 -03:00
haoyuan a1f20cbdf4 fix typo 2016-08-08 19:33:51 +08:00
Rick Buskens 77bda6fd16 Updated configuration to spawn multiple threads for processing 2016-08-01 15:08:11 -04:00
David McMahon ef0c9f0c5b Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
Alex Robinson bb51c94172 Add collection of the new glbc and cluster-autoscaler logs 2016-06-03 15:50:52 +00:00
Mike Zupan dc643d2592 Fixing a simple typo 2016-03-06 23:49:35 -08:00
Alex Robinson 29eb6020fb Merge pull request #22074 from a-robinson/images
Update the logging images to parse sub-second docker timestamps.
2016-03-04 09:04:33 -08:00
Alex Robinson feea376b5c Update the logging images to parse sub-second docker timestamps. 2016-03-03 10:21:30 -08:00
Alex Robinson f8965a6f41 Fix parsing of kube logs to handle logs split across lines.
I didn't expect glog to split single log statements onto multiple lines,
but apparently it does if they're long enough. This groups them back
together appropriately.
2016-02-26 22:58:46 +00:00
Christopher Eck c34ecf7100 Enable fluentd to parse the sub-second portion of timestamps coming from docker logs. 2016-02-18 11:07:27 -08:00
Alex Robinson e5c5f76402 Update the default fluentd images to parse system logs. 2016-02-04 23:29:27 +00:00
Alex Robinson 592da85a81 Parse system logs into structured messages in fluentd.
This allows you to filter based on components of the log,
like their severity, in the developers console or elasticsearch.
2016-02-04 08:14:59 +00:00
Paul Morie b672785d72 Add boilerplate checks for Dockerfiles 2016-02-03 18:35:26 -05:00
Paul Morie 05bd107301 Add boilerplate checks for Makefiles 2016-02-03 18:35:26 -05:00
Alex Robinson 6acf297231 Update the fluentd versions to include fixes for #19405. 2016-01-27 00:30:53 +00:00
Alex Robinson ac13e851e2 Don't let fluentd pipe its own logs directly back into itself. 2016-01-27 00:30:53 +00:00
Alex Robinson 95a4ea3fc8 Increase the fluentd buffer chunk size to improve write throughput.
Also reduce the max wait between retries, 30 seconds should be more than
enough backoff.
2016-01-27 00:30:50 +00:00
Matt Moore b750d1dddc Drop the beta for GCR v2 images.
beta.gcr.io is no longer needed to pull through v2.
2015-11-14 12:19:02 -08:00
Matt Moore eeb4eeb17c Move pause and fluentd-elasticsearch to GCR v2.
This scopes down the initially ambitious PR:
https://github.com/kubernetes/kubernetes/pull/14960 to replace just
`pause` and `fluentd-elasticsearch` to come through `beta.gcr.io`.

The v2 versions have been pushed under new tags, `pause:2.0` and
`fluentd-elastisearch:1.12`.

NOTE: `beta.gcr.io` will still serve images using v1 until they are repushed with v2.  Pulls through `gcr.io` will still work after pushing through `beta.gcr.io`, but will be served over v1 (via compat logic).
2015-10-06 16:39:07 -07:00
Eric Paris 04fc8ae3dd s|github.com/GoogleCloudPlatform/kubernetes|github.com/kubernetes/kubernetes| 2015-09-03 10:10:11 -04:00
Jimmi Dyson e07da97447 Switch to fluentd kubernetes metadata plugin to enrich logs with more searchable kubernetes metadata 2015-09-02 10:33:15 +01:00
Alex Robinson d02dc64036 Let fluentd-es log by not redirecting to a file within the container
and only using -q (warning logs only) rather than -qq (error logs only).
2015-08-27 21:41:44 +00:00
Alex Robinson f6f8cb07ae Transfer maintainer responsibilities for fluentd configs/images. 2015-08-26 21:48:51 +00:00
Satnam Singh 278a3911fd Fetch fluentd td agent securely 2015-08-19 13:34:03 -07:00
Alex Robinson c887ad75b8 Stop collecting monit logs via fluentd. 2015-08-07 00:12:21 +00:00
Alex Robinson cc02e50fc7 Collect more of the VM's log files (monit, salt) in fluentd-es. 2015-08-05 00:22:38 +00:00
Alex Robinson 69ab7026af Update fluentd-es to collect etcd logs and remove merge conflict cruft. 2015-07-30 22:21:01 +00:00
Brendan Burns 988aa6fdf6 Move things into a 'kube-system' namespace. 2015-07-06 15:08:23 -07:00
Alex Robinson 4bfbd5f553 Add limits to the fluentd buffer size. 2015-07-03 23:24:08 +00:00
Alex Robinson 8855371763 Configure fluentd to collect additional log files, including master log files.
This also merges the output blocks together because there's no compelling
reason to have a separate output block for each input.
2015-07-01 20:07:47 +00:00
Tim St. Clair f5830edbbc s/gcloud preview docker/gcloud docker/ 2015-06-18 12:27:08 -07:00
Jeff Lowdermilk 553f9f822b Add ga-beacon analytics to gendocs scripts
hack/run-gendocs.sh puts ga-beacon analytics link into all md files,
hack/verify-gendocs.sh verifies presence of link.
2015-05-15 18:56:38 -07:00
Satnam Singh 95007d7217 Improve comment in Fluentd to ES logging config file 2015-04-30 13:42:30 -07:00
Satnam Singh e65f0aafd5 Trim cluster log tags to pod name and container name 2015-04-29 18:24:13 -07:00
Alex Robinson ada9242155 Remove unused env variable from the fluentd dockerfiles and make their apt-get
arguments consistent.
2015-04-27 20:47:38 +00:00
Satnam Singh 2444c1f943 Propagate pod and container name for log files 2015-04-22 15:08:51 -07:00
Satnam Singh 73a9ab2928 Rename logging image to use google_containers 2015-03-31 16:00:52 -07:00