Commit Graph

22 Commits (c83a9246281d9eb6915e591673defc05d8590c00)

Author SHA1 Message Date
Rick Buskens 77bda6fd16 Updated configuration to spawn multiple threads for processing 2016-08-01 15:08:11 -04:00
Alex Robinson 19bf9d0624 Support journal logs in fluentd-gcp.
Only run the systemd-journal plugin when on a platform that requests it.
The plugin crashes the fluentd process if the journal isn't present, so
it can't just be run blindly in all configurations.
2016-06-24 00:30:32 +00:00
Alex Robinson bb51c94172 Add collection of the new glbc and cluster-autoscaler logs 2016-06-03 15:50:52 +00:00
Alex Robinson 3216204668 Update to the latest version of google-fluentd.
It includes some performance improvements for parsing JSON (which is
very important for us, since all Docker logs are JSON) as well as a
couple new settings, like forcing of a flush of multiline logs after a
time period rather than having to wait until a new log is seen before
feeling confident flushing the previous one.
2016-03-05 00:18:31 +00:00
Alex Robinson f8965a6f41 Fix parsing of kube logs to handle logs split across lines.
I didn't expect glog to split single log statements onto multiple lines,
but apparently it does if they're long enough. This groups them back
together appropriately.
2016-02-26 22:58:46 +00:00
Christopher Eck c34ecf7100 Enable fluentd to parse the sub-second portion of timestamps coming from docker logs. 2016-02-18 11:07:27 -08:00
Alex Robinson 592da85a81 Parse system logs into structured messages in fluentd.
This allows you to filter based on components of the log,
like their severity, in the developers console or elasticsearch.
2016-02-04 08:14:59 +00:00
Alex Robinson ac13e851e2 Don't let fluentd pipe its own logs directly back into itself. 2016-01-27 00:30:53 +00:00
Alex Robinson 95a4ea3fc8 Increase the fluentd buffer chunk size to improve write throughput.
Also reduce the max wait between retries, 30 seconds should be more than
enough backoff.
2016-01-27 00:30:50 +00:00
Alex Robinson 8040f75594 Update fluentd-gcp to use a new google-fluentd build with my recent
changes to fluent-plugin-google-cloud to attach Kubernetes metadata to
logs.

Along with this, separate logs from containers in the cluster out from
logs from the daemons running on the node by instantiating two instances
of the output plugin, one which uses the new metadata (for containers)
and one which doesn't (for things like docker and the kubelet).
2015-09-15 23:14:43 +00:00
Alex Robinson 614b1cf3c1 Replace /varlog with /var/log in the fluentd-gcp config to match the
recently updated fluent-es config.
2015-09-11 16:52:49 -07:00
Alex Robinson f6f8cb07ae Transfer maintainer responsibilities for fluentd configs/images. 2015-08-26 21:48:51 +00:00
Alex Robinson c887ad75b8 Stop collecting monit logs via fluentd. 2015-08-07 00:12:21 +00:00
Alex Robinson 7430181d65 Collect more of the VM's log files (monit, salt) in fluentd-gcp. 2015-08-05 00:21:47 +00:00
Alex Robinson 9e18c29e9f Make fluentd-gcp collect etcd's logs on the master. 2015-07-30 22:20:57 +00:00
Alex Robinson 4bfbd5f553 Add limits to the fluentd buffer size. 2015-07-03 23:24:08 +00:00
Alex Robinson 8855371763 Configure fluentd to collect additional log files, including master log files.
This also merges the output blocks together because there's no compelling
reason to have a separate output block for each input.
2015-07-01 20:07:47 +00:00
Satnam Singh 33311b46a2 Adjust Fluentd source format for Docker files to be JSON for Cloud Logging 2015-06-15 10:22:27 -07:00
Satnam Singh 941a457851 Trim Fleuntd tag for Cloud Logging 2015-04-30 13:39:35 -07:00
Satnam Singh 20a0905d61 New Fluentd to Cloud Logging Dockerfile with args 2015-04-28 10:30:32 -07:00
Satnam Singh 2444c1f943 Propagate pod and container name for log files 2015-04-22 15:08:51 -07:00
Alex Robinson 059a8c92bd Move the logging-related directories to where I think they belong.
1. Move fluentd-gcp to be a core cluster addon, rather than a contrib.
2. Get rid of the synthetic logger under contrib, since the exact same
synthetic logger was also included in the logging-demo.
3. Move the logging-demo to examples, since it's effectively an example.

We should also consider adding on a GCP section to the logging-demo
example :)
2015-04-17 23:59:50 +00:00