Commit Graph

56 Commits (def802272904c04f0f3f736c8616bc68fb1edf8f)

Author SHA1 Message Date
Piotr Szczesniak c00e57789d Added upgrade story from manifest pod to ds 2016-12-15 13:48:32 +01:00
Piotr Szczesniak a52637f09f Migrated fluentd to daemon set 2016-12-15 13:48:32 +01:00
Mik Vyatskov e52c3e77e2 Avoid exporting fluentd-gcp own logs 2016-12-14 14:43:05 +01:00
Mik Vyatskov 6215089796 Unify fluentd-gcp configurations 2016-11-30 14:47:24 +01:00
Mik Vyatskov ba19a27974 Bump version of the fluentd-gcp image 2016-11-22 11:12:28 +01:00
Mik Vyatskov b5b47620f5 Add kube-proxy logs to fluentd configs 2016-11-21 15:43:28 +01:00
Mik Vyatskov aa31f7bc02 Bump fluentd-gcp image version 2016-11-17 21:21:10 +01:00
Mik Vyatskov cf845eda7c Remove fixed docker daemon log time format in fluentd configs 2016-11-17 15:52:07 +01:00
Mik Vyatskov 279e20ed13 Fix flunetd-gcp image Dockerfile 2016-11-08 15:14:09 +01:00
Mik Vyatskov d478307106 Fix config file names inside fluentd-gcp image 2016-11-07 20:31:12 +01:00
Kubernetes Submit Queue 04a81cdd3e Merge pull request #36363 from Crassirostris/fluentd-gcp-image-build-fix
Automatic merge from submit-queue

Fix fluentd-gcp Dockerfile to reduce image size

Change reduces image size by 150MB.

@piosz
2016-11-07 07:56:11 -08:00
Mik Vyatskov d3465e5b8c Add rescheduler logs to the fluentd-gcp configuration 2016-11-07 15:10:52 +01:00
Mik Vyatskov 220168c9aa Fix fluentd-gcp Dockerfile to reduce image size 2016-11-07 15:01:38 +01:00
Mik Vyatskov b4c7dfa674 Switched from google-fluentd to configuring fluentd from scratch 2016-11-02 12:08:36 +01:00
Mik Vyatskov 24361fce06 Update fluentd-gcp configuration 2016-11-02 12:06:01 +01:00
Jess Frazelle cde5559d09
Update `gcloud docker` commands to use `gcloud docker -- ARGS`
We can then avoid the following warning:
```
WARNING: The '--' argument must be specified between gcloud specific args on the left and DOCKER_ARGS on the right. IMPORTANT: previously, commands allowed the omission of the --, and unparsed arguments were treated as implementation args. This usage is being deprecated and will be removed in March 2017.
This will be strictly enforced in March 2017. Use 'gcloud beta docker' to see new behavior.
```

Signed-off-by: Jess Frazelle <acidburn@google.com>
2016-10-10 13:42:34 -07:00
Manuel de Brito Fontes e2c50152dd Remove kubernetes metadata plugin 2016-08-23 08:21:46 -03:00
Manuel de Brito Fontes 5753a3196c Reduce the size of fluentd images 2016-08-23 08:21:45 -03:00
Rick Buskens 77bda6fd16 Updated configuration to spawn multiple threads for processing 2016-08-01 15:08:11 -04:00
David McMahon ef0c9f0c5b Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
Alex Robinson 19bf9d0624 Support journal logs in fluentd-gcp.
Only run the systemd-journal plugin when on a platform that requests it.
The plugin crashes the fluentd process if the journal isn't present, so
it can't just be run blindly in all configurations.
2016-06-24 00:30:32 +00:00
Alex Robinson bb51c94172 Add collection of the new glbc and cluster-autoscaler logs 2016-06-03 15:50:52 +00:00
Alex Robinson 3216204668 Update to the latest version of google-fluentd.
It includes some performance improvements for parsing JSON (which is
very important for us, since all Docker logs are JSON) as well as a
couple new settings, like forcing of a flush of multiline logs after a
time period rather than having to wait until a new log is seen before
feeling confident flushing the previous one.
2016-03-05 00:18:31 +00:00
Alex Robinson 29eb6020fb Merge pull request #22074 from a-robinson/images
Update the logging images to parse sub-second docker timestamps.
2016-03-04 09:04:33 -08:00
Alex Robinson feea376b5c Update the logging images to parse sub-second docker timestamps. 2016-03-03 10:21:30 -08:00
Alex Robinson f8965a6f41 Fix parsing of kube logs to handle logs split across lines.
I didn't expect glog to split single log statements onto multiple lines,
but apparently it does if they're long enough. This groups them back
together appropriately.
2016-02-26 22:58:46 +00:00
Christopher Eck c34ecf7100 Enable fluentd to parse the sub-second portion of timestamps coming from docker logs. 2016-02-18 11:07:27 -08:00
Alex Robinson e5c5f76402 Update the default fluentd images to parse system logs. 2016-02-04 23:29:27 +00:00
Alex Robinson 592da85a81 Parse system logs into structured messages in fluentd.
This allows you to filter based on components of the log,
like their severity, in the developers console or elasticsearch.
2016-02-04 08:14:59 +00:00
Paul Morie b672785d72 Add boilerplate checks for Dockerfiles 2016-02-03 18:35:26 -05:00
Paul Morie 05bd107301 Add boilerplate checks for Makefiles 2016-02-03 18:35:26 -05:00
Alex Robinson 6acf297231 Update the fluentd versions to include fixes for #19405. 2016-01-27 00:30:53 +00:00
Alex Robinson ac13e851e2 Don't let fluentd pipe its own logs directly back into itself. 2016-01-27 00:30:53 +00:00
Alex Robinson 95a4ea3fc8 Increase the fluentd buffer chunk size to improve write throughput.
Also reduce the max wait between retries, 30 seconds should be more than
enough backoff.
2016-01-27 00:30:50 +00:00
Alex Robinson f0ebe4066f Update fluentd-gcp to include all recent improvements that are in the
new google-fluentd 1.5.4-1 build.
2015-10-29 11:54:07 +00:00
Alex Robinson 8040f75594 Update fluentd-gcp to use a new google-fluentd build with my recent
changes to fluent-plugin-google-cloud to attach Kubernetes metadata to
logs.

Along with this, separate logs from containers in the cluster out from
logs from the daemons running on the node by instantiating two instances
of the output plugin, one which uses the new metadata (for containers)
and one which doesn't (for things like docker and the kubelet).
2015-09-15 23:14:43 +00:00
Alex Robinson f691dca96e Don't install unused configs in the fluentd-gcp Dockerfile. 2015-09-11 16:52:52 -07:00
Alex Robinson 614b1cf3c1 Replace /varlog with /var/log in the fluentd-gcp config to match the
recently updated fluent-es config.
2015-09-11 16:52:49 -07:00
Eric Paris 04fc8ae3dd s|github.com/GoogleCloudPlatform/kubernetes|github.com/kubernetes/kubernetes| 2015-09-03 10:10:11 -04:00
Alex Robinson b3154a7815 Remove the garbage log file in the fluentd-gcp that gets filled with
gunk when installing the google-fluentd agent.

Also let it log things by not redirecting to a file within the container
and only using -q (warning logs only) rather than -qq (error logs only).
2015-08-27 21:41:44 +00:00
Alex Robinson f6f8cb07ae Transfer maintainer responsibilities for fluentd configs/images. 2015-08-26 21:48:51 +00:00
Alex Robinson c887ad75b8 Stop collecting monit logs via fluentd. 2015-08-07 00:12:21 +00:00
Alex Robinson 7430181d65 Collect more of the VM's log files (monit, salt) in fluentd-gcp. 2015-08-05 00:21:47 +00:00
Alex Robinson 9e18c29e9f Make fluentd-gcp collect etcd's logs on the master. 2015-07-30 22:20:57 +00:00
Alex Robinson 0889307522 Update the fluentd-gcp images to include their latest Dockerfile changes.
Their Dockerfiles were changed in #11768.
2015-07-24 01:27:42 +00:00
Todd Derr 98ed2e8606 Remove gem update for fluent-plugin-google-cloud
This was originally submitted to pick up v0.3.1 of the cloud logging
plugin which had a fix for the name 'metadata' failing to resolve.
Since new releases of google-fluentd have this fix, it is no longer
required.

I've done some additional testing of 'gem update' behavior in the interim
and I think it is ok to use in targeted situations, but we should not be
doing an unconstrained update in general. The issue is that updating a
gem may bring new dependencies, some of those dependencies may include
native code, so it may try to launch a compiler, which isn't desirable
and prone to failure.

If we do need to grab an updated gem in the future we should specify an
explicit version and the --minimal-deps flag.
2015-07-23 15:06:56 -04:00
Alex Robinson 4bfbd5f553 Add limits to the fluentd buffer size. 2015-07-03 23:24:08 +00:00
Alex Robinson 8855371763 Configure fluentd to collect additional log files, including master log files.
This also merges the output blocks together because there's no compelling
reason to have a separate output block for each input.
2015-07-01 20:07:47 +00:00
Tim St. Clair f5830edbbc s/gcloud preview docker/gcloud docker/ 2015-06-18 12:27:08 -07:00
Satnam Singh 33311b46a2 Adjust Fluentd source format for Docker files to be JSON for Cloud Logging 2015-06-15 10:22:27 -07:00