Discussion is in the comments on 7260403fdd
and fix consists of skipping first 3 lines while creating a "new" log file.
This should be sufficient to make fail2ban detect "log rotation" and thus
function properly
Somehow on that elderly squeeze Debian sparc box, I got error that self.__notifier
was not defined. So first I did define it now in the constructor, but mystery remains
how come it was not defined -- wasn"t run() then run (where it is defined)?
Anyways -- conditioning on it being defined might be safer may be?
Not sure (need to go to sleep) if with this change but on this box I also run from time
to time either into stalling of fail2ban-testcases and refusing to exit normally or
======================================================================
ERROR: test_move_into_file_after_removed (testcases.filtertestcase.MonitorFailures<FilterPyinotify>(/tmp/monitorfailures_FilterPyinotifypcHmMJfail2ban))
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/yoh/deb/gits/fail2ban/testcases/filtertestcase.py", line 473, in tearDown
self.filter.stop()
File "/home/yoh/deb/gits/fail2ban/server/filterpyinotify.py", line 196, in stop
self.__notifier.stop()
File "/usr/lib/pymodules/python2.6/pyinotify.py", line 1315, in stop
threading.Thread.join(self)
File "/usr/lib/python2.6/threading.py", line 633, in join
raise RuntimeError("cannot join thread before it is started")
RuntimeError: cannot join thread before it is started
that is with pyinotify 0.8.9-1 so could quite be "related" to its age.
When a file is deleted, the watcher for it is automatically removed.
FilterPyinotify is detecting a new file being created with the same
name, and in turn attempts to remove the watch for the deleted file
(which has already been removed automatically).
Also, IN_IGNORED events are generated when a file is deleted, but these
weren't being caught, causing an non-existent file path to be passed to
FilterPyinotify._process_file (which caught the exceptions in
Filter.getFailures).
Realized --print-all-{missed,ignored} aren't meant to be simple on/off,
but are meant to enable printing all lines even when there are more
than 20.
This restores the behavior of --print-all-*, but preserves the
memory-savings when --print-all-* are not specified.
Also adds --print-no-{missed,ignored} which can be used to suppress
printing of a line type entirely.
When testing proccessing a large logfile (430 MB, 2M lines) without
--print-all-missed or --print-all-ignored, fail2ban-regex consumes
just over 2 GB of RAM before completing. Even though it does not need
to retain any of the missed or ignored log lines, it does anyway.
With this patch, memory use never grows beyond about 11 MB (unless
--print-all-* are enabled).
Previously, fail2ban-regex always printed both regardless of whether
those options were present or not.
This commit duplicates the patch I submitted at
https://sourceforge.net/p/fail2ban/patches/25/
Yaroslav Halchenko