otherwise years present in the freshly contributed by Dan apache regexes do not match
although should have. I had also to adjust failing now vsftpd test
Otherwise <HOST> regexp might swallow period in the sentence right after the address.
I have decided to enforce alphanumeric instead of switching to non-greedy +? ... because
I think it is closer to what we actually want here
This avoids duplication of code across fail2ban-regex and samples test
cases. This also now more neatly resolves the issue of double counting
date templates matches in fail2ban-regex.
In addition, the samples test cases now also print a warning message
that not all regexs have samples for them, with future plan to change
this to an assertion.
IMHO there is no good reason to capture only gaierror.
on my network it was consistent to error out with
======================================================================
ERROR: testIgnoreIPNOK (testcases.filtertestcase.IgnoreIP)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/yoh/deb/gits/fail2ban/testcases/filtertestcase.py", line 166, in testIgnoreIPNOK
self.assertFalse(self.filter.inIgnoreIPList(ip))
File "/home/yoh/deb/gits/fail2ban/server/filter.py", line 277, in inIgnoreIPList
ips = DNSUtils.dnsToIp(i)
File "/home/yoh/deb/gits/fail2ban/server/filter.py", line 625, in dnsToIp
return socket.gethostbyname_ex(dns)[2]
error: [Errno 11] Resource temporarily unavailable
with this commit tests would pass normally as they should
Previous code would store md5sum of an empty line as the one
identifying the monitored file. That file then was read and possibly
failures were found. Upon next "container.open()", md5 digest of now
present first line was compared against previous digest of an empty
line, which was different, thus file was assumed to be rotated and all
the log lines were read once again.
The History
-----------
In rare cases various tests failed somewhat consistently. Below you
can find one case in test_move_file where such failure really made no
sense -- we should have not had 4 failures by that point.
Fail2ban 0.8.10.dev test suite. Python 2.4.6 (#2, Sep 25 2009, 22:22:06) [GCC 4.3.4]. Please wait...
I: Skipping gamin backend testing. Got exception 'No module named gamin'
I: Skipping pyinotify backend testing. Got exception 'No module named pyinotify'
D: fn=/home/yoh/test/monitorfailures_FilterPoll9nUKoCfail2ban-0 hashes=d41d8cd98f00b204e9800998ecf8427e/d41d8cd98f00b204e9800998ecf8427e inos=5398862/5398862 pos=0 rotate=False
D: Adding a ticket for ('193.168.0.128', 1124013599.0, ['Aug 14 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 193.168.0.128\n'])
D: Adding a ticket for ('193.168.0.128', 1124013599.0, ['Aug 14 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 193.168.0.128\n'])
D: Closed <closed file '/home/yoh/test/monitorfailures_FilterPoll9nUKoCfail2ban-0', mode 'r' at 0x7f472b2efdc8> with pos 1231
D: fn=/home/yoh/test/monitorfailures_FilterPoll9nUKoCfail2ban-0 hashes=d41d8cd98f00b204e9800998ecf8427e/aeb4e73e6922a746d027eb365ece2149 inos=5398862/5398862 pos=1231 rotate=True
D: Adding a ticket for ('193.168.0.128', 1124013599.0, ['Aug 14 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 193.168.0.128\n'])
D: Adding a ticket for ('193.168.0.128', 1124013599.0, ['Aug 14 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 193.168.0.128\n'])
D: Closed <closed file '/home/yoh/test/monitorfailures_FilterPoll9nUKoCfail2ban-0', mode 'r' at 0x7f472b2ef558> with pos 1231
F
======================================================================
FAIL: test_move_file (testcases.filtertestcase.MonitorFailures<FilterPoll>(/home/yoh/test/monitorfailures_FilterPoll9nUKoCfail2ban))
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/yoh/deb/gits/fail2ban/testcases/filtertestcase.py", line 451, in test_move_file
"Queue must be empty but it is not: %s."
AssertionError: Queue must be empty but it is not: server.ticket.FailTicket: ip=193.168.0.128 time=1124013599.0 #attempts=4.
----------------------------------------------------------------------
N.B.1 I preserved here and in the code corresponding additional debug
print statements, which are commented out by default. sensible md5
digest was generated by using hexdigest() instead of current a bit
faster digest(). Running tests with all the debug output simply
breaks the race loose and failure doesn't trigger.
N.B.2 d41d8cd98f00b204e9800998ecf8427e is an md5sum of an empty
string, and aeb4e73e6922a746d027eb365ece2149 of the first line in that
file.
mtime alone is a poor measure here as many of our tests shown -- on older Pythons and some file systems
mtime might be reported only with precision up to a second. If file gets rotated fast, or there are new
modifications within the same second, fail2ban might miss them with Polling backend if judging only by
mtime. With this modification we will track also inode and size which are all indicative of a file
change.
Origin: https://github.com/lenrico/fail2ban
Squashing was done via rebase -i 1524b076d6
to eliminate massive assp sample log file originally added
fixed test date thx to steven
tight control of the filter for ASSP
as yaroslav wishes
as daniel desires
changed from DateASSPlike class to DateStrptime
fixed little things
added new date format support for ASSP SMTP Proxy
* 'master' of git://github.com/fail2ban/fail2ban:
BF: log error only if there were missed config files that couldn't be read
DOC: missing cinfo tags are ok. Log error for self referencing definitions
DOC: s/defination/definition/g learn to spell
ENH: remove stats of config files and use results of SafeConfigParserWithIncludes.read to facilitate meaningful error messages
DOC: ChangeLog for recursive tag substition
ENH: allow recursive tag substitution in action files.
is None is generally faster than == and from looking at those places
should be adequate.
Also while at those files removed unneded duplicate author listing +
expanded copyright/authors with myself where applicable
* 'Support_for_mysql_log_example' of https://github.com/arto-p/fail2ban:
Added testcase for MySQL date format to testcases/datedetectortestcase.py and example of MySQL log file.
Added support for MySQL logfiles
Conflicts:
testcases/datedetectortestcase.py -- conflictde with other added test cases
* 001-fail2ban-server-socket-close-on-exec-no-leak.diff
Add code that marks server and client sockets with FD_CLOEXEC flags.
Avoid leaking file descriptors to processes spawned when handling
fail2ban actions (ex: iptables).
Unix sockets managed by fail2ban-server don't need to be passed to any
child process. Fail2ban already uses the FD_CLOEXEC flags in the filter
code.
This patch also avoids giving iptables access to fail2ban UNIX socket in
a SELinux environment (A sane SELinux policy should trigger an audit
event because "iptables" will be given read/write access to the fail2ban
control socket).
Some random references related to this bug:
http://sourceforge.net/tracker/?func=detail&atid=689044&aid=2086568&group_id=121032http://www.redhat.com/archives/fedora-selinux-list/2009-June/msg00124.htmlhttp://forums.fedoraforum.org/showthread.php?t=234230
* 002-fail2ban-filters-close-on-exec-typo-fix.diff
There is a typo in the fail2ban server/filter.py source code. The
FD_CLOEXEC is correctly set but additional *random* flags are also set.
It has no side-effect as long as the fd doesn't match a valid flag :)
"fcntl.fcntl(fd, fcntl.F_SETFD, fd | fcntl.FD_CLOEXEC)" <== the 3rd
parameter should be flags, not a file descriptor.
* 003-fail2ban-gamin-socket-close-on-exec-no-leak.diff
Add code that marks the Gamin monitor file descriptor with FD_CLOEXEC
flags. Avoid leaking file descriptors to processes spawned when handling
fail2ban actions (ex: iptables).
---
File descriptors in action process before patches:
dr-x------ 2 root root 0 .
dr-xr-xr-x 8 root root 0 ..
lr-x------ 1 root root 64 0 -> /dev/null <== OK
l-wx------ 1 root root 64 1 -> /tmp/test.log <== used by test action
lrwx------ 1 root root 64 2 -> /dev/null <== OK
lrwx------ 1 root root 64 3 -> socket:[116361] <== NOK (fail2ban.sock leak)
lr-x------ 1 root root 64 4 -> /proc/20090/fd <== used by test action
l-wx------ 1 root root 64 5 -> /var/log/fail2ban.log <== OK
lrwx------ 1 root root 64 6 -> socket:[115608] <== NOK (gamin sock leak)
File descriptors in action process after patches:
dr-x------ 2 root root 0 .
dr-xr-xr-x 8 root root 0 ..
lr-x------ 1 root root 64 0 -> /dev/null <== OK
l-wx------ 1 root root 64 1 -> /tmp/test.log <== used by test action
lrwx------ 1 root root 64 2 -> /dev/null <== OK
lr-x------ 1 root root 64 3 -> /proc/18284/fd <== used by test action
l-wx------ 1 root root 64 5 -> /var/log/fail2ban.log <== OK
Now it should be evaluated only if logging mode is debug or "lower", and also
should happen not only on initial detections but for every added failure.
It also reports counts per each IP now
* 'master' of https://github.com/sebres/fail2ban:
invalid date recognition, irregular because of sorting template list, now via setRegex
invalid date recognition, irregular because of sorting template list (sometimes not reproducible by fail2ban-regex, cause will be not sorted)
* 'datedetector-testcase' of https://github.com/kwirk/fail2ban:
Correct datedetector testcases for TAI64N format
Additional date/time formats for datedetector testcases
Also made a note that some workarounds for certain date/time formats
may cause week/year day to be incorrect. This does not affect
functionality of fail2ban so no urgency to fix.
Yaroslav Halchenko