From ee786671aa5824d908275398ccdcd17cc0ad099d Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Sat, 15 Jun 2013 13:17:09 +1000 Subject: [PATCH 01/51] DOC: developing filters without DoS --- DEVELOP | 87 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 85 insertions(+), 2 deletions(-) diff --git a/DEVELOP b/DEVELOP index bebc12b3..474ddc81 100644 --- a/DEVELOP +++ b/DEVELOP @@ -41,8 +41,91 @@ Filters example.com/example.org used for DNS names * Ensure ./fail2ban-regex testcases/files/logs/{samplelog} config/filter.d/{filter}.conf has matches for EVERY regex -* Ensure regexs end with a $ and are restrictive as possible. E.g. not .* if - [0-9]+ is sufficient +* Ensure regexs start with a ^ and are restrictive as possible. E.g. not .* if + \d+ is sufficient +* Use the functionality of regexs http://docs.python.org/2/library/re.html +* Take a look at the source code of the application. You may see optional or + extra log messages, or parts there of, that need to form part of your regex. + +If you only have a basic knowledge of regular repressions read +http://docs.python.org/2/library/re.html first. + +Filter Security +--------------- + +Poor filter regular expressions are suseptable to DoS attacks. + +When a remote user has the ability to introduce text that will match the +filter regex such that the inserted text matches the part they have the +ability to deny any host they choose. + +So the part must be anchored on text generated by the application and not +the user. Ideally this should anchor to the beginning and end of the log line +however as more applications log at the beginning than the end, achoring the +beginning is more important. + +When creating a regex that extends back to the begining remember the date part +has been removed within fail2ban so theres no need to match that. If the format +is like ' error 1.2.3.4 is evil' then you will need to match the < at +the start so here the regex would start like '^<> is evil$'. + +Some applications log spaces at the end. If you're not sure add \s*$ as the +end part of the regex. + +Examples of poor filters +------------------------ + +1. Too restrictive + +We find a log message: + + Apr-07-13 07:08:36 Invalid command fial2ban from 1.2.3.4 + +We make a failregex + + ^Invalid command \S+ from + +Now think evil. The user does the command 'blah from 1.2.3.44' + +The program diliently logs: + + Apr-07-13 07:08:36 Invalid command blah from 1.2.3.44 from 1.2.3.4 + +And fail2ban matches 1.2.3.44 as the IP that it ban. A DoS attack was successful. + +The fix here is that the command can be anything so .* is approprate. + + ^Invalid command .* from + +Here the .* will match until the end of the string. Then realise it has more to +match, i.e. "from " and go back until it find this. Then it will ban +1.2.3.4 correctly. Since the is always at the end, end the regex witha $ + + ^Invalid command .* from $ + +Note if we'd just had the expression: + + ^Invalid command \S+ from $ + +Then provided the user put a space in their command they would have never been +banned. + +2. Applicaiton generates two identical log messages with different meanings + +If the application generates the following two messages under different +circmstances: + + client : authentication failed + client : authentication failed + + +Then its obvious that a regex of "^client : authentication +failed$" will still cause problems if the user can trigger the second +log message with a of 123.1.1.1. + +Here there's nothing to do except request/change the application so it logs +messages differently. + Code Testing ============ From fa7a105483e6653d45830ae0336c84daba6899fc Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Thu, 27 Jun 2013 09:16:14 +1000 Subject: [PATCH 02/51] ENH: filter.d/asterisk - consolidate log prefix regex and add a few fail messages --- ChangeLog | 2 ++ THANKS | 1 + config/filter.d/asterisk.conf | 29 ++++++++++++++++------------- testcases/files/logs/asterisk | 6 ++++++ 4 files changed, 25 insertions(+), 13 deletions(-) diff --git a/ChangeLog b/ChangeLog index a6bc4e21..44744a76 100644 --- a/ChangeLog +++ b/ChangeLog @@ -22,6 +22,8 @@ ver. 0.8.11 (2013/XX/XXX) - wanna-be-released Daniel Black & Georgiy Mernov * filter.d/exim.conf -- regex hardening and extra failure examples in sample logs + Daniel Black & Sebastian Arcus + * filter.d/asterisk -- more regexes Yaroslav Halchenko * fail2ban-regex -- refactored to provide more details (missing and ignored lines, control over logging, etc) while maintaining look&feel. diff --git a/THANKS b/THANKS index 47c3e999..c549d77f 100644 --- a/THANKS +++ b/THANKS @@ -40,6 +40,7 @@ Raphaël Marichez René Berber Robert Edeker Russell Odom +Sebastian Arcus Sireyessire silviogarbes Stephen Gildea diff --git a/config/filter.d/asterisk.conf b/config/filter.d/asterisk.conf index a8f65e09..6d351f8b 100644 --- a/config/filter.d/asterisk.conf +++ b/config/filter.d/asterisk.conf @@ -17,19 +17,22 @@ before = common.conf # Notes.: regex to match the password failures messages in the logfile. # Values: TEXT # -failregex = ^\[\]\s*NOTICE%(__pid_re)s \S+: Registration from '[^']*' failed for '(:\d+)?' - Wrong password$ - ^\[\]\s*NOTICE%(__pid_re)s \S+: Registration from '[^']*' failed for '(:\d+)?' - No matching peer found$ - ^\[\]\s*NOTICE%(__pid_re)s \S+: Registration from '[^']*' failed for '(:\d+)?' - Username/auth name mismatch$ - ^\[\]\s*NOTICE%(__pid_re)s \S+: Registration from '[^']*' failed for '(:\d+)?' - Device does not match ACL$ - ^\[\]\s*NOTICE%(__pid_re)s \S+: Registration from '[^']*' failed for '(:\d+)?' - Peer is not supposed to register$ - ^\[\]\s*NOTICE%(__pid_re)s \S+: Registration from '[^']*' failed for '(:\d+)?' - ACL error \(permit/deny\)$ - ^\[\]\s*NOTICE%(__pid_re)s \S+: Registration from '[^']*' failed for '(:\d+)?' - Not a local domain$ - ^\[\]\s*NOTICE%(__pid_re)s\[\S+\] \S+: Call from '[^']*' \(:\d+\) to extension '\d+' rejected because extension not found in context 'default'\.$ - ^\[\]\s*NOTICE%(__pid_re)s \S+: Host failed to authenticate as '[^']*'$ - ^\[\]\s*NOTICE%(__pid_re)s \S+: No registration for peer '[^']*' \(from \)$ - ^\[\]\s*NOTICE%(__pid_re)s \S+: Host failed MD5 authentication for '[^']*' \([^)]+\)$ - ^\[\]\s*NOTICE%(__pid_re)s \S+: Failed to authenticate user [^@]+@\S*$ - ^\[\]\s*SECURITY%(__pid_re)s \S+: SecurityEvent="InvalidAccountID",EventTV="[\d-]+",Severity="[\w]+",Service="[\w]+",EventVersion="\d+",AccountID="\d+",SessionID="0x[\da-f]+",LocalAddress="IPV[46]/(UD|TC)P/[\da-fA-F:.]+/\d+",RemoteAddress="IPV[46]/(UD|TC)P//\d+"$ +log_prefix= \[\]\s*(?:NOTICE|SECURITY)%(__pid_re)s:?(?:\[\S+\d*\])? \S+:\d* + +failregex = ^%(log_prefix)s Registration from '[^']*' failed for '(:\d+)?' - Wrong password$ + ^%(log_prefix)s Registration from '[^']*' failed for '(:\d+)?' - No matching peer found$ + ^%(log_prefix)s Registration from '[^']*' failed for '(:\d+)?' - Username/auth name mismatch$ + ^%(log_prefix)s Registration from '[^']*' failed for '(:\d+)?' - Device does not match ACL$ + ^%(log_prefix)s Registration from '[^']*' failed for '(:\d+)?' - Peer is not supposed to register$ + ^%(log_prefix)s Registration from '[^']*' failed for '(:\d+)?' - ACL error \(permit/deny\)$ + ^%(log_prefix)s Registration from '[^']*' failed for '(:\d+)?' - Not a local domain$ + ^%(log_prefix)s Call from '[^']*' \(:\d+\) to extension '\d+' rejected because extension not found in context 'default'\.$ + ^%(log_prefix)s Host failed to authenticate as '[^']*'$ + ^%(log_prefix)s No registration for peer '[^']*' \(from \)$ + ^%(log_prefix)s Host failed MD5 authentication for '[^']*' \([^)]+\)$ + ^%(log_prefix)s Failed to authenticate user [^@]+@\S*$ + ^%(log_prefix)s (?:handle_request_subscribe: )?Sending fake auth rejection for (device|user) \d*>;tag=\w+\S* + ^%(log_prefix)s SecurityEvent="(FailedACL|InvalidAccountID|ChallengeResponseFailed|InvalidPassword)",EventTV="[\d-]+",Severity="[\w]+",Service="[\w]+",EventVersion="\d+",AccountID="\d+",SessionID="0x[\da-f]+",LocalAddress="IPV[46]/(UD|TC)P/[\da-fA-F:.]+/\d+",RemoteAddress="IPV[46]/(UD|TC)P//\d+"$ # Option: ignoreregex # Notes.: regex to ignore. If this regex matches, the line is ignored. diff --git a/testcases/files/logs/asterisk b/testcases/files/logs/asterisk index 45b69304..e2fe3b6e 100644 --- a/testcases/files/logs/asterisk +++ b/testcases/files/logs/asterisk @@ -13,3 +13,9 @@ [2013-03-26 15:47:54] NOTICE[1237] chan_sip.c: Registration from '"100"sip:100@1.2.3.4' failed for '1.2.3.4:23930' - No matching peer found [2013-05-13 07:10:53] SECURITY[1204] res_security_log.c: SecurityEvent="InvalidAccountID",EventTV="1368439853-500975",Severity="Error",Service="SIP",EventVersion="1",AccountID="00972599580679",SessionID="0x7f8ecc0421f8",LocalAddress="IPV4/UDP/1.2.3.4/5060",RemoteAddress="IPV4/UDP/1.2.3.4/5070" [2013-06-10 18:15:03] NOTICE[2723] chan_sip.c: Registration from '"100"' failed for '1.2.3.4' - Not a local domain +# http://forum.4psa.com/showthread.php?t=6601 +[2009-12-22 16:35:24] NOTICE[6163] chan_sip.c: Sending fake auth rejection for device ;tag=e3793a95e1acbc69o +# http://www.freepbx.org/forum/general-help/fake-auth-rejection +[2009-12-22 16:35:24] NOTICE[1570][C-00000086] chan_sip.c: Sending fake auth rejection for device 1022;tag=5d8b6f92 +# http://www.spinics.net/lists/asterisk/msg127381.html +[2009-12-22 16:35:24] NOTICE[14916]: chan_sip.c:15644 handle_request_subscribe: Sending fake auth rejection for user ;tag=6pwd6erg54 From 1b170b2aef828899ad6596ada0c379c6e07a620e Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Fri, 28 Jun 2013 09:34:12 -0400 Subject: [PATCH 03/51] BF: support apache 2.4 more detailed error log format. Close #268 --- ChangeLog | 2 ++ config/filter.d/apache-common.conf | 6 +++++- testcases/files/logs/apache-nohome | 4 ++++ 3 files changed, 11 insertions(+), 1 deletion(-) create mode 100644 testcases/files/logs/apache-nohome diff --git a/ChangeLog b/ChangeLog index 45c6f343..e5c38eb1 100644 --- a/ChangeLog +++ b/ChangeLog @@ -13,6 +13,8 @@ ver. 0.8.11 (2013/XX/XXX) - wanna-be-released - Fixes: Yaroslav Halchenko * filter.d/common.conf -- make colon after [daemon] optional. Closes gh-267 + * filter.d/apache-common.conf -- support apache 2.4 more detailed error + log format. Closes gh-268 - New Features: Daniel Black & ykimon * filter.d/3proxy.conf -- filter added diff --git a/config/filter.d/apache-common.conf b/config/filter.d/apache-common.conf index c3829e2f..cc35ae5f 100644 --- a/config/filter.d/apache-common.conf +++ b/config/filter.d/apache-common.conf @@ -14,4 +14,8 @@ after = apache-common.local [DEFAULT] # Common prefix for [error] apache messages which also would include -_apache_error_client = \[[^]]+\] \[error\] \[client \] +# Depending on the version it could be +# 2.2: [Sat Jun 01 11:23:08 2013] [error] [client 1.2.3.4] +# 2.4: [Thu Jun 27 11:55:44.569531 2013] [core:info] [pid 4101:tid 2992634688] [client 1.2.3.4:46652] +# Reference: https://github.com/fail2ban/fail2ban/issues/268 +_apache_error_client = \[[^]]+\] \[(error|core:\S+)\]( \[pid \d+:\S+ \d+\])? \[client (:\d{1,5})?\]( \S+:)? diff --git a/testcases/files/logs/apache-nohome b/testcases/files/logs/apache-nohome new file mode 100644 index 00000000..e8ef56bc --- /dev/null +++ b/testcases/files/logs/apache-nohome @@ -0,0 +1,4 @@ +# Apache 2.2 +[Sat Jun 01 11:23:08 2013] [error] [client 1.2.3.4] File does not exist: /xxx/~ +# Apache 2.4 +[Thu Jun 27 11:55:44.569531 2013] [core:info] [pid 4101:tid 2992634688] [client 192.0.2.12:46652] AH00128: File does not exist: /xxx/~ From 0086a7edabaffc0524ed8dd02e679c9cdcd577fb Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Sat, 29 Jun 2013 11:30:37 +1000 Subject: [PATCH 04/51] ENH: missed a $ --- config/filter.d/asterisk.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/filter.d/asterisk.conf b/config/filter.d/asterisk.conf index 6d351f8b..5e775430 100644 --- a/config/filter.d/asterisk.conf +++ b/config/filter.d/asterisk.conf @@ -31,7 +31,7 @@ failregex = ^%(log_prefix)s Registration from '[^']*' failed for '(:\d+)?' ^%(log_prefix)s No registration for peer '[^']*' \(from \)$ ^%(log_prefix)s Host failed MD5 authentication for '[^']*' \([^)]+\)$ ^%(log_prefix)s Failed to authenticate user [^@]+@\S*$ - ^%(log_prefix)s (?:handle_request_subscribe: )?Sending fake auth rejection for (device|user) \d*>;tag=\w+\S* + ^%(log_prefix)s (?:handle_request_subscribe: )?Sending fake auth rejection for (device|user) \d*>;tag=\w+\S*$ ^%(log_prefix)s SecurityEvent="(FailedACL|InvalidAccountID|ChallengeResponseFailed|InvalidPassword)",EventTV="[\d-]+",Severity="[\w]+",Service="[\w]+",EventVersion="\d+",AccountID="\d+",SessionID="0x[\da-f]+",LocalAddress="IPV[46]/(UD|TC)P/[\da-fA-F:.]+/\d+",RemoteAddress="IPV[46]/(UD|TC)P//\d+"$ # Option: ignoreregex From c2696fe641c92002b1535bb5ea76d47d442547a3 Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Sun, 30 Jun 2013 15:03:13 +1000 Subject: [PATCH 05/51] DOC: enhance development doc to show how CVE-2013-2178 was done --- DEVELOP | 45 +++++++++++++++++++++++++++++++++++++-------- 1 file changed, 37 insertions(+), 8 deletions(-) diff --git a/DEVELOP b/DEVELOP index 474ddc81..2731fd13 100644 --- a/DEVELOP +++ b/DEVELOP @@ -56,13 +56,21 @@ Filter Security Poor filter regular expressions are suseptable to DoS attacks. When a remote user has the ability to introduce text that will match the -filter regex such that the inserted text matches the part they have the +filter regex, such that the inserted text matches the part, they have the ability to deny any host they choose. -So the part must be anchored on text generated by the application and not -the user. Ideally this should anchor to the beginning and end of the log line -however as more applications log at the beginning than the end, achoring the -beginning is more important. +So the part must be anchored on text generated by the application, and not +the user, to a sufficient extent that the user cannot insert the entire text. + +Filters are matched against the log line with their date removed. + +Ideally filter regex should anchor to the beginning and end of the log line +however as more applications log at the beginning than the end, achoring the +beginning is more important. If the log file used by the application is shared +with other applications, like system logs, ensure the other application that +use that log file do not log user generated text at the beginning of the line, +or, if they do, ensure the regexs of the filter are sufficient to mitigate the +risk of insertion. When creating a regex that extends back to the begining remember the date part has been removed within fail2ban so theres no need to match that. If the format @@ -99,7 +107,7 @@ The fix here is that the command can be anything so .* is approprate. Here the .* will match until the end of the string. Then realise it has more to match, i.e. "from " and go back until it find this. Then it will ban -1.2.3.4 correctly. Since the is always at the end, end the regex witha $ +1.2.3.4 correctly. Since the is always at the end, end the regex with a $. ^Invalid command .* from $ @@ -110,7 +118,28 @@ Note if we'd just had the expression: Then provided the user put a space in their command they would have never been banned. -2. Applicaiton generates two identical log messages with different meanings +2. Filter regex can match other user injected data + +From the apache vulnerability CVE-2013-2178 +( original ref: https://vndh.net/note:fail2ban-089-denial-service ). + +An example bad regex for apache: + + failregex = [[]client []] user .* not found + +Since the user can do a get request on: + + GET /[client%20192.168.0.1]%20user%20root%20not%20found HTTP/1.0 +Host: remote.site + +Now the log line will be: + + [Sat Jun 01 02:17:42 2013] [error] [client 192.168.33.1] File does not exist: /srv/http/site/[client 192.168.0.1] user root not found + +As this log line doesn't match other expressions hence it matches the above +regex and blocks 192.168.33.1 as a denial of service from the HTTP requester. + +3. Applicaiton generates two identical log messages with different meanings If the application generates the following two messages under different circmstances: @@ -119,7 +148,7 @@ circmstances: client : authentication failed -Then its obvious that a regex of "^client : authentication +Then it's obvious that a regex of "^client : authentication failed$" will still cause problems if the user can trigger the second log message with a of 123.1.1.1. From 3b76fc79f9a6269aca915fc33ea15fcef1709ffb Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Mon, 1 Jul 2013 21:12:51 +1000 Subject: [PATCH 06/51] BF: fix dovecot filter for when no TLS is enabled on pop/imap --- ChangeLog | 2 ++ THANKS | 1 + config/filter.d/dovecot.conf | 2 +- testcases/files/logs/dovecot | 1 + 4 files changed, 5 insertions(+), 1 deletion(-) diff --git a/ChangeLog b/ChangeLog index 45c6f343..c11215a7 100644 --- a/ChangeLog +++ b/ChangeLog @@ -13,6 +13,8 @@ ver. 0.8.11 (2013/XX/XXX) - wanna-be-released - Fixes: Yaroslav Halchenko * filter.d/common.conf -- make colon after [daemon] optional. Closes gh-267 + Daniel Black & Мернов Георгий + * filter.d/dovecot.conf -- Fix when no TLS enabled - line doesn't end in , - New Features: Daniel Black & ykimon * filter.d/3proxy.conf -- filter added diff --git a/THANKS b/THANKS index f8b18daa..eac3c6f0 100644 --- a/THANKS +++ b/THANKS @@ -33,6 +33,7 @@ Mark Edgington Markus Hoffmann Marvin Rouge mEDI +Мернов Георгий Michael C. Haller Michael Hanselmann NickMunger diff --git a/config/filter.d/dovecot.conf b/config/filter.d/dovecot.conf index dd4c35ba..e3702fcf 100644 --- a/config/filter.d/dovecot.conf +++ b/config/filter.d/dovecot.conf @@ -17,7 +17,7 @@ _daemon = dovecot(-auth)? # Values: TEXT # failregex = ^%(__prefix_line)s(pam_unix(?:\(\S+\))?:)?\s+authentication failure; logname=\S* uid=\S* euid=\S* tty=dovecot ruser=\S* rhost=(\s+user=\S*)?\s*$ - ^%(__prefix_line)s(pop3|imap)-login: (Info: )?(Aborted login|Disconnected)(: Inactivity)? \((no auth attempts|auth failed, \d+ attempts|tried to use disabled \S+ auth)\):( user=<\S+>,)?( method=\S+,)? rip=, lip=(\d{1,3}\.){3}\d{1,3},( TLS( handshaking)?(: Disconnected)?)?\s*$ + ^%(__prefix_line)s(pop3|imap)-login: (Info: )?(Aborted login|Disconnected)(: Inactivity)? \((no auth attempts|auth failed, \d+ attempts|tried to use disabled \S+ auth)\):( user=<\S+>,)?( method=\S+,)? rip=, lip=(?:\d{1,3}\.){3}\d{1,3}(, TLS( handshaking)?(: Disconnected)?)?\s*$ # Option: ignoreregex # Notes.: regex to ignore. If this regex matches, the line is ignored. diff --git a/testcases/files/logs/dovecot b/testcases/files/logs/dovecot index f904a8fe..01df0af3 100644 --- a/testcases/files/logs/dovecot +++ b/testcases/files/logs/dovecot @@ -12,4 +12,5 @@ Jun 14 00:48:21 platypus dovecot: imap-login: Disconnected (auth failed, 1 attem Jun 13 20:48:11 platypus dovecot: pop3-login: Disconnected (no auth attempts): rip=121.44.24.254, lip=113.212.99.194, TLS: Disconnected Jun 13 21:48:06 platypus dovecot: pop3-login: Disconnected: Inactivity (no auth attempts): rip=180.200.180.81, lip=113.212.99.194, TLS Jun 13 20:20:21 platypus dovecot: imap-login: Disconnected (no auth attempts): rip=180.189.168.166, lip=113.212.99.194, TLS handshaking: Disconnected +Jun 23 00:52:43 vhost1-ua dovecot: pop3-login: Disconnected: Inactivity (auth failed, 1 attempts): user=, method=PLAIN, rip=193.95.245.163, lip=176.214.13.210 From 72f9e6a51ecf12abb92d6d14349f3558c5e60c7c Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Mon, 1 Jul 2013 21:50:35 +1000 Subject: [PATCH 07/51] ENH/TST: more samples and rejection types for sender verify fail and rejected RCPT --- ChangeLog | 2 +- config/filter.d/exim.conf | 4 ++-- testcases/files/logs/exim | 6 ++++++ 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/ChangeLog b/ChangeLog index c11215a7..549eb7f7 100644 --- a/ChangeLog +++ b/ChangeLog @@ -22,7 +22,7 @@ ver. 0.8.11 (2013/XX/XXX) - wanna-be-released Daniel Black * filter.d/{asterisk,assp,dovecot,proftpd}.conf -- regex hardening and extra failure examples in sample logs - Daniel Black & Georgiy Mernov & ftoppi + Daniel Black & Georgiy Mernov & ftoppi & Мернов Георгий * filter.d/exim.conf -- regex hardening and extra failure examples in sample logs Yaroslav Halchenko diff --git a/config/filter.d/exim.conf b/config/filter.d/exim.conf index 252da07a..fc744dd3 100644 --- a/config/filter.d/exim.conf +++ b/config/filter.d/exim.conf @@ -18,10 +18,10 @@ host_info = H=([\w.-]+ )?(\(\S+\) )?\[\](:\d+)? (?:I=\[\S+\]:\d+ )?(?:U=\S+ )?(P=e?smtp )? pid = ( \[\d+\])? -failregex = ^%(pid)s %(host_info)ssender verify fail for <\S+>: Unrouteable address\s*$ +failregex = ^%(pid)s %(host_info)ssender verify fail for <\S+>: (?:Unknown user|Unrouteable address|all relevant MX records point to non-existent hosts)\s*$ ^%(pid)s \S+ F=(?:<>|\S+@\S+) %(host_info)s(?:temporarily )?rejected by local_scan\(\): .{0,256}$ ^%(pid)s login authenticator failed for (\S+ )?\(\S+\) \[\]: 535 Incorrect authentication data( \(set_id=.*\)|: \d+ Time\(s\))?\s*$ - ^%(pid)s %(host_info)sF=(?:<>|[^@]+@\S+) rejected RCPT [^@]+@\S+: (rejected found in dnsbl \S+|relay not permitted)\s*$ + ^%(pid)s %(host_info)sF=(?:<>|[^@]+@\S+) rejected RCPT [^@]+@\S+: (rejected found in dnsbl \S+|relay not permitted|Sender verify failed|Unknown user)\s*$ ^%(pid)s \S+ %(host_info)sF=(?:<>|[^@]+@\S+) rejected after DATA: This message contains a virus \(\S+\)\.\s*$ ^%(pid)s SMTP protocol synchronization error \(.*\): rejected (connection from|"\S+") %(host_info)s(next )?input=".*"\s*$ ^%(pid)s SMTP call from \S+ \[\](:\d+)? (I=\[\S+\]:\d+ )?dropped: too many nonmail commands \(last was "\S+"\)\s*$ diff --git a/testcases/files/logs/exim b/testcases/files/logs/exim index ebcb9726..792f949e 100644 --- a/testcases/files/logs/exim +++ b/testcases/files/logs/exim @@ -18,3 +18,9 @@ 2013-06-02 06:54:20 [13314] SMTP protocol synchronization error (input sent without waiting for greeting): rejected connection from H=[211.148.195.192]:25936 I=[1.2.3.4]:25 input="GET / HTTP/1.1\r\n\r\n" 2013-06-02 09:05:48 [18505] SMTP protocol synchronization error (next input sent too soon: pipelining was not advertised): rejected "RSET" H=ba77.mx83.fr [82.96.160.77]:58302 I=[1.2.3.4]:25 next input="QUIT\r\n" 2013-06-02 09:22:05 [19591] SMTP call from pc012-6201.spo.scu.edu.tw [163.14.21.161]:3767 I=[1.2.3.4]:25 dropped: too many nonmail commands (last was "RSET") +2013-06-02 15:06:18 H=(VM-WIN2K3-1562) [46.20.35.114] sender verify fail for : Unknown user +2013-06-07 02:02:09 H=treeladders.kiev.ua [91.232.21.92] sender verify fail for : all relevant MX records point to non-existent hosts +2013-06-15 16:34:55 H=mx.tillions.com [182.18.24.93] F= rejected RCPT : Sender verify failed +2013-06-15 16:36:49 H=altmx.marsukov.com [111.67.203.116] F= rejected RCPT : Unknown user +#2013-06-16 02:50:43 H=dbs.marsukov.com [111.67.203.114] F= rejected RCPT : rejected because 111.67.203.114 is in a black list at dnsbl.sorbs.net\nCurrently Sending Spam See: http://www.sorbs.net/lookup.shtml?111.67.203.114 + From 9757e1df2b84bd40a42d52eb4ede6890092998a9 Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Mon, 1 Jul 2013 21:53:05 +1000 Subject: [PATCH 08/51] ENH: make groupings non-capturing --- config/filter.d/exim.conf | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/config/filter.d/exim.conf b/config/filter.d/exim.conf index fc744dd3..9e8491b7 100644 --- a/config/filter.d/exim.conf +++ b/config/filter.d/exim.conf @@ -20,11 +20,11 @@ pid = ( \[\d+\])? failregex = ^%(pid)s %(host_info)ssender verify fail for <\S+>: (?:Unknown user|Unrouteable address|all relevant MX records point to non-existent hosts)\s*$ ^%(pid)s \S+ F=(?:<>|\S+@\S+) %(host_info)s(?:temporarily )?rejected by local_scan\(\): .{0,256}$ - ^%(pid)s login authenticator failed for (\S+ )?\(\S+\) \[\]: 535 Incorrect authentication data( \(set_id=.*\)|: \d+ Time\(s\))?\s*$ - ^%(pid)s %(host_info)sF=(?:<>|[^@]+@\S+) rejected RCPT [^@]+@\S+: (rejected found in dnsbl \S+|relay not permitted|Sender verify failed|Unknown user)\s*$ + ^%(pid)s login authenticator failed for (?:\S+ )?\(\S+\) \[\]: 535 Incorrect authentication data(?: \(set_id=.*\)|: \d+ Time\(s\))?\s*$ + ^%(pid)s %(host_info)sF=(?:<>|[^@]+@\S+) rejected RCPT [^@]+@\S+: (?:rejected found in dnsbl \S+|relay not permitted|Sender verify failed|Unknown user)\s*$ ^%(pid)s \S+ %(host_info)sF=(?:<>|[^@]+@\S+) rejected after DATA: This message contains a virus \(\S+\)\.\s*$ - ^%(pid)s SMTP protocol synchronization error \(.*\): rejected (connection from|"\S+") %(host_info)s(next )?input=".*"\s*$ - ^%(pid)s SMTP call from \S+ \[\](:\d+)? (I=\[\S+\]:\d+ )?dropped: too many nonmail commands \(last was "\S+"\)\s*$ + ^%(pid)s SMTP protocol synchronization error \(.*\): rejected (?:connection from|"\S+") %(host_info)s(?:next )?input=".*"\s*$ + ^%(pid)s SMTP call from \S+ \[\](:\d+)? (?:I=\[\S+\]:\d+ )?dropped: too many nonmail commands \(last was "\S+"\)\s*$ # Option: ignoreregex # Notes.: regex to ignore. If this regex matches, the line is ignored. From ca996ace5e9bed218223da7fd96eff6606e64c04 Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Mon, 1 Jul 2013 21:56:02 +1000 Subject: [PATCH 09/51] ENH: remove temporary failures from local_scan in line with comments in gh-258 --- config/filter.d/exim.conf | 2 +- testcases/files/logs/exim | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/config/filter.d/exim.conf b/config/filter.d/exim.conf index 9e8491b7..49a7d2c4 100644 --- a/config/filter.d/exim.conf +++ b/config/filter.d/exim.conf @@ -19,7 +19,7 @@ host_info = H=([\w.-]+ )?(\(\S+\) )?\[\](:\d+)? (?:I=\[\S+\]:\d+ )?(?:U=\S pid = ( \[\d+\])? failregex = ^%(pid)s %(host_info)ssender verify fail for <\S+>: (?:Unknown user|Unrouteable address|all relevant MX records point to non-existent hosts)\s*$ - ^%(pid)s \S+ F=(?:<>|\S+@\S+) %(host_info)s(?:temporarily )?rejected by local_scan\(\): .{0,256}$ + ^%(pid)s \S+ F=(?:<>|\S+@\S+) %(host_info)srejected by local_scan\(\): .{0,256}$ ^%(pid)s login authenticator failed for (?:\S+ )?\(\S+\) \[\]: 535 Incorrect authentication data(?: \(set_id=.*\)|: \d+ Time\(s\))?\s*$ ^%(pid)s %(host_info)sF=(?:<>|[^@]+@\S+) rejected RCPT [^@]+@\S+: (?:rejected found in dnsbl \S+|relay not permitted|Sender verify failed|Unknown user)\s*$ ^%(pid)s \S+ %(host_info)sF=(?:<>|[^@]+@\S+) rejected after DATA: This message contains a virus \(\S+\)\.\s*$ diff --git a/testcases/files/logs/exim b/testcases/files/logs/exim index 792f949e..2a59ecdc 100644 --- a/testcases/files/logs/exim +++ b/testcases/files/logs/exim @@ -5,9 +5,9 @@ 2013-06-12 13:18:11 login authenticator failed for (USER-KVI9FGS9KP) [101.66.165.86]: 535 Incorrect authentication data 2013-06-10 10:10:59 H=ufficioestampa.it (srv.ufficioestampa.it) [193.169.56.211] sender verify fail for : Unrouteable address # http://forum.lissyara.su/viewtopic.php?f=20&t=2985 -2010-11-24 21:48:41 1PLKOW-00046U-EW F=wvhluo@droolindog.com H=93-143-146-237.adsl.net.t-com.hr (droolindog.com) [93.143.146.237] I=[10.10.10.32]:25 P=esmtp temporarily rejected by local_scan(): Temporary local problem +#2010-11-24 21:48:41 1PLKOW-00046U-EW F=wvhluo@droolindog.com H=93-143-146-237.adsl.net.t-com.hr (droolindog.com) [93.143.146.237] I=[10.10.10.32]:25 P=esmtp temporarily rejected by local_scan(): Temporary local problem # http://us.generation-nt.com/answer/exim-spamassassin-2010-0-x64-help-204020461.html -2011-07-07 15:44:16 1QexIu-0006dj-PX F=XXXXXX@XXXXXXXXXXXX H=localhost (saf.bio.caltech.edu) [127.0.0.1] P=esmtp temporarily rejected by local_scan(): Local configuration error - local_scan() library failure/usr/lib/exim/sa-exim.so: cannot open shared object file: No such file or directory +#2011-07-07 15:44:16 1QexIu-0006dj-PX F=XXXXXX@XXXXXXXXXXXX H=localhost (saf.bio.caltech.edu) [127.0.0.1] P=esmtp temporarily rejected by local_scan(): Local configuration error - local_scan() library failure/usr/lib/exim/sa-exim.so: cannot open shared object file: No such file or directory # http://www.clues.ltd.uk/howto/debian-sa-fprot-HOWTO.html 2004-01-18 07:15:35 1Ai79e-0000Dq-8i F=uzwltcmwto24@melissacam.biz H=lsanca1-ar3-4-47-028-040.lsanca1.elnk.dsl.genuity.net [4.47.28.40] P=smtp rejected by local_scan(): Rejected: hits=7.5 required=5.0 trigger=5.0 # https://github.com/fail2ban/fail2ban/pull/251#issuecomment-19493875 From c7d64c3c7f9dee7b300f97e6c6f444b7ca6e8c10 Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Mon, 1 Jul 2013 21:58:03 +1000 Subject: [PATCH 10/51] TST: url reference fix --- testcases/files/logs/exim | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/testcases/files/logs/exim b/testcases/files/logs/exim index 2a59ecdc..96b901ab 100644 --- a/testcases/files/logs/exim +++ b/testcases/files/logs/exim @@ -4,7 +4,7 @@ 2013-06-12 03:57:58 login authenticator failed for (ylmf-pc) [120.196.140.45]: 535 Incorrect authentication data: 1 Time(s) 2013-06-12 13:18:11 login authenticator failed for (USER-KVI9FGS9KP) [101.66.165.86]: 535 Incorrect authentication data 2013-06-10 10:10:59 H=ufficioestampa.it (srv.ufficioestampa.it) [193.169.56.211] sender verify fail for : Unrouteable address -# http://forum.lissyara.su/viewtopic.php?f=20&t=2985 +# http://forum.lissyara.su/viewtopic.php?f=20&t=29857 #2010-11-24 21:48:41 1PLKOW-00046U-EW F=wvhluo@droolindog.com H=93-143-146-237.adsl.net.t-com.hr (droolindog.com) [93.143.146.237] I=[10.10.10.32]:25 P=esmtp temporarily rejected by local_scan(): Temporary local problem # http://us.generation-nt.com/answer/exim-spamassassin-2010-0-x64-help-204020461.html #2011-07-07 15:44:16 1QexIu-0006dj-PX F=XXXXXX@XXXXXXXXXXXX H=localhost (saf.bio.caltech.edu) [127.0.0.1] P=esmtp temporarily rejected by local_scan(): Local configuration error - local_scan() library failure/usr/lib/exim/sa-exim.so: cannot open shared object file: No such file or directory From 4777cfd4e7d45c13cf6f7fcfcf2aadaed323caa6 Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Tue, 2 Jul 2013 20:03:16 +1000 Subject: [PATCH 11/51] ENH: split out exim-spam into speparate filter --- config/filter.d/exim-common.conf | 17 +++++++++++++++++ config/filter.d/exim-spam.conf | 29 +++++++++++++++++++++++++++++ config/filter.d/exim.conf | 31 +++++++++++++++---------------- testcases/files/logs/exim | 6 +++--- 4 files changed, 64 insertions(+), 19 deletions(-) create mode 100644 config/filter.d/exim-common.conf create mode 100644 config/filter.d/exim-spam.conf diff --git a/config/filter.d/exim-common.conf b/config/filter.d/exim-common.conf new file mode 100644 index 00000000..79d6ffbb --- /dev/null +++ b/config/filter.d/exim-common.conf @@ -0,0 +1,17 @@ +# Fail2Ban configuration file for exim +# +# Author: Daniel Black +# + +[INCLUDES] + +# Load customizations if any available +# +after = exim-common.local + +[Definition] + +# From exim source code: ./src/receive.c:add_host_info_for_log +host_info = H=([\w.-]+ )?(\(\S+\) )?\[\](:\d+)? (I=\[\S+\]:\d+ )?(U=\S+ )?(P=e?smtp )? +pid = ( \[\d+\])? + diff --git a/config/filter.d/exim-spam.conf b/config/filter.d/exim-spam.conf new file mode 100644 index 00000000..55a6f5dd --- /dev/null +++ b/config/filter.d/exim-spam.conf @@ -0,0 +1,29 @@ +# Fail2Ban configuration file +# +# Author: Cyril Jaquier +# Daniel Black (rewrote with strong regexs) +# + + +[INCLUDES] + +# Read common prefixes. If any customizations available -- read them from +# exim-common.local +before = exim-common.conf + + +[Definition] + +# Option: failregex +# Notes.: This includes the spam rejection messages of exim. +# Note the %(host_info) defination contains a match + +failregex = ^%(pid)s \S+ F=(<>|\S+@\S+) %(host_info)srejected by local_scan\(\): .{0,256}$ + ^%(pid)s %(host_info)sF=(<>|[^@]+@\S+) rejected RCPT [^@]+@\S+: .*dnsbl.*\s*$ + ^%(pid)s \S+ %(host_info)sF=(<>|[^@]+@\S+) rejected after DATA: This message contains a virus \(\S+\)\.\s*$ + +# Option: ignoreregex +# Notes.: regex to ignore. If this regex matches, the line is ignored. +# Values: TEXT +# +ignoreregex = diff --git a/config/filter.d/exim.conf b/config/filter.d/exim.conf index 49a7d2c4..63b0fa1d 100644 --- a/config/filter.d/exim.conf +++ b/config/filter.d/exim.conf @@ -4,27 +4,26 @@ # Daniel Black (rewrote with strong regexs) # + +[INCLUDES] + +# Read common prefixes. If any customizations available -- read them from +# exim-common.local +before = exim-common.conf + + [Definition] # Option: failregex -# Notes.: regex to match the password failures messages in the logfile. The -# host must be matched by a group named "host". The tag "" can -# be used for standard IP/hostname matching and is only an alias for -# (?:::f{4,6}:)?(?P[\w\-.^_]+) -# Values: TEXT -# - -# From exim source code: ./src/receive.c:add_host_info_for_log -host_info = H=([\w.-]+ )?(\(\S+\) )?\[\](:\d+)? (?:I=\[\S+\]:\d+ )?(?:U=\S+ )?(P=e?smtp )? -pid = ( \[\d+\])? +# Notes.: This includes the rejection messages of exim. For spam and filter +# related bans use the exim-spam.conf +# Note the %(host_info) defination contains a match failregex = ^%(pid)s %(host_info)ssender verify fail for <\S+>: (?:Unknown user|Unrouteable address|all relevant MX records point to non-existent hosts)\s*$ - ^%(pid)s \S+ F=(?:<>|\S+@\S+) %(host_info)srejected by local_scan\(\): .{0,256}$ - ^%(pid)s login authenticator failed for (?:\S+ )?\(\S+\) \[\]: 535 Incorrect authentication data(?: \(set_id=.*\)|: \d+ Time\(s\))?\s*$ - ^%(pid)s %(host_info)sF=(?:<>|[^@]+@\S+) rejected RCPT [^@]+@\S+: (?:rejected found in dnsbl \S+|relay not permitted|Sender verify failed|Unknown user)\s*$ - ^%(pid)s \S+ %(host_info)sF=(?:<>|[^@]+@\S+) rejected after DATA: This message contains a virus \(\S+\)\.\s*$ - ^%(pid)s SMTP protocol synchronization error \(.*\): rejected (?:connection from|"\S+") %(host_info)s(?:next )?input=".*"\s*$ - ^%(pid)s SMTP call from \S+ \[\](:\d+)? (?:I=\[\S+\]:\d+ )?dropped: too many nonmail commands \(last was "\S+"\)\s*$ + ^%(pid)s login authenticator failed for (\S+ )?\(\S+\) \[\]: 535 Incorrect authentication data( \(set_id=.*\)|: \d+ Time\(s\))?\s*$ + ^%(pid)s %(host_info)sF=(<>|[^@]+@\S+) rejected RCPT [^@]+@\S+: (relay not permitted|Sender verify failed|Unknown user)\s*$ + ^%(pid)s SMTP protocol synchronization error \(.*\): rejected (connection from|"\S+") %(host_info)s(next )?input=".*"\s*$ + ^%(pid)s SMTP call from \S+ \[\](:\d+)? (I=\[\S+\]:\d+ )?dropped: too many nonmail commands \(last was "\S+"\)\s*$ # Option: ignoreregex # Notes.: regex to ignore. If this regex matches, the line is ignored. diff --git a/testcases/files/logs/exim b/testcases/files/logs/exim index 96b901ab..95ca621f 100644 --- a/testcases/files/logs/exim +++ b/testcases/files/logs/exim @@ -5,9 +5,9 @@ 2013-06-12 13:18:11 login authenticator failed for (USER-KVI9FGS9KP) [101.66.165.86]: 535 Incorrect authentication data 2013-06-10 10:10:59 H=ufficioestampa.it (srv.ufficioestampa.it) [193.169.56.211] sender verify fail for : Unrouteable address # http://forum.lissyara.su/viewtopic.php?f=20&t=29857 -#2010-11-24 21:48:41 1PLKOW-00046U-EW F=wvhluo@droolindog.com H=93-143-146-237.adsl.net.t-com.hr (droolindog.com) [93.143.146.237] I=[10.10.10.32]:25 P=esmtp temporarily rejected by local_scan(): Temporary local problem +# 2010-11-24 21:48:41 1PLKOW-00046U-EW F=wvhluo@droolindog.com H=93-143-146-237.adsl.net.t-com.hr (droolindog.com) [93.143.146.237] I=[10.10.10.32]:25 P=esmtp temporarily rejected by local_scan(): Temporary local problem # http://us.generation-nt.com/answer/exim-spamassassin-2010-0-x64-help-204020461.html -#2011-07-07 15:44:16 1QexIu-0006dj-PX F=XXXXXX@XXXXXXXXXXXX H=localhost (saf.bio.caltech.edu) [127.0.0.1] P=esmtp temporarily rejected by local_scan(): Local configuration error - local_scan() library failure/usr/lib/exim/sa-exim.so: cannot open shared object file: No such file or directory +# 2011-07-07 15:44:16 1QexIu-0006dj-PX F=XXXXXX@XXXXXXXXXXXX H=localhost (saf.bio.caltech.edu) [127.0.0.1] P=esmtp temporarily rejected by local_scan(): Local configuration error - local_scan() library failure/usr/lib/exim/sa-exim.so: cannot open shared object file: No such file or directory # http://www.clues.ltd.uk/howto/debian-sa-fprot-HOWTO.html 2004-01-18 07:15:35 1Ai79e-0000Dq-8i F=uzwltcmwto24@melissacam.biz H=lsanca1-ar3-4-47-028-040.lsanca1.elnk.dsl.genuity.net [4.47.28.40] P=smtp rejected by local_scan(): Rejected: hits=7.5 required=5.0 trigger=5.0 # https://github.com/fail2ban/fail2ban/pull/251#issuecomment-19493875 @@ -22,5 +22,5 @@ 2013-06-07 02:02:09 H=treeladders.kiev.ua [91.232.21.92] sender verify fail for : all relevant MX records point to non-existent hosts 2013-06-15 16:34:55 H=mx.tillions.com [182.18.24.93] F= rejected RCPT : Sender verify failed 2013-06-15 16:36:49 H=altmx.marsukov.com [111.67.203.116] F= rejected RCPT : Unknown user -#2013-06-16 02:50:43 H=dbs.marsukov.com [111.67.203.114] F= rejected RCPT : rejected because 111.67.203.114 is in a black list at dnsbl.sorbs.net\nCurrently Sending Spam See: http://www.sorbs.net/lookup.shtml?111.67.203.114 +2013-06-16 02:50:43 H=dbs.marsukov.com [111.67.203.114] F= rejected RCPT : rejected because 111.67.203.114 is in a black list at dnsbl.sorbs.net\nCurrently Sending Spam See: http://www.sorbs.net/lookup.shtml?111.67.203.114 From aebd24ec5485dacc5146728fa22387340b51099e Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Tue, 2 Jul 2013 20:09:27 +1000 Subject: [PATCH 12/51] BF: replace with ed so its cross platform, fixes permission problem gh-266, and Yaroslav doesn't revert to perl --- ChangeLog | 2 ++ config/action.d/hostsdeny.conf | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/ChangeLog b/ChangeLog index e5c38eb1..51835de4 100644 --- a/ChangeLog +++ b/ChangeLog @@ -15,6 +15,8 @@ ver. 0.8.11 (2013/XX/XXX) - wanna-be-released * filter.d/common.conf -- make colon after [daemon] optional. Closes gh-267 * filter.d/apache-common.conf -- support apache 2.4 more detailed error log format. Closes gh-268 + Daniel Black + * action.d/hostsdeny -- ensure permissions are fixed - gh-266 - New Features: Daniel Black & ykimon * filter.d/3proxy.conf -- filter added diff --git a/config/action.d/hostsdeny.conf b/config/action.d/hostsdeny.conf index 50a4545c..36e34948 100644 --- a/config/action.d/hostsdeny.conf +++ b/config/action.d/hostsdeny.conf @@ -39,7 +39,7 @@ actionban = IP= && # Tags: See jail.conf(5) man page # Values: CMD # -actionunban = IP= && sed /ALL:\ $IP/d > .new && mv .new +actionunban = echo "/ALL: $/
d
w
q" | ed [Init] From e6823149a1d4916a80594970fe8f7f7845310e88 Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Tue, 2 Jul 2013 20:16:43 +1000 Subject: [PATCH 13/51] ENH: remove non-capturing groups for readibility --- config/filter.d/dovecot.conf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/config/filter.d/dovecot.conf b/config/filter.d/dovecot.conf index e3702fcf..f111859f 100644 --- a/config/filter.d/dovecot.conf +++ b/config/filter.d/dovecot.conf @@ -16,8 +16,8 @@ _daemon = dovecot(-auth)? # first regex is essentially a copy of pam-generic.conf # Values: TEXT # -failregex = ^%(__prefix_line)s(pam_unix(?:\(\S+\))?:)?\s+authentication failure; logname=\S* uid=\S* euid=\S* tty=dovecot ruser=\S* rhost=(\s+user=\S*)?\s*$ - ^%(__prefix_line)s(pop3|imap)-login: (Info: )?(Aborted login|Disconnected)(: Inactivity)? \((no auth attempts|auth failed, \d+ attempts|tried to use disabled \S+ auth)\):( user=<\S+>,)?( method=\S+,)? rip=, lip=(?:\d{1,3}\.){3}\d{1,3}(, TLS( handshaking)?(: Disconnected)?)?\s*$ +failregex = ^%(__prefix_line)s(pam_unix(\(\S+\))?:)?\s+authentication failure; logname=\S* uid=\S* euid=\S* tty=dovecot ruser=\S* rhost=(\s+user=\S*)?\s*$ + ^%(__prefix_line)s(pop3|imap)-login: (Info: )?(Aborted login|Disconnected)(: Inactivity)? \((no auth attempts|auth failed, \d+ attempts|tried to use disabled \S+ auth)\):( user=<\S+>,)?( method=\S+,)? rip=, lip=(\d{1,3}\.){3}\d{1,3}(, TLS( handshaking)?(: Disconnected)?)?\s*$ # Option: ignoreregex # Notes.: regex to ignore. If this regex matches, the line is ignored. From da594075f3f59476ef055a6387634468fa839036 Mon Sep 17 00:00:00 2001 From: Alexander Dietrich Date: Tue, 2 Jul 2013 20:30:41 +0200 Subject: [PATCH 14/51] Move sendmail settings to common file, make sender name configurable --- config/action.d/sendmail-buffered.conf | 18 +++++++----------- config/action.d/sendmail-common.conf | 23 +++++++++++++++++++++++ config/action.d/sendmail-whois-lines.conf | 18 +++++++----------- config/action.d/sendmail-whois.conf | 18 +++++++----------- config/action.d/sendmail.conf | 18 +++++++----------- 5 files changed, 51 insertions(+), 44 deletions(-) create mode 100644 config/action.d/sendmail-common.conf diff --git a/config/action.d/sendmail-buffered.conf b/config/action.d/sendmail-buffered.conf index bec1e91c..f5ca6c10 100644 --- a/config/action.d/sendmail-buffered.conf +++ b/config/action.d/sendmail-buffered.conf @@ -4,6 +4,10 @@ # # +[INCLUDES] + +before = sendmail-common.conf + [Definition] # Option: actionstart @@ -11,7 +15,7 @@ # Values: CMD # actionstart = printf %%b "Subject: [Fail2Ban] : started - From: Fail2Ban <> + From: <> To: \n Hi,\n The jail has been started successfully.\n @@ -25,7 +29,7 @@ actionstart = printf %%b "Subject: [Fail2Ban] : started # actionstop = if [ -f ]; then printf %%b "Subject: [Fail2Ban] : summary - From: Fail2Ban <> + From: <> To: \n Hi,\n These hosts have been banned by Fail2Ban.\n @@ -58,7 +62,7 @@ actionban = printf %%b "`date`: ( failures)\n" >> LINE=$( wc -l | awk '{ print $1 }' ) if [ $LINE -ge ]; then printf %%b "Subject: [Fail2Ban] : summary - From: Fail2Ban <> + From: <> To: \n Hi,\n These hosts have been banned by Fail2Ban.\n @@ -82,14 +86,6 @@ actionunban = # name = default -# Destination/Addressee of the mail -# -dest = root - -# Sender of the mail -# -sender = fail2ban - # Default number of lines that are buffered # lines = 5 diff --git a/config/action.d/sendmail-common.conf b/config/action.d/sendmail-common.conf new file mode 100644 index 00000000..e2820470 --- /dev/null +++ b/config/action.d/sendmail-common.conf @@ -0,0 +1,23 @@ +# Fail2Ban configuration file +# +# Common settings for sendmail actions +# +# Users can override the defaults in sendmail-common.local + +[INCLUDES] + +after = sendmail-common.local + +[Init] + +# Recipient mail address +# +dest = root + +# Sender mail address +# +sender = fail2ban + +# Sender display name +# +sendername = Fail2Ban diff --git a/config/action.d/sendmail-whois-lines.conf b/config/action.d/sendmail-whois-lines.conf index bc5074c6..2cb27bd2 100644 --- a/config/action.d/sendmail-whois-lines.conf +++ b/config/action.d/sendmail-whois-lines.conf @@ -4,6 +4,10 @@ # # +[INCLUDES] + +before = sendmail-common.conf + [Definition] # Option: actionstart @@ -12,7 +16,7 @@ # actionstart = printf %%b "Subject: [Fail2Ban] : started Date: `LC_TIME=C date -u +"%%a, %%d %%h %%Y %%T +0000"` - From: Fail2Ban <> + From: <> To: \n Hi,\n The jail has been started successfully.\n @@ -25,7 +29,7 @@ actionstart = printf %%b "Subject: [Fail2Ban] : started # actionstop = printf %%b "Subject: [Fail2Ban] : stopped Date: `LC_TIME=C date -u +"%%a, %%d %%h %%Y %%T +0000"` - From: Fail2Ban <> + From: <> To: \n Hi,\n The jail has been stopped.\n @@ -46,7 +50,7 @@ actioncheck = # actionban = printf %%b "Subject: [Fail2Ban] : banned Date: `LC_TIME=C date -u +"%%a, %%d %%h %%Y %%T +0000"` - From: Fail2Ban <> + From: <> To: \n Hi,\n The IP has just been banned by Fail2Ban after @@ -72,14 +76,6 @@ actionunban = # name = default -# Destination/Addressee of the mail -# -dest = root - -# Sender of the mail -# -sender = fail2ban - # Path to the log files which contain relevant lines for the abuser IP # logpath = /dev/null diff --git a/config/action.d/sendmail-whois.conf b/config/action.d/sendmail-whois.conf index 0d1fd97e..b111e19f 100644 --- a/config/action.d/sendmail-whois.conf +++ b/config/action.d/sendmail-whois.conf @@ -4,6 +4,10 @@ # # +[INCLUDES] + +before = sendmail-common.conf + [Definition] # Option: actionstart @@ -12,7 +16,7 @@ # actionstart = printf %%b "Subject: [Fail2Ban] : started Date: `LC_TIME=C date -u +"%%a, %%d %%h %%Y %%T +0000"` - From: Fail2Ban <> + From: <> To: \n Hi,\n The jail has been started successfully.\n @@ -25,7 +29,7 @@ actionstart = printf %%b "Subject: [Fail2Ban] : started # actionstop = printf %%b "Subject: [Fail2Ban] : stopped Date: `LC_TIME=C date -u +"%%a, %%d %%h %%Y %%T +0000"` - From: Fail2Ban <> + From: <> To: \n Hi,\n The jail has been stopped.\n @@ -46,7 +50,7 @@ actioncheck = # actionban = printf %%b "Subject: [Fail2Ban] : banned Date: `LC_TIME=C date -u +"%%a, %%d %%h %%Y %%T +0000"` - From: Fail2Ban <> + From: <> To: \n Hi,\n The IP has just been banned by Fail2Ban after @@ -70,11 +74,3 @@ actionunban = # name = default -# Destination/Addressee of the mail -# -dest = root - -# Sender of the mail -# -sender = fail2ban - diff --git a/config/action.d/sendmail.conf b/config/action.d/sendmail.conf index 8054050d..55d388fc 100644 --- a/config/action.d/sendmail.conf +++ b/config/action.d/sendmail.conf @@ -4,6 +4,10 @@ # # +[INCLUDES] + +before = sendmail-common.conf + [Definition] # Option: actionstart @@ -12,7 +16,7 @@ # actionstart = printf %%b "Subject: [Fail2Ban] : started Date: `LC_TIME=C date -u +"%%a, %%d %%h %%Y %%T +0000"` - From: Fail2Ban <> + From: <> To: \n Hi,\n The jail has been started successfully.\n @@ -25,7 +29,7 @@ actionstart = printf %%b "Subject: [Fail2Ban] : started # actionstop = printf %%b "Subject: [Fail2Ban] : stopped Date: `LC_TIME=C date -u +"%%a, %%d %%h %%Y %%T +0000"` - From: Fail2Ban <> + From: <> To: \n Hi,\n The jail has been stopped.\n @@ -46,7 +50,7 @@ actioncheck = # actionban = printf %%b "Subject: [Fail2Ban] : banned Date: `LC_TIME=C date -u +"%%a, %%d %%h %%Y %%T +0000"` - From: Fail2Ban <> + From: <> To: \n Hi,\n The IP has just been banned by Fail2Ban after @@ -68,11 +72,3 @@ actionunban = # name = default -# Destination/Addressee of the mail -# -dest = root - -# Sender of the mail -# -sender = fail2ban - From 8f3671bc94c604ae33a7592d0e72220cbc00dd52 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 2 Jul 2013 17:10:00 -0400 Subject: [PATCH 15/51] BF: figure out minimal sleep time needed for mtime changes to get detected. Close #223, and probably #103 --- testcases/filtertestcase.py | 22 +++++----------- testcases/utils.py | 50 ++++++++++++++++++++++++++++++++++++- 2 files changed, 55 insertions(+), 17 deletions(-) diff --git a/testcases/filtertestcase.py b/testcases/filtertestcase.py index 1657f694..2381c942 100644 --- a/testcases/filtertestcase.py +++ b/testcases/filtertestcase.py @@ -39,6 +39,8 @@ from server.failmanager import FailManagerEmpty # Useful helpers # +from utils import mtimesleep + # yoh: per Steven Hiscocks's insight while troubleshooting # https://github.com/fail2ban/fail2ban/issues/103#issuecomment-15542836 # adding a sufficiently large buffer might help to guarantee that @@ -68,18 +70,6 @@ def _killfile(f, name): _killfile(None, name + '.bak') -def _sleep_4_poll(): - """PollFilter relies on file timestamps - so we might need to - sleep to guarantee that they differ - """ - if sys.version_info[:2] <= (2,4): - # on old Python st_mtime is int, so we should give - # at least 1 sec so polling filter could detect - # the change - time.sleep(1.) - else: - time.sleep(0.1) - def _assert_equal_entries(utest, found, output, count=None): """Little helper to unify comparisons with the target entries @@ -237,14 +227,14 @@ class LogFileMonitor(unittest.TestCase): # but not any longer self.assertTrue(self.notModified()) self.assertTrue(self.notModified()) - _sleep_4_poll() # to guarantee freshier mtime + mtimesleep() # to guarantee freshier mtime for i in range(4): # few changes # unless we write into it self.file.write("line%d\n" % i) self.file.flush() self.assertTrue(self.isModified()) self.assertTrue(self.notModified()) - _sleep_4_poll() # to guarantee freshier mtime + mtimesleep() # to guarantee freshier mtime os.rename(self.name, self.name + '.old') # we are not signaling as modified whenever # it gets away @@ -252,7 +242,7 @@ class LogFileMonitor(unittest.TestCase): f = open(self.name, 'a') self.assertTrue(self.isModified()) self.assertTrue(self.notModified()) - _sleep_4_poll() + mtimesleep() f.write("line%d\n" % i) f.flush() self.assertTrue(self.isModified()) @@ -398,7 +388,7 @@ def get_monitor_failures_testcase(Filter_): # actions might be happening too fast in the tests, # sleep a bit to guarantee reliable time stamps if isinstance(self.filter, FilterPoll): - _sleep_4_poll() + mtimesleep() def isEmpty(self, delay=0.4): # shorter wait time for not modified status diff --git a/testcases/utils.py b/testcases/utils.py index 87aab915..b8d467bf 100644 --- a/testcases/utils.py +++ b/testcases/utils.py @@ -22,7 +22,7 @@ __author__ = "Yaroslav Halchenko" __copyright__ = "Copyright (c) 2013 Yaroslav Halchenko" __license__ = "GPL" -import logging, os, re, traceback +import logging, os, re, tempfile, sys, time, traceback from os.path import basename, dirname # @@ -100,3 +100,51 @@ class FormatterWithTraceBack(logging.Formatter): def format(self, record): record.tbc = record.tb = self._tb() return logging.Formatter.format(self, record) + + +class MTimeSleep(object): + """Sleep minimal duration needed to resolve changes in mtime of files in TMPDIR + + mtime resolution depends on Python version AND underlying filesystem + """ + def __init__(self): + self._sleep = None + + @staticmethod + def _get_good_sleep(): + times = [1., 2., 5., 10.] + # we know that older Pythons simply have no ability to resolve + # at < sec level. + if sys.version_info[:2] > (2, 4): + times = [0.01, 0.1] + times + ffid, name = tempfile.mkstemp() + tfile = os.fdopen(ffid, 'w') + + for stime in times: + prev_stat, dt = "", 0. + # needs to be done 3 times (not clear why) + for i in xrange(3): + stat2 = os.stat(name) + if prev_stat: + dt = (stat2.st_mtime - prev_stat.st_mtime) + prev_stat = stat2 + tfile.write("LOAD\n") + tfile.flush() + time.sleep(stime) + if dt: + break + if not dt: + import logging + logSys = logging.getLogger("fail2ban.tests") + #from warnings import warn + logSys.warn("Could not deduce appropriate sleep time for tests. " + "Maximal tested one of %f sec will be used." % stime) + os.unlink(name) + return stime + + def __call__(self): + if self._sleep is None: + self._sleep = self._get_good_sleep() + time.sleep(self._sleep) + +mtimesleep = MTimeSleep() From 591590860ac3077d9ec24af3c0f998df15820541 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 2 Jul 2013 17:11:24 -0400 Subject: [PATCH 16/51] BF: setSleepTime -- would barf since value is not str (wasn't used/tested) --- server/jailthread.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/server/jailthread.py b/server/jailthread.py index 11d1a82b..ca6effae 100644 --- a/server/jailthread.py +++ b/server/jailthread.py @@ -56,7 +56,7 @@ class JailThread(Thread): def setSleepTime(self, value): self.__sleepTime = value - logSys.info("Set sleeptime = " + value) + logSys.info("Set sleeptime %s" % value) ## # Get the time that the thread sleeps. From e9c8a51ce492d2ee225e514395642f50ae8a64d6 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 2 Jul 2013 17:26:41 -0400 Subject: [PATCH 17/51] ENH: further tighten up detection of mtimesleep duration + log what was assigned --- testcases/utils.py | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/testcases/utils.py b/testcases/utils.py index b8d467bf..6107516b 100644 --- a/testcases/utils.py +++ b/testcases/utils.py @@ -112,11 +112,12 @@ class MTimeSleep(object): @staticmethod def _get_good_sleep(): - times = [1., 2., 5., 10.] + logSys = logging.getLogger("fail2ban.tests") + times = [1.5, 2., 5., 10.] # we know that older Pythons simply have no ability to resolve # at < sec level. if sys.version_info[:2] > (2, 4): - times = [0.01, 0.1] + times + times = [0.1] + times ffid, name = tempfile.mkstemp() tfile = os.fdopen(ffid, 'w') @@ -131,14 +132,19 @@ class MTimeSleep(object): tfile.write("LOAD\n") tfile.flush() time.sleep(stime) - if dt: - break + + # check dt but also verify that we are not getting 'quick' + # stime simply by chance of catching second increment + if dt and \ + not (stime < 1 and int(stat2.st_mtime) == stat2.st_mtime): + break if not dt: - import logging - logSys = logging.getLogger("fail2ban.tests") #from warnings import warn logSys.warn("Could not deduce appropriate sleep time for tests. " "Maximal tested one of %f sec will be used." % stime) + else: + logSys.debug("It was needed a sleep of %f to detect dt=%f mtime change" + % (stime, dt)) os.unlink(name) return stime From d6dece490043671604ddf9710590e294fd2c9095 Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Wed, 3 Jul 2013 07:42:47 +1000 Subject: [PATCH 18/51] ENH: Split log and provide jail examples --- config/jail.conf | 11 +++++++++++ testcases/files/logs/exim | 7 ------- testcases/files/logs/exim-spam | 12 ++++++++++++ 3 files changed, 23 insertions(+), 7 deletions(-) create mode 100644 testcases/files/logs/exim-spam diff --git a/config/jail.conf b/config/jail.conf index c999cc7b..b3b93e53 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -408,3 +408,14 @@ filter = 3proxy action = iptables-multiport[name=3proxy, port=3128, protocol=tcp] logpath = /var/log/3proxy.log +[exim] +enabled = false +filter = exim +action = iptables-multiport[name=exim,port="25,465,587"] +logpath = /var/log/exim/mainlog + +[exim-spam] +enabled = false +filter = exim-spam +action = iptables-multiport[name=exim-spam,port="25,465,587"] +logpath = /var/log/exim/mainlog diff --git a/testcases/files/logs/exim b/testcases/files/logs/exim index 95ca621f..0d5ae51d 100644 --- a/testcases/files/logs/exim +++ b/testcases/files/logs/exim @@ -8,13 +8,7 @@ # 2010-11-24 21:48:41 1PLKOW-00046U-EW F=wvhluo@droolindog.com H=93-143-146-237.adsl.net.t-com.hr (droolindog.com) [93.143.146.237] I=[10.10.10.32]:25 P=esmtp temporarily rejected by local_scan(): Temporary local problem # http://us.generation-nt.com/answer/exim-spamassassin-2010-0-x64-help-204020461.html # 2011-07-07 15:44:16 1QexIu-0006dj-PX F=XXXXXX@XXXXXXXXXXXX H=localhost (saf.bio.caltech.edu) [127.0.0.1] P=esmtp temporarily rejected by local_scan(): Local configuration error - local_scan() library failure/usr/lib/exim/sa-exim.so: cannot open shared object file: No such file or directory -# http://www.clues.ltd.uk/howto/debian-sa-fprot-HOWTO.html -2004-01-18 07:15:35 1Ai79e-0000Dq-8i F=uzwltcmwto24@melissacam.biz H=lsanca1-ar3-4-47-028-040.lsanca1.elnk.dsl.genuity.net [4.47.28.40] P=smtp rejected by local_scan(): Rejected: hits=7.5 required=5.0 trigger=5.0 -# https://github.com/fail2ban/fail2ban/pull/251#issuecomment-19493875 -2013-06-15 11:19:33 [2249] H=([2.181.148.95]) [2.181.148.95]:52391 I=[1.2.3.4]:25 F=fantasizesg4@google.com rejected RCPT some@email.com: rejected found in dnsbl zen.spamhaus.org 2013-06-10 18:33:32 [10099] H=(yakult.com.tw) [202.132.70.178]:3755 I=[1.2.3.4]:25 F=menacedsj04@listserv.eurasia.org rejected RCPT dir@ml3.ru: relay not permitted -2013-06-09 10:21:28 [14127] 1UlasQ-0003fr-45 F=mcorporation4@aol.com H=(mail38.fssprus.ru) [46.254.240.82]:43671 I=[1.2.3.4]:25 P=esmtp rejected by local_scan(): Rejected -2013-06-15 11:20:36 [2516] 1Unmew-0000ea-SE H=egeftech.static.otenet.gr [83.235.177.148]:32706 I=[1.2.3.4]:25 F=auguriesvbd40@google.com rejected after DATA: This message contains a virus (Sanesecurity.Junk.39934.UNOFFICIAL). 2013-06-02 06:54:20 [13314] SMTP protocol synchronization error (input sent without waiting for greeting): rejected connection from H=[211.148.195.192]:25936 I=[1.2.3.4]:25 input="GET / HTTP/1.1\r\n\r\n" 2013-06-02 09:05:48 [18505] SMTP protocol synchronization error (next input sent too soon: pipelining was not advertised): rejected "RSET" H=ba77.mx83.fr [82.96.160.77]:58302 I=[1.2.3.4]:25 next input="QUIT\r\n" 2013-06-02 09:22:05 [19591] SMTP call from pc012-6201.spo.scu.edu.tw [163.14.21.161]:3767 I=[1.2.3.4]:25 dropped: too many nonmail commands (last was "RSET") @@ -22,5 +16,4 @@ 2013-06-07 02:02:09 H=treeladders.kiev.ua [91.232.21.92] sender verify fail for : all relevant MX records point to non-existent hosts 2013-06-15 16:34:55 H=mx.tillions.com [182.18.24.93] F= rejected RCPT : Sender verify failed 2013-06-15 16:36:49 H=altmx.marsukov.com [111.67.203.116] F= rejected RCPT : Unknown user -2013-06-16 02:50:43 H=dbs.marsukov.com [111.67.203.114] F= rejected RCPT : rejected because 111.67.203.114 is in a black list at dnsbl.sorbs.net\nCurrently Sending Spam See: http://www.sorbs.net/lookup.shtml?111.67.203.114 diff --git a/testcases/files/logs/exim-spam b/testcases/files/logs/exim-spam new file mode 100644 index 00000000..535309f5 --- /dev/null +++ b/testcases/files/logs/exim-spam @@ -0,0 +1,12 @@ +# http://forum.lissyara.su/viewtopic.php?f=20&t=29857 +# 2010-11-24 21:48:41 1PLKOW-00046U-EW F=wvhluo@droolindog.com H=93-143-146-237.adsl.net.t-com.hr (droolindog.com) [93.143.146.237] I=[10.10.10.32]:25 P=esmtp temporarily rejected by local_scan(): Temporary local problem +# http://us.generation-nt.com/answer/exim-spamassassin-2010-0-x64-help-204020461.html +# 2011-07-07 15:44:16 1QexIu-0006dj-PX F=XXXXXX@XXXXXXXXXXXX H=localhost (saf.bio.caltech.edu) [127.0.0.1] P=esmtp temporarily rejected by local_scan(): Local configuration error - local_scan() library failure/usr/lib/exim/sa-exim.so: cannot open shared object file: No such file or directory +# http://www.clues.ltd.uk/howto/debian-sa-fprot-HOWTO.html +2004-01-18 07:15:35 1Ai79e-0000Dq-8i F=uzwltcmwto24@melissacam.biz H=lsanca1-ar3-4-47-028-040.lsanca1.elnk.dsl.genuity.net [4.47.28.40] P=smtp rejected by local_scan(): Rejected: hits=7.5 required=5.0 trigger=5.0 +# https://github.com/fail2ban/fail2ban/pull/251#issuecomment-19493875 +2013-06-15 11:19:33 [2249] H=([2.181.148.95]) [2.181.148.95]:52391 I=[1.2.3.4]:25 F=fantasizesg4@google.com rejected RCPT some@email.com: rejected found in dnsbl zen.spamhaus.org +2013-06-09 10:21:28 [14127] 1UlasQ-0003fr-45 F=mcorporation4@aol.com H=(mail38.fssprus.ru) [46.254.240.82]:43671 I=[1.2.3.4]:25 P=esmtp rejected by local_scan(): Rejected +2013-06-15 11:20:36 [2516] 1Unmew-0000ea-SE H=egeftech.static.otenet.gr [83.235.177.148]:32706 I=[1.2.3.4]:25 F=auguriesvbd40@google.com rejected after DATA: This message contains a virus (Sanesecurity.Junk.39934.UNOFFICIAL). +2013-06-16 02:50:43 H=dbs.marsukov.com [111.67.203.114] F= rejected RCPT : rejected because 111.67.203.114 is in a black list at dnsbl.sorbs.net\nCurrently Sending Spam See: http://www.sorbs.net/lookup.shtml?111.67.203.114 + From 256f60adae92933b2fa7e9099103b95a54b90478 Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Wed, 3 Jul 2013 09:01:24 +1000 Subject: [PATCH 19/51] DOC: improved log message --- ChangeLog | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/ChangeLog b/ChangeLog index 51835de4..5331e4a0 100644 --- a/ChangeLog +++ b/ChangeLog @@ -16,7 +16,8 @@ ver. 0.8.11 (2013/XX/XXX) - wanna-be-released * filter.d/apache-common.conf -- support apache 2.4 more detailed error log format. Closes gh-268 Daniel Black - * action.d/hostsdeny -- ensure permissions are fixed - gh-266 + * action.d/hostsdeny -- Switched to use 'ed' available across all + platforms to ensure permissions are the same before and after a ban - gh-266 - New Features: Daniel Black & ykimon * filter.d/3proxy.conf -- filter added From 2ffc14359722acff7ddb19f4f1758dab69fb6f8e Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 2 Jul 2013 19:49:41 -0400 Subject: [PATCH 20/51] ENH: more of heavydebug'ing for FilterPoll --- server/filterpoll.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/server/filterpoll.py b/server/filterpoll.py index 20cde0ca..4acecd42 100644 --- a/server/filterpoll.py +++ b/server/filterpoll.py @@ -30,6 +30,11 @@ from mytime import MyTime import time, logging, os +def _ctime(t): + """Given time in seconds provide string representation with milliseconds + """ + return "%s%.3f" %(time.strftime("%X", time.localtime(t)), (t-int(t))) + # Gets the instance of the logger. logSys = logging.getLogger("fail2ban.filter") @@ -84,6 +89,9 @@ class FilterPoll(FileFilter): def run(self): self.setActive(True) while self._isActive(): + if logSys.getEffectiveLevel() <= 6: + logSys.log(6, "Woke up idle=%s with %d files monitored", + self.getIdle(), len(self.getLogPath())) if not self.getIdle(): # Get file modification for container in self.getLogPath(): @@ -119,6 +127,12 @@ class FilterPoll(FileFilter): try: logStats = os.stat(filename) self.__file404Cnt[filename] = 0 + if logSys.getEffectiveLevel() <= 7: + # we do not want to waste time on strftime etc if not necessary + dt = logStats.st_mtime - self.__lastModTime[filename] + logSys.log(7, "Checking %s for being modified. Previous/current mtimes: %s / %s. dt: %s", + filename, _ctime(self.__lastModTime[filename]), _ctime(logStats.st_mtime), dt) + # os.system("stat %s | grep Modify" % filename) if self.__lastModTime[filename] == logStats.st_mtime: return False else: From 8c125b6053d137132bacfa7c25cb84e261cd9894 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 2 Jul 2013 19:50:22 -0400 Subject: [PATCH 21/51] ENH: do not sleep 1 sec only on older Pythons while testing filters --- testcases/filtertestcase.py | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/testcases/filtertestcase.py b/testcases/filtertestcase.py index 2381c942..f2f6eb50 100644 --- a/testcases/filtertestcase.py +++ b/testcases/filtertestcase.py @@ -109,10 +109,9 @@ def _copy_lines_between_files(fin, fout, n=None, skip=0, mode='a', terminal_line Returns open fout """ - if sys.version_info[:2] <= (2,4): # pragma: no cover - # on old Python st_mtime is int, so we should give at least 1 sec so - # polling filter could detect the change - time.sleep(1) + # on older Pythons and on some FSs st_mtime is int, so we should + # give it some time so polling filter could detect the change + mtimesleep() if isinstance(fin, str): # pragma: no branch - only used with str in test cases fin = open(fin, 'r') # Skip From f340e5c0f5a74a6bb0b515d9e25dec57f28467c6 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 2 Jul 2013 20:43:51 -0400 Subject: [PATCH 22/51] ENH(+possibly BF): for FilterPoll rely not only on mtime but also ino and size to assess if file was modified mtime alone is a poor measure here as many of our tests shown -- on older Pythons and some file systems mtime might be reported only with precision up to a second. If file gets rotated fast, or there are new modifications within the same second, fail2ban might miss them with Polling backend if judging only by mtime. With this modification we will track also inode and size which are all indicative of a file change. --- server/filterpoll.py | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/server/filterpoll.py b/server/filterpoll.py index 4acecd42..4cd7cb48 100644 --- a/server/filterpoll.py +++ b/server/filterpoll.py @@ -57,7 +57,7 @@ class FilterPoll(FileFilter): FileFilter.__init__(self, jail) self.__modified = False ## The time of the last modification of the file. - self.__lastModTime = dict() + self.__prevStats = dict() self.__file404Cnt = dict() logSys.debug("Created FilterPoll") @@ -67,7 +67,7 @@ class FilterPoll(FileFilter): # @param path log file path def _addLogPath(self, path): - self.__lastModTime[path] = 0 + self.__prevStats[path] = (0, None, None) # mtime, ino, size self.__file404Cnt[path] = 0 ## @@ -76,7 +76,7 @@ class FilterPoll(FileFilter): # @param path the log file to delete def _delLogPath(self, path): - del self.__lastModTime[path] + del self.__prevStats[path] del self.__file404Cnt[path] ## @@ -126,18 +126,20 @@ class FilterPoll(FileFilter): def isModified(self, filename): try: logStats = os.stat(filename) + stats = logStats.st_mtime, logStats.st_ino, logStats.st_size + pstats = self.__prevStats[filename] self.__file404Cnt[filename] = 0 if logSys.getEffectiveLevel() <= 7: # we do not want to waste time on strftime etc if not necessary - dt = logStats.st_mtime - self.__lastModTime[filename] - logSys.log(7, "Checking %s for being modified. Previous/current mtimes: %s / %s. dt: %s", - filename, _ctime(self.__lastModTime[filename]), _ctime(logStats.st_mtime), dt) + dt = logStats.st_mtime - pstats[0] + logSys.log(7, "Checking %s for being modified. Previous/current stats: %s / %s. dt: %s", + filename, pstats, stats, dt) # os.system("stat %s | grep Modify" % filename) - if self.__lastModTime[filename] == logStats.st_mtime: + if pstats == stats: return False else: - logSys.debug(filename + " has been modified") - self.__lastModTime[filename] = logStats.st_mtime + logSys.debug("%s has been modified", filename) + self.__prevStats[filename] = stats return True except OSError, e: logSys.error("Unable to get stat on %s because of: %s" From 052e7ff9da53865a38c28631a6857112ad21bbad Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 2 Jul 2013 20:44:28 -0400 Subject: [PATCH 23/51] ENH: deprecate sophisticated MTimeSleep in favor of no sleeping at all all invocations of mtimesleep() are left in the tests for now --- testcases/utils.py | 57 ++++------------------------------------------ 1 file changed, 4 insertions(+), 53 deletions(-) diff --git a/testcases/utils.py b/testcases/utils.py index 6107516b..643c9ad1 100644 --- a/testcases/utils.py +++ b/testcases/utils.py @@ -101,56 +101,7 @@ class FormatterWithTraceBack(logging.Formatter): record.tbc = record.tb = self._tb() return logging.Formatter.format(self, record) - -class MTimeSleep(object): - """Sleep minimal duration needed to resolve changes in mtime of files in TMPDIR - - mtime resolution depends on Python version AND underlying filesystem - """ - def __init__(self): - self._sleep = None - - @staticmethod - def _get_good_sleep(): - logSys = logging.getLogger("fail2ban.tests") - times = [1.5, 2., 5., 10.] - # we know that older Pythons simply have no ability to resolve - # at < sec level. - if sys.version_info[:2] > (2, 4): - times = [0.1] + times - ffid, name = tempfile.mkstemp() - tfile = os.fdopen(ffid, 'w') - - for stime in times: - prev_stat, dt = "", 0. - # needs to be done 3 times (not clear why) - for i in xrange(3): - stat2 = os.stat(name) - if prev_stat: - dt = (stat2.st_mtime - prev_stat.st_mtime) - prev_stat = stat2 - tfile.write("LOAD\n") - tfile.flush() - time.sleep(stime) - - # check dt but also verify that we are not getting 'quick' - # stime simply by chance of catching second increment - if dt and \ - not (stime < 1 and int(stat2.st_mtime) == stat2.st_mtime): - break - if not dt: - #from warnings import warn - logSys.warn("Could not deduce appropriate sleep time for tests. " - "Maximal tested one of %f sec will be used." % stime) - else: - logSys.debug("It was needed a sleep of %f to detect dt=%f mtime change" - % (stime, dt)) - os.unlink(name) - return stime - - def __call__(self): - if self._sleep is None: - self._sleep = self._get_good_sleep() - time.sleep(self._sleep) - -mtimesleep = MTimeSleep() +def mtimesleep(): + # no sleep now should be necessary since polling tracks now not only + # mtime but also ino and size + pass From 8d25e83f5d154fd2adcaa3cac5f883a8f87b56ad Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 2 Jul 2013 23:37:23 -0400 Subject: [PATCH 24/51] BF: race condition -- file should not be read unless it is not empty Previous code would store md5sum of an empty line as the one identifying the monitored file. That file then was read and possibly failures were found. Upon next "container.open()", md5 digest of now present first line was compared against previous digest of an empty line, which was different, thus file was assumed to be rotated and all the log lines were read once again. The History ----------- In rare cases various tests failed somewhat consistently. Below you can find one case in test_move_file where such failure really made no sense -- we should have not had 4 failures by that point. Fail2ban 0.8.10.dev test suite. Python 2.4.6 (#2, Sep 25 2009, 22:22:06) [GCC 4.3.4]. Please wait... I: Skipping gamin backend testing. Got exception 'No module named gamin' I: Skipping pyinotify backend testing. Got exception 'No module named pyinotify' D: fn=/home/yoh/test/monitorfailures_FilterPoll9nUKoCfail2ban-0 hashes=d41d8cd98f00b204e9800998ecf8427e/d41d8cd98f00b204e9800998ecf8427e inos=5398862/5398862 pos=0 rotate=False D: Adding a ticket for ('193.168.0.128', 1124013599.0, ['Aug 14 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 193.168.0.128\n']) D: Adding a ticket for ('193.168.0.128', 1124013599.0, ['Aug 14 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 193.168.0.128\n']) D: Closed with pos 1231 D: fn=/home/yoh/test/monitorfailures_FilterPoll9nUKoCfail2ban-0 hashes=d41d8cd98f00b204e9800998ecf8427e/aeb4e73e6922a746d027eb365ece2149 inos=5398862/5398862 pos=1231 rotate=True D: Adding a ticket for ('193.168.0.128', 1124013599.0, ['Aug 14 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 193.168.0.128\n']) D: Adding a ticket for ('193.168.0.128', 1124013599.0, ['Aug 14 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 193.168.0.128\n']) D: Closed with pos 1231 F ====================================================================== FAIL: test_move_file (testcases.filtertestcase.MonitorFailures(/home/yoh/test/monitorfailures_FilterPoll9nUKoCfail2ban)) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/yoh/deb/gits/fail2ban/testcases/filtertestcase.py", line 451, in test_move_file "Queue must be empty but it is not: %s." AssertionError: Queue must be empty but it is not: server.ticket.FailTicket: ip=193.168.0.128 time=1124013599.0 #attempts=4. ---------------------------------------------------------------------- N.B.1 I preserved here and in the code corresponding additional debug print statements, which are commented out by default. sensible md5 digest was generated by using hexdigest() instead of current a bit faster digest(). Running tests with all the debug output simply breaks the race loose and failure doesn't trigger. N.B.2 d41d8cd98f00b204e9800998ecf8427e is an md5sum of an empty string, and aeb4e73e6922a746d027eb365ece2149 of the first line in that file. --- server/filter.py | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/server/filter.py b/server/filter.py index 59e612ed..b7c2d091 100644 --- a/server/filter.py +++ b/server/filter.py @@ -21,6 +21,8 @@ __author__ = "Cyril Jaquier and Fail2Ban Contributors" __copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2011-2013 Yaroslav Halchenko" __license__ = "GPL" +import sys + from failmanager import FailManagerEmpty from failmanager import FailManager from ticket import FailTicket @@ -322,6 +324,7 @@ class Filter(JailThread): logSys.debug("Ignore %s" % ip) continue logSys.debug("Found %s" % ip) + ## print "D: Adding a ticket for %s" % ((ip, unixTime, [line]),) self.failManager.addFailure(FailTicket(ip, unixTime, [line])) ## @@ -477,7 +480,7 @@ class FileFilter(Filter): return False # Try to open log file. try: - container.open() + has_content = container.open() # see http://python.org/dev/peps/pep-3151/ except IOError, e: logSys.error("Unable to open %s" % filename) @@ -492,7 +495,12 @@ class FileFilter(Filter): logSys.exception(e) return False - while True: + # yoh: has_content is just a bool, so do not expect it to + # change -- loop is exited upon break, and is not entered at + # all if upon container opening that one was empty. If we + # start reading tested to be empty container -- race condition + # might occur leading at least to tests failures. + while has_content: line = container.readline() if (line == "") or not self._isActive(): # The jail reached the bottom or has been stopped @@ -555,10 +563,20 @@ class FileContainer: fd = self.__handler.fileno() flags = fcntl.fcntl(fd, fcntl.F_GETFD) fcntl.fcntl(fd, fcntl.F_SETFD, flags | fcntl.FD_CLOEXEC) + # Stat the file before even attempting to read it + stats = os.fstat(self.__handler.fileno()) + if not stats.st_size: + # yoh: so it is still an empty file -- nothing should be + # read from it yet + # print "D: no content -- return" + return False firstLine = self.__handler.readline() # Computes the MD5 of the first line. myHash = md5sum(firstLine).digest() - stats = os.fstat(self.__handler.fileno()) + ## print "D: fn=%s hashes=%s/%s inos=%s/%s pos=%s rotate=%s" % ( + ## self.__filename, self.__hash, myHash, stats.st_ino, self.__ino, self.__pos, + ## self.__hash != myHash or self.__ino != stats.st_ino) + ## sys.stdout.flush() # Compare hash and inode if self.__hash != myHash or self.__ino != stats.st_ino: logSys.debug("Log rotation detected for %s" % self.__filename) @@ -567,6 +585,7 @@ class FileContainer: self.__pos = 0 # Sets the file pointer to the last position. self.__handler.seek(self.__pos) + return True def readline(self): if self.__handler is None: @@ -580,6 +599,8 @@ class FileContainer: # Closes the file. self.__handler.close() self.__handler = None + ## print "D: Closed %s with pos %d" % (handler, self.__pos) + ## sys.stdout.flush() From 47ac39fb3451c1d29141aefb452124c05653ef87 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 2 Jul 2013 23:37:41 -0400 Subject: [PATCH 25/51] TST: minor enhancement to test failure msg --- testcases/filtertestcase.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/testcases/filtertestcase.py b/testcases/filtertestcase.py index f2f6eb50..d918bbf7 100644 --- a/testcases/filtertestcase.py +++ b/testcases/filtertestcase.py @@ -447,7 +447,9 @@ def get_monitor_failures_testcase(Filter_): self.file = _copy_lines_between_files(GetFailures.FILENAME_01, self.name, n=14, mode='w') # Poll might need more time - self.assertTrue(self.isEmpty(4 + int(isinstance(self.filter, FilterPoll))*2)) + self.assertTrue(self.isEmpty(4 + int(isinstance(self.filter, FilterPoll))*2), + "Queue must be empty but it is not: %s." + % (', '.join([str(x) for x in self.jail.queue]))) self.assertRaises(FailManagerEmpty, self.filter.failManager.toBan) self.assertEqual(self.filter.failManager.getFailTotal(), 2) From 5d7ab9e7fb99287dae3d050ba04a7f5f046f4598 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 2 Jul 2013 23:38:27 -0400 Subject: [PATCH 26/51] DOC: Changelog for preceding changes --- ChangeLog | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/ChangeLog b/ChangeLog index 39c0a5aa..dc08641d 100644 --- a/ChangeLog +++ b/ChangeLog @@ -7,7 +7,7 @@ Fail2Ban (version 0.8.10) 2013/06/12 ================================================================================ -ver. 0.8.11 (2013/XX/XXX) - wanna-be-released +ver. 0.8.11 (2013/XX/XXX) - loves-unittests ----------- - Fixes: @@ -15,6 +15,15 @@ ver. 0.8.11 (2013/XX/XXX) - wanna-be-released * filter.d/common.conf -- make colon after [daemon] optional. Closes gh-267 * filter.d/apache-common.conf -- support apache 2.4 more detailed error log format. Closes gh-268 + * Backends changes detection and parsing. Close gh-223 and gh-103: + - Polling backend: detect changes in the files not only based on + mtime, but also on the size and inode. It should allow for + better detection of changes and log rotations on busy servers, + older python 2.4, and file systems with precision of mtime only + up to a second (e.g. ext3). + - All backends, possible race condition: do not read from a file + initially reported empty. Originally could have lead to + accounting for detected log lines multiple times. Daniel Black & Мернов Георгий * filter.d/dovecot.conf -- Fix when no TLS enabled - line doesn't end in , - New Features: From 5df6796e6943807e17b0e5ea962613af8f7b189f Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 2 Jul 2013 23:51:06 -0400 Subject: [PATCH 27/51] ENH: DNS resolution -- catch parent exception IMHO there is no good reason to capture only gaierror. on my network it was consistent to error out with ====================================================================== ERROR: testIgnoreIPNOK (testcases.filtertestcase.IgnoreIP) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/yoh/deb/gits/fail2ban/testcases/filtertestcase.py", line 166, in testIgnoreIPNOK self.assertFalse(self.filter.inIgnoreIPList(ip)) File "/home/yoh/deb/gits/fail2ban/server/filter.py", line 277, in inIgnoreIPList ips = DNSUtils.dnsToIp(i) File "/home/yoh/deb/gits/fail2ban/server/filter.py", line 625, in dnsToIp return socket.gethostbyname_ex(dns)[2] error: [Errno 11] Resource temporarily unavailable with this commit tests would pass normally as they should --- server/filter.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/server/filter.py b/server/filter.py index b7c2d091..6cd37ca1 100644 --- a/server/filter.py +++ b/server/filter.py @@ -623,9 +623,9 @@ class DNSUtils: """ try: return socket.gethostbyname_ex(dns)[2] - except socket.gaierror: - logSys.warn("Unable to find a corresponding IP address for %s" - % dns) + except socket.error, e: + logSys.warn("Unable to find a corresponding IP address for %s: %s" + % (dns, e)) return list() dnsToIp = staticmethod(dnsToIp) From 404574499d9d76c04e07c36b09a1fbcfb1361795 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 2 Jul 2013 23:52:37 -0400 Subject: [PATCH 28/51] BF: fail2ban-testcases -- use full "warning" instead of warn for the verbosity dictionary --- fail2ban-testcases | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fail2ban-testcases b/fail2ban-testcases index fc5cc146..89fc7deb 100755 --- a/fail2ban-testcases +++ b/fail2ban-testcases @@ -79,7 +79,7 @@ logSys = logging.getLogger("fail2ban") verbosity = {'heavydebug': 4, 'debug': 3, 'info': 2, - 'warn': 1, + 'warning': 1, 'error': 1, 'fatal': 0, None: 1}[opts.log_level] From e282d6b1c7aa15281be3fb748c11ad70ca5dd035 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Wed, 3 Jul 2013 00:09:39 -0400 Subject: [PATCH 29/51] ENH: Remove unused any longer _ctime helper --- server/filterpoll.py | 5 ----- 1 file changed, 5 deletions(-) diff --git a/server/filterpoll.py b/server/filterpoll.py index 4cd7cb48..8b4038b8 100644 --- a/server/filterpoll.py +++ b/server/filterpoll.py @@ -30,11 +30,6 @@ from mytime import MyTime import time, logging, os -def _ctime(t): - """Given time in seconds provide string representation with milliseconds - """ - return "%s%.3f" %(time.strftime("%X", time.localtime(t)), (t-int(t))) - # Gets the instance of the logger. logSys = logging.getLogger("fail2ban.filter") From 2155f6bfa58ed820d95ce50ce7488d6c43c31a20 Mon Sep 17 00:00:00 2001 From: Alexander Dietrich Date: Thu, 4 Jul 2013 08:57:52 +0200 Subject: [PATCH 30/51] Update ChangeLog and jail.conf example --- ChangeLog | 3 +++ config/jail.conf | 2 ++ 2 files changed, 5 insertions(+) diff --git a/ChangeLog b/ChangeLog index 39c0a5aa..dd5426a2 100644 --- a/ChangeLog +++ b/ChangeLog @@ -35,6 +35,9 @@ ver. 0.8.11 (2013/XX/XXX) - wanna-be-released * fail2ban-client -- log to standard error. Closes gh-264 * Fail to configure if not a single log file was found for an enabled jail. Closes gh-63 + Alexander Dietrich + * action.d/sendmail-common.conf -- added common sendmail settings file + and made the sender display name configurable ver. 0.8.10 (2013/06/12) - wanna-be-secure ----------- diff --git a/config/jail.conf b/config/jail.conf index c999cc7b..e4e7e5b0 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -66,6 +66,8 @@ enabled = false filter = sshd action = iptables[name=SSH, port=ssh, protocol=tcp] sendmail-whois[name=SSH, dest=you@example.com, sender=fail2ban@example.com] +# Alternative example: +# sendmail-whois[name=SSH, sendername="Fail2Ban - example.com"] logpath = /var/log/sshd.log maxretry = 5 From 04b8069ceed9212ef9b72ec2ea3d8b8030a31685 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Fri, 5 Jul 2013 10:12:29 -0400 Subject: [PATCH 31/51] ENH: adjust sendmail-whois 'active' example to have also sendername in it --- config/jail.conf | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/config/jail.conf b/config/jail.conf index e4e7e5b0..740db30c 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -65,9 +65,7 @@ usedns = warn enabled = false filter = sshd action = iptables[name=SSH, port=ssh, protocol=tcp] - sendmail-whois[name=SSH, dest=you@example.com, sender=fail2ban@example.com] -# Alternative example: -# sendmail-whois[name=SSH, sendername="Fail2Ban - example.com"] + sendmail-whois[name=SSH, dest=you@example.com, sender=fail2ban@example.com, sendername="Fail2Ban"] logpath = /var/log/sshd.log maxretry = 5 From bfa2b9dec3bb5414e8a8f46d518ec3bce2fbec1a Mon Sep 17 00:00:00 2001 From: Steven Hiscocks Date: Fri, 5 Jul 2013 18:36:02 +0100 Subject: [PATCH 32/51] ENH: dovecot filter additions for session, time value and blank user --- ChangeLog | 3 +++ config/filter.d/dovecot.conf | 2 +- testcases/files/logs/dovecot | 2 ++ 3 files changed, 6 insertions(+), 1 deletion(-) diff --git a/ChangeLog b/ChangeLog index dd5426a2..c91d6bcd 100644 --- a/ChangeLog +++ b/ChangeLog @@ -38,6 +38,9 @@ ver. 0.8.11 (2013/XX/XXX) - wanna-be-released Alexander Dietrich * action.d/sendmail-common.conf -- added common sendmail settings file and made the sender display name configurable + Steven Hiscocks + * filter.d/dovecot - Addition of session, time values and possible blank + user ver. 0.8.10 (2013/06/12) - wanna-be-secure ----------- diff --git a/config/filter.d/dovecot.conf b/config/filter.d/dovecot.conf index f111859f..2143e224 100644 --- a/config/filter.d/dovecot.conf +++ b/config/filter.d/dovecot.conf @@ -17,7 +17,7 @@ _daemon = dovecot(-auth)? # Values: TEXT # failregex = ^%(__prefix_line)s(pam_unix(\(\S+\))?:)?\s+authentication failure; logname=\S* uid=\S* euid=\S* tty=dovecot ruser=\S* rhost=(\s+user=\S*)?\s*$ - ^%(__prefix_line)s(pop3|imap)-login: (Info: )?(Aborted login|Disconnected)(: Inactivity)? \((no auth attempts|auth failed, \d+ attempts|tried to use disabled \S+ auth)\):( user=<\S+>,)?( method=\S+,)? rip=, lip=(\d{1,3}\.){3}\d{1,3}(, TLS( handshaking)?(: Disconnected)?)?\s*$ + ^%(__prefix_line)s(pop3|imap)-login: (Info: )?(Aborted login|Disconnected)(: Inactivity)? \(((no auth attempts|auth failed, \d+ attempts)( in \d+ secs)?|tried to use disabled \S+ auth)\):( user=<\S*>,)?( method=\S+,)? rip=, lip=(\d{1,3}\.){3}\d{1,3}(, session=<\w+>)?(, TLS( handshaking)?(: Disconnected)?)?\s*$ # Option: ignoreregex # Notes.: regex to ignore. If this regex matches, the line is ignored. diff --git a/testcases/files/logs/dovecot b/testcases/files/logs/dovecot index 01df0af3..b4d978b6 100644 --- a/testcases/files/logs/dovecot +++ b/testcases/files/logs/dovecot @@ -14,3 +14,5 @@ Jun 13 21:48:06 platypus dovecot: pop3-login: Disconnected: Inactivity (no auth Jun 13 20:20:21 platypus dovecot: imap-login: Disconnected (no auth attempts): rip=180.189.168.166, lip=113.212.99.194, TLS handshaking: Disconnected Jun 23 00:52:43 vhost1-ua dovecot: pop3-login: Disconnected: Inactivity (auth failed, 1 attempts): user=, method=PLAIN, rip=193.95.245.163, lip=176.214.13.210 +Jul 02 13:49:31 hostname dovecot[442]: pop3-login: Aborted login (auth failed, 1 attempts in 17 secs): user=, method=PLAIN, rip=192.51.100.13, lip=203.0.113.17, session= +Jul 02 13:49:32 hostname dovecot[442]: pop3-login: Disconnected (no auth attempts in 58 secs): user=<>, rip=192.51.100.13, lip=203.0.113.17, session= From 619603fe054765da7bcfed374c55ce07c142abeb Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Sun, 7 Jul 2013 17:48:20 +1000 Subject: [PATCH 33/51] BF: match asterisk InvalidPassword correctly --- config/filter.d/asterisk.conf | 2 +- testcases/files/logs/asterisk | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/config/filter.d/asterisk.conf b/config/filter.d/asterisk.conf index 5e775430..e4e5de5f 100644 --- a/config/filter.d/asterisk.conf +++ b/config/filter.d/asterisk.conf @@ -32,7 +32,7 @@ failregex = ^%(log_prefix)s Registration from '[^']*' failed for '(:\d+)?' ^%(log_prefix)s Host failed MD5 authentication for '[^']*' \([^)]+\)$ ^%(log_prefix)s Failed to authenticate user [^@]+@\S*$ ^%(log_prefix)s (?:handle_request_subscribe: )?Sending fake auth rejection for (device|user) \d*>;tag=\w+\S*$ - ^%(log_prefix)s SecurityEvent="(FailedACL|InvalidAccountID|ChallengeResponseFailed|InvalidPassword)",EventTV="[\d-]+",Severity="[\w]+",Service="[\w]+",EventVersion="\d+",AccountID="\d+",SessionID="0x[\da-f]+",LocalAddress="IPV[46]/(UD|TC)P/[\da-fA-F:.]+/\d+",RemoteAddress="IPV[46]/(UD|TC)P//\d+"$ + ^%(log_prefix)s SecurityEvent="(FailedACL|InvalidAccountID|ChallengeResponseFailed|InvalidPassword)",EventTV="[\d-]+",Severity="[\w]+",Service="[\w]+",EventVersion="\d+",AccountID="\d+",SessionID="0x[\da-f]+",LocalAddress="IPV[46]/(UD|TC)P/[\da-fA-F:.]+/\d+",RemoteAddress="IPV[46]/(UD|TC)P//\d+"(,Challenge="\w+",ReceivedChallenge="\w+")?(,ReceivedHash="[\da-f]+")?$ # Option: ignoreregex # Notes.: regex to ignore. If this regex matches, the line is ignored. diff --git a/testcases/files/logs/asterisk b/testcases/files/logs/asterisk index e2fe3b6e..b4ecf974 100644 --- a/testcases/files/logs/asterisk +++ b/testcases/files/logs/asterisk @@ -19,3 +19,4 @@ [2009-12-22 16:35:24] NOTICE[1570][C-00000086] chan_sip.c: Sending fake auth rejection for device 1022;tag=5d8b6f92 # http://www.spinics.net/lists/asterisk/msg127381.html [2009-12-22 16:35:24] NOTICE[14916]: chan_sip.c:15644 handle_request_subscribe: Sending fake auth rejection for user ;tag=6pwd6erg54 +[2013-07-06 09:09:25] SECURITY[3308] res_security_log.c: SecurityEvent="InvalidPassword",EventTV="1373098165-824497",Severity="Error",Service="SIP",EventVersion="2",AccountID="972592891005",SessionID="0x88aab6c",LocalAddress="IPV4/UDP/92.28.73.180/5060",RemoteAddress="IPV4/UDP/141.255.164.106/5084",Challenge="41d26de5",ReceivedChallenge="41d26de5",ReceivedHash="7a6a3a2e95a05260aee612896e1b4a39" From 1f5097649c55925e7b51c3156d5e0fdd0627a21e Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Tue, 9 Jul 2013 08:20:13 +1000 Subject: [PATCH 34/51] DOC: ChangeLog for exim-spam.conf filter and tabs->spaces in changelog --- ChangeLog | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/ChangeLog b/ChangeLog index 549eb7f7..4a28f815 100644 --- a/ChangeLog +++ b/ChangeLog @@ -13,11 +13,14 @@ ver. 0.8.11 (2013/XX/XXX) - wanna-be-released - Fixes: Yaroslav Halchenko * filter.d/common.conf -- make colon after [daemon] optional. Closes gh-267 - Daniel Black & Мернов Георгий - * filter.d/dovecot.conf -- Fix when no TLS enabled - line doesn't end in , + Daniel Black & Мернов Георгий + * filter.d/dovecot.conf -- Fix when no TLS enabled - line doesn't end in , - New Features: Daniel Black & ykimon * filter.d/3proxy.conf -- filter added + Daniel Black + * filter.d/exim-spam.conf -- a splitout of exim's spam regexes + with additions for greater control over filtering spam. - Enhancements: Daniel Black * filter.d/{asterisk,assp,dovecot,proftpd}.conf -- regex hardening From 174e3dba6dc5bfc44d34f02c68f5dee459cc7b92 Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Tue, 9 Jul 2013 08:36:53 +1000 Subject: [PATCH 35/51] DOC: Note on new dependency - ed for hosts_deny --- ChangeLog | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/ChangeLog b/ChangeLog index 5331e4a0..d02601c1 100644 --- a/ChangeLog +++ b/ChangeLog @@ -16,8 +16,9 @@ ver. 0.8.11 (2013/XX/XXX) - wanna-be-released * filter.d/apache-common.conf -- support apache 2.4 more detailed error log format. Closes gh-268 Daniel Black - * action.d/hostsdeny -- Switched to use 'ed' available across all - platforms to ensure permissions are the same before and after a ban - gh-266 + * action.d/hostsdeny -- NOTE: new dependancy 'ed'. Switched to use 'ed' across + all platforms to ensure permissions are the same before and after a ban - + closes gh-266 - New Features: Daniel Black & ykimon * filter.d/3proxy.conf -- filter added From 424da926010cbc31e4849704b3c7b32e2e87f854 Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Tue, 9 Jul 2013 08:51:11 +1000 Subject: [PATCH 36/51] DOC: close message for commits. --- DEVELOP | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/DEVELOP b/DEVELOP index 2731fd13..2d7a043b 100644 --- a/DEVELOP +++ b/DEVELOP @@ -248,6 +248,10 @@ Use the following tags in your commit messages: Multiple tags could be joined with +, e.g. "BF+TST:". +Use the text "closes #333"/"resolves #333 "/"fixes #333" where 333 represents +an issue that is closed. Other text and details in link below. +See: https://help.github.com/articles/closing-issues-via-commit-messages + Adding Actions -------------- From 606e97683baf3216be2c521305b21122f500a586 Mon Sep 17 00:00:00 2001 From: Steven Hiscocks Date: Fri, 12 Jul 2013 23:34:04 +0100 Subject: [PATCH 37/51] BF: jail.conf multiport actions previously using single port iptables --- config/jail.conf | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/config/jail.conf b/config/jail.conf index c41d2c1b..e3b92038 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -229,7 +229,7 @@ logpath = /var/log/apache2/error_log enabled = false filter = roundcube-auth -action = iptables[name=RoundCube, port="http,https"] +action = iptables-multiport[name=RoundCube, port="http,https"] logpath = /var/log/roundcube/userlogins @@ -241,7 +241,7 @@ enabled = false filter = sogo-auth # without proxy this would be: # port = 20000 -action = iptables[name=SOGo, port="http,https"] +action = iptables-multiport[name=SOGo, port="http,https"] logpath = /var/log/sogo/sogo.log # Ban attackers that try to use PHP's URL-fopen() functionality @@ -251,7 +251,7 @@ logpath = /var/log/sogo/sogo.log [php-url-fopen] enabled = false -action = iptables[name=php-url-open, port="http,https"] +action = iptables-multiport[name=php-url-open, port="http,https"] filter = php-url-fopen logpath = /var/www/*/logs/access_log maxretry = 1 @@ -267,7 +267,7 @@ maxretry = 1 enabled = false filter = lighttpd-fastcgi -action = iptables[name=lighttpd-fastcgi, port="http,https"] +action = iptables-multiport[name=lighttpd-fastcgi, port="http,https"] # adapt the following two items as needed logpath = /var/log/lighttpd/error.log maxretry = 2 @@ -279,7 +279,7 @@ maxretry = 2 enabled = false filter = lighttpd-auth -action = iptables[name=lighttpd-auth, port="http,https"] +action = iptables-multiport[name=lighttpd-auth, port="http,https"] # adapt the following two items as needed logpath = /var/log/lighttpd/error.log maxretry = 2 @@ -405,7 +405,7 @@ maxretry=5 enabled = false filter = 3proxy -action = iptables-multiport[name=3proxy, port=3128, protocol=tcp] +action = iptables[name=3proxy, port=3128, protocol=tcp] logpath = /var/log/3proxy.log [exim] From 40f67c64b8d8d25a6d887ca83c9527549113ef41 Mon Sep 17 00:00:00 2001 From: Steven Hiscocks Date: Sat, 13 Jul 2013 23:03:01 +0100 Subject: [PATCH 38/51] TST: Test sample logs' entries are matched by filter regexs --- fail2ban-testcases | 3 + server/filter.py | 9 +- testcases/files/logs/3proxy | 3 + testcases/files/logs/apache-auth | 2 + testcases/files/logs/apache-nohome | 2 + testcases/files/logs/apache-noscript | 1 + testcases/files/logs/apache-overflows | 2 + testcases/files/logs/assp | 12 +++ testcases/files/logs/asterisk | 18 ++++ testcases/files/logs/dovecot | 17 +++- testcases/files/logs/exim | 12 +++ testcases/files/logs/exim-spam | 5 + .../files/logs/{lighttpd => lighttpd-auth} | 2 + .../files/logs/{mysqld.log => mysqld-auth} | 6 ++ testcases/files/logs/named-refused | 6 ++ testcases/files/logs/pam-generic | 7 ++ testcases/files/logs/postfix | 1 + testcases/files/logs/proftpd | 6 ++ testcases/files/logs/pure-ftpd | 2 + testcases/files/logs/roundcube-auth | 2 + testcases/files/logs/sasl | 2 + testcases/files/logs/sogo-auth | 14 +++ testcases/files/logs/sshd | 28 ++++++ testcases/files/logs/sshd-ddos | 1 + testcases/files/logs/vsftpd | 3 + testcases/files/logs/webmin-auth | 2 + testcases/files/logs/{wu-ftpd => wuftpd} | 1 + testcases/samplestestcase.py | 98 +++++++++++++++++++ 28 files changed, 261 insertions(+), 6 deletions(-) rename testcases/files/logs/{lighttpd => lighttpd-auth} (60%) rename testcases/files/logs/{mysqld.log => mysqld-auth} (51%) rename testcases/files/logs/{wu-ftpd => wuftpd} (60%) create mode 100644 testcases/samplestestcase.py diff --git a/fail2ban-testcases b/fail2ban-testcases index 89fc7deb..278dd890 100755 --- a/fail2ban-testcases +++ b/fail2ban-testcases @@ -32,6 +32,7 @@ from testcases import banmanagertestcase from testcases import clientreadertestcase from testcases import failmanagertestcase from testcases import filtertestcase +from testcases import samplestestcase from testcases import servertestcase from testcases import datedetectortestcase from testcases import actiontestcase @@ -170,6 +171,8 @@ tests.addTest(unittest.makeSuite(filtertestcase.JailTests)) # DateDetector tests.addTest(unittest.makeSuite(datedetectortestcase.DateDetectorTest)) +# Filter Regex tests with sample logs +tests.addTest(unittest.makeSuite(samplestestcase.FilterSamplesRegex)) # # Extensive use-tests of different available filters backends diff --git a/server/filter.py b/server/filter.py index 6cd37ca1..d0390e62 100644 --- a/server/filter.py +++ b/server/filter.py @@ -284,7 +284,7 @@ class Filter(JailThread): return False - def processLine(self, line): + def processLine(self, line, returnRawHost=False): """Split the time portion from log msg and return findFailures on them """ try: @@ -306,7 +306,7 @@ class Filter(JailThread): else: timeLine = l logLine = l - return self.findFailure(timeLine, logLine) + return self.findFailure(timeLine, logLine, returnRawHost) def processLineAndAdd(self, line): """Processes the line for failures and populates failManager @@ -348,7 +348,7 @@ class Filter(JailThread): # to find the logging time. # @return a dict with IP and timestamp. - def findFailure(self, timeLine, logLine): + def findFailure(self, timeLine, logLine, returnRawHost=False): failList = list() # Checks if we must ignore this line. if self.ignoreLine(logLine): @@ -371,6 +371,9 @@ class Filter(JailThread): else: try: host = failRegex.getHost() + if returnRawHost: + failList.append([host, date]) + break ipMatch = DNSUtils.textToIp(host, self.__useDns) if ipMatch: for ip in ipMatch: diff --git a/testcases/files/logs/3proxy b/testcases/files/logs/3proxy index 2967c9bf..614600ef 100644 --- a/testcases/files/logs/3proxy +++ b/testcases/files/logs/3proxy @@ -1,3 +1,6 @@ +# failJSON: { "time": "2013-06-11T02:09:40", "match": true , "host": "1.2.3.4" } 11-06-2013 02:09:40 +0300 PROXY.3128 00004 - 1.2.3.4:28783 0.0.0.0:0 0 0 0 GET http://www.yandex.ua/?ncrnd=2169807731 HTTP/1.1 +# failJSON: { "time": "2013-06-11T02:09:43", "match": true , "host": "1.2.3.4" } 11-06-2013 02:09:43 +0300 PROXY.3128 00005 ewr 1.2.3.4:28788 0.0.0.0:0 0 0 0 GET http://www.yandex.ua/?ncrnd=2169807731 HTTP/1.1 +# failJSON: { "time": "2013-06-13T01:39:34", "match": true , "host": "1.2.3.4" } 13-06-2013 01:39:34 +0300 PROXY.3128 00508 - 1.2.3.4:28938 0.0.0.0:0 0 0 0 diff --git a/testcases/files/logs/apache-auth b/testcases/files/logs/apache-auth index cf0f6d30..6035db3f 100644 --- a/testcases/files/logs/apache-auth +++ b/testcases/files/logs/apache-auth @@ -1,5 +1,7 @@ # Should not match -- DoS vector https://vndh.net/note:fail2ban-089-denial-service +# failJSON: { "match": false } [Sat Jun 01 02:17:42 2013] [error] [client 192.168.33.1] File does not exist: /srv/http/site/[client 192.168.0.1] user root not found # should match +# failJSON: { "time": "2005-06-01T02:17:42", "match": true , "host": "192.168.0.2" } [Sat Jun 01 02:17:42 2013] [error] [client 192.168.0.2] user root not found diff --git a/testcases/files/logs/apache-nohome b/testcases/files/logs/apache-nohome index e8ef56bc..aea0d816 100644 --- a/testcases/files/logs/apache-nohome +++ b/testcases/files/logs/apache-nohome @@ -1,4 +1,6 @@ # Apache 2.2 +# failJSON: { "time": "2005-06-01T11:23:08", "match": true , "host": "1.2.3.4" } [Sat Jun 01 11:23:08 2013] [error] [client 1.2.3.4] File does not exist: /xxx/~ # Apache 2.4 +# failJSON: { "time": "2005-06-27T11:55:44", "match": true , "host": "192.0.2.12" } [Thu Jun 27 11:55:44.569531 2013] [core:info] [pid 4101:tid 2992634688] [client 192.0.2.12:46652] AH00128: File does not exist: /xxx/~ diff --git a/testcases/files/logs/apache-noscript b/testcases/files/logs/apache-noscript index 5d5d35ff..e08b3468 100644 --- a/testcases/files/logs/apache-noscript +++ b/testcases/files/logs/apache-noscript @@ -1 +1,2 @@ +# failJSON: { "time": "2005-06-09T07:57:47", "match": true , "host": "192.0.43.10" } [Sun Jun 09 07:57:47 2013] [error] [client 192.0.43.10] script '/usr/lib/cgi-bin/gitweb.cgiwp-login.php' not found or unable to stat diff --git a/testcases/files/logs/apache-overflows b/testcases/files/logs/apache-overflows index 18a44bfc..1af377f1 100644 --- a/testcases/files/logs/apache-overflows +++ b/testcases/files/logs/apache-overflows @@ -1,2 +1,4 @@ +# failJSON: { "time": "2005-03-16T15:39:29", "match": true , "host": "58.179.109.179" } [Tue Mar 16 15:39:29 2010] [error] [client 58.179.109.179] Invalid URI in request \xf9h\xa9\xf3\x88\x8cXKj \xbf-l*4\x87n\xe4\xfe\xd4\x1d\x06\x8c\xf8m\\rS\xf6n\xeb\x8 +# failJSON: { "time": "2005-03-15T15:44:47", "match": true , "host": "121.222.2.133" } [Mon Mar 15 15:44:47 2010] [error] [client 121.222.2.133] Invalid URI in request n\xed*\xbe*\xab\xefd\x80\xb5\xae\xf6\x01\x10M?\xf2\xce\x13\x9c\xd7\xa0N\xa7\xdb%0\xde\xe0\xfc\xd2\xa0\xfe\xe9w\xee\xc4`v\x9b[{\x0c:\xcb\x93\xc6\xa0\x93\x9c`l\\\x8d\xc9 diff --git a/testcases/files/logs/assp b/testcases/files/logs/assp index 99363001..2c658eb9 100644 --- a/testcases/files/logs/assp +++ b/testcases/files/logs/assp @@ -1,13 +1,25 @@ +# failJSON: { "time": "2013-04-07T07:08:36", "match": true , "host": "68.171.223.68" } Apr-07-13 07:08:36 [SSL-out] 68.171.223.68 SSL negotiation with client failed: SSL accept attempt failed with unknown errorerror:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol; +# failJSON: { "time": "2013-04-07T07:08:36", "match": true , "host": "68.171.223.68" } Apr-07-13 07:08:36 [SSL-out] 68.171.223.68 SSL negotiation with client failed: SSL accept attempt failed with unknown errorerror:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol; +# failJSON: { "time": "2013-04-07T07:10:37", "match": true , "host": "68.171.223.68" } Apr-07-13 07:10:37 [SSL-out] 68.171.223.68 SSL negotiation with client failed: SSL accept attempt failed with unknown errorerror:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol; +# failJSON: { "time": "2013-04-07T07:12:37", "match": true , "host": "68.171.223.68" } Apr-07-13 07:12:37 [SSL-out] 68.171.223.68 SSL negotiation with client failed: SSL accept attempt failed with unknown errorerror:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol; +# failJSON: { "time": "2013-04-07T07:14:36", "match": true , "host": "68.171.223.68" } Apr-07-13 07:14:36 [SSL-out] 68.171.223.68 SSL negotiation with client failed: SSL accept attempt failed with unknown errorerror:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol; +# failJSON: { "time": "2013-04-27T02:25:09", "match": true , "host": "217.194.197.97" } Apr-27-13 02:25:09 Blocking 217.194.197.97 - too much AUTH errors (8); +# failJSON: { "time": "2013-04-27T02:25:09", "match": true , "host": "217.194.197.97" } Apr-27-13 02:25:09 Blocking 217.194.197.97 - too much AUTH errors (9); +# failJSON: { "time": "2013-04-27T02:25:09", "match": true , "host": "217.194.197.97" } Apr-27-13 02:25:09 Blocking 217.194.197.97 - too much AUTH errors (10); +# failJSON: { "time": "2013-04-27T02:25:10", "match": true , "host": "217.194.197.97" } Apr-27-13 02:25:10 [SSL-out] 217.194.197.97 max sender authentication errors (5) exceeded -- dropping connection - after reply: 535 5.7.8 Error: authentication failed: UGFzc3dvcmQ6; +# failJSON: { "time": "2013-04-27T02:25:10", "match": true , "host": "217.194.197.97" } Apr-27-13 02:25:10 [SSL-out] 217.194.197.97 max sender authentication errors (5) exceeded -- dropping connection - after reply: 535 5.7.8 Error: authentication failed: UGFzc3dvcmQ6; +# failJSON: { "time": "2013-04-27T02:25:10", "match": true , "host": "217.194.197.97" } Apr-27-13 02:25:10 [SSL-out] 217.194.197.97 max sender authentication errors (5) exceeded -- dropping connection - after reply: 535 5.7.8 Error: authentication failed: UGFzc3dvcmQ6; +# failJSON: { "time": "2013-04-27T02:25:11", "match": true , "host": "217.194.197.97" } Apr-27-13 02:25:11 [SSL-out] 217.194.197.97 max sender authentication errors (5) exceeded -- dropping connection - after reply: 535 5.7.8 Error: authentication failed: UGFzc3dvcmQ6; diff --git a/testcases/files/logs/asterisk b/testcases/files/logs/asterisk index b4ecf974..ec2fec7c 100644 --- a/testcases/files/logs/asterisk +++ b/testcases/files/logs/asterisk @@ -1,22 +1,40 @@ # Sample log files for asterisk +# failJSON: { "time": "2012-02-13T17:21:54", "match": true , "host": "1.2.3.4" } [2012-02-13 17:21:54] NOTICE[1638] chan_sip.c: Registration from '' failed for '1.2.3.4' - Wrong password +# failJSON: { "time": "2012-02-13T17:18:22", "match": true , "host": "1.2.3.4" } [2012-02-13 17:18:22] NOTICE[1638] chan_sip.c: Registration from '' failed for '1.2.3.4' - No matching peer found +# failJSON: { "time": "2012-02-13T17:21:21", "match": true , "host": "1.2.3.4" } [2012-02-13 17:21:21] NOTICE[1638] chan_sip.c: Registration from '' failed for '1.2.3.4' - Username/auth name mismatch +# failJSON: { "time": "2012-02-13T17:32:01", "match": true , "host": "1.2.3.4" } [2012-02-13 17:32:01] NOTICE[1638] chan_sip.c: Registration from '' failed for '1.2.3.4' - Device does not match ACL +# failJSON: { "time": "2012-02-13T17:34:10", "match": true , "host": "1.2.3.4" } [2012-02-13 17:34:10] NOTICE[1638] chan_sip.c: Registration from '' failed for '1.2.3.4' - Peer is not supposed to register +# failJSON: { "time": "2012-02-13T17:36:23", "match": true , "host": "1.2.3.4" } [2012-02-13 17:36:23] NOTICE[1638] chan_sip.c: Registration from '' failed for '1.2.3.4' - ACL error (permit/deny) +# failJSON: { "time": "2012-02-13T17:53:59", "match": true , "host": "1.2.3.4" } [2012-02-13 17:53:59] NOTICE[1638] chan_iax2.c: Host 1.2.3.4 failed to authenticate as 'Fail2ban' +# failJSON: { "time": "2012-02-13T17:39:20", "match": true , "host": "1.2.3.4" } [2012-02-13 17:39:20] NOTICE[1638] chan_iax2.c: No registration for peer 'Fail2ban' (from 1.2.3.4) +# failJSON: { "time": "2012-02-13T17:44:26", "match": true , "host": "1.2.3.4" } [2012-02-13 17:44:26] NOTICE[1638] chan_iax2.c: Host 1.2.3.4 failed MD5 authentication for 'Fail2ban' (e7df7cd2ca07f4f1ab415d457a6e1c13 != 53ac4bc41ee4ec77888ed4aa50677247) +# failJSON: { "time": "2012-02-13T17:37:07", "match": true , "host": "1.2.3.4" } [2012-02-13 17:37:07] NOTICE[1638] chan_sip.c: Failed to authenticate user "Fail2ban" ;tag=1r698745234 +# failJSON: { "time": "2013-02-05T23:44:42", "match": true , "host": "1.2.3.4" } [2013-02-05 23:44:42] NOTICE[436][C-00000fa9] chan_sip.c: Call from '' (1.2.3.4:10836) to extension '0972598285108' rejected because extension not found in context 'default'. +# failJSON: { "time": "2013-03-26T15:47:54", "match": true , "host": "1.2.3.4" } [2013-03-26 15:47:54] NOTICE[1237] chan_sip.c: Registration from '"100"sip:100@1.2.3.4' failed for '1.2.3.4:23930' - No matching peer found +# failJSON: { "time": "2013-05-13T07:10:53", "match": true , "host": "1.2.3.4" } [2013-05-13 07:10:53] SECURITY[1204] res_security_log.c: SecurityEvent="InvalidAccountID",EventTV="1368439853-500975",Severity="Error",Service="SIP",EventVersion="1",AccountID="00972599580679",SessionID="0x7f8ecc0421f8",LocalAddress="IPV4/UDP/1.2.3.4/5060",RemoteAddress="IPV4/UDP/1.2.3.4/5070" +# failJSON: { "time": "2013-06-10T18:15:03", "match": true , "host": "1.2.3.4" } [2013-06-10 18:15:03] NOTICE[2723] chan_sip.c: Registration from '"100"' failed for '1.2.3.4' - Not a local domain # http://forum.4psa.com/showthread.php?t=6601 +# failJSON: { "time": "2009-12-22T16:35:24", "match": true , "host": "192.168.2.102" } [2009-12-22 16:35:24] NOTICE[6163] chan_sip.c: Sending fake auth rejection for device ;tag=e3793a95e1acbc69o # http://www.freepbx.org/forum/general-help/fake-auth-rejection +# failJSON: { "time": "2009-12-22T16:35:24", "match": true , "host": "192.168.2.102" } [2009-12-22 16:35:24] NOTICE[1570][C-00000086] chan_sip.c: Sending fake auth rejection for device 1022;tag=5d8b6f92 # http://www.spinics.net/lists/asterisk/msg127381.html +# failJSON: { "time": "2009-12-22T16:35:24", "match": true , "host": "192.168.2.102" } [2009-12-22 16:35:24] NOTICE[14916]: chan_sip.c:15644 handle_request_subscribe: Sending fake auth rejection for user ;tag=6pwd6erg54 +# failJSON: { "time": "2013-07-06T09:09:25", "match": true , "host": "141.255.164.106" } [2013-07-06 09:09:25] SECURITY[3308] res_security_log.c: SecurityEvent="InvalidPassword",EventTV="1373098165-824497",Severity="Error",Service="SIP",EventVersion="2",AccountID="972592891005",SessionID="0x88aab6c",LocalAddress="IPV4/UDP/92.28.73.180/5060",RemoteAddress="IPV4/UDP/141.255.164.106/5084",Challenge="41d26de5",ReceivedChallenge="41d26de5",ReceivedHash="7a6a3a2e95a05260aee612896e1b4a39" diff --git a/testcases/files/logs/dovecot b/testcases/files/logs/dovecot index 01df0af3..19297912 100644 --- a/testcases/files/logs/dovecot +++ b/testcases/files/logs/dovecot @@ -1,16 +1,27 @@ +# failJSON: { "time": "2010-09-16T07:51:00", "match": true , "host": "80.187.101.33" } @400000004c91b044077a9e94 imap-login: Info: Aborted login (auth failed, 1 attempts): user=, method=CRAM-MD5, rip=80.187.101.33, lip=80.254.129.240, TLS -@e040c6d8a3bfa62d358083300119c259cd44dcd0 dovecot-auth: pam_unix(dovecot:auth): authentication failure; logname= uid=0 euid=0 tty=dovecot ruser=web rhost=176.61.140.224 +# failJSON: { "time": "2010-09-16T07:51:00", "match": true , "host": "176.61.140.224" } +@400000004c91b044077a9e94 dovecot-auth: pam_unix(dovecot:auth): authentication failure; logname= uid=0 euid=0 tty=dovecot ruser=web rhost=176.61.140.224 # Above example with injected rhost into ruser -- should not match for 1.2.3.4 -@e040c6d8a3bfa62d358083300119c259cd44dcd0 dovecot-auth: pam_unix(dovecot:auth): authentication failure; logname= uid=0 euid=0 tty=dovecot ruser=rhost=1.2.3.4 rhost=192.0.43.10 -@e040c6d8a3bfa62d358083300119c259cd44dcd0 dovecot-auth: pam_unix(dovecot:auth): authentication failure; logname= uid=0 euid=0 tty=dovecot ruser=root rhost=176.61.140.225 user=root +# failJSON: { "time": "2010-09-16T07:51:00", "match": true , "host": "192.0.43.10" } +@400000004c91b044077a9e94 dovecot-auth: pam_unix(dovecot:auth): authentication failure; logname= uid=0 euid=0 tty=dovecot ruser=rhost=1.2.3.4 rhost=192.0.43.10 +# failJSON: { "time": "2010-09-16T07:51:00", "match": true , "host": "176.61.140.225" } +@400000004c91b044077a9e94 dovecot-auth: pam_unix(dovecot:auth): authentication failure; logname= uid=0 euid=0 tty=dovecot ruser=root rhost=176.61.140.225 user=root +# failJSON: { "time": "2004-12-12T11:19:11", "match": true , "host": "190.210.136.21" } Dec 12 11:19:11 dunnart dovecot: pop3-login: Aborted login (tried to use disabled plaintext auth): rip=190.210.136.21, lip=113.212.99.193 +# failJSON: { "time": "2005-06-13T16:30:54", "match": true , "host": "49.176.98.87" } Jun 13 16:30:54 platypus dovecot: imap-login: Disconnected (auth failed, 2 attempts): user=, method=PLAIN, rip=49.176.98.87, lip=113.212.99.194, TLS +# failJSON: { "time": "2005-06-14T00:48:21", "match": true , "host": "59.167.242.100" } Jun 14 00:48:21 platypus dovecot: imap-login: Disconnected (auth failed, 1 attempts): method=PLAIN, rip=59.167.242.100, lip=113.212.99.194, TLS: Disconnected +# failJSON: { "time": "2005-06-13T20:48:11", "match": true , "host": "121.44.24.254" } Jun 13 20:48:11 platypus dovecot: pop3-login: Disconnected (no auth attempts): rip=121.44.24.254, lip=113.212.99.194, TLS: Disconnected +# failJSON: { "time": "2005-06-13T21:48:06", "match": true , "host": "180.200.180.81" } Jun 13 21:48:06 platypus dovecot: pop3-login: Disconnected: Inactivity (no auth attempts): rip=180.200.180.81, lip=113.212.99.194, TLS +# failJSON: { "time": "2005-06-13T20:20:21", "match": true , "host": "180.189.168.166" } Jun 13 20:20:21 platypus dovecot: imap-login: Disconnected (no auth attempts): rip=180.189.168.166, lip=113.212.99.194, TLS handshaking: Disconnected +# failJSON: { "time": "2005-06-23T00:52:43", "match": true , "host": "193.95.245.163" } Jun 23 00:52:43 vhost1-ua dovecot: pop3-login: Disconnected: Inactivity (auth failed, 1 attempts): user=, method=PLAIN, rip=193.95.245.163, lip=176.214.13.210 diff --git a/testcases/files/logs/exim b/testcases/files/logs/exim index 0d5ae51d..95c70d25 100644 --- a/testcases/files/logs/exim +++ b/testcases/files/logs/exim @@ -1,19 +1,31 @@ # From IRC 2013-01-04 +# failJSON: { "time": "2013-01-04T17:03:46", "match": true , "host": "24.106.174.74" } 2013-01-04 17:03:46 login authenticator failed for rrcs-24-106-174-74.se.biz.rr.com ([192.168.2.33]) [24.106.174.74]: 535 Incorrect authentication data (set_id=brian) # From IRC 2013-06-13 XATRIX (Georgiy Mernov) +# failJSON: { "time": "2013-06-12T03:57:58", "match": true , "host": "120.196.140.45" } 2013-06-12 03:57:58 login authenticator failed for (ylmf-pc) [120.196.140.45]: 535 Incorrect authentication data: 1 Time(s) +# failJSON: { "time": "2013-06-12T13:18:11", "match": true , "host": "101.66.165.86" } 2013-06-12 13:18:11 login authenticator failed for (USER-KVI9FGS9KP) [101.66.165.86]: 535 Incorrect authentication data +# failJSON: { "time": "2013-06-10T10:10:59", "match": true , "host": "193.169.56.211" } 2013-06-10 10:10:59 H=ufficioestampa.it (srv.ufficioestampa.it) [193.169.56.211] sender verify fail for : Unrouteable address # http://forum.lissyara.su/viewtopic.php?f=20&t=29857 # 2010-11-24 21:48:41 1PLKOW-00046U-EW F=wvhluo@droolindog.com H=93-143-146-237.adsl.net.t-com.hr (droolindog.com) [93.143.146.237] I=[10.10.10.32]:25 P=esmtp temporarily rejected by local_scan(): Temporary local problem # http://us.generation-nt.com/answer/exim-spamassassin-2010-0-x64-help-204020461.html # 2011-07-07 15:44:16 1QexIu-0006dj-PX F=XXXXXX@XXXXXXXXXXXX H=localhost (saf.bio.caltech.edu) [127.0.0.1] P=esmtp temporarily rejected by local_scan(): Local configuration error - local_scan() library failure/usr/lib/exim/sa-exim.so: cannot open shared object file: No such file or directory +# failJSON: { "time": "2013-06-10T18:33:32", "match": true , "host": "202.132.70.178" } 2013-06-10 18:33:32 [10099] H=(yakult.com.tw) [202.132.70.178]:3755 I=[1.2.3.4]:25 F=menacedsj04@listserv.eurasia.org rejected RCPT dir@ml3.ru: relay not permitted +# failJSON: { "time": "2013-06-02T06:54:20", "match": true , "host": "211.148.195.192" } 2013-06-02 06:54:20 [13314] SMTP protocol synchronization error (input sent without waiting for greeting): rejected connection from H=[211.148.195.192]:25936 I=[1.2.3.4]:25 input="GET / HTTP/1.1\r\n\r\n" +# failJSON: { "time": "2013-06-02T09:05:48", "match": true , "host": "82.96.160.77" } 2013-06-02 09:05:48 [18505] SMTP protocol synchronization error (next input sent too soon: pipelining was not advertised): rejected "RSET" H=ba77.mx83.fr [82.96.160.77]:58302 I=[1.2.3.4]:25 next input="QUIT\r\n" +# failJSON: { "time": "2013-06-02T09:22:05", "match": true , "host": "163.14.21.161" } 2013-06-02 09:22:05 [19591] SMTP call from pc012-6201.spo.scu.edu.tw [163.14.21.161]:3767 I=[1.2.3.4]:25 dropped: too many nonmail commands (last was "RSET") +# failJSON: { "time": "2013-06-02T15:06:18", "match": true , "host": "46.20.35.114" } 2013-06-02 15:06:18 H=(VM-WIN2K3-1562) [46.20.35.114] sender verify fail for : Unknown user +# failJSON: { "time": "2013-06-07T02:02:09", "match": true , "host": "91.232.21.92" } 2013-06-07 02:02:09 H=treeladders.kiev.ua [91.232.21.92] sender verify fail for : all relevant MX records point to non-existent hosts +# failJSON: { "time": "2013-06-15T16:34:55", "match": true , "host": "182.18.24.93" } 2013-06-15 16:34:55 H=mx.tillions.com [182.18.24.93] F= rejected RCPT : Sender verify failed +# failJSON: { "time": "2013-06-15T16:36:49", "match": true , "host": "111.67.203.116" } 2013-06-15 16:36:49 H=altmx.marsukov.com [111.67.203.116] F= rejected RCPT : Unknown user diff --git a/testcases/files/logs/exim-spam b/testcases/files/logs/exim-spam index 535309f5..46b915d1 100644 --- a/testcases/files/logs/exim-spam +++ b/testcases/files/logs/exim-spam @@ -3,10 +3,15 @@ # http://us.generation-nt.com/answer/exim-spamassassin-2010-0-x64-help-204020461.html # 2011-07-07 15:44:16 1QexIu-0006dj-PX F=XXXXXX@XXXXXXXXXXXX H=localhost (saf.bio.caltech.edu) [127.0.0.1] P=esmtp temporarily rejected by local_scan(): Local configuration error - local_scan() library failure/usr/lib/exim/sa-exim.so: cannot open shared object file: No such file or directory # http://www.clues.ltd.uk/howto/debian-sa-fprot-HOWTO.html +# failJSON: { "time": "2004-01-18T07:15:35", "match": true , "host": "4.47.28.40" } 2004-01-18 07:15:35 1Ai79e-0000Dq-8i F=uzwltcmwto24@melissacam.biz H=lsanca1-ar3-4-47-028-040.lsanca1.elnk.dsl.genuity.net [4.47.28.40] P=smtp rejected by local_scan(): Rejected: hits=7.5 required=5.0 trigger=5.0 # https://github.com/fail2ban/fail2ban/pull/251#issuecomment-19493875 +# failJSON: { "time": "2013-06-15T11:19:33", "match": true , "host": "2.181.148.95" } 2013-06-15 11:19:33 [2249] H=([2.181.148.95]) [2.181.148.95]:52391 I=[1.2.3.4]:25 F=fantasizesg4@google.com rejected RCPT some@email.com: rejected found in dnsbl zen.spamhaus.org +# failJSON: { "time": "2013-06-09T10:21:28", "match": true , "host": "46.254.240.82" } 2013-06-09 10:21:28 [14127] 1UlasQ-0003fr-45 F=mcorporation4@aol.com H=(mail38.fssprus.ru) [46.254.240.82]:43671 I=[1.2.3.4]:25 P=esmtp rejected by local_scan(): Rejected +# failJSON: { "time": "2013-06-15T11:20:36", "match": true , "host": "83.235.177.148" } 2013-06-15 11:20:36 [2516] 1Unmew-0000ea-SE H=egeftech.static.otenet.gr [83.235.177.148]:32706 I=[1.2.3.4]:25 F=auguriesvbd40@google.com rejected after DATA: This message contains a virus (Sanesecurity.Junk.39934.UNOFFICIAL). +# failJSON: { "time": "2013-06-16T02:50:43", "match": true , "host": "111.67.203.114" } 2013-06-16 02:50:43 H=dbs.marsukov.com [111.67.203.114] F= rejected RCPT : rejected because 111.67.203.114 is in a black list at dnsbl.sorbs.net\nCurrently Sending Spam See: http://www.sorbs.net/lookup.shtml?111.67.203.114 diff --git a/testcases/files/logs/lighttpd b/testcases/files/logs/lighttpd-auth similarity index 60% rename from testcases/files/logs/lighttpd rename to testcases/files/logs/lighttpd-auth index c3cfcb75..745a2462 100644 --- a/testcases/files/logs/lighttpd +++ b/testcases/files/logs/lighttpd-auth @@ -1,3 +1,5 @@ #authentification failure (mod_auth) +# failJSON: { "time": "2011-12-25T17:09:20", "match": true , "host": "4.4.4.4" } 2011-12-25 17:09:20: (http_auth.c.875) password doesn't match for /gitweb/ username: francois, IP: 4.4.4.4 +# failJSON: { "time": "2012-09-26T10:24:35", "match": true , "host": "4.4.4.4" } 2012-09-26 10:24:35: (http_auth.c.1136) digest: auth failed for xxx : wrong password, IP: 4.4.4.4 diff --git a/testcases/files/logs/mysqld.log b/testcases/files/logs/mysqld-auth similarity index 51% rename from testcases/files/logs/mysqld.log rename to testcases/files/logs/mysqld-auth index b3a73078..757ecb66 100644 --- a/testcases/files/logs/mysqld.log +++ b/testcases/files/logs/mysqld-auth @@ -1,6 +1,12 @@ +# failJSON: { "time": "2013-03-24T00:04:00", "match": true , "host": "192.168.1.35" } 130324 0:04:00 [Warning] Access denied for user 'root'@'192.168.1.35' (using password: NO) +# failJSON: { "time": "2013-03-24T08:24:09", "match": true , "host": "220.95.238.171" } 130324 8:24:09 [Warning] Access denied for user 'root'@'220.95.238.171' (using password: NO) +# failJSON: { "time": "2013-03-24T17:56:13", "match": true , "host": "61.160.223.112" } 130324 17:56:13 [Warning] Access denied for user 'root'@'61.160.223.112' (using password: NO) +# failJSON: { "time": "2013-03-24T17:56:14", "match": true , "host": "61.160.223.112" } 130324 17:56:14 [Warning] Access denied for user 'root'@'61.160.223.112' (using password: YES) +# failJSON: { "time": "2013-03-24T19:01:39", "match": true , "host": "61.147.108.35" } 130324 19:01:39 [Warning] Access denied for user 'root'@'61.147.108.35' (using password: NO) +# failJSON: { "time": "2013-03-24T19:01:40", "match": true , "host": "61.147.108.35" } 130324 19:01:40 [Warning] Access denied for user 'root'@'61.147.108.35' (using password: YES) diff --git a/testcases/files/logs/named-refused b/testcases/files/logs/named-refused index 130e7417..6f6092e2 100644 --- a/testcases/files/logs/named-refused +++ b/testcases/files/logs/named-refused @@ -1,6 +1,12 @@ +# failJSON: { "time": "2005-07-24T14:16:55", "match": true , "host": "194.145.196.18" } Jul 24 14:16:55 raid5 named[3935]: client 194.145.196.18#4795: query 'ricreig.com/NS/IN' denied +# failJSON: { "time": "2005-07-24T14:16:56", "match": true , "host": "62.123.164.113" } Jul 24 14:16:56 raid5 named[3935]: client 62.123.164.113#32768: query 'ricreig.com/NS/IN' denied +# failJSON: { "time": "2005-07-24T14:17:13", "match": true , "host": "148.160.29.6" } Jul 24 14:17:13 raid5 named[3935]: client 148.160.29.6#33081: query (cache) 'geo-mueller.de/NS/IN' denied +# failJSON: { "time": "2005-07-24T14:20:25", "match": true , "host": "148.160.29.6" } Jul 24 14:20:25 raid5 named[3935]: client 148.160.29.6#33081: query (cache) 'shivaree.de/NS/IN' denied +# failJSON: { "time": "2005-07-24T14:23:36", "match": true , "host": "148.160.29.6" } Jul 24 14:23:36 raid5 named[3935]: client 148.160.29.6#33081: query (cache) 'mietberatung.de/NS/IN' denied +# failJSON: { "time": "2005-07-24T14:23:36", "match": true , "host": "62.109.4.89" } Jul 24 14:23:36 raid5 named[3935]: client 62.109.4.89#9334: view external: query (cache) './NS/IN' denied diff --git a/testcases/files/logs/pam-generic b/testcases/files/logs/pam-generic index d84ab153..dc7efeee 100644 --- a/testcases/files/logs/pam-generic +++ b/testcases/files/logs/pam-generic @@ -1,7 +1,14 @@ +# failJSON: { "time": "2005-02-07T15:10:42", "match": true , "host": "192.168.1.1" } Feb 7 15:10:42 example pure-ftpd: (pam_unix) authentication failure; logname= uid=0 euid=0 tty=pure-ftpd ruser=sample-user rhost=192.168.1.1 +# failJSON: { "time": "2005-05-12T09:47:54", "match": true , "host": "71-13-115-12.static.mdsn.wi.charter.com" } May 12 09:47:54 vaio sshd[16004]: (pam_unix) authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=71-13-115-12.static.mdsn.wi.charter.com user=root +# failJSON: { "time": "2005-05-12T09:48:03", "match": true , "host": "71-13-115-12.static.mdsn.wi.charter.com" } May 12 09:48:03 vaio sshd[16021]: (pam_unix) authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=71-13-115-12.static.mdsn.wi.charter.com +# failJSON: { "time": "2005-05-15T18:02:12", "match": true , "host": "66.232.129.62" } May 15 18:02:12 localhost proftpd: (pam_unix) authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=66.232.129.62 user=mark +# failJSON: { "time": "2004-11-25T17:12:13", "match": true , "host": "192.168.10.3" } Nov 25 17:12:13 webmail pop(pam_unix)[4920]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=192.168.10.3 user=mailuser +# failJSON: { "time": "2005-07-19T18:11:26", "match": true , "host": "www3.google.com" } Jul 19 18:11:26 srv2 vsftpd: pam_unix(vsftpd:auth): authentication failure; logname= uid=0 euid=0 tty=ftp ruser=an8767 rhost=www3.google.com +# failJSON: { "time": "2005-07-19T18:11:26", "match": true , "host": "www3.google.com" } Jul 19 18:11:26 srv2 vsftpd: pam_unix: authentication failure; logname= uid=0 euid=0 tty=ftp ruser=an8767 rhost=www3.google.com diff --git a/testcases/files/logs/postfix b/testcases/files/logs/postfix index 28ff95a4..50e3a360 100644 --- a/testcases/files/logs/postfix +++ b/testcases/files/logs/postfix @@ -1,3 +1,4 @@ # per https://github.com/fail2ban/fail2ban/issues/125 # and https://github.com/fail2ban/fail2ban/issues/126 +# failJSON: { "time": "2005-02-21T09:21:54", "match": true , "host": "192.0.43.10" } Feb 21 09:21:54 xxx postfix/smtpd[14398]: NOQUEUE: reject: RCPT from example.com[192.0.43.10]: 450 4.7.1 : Helo command rejected: Host not found; from=<> to=<> proto=ESMTP helo= diff --git a/testcases/files/logs/proftpd b/testcases/files/logs/proftpd index aaaf0295..404ebfaf 100644 --- a/testcases/files/logs/proftpd +++ b/testcases/files/logs/proftpd @@ -1,8 +1,14 @@ +# failJSON: { "time": "2005-01-10T00:00:00", "match": true , "host": "123.123.123.123" } Jan 10 00:00:00 myhost proftpd[12345] myhost.domain.com (123.123.123.123[123.123.123.123]): USER username (Login failed): User in /etc/ftpusers +# failJSON: { "time": "2005-02-01T00:00:00", "match": true , "host": "123.123.123.123" } Feb 1 00:00:00 myhost proftpd[12345] myhost.domain.com (123.123.123.123[123.123.123.123]): USER username: no such user found from 123.123.123.123 [123.123.123.123] to 234.234.234.234:21 +# failJSON: { "time": "2005-06-09T07:30:58", "match": true , "host": "67.227.224.66" } Jun 09 07:30:58 platypus.ace-hosting.com.au proftpd[11864] platypus.ace-hosting.com.au (mail.bloodymonster.net[::ffff:67.227.224.66]): USER username (Login failed): Incorrect password. +# failJSON: { "time": "2005-06-09T11:15:43", "match": true , "host": "101.71.143.238" } Jun 09 11:15:43 platypus.ace-hosting.com.au proftpd[17424] platypus.ace-hosting.com.au (::ffff:101.71.143.238[::ffff:101.71.143.238]): USER god: no such user found from ::ffff:101.71.143.238 [::ffff:101.71.143.238] to ::ffff:123.212.99.194:21 +# failJSON: { "time": "2005-06-13T22:07:23", "match": true , "host": "59.167.242.100" } Jun 13 22:07:23 platypus.ace-hosting.com.au proftpd[15719] platypus.ace-hosting.com.au (::ffff:59.167.242.100[::ffff:59.167.242.100]): SECURITY VIOLATION: root login attempted. +# failJSON: { "time": "2005-06-14T00:09:59", "match": true , "host": "59.167.242.100" } Jun 14 00:09:59 platypus.ace-hosting.com.au proftpd[17839] platypus.ace-hosting.com.au (::ffff:59.167.242.100[::ffff:59.167.242.100]): USER platypus.ace-hosting.com.au proftpd[17424] platypus.ace-hosting.com.au (hihoinjection[1.2.3.44]): no such user found from ::ffff:59.167.242.100 [::ffff:59.167.242.100] to ::ffff:113.212.99.194:21 diff --git a/testcases/files/logs/pure-ftpd b/testcases/files/logs/pure-ftpd index 4b4e3455..ef49b2ff 100644 --- a/testcases/files/logs/pure-ftpd +++ b/testcases/files/logs/pure-ftpd @@ -1,2 +1,4 @@ +# failJSON: { "time": "2005-01-31T16:54:07", "match": true , "host": "24.79.92.194" } Jan 31 16:54:07 desktop pure-ftpd: (?@24.79.92.194) [WARNING] Authentication failed for user [Administrator] +# failJSON: { "time": "2004-11-05T18:54:02", "match": true , "host": "server202181210195.ixlink.net" } Nov 5 18:54:02 pure-ftpd: (?@server202181210195.ixlink.net) [WARNING] Authentication failed for user [Administrator] diff --git a/testcases/files/logs/roundcube-auth b/testcases/files/logs/roundcube-auth index 04e0faf5..dec8051e 100644 --- a/testcases/files/logs/roundcube-auth +++ b/testcases/files/logs/roundcube-auth @@ -1,2 +1,4 @@ +# failJSON: { "time": "2013-01-22T22:28:21", "match": true , "host": "192.0.43.10" } [22-Jan-2013 22:28:21 +0200]: FAILED login for user1 from 192.0.43.10 +# failJSON: { "time": "2005-05-26T07:12:40", "match": true , "host": "10.1.1.47" } May 26 07:12:40 hamster roundcube: IMAP Error: Login failed for sales@example.com from 10.1.1.47 diff --git a/testcases/files/logs/sasl b/testcases/files/logs/sasl index 18c5ff14..ca306098 100644 --- a/testcases/files/logs/sasl +++ b/testcases/files/logs/sasl @@ -1,5 +1,7 @@ #1 Example from postfix from dbts #507990 +# failJSON: { "time": "2004-12-02T22:24:22", "match": true , "host": "114.44.142.233" } Dec 2 22:24:22 hel postfix/smtpd[7676]: warning: 114-44-142-233.dynamic.hinet.net[114.44.142.233]: SASL CRAM-MD5 authentication failed: PDc3OTEwNTkyNTEyMzA2NDIuMTIyODI1MzA2MUBoZWw+ #2 Example from postfix from dbts #573314 +# failJSON: { "time": "2005-03-10T13:33:30", "match": true , "host": "1.1.1.1" } Mar 10 13:33:30 gandalf postfix/smtpd[3937]: warning: HOSTNAME[1.1.1.1]: SASL LOGIN authentication failed: authentication failure diff --git a/testcases/files/logs/sogo-auth b/testcases/files/logs/sogo-auth index e4fb9409..02a69c6d 100644 --- a/testcases/files/logs/sogo-auth +++ b/testcases/files/logs/sogo-auth @@ -1,17 +1,31 @@ # yoh: Kept original apache log lines as well, just in case they might come useful # for (testing) multiline regular expressions which would further constraint # SOGo log lines +# failJSON: { "match": false } Mar 24 08:58:32 sogod [26818]: <0x0xb8537990[LDAPSource]> NAME:LDAPException REASON:operation bind failed: Invalid credentials (0x31) INFO:{login = "uid=hack0r,ou=users,dc=mail,dc=example,dc=org"; } +# failJSON: { "time": "2005-03-24T08:58:32", "match": true , "host": "173.194.44.31" } Mar 24 08:58:32 sogod [26818]: SOGoRootPage Login from '173.194.44.31' for user 'hack0r' might not have worked - password policy: 65535 grace: -1 expire: -1 bound: 0 +# failJSON: { "match": false } 173.194.44.31 - - [24/Mar/2013:08:58:32 GMT] "POST /SOGo/connect HTTP/1.1" 403 34/38 0.311 - - 2M +# failJSON: { "match": false } Mar 24 08:58:40 sogod [26818]: <0x0xb8537990[LDAPSource]> NAME:LDAPException REASON:operation bind failed: Invalid credentials (0x31) INFO:{login = "uid=kiddy,ou=users,dc=mail,dc=example,dc=org"; } +# failJSON: { "time": "2005-03-24T08:58:40", "match": true , "host": "173.194.44.31" } Mar 24 08:58:40 sogod [26818]: SOGoRootPage Login from '173.194.44.31' for user 'kiddy' might not have worked - password policy: 65535 grace: -1 expire: -1 bound: 0 +# failJSON: { "match": false } 173.194.44.31 - - [24/Mar/2013:08:58:40 GMT] "POST /SOGo/connect HTTP/1.1" 403 34/37 0.007 - - 32K +# failJSON: { "match": false } Mar 24 08:58:50 sogod [26818]: <0x0xb8537990[LDAPSource]> NAME:LDAPException REASON:operation bind failed: Invalid credentials (0x31) INFO:{login = "uid=plsBanMe,ou=users,dc=mail,dc=example,dc=org"; } +# failJSON: { "time": "2005-03-24T08:58:50", "match": true , "host": "173.194.44.31" } Mar 24 08:58:50 sogod [26818]: SOGoRootPage Login from '173.194.44.31' for user 'plsBanMe' might not have worked - password policy: 65535 grace: -1 expire: -1 bound: 0 +# failJSON: { "match": false } 173.194.44.31 - - [24/Mar/2013:08:58:50 GMT] "POST /SOGo/connect HTTP/1.1" 403 34/40 0.008 - - 0 +# failJSON: { "match": false } Mar 24 08:58:59 sogod [26818]: <0x0xb8537990[LDAPSource]> NAME:LDAPException REASON:operation bind failed: Invalid credentials (0x31) INFO:{login = "uid=root,ou=users,dc=mail,dc=example,dc=org"; } +# failJSON: { "time": "2005-03-24T08:58:59", "match": true , "host": "173.194.44.31" } Mar 24 08:58:59 sogod [26818]: SOGoRootPage Login from '173.194.44.31' for user 'root' might not have worked - password policy: 65535 grace: -1 expire: -1 bound: 0 +# failJSON: { "match": false } 173.194.44.31 - - [24/Mar/2013:08:58:59 GMT] "POST /SOGo/connect HTTP/1.1" 403 34/36 0.007 - - 0 +# failJSON: { "match": false } Mar 24 08:59:04 sogod [26818]: <0x0xb8537990[LDAPSource]> NAME:LDAPException REASON:operation bind failed: Invalid credentials (0x31) INFO:{login = "uid=admin,ou=users,dc=mail,dc=example,dc=org"; } +# failJSON: { "time": "2005-03-24T08:59:04", "match": true , "host": "173.194.44.31" } Mar 24 08:59:04 sogod [26818]: SOGoRootPage Login from '173.194.44.31' for user 'admin' might not have worked - password policy: 65535 grace: -1 expire: -1 bound: 0 diff --git a/testcases/files/logs/sshd b/testcases/files/logs/sshd index 1943fb37..ff97c5a5 100644 --- a/testcases/files/logs/sshd +++ b/testcases/files/logs/sshd @@ -1,51 +1,79 @@ #1 +# failJSON: { "time": "2005-06-21T16:47:48", "match": true , "host": "192.030.0.6" } Jun 21 16:47:48 digital-mlhhyiqscv sshd[13709]: error: PAM: Authentication failure for myhlj1374 from 192.030.0.6 +# failJSON: { "time": "2005-05-29T20:56:52", "match": true , "host": "example.com" } May 29 20:56:52 imago sshd[28732]: error: PAM: Authentication failure for stefanor from example.com #2 +# failJSON: { "time": "2005-02-25T14:34:10", "match": true , "host": "194.117.26.69" } Feb 25 14:34:10 belka sshd[31602]: Failed password for invalid user ROOT from 194.117.26.69 port 50273 ssh2 +# failJSON: { "time": "2005-02-25T14:34:10", "match": true , "host": "194.117.26.70" } Feb 25 14:34:10 belka sshd[31602]: Failed password for invalid user ROOT from 194.117.26.70 port 12345 #3 +# failJSON: { "time": "2005-01-05T01:31:41", "match": true , "host": "1.2.3.4" } Jan 5 01:31:41 www sshd[1643]: ROOT LOGIN REFUSED FROM 1.2.3.4 +# failJSON: { "time": "2005-01-05T01:31:41", "match": true , "host": "1.2.3.4" } Jan 5 01:31:41 www sshd[1643]: ROOT LOGIN REFUSED FROM ::ffff:1.2.3.4 #4 +# failJSON: { "time": "2005-07-20T14:42:11", "match": true , "host": "211.114.51.213" } Jul 20 14:42:11 localhost sshd[22708]: Invalid user ftp from 211.114.51.213 #5 new filter introduced after looking at 44087D8C.9090407@bluewin.ch # yoh: added ':' after [sshd] since the case without is not really common any more +# failJSON: { "time": "2005-03-03T00:17:22", "match": true , "host": "211.188.220.49" } Mar 3 00:17:22 [sshd]: User root from 211.188.220.49 not allowed because not listed in AllowUsers +# failJSON: { "time": "2005-02-25T14:34:11", "match": true , "host": "example.com" } Feb 25 14:34:11 belka sshd[31607]: User root from example.com not allowed because not listed in AllowUsers #6 ew filter introduced thanks to report Guido Bozzetto +# failJSON: { "time": "2004-11-11T23:33:27", "match": true , "host": "218.249.210.161" } Nov 11 23:33:27 Server sshd[5174]: refused connect from _U2FsdGVkX19P3BCJmFBHhjLza8BcMH06WCUVwttMHpE=_@::ffff:218.249.210.161 (::ffff:218.249.210.161) #7 added exclamation mark to BREAK-IN # Now should be a negative since we decided not to catch those +# failJSON: { "match": false } Oct 15 19:51:35 server sshd[7592]: Address 1.2.3.4 maps to 1234.bbbbbb.com, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT +# failJSON: { "match": false } Oct 15 19:51:35 server sshd[7592]: Address 1.2.3.4 maps to 1234.bbbbbb.com, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT! #8 DenyUsers https://github.com/fail2ban/fail2ban/issues/47 +# failJSON: { "time": "2005-04-16T22:01:15", "match": true , "host": "46.45.128.3" } Apr 16 22:01:15 al-ribat sshd[5154]: User root from 46.45.128.3 not allowed because listed in DenyUsers #9 OpenSolaris patch - pull https://github.com/fail2ban/fail2ban/pull/182 +# failJSON: { "time": "2005-03-29T05:59:23", "match": true , "host": "205.186.180.55" } Mar 29 05:59:23 dusky sshd[20878]: [ID 800047 auth.info] Failed keyboard-interactive for from 205.186.180.55 port 42742 ssh2 +# failJSON: { "time": "2005-03-29T05:20:09", "match": true , "host": "205.186.180.30" } Mar 29 05:20:09 dusky sshd[19558]: [ID 800047 auth.info] Failed keyboard-interactive for james from 205.186.180.30 port 54520 ssh2 #10 OSX syslog error +# failJSON: { "time": "2005-04-29T17:16:20", "match": true , "host": "example.com" } Apr 29 17:16:20 Jamess-iMac.local sshd[62312]: error: PAM: authentication error for james from example.com via 192.168.1.201 +# failJSON: { "time": "2005-04-29T20:11:08", "match": true , "host": "205.186.180.35" } Apr 29 20:11:08 Jamess-iMac.local sshd[63814]: [ID 800047 auth.info] Failed keyboard-interactive for from 205.186.180.35 port 42742 ssh2 +# failJSON: { "time": "2005-04-29T20:12:08", "match": true , "host": "205.186.180.22" } Apr 29 20:12:08 Jamess-iMac.local sshd[63814]: [ID 800047 auth.info] Failed keyboard-interactive for james from 205.186.180.22 port 54520 ssh2 +# failJSON: { "time": "2005-04-29T20:13:08", "match": true , "host": "205.186.180.42" } Apr 29 20:13:08 Jamess-iMac.local sshd[63814]: Failed keyboard-interactive for james from 205.186.180.42 port 54520 ssh2 +# failJSON: { "time": "2005-04-29T20:14:08", "match": true , "host": "205.186.180.44" } Apr 29 20:14:08 Jamess-iMac.local sshd[63814]: Failed keyboard-interactive for from 205.186.180.44 port 42742 ssh2 +# failJSON: { "time": "2005-04-30T01:42:12", "match": true , "host": "205.186.180.77" } Apr 30 01:42:12 Jamess-iMac.local sshd[2554]: Failed keyboard-interactive/pam for invalid user jamedds from 205.186.180.77 port 33723 ssh2 +# failJSON: { "time": "2005-04-29T12:53:38", "match": true , "host": "205.186.180.88" } Apr 29 12:53:38 Jamess-iMac.local sshd[47831]: error: PAM: authentication failure for james from 205.186.180.88 via 192.168.1.201 +# failJSON: { "time": "2005-04-29T13:53:38", "match": true , "host": "205.186.180.99" } Apr 29 13:53:38 Jamess-iMac.local sshd[47831]: error: PAM: Authentication failure for james from 205.186.180.99 via 192.168.1.201 +# failJSON: { "time": "2005-04-29T15:53:38", "match": true , "host": "205.186.180.100" } Apr 29 15:53:38 Jamess-iMac.local sshd[47831]: error: PAM: Authentication error for james from 205.186.180.100 via 192.168.1.201 +# failJSON: { "time": "2005-04-29T16:53:38", "match": true , "host": "205.186.180.101" } Apr 29 16:53:38 Jamess-iMac.local sshd[47831]: error: PAM: authentication error for james from 205.186.180.101 via 192.168.1.201 +# failJSON: { "time": "2005-04-29T17:53:38", "match": true , "host": "205.186.180.102" } Apr 29 17:53:38 Jamess-iMac.local sshd[47831]: error: PAM: authentication error for james from 205.186.180.102 +# failJSON: { "time": "2005-04-29T18:53:38", "match": true , "host": "205.186.180.103" } Apr 29 18:53:38 Jamess-iMac.local sshd[47831]: error: PAM: authentication error for james from 205.186.180.103 #11 https://github.com/fail2ban/fail2ban/issues/267 There might be no colon after [daemon] +# failJSON: { "time": "2005-06-25T23:53:34", "match": true , "host": "1.2.3.4" } Jun 25 23:53:34 [sshd] User root from 1.2.3.4 not allowed because not listed in AllowUsers diff --git a/testcases/files/logs/sshd-ddos b/testcases/files/logs/sshd-ddos index d71c6bb2..452abbde 100644 --- a/testcases/files/logs/sshd-ddos +++ b/testcases/files/logs/sshd-ddos @@ -1,2 +1,3 @@ # http://forums.powervps.com/showthread.php?t=1667 +# failJSON: { "time": "2005-06-07T01:10:56", "match": true , "host": "69.61.56.114" } Jun 7 01:10:56 host sshd[5937]: Did not receive identification string from 69.61.56.114 diff --git a/testcases/files/logs/vsftpd b/testcases/files/logs/vsftpd index a8b6a4cf..f3fb997f 100644 --- a/testcases/files/logs/vsftpd +++ b/testcases/files/logs/vsftpd @@ -1,7 +1,10 @@ #1 PAM based +# failJSON: { "time": "2004-10-11T01:06:47", "match": true , "host": "209.67.1.67" } Oct 11 01:06:47 ServerJV vsftpd: (pam_unix) authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=209.67.1.67 +# failJSON: { "time": "2005-02-06T12:02:29", "match": true , "host": "64.168.103.1" } Feb 6 12:02:29 server vsftpd(pam_unix)[15522]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=64.168.103.1 user=user1 #2 Internal +# failJSON: { "time": "2005-01-19T12:20:33", "match": true , "host": "64.106.46.98" } Fri Jan 19 12:20:33 2007 [pid 27202] [anonymous] FAIL LOGIN: Client "64.106.46.98" diff --git a/testcases/files/logs/webmin-auth b/testcases/files/logs/webmin-auth index 86d3f321..424c114d 100644 --- a/testcases/files/logs/webmin-auth +++ b/testcases/files/logs/webmin-auth @@ -1,7 +1,9 @@ #Webmin authentication failures from /var/log/auth.log #1 User exists, bad password +# failJSON: { "time": "2004-12-13T08:15:18", "match": true , "host": "89.2.49.230" } Dec 13 08:15:18 sb1 webmin[25875]: Invalid login as root from 89.2.49.230 #2 User does not exists +# failJSON: { "time": "2004-12-12T23:14:19", "match": true , "host": "188.40.105.142" } Dec 12 23:14:19 sb1 webmin[22134]: Non-existent login as robert from 188.40.105.142 diff --git a/testcases/files/logs/wu-ftpd b/testcases/files/logs/wuftpd similarity index 60% rename from testcases/files/logs/wu-ftpd rename to testcases/files/logs/wuftpd index b6b41613..22ac0303 100644 --- a/testcases/files/logs/wu-ftpd +++ b/testcases/files/logs/wuftpd @@ -1,2 +1,3 @@ # This login line is from syslog +# failJSON: { "time": "2004-10-06T09:59:26", "match": true , "host": "202.108.145.173" } Oct 6 09:59:26 myserver wu-ftpd[18760]: failed login from hj-145-173-a8.bta.net.cn [202.108.145.173] diff --git a/testcases/samplestestcase.py b/testcases/samplestestcase.py new file mode 100644 index 00000000..2bb73ca1 --- /dev/null +++ b/testcases/samplestestcase.py @@ -0,0 +1,98 @@ +# emacs: -*- mode: python; py-indent-offset: 4; indent-tabs-mode: t -*- +# vi: set ft=python sts=4 ts=4 sw=4 noet : + +# This file is part of Fail2Ban. +# +# Fail2Ban is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# Fail2Ban is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with Fail2Ban; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +# Fail2Ban developers + +__copyright__ = "Copyright (c) 2013 Steven Hiscocks" +__license__ = "GPL" + +import unittest, os, fileinput, re, json, datetime + +from server.filter import Filter +from client.filterreader import FilterReader + +TEST_FILES_DIR = os.path.join(os.path.dirname(__file__), "files") +CONFIG_DIR = "config" + +class FilterSamplesRegex(unittest.TestCase): + + def setUp(self): + """Call before every test case.""" + self.filter = Filter(None) + self.filter.setActive(True) + + def tearDown(self): + """Call after every test case.""" + +def testSampleRegexsFactory(name): + def testFilter(self): + + # Check filter exists + filterConf = FilterReader(name, "jail", basedir=CONFIG_DIR) + filterConf.read() + filterConf.getOptions({}) + + for opt in filterConf.convert(): + if opt[2] == "addfailregex": + self.filter.addFailRegex(opt[3]) + + logFile = fileinput.FileInput( + os.path.join(TEST_FILES_DIR, "logs", name)) + for line in logFile: + jsonREMatch = re.match("^# ?failJSON:(.+)$", line) + if jsonREMatch: + try: + faildata = json.loads(jsonREMatch.group(1)) + except ValueError, e: + raise ValueError("%s: %s:%i" % + (e, logFile.filename(), logFile.filelineno())) + line = next(logFile) + elif line.startswith("#") or not line.strip(): + continue + else: + faildata = {} + + ret = self.filter.processLine(line, returnRawHost=True) + if not ret: + # Check line is flagged as none match + self.assertFalse(faildata.get('match', True), + "Line not matched when should have: %s:%i" % + (logFile.filename(), logFile.filelineno())) + elif ret: + # Check line is flagged to match + self.assertTrue(faildata.get('match', False), + "Line matched when shouldn't have: %s:%i" % + (logFile.filename(), logFile.filelineno())) + self.assertEqual(len(ret), 1) + # Verify timestamp and host as expected + host, time = ret[0] + self.assertEqual(host, faildata.get("host", None)) + self.assertEqual( + datetime.datetime.fromtimestamp(time), + datetime.datetime.strptime( + faildata.get("time", None), "%Y-%m-%dT%H:%M:%S")) + + return testFilter + +for filter_ in os.listdir(os.path.join(TEST_FILES_DIR, "logs")): + if os.path.isfile(os.path.join(TEST_FILES_DIR, "logs", filter_)): + setattr( + FilterSamplesRegex, + "testSampleRegexs%s" % filter_.upper(), + testSampleRegexsFactory(filter_)) From 09850d6ba52c7c1fbbe007cfa11348ca60dd7ee1 Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Sun, 14 Jul 2013 10:25:06 +1000 Subject: [PATCH 39/51] DOC: shorten example and provide clarifcation and spelling fixes --- README.Solaris | 23 +++-------------------- 1 file changed, 3 insertions(+), 20 deletions(-) diff --git a/README.Solaris b/README.Solaris index 10a5f88c..86f56241 100644 --- a/README.Solaris +++ b/README.Solaris @@ -69,27 +69,10 @@ FAIL2BAN CONFIGURATION OPT: Create /etc/fail2ban/fail2ban.local containing: -# Fail2Ban main configuration file -# -# Comments: use '#' for comment lines and ';' (following a space) for inline comments -# -# Changes: in most of the cases you should not modify this -# file, but provide customizations in fail2ban.local file, e.g.: -# -# [Definition] -# loglevel = 4 +# Fail2Ban configuration file for logging fail2ban on Solaris # [Definition] -# Option: logtarget -# Notes.: Set the log target. This could be a file, SYSLOG, STDERR or STDOUT. -# Only one log target can be specified. -# If you change logtarget from the default value and you are -# using logrotate -- also adjust or disable rotation in the -# corresponding configuration file -# (e.g. /etc/logrotate.d/fail2ban on Debian systems) -# Values: STDOUT STDERR SYSLOG file Default: /var/log/fail2ban.log -# logtarget = /var/adm/fail2ban.log @@ -105,7 +88,7 @@ ignoreregex = for myuser from logpath = /var/adm/auth.log Set the sendmail dest address to something useful or drop the line to stop it spamming you. -Set 'myuser' to your username to avoid banning yourself or drop it. +Set 'myuser' to your username to avoid banning yourself or remove the line. START (OR RESTART) FAIL2BAN @@ -128,7 +111,7 @@ GOTCHAS AND FIXMES svcadm enable fail2ban * If svcs -xv says that fail2ban failed to start or svcs says it's in maintenance mode - chcek /var/svc/log/network-fail2ban:default.log for clues. + check /var/svc/log/network-fail2ban:default.log for clues. Check permissions on /var/adm, /var/adm/auth.log /var/adm/fail2ban.log and /var/run/fail2ban You may need to: From 94376bfbe1a7a097876c21582558425db500bf93 Mon Sep 17 00:00:00 2001 From: Steven Hiscocks Date: Sun, 14 Jul 2013 11:15:45 +0100 Subject: [PATCH 40/51] TST: Handle lack of `json` library in python2.5 for samples test case --- fail2ban-testcases | 16 +++++++++++++--- testcases/samplestestcase.py | 8 +++++++- 2 files changed, 20 insertions(+), 4 deletions(-) diff --git a/fail2ban-testcases b/fail2ban-testcases index 278dd890..8ba697b3 100755 --- a/fail2ban-testcases +++ b/fail2ban-testcases @@ -27,17 +27,26 @@ __license__ = "GPL" import unittest, logging, sys, time, os +if sys.version_info >= (2, 6): + import json +else: + try: + import simplejson as json + except ImportError: + json = None + from common.version import version from testcases import banmanagertestcase from testcases import clientreadertestcase from testcases import failmanagertestcase from testcases import filtertestcase -from testcases import samplestestcase from testcases import servertestcase from testcases import datedetectortestcase from testcases import actiontestcase from testcases import sockettestcase from testcases import misctestcase +if json: + from testcases import samplestestcase from testcases.utils import FormatterWithTraceBack from server.mytime import MyTime @@ -171,8 +180,9 @@ tests.addTest(unittest.makeSuite(filtertestcase.JailTests)) # DateDetector tests.addTest(unittest.makeSuite(datedetectortestcase.DateDetectorTest)) -# Filter Regex tests with sample logs -tests.addTest(unittest.makeSuite(samplestestcase.FilterSamplesRegex)) +if json: + # Filter Regex tests with sample logs + tests.addTest(unittest.makeSuite(samplestestcase.FilterSamplesRegex)) # # Extensive use-tests of different available filters backends diff --git a/testcases/samplestestcase.py b/testcases/samplestestcase.py index 2bb73ca1..656bbffa 100644 --- a/testcases/samplestestcase.py +++ b/testcases/samplestestcase.py @@ -22,7 +22,13 @@ __copyright__ = "Copyright (c) 2013 Steven Hiscocks" __license__ = "GPL" -import unittest, os, fileinput, re, json, datetime +import unittest, sys, os, fileinput, re, datetime + +if sys.version_info >= (2, 6): + import json +else: + import simplejson as json + next = lambda x: x.next() from server.filter import Filter from client.filterreader import FilterReader From 1116f23151706c89e923d7e4db2b818591c27614 Mon Sep 17 00:00:00 2001 From: Steven Hiscocks Date: Sun, 14 Jul 2013 18:19:16 +0100 Subject: [PATCH 41/51] TST: Sample log regex test now warns if no log for a filter Also checks that at least some tests are present --- fail2ban-testcases | 2 ++ testcases/samplestestcase.py | 44 +++++++++++++++++++++++++++--------- 2 files changed, 35 insertions(+), 11 deletions(-) diff --git a/fail2ban-testcases b/fail2ban-testcases index 8ba697b3..5aed6e43 100755 --- a/fail2ban-testcases +++ b/fail2ban-testcases @@ -183,6 +183,8 @@ tests.addTest(unittest.makeSuite(datedetectortestcase.DateDetectorTest)) if json: # Filter Regex tests with sample logs tests.addTest(unittest.makeSuite(samplestestcase.FilterSamplesRegex)) +else: + print "I: Skipping filter samples testing. No simplejson/json module" # # Extensive use-tests of different available filters backends diff --git a/testcases/samplestestcase.py b/testcases/samplestestcase.py index 656bbffa..6b0e9ae0 100644 --- a/testcases/samplestestcase.py +++ b/testcases/samplestestcase.py @@ -22,7 +22,7 @@ __copyright__ = "Copyright (c) 2013 Steven Hiscocks" __license__ = "GPL" -import unittest, sys, os, fileinput, re, datetime +import unittest, sys, os, fileinput, re, datetime, inspect if sys.version_info >= (2, 6): import json @@ -46,6 +46,14 @@ class FilterSamplesRegex(unittest.TestCase): def tearDown(self): """Call after every test case.""" + def testFiltersPresent(self): + """Check to ensure some tests exist""" + self.assertTrue( + len([test for test in inspect.getmembers(self) + if test[0].startswith('testSampleRegexs')]) + >= 10, + "Expected more FilterSampleRegexs tests") + def testSampleRegexsFactory(name): def testFilter(self): @@ -58,8 +66,22 @@ def testSampleRegexsFactory(name): if opt[2] == "addfailregex": self.filter.addFailRegex(opt[3]) + if not self.filter.getFailRegex(): + # No fail regexs set: likely just common file for includes. + return + + # TODO: Remove exception handling once sample logs obtained for all + try: + self.assertTrue( + os.path.isfile(os.path.join(TEST_FILES_DIR, "logs", name)), + "No sample log file available for '%s' filter" % name) + except AssertionError: + print "I: No sample log file available for '%s' filter" % name + return + logFile = fileinput.FileInput( os.path.join(TEST_FILES_DIR, "logs", name)) + for line in logFile: jsonREMatch = re.match("^# ?failJSON:(.+)$", line) if jsonREMatch: @@ -78,13 +100,13 @@ def testSampleRegexsFactory(name): if not ret: # Check line is flagged as none match self.assertFalse(faildata.get('match', True), - "Line not matched when should have: %s:%i" % - (logFile.filename(), logFile.filelineno())) + "Line not matched when should have: %s:%i %r" % + (logFile.filename(), logFile.filelineno(), line)) elif ret: # Check line is flagged to match self.assertTrue(faildata.get('match', False), - "Line matched when shouldn't have: %s:%i" % - (logFile.filename(), logFile.filelineno())) + "Line matched when shouldn't have: %s:%i %r" % + (logFile.filename(), logFile.filelineno(), line)) self.assertEqual(len(ret), 1) # Verify timestamp and host as expected host, time = ret[0] @@ -96,9 +118,9 @@ def testSampleRegexsFactory(name): return testFilter -for filter_ in os.listdir(os.path.join(TEST_FILES_DIR, "logs")): - if os.path.isfile(os.path.join(TEST_FILES_DIR, "logs", filter_)): - setattr( - FilterSamplesRegex, - "testSampleRegexs%s" % filter_.upper(), - testSampleRegexsFactory(filter_)) +for filter_ in os.listdir(os.path.join(CONFIG_DIR, "filter.d")): + filterName = filter_.rpartition(".")[0] + setattr( + FilterSamplesRegex, + "testSampleRegexs%s" % filterName.upper(), + testSampleRegexsFactory(filterName)) From 2c8747cc76e88a58deb93d94fe39e51eaa9b8852 Mon Sep 17 00:00:00 2001 From: Steven Hiscocks Date: Mon, 15 Jul 2013 18:40:32 +0100 Subject: [PATCH 42/51] BF: fail2ban-regex date detector template hits count now correct closes #295 --- fail2ban-regex | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/fail2ban-regex b/fail2ban-regex index 78716a43..e75cb2a3 100755 --- a/fail2ban-regex +++ b/fail2ban-regex @@ -46,6 +46,7 @@ from client.configparserinc import SafeConfigParserWithIncludes from ConfigParser import NoOptionError, NoSectionError, MissingSectionHeaderError from server.filter import Filter from server.failregex import RegexException +from server.datedetector import DateDetector from testcases.utils import FormatterWithTraceBack # Gets the instance of the logger. @@ -172,6 +173,8 @@ class Fail2banRegex(object): self._ignoreregex = list() self._failregex = list() self._line_stats = LineStats() + self._dateDetector = DateDetector() + self._dateDetector.addDefaultTemplate() def readRegex(self, value, regextype): @@ -256,6 +259,9 @@ class Fail2banRegex(object): if line.startswith('# ') or not line.strip(): # skip comment and empty lines continue + + self._dateDetector.matchTime(line) + is_ignored = fail2banRegex.testIgnoreRegex(line) if is_ignored: self._line_stats.ignored_lines.append(line) @@ -312,7 +318,7 @@ class Fail2banRegex(object): print "\nDate template hits:" out = [] - for template in self._filter.dateDetector.getTemplates(): + for template in self._dateDetector.getTemplates(): if self._verbose or template.getHits(): out.append("[%d] %s" % (template.getHits(), template.getName())) pprint_list(out, "[# of hits] date format") From 5bd186b85495319f717bfc94b8eb8888003bff43 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Mon, 15 Jul 2013 13:52:42 -0400 Subject: [PATCH 43/51] ENH(minor): fail2ban-regex comment line doesn't have to have a space after leading # --- fail2ban-regex | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fail2ban-regex b/fail2ban-regex index e75cb2a3..3326e328 100755 --- a/fail2ban-regex +++ b/fail2ban-regex @@ -256,7 +256,7 @@ class Fail2banRegex(object): def process(self, test_lines): for line in test_lines: - if line.startswith('# ') or not line.strip(): + if line.startswith('#') or not line.strip(): # skip comment and empty lines continue From 1a2b6442a069a0aa7f86ff21e4a9dee55b893c1b Mon Sep 17 00:00:00 2001 From: Steven Hiscocks Date: Mon, 15 Jul 2013 22:16:40 +0100 Subject: [PATCH 44/51] ENH+BF+TST: Filter now returns reference to failregex and ignoreregex This avoids duplication of code across fail2ban-regex and samples test cases. This also now more neatly resolves the issue of double counting date templates matches in fail2ban-regex. In addition, the samples test cases now also print a warning message that not all regexs have samples for them, with future plan to change this to an assertion. --- fail2ban-regex | 85 +++++++++++++++--------------------- server/datedetector.py | 1 + server/datetemplate.py | 5 ++- server/filter.py | 42 ++++++++++-------- testcases/samplestestcase.py | 21 +++++++-- 5 files changed, 79 insertions(+), 75 deletions(-) diff --git a/fail2ban-regex b/fail2ban-regex index 3326e328..2c050e0d 100755 --- a/fail2ban-regex +++ b/fail2ban-regex @@ -46,7 +46,6 @@ from client.configparserinc import SafeConfigParserWithIncludes from ConfigParser import NoOptionError, NoSectionError, MissingSectionHeaderError from server.filter import Filter from server.failregex import RegexException -from server.datedetector import DateDetector from testcases.utils import FormatterWithTraceBack # Gets the instance of the logger. @@ -130,7 +129,7 @@ class RegexStat(object): return self._failregex def appendIP(self, value): - self._ipList.extend(value) + self._ipList.append(value) def getIPList(self): return self._ipList @@ -173,8 +172,6 @@ class Fail2banRegex(object): self._ignoreregex = list() self._failregex = list() self._line_stats = LineStats() - self._dateDetector = DateDetector() - self._dateDetector.addDefaultTemplate() def readRegex(self, value, regextype): @@ -204,53 +201,39 @@ class Fail2banRegex(object): regex_values = [RegexStat(value)] setattr(self, "_" + regex, regex_values) + for regex in regex_values: + getattr( + self._filter, + 'add%sRegex' % regextype.title())(regex.getFailRegex()) return True def testIgnoreRegex(self, line): found = False - for regex in self._ignoreregex: - try: - self._filter.addIgnoreRegex(regex.getFailRegex()) - try: - ret = self._filter.ignoreLine(line) - if ret: - found = True - regex.inc() - except RegexException, e: - print e - return False - finally: - self._filter.delIgnoreRegex(0) + try: + ret = self._filter.ignoreLine(line) + if ret is not None: + found = True + regex = self._ignoreregex[ret].inc() + except RegexException, e: + print e + return False return found def testRegex(self, line): - found = False - for regex in self._ignoreregex: - self._filter.addIgnoreRegex(regex.getFailRegex()) - for regex in self._failregex: - try: - self._filter.addFailRegex(regex.getFailRegex()) - try: - ret = self._filter.processLine(line) - if len(ret): - if found == True: - ret[0].append(True) - else: - found = True - ret[0].append(False) - regex.inc() - regex.appendIP(ret) - except RegexException, e: - print e - return False - except IndexError: - print "Sorry, but no found in regex" - return False - finally: - self._filter.delFailRegex(0) - for regex in self._ignoreregex: - self._filter.delIgnoreRegex(0) - return found + try: + ret = self._filter.processLine(line, checkAllRegex=True) + for match in ret: + match.append(len(ret)>1) + regex = self._failregex[match[0]] + regex.inc() + regex.appendIP(match) + except RegexException, e: + print e + return False + except IndexError: + print "Sorry, but no found in regex" + return False + return len(ret) > 0 def process(self, test_lines): @@ -259,9 +242,6 @@ class Fail2banRegex(object): if line.startswith('#') or not line.strip(): # skip comment and empty lines continue - - self._dateDetector.matchTime(line) - is_ignored = fail2banRegex.testIgnoreRegex(line) if is_ignored: self._line_stats.ignored_lines.append(line) @@ -302,10 +282,13 @@ class Fail2banRegex(object): if self._verbose and len(failregex.getIPList()): for ip in failregex.getIPList(): - timeTuple = time.localtime(ip[1]) + timeTuple = time.localtime(ip[2]) timeString = time.strftime("%a %b %d %H:%M:%S %Y", timeTuple) - out.append(" %s %s%s" % ( - ip[0], timeString, ip[2] and " (already matched)" or "")) + out.append( + " %s %s%s" % ( + ip[1], + timeString, + ip[3] and " (multiple regex matched)" or "")) print "\n%s: %d total" % (title, total) pprint_list(out, " #) [# of hits] regular expression") @@ -318,7 +301,7 @@ class Fail2banRegex(object): print "\nDate template hits:" out = [] - for template in self._dateDetector.getTemplates(): + for template in self._filter.dateDetector.getTemplates(): if self._verbose or template.getHits(): out.append("[%d] %s" % (template.getHits(), template.getName())) pprint_list(out, "[# of hits] date format") diff --git a/server/datedetector.py b/server/datedetector.py index 0c8b4df2..0ed9e00a 100644 --- a/server/datedetector.py +++ b/server/datedetector.py @@ -174,6 +174,7 @@ class DateDetector: match = template.matchDate(line) if not match is None: logSys.debug("Matched time template %s" % template.getName()) + template.incHits() return match return None finally: diff --git a/server/datetemplate.py b/server/datetemplate.py index 8c49aa15..86eeee8e 100644 --- a/server/datetemplate.py +++ b/server/datetemplate.py @@ -59,11 +59,12 @@ class DateTemplate: def getHits(self): return self.__hits + + def incHits(self): + self.__hits += 1 def matchDate(self, line): dateMatch = self.__cRegex.search(line) - if not dateMatch is None: - self.__hits += 1 return dateMatch def getDate(self, line): diff --git a/server/filter.py b/server/filter.py index d0390e62..ae12968e 100644 --- a/server/filter.py +++ b/server/filter.py @@ -284,7 +284,7 @@ class Filter(JailThread): return False - def processLine(self, line, returnRawHost=False): + def processLine(self, line, returnRawHost=False, checkAllRegex=False): """Split the time portion from log msg and return findFailures on them """ try: @@ -306,14 +306,15 @@ class Filter(JailThread): else: timeLine = l logLine = l - return self.findFailure(timeLine, logLine, returnRawHost) + return self.findFailure(timeLine, logLine, returnRawHost, checkAllRegex) def processLineAndAdd(self, line): """Processes the line for failures and populates failManager """ for element in self.processLine(line): - ip = element[0] - unixTime = element[1] + failregex = element[0] + ip = element[1] + unixTime = element[2] logSys.debug("Processing line with time:%s and ip:%s" % (unixTime, ip)) if unixTime < MyTime.time() - self.getFindTime(): @@ -335,11 +336,11 @@ class Filter(JailThread): # @return: a boolean def ignoreLine(self, line): - for ignoreRegex in self.__ignoreRegex: + for ignoreRegexIndex, ignoreRegex in enumerate(self.__ignoreRegex): ignoreRegex.search(line) if ignoreRegex.hasMatched(): - return True - return False + return ignoreRegexIndex + return None ## # Finds the failure in a line given split into time and log parts. @@ -348,18 +349,19 @@ class Filter(JailThread): # to find the logging time. # @return a dict with IP and timestamp. - def findFailure(self, timeLine, logLine, returnRawHost=False): + def findFailure(self, timeLine, logLine, + returnRawHost=False, checkAllRegex=False): failList = list() # Checks if we must ignore this line. - if self.ignoreLine(logLine): + if self.ignoreLine(logLine) is not None: # The ignoreregex matched. Return. return failList + date = self.dateDetector.getUnixTime(timeLine) # Iterates over all the regular expressions. - for failRegex in self.__failRegex: + for failRegexIndex, failRegex in enumerate(self.__failRegex): failRegex.search(logLine) if failRegex.hasMatched(): # The failregex matched. - date = self.dateDetector.getUnixTime(timeLine) logSys.log(7, "Date: %r, message: %r", timeLine, logLine) if date is None: @@ -372,14 +374,16 @@ class Filter(JailThread): try: host = failRegex.getHost() if returnRawHost: - failList.append([host, date]) - break - ipMatch = DNSUtils.textToIp(host, self.__useDns) - if ipMatch: - for ip in ipMatch: - failList.append([ip, date]) - # We matched a regex, it is enough to stop. - break + failList.append([failRegexIndex, host, date]) + if not checkAllRegex: + break + else: + ipMatch = DNSUtils.textToIp(host, self.__useDns) + if ipMatch: + for ip in ipMatch: + failList.append([failRegexIndex, ip, date]) + if not checkAllRegex: + break except RegexException, e: # pragma: no cover - unsure if reachable logSys.error(e) return failList diff --git a/testcases/samplestestcase.py b/testcases/samplestestcase.py index 6b0e9ae0..a52873b9 100644 --- a/testcases/samplestestcase.py +++ b/testcases/samplestestcase.py @@ -82,6 +82,7 @@ def testSampleRegexsFactory(name): logFile = fileinput.FileInput( os.path.join(TEST_FILES_DIR, "logs", name)) + regexsUsed = set() for line in logFile: jsonREMatch = re.match("^# ?failJSON:(.+)$", line) if jsonREMatch: @@ -96,7 +97,8 @@ def testSampleRegexsFactory(name): else: faildata = {} - ret = self.filter.processLine(line, returnRawHost=True) + ret = self.filter.processLine( + line, returnRawHost=True, checkAllRegex=True) if not ret: # Check line is flagged as none match self.assertFalse(faildata.get('match', True), @@ -107,15 +109,28 @@ def testSampleRegexsFactory(name): self.assertTrue(faildata.get('match', False), "Line matched when shouldn't have: %s:%i %r" % (logFile.filename(), logFile.filelineno(), line)) - self.assertEqual(len(ret), 1) + self.assertEqual(len(ret), 1, "Multiple regexs matched") # Verify timestamp and host as expected - host, time = ret[0] + failregex, host, time = ret[0] self.assertEqual(host, faildata.get("host", None)) self.assertEqual( datetime.datetime.fromtimestamp(time), datetime.datetime.strptime( faildata.get("time", None), "%Y-%m-%dT%H:%M:%S")) + regexsUsed.add(failregex) + + # TODO: Remove exception handling once all regexs have samples + for failRegexIndex, failRegex in enumerate(self.filter.getFailRegex()): + try: + self.assertTrue( + failRegexIndex in regexsUsed, + "Regex for filter '%s' has no samples: %i: %r" % + (name, failRegexIndex, failRegex)) + except AssertionError: + print "I: Regex for filter '%s' has no samples: %i: %r" % ( + name, failRegexIndex, failRegex) + return testFilter for filter_ in os.listdir(os.path.join(CONFIG_DIR, "filter.d")): From 148cbd8d2a2d3cab6815fc100a4e635c0e6a10ed Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 9 Jul 2013 12:47:41 -0400 Subject: [PATCH 45/51] ENH: heavier debugging -- log split date/log line even for no match. Log matching regex upon match --- server/failregex.py | 3 ++- server/filter.py | 7 ++++--- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/server/failregex.py b/server/failregex.py index 957c77db..d302370b 100644 --- a/server/failregex.py +++ b/server/failregex.py @@ -50,7 +50,8 @@ class Regex: except sre_constants.error: raise RegexException("Unable to compile regular expression '%s'" % regex) - + def __str__(self): + return "%s(%r)" % (self.__class__.__name__, self._regex) ## # Gets the regular expression. # diff --git a/server/filter.py b/server/filter.py index d0390e62..54efb619 100644 --- a/server/filter.py +++ b/server/filter.py @@ -294,7 +294,7 @@ class Filter(JailThread): l = line l = l.rstrip('\r\n') - logSys.log(5, "Working on line %r", l) + logSys.log(7, "Working on line %r", l) timeMatch = self.dateDetector.matchTime(l) if timeMatch: # Lets split into time part and log part of the line @@ -349,19 +349,20 @@ class Filter(JailThread): # @return a dict with IP and timestamp. def findFailure(self, timeLine, logLine, returnRawHost=False): + logSys.log(5, "Date: %r, message: %r", timeLine, logLine) failList = list() # Checks if we must ignore this line. if self.ignoreLine(logLine): # The ignoreregex matched. Return. + logSys.log(7, "Matched ignoreregex and was ignored") return failList # Iterates over all the regular expressions. for failRegex in self.__failRegex: failRegex.search(logLine) if failRegex.hasMatched(): # The failregex matched. + logSys.log(7, "Matched %s", failRegex) date = self.dateDetector.getUnixTime(timeLine) - logSys.log(7, "Date: %r, message: %r", - timeLine, logLine) if date is None: logSys.debug("Found a match for %r but no valid date/time " "found for %r. Please file a detailed issue on" From 8add63c733f997feb0a3dd321cdf205ce22c3f04 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 9 Jul 2013 12:48:12 -0400 Subject: [PATCH 46/51] ENH: anchor roundcube-auth at the beginning as well --- config/filter.d/roundcube-auth.conf | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/config/filter.d/roundcube-auth.conf b/config/filter.d/roundcube-auth.conf index 7b153f44..77d872fb 100644 --- a/config/filter.d/roundcube-auth.conf +++ b/config/filter.d/roundcube-auth.conf @@ -4,6 +4,10 @@ # # +[INCLUDES] + +before = common.conf + [Definition] # Option: failregex @@ -13,7 +17,7 @@ # (?:::f{4,6}:)?(?P[\w\-.^_]+) # Values: TEXT # -failregex = (FAILED login|Login failed) for .* from \s*$ +failregex = ^\s*(\[(\s\+[0-9]{4})?\])?(%(__hostname)s roundcube: IMAP Error)?: (FAILED login|Login failed) for .* from \s*$ # Option: ignoreregex # Notes.: regex to ignore. If this regex matches, the line is ignored. From 0a02cfe9e8e65e68ec438bcd1c10fb7493f3e842 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 16 Jul 2013 14:39:42 -0400 Subject: [PATCH 47/51] ENH: must end with alphanumeric \w (not a dot or a dash etc) Otherwise regexp might swallow period in the sentence right after the address. I have decided to enforce alphanumeric instead of switching to non-greedy +? ... because I think it is closer to what we actually want here --- server/failregex.py | 2 +- testcases/servertestcase.py | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/server/failregex.py b/server/failregex.py index d302370b..bfa50575 100644 --- a/server/failregex.py +++ b/server/failregex.py @@ -41,7 +41,7 @@ class Regex: self._matchCache = None # Perform shortcuts expansions. # Replace "" with default regular expression for host. - regex = regex.replace("", "(?:::f{4,6}:)?(?P[\w\-.^_]+)") + regex = regex.replace("", "(?:::f{4,6}:)?(?P[\w\-.^_]*\w)") if regex.lstrip() == '': raise RegexException("Cannot add empty regex") try: diff --git a/testcases/servertestcase.py b/testcases/servertestcase.py index 0a5593e3..07cc373c 100644 --- a/testcases/servertestcase.py +++ b/testcases/servertestcase.py @@ -334,9 +334,9 @@ class Transmitter(TransmitterBase): "failed attempt from again", ], [ - "user john at (?:::f{4,6}:)?(?P[\w\-.^_]+)", - "Admin user login from (?:::f{4,6}:)?(?P[\w\-.^_]+)", - "failed attempt from (?:::f{4,6}:)?(?P[\w\-.^_]+) again", + "user john at (?:::f{4,6}:)?(?P[\w\-.^_]*\\w)", + "Admin user login from (?:::f{4,6}:)?(?P[\w\-.^_]*\\w)", + "failed attempt from (?:::f{4,6}:)?(?P[\w\-.^_]*\\w) again", ], self.jailName ) @@ -359,7 +359,7 @@ class Transmitter(TransmitterBase): ], [ "user john", - "Admin user login from (?:::f{4,6}:)?(?P[\w\-.^_]+)", + "Admin user login from (?:::f{4,6}:)?(?P[\w\-.^_]*\\w)", "Dont match me!", ], self.jailName From f6a8a04cf34b11e317ac8b90970f9ae6518980e9 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 16 Jul 2013 14:45:59 -0400 Subject: [PATCH 48/51] ENH: roundcube-auth - adopt for current format with trailing error message. thanks @kwirk for the review/feedback I also used non-greedy .*? for the login portion since not sure if space could be there and trying to minimize possibility of reacting on injected "from " somewhere within the trailing .* --- config/filter.d/roundcube-auth.conf | 2 +- testcases/files/logs/roundcube-auth | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/config/filter.d/roundcube-auth.conf b/config/filter.d/roundcube-auth.conf index 77d872fb..fe669f66 100644 --- a/config/filter.d/roundcube-auth.conf +++ b/config/filter.d/roundcube-auth.conf @@ -17,7 +17,7 @@ before = common.conf # (?:::f{4,6}:)?(?P[\w\-.^_]+) # Values: TEXT # -failregex = ^\s*(\[(\s\+[0-9]{4})?\])?(%(__hostname)s roundcube: IMAP Error)?: (FAILED login|Login failed) for .* from \s*$ +failregex = ^\s*(\[(\s\+[0-9]{4})?\])?(%(__hostname)s roundcube: IMAP Error)?: (FAILED login|Login failed) for .*? from (\. AUTHENTICATE .*)?\s*$ # Option: ignoreregex # Notes.: regex to ignore. If this regex matches, the line is ignored. diff --git a/testcases/files/logs/roundcube-auth b/testcases/files/logs/roundcube-auth index dec8051e..7c16efbd 100644 --- a/testcases/files/logs/roundcube-auth +++ b/testcases/files/logs/roundcube-auth @@ -2,3 +2,5 @@ [22-Jan-2013 22:28:21 +0200]: FAILED login for user1 from 192.0.43.10 # failJSON: { "time": "2005-05-26T07:12:40", "match": true , "host": "10.1.1.47" } May 26 07:12:40 hamster roundcube: IMAP Error: Login failed for sales@example.com from 10.1.1.47 +# failJSON: { "time": "2005-07-11T03:06:37", "match": true , "host": "1.2.3.4" } +Jul 11 03:06:37 myhostname roundcube: IMAP Error: Login failed for admin from 1.2.3.4. AUTHENTICATE PLAIN: A0002 NO Login failed. in /usr/share/roundcube/program/include/rcube_imap.php on line 205 (POST /wmail/?_task=login&_action=login) From 90ec82669c696894dfeb0d1c71dc01f415be15e7 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 16 Jul 2013 15:10:41 -0400 Subject: [PATCH 49/51] DOC: changelog entries for preceeding changes --- ChangeLog | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/ChangeLog b/ChangeLog index 16c93db1..a6a6ab23 100644 --- a/ChangeLog +++ b/ChangeLog @@ -29,7 +29,7 @@ ver. 0.8.11 (2013/XX/XXX) - loves-unittests Daniel Black * action.d/hostsdeny -- NOTE: new dependancy 'ed'. Switched to use 'ed' across all platforms to ensure permissions are the same before and after a ban - - closes gh-266 + closes gh-266 - New Features: Daniel Black & ykimon * filter.d/3proxy.conf -- filter added @@ -51,6 +51,8 @@ ver. 0.8.11 (2013/XX/XXX) - loves-unittests * fail2ban-client -- log to standard error. Closes gh-264 * Fail to configure if not a single log file was found for an enabled jail. Closes gh-63 + * is now enforced to end with an alphanumeric + * filter.d/roundcube-auth.conf -- anchored version Alexander Dietrich * action.d/sendmail-common.conf -- added common sendmail settings file and made the sender display name configurable From c2bdfefb626899c7c249dfc37e93c994377b568f Mon Sep 17 00:00:00 2001 From: Steven Hiscocks Date: Tue, 16 Jul 2013 20:58:44 +0100 Subject: [PATCH 50/51] DOC: Comment to fail2ban-regex - flagging lines matched multiple regexs --- fail2ban-regex | 2 ++ 1 file changed, 2 insertions(+) diff --git a/fail2ban-regex b/fail2ban-regex index 2c050e0d..e19b1bc9 100755 --- a/fail2ban-regex +++ b/fail2ban-regex @@ -223,6 +223,8 @@ class Fail2banRegex(object): try: ret = self._filter.processLine(line, checkAllRegex=True) for match in ret: + # Append True/False flag depending if line was matched by + # more than one regex match.append(len(ret)>1) regex = self._failregex[match[0]] regex.inc() From ba29f6bef3cb1757155e7a2bae475eac6afb3aa0 Mon Sep 17 00:00:00 2001 From: Steven Hiscocks Date: Tue, 16 Jul 2013 21:11:10 +0100 Subject: [PATCH 51/51] DOC: Update doc in reference to changes for sample testcases --- DEVELOP | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/DEVELOP b/DEVELOP index 2d7a043b..b0f7b47e 100644 --- a/DEVELOP +++ b/DEVELOP @@ -39,8 +39,9 @@ Filters * Include sample logs with 1.2.3.4 used for IP addresses and example.com/example.org used for DNS names -* Ensure ./fail2ban-regex testcases/files/logs/{samplelog} config/filter.d/{filter}.conf - has matches for EVERY regex +* Ensure sample log is provided in testcases/files/logs/ with same name as the + filter. Each log line should include match meta data for time & IP above + every line (see other sample log files for examples) * Ensure regexs start with a ^ and are restrictive as possible. E.g. not .* if \d+ is sufficient * Use the functionality of regexs http://docs.python.org/2/library/re.html