Merge remote-tracking branch 'remotes/gh-upstream/0.10' into 0.10-full

ChangeLog

@@ -15,8 +15,18 @@ TODO: implementing of options resp. other tasks from PR #1346
 ### Fixes
 * `filter.d/pam-generic.conf`:
   - [grave] injection on user name to host fixed
+* `filter.d/sshd.conf`:
+  - rewritten using `prefregex` and used MLFID-related multi-line parsing
+    (by using tag `<F-MLFID>` instead of buffering with `maxlines`);
+  - optional parameter `mode` rewritten: normal (default), ddos, extra or aggressive (combines all),
+    see sshd for regex details)
+* filter.d/sendmail-reject.conf:
+  - rewritten using `prefregex` and used MLFID-related multi-line parsing;
+  - optional parameter `mode` introduced: normal (default), extra or aggressive
 * `action.d/complain.conf`
   - fixed using new tag `<ip-rev>` (sh/dash compliant now)
+* `action.d/sendmail-geoip-lines.conf`
+  - fixed using new tag `<ip-host>` (without external command execution)
 
 ### New Features
 * New Actions:
@@ -44,7 +54,16 @@ TODO: implementing of options resp. other tasks from PR #1346
   to re.sub with callable)
 * substituteRecursiveTags optimization + moved in helpers facilities (because currently used 
   commonly in server and in client)
-* Provides new tag `<ip-rev>` for PTR reversed representation of IP address
+* New tags (usable in actions):
+  - `<fid>` - failure identifier (if raw resp. failures without IP address)
+  - `<ip-rev>` - PTR reversed representation of IP address
+  - `<ip-host>` - host name of the IP address
+  - `<F-...>` - interpolates to the corresponding filter group capture `...`
+* Allow to use filter options by `fail2ban-regex`, example:
+  fail2ban-regex text.log "sshd[mode=aggressive]"
+* Samples test case factory extended with filter options - dict in JSON to control 
+  filter options (e. g. mode, etc.):
+  # filterOptions: {"mode": "aggressive"}
 
 
 ver. 0.10.0-alpha-1 (2016/07/14) - ipv6-support-etc
@@ -128,6 +147,10 @@ ver. 0.10.0-alpha-1 (2016/07/14) - ipv6-support-etc
     if configuration is clean (fails by wrong configured jails if option `-t` specified)
 * New command action parameter `actionrepair` - command executed in order to restore
   sane environment in error case of `actioncheck`.
+* Reporting via abuseipdb.com:
+  - Bans can now be reported to abuseipdb
+  - Catagories must be set in the config
+  - Relevant log lines included in report
 
 ### Enhancements
 * Huge increasing of fail2ban performance and especially test-cases performance (see gh-1109)
@@ -268,6 +291,8 @@ releases.
       and suffix (logged from several ssh versions), according to gh-1206;
 * filter.d/suhosin.conf
     - greedy catch-all before `<HOST>` fixed (potential vulnerability)
+* filter.d/cyrus-imap.conf
+    - accept entries without login-info resp. hostname before IP address (gh-1707)
 * Filter tests extended with check of all config-regexp, that contains greedy catch-all
   before `<HOST>`, that is hard-anchored at end or precise  sub expression after `<HOST>`
 

MANIFEST

@@ -138,7 +138,6 @@ config/filter.d/solid-pop3d.conf
 config/filter.d/squid.conf
 config/filter.d/squirrelmail.conf
 config/filter.d/sshd.conf
-config/filter.d/sshd-ddos.conf
 config/filter.d/stunnel.conf
 config/filter.d/suhosin.conf
 config/filter.d/tine20.conf
@@ -329,7 +328,6 @@ fail2ban/tests/files/logs/solid-pop3d
 fail2ban/tests/files/logs/squid
 fail2ban/tests/files/logs/squirrelmail
 fail2ban/tests/files/logs/sshd
-fail2ban/tests/files/logs/sshd-ddos
 fail2ban/tests/files/logs/stunnel
 fail2ban/tests/files/logs/suhosin
 fail2ban/tests/files/logs/tine20

THANKS

@@ -16,6 +16,7 @@ Alexander Koeppe (IPv6 support)
 Alexandre Perrin (kAworu)
 Amir Caspi
 Amy
+Andrew James Collett (ajcollett)
 Andrew St. Jean
 Andrey G. Grozin
 Andy Fragen
@@ -111,6 +112,7 @@ Sean DuBois
 Sebastian Arcus
 Serg G. Brester (sebres)
 Sergey Safarov
+Shaun C.
 Sireyessire
 silviogarbes
 Stefan Tatschner

config/action.d/abuseipdb.conf

@@ -0,0 +1,105 @@
+# Fail2ban configuration file
+#
+# Action to report IP address to abuseipdb.com
+# You must sign up to obtain an API key from abuseipdb.com.
+#
+# NOTE: These reports may include sensitive Info.
+# If you want cleaner reports that ensure no user data see the helper script at the below website.
+#
+# IMPORTANT:
+#
+# Reporting an IP of abuse is a serious complaint. Make sure that it is
+# serious. Fail2ban developers and network owners recommend you only use this
+# action for:
+#   * The recidive where the IP has been banned multiple times
+#   * Where maxretry has been set quite high, beyond the normal user typing
+#     password incorrectly.
+#   * For filters that have a low likelihood of receiving human errors
+#
+# This action relies on a api_key being added to the above action conf,
+# and the appropriate categories set.
+#
+# Example, for ssh bruteforce (in section [sshd] of `jail.local`): 
+#   action = %(known/action)s
+#            %(action_abuseipdb)s[abuseipdb_apikey="my-api-key", abuseipdb_category="18,22"]
+#
+# See below for catagories.
+#
+# Original Ref: https://wiki.shaunc.com/wikka.php?wakka=ReportingToAbuseIPDBWithFail2Ban
+# Added to fail2ban by Andrew James Collett (ajcollett)
+
+## abuseIPDB Catagories, `the abuseipdb_category` MUST be set in the jail.conf action call.
+# Example, for ssh bruteforce: action = %(action_abuseipdb)s[abuseipdb_category="18,22"]
+# ID	Title	Description
+# 3	  Fraud Orders
+# 4	  DDoS Attack
+# 9	  Open Proxy
+# 10	Web Spam
+# 11	Email Spam
+# 14	Port Scan
+# 18	Brute-Force
+# 19	Bad Web Bot
+# 20	Exploited Host
+# 21	Web App Attack
+# 22	SSH	Secure Shell (SSH) abuse. Use this category in combination with more specific categories.
+# 23	IoT Targeted
+# See https://abuseipdb.com/categories for more descriptions
+
+[Definition]
+
+# Option:  actionstart
+# Notes.:  command executed once at the start of Fail2Ban.
+# Values:  CMD
+#
+actionstart =
+
+# Option:  actionstop
+# Notes.:  command executed once at the end of Fail2Ban
+# Values:  CMD
+#
+actionstop =
+
+# Option:  actioncheck
+# Notes.:  command executed once before each actionban command
+# Values:  CMD
+#
+actioncheck =
+
+# Option:  actionban
+# Notes.:  command executed when banning an IP. Take care that the
+#          command is executed with Fail2Ban user rights.
+#
+#          ** IMPORTANT! **
+#
+#          By default, this posts directly to AbuseIPDB's API, unfortunately
+#          this results in a lot of backslashes/escapes appearing in the
+#          reports. This also may include info like your hostname.
+#          If you have your own web server with PHP available, you can
+#          use my (Shaun's) helper PHP script by commenting out the first #actionban
+#          line below, uncommenting the second one, and pointing the URL at
+#          wherever you install the helper script. For the PHP helper script, see
+#          <https://wiki.shaunc.com/wikka.php?wakka=ReportingToAbuseIPDBWithFail2Ban>
+#
+#          --ciphers ecdhe_ecdsa_aes_256_sha is used to workaround a
+#          "NSS error -12286" from curl as it attempts to connect using
+#          SSLv3. See https://www.centos.org/forums/viewtopic.php?t=52732
+# Tags:    See jail.conf(5) man page
+# Values:  CMD
+#
+actionban = curl --fail --ciphers ecdhe_ecdsa_aes_256_sha --data 'key=<abuseipdb_apikey>' --data-urlencode 'comment=<matches>' --data 'ip=<ip>' --data 'category=<abuseipdb_category>' "https://www.abuseipdb.com/report/json"
+
+# Option:  actionunban
+# Notes.:  command executed when unbanning an IP. Take care that the
+#          command is executed with Fail2Ban user rights.
+# Tags:    See jail.conf(5) man page
+# Values:  CMD
+#
+actionunban =
+
+[Init]
+# Option:  abuseipdb_apikey
+# Notes    Your API key from abuseipdb.com
+# Values:  STRING  Default: None
+# Register for abuseipdb [https://www.abuseipdb.com], get api key and set below.
+# You will need to set the catagory in the action call.
+abuseipdb_apikey =

config/action.d/sendmail-geoip-lines.conf

@@ -36,7 +36,7 @@ actionban = ( printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from `uname -n
             http://whois.domaintools.com/<ip>\n\n
             Country:`geoiplookup -f /usr/share/GeoIP/GeoIP.dat "<ip>" | cut -d':' -f2-`
             AS:`geoiplookup -f /usr/share/GeoIP/GeoIPASNum.dat "<ip>" | cut -d':' -f2-`
-            hostname: `host -t A <ip> 2>&1`\n\n
+            hostname: <ip-host>\n\n
             Lines containing failures of <ip>\n";
             %(_grep_logs)s;
             printf %%b "\n

config/filter.d/cyrus-imap.conf

@@ -13,7 +13,7 @@ before = common.conf
 
 _daemon = (?:cyrus/)?(?:imap(d|s)?|pop3(d|s)?)
 
-failregex = ^%(__prefix_line)sbadlogin: \S+ ?\[<HOST>\] \S+ .*?\[?SASL\(-13\): (authentication failure|user not found): .*\]?$
+failregex = ^%(__prefix_line)sbadlogin: [^\[]*\[<HOST>\] \S+ .*?\[?SASL\(-13\): (authentication failure|user not found): .*\]?$
 
 ignoreregex = 
 

config/filter.d/sendmail-reject.conf

@@ -21,30 +21,45 @@ before = common.conf
 
 _daemon = (?:(sm-(mta|acceptingconnections)|sendmail))
 
-failregex = ^%(__prefix_line)s\w{14}: ruleset=check_rcpt, arg1=(?P<email><\S+@\S+>), relay=(\S+ )?\[<HOST>\]( \(may be forged\))?, reject=(550 5\.7\.1 (?P=email)\.\.\. Relaying denied\. (IP name possibly forged \[(\d+\.){3}\d+\]|Proper authentication required\.|IP name lookup failed \[(\d+\.){3}\d+\])|553 5\.1\.8 (?P=email)\.\.\. Domain of sender address \S+ does not exist|550 5\.[71]\.1 (?P=email)\.\.\. (Rejected: .*|User unknown))$
+prefregex = ^<F-MLFID>%(__prefix_line)s(?:\w{14}: )?</F-MLFID><F-CONTENT>.+</F-CONTENT>$
-            ^%(__prefix_line)sruleset=check_relay, arg1=(?P<dom>\S+), arg2=<HOST>, relay=((?P=dom) )?\[(\d+\.){3}\d+\]( \(may be forged\))?, reject=421 4\.3\.2 (Connection rate limit exceeded\.|Too many open connections\.)$
-            ^%(__prefix_line)s\w{14}: rejecting commands from (\S* )?\[<HOST>\] due to pre-greeting traffic after \d+ seconds$
-            ^%(__prefix_line)s\w{14}: (\S+ )?\[<HOST>\]: ((?i)expn|vrfy) \S+ \[rejected\]$
-            ^(?P<__prefix>%(__prefix_line)s\w+: )<[^@]+@[^>]+>\.\.\. No such user here$<SKIPLINES>^(?P=__prefix)from=<[^@]+@[^>]+>, size=\d+, class=\d+, nrcpts=\d+, bodytype=\w+, proto=E?SMTP, daemon=MTA, relay=\S+ \[<HOST>\]$
 
+cmnfailre = ^ruleset=check_rcpt, arg1=(?P<email><\S+@\S+>), relay=(\S+ )?\[<HOST>\](?: \(may be forged\))?, reject=(550 5\.7\.1 (?P=email)\.\.\. Relaying denied\. (IP name possibly forged \[(\d+\.){3}\d+\]|Proper authentication required\.|IP name lookup failed \[(\d+\.){3}\d+\])|553 5\.1\.8 (?P=email)\.\.\. Domain of sender address \S+ does not exist|550 5\.[71]\.1 (?P=email)\.\.\. (Rejected: .*|User unknown))$
+            ^ruleset=check_relay, arg1=(?P<dom>\S+), arg2=<HOST>, relay=((?P=dom) )?\[(\d+\.){3}\d+\](?: \(may be forged\))?, reject=421 4\.3\.2 (Connection rate limit exceeded\.|Too many open connections\.)$
+            ^rejecting commands from (\S* )?\[<HOST>\] due to pre-greeting traffic after \d+ seconds$
+            ^(?:\S+ )?\[<HOST>\]: (?:(?i)expn|vrfy) \S+ \[rejected\]$
+            ^<[^@]+@[^>]+>\.\.\. No such user here$
+            ^<F-NOFAIL>from=<[^@]+@[^>]+></F-NOFAIL>, size=\d+, class=\d+, nrcpts=\d+, bodytype=\w+, proto=E?SMTP, daemon=MTA, relay=\S+ \[<HOST>\]$
 
-ignoreregex =
+mdre-normal =
 
+mdre-extra = ^(?:\S+ )?\[<HOST>\](?: \(may be forged\))? did not issue (?:[A-Z]{4}[/ ]?)+during connection to M(?:TA|SP)(?:-\w+)?$
 
-[Init]
+mdre-aggressive = %(mdre-extra)s
+
+failregex = %(cmnfailre)s
+            <mdre-<mode>>
+
+# Parameter "mode": normal (default), extra or aggressive
+# Usage example (for jail.local):
+#   [sendmail-reject]
+#   filter = sendmail-reject[mode=extra]
+#
+mode = normal
+
+ignoreregex = 
 
-# "maxlines" is number of log lines to buffer for multi-line regex searches
-maxlines = 10
 
 # DEV NOTES:
 # 
-# Regarding the last multiline regex:
+# Regarding the multiline regex:
 #
-# There can be a nunber of non-related lines between the first and second part
+# "No such user" lines generate a failure and needs to be matched together with
-# of this regex maxlines of 10 is quite generious. Only one of the 
+# another line with the HOST, therefore no-failure line was added as regex, that
-# "No such user" lines needs to be matched before the line with the HOST.
+# contains HOST (see line with tag <F-NOFAIL>).
 #
-# Note the capture __prefix, includes both the __prefix_lines (which includes
+# Note the capture <F-MLFID>, includes both the __prefix_lines (which includes
-# the sendmail PID), but also the \w+ which the the sendmail assigned mail ID.
+# the sendmail PID), but also the `\w{14}` which the the sendmail assigned 
+# mail ID (todo: check this is necessary, possible obsolete).
 #
-# Author: Daniel Black and Fabian Wenk
+# Author: Daniel Black, Fabian Wenk and Sergey Brester aka sebres.
+# Rewritten using prefregex by Serg G. Brester.

config/filter.d/sshd-aggressive.conf

@@ -1,11 +0,0 @@
-# Fail2Ban aggressive ssh filter for at attempted exploit
-#
-# Includes failregex of both sshd and sshd-ddos filters
-#
-[INCLUDES]
-
-before = sshd.conf
-
-[Definition]
-
-mode = %(aggressive)s

config/filter.d/sshd-ddos.conf

@@ -1,17 +0,0 @@
-# Fail2Ban ssh filter for at attempted exploit
-#
-# The regex here also relates to a exploit:
-#
-#  http://www.securityfocus.com/bid/17958/exploit
-#  The example code here shows the pushing of the exploit straight after
-#  reading the server version. This is where the client version string normally
-#  pushed. As such the server will read this unparsible information as
-#  "Did not receive identification string".
-
-[INCLUDES]
-
-before = sshd.conf
-
-[Definition]
-
-mode = %(ddos)s

config/filter.d/sshd.conf

@@ -24,45 +24,59 @@ __pref = (?:(?:error|fatal): (?:PAM: )?)?
 __suff = (?: \[preauth\])?\s*
 __on_port_opt = (?: port \d+)?(?: on \S+(?: port \d+)?)?
 
-prefregex = ^<F-MLFID>%(__prefix_line)s</F-MLFID>%(__pref)s<F-CONTENT>.+</F-CONTENT>$
-
-mode = %(normal)s
-
-normal = ^[aA]uthentication (?:failure|error|failed) for <F-USER>.*</F-USER> from <HOST>( via \S+)?\s*%(__suff)s$
-         ^User not known to the underlying authentication module for <F-USER>.*</F-USER> from <HOST>\s*%(__suff)s$
-         ^Failed \S+ for (?P<cond_inv>invalid user )?<F-USER>(?P<cond_user>\S+)|(?(cond_inv)(?:(?! from ).)*?|[^:]+)</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$)
-         ^<F-USER>ROOT</F-USER> LOGIN REFUSED.* FROM <HOST>\s*%(__suff)s$
-         ^[iI](?:llegal|nvalid) user <F-USER>.*?</F-USER> from <HOST>%(__on_port_opt)s\s*$
-         ^User <F-USER>.+</F-USER> from <HOST> not allowed because not listed in AllowUsers\s*%(__suff)s$
-         ^User <F-USER>.+</F-USER> from <HOST> not allowed because listed in DenyUsers\s*%(__suff)s$
-         ^User <F-USER>.+</F-USER> from <HOST> not allowed because not in any group\s*%(__suff)s$
-         ^refused connect from \S+ \(<HOST>\)\s*%(__suff)s$
-         ^Received disconnect from <HOST>%(__on_port_opt)s:\s*3: .*: Auth fail%(__suff)s$
-         ^User <F-USER>.+</F-USER> from <HOST> not allowed because a group is listed in DenyGroups\s*%(__suff)s$
-         ^User <F-USER>.+</F-USER> from <HOST> not allowed because none of user's groups are listed in AllowGroups\s*%(__suff)s$
-         ^pam_unix\(sshd:auth\):\s+authentication failure;\s*logname=\S*\s*uid=\d*\s*euid=\d*\s*tty=\S*\s*ruser=<F-USER>\S*</F-USER>\s*rhost=<HOST>\s.*%(__suff)s$
-         ^(error: )?maximum authentication attempts exceeded for <F-USER>.*</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)? \[preauth\]$
-         ^User <F-USER>.+</F-USER> not allowed because account is locked%(__suff)s
-         ^Disconnecting: Too many authentication failures for <F-USER>.+?</F-USER>%(__suff)s
-         ^<F-NOFAIL>Received disconnect</F-NOFAIL> from <HOST>: 11:
-         ^<F-NOFAIL>Connection closed</F-NOFAIL> by <HOST>%(__suff)s$
-
-ddos =   ^Did not receive identification string from <HOST>%(__suff)s$
-         ^Received disconnect from <HOST>%(__on_port_opt)s:\s*14: No supported authentication methods available%(__suff)s$
-         ^Unable to negotiate with <HOST>%(__on_port_opt)s: no matching (?:cipher|key exchange method) found.
-         ^Unable to negotiate a (?:cipher|key exchange method)%(__suff)s$
-         ^<F-NOFAIL>SSH: Server;Ltype:</F-NOFAIL> (?:Authname|Version|Kex);Remote: <HOST>-\d+;[A-Z]\w+:
-         ^Read from socket failed: Connection reset by peer \[preauth\]
-
-common = ^<F-NOFAIL>Connection from</F-NOFAIL> <HOST>
-
-aggressive = %(normal)s
-             %(ddos)s
-
 [Definition]
 
-failregex = %(mode)s
+prefregex = ^<F-MLFID>%(__prefix_line)s</F-MLFID>%(__pref)s<F-CONTENT>.+</F-CONTENT>$
-            %(common)s
+
+cmnfailre = ^[aA]uthentication (?:failure|error|failed) for <F-USER>.*</F-USER> from <HOST>( via \S+)?\s*%(__suff)s$
+            ^User not known to the underlying authentication module for <F-USER>.*</F-USER> from <HOST>\s*%(__suff)s$
+            ^Failed \S+ for (?P<cond_inv>invalid user )?<F-USER>(?P<cond_user>\S+)|(?(cond_inv)(?:(?! from ).)*?|[^:]+)</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$)
+            ^<F-USER>ROOT</F-USER> LOGIN REFUSED.* FROM <HOST>\s*%(__suff)s$
+            ^[iI](?:llegal|nvalid) user <F-USER>.*?</F-USER> from <HOST>%(__on_port_opt)s\s*$
+            ^User <F-USER>.+</F-USER> from <HOST> not allowed because not listed in AllowUsers\s*%(__suff)s$
+            ^User <F-USER>.+</F-USER> from <HOST> not allowed because listed in DenyUsers\s*%(__suff)s$
+            ^User <F-USER>.+</F-USER> from <HOST> not allowed because not in any group\s*%(__suff)s$
+            ^refused connect from \S+ \(<HOST>\)\s*%(__suff)s$
+            ^Received disconnect from <HOST>%(__on_port_opt)s:\s*3: .*: Auth fail%(__suff)s$
+            ^User <F-USER>.+</F-USER> from <HOST> not allowed because a group is listed in DenyGroups\s*%(__suff)s$
+            ^User <F-USER>.+</F-USER> from <HOST> not allowed because none of user's groups are listed in AllowGroups\s*%(__suff)s$
+            ^pam_unix\(sshd:auth\):\s+authentication failure;\s*logname=\S*\s*uid=\d*\s*euid=\d*\s*tty=\S*\s*ruser=<F-USER>\S*</F-USER>\s*rhost=<HOST>\s.*%(__suff)s$
+            ^(error: )?maximum authentication attempts exceeded for <F-USER>.*</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)? \[preauth\]$
+            ^User <F-USER>.+</F-USER> not allowed because account is locked%(__suff)s
+            ^Disconnecting: Too many authentication failures for <F-USER>.+?</F-USER>%(__suff)s
+            ^<F-NOFAIL>Received disconnect</F-NOFAIL> from <HOST>: 11:
+            ^<F-NOFAIL>Connection closed</F-NOFAIL> by <HOST>%(__suff)s$
+
+mdre-normal =
+
+mdre-ddos = ^Did not receive identification string from <HOST>%(__suff)s$
+            ^<F-NOFAIL>SSH: Server;Ltype:</F-NOFAIL> (?:Authname|Version|Kex);Remote: <HOST>-\d+;[A-Z]\w+:
+            ^Read from socket failed: Connection reset by peer \[preauth\]
+
+mdre-extra = ^Received disconnect from <HOST>%(__on_port_opt)s:\s*14: No supported authentication methods available%(__suff)s$
+            ^Unable to negotiate with <HOST>%(__on_port_opt)s: no matching (?:cipher|key exchange method) found.
+            ^Unable to negotiate a (?:cipher|key exchange method)%(__suff)s$
+
+mdre-aggressive = %(mdre-ddos)s
+                  %(mdre-extra)s
+
+cfooterre = ^<F-NOFAIL>Connection from</F-NOFAIL> <HOST>
+
+failregex = %(cmnfailre)s
+            <mdre-<mode>>
+            %(cfooterre)s
+
+# Parameter "mode": normal (default), ddos, extra or aggressive (combines all)
+# Usage example (for jail.local):
+#   [sshd]
+#   mode = extra
+#   # or another jail (rewrite filter parameters of jail):
+#   [sshd-aggressive]
+#   filter = sshd[mode=aggressive]
+#
+mode = normal
+
+#filter = sshd[mode=aggressive]
 
 ignoreregex = 
 
@@ -79,5 +93,5 @@ datepattern = {^LN-BEG}
 #   and later catch-all's could contain user-provided input, which need to be greedily
 #   matched away first.
 #
-# Author: Cyril Jaquier, Yaroslav Halchenko, Petr Voralek, Daniel Black
+# Author: Cyril Jaquier, Yaroslav Halchenko, Petr Voralek, Daniel Black and Sergey Brester aka sebres
-
+# Rewritten using prefregex (and introduced "mode" parameter) by Serg G. Brester.

config/jail.conf

@@ -245,6 +245,12 @@ action_badips = badips.py[category="%(__name__)s", banaction="%(banaction)s", ag
 #
 action_badips_report = badips[category="%(__name__)s", agent="%(fail2ban_agent)s"]
 
+# Report ban via abuseipdb.com.
+#
+# See action.d/abuseipdb.conf for usage example and details.
+#
+action_abuseipdb = abuseipdb
+
 # Choose default action.  To change, just override value of 'action' with the
 # interpolation to the chosen action shortcut (e.g.  action_mw, action_mwl, etc) in jail.local
 # globally (section [DEFAULT]) or per specific section
@@ -261,17 +267,11 @@ action = %(action_)s
 
 [sshd]
 
-# To use more aggressive sshd filter (inclusive sshd-ddos failregex):
+# To use more aggressive sshd modes set filter parameter "mode" in jail.local:
-#filter = sshd-aggressive
+# normal (default), ddos, extra or aggressive (combines all).
-port    = ssh
+# See "tests/files/logs/sshd" or "filter.d/sshd.conf" for usage example and details.
-logpath = %(sshd_log)s
+mode    = normal
-backend = %(sshd_backend)s
+filter  = sshd[mode=%(mode)s]
-
-
-[sshd-ddos]
-# This jail corresponds to the standard configuration in Fail2ban.
-# The mail-whois action send a notification e-mail with a whois request
-# in the body.
 port    = ssh
 logpath = %(sshd_log)s
 backend = %(sshd_backend)s
@@ -587,7 +587,11 @@ backend = %(syslog_backend)s
 
 
 [sendmail-reject]
-
+# To use more aggressive modes set filter parameter "mode" in jail.local:
+# normal (default), extra or aggressive
+# See "tests/files/logs/sendmail-reject" or "filter.d/sendmail-reject.conf" for usage example and details.
+mode    = normal
+filter  = sendmail-reject[mode=%(mode)s]
 port     = smtp,465,submission
 logpath  = %(syslog_mail)s
 backend  = %(syslog_backend)s

fail2ban/client/configreader.py

@@ -176,6 +176,8 @@ class ConfigReaderUnshared(SafeConfigParserWithIncludes):
 		if not os.path.exists(self._basedir):
 			raise ValueError("Base configuration directory %s does not exist "
 							  % self._basedir)
+		if filename.startswith("./"): # pragma: no cover
+			filename = os.path.abspath(filename)
 		basename = os.path.join(self._basedir, filename)
 		logSys.debug("Reading configs for %s under %s " , filename, self._basedir)
 		config_files = [ basename + ".conf" ]
@@ -277,6 +279,8 @@ class DefinitionInitConfigReader(ConfigReader):
 	
 	def __init__(self, file_, jailName, initOpts, **kwargs):
 		ConfigReader.__init__(self, **kwargs)
+		if file_.startswith("./"): # pragma: no cover
+			file_ = os.path.abspath(file_)
 		self.setFile(file_)
 		self.setJailName(jailName)
 		self._initOpts = initOpts

fail2ban/client/fail2banregex.py

@@ -120,6 +120,8 @@ Report bugs to https://github.com/fail2ban/fail2ban/issues
 				version="%prog " + version)
 
 	p.add_options([
+		Option("-c", "--config", default='/etc/fail2ban',
+			   help="set alternate config directory"),
 		Option("-d", "--datepattern",
 			   help="set custom pattern used to match date/times"),
 		Option("-e", "--encoding", default=PREFER_ENC,
@@ -271,24 +273,60 @@ class Fail2banRegex(object):
 	def readRegex(self, value, regextype):
 		assert(regextype in ('fail', 'ignore'))
 		regex = regextype + 'regex'
-		if regextype == 'fail' and (os.path.isfile(value) or os.path.isfile(value + '.conf')):
+		# try to check - we've case filter?[options...]?:
-			if os.path.basename(os.path.dirname(value)) == 'filter.d':
+		basedir = self._opts.config
+		fltFile = None
+		fltOpt = {}
+		if regextype == 'fail':
+			fltName, fltOpt = JailReader.extractOptions(value)
+			if fltName is not None:
+				if "." in fltName[~5:]:
+					tryNames = (fltName,)
+				else:
+					tryNames = (fltName, fltName + '.conf', fltName + '.local')
+				for fltFile in tryNames:
+					if not "/" in fltFile:
+						if os.path.basename(basedir) == 'filter.d':
+							fltFile = os.path.join(basedir, fltFile)
+						else:
+							fltFile = os.path.join(basedir, 'filter.d', fltFile)
+					else:
+						basedir = os.path.dirname(fltFile)
+					if os.path.isfile(fltFile):
+						break
+					fltFile = None
+		# if it is filter file:
+		if fltFile is not None:
+			if (basedir == self._opts.config
+				or os.path.basename(basedir) == 'filter.d'
+				or ("." not in fltName[~5:] and "/" not in fltName)
+			):
 				## within filter.d folder - use standard loading algorithm to load filter completely (with .local etc.):
-				basedir = os.path.dirname(os.path.dirname(value))
+				if os.path.basename(basedir) == 'filter.d':
-				value = os.path.splitext(os.path.basename(value))[0]
+					basedir = os.path.dirname(basedir)
-				output( "Use %11s filter file : %s, basedir: %s" % (regex, value, basedir) )
+				fltName = os.path.splitext(os.path.basename(fltName))[0]
-				reader = FilterReader(value, 'fail2ban-regex-jail', {}, share_config=self.share_config, basedir=basedir)
+				output( "Use %11s filter file : %s, basedir: %s" % (regex, fltName, basedir) )
-				if not reader.read(): # pragma: no cover
+			else:
-					output( "ERROR: failed to load filter %s" % value )
-					return False
-			else: # pragma: no cover
 				## foreign file - readexplicit this file and includes if possible:
-				output( "Use %11s file : %s" % (regex, value) )
+				output( "Use %11s file : %s" % (regex, fltName) )
-				reader = FilterReader(value, 'fail2ban-regex-jail', {}, share_config=self.share_config)
+				basedir = None
-				reader.setBaseDir(None)
+			if fltOpt:
-				if not reader.readexplicit(): 
+				output( "Use   filter options : %r" % fltOpt )
-					output( "ERROR: failed to read %s" % value )
+			reader = FilterReader(fltName, 'fail2ban-regex-jail', fltOpt, share_config=self.share_config, basedir=basedir)
-					return False
+			ret = None
+			try:
+				if basedir is not None:
+					ret = reader.read()
+				else:
+					## foreign file - readexplicit this file and includes if possible:
+					reader.setBaseDir(None)
+					ret = reader.readexplicit()
+			except Exception as e:
+				output("Wrong config file: %s" % (str(e),))
+				if self._verbose: raise(e)
+			if not ret:
+				output( "ERROR: failed to load filter %s" % value )
+				return False
 			reader.getOptions(None)
 			readercommands = reader.convert()
 

fail2ban/client/jailreader.py

@@ -43,7 +43,7 @@ logSys = getLogger(__name__)
 class JailReader(ConfigReader):
 	
 	# regex, to extract list of options:
-	optionCRE = re.compile(r"^([\w\-_\.]+)(?:\[(.*)\])?\s*$", re.DOTALL)
+	optionCRE = re.compile(r"^([^\[]+)(?:\[(.*)\])?\s*$", re.DOTALL)
 	# regex, to iterate over single option in option list, syntax:
 	# `action = act[p1="...", p2='...', p3=...]`, where the p3=... not contains `,` or ']'
 	# since v0.10 separator extended with `]\s*[` for support of multiple option groups, syntax 

fail2ban/server/actions.py

@@ -292,6 +292,7 @@ class Actions(JailThread, Mapping):
 		AI_DICT = {
 			"ip":				lambda self: self.__ticket.getIP(),
 			"ip-rev":		lambda self: self['ip'].getPTR(''),
+			"ip-host":	lambda self: self['ip'].getHost(),
 			"fid":			lambda self: self.__ticket.getID(),
 			"failures":	lambda self: self.__ticket.getAttempt(),
 			"time":			lambda self: self.__ticket.getTime(),

fail2ban/server/ipdns.py

@@ -376,6 +376,11 @@ class IPAddr(object):
 
 		return "%s.%s" % (".".join(reversed(exploded_ip)), suffix)
 
+	def getHost(self):
+		"""Return the host name (DNS) of the provided IP address object
+		"""
+		return DNSUtils.ipToName(self.ntoa)
+
 	@property
 	def isIPv4(self):
 		"""Either the IP object is of address family AF_INET

fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf

@@ -23,9 +23,9 @@ __prefix_line_sl = %(__prefix_line)s%(__pref)s
 __prefix_line_ml1 = (?P<__prefix>%(__prefix_line)s)%(__pref)s
 __prefix_line_ml2 = %(__suff)s$<SKIPLINES>^(?P=__prefix)%(__pref)s
 
-mode = %(normal)s
+[Definition]
 
-normal = ^%(__prefix_line_sl)s[aA]uthentication (?:failure|error|failed) for .* from <HOST>( via \S+)?\s*%(__suff)s$
+cmnfailre = ^%(__prefix_line_sl)s[aA]uthentication (?:failure|error|failed) for .* from <HOST>( via \S+)?\s*%(__suff)s$
          ^%(__prefix_line_sl)sUser not known to the underlying authentication module for .* from <HOST>\s*%(__suff)s$
          ^%(__prefix_line_sl)sFailed \S+ for (?P<cond_inv>invalid user )?(?P<user>(?P<cond_user>\S+)|(?(cond_inv)(?:(?! from ).)*?|[^:]+)) from <HOST>%(__on_port_opt)s(?: ssh\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$)
          ^%(__prefix_line_sl)sROOT LOGIN REFUSED.* FROM <HOST>\s*%(__suff)s$
@@ -43,18 +43,30 @@ normal = ^%(__prefix_line_sl)s[aA]uthentication (?:failure|error|failed) for .*
          ^%(__prefix_line_ml1)sDisconnecting: Too many authentication failures for .+?%(__prefix_line_ml2)sConnection closed by <HOST>%(__suff)s$
          ^%(__prefix_line_ml1)sConnection from <HOST>%(__on_port_opt)s%(__prefix_line_ml2)sDisconnecting: Too many authentication failures for .+%(__suff)s$
 
-ddos =   ^%(__prefix_line_sl)sDid not receive identification string from <HOST>%(__suff)s$
+mdre-normal =
-         ^%(__prefix_line_sl)sReceived disconnect from <HOST>%(__on_port_opt)s:\s*14: No supported authentication methods available%(__suff)s$
-         ^%(__prefix_line_sl)sUnable to negotiate with <HOST>%(__on_port_opt)s: no matching (?:cipher|key exchange method) found.
-         ^%(__prefix_line_ml1)sConnection from <HOST>%(__on_port_opt)s%(__prefix_line_ml2)sUnable to negotiate a (?:cipher|key exchange method)%(__suff)s$
-         ^%(__prefix_line_ml1)sSSH: Server;Ltype: (?:Authname|Version|Kex);Remote: <HOST>-\d+;[A-Z]\w+:.*%(__prefix_line_ml2)sRead from socket failed: Connection reset by peer%(__suff)s$
 
-aggressive = %(normal)s
+mdre-ddos =  ^%(__prefix_line_sl)sDid not receive identification string from <HOST>%(__suff)s$
-             %(ddos)s
+             ^%(__prefix_line_ml1)sSSH: Server;Ltype: (?:Authname|Version|Kex);Remote: <HOST>-\d+;[A-Z]\w+:.*%(__prefix_line_ml2)sRead from socket failed: Connection reset by peer%(__suff)s$
 
-[Definition]
+mdre-extra = ^%(__prefix_line_sl)sReceived disconnect from <HOST>%(__on_port_opt)s:\s*14: No supported authentication methods available%(__suff)s$
+             ^%(__prefix_line_sl)sUnable to negotiate with <HOST>%(__on_port_opt)s: no matching (?:cipher|key exchange method) found.
+             ^%(__prefix_line_ml1)sConnection from <HOST>%(__on_port_opt)s%(__prefix_line_ml2)sUnable to negotiate a (?:cipher|key exchange method)%(__suff)s$
 
-failregex = %(mode)s
+mdre-aggressive = %(mdre-ddos)s
+                  %(mdre-extra)s
+
+failregex = %(cmnfailre)s
+            <mdre-<mode>>
+
+# Parameter "mode": normal (default), ddos, extra or aggressive (combines all)
+# Usage example (for jail.local):
+#   [sshd]
+#   mode = extra
+#   # or another jail (rewrite filter parameters of jail):
+#   [sshd-aggressive]
+#   filter = sshd[mode=aggressive]
+#
+mode = normal
 
 ignoreregex = 
 

fail2ban/tests/fail2banregextestcase.py

@@ -209,7 +209,8 @@ class Fail2banRegexTest(LogCaptureTestCase):
 		(opts, args, fail2banRegex) = _Fail2banRegex(
 			"-l", "notice", # put down log-level, because of too many debug-messages
 			"-v", "--verbose-date", "--print-all-matched",
-			Fail2banRegexTest.FILENAME_SSHD, Fail2banRegexTest.FILTER_SSHD
+			"-c", CONFIG_DIR,
+			Fail2banRegexTest.FILENAME_SSHD, "sshd"
 		)
 		self.assertTrue(fail2banRegex.start(args))
 		# test failure line and not-failure lines both presents:
@@ -220,7 +221,8 @@ class Fail2banRegexTest(LogCaptureTestCase):
 		(opts, args, fail2banRegex) = _Fail2banRegex(
 			"-l", "notice", # put down log-level, because of too many debug-messages
 			"--print-all-matched",
-			Fail2banRegexTest.FILENAME_ZZZ_SSHD, Fail2banRegexTest.FILTER_SSHD
+			"-c", CONFIG_DIR,
+			Fail2banRegexTest.FILENAME_ZZZ_SSHD, "sshd.conf[mode=normal]"
 		)
 		self.assertTrue(fail2banRegex.start(args))
 		# test failure line and all not-failure lines presents:
@@ -234,7 +236,8 @@ class Fail2banRegexTest(LogCaptureTestCase):
 		(opts, args, fail2banRegex) = _Fail2banRegex(
 			"-l", "notice", # put down log-level, because of too many debug-messages
 			"--print-all-matched", "--print-all-missed",
-			Fail2banRegexTest.FILENAME_ZZZ_SSHD, Fail2banRegexTest.FILTER_ZZZ_SSHD
+			"-c", os.path.dirname(Fail2banRegexTest.FILTER_ZZZ_SSHD),
+			Fail2banRegexTest.FILENAME_ZZZ_SSHD, os.path.basename(Fail2banRegexTest.FILTER_ZZZ_SSHD)
 		)
 		self.assertTrue(fail2banRegex.start(args))
 		# test "failure" line presents (2nd part only, because multiline fewer precise):
@@ -245,10 +248,17 @@ class Fail2banRegexTest(LogCaptureTestCase):
 		# by the way test of ignoreregex (specified in filter file)...
 		(opts, args, fail2banRegex) = _Fail2banRegex(
 			"-l", "notice", # put down log-level, because of too many debug-messages
-			Fail2banRegexTest.FILENAME_ZZZ_GEN, Fail2banRegexTest.FILTER_ZZZ_GEN
+			Fail2banRegexTest.FILENAME_ZZZ_GEN, Fail2banRegexTest.FILTER_ZZZ_GEN+"[mode=test]"
 		)
 		self.assertTrue(fail2banRegex.start(args))
 
+	def testWrongFilterFile(self):
+		# use test log as filter file to cover eror cases...
+		(opts, args, fail2banRegex) = _Fail2banRegex(
+			Fail2banRegexTest.FILENAME_ZZZ_GEN, Fail2banRegexTest.FILENAME_ZZZ_GEN
+		)
+		self.assertFalse(fail2banRegex.start(args))
+
 	def _reset(self):
 		# reset global warn-counter:
 		from ..server.filter import _decode_line_warn

fail2ban/tests/files/logs/cyrus-imap

@@ -16,3 +16,6 @@ Dec 30 16:03:27 somehost imapd[2517]: badlogin: local-somehost[1.2.3.4] OTP [SAS
 Jul 17 22:55:56 derry cyrus/imaps[7568]: badlogin: serafinat.xxxxxx [1.2.3.4] plain [SASL(-13): user not found: user: pressy@derry property: cmusaslsecretPLAIN not found in sasldb]
 # failJSON: { "time": "2005-07-18T16:46:42", "match": true , "host": "1.2.3.4" }
 Jul 18 16:46:42 derry cyrus/imaps[27449]: badlogin: serafinat.xxxxxx [1.2.3.4] PLAIN [SASL(-13): user not found: Password verification failed]
+
+# failJSON: { "time": "2005-03-08T05:25:21", "match": true , "host": "192.0.2.4", "desc": "entry without loginname/hostname before IP" }
+Mar  8 05:25:21 host imap[22130]: badlogin: [192.0.2.4] plain [SASL(-13): authentication failure: Password verification failed]

fail2ban/tests/files/logs/sendmail-reject

@@ -1,3 +1,5 @@
+# normal mode # filterOptions: {"mode": "normal"}
+
 # failJSON: { "time": "2005-02-25T03:01:10", "match": true , "host": "128.68.136.133" }
 Feb 25 03:01:10 kismet sm-acceptingconnections[27713]: s1P819mk027713: ruleset=check_rcpt, arg1=<asservnew@freemailhost.ru>, relay=128-68-136-133.broadband.corbina.ru [128.68.136.133], reject=550 5.7.1 <asservnew@freemailhost.ru>... Relaying denied. Proper authentication required.
 
@@ -69,20 +71,27 @@ Feb 22 14:02:44 batman sm-mta[4030]: s1MD2hsd004030: rrcs-24-73-201-194.se.biz.r
 
 
 # failJSON: { "match": false }
-Nov  3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026254: <arhipov@domain.com>... No such user here
+Nov  3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026250: <arhipov@domain.com>... No such user here
 # failJSON: { "match": false }
-Nov  3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026254: <anatoliy@domain.com>... No such user here
+Nov  3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026251: <anatoliy@domain.com>... No such user here
 # failJSON: { "match": false }
-Nov  3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026254: <artem@domain.com>... No such user here
+Nov  3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026252: <artem@domain.com>... No such user here
 # failJSON: { "match": false }
-Nov  3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026254: <anto@domain.com>... No such user here
+Nov  3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026252: <anto@domain.com>... No such user here
+
 # failJSON: { "match": false }
-Nov  3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026254: <anton@domain.com>... No such user here
+Nov  3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026254: <davaojk25@domain.com>... No such user here
 # failJSON: { "time": "2004-11-03T11:35:30", "match": true , "host": "95.32.23.163" }
 Nov  3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026254: from=<davaojk25@domain.com>, size=0, class=0, nrcpts=0, bodytype=8BITMIME, proto=ESMTP, daemon=MTA, relay=163.23.32.95.dsl-dynamic.vsi.ru [95.32.23.163]
 
 # failJSON: { "match": false }
-Nov  3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026254: <anton@domain.com>... No such user here
+Nov  3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026252: <anton@domain.com>... No such user here
-# Different mail ID shouldn't match
+# failJSON: { "match": false, "desc": "Different mail ID shouldn't match" }
-# failJSON: { "match": false }
+Nov  3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026255: from=<anton@domain.com>, size=0, class=0, nrcpts=0, bodytype=8BITMIME, proto=ESMTP, daemon=MTA, relay=163.23.32.95.dsl-dynamic.vsi.ru [95.32.23.163]
-Nov  3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026255: from=<davaojk25@domain.com>, size=0, class=0, nrcpts=0, bodytype=8BITMIME, proto=ESMTP, daemon=MTA, relay=163.23.32.95.dsl-dynamic.vsi.ru [95.32.23.163]
+
+# filterOptions: {"mode": "extra"}
+
+# failJSON: { "time": "2005-03-06T16:55:28", "match": true , "host": "192.0.2.194", "desc": "wrong resp. non RFC compiant (ddos prelude?), MTA-mode" }
+Mar  6 16:55:28 s192-168-0-1 sm-mta[20949]: v26LtRA0020949: some-host-24.example.org [192.0.2.194] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA
+# failJSON: { "time": "2005-03-07T15:04:37", "match": true , "host": "192.0.2.195", "desc": "wrong resp. non RFC compiant (ddos prelude?), MSP-mode, (may be forged)" }
+Mar  7 15:04:37 s192-168-0-1 sm-mta[18624]: v27K4Vj8018624: some-host-24.example.org [192.0.2.195] (may be forged) did not issue MAIL/EXPN/VRFY/ETRN during connection to MSP-v4

fail2ban/tests/files/logs/sshd

@@ -183,3 +183,49 @@ Apr 27 13:02:04 host sshd[29116]: Received disconnect from 1.2.3.4: 11: Normal S
 # Match sshd auth errors on OpenSUSE systems
 # failJSON: { "time": "2015-04-16T20:02:50", "match": true , "host": "222.186.21.217", "desc": "Authentication for user failed" }
 2015-04-16T18:02:50.321974+00:00 host sshd[2716]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.186.21.217  user=root
+
+# filterOptions: {"mode": "ddos"}
+
+# http://forums.powervps.com/showthread.php?t=1667
+# failJSON: { "time": "2005-06-07T01:10:56", "match": true , "host": "69.61.56.114" }
+Jun 7 01:10:56 host sshd[5937]: Did not receive identification string from 69.61.56.114
+
+# gh-864(1):
+# failJSON: { "match": false }
+Nov 24 23:46:39 host sshd[32686]: SSH: Server;Ltype: Version;Remote: 127.0.0.1-1780;Protocol: 2.0;Client: libssh2_1.4.3
+# failJSON: { "time": "2004-11-24T23:46:43", "match": true , "host": "127.0.0.1", "desc": "Multiline for connection reset by peer (1)" }
+Nov 24 23:46:43 host sshd[32686]: fatal: Read from socket failed: Connection reset by peer [preauth]
+
+# gh-864(2):
+# failJSON: { "match": false }
+Nov 24 23:46:40 host sshd[32686]: SSH: Server;Ltype: Kex;Remote: 127.0.0.1-1780;Enc: aes128-ctr;MAC: hmac-sha1;Comp: none [preauth]
+# failJSON: { "time": "2004-11-24T23:46:43", "match": true , "host": "127.0.0.1", "desc": "Multiline for connection reset by peer (2)" }
+Nov 24 23:46:43 host sshd[32686]: fatal: Read from socket failed: Connection reset by peer [preauth]
+
+# gh-864(3):
+# failJSON: { "match": false }
+Nov 24 23:46:41 host sshd[32686]: SSH: Server;Ltype: Authname;Remote: 127.0.0.1-1780;Name: root [preauth]
+# failJSON: { "time": "2004-11-24T23:46:43", "match": true , "host": "127.0.0.1", "desc": "Multiline for connection reset by peer (3)" }
+Nov 24 23:46:43 host sshd[32686]: fatal: Read from socket failed: Connection reset by peer [preauth]
+
+# filterOptions: {"mode": "extra"}
+
+# several other cases from gh-864:
+# failJSON: { "time": "2004-11-25T01:34:12", "match": true , "host": "127.0.0.1", "desc": "No supported authentication methods" }
+Nov 25 01:34:12 srv sshd[123]: Received disconnect from 127.0.0.1: 14: No supported authentication methods available [preauth]
+# failJSON: { "time": "2004-11-25T01:35:13", "match": true , "host": "127.0.0.1", "desc": "No supported authentication methods" }
+Nov 25 01:35:13 srv sshd[123]: error: Received disconnect from 127.0.0.1: 14: No supported authentication methods available [preauth]
+# failJSON: { "time": "2004-11-25T01:35:14", "match": true , "host": "192.168.2.92", "desc": "Optional space after port" }
+Nov 25 01:35:14 srv sshd[3625]: error: Received disconnect from 192.168.2.92 port 1684:14: No supported authentication methods available [preauth]
+
+# gh-1545:
+# failJSON: { "time": "2004-11-26T13:03:29", "match": true , "host": "192.0.2.1", "desc": "No matching cipher" }
+Nov 26 13:03:29 srv sshd[45]: Unable to negotiate with 192.0.2.1 port 55419: no matching cipher found. Their offer: aes256-cbc,rijndael-cbc@lysator.liu.se,aes192-cbc,aes128-cbc,arcfour128,arcfour,3des-cbc,none [preauth]
+
+# gh-1117:
+# failJSON: { "time": "2004-11-26T13:03:30", "match": true , "host": "192.0.2.2", "desc": "No matching key exchange method" }
+Nov 26 13:03:30 srv sshd[45]: fatal: Unable to negotiate with 192.0.2.2 port 55419: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1
+# failJSON: { "match": false }
+Nov 26 15:03:30 host sshd[22440]: Connection from 192.0.2.3 port 39678 on 192.168.1.9 port 22
+# failJSON: { "time": "2004-11-26T15:03:31", "match": true , "host": "192.0.2.3", "desc": "Multiline - no matching key exchange method" }
+Nov 26 15:03:31 host sshd[22440]: fatal: Unable to negotiate a key exchange method [preauth]

fail2ban/tests/files/logs/sshd-aggressive

@@ -1,3 +0,0 @@
-# sshd-aggressive includes sshd and sshd-ddos failregex's:
-# addFILE: "sshd"
-# addFILE: "sshd-ddos"

fail2ban/tests/files/logs/sshd-ddos

@@ -1,41 +0,0 @@
-# http://forums.powervps.com/showthread.php?t=1667
-# failJSON: { "time": "2005-06-07T01:10:56", "match": true , "host": "69.61.56.114" }
-Jun 7 01:10:56 host sshd[5937]: Did not receive identification string from 69.61.56.114
-
-# gh-864(1):
-# failJSON: { "match": false }
-Nov 24 23:46:39 host sshd[32686]: SSH: Server;Ltype: Version;Remote: 127.0.0.1-1780;Protocol: 2.0;Client: libssh2_1.4.3
-# failJSON: { "time": "2004-11-24T23:46:43", "match": true , "host": "127.0.0.1", "desc": "Multiline for connection reset by peer (1)" }
-Nov 24 23:46:43 host sshd[32686]: fatal: Read from socket failed: Connection reset by peer [preauth]
-
-# gh-864(2):
-# failJSON: { "match": false }
-Nov 24 23:46:40 host sshd[32686]: SSH: Server;Ltype: Kex;Remote: 127.0.0.1-1780;Enc: aes128-ctr;MAC: hmac-sha1;Comp: none [preauth]
-# failJSON: { "time": "2004-11-24T23:46:43", "match": true , "host": "127.0.0.1", "desc": "Multiline for connection reset by peer (2)" }
-Nov 24 23:46:43 host sshd[32686]: fatal: Read from socket failed: Connection reset by peer [preauth]
-
-# gh-864(3):
-# failJSON: { "match": false }
-Nov 24 23:46:41 host sshd[32686]: SSH: Server;Ltype: Authname;Remote: 127.0.0.1-1780;Name: root [preauth]
-# failJSON: { "time": "2004-11-24T23:46:43", "match": true , "host": "127.0.0.1", "desc": "Multiline for connection reset by peer (3)" }
-Nov 24 23:46:43 host sshd[32686]: fatal: Read from socket failed: Connection reset by peer [preauth]
-
-# several other cases from gh-864:
-# failJSON: { "time": "2004-11-25T01:34:12", "match": true , "host": "127.0.0.1", "desc": "No supported authentication methods" }
-Nov 25 01:34:12 srv sshd[123]: Received disconnect from 127.0.0.1: 14: No supported authentication methods available [preauth]
-# failJSON: { "time": "2004-11-25T01:35:13", "match": true , "host": "127.0.0.1", "desc": "No supported authentication methods" }
-Nov 25 01:35:13 srv sshd[123]: error: Received disconnect from 127.0.0.1: 14: No supported authentication methods available [preauth]
-# failJSON: { "time": "2004-11-25T01:35:14", "match": true , "host": "192.168.2.92", "desc": "Optional space after port" }
-Nov 25 01:35:14 srv sshd[3625]: error: Received disconnect from 192.168.2.92 port 1684:14: No supported authentication methods available [preauth]
-
-# gh-1545:
-# failJSON: { "time": "2004-11-26T13:03:29", "match": true , "host": "192.0.2.1", "desc": "No matching cipher" }
-Nov 26 13:03:29 srv sshd[45]: Unable to negotiate with 192.0.2.1 port 55419: no matching cipher found. Their offer: aes256-cbc,rijndael-cbc@lysator.liu.se,aes192-cbc,aes128-cbc,arcfour128,arcfour,3des-cbc,none [preauth]
-
-# gh-1117:
-# failJSON: { "time": "2004-11-26T13:03:30", "match": true , "host": "192.0.2.2", "desc": "No matching key exchange method" }
-Nov 26 13:03:30 srv sshd[45]: fatal: Unable to negotiate with 192.0.2.2 port 55419: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1
-# failJSON: { "match": false }
-Nov 26 15:03:30 host sshd[22440]: Connection from 192.0.2.3 port 39678 on 192.168.1.9 port 22
-# failJSON: { "time": "2004-11-26T15:03:31", "match": true , "host": "192.0.2.3", "desc": "Multiline - no matching key exchange method" }
-Nov 26 15:03:31 host sshd[22440]: fatal: Unable to negotiate a key exchange method [preauth]

fail2ban/tests/samplestestcase.py

@@ -49,12 +49,7 @@ class FilterSamplesRegex(unittest.TestCase):
 	def setUp(self):
 		"""Call before every test case."""
 		super(FilterSamplesRegex, self).setUp()
-		self.filter = Filter(None)
+		self.filter = None
-		self.filter.returnRawHost = True
-		self.filter.checkAllRegex = True
-		self.filter.checkFindTime = False
-		self.filter.active = True
-
 		setUpMyTime()
 
 	def tearDown(self):
@@ -83,11 +78,15 @@ class FilterSamplesRegex(unittest.TestCase):
 			RE_WRONG_GREED.search('non-greedy .+? test' + RE_HOST + ' test vary catch-all .* anchored$'))
 
 
-def testSampleRegexsFactory(name, basedir):
+	def _readFilter(self, name, basedir, opts=None):
-	def testFilter(self):
+		self.filter = Filter(None)
-
+		self.filter.returnRawHost = True
+		self.filter.checkAllRegex = True
+		self.filter.checkFindTime = False
+		self.filter.active = True
+		if opts is None: opts = dict()
 		# Check filter exists
-		filterConf = FilterReader(name, "jail", {}, 
+		filterConf = FilterReader(name, "jail", opts,
 			basedir=basedir, share_config=unittest.F2B.share_config)
 		self.assertEqual(filterConf.getFile(), name)
 		self.assertEqual(filterConf.getJailName(), "jail")
@@ -113,6 +112,17 @@ def testSampleRegexsFactory(name, basedir):
 				elif opt[2] == "datepattern":
 					self.filter.setDatePattern(optval)
 
+		# test regexp contains greedy catch-all before <HOST>, that is
+		# not hard-anchored at end or has not precise sub expression after <HOST>:
+		for fr in self.filter.getFailRegex():
+			if RE_WRONG_GREED.search(fr): # pragma: no cover
+				raise AssertionError("Following regexp of \"%s\" contains greedy catch-all before <HOST>, "
+					"that is not hard-anchored at end or has not precise sub expression after <HOST>:\n%s" %
+					(name, str(fr).replace(RE_HOST, '<HOST>')))
+
+def testSampleRegexsFactory(name, basedir):
+	def testFilter(self):
+
 		self.assertTrue(
 			os.path.isfile(os.path.join(TEST_FILES_DIR, "logs", name)),
 			"No sample log file available for '%s' filter" % name)
@@ -125,22 +135,21 @@ def testSampleRegexsFactory(name, basedir):
 			logFile = fileinput.FileInput(os.path.join(TEST_FILES_DIR, "logs",
 				filename))
 
-			# test regexp contains greedy catch-all before <HOST>, that is
-			# not hard-anchored at end or has not precise sub expression after <HOST>:
-			for fr in self.filter.getFailRegex():
-				if RE_WRONG_GREED.search(fr): # pragma: no cover
-					raise AssertionError("Following regexp of \"%s\" contains greedy catch-all before <HOST>, "
-						"that is not hard-anchored at end or has not precise sub expression after <HOST>:\n%s" %
-						(name, str(fr).replace(RE_HOST, '<HOST>')))
-
 			for line in logFile:
-				jsonREMatch = re.match("^# ?(failJSON|addFILE):(.+)$", line)
+				jsonREMatch = re.match("^#+ ?(failJSON|filterOptions|addFILE):(.+)$", line)
 				if jsonREMatch:
 					try:
 						faildata = json.loads(jsonREMatch.group(2))
+						# filterOptions - dict in JSON to control filter options (e. g. mode, etc.):
+						if jsonREMatch.group(1) == 'filterOptions':
+							self.filter = None
+							self._readFilter(name, basedir, opts=faildata)
+							continue
+						# addFILE - filename to "include" test-files should be additionally parsed:
 						if jsonREMatch.group(1) == 'addFILE':
 							filenames.append(faildata)
 							continue
+						# failJSON - faildata contains info of the failure to check it.
 					except ValueError as e:
 						raise ValueError("%s: %s:%i" %
 							(e, logFile.filename(), logFile.filelineno()))
@@ -150,6 +159,9 @@ def testSampleRegexsFactory(name, basedir):
 				else:
 					faildata = {}
 
+				if self.filter is None:
+					self._readFilter(name, basedir, opts=None)
+
 				try:
 					ret = self.filter.processLine(line)
 					if not ret:

fail2ban/tests/servertestcase.py

@@ -38,7 +38,9 @@ from ..server.server import Server
 from ..server.ipdns import IPAddr
 from ..server.jail import Jail
 from ..server.jailthread import JailThread
+from ..server.ticket import BanTicket
 from ..server.utils import Utils
+from .dummyjail import DummyJail
 from .utils import LogCaptureTestCase
 from ..helpers import getLogger, PREFER_ENC
 from .. import version
@@ -1686,7 +1688,7 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 				# complain --
 				('j-complain-abuse', 
 					'complain['
-					  'name=%(__name__)s, grepopts="-m 1", grepmax=2, mailcmd="mail -s",' +
+					  'name=%(__name__)s, grepopts="-m 1", grepmax=2, mailcmd="mail -s Hostname: <ip-host> - ",' +
 					  # test reverse ip:
 					  'debug=1,' +
 						# 2 logs to test grep from multiple logs:
@@ -1701,14 +1703,14 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 						'testcase01.log:Dec 31 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 87.142.124.10',
 						'testcase01a.log:Dec 31 11:55:01 [sshd] error: PAM: Authentication failure for test from 87.142.124.10',
 						# both abuse mails should be separated with space:
-						'mail -s Abuse from 87.142.124.10 abuse-1@abuse-test-server abuse-2@abuse-test-server',
+						'mail -s Hostname: test-host - Abuse from 87.142.124.10 abuse-1@abuse-test-server abuse-2@abuse-test-server',
 					),
 					'ip6-ban': (
 						# test reverse ip:
 						'try to resolve 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.abuse-contacts.abusix.org',
 						'Lines containing failures of 2001:db8::1 (max 2)',
 						# both abuse mails should be separated with space:
-						'mail -s Abuse from 2001:db8::1 abuse-1@abuse-test-server abuse-2@abuse-test-server',
+						'mail -s Hostname: test-host - Abuse from 2001:db8::1 abuse-1@abuse-test-server abuse-2@abuse-test-server',
 					),
 				}),
 			)
@@ -1732,6 +1734,7 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 
 			ipv4 = IPAddr('87.142.124.10')
 			ipv6 = IPAddr('2001:db8::1');
+			dmyjail = DummyJail()
 			for jail, act, tests in testJailsActions:
 				# print(jail, jails[jail])
 				for a in jails[jail].actions:
@@ -1745,7 +1748,8 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 					for (test, ip) in (('ip4-ban', ipv4), ('ip6-ban', ipv6)):
 						if not tests.get(test): continue
 						self.pruneLog('# === %s ===' % test)
-						ticket = _actions.CallingMap({
+						ticket = BanTicket(ip)
-							'ip': ip, 'ip-rev': lambda self: self['ip'].getPTR(''), 'failures': 100,})
+						ticket.setAttempt(100)
+						ticket = _actions.Actions.ActionInfo(ticket, dmyjail)
 						action.ban(ticket)
 						self.assertLogged(*tests[test], all=True)

fail2ban/tests/utils.py

@@ -273,6 +273,9 @@ def initTests(opts):
 		c.set('192.0.2.%s' % i, None)
 		c.set('198.51.100.%s' % i, None)
 		c.set('203.0.113.%s' % i, None)
+		c.set('2001:db8::%s' %i, 'test-host')
+	# some legal ips used in our test cases (prevent slow dns-resolving and failures if will be changed later):
+	c.set('87.142.124.10', 'test-host')
 	if unittest.F2B.no_network: # pragma: no cover
 		# precache all wrong dns to ip's used in test cases:
 		c = DNSUtils.CACHE_nameToIp

man/fail2ban-regex.1

@@ -27,6 +27,9 @@ a string representing a 'failregex'
 .TP
 filename
 path to a filter file (filter.d/sshd.conf)
+.TP
+filtername[option=value, ..., option=value]
+short path to a filter relative filter.d in configuration base (sshd[mode=aggressive])
 .SS "IGNOREREGEX:"
 .TP
 string
@@ -42,6 +45,9 @@ show program's version number and exit
 \fB\-h\fR, \fB\-\-help\fR
 show this help message and exit
 .TP
+\fB\-c\fR CONFIGBASE, \fB\-\-config\fR=\fI\,CONFIGBASE\/\fR
+set alternate config base directory (default /etc/fail2ban)
+.TP
 \fB\-d\fR DATEPATTERN, \fB\-\-datepattern\fR=\fI\,DATEPATTERN\/\fR
 set custom pattern used to match date/times
 .TP