Merge branch 'enh-rel0.9.6' into debian

* enh-rel0.9.6: (60 commits) updated man pages ENH: prep for 0.9.6 release (as of tomorrow) BF: added missing entires into MANIFEST Update ChangeLog ChangeLog entry added + jail.conf review code review, makes the test cases workable, added dev-notes ChangeLog update `filter.d/apache-modsecurity.conf` - fixed for newer version (one space, closes gh-1626) reviewed and optimized: - non-greedy catch-all replaced for safer match - unneeded catch-all anchoring removed - non-capturing groups filter.d/dovecot.conf update: - fixes failregex, that ignores failures through some irrelevant info (closes #1623); - ignores whole additionally irrelevant info in anchored regex before fixed failure data `\((?:auth failed, \d+ attempts( in \d+ secs)?|tried to use (disabled|disallowed) \S+ auth)\)` - review, IPv6 compatibility fix, non-capturing groups Update jail.conf Use Fedora's backend-settings for openSUSE amend after code review of merge gh-1581 Make changes and add test file Add Mongodb-auth filter and jail Update FILTERS filter.d/sshd.conf: Match 'Invalid user' with 'port \d*' ChangeLog entry added filter.d/sendmail-reject.conf: double space (should be by missing dns-host only) Closes #1578 Update Changelog to reflect the new np.conf action Create npf.conf for the NPF packet filter ...
2016-12-09 09:37:33 -05:00 · 2016-12-09 09:37:33 -05:00 · 623bb39ca6
parent 8f42580c05 3605155978
commit 623bb39ca6
68 changed files with 1146 additions and 319 deletions
--- a/.travis.yml
+++ b/.travis.yml
@ -41,6 +41,8 @@ script:
  - if [[ "$F2B_PY_3" ]]; then coverage run bin/fail2ban-testcases; fi
  # Use $VENV_BIN (not python) or else sudo will always run the system's python (2.7)
  - sudo $VENV_BIN/pip install .
+  # Doc files should get installed on Travis under Linux
+  - test -e /usr/share/doc/fail2ban/FILTERS
 after_success:
  - coveralls
  - codecov
--- a/74
+++ b/74
@ -6,13 +6,85 @@
 Fail2Ban: Changelog
 ===================

-ver. 0.9.5 (2016/07/15) - old-not-obsolete
+ver. 0.9.6 (2016/12/10) - stretch-is-coming
 -----------

 0.9.x line is no longer heavily developed.  If you are interested in
 new features (e.g. IPv6 support), please consider 0.10 branch and its
 releases.

+### Fixes
+* Misleading add resp. enable of (already available) jail in database, that
+  induced a subsequent error: last position of log file will be never retrieved (gh-795)
+* Fixed a distribution related bug within testReadStockJailConfForceEnabled
+  (e.g. test-cases faults on Fedora, see gh-1353)
+* Fixed pythonic filters and test scripts (running via wrong python version,
+  uses "fail2ban-python" now);
+* Fixed test case "testSetupInstallRoot" for not default python version (also
+  using direct call, out of virtualenv);
+* Fixed ambiguous wrong recognized date pattern resp. its optional parts (see gh-1512);
+* FIPS compliant, use sha1 instead of md5 if it not allowed (see gh-1540)
+* Monit config: scripting is not supported in path (gh-1556)
+* `filter.d/apache-modsecurity.conf`
+    - Fixed for newer version (one space, gh-1626), optimized: non-greedy catch-all
+      replaced for safer match, unneeded catch-all anchoring removed, non-capturing
+* `filter.d/asterisk.conf`
+    - Fixed to match different asterisk log prefix (source file: method:)
+* `filter.d/dovecot.conf`
+    - Fixed failregex ignores failures through some not relevant info (gh-1623)
+* `filter.d/ignorecommands/apache-fakegooglebot`
+    - Fixed error within apache-fakegooglebot, that will be called
+      with wrong python version (gh-1506)
+* `filter.d/assp.conf`
+    - Extended failregex and test cases to handle ASSP V1 and V2 (gh-1494)
+* `filter.d/postfix-sasl.conf`
+    - Allow for having no trailing space after 'failed:' (gh-1497)
+* `filter.d/vsftpd.conf`
+    - Optional reason part in message after FAIL LOGIN (gh-1543)
+* `filter.d/sendmail-reject.conf`
+    - removed mandatory double space (if dns-host available, gh-1579)
+* filter.d/sshd.conf
+    - recognized "Failed publickey for" (gh-1477);
+    - optimized failregex to match all of "Failed any-method for ... from <HOST>" (gh-1479)
+    - eliminated possible complex injections (on user-name resp. auth-info, see gh-1479)
+    - optional port part after host (see gh-1533, gh-1581)
+
+### New Features
+* New Actions:
+    - `action.d/npf.conf` for NPF, the latest packet filter for NetBSD
+* New Filters:
+    - `filter.d/mongodb-auth.conf` for MongoDB (document-oriented NoSQL database engine)
+      (gh-1586, gh-1606 and gh-1607)
+
+### Enhancements
+* DateTemplate regexp extended with the word-end boundary, additionally to
+  word-start boundary
+* Introduces new command "fail2ban-python", as automatically created symlink to
+  python executable, where fail2ban currently installed (resp. its modules are located):
+    - allows to use the same version, fail2ban currently running, e.g. in
+      external scripts just via replace python with fail2ban-python:
+      ```diff
+      -#!/usr/bin/env python
+      +#!/usr/bin/env fail2ban-python
+      ```
+    - always the same pickle protocol
+    - the same (and also guaranteed available) fail2ban modules
+    - simplified stand-alone install, resp. stand-alone installation possibility
+      via setup (like gh-1487) is getting closer
+* Several test cases rewritten using new methods assertIn, assertNotIn
+* New forward compatibility method assertRaisesRegexp (normally python >= 2.7).
+  Methods assertIn, assertNotIn, assertRaisesRegexp, assertLogged, assertNotLogged
+  are test covered now
+* Jail configuration extended with new syntax to pass options to the backend (see gh-1408),
+  examples:
+    - `backend = systemd[journalpath=/run/log/journal/machine-1]`
+    - `backend = systemd[journalfiles="/run/log/journal/machine-1/system.journal, /run/log/journal/machine-1/user.journal"]`
+    - `backend = systemd[journalflags=2]`
+
+
+ver. 0.9.5 (2016/07/15) - old-not-obsolete
+-----------
+
 ### Fixes
 * `filter.d/monit.conf`
    - Extended failregex with new monit "access denied" version (gh-1355)
--- a/2
+++ b/2
@ -227,7 +227,7 @@ Regular expressions (failregex, ignoreregex) assume that the date/time has been
 removed from the log line (this is just how fail2ban works internally ATM).

 If the format is like '<date...> error 1.2.3.4 is evil' then you need to match
-the < at the start so regex should be similar to '^<> <HOST> is evil$' using
+the <> at the start so regex should be similar to '^<> error <HOST> is evil$' using
 <HOST> where the IP/domain name appears in the log line.

 The following general rules apply to regular expressions:
--- a/19
+++ b/19
@ -33,12 +33,14 @@ config/action.d/iptables-new.conf
 config/action.d/iptables-xt_recent-echo.conf
 config/action.d/mail-buffered.conf
 config/action.d/mail.conf
+config/action.d/mail-whois-common.conf
 config/action.d/mail-whois.conf
 config/action.d/mail-whois-lines.conf
 config/action.d/mynetwatchman.conf
 config/action.d/nftables-allports.conf
 config/action.d/nftables-common.conf
 config/action.d/nftables-multiport.conf
+config/action.d/npf.conf
 config/action.d/nsupdate.conf
 config/action.d/osx-afctl.conf
 config/action.d/osx-ipfw.conf
@ -54,6 +56,7 @@ config/action.d/sendmail-whois-ipmatches.conf
 config/action.d/sendmail-whois-lines.conf
 config/action.d/sendmail-whois-matches.conf
 config/action.d/shorewall.conf
+config/action.d/shorewall-ipset-proto6.conf
 config/action.d/smtp.py
 config/action.d/symbiosis-blacklist-allports.conf
 config/action.d/ufw.conf
@ -69,6 +72,7 @@ config/filter.d/apache-modsecurity.conf
 config/filter.d/apache-nohome.conf
 config/filter.d/apache-noscript.conf
 config/filter.d/apache-overflows.conf
+config/filter.d/apache-pass.conf
 config/filter.d/apache-shellshock.conf
 config/filter.d/assp.conf
 config/filter.d/asterisk.conf
@ -81,11 +85,13 @@ config/filter.d/cyrus-imap.conf
 config/filter.d/directadmin.conf
 config/filter.d/dovecot.conf
 config/filter.d/dropbear.conf
+config/filter.d/drupal-auth.conf
 config/filter.d/ejabberd-auth.conf
 config/filter.d/exim-common.conf
 config/filter.d/exim.conf
 config/filter.d/exim-spam.conf
 config/filter.d/freeswitch.conf
+config/filter.d/froxlor-auth.conf
 config/filter.d/groupoffice.conf
 config/filter.d/gssftpd.conf
 config/filter.d/guacamole.conf
@ -95,6 +101,7 @@ config/filter.d/ignorecommands
 config/filter.d/ignorecommands/apache-fakegooglebot
 config/filter.d/kerio.conf
 config/filter.d/lighttpd-auth.conf
+config/filter.d/mongodb-auth.conf
 config/filter.d/monit.conf
 config/filter.d/murmur.conf
 config/filter.d/mysqld-auth.conf
@ -150,6 +157,7 @@ config/paths-opensuse.conf
 config/paths-osx.conf
 CONTRIBUTING.md
 COPYING
+.coveragerc
 DEVELOP
 doc/run-rootless.txt
 fail2ban-2to3
@ -194,6 +202,7 @@ fail2ban/server/server.py
 fail2ban/server/strptime.py
 fail2ban/server/ticket.py
 fail2ban/server/transmitter.py
+fail2ban/setup.py
 fail2ban-testcases-all
 fail2ban-testcases-all-python3
 fail2ban/tests/action_d/__init__.py
@ -205,6 +214,7 @@ fail2ban/tests/banmanagertestcase.py
 fail2ban/tests/clientreadertestcase.py
 fail2ban/tests/config/action.d/brokenaction.conf
 fail2ban/tests/config/fail2ban.conf
+fail2ban/tests/config/filter.d/common.conf
 fail2ban/tests/config/filter.d/simple.conf
 fail2ban/tests/config/filter.d/test.conf
 fail2ban/tests/config/filter.d/test.local
@ -256,6 +266,7 @@ fail2ban/tests/files/logs/apache-modsecurity
 fail2ban/tests/files/logs/apache-nohome
 fail2ban/tests/files/logs/apache-noscript
 fail2ban/tests/files/logs/apache-overflows
+fail2ban/tests/files/logs/apache-pass
 fail2ban/tests/files/logs/apache-shellshock
 fail2ban/tests/files/logs/assp
 fail2ban/tests/files/logs/asterisk
@ -269,10 +280,12 @@ fail2ban/tests/files/logs/cyrus-imap
 fail2ban/tests/files/logs/directadmin
 fail2ban/tests/files/logs/dovecot
 fail2ban/tests/files/logs/dropbear
+fail2ban/tests/files/logs/drupal-auth
 fail2ban/tests/files/logs/ejabberd-auth
 fail2ban/tests/files/logs/exim
 fail2ban/tests/files/logs/exim-spam
 fail2ban/tests/files/logs/freeswitch
+fail2ban/tests/files/logs/froxlor-auth
 fail2ban/tests/files/logs/groupoffice
 fail2ban/tests/files/logs/gssftpd
 fail2ban/tests/files/logs/guacamole
@ -280,6 +293,7 @@ fail2ban/tests/files/logs/haproxy-http-auth
 fail2ban/tests/files/logs/horde
 fail2ban/tests/files/logs/kerio
 fail2ban/tests/files/logs/lighttpd-auth
+fail2ban/tests/files/logs/mongodb-auth
 fail2ban/tests/files/logs/monit
 fail2ban/tests/files/logs/murmur
 fail2ban/tests/files/logs/mysqld-auth
@ -356,6 +370,8 @@ files/gentoo-confd
 files/gentoo-initd
 files/ipmasq-ZZZzzz_fail2ban.rul
 files/logwatch/fail2ban
+files/logwatch/fail2ban-0.8.log
+files/logwatch/fail2ban-0.9.log
 files/macosx-initd
 files/monit/fail2ban
 files/nagios/check_fail2ban
@ -373,8 +389,11 @@ man/fail2ban-regex.1
 man/fail2ban-regex.h2m
 man/fail2ban-server.1
 man/fail2ban-server.h2m
+man/fail2ban-testcases.1
+man/fail2ban-testcases.h2m
 man/generate-man
 man/jail.conf.5
+.pylintrc
 README.md
 README.Solaris
 RELEASE
--- a/README.md
+++ b/README.md
@ -2,7 +2,7 @@
                        / _|__ _(_) |_  ) |__  __ _ _ _  
                       |  _/ _` | | |/ /| '_ \/ _` | ' \ 
                       |_| \__,_|_|_/___|_.__/\__,_|_||_|
-                       v0.9.5                  2016/07/15
+                       v0.9.6                  2016/12/10

 ## Fail2Ban: ban hosts that cause multiple authentication errors

@ -39,8 +39,8 @@ Optional:

 To install, just do:

-    tar xvfj fail2ban-0.9.5.tar.bz2
-    cd fail2ban-0.9.5
+    tar xvfj fail2ban-0.9.6.tar.bz2
+    cd fail2ban-0.9.6
    python setup.py install

 This will install Fail2Ban into the python library directory. The executable
--- a/8
+++ b/8
@ -53,7 +53,7 @@ Preparation

  or an alternative for comparison with previous release

-     git diff 0.9.5 | grep -B2 'index 0000000..' | grep -B1 'new file mode' | sed -n -e '/^diff /s,.* b/,,gp' >> MANIFEST
+     git diff 0.9.6 | grep -B2 'index 0000000..' | grep -B1 'new file mode' | sed -n -e '/^diff /s,.* b/,,gp' >> MANIFEST
     sort MANIFEST | uniq | sponge MANIFEST

 * Run::
@ -70,7 +70,7 @@ Preparation

 * clean up current directory::

-    diff -rul --exclude \*.pyc . /tmp/fail2ban-0.9.5/
+    diff -rul --exclude \*.pyc . /tmp/fail2ban-0.9.6/

  * Only differences should be files that you don't want distributed.

@ -83,7 +83,7 @@ Preparation

  * To generate a list of committers use e.g.::

-      git shortlog -sn 0.9.5.. | sed -e 's,^[ 0-9\t]*,,g' | tr '\n' '\|' | sed -e 's:|:, :g'
+      git shortlog -sn 0.9.6.. | sed -e 's,^[ 0-9\t]*,,g' | tr '\n' '\|' | sed -e 's:|:, :g'

  * Ensure the top of the ChangeLog has the right version and current date.
  * Ensure the top entry of the ChangeLog has the right version and current date.
@ -106,7 +106,7 @@ Preparation
 * Tag the release by using a signed (and annotated) tag.  Cut/paste
  release ChangeLog entry as tag annotation::

-    git tag -s 0.9.5
+    git tag -s 0.9.6

 Pre Release
 ===========
--- a/1
+++ b/1
@ -119,6 +119,7 @@ Thomas Mayer
 Tom Pike
 Tom Hendrikx
 Tomas Pihl
+Thomas Skierlo (phaleas)
 Tony Lawrence
 Tomasz Ciolek
 Tyler
--- a/bin/fail2ban-client
+++ b/bin/fail2ban-client
@ -176,7 +176,7 @@ class Fail2banClient:
 					if showRet:
 						self.__logSocketError()
 					return False
-				except Exception, e:
+				except Exception as e:
 					if showRet:
 						logSys.error(e)
 					return False
@ -429,7 +429,7 @@ class Fail2banClient:
 						elif not cmd == "":
 							try:
 								self.__processCommand(shlex.split(cmd))
-							except Exception, e:
+							except Exception as e:
 								logSys.error(e)
 			except (EOFError, KeyboardInterrupt):
 				print
@ -451,7 +451,7 @@ class Fail2banClient:
 			ret = self.__configurator.getOptions(jail)
 			self.__configurator.convertToProtocol()
 			self.__stream = self.__configurator.getConfigStream()
-		except Exception, e:
+		except Exception as e:
 			logSys.error("Failed during configuration: %s" % e)
 			ret = False
 		return ret
--- a/bin/fail2ban-server
+++ b/bin/fail2ban-server
@ -127,7 +127,7 @@ class Fail2banServer:
 								self.__conf["pidfile"],
 								self.__conf["force"])
 			return True
-		except Exception, e:
+		except Exception as e:
 			logSys.exception(e)
 			self.__server.quit()
 			return False
--- a/bin/fail2ban-testcases
+++ b/bin/fail2ban-testcases
@ -39,10 +39,18 @@ from fail2ban.version import version

 from fail2ban.tests.utils import gatherTests
 from fail2ban.helpers import FormatterWithTraceBack, getLogger
+from fail2ban.setup import updatePyExec
 from fail2ban.server.mytime import MyTime

 from optparse import OptionParser, Option

+# Update fail2ban-python env to current python version (where f2b-modules located/installed)
+bindir = os.path.dirname(
+	# __file__ seems to be overwritten sometimes on some python versions (e.g. bug of 2.6 by running under cProfile, etc.):
+	sys.argv[0] if os.path.basename(sys.argv[0]) == 'fail2ban-testcases' else __file__
+)
+updatePyExec(bindir)
+
 def get_opt_parser():
 	# use module docstring for help output
 	p = OptionParser(
--- a/config/action.d/badips.conf
+++ b/config/action.d/badips.conf
@ -1,6 +1,6 @@
 # Fail2ban reporting to badips.com
 #
-# Note: This reports and IP only and does not actually ban traffic. Use 
+# Note: This reports an IP only and does not actually ban traffic. Use
 # another action in the same jail if you want bans to occur.
 #
 # Set the category to the appropriate value before use.
--- a/config/action.d/npf.conf
+++ b/config/action.d/npf.conf
@ -0,0 +1,61 @@
+# Fail2Ban configuration file
+#
+# NetBSD npf ban/unban
+#
+# Author: Nils Ratusznik <nils@NetBSD.org>
+# Based on pf.conf action file
+#
+
+[Definition]
+
+# Option:  actionstart
+# Notes.:  command executed once at the start of Fail2Ban.
+# Values:  CMD
+#
+# we don't enable NPF automatically, as it will be enabled elsewhere
+actionstart = 
+
+
+# Option:  actionstop
+# Notes.:  command executed once at the end of Fail2Ban
+# Values:  CMD
+#
+# we don't disable NPF automatically either
+actionstop = 
+
+
+# Option:  actioncheck
+# Notes.:  command executed once before each actionban command
+# Values:  CMD
+#
+actioncheck = 
+
+
+# Option:  actionban
+# Notes.:  command executed when banning an IP. Take care that the
+#          command is executed with Fail2Ban user rights.
+# Tags:    <ip>  IP address
+#          <failures>  number of failures
+#          <time>  unix timestamp of the ban time
+# Values:  CMD
+#
+actionban = /sbin/npfctl table <tablename> add <ip>
+
+
+# Option:  actionunban
+# Notes.:  command executed when unbanning an IP. Take care that the
+#          command is executed with Fail2Ban user rights.
+# Tags:    <ip>  IP address
+#          <failures>  number of failures
+#          <time>  unix timestamp of the ban time
+# Values:  CMD
+#
+# note -r option used to remove matching rule
+actionunban = /sbin/npfctl table <tablename> rem <ip>
+
+[Init]
+# Option:  tablename
+# Notes.:  The pf table name.
+# Values:  [ STRING ]
+#
+tablename = fail2ban
--- a/config/filter.d/apache-modsecurity.conf
+++ b/config/filter.d/apache-modsecurity.conf
@ -10,9 +10,10 @@ before = apache-common.conf
 [Definition]


-failregex = ^%(_apache_error_client)s ModSecurity:  (\[.*?\] )*Access denied with code [45]\d\d.*$
+failregex = ^%(_apache_error_client)s ModSecurity:\s+(?:\[(?:\w+ \"[^\"]*\"|[^\]]*)\]\s*)*Access denied with code [45]\d\d

 ignoreregex = 

 # https://github.com/SpiderLabs/ModSecurity/wiki/ModSecurity-2-Data-Formats
 # Author: Daniel Black
+#         Sergey G. Brester aka sebres (review, optimization)
--- a/config/filter.d/assp.conf
+++ b/config/filter.d/assp.conf
@ -1,24 +1,43 @@
-# Fail2Ban filter for Anti-Spam SMTP Proxy Server also known as ASSP
-# 
-#    Honmepage:   http://www.magicvillage.de/~Fritz_Borgstedt/assp/0003D91C-8000001C/
-#    ProjektSite: http://sourceforge.net/projects/assp/?source=directory
+# Fail2Ban filter for Anti-Spam SMTP Proxy Server (ASSP)
+#    Filter works in theory for both ASSP V1 and V2. Recommended ASSP is V2.5.1 or later.
+#    Support for ASSP V1 ended in 2014 so if you are still running ASSP V1 an immediate upgrade is recommended.
+#
+#    Homepage:    http://sourceforge.net/projects/assp/
+#    ProjectSite: http://sourceforge.net/projects/assp/?source=directory
 #
 #

 [Definition] 
+# Note: First three failregex matches below are for ASSP V1 with the remaining being designed for V2. Deleting the V1 regex is recommended but I left it in for compatibilty reasons.

 __assp_actions = (?:dropping|refusing)

 failregex = ^(:? \[SSL-out\])? <HOST> max sender authentication errors \(\d{,3}\) exceeded -- %(__assp_actions)s connection - after reply: \d{3} \d{1}\.\d{1}.\d{1} Error: authentication failed: \w+;$
 			^(?: \[SSL-out\])? <HOST> SSL negotiation with client failed: SSL accept attempt failed with unknown error.*:unknown protocol;$
 			^ Blocking <HOST> - too much AUTH errors \(\d{,3}\);$
+			^\s*(?:[\w\-]+\s+)*(?:\[\S+\]\s+)*<HOST> (?:\<\S+@\S+\.\S+\> )*(?:to: \S+@\S+\.\S+ )*relay attempt blocked for(?: \(parsing\))?: \S+$
+			^\s*(?:[\w\-]+\s+)*(?:\[\S+\]\s+)*<HOST> \[SMTP Error\] 535 5\.7\.8 Error: authentication failed:\s+(?:\S+|Connection lost to authentication server|Invalid authentication mechanism|Invalid base64 data in continued response)?$

 ignoreregex = 

 # DEV Notes:
+# V1 Examples matches:
+#   Apr-27-13 02:33:09 Blocking 217.194.197.97 - too much AUTH errors (41);
+#   Dec-29-12 17:10:31 [SSL-out] 200.247.87.82 SSL negotiation with client failed: SSL accept attempt failed with unknown errorerror:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol;
+#   Dec-30-12 04:01:47 [SSL-out] 81.82.232.66 max sender authentication errors (5) exceeded 
 #
-# Examples: Apr-27-13 02:33:09 Blocking 217.194.197.97 - too much AUTH errors (41);
-#           Dec-29-12 17:10:31 [SSL-out] 200.247.87.82 SSL negotiation with client failed: SSL accept attempt failed with unknown errorerror:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol;
-#           Dec-30-12 04:01:47 [SSL-out] 81.82.232.66 max sender authentication errors (5) exceeded 
+# V2 Examples matches:
+#   Jul-29-16 16:49:52 m1-25391-06124 [Worker_1] [TLS-out] [RelayAttempt] 0.0.0.0 <user@example.com> to: user@example.org relay attempt blocked for: someone@example.org
+#   Jul-30-16 16:59:42 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: UGFzc3dvcmQ6
+#   Jul-30-16 00:15:36 m1-52131-09651 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: UGFzc3dvcmQ6
+#   Jul-31-16 06:45:59 [Worker_1] [TLS-in] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed:
+#   Jan-05-16 08:38:49 m1-01129-09140 [Worker_1] [TLS-in] [TLS-out] [RelayAttempt] 0.0.0.0 <user@example.com> relay attempt blocked for (parsing): <user2@example>
+#   Jun-12-16 16:43:37 m1-64217-12013 [Worker_1] [TLS-in] [TLS-out] [RelayAttempt] 0.0.0.0 <user@example.com> to: user2@example.com relay attempt blocked for (parsing): <a.notheruser69@example.c>
+#   Jan-22-16 22:25:51 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: Invalid authentication mechanism
+#   Mar-19-16 13:42:20 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: Invalid base64 data in continued response
+#   Jul-18-16 16:54:21 [Worker_2] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: Connection lost to authentication server
+#   Jul-18-16 17:14:23 m1-76453-02949 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: Connection lost to authentication server
+
 #
 # Author: Enrico Labedzki (enrico.labedzki@deiwos.de)
+# V2 Filters: Robert Hardy (rhardy@webcon.ca)
--- a/config/filter.d/asterisk.conf
+++ b/config/filter.d/asterisk.conf
@ -16,7 +16,7 @@ __pid_re = (?:\[\d+\])
 iso8601 = \d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+[+-]\d{4}

 # All Asterisk log messages begin like this:
-log_prefix= (?:NOTICE|SECURITY|WARNING)%(__pid_re)s:?(?:\[C-[\da-f]*\])? [^:]+:\d*( in \w+:)?
+log_prefix= (?:NOTICE|SECURITY|WARNING)%(__pid_re)s:?(?:\[C-[\da-f]*\])? [^:]+:\d*(?:(?: in)? \w+:)?

 failregex = ^%(__prefix_line)s%(log_prefix)s Registration from '[^']*' failed for '<HOST>(:\d+)?' - (Wrong password|Username/auth name mismatch|No matching peer found|Not a local domain|Device does not match ACL|Peer is not supposed to register|ACL error \(permit/deny\)|Not a local domain)$
            ^%(__prefix_line)s%(log_prefix)s Call from '[^']*' \(<HOST>:\d+\) to extension '[^']*' rejected because extension not found in context
--- a/config/filter.d/dovecot.conf
+++ b/config/filter.d/dovecot.conf
@ -9,11 +9,11 @@ before = common.conf

 _daemon = (auth|dovecot(-auth)?|auth-worker)

-failregex = ^%(__prefix_line)s(%(__pam_auth)s(\(dovecot:auth\))?:)?\s+authentication failure; logname=\S* uid=\S* euid=\S* tty=dovecot ruser=\S* rhost=<HOST>(\s+user=\S*)?\s*$
-            ^%(__prefix_line)s(pop3|imap)-login: (Info: )?(Aborted login|Disconnected)(: Inactivity)? \(((auth failed, \d+ attempts)( in \d+ secs)?|tried to use (disabled|disallowed) \S+ auth)\):( user=<\S*>,)?( method=\S+,)? rip=<HOST>(, lip=(\d{1,3}\.){3}\d{1,3})?(, TLS( handshaking(: SSL_accept\(\) failed: error:[\dA-F]+:SSL routines:[TLS\d]+_GET_CLIENT_HELLO:unknown protocol)?)?(: Disconnected)?)?(, session=<\S+>)?\s*$
-            ^%(__prefix_line)s(Info|dovecot: auth\(default\)|auth-worker\(\d+\)): pam\(\S+,<HOST>\): pam_authenticate\(\) failed: (User not known to the underlying authentication module: \d+ Time\(s\)|Authentication failure \(password mismatch\?\))\s*$
-            ^%(__prefix_line)s(auth|auth-worker\(\d+\)): (pam|passwd-file)\(\S+,<HOST>\): unknown user\s*$
-            ^%(__prefix_line)s(auth|auth-worker\(\d+\)): Info: ldap\(\S*,<HOST>,\S*\): invalid credentials\s*$
+failregex = ^%(__prefix_line)s(?:%(__pam_auth)s(?:\(dovecot:auth\))?:)?\s+authentication failure; logname=\S* uid=\S* euid=\S* tty=dovecot ruser=\S* rhost=<HOST>(?:\s+user=\S*)?\s*$
+            ^%(__prefix_line)s(?:pop3|imap)-login: (?:Info: )?(?:Aborted login|Disconnected)(?::(?: [^ \(]+)+)? \((?:auth failed, \d+ attempts( in \d+ secs)?|tried to use (disabled|disallowed) \S+ auth)\):( user=<[^>]+>,)?( method=\S+,)? rip=<HOST>(?:, lip=\S+)?(?:, TLS(?: handshaking(?:: SSL_accept\(\) failed: error:[\dA-F]+:SSL routines:[TLS\d]+_GET_CLIENT_HELLO:unknown protocol)?)?(: Disconnected)?)?(, session=<\S+>)?\s*$
+            ^%(__prefix_line)s(?:Info|dovecot: auth\(default\)|auth-worker\(\d+\)): pam\(\S+,<HOST>\): pam_authenticate\(\) failed: (User not known to the underlying authentication module: \d+ Time\(s\)|Authentication failure \(password mismatch\?\))\s*$
+            ^%(__prefix_line)s(?:auth|auth-worker\(\d+\)): (?:pam|passwd-file)\(\S+,<HOST>\): unknown user\s*$
+            ^%(__prefix_line)s(?:auth|auth-worker\(\d+\)): Info: ldap\(\S*,<HOST>,\S*\): invalid credentials\s*$

 ignoreregex = 

@ -30,3 +30,4 @@ journalmatch = _SYSTEMD_UNIT=dovecot.service
 # Author: Martin Waschbuesch
 #         Daniel Black (rewrote with begin and end anchors)
 #         Martin O'Neal (added LDAP authentication failure regex)
+#         Sergey G. Brester aka sebres (reviewed, optimized, IPv6-compatibility)
--- a/config/filter.d/ignorecommands/apache-fakegooglebot
+++ b/config/filter.d/ignorecommands/apache-fakegooglebot
@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env fail2ban-python
 # Inspired by https://isc.sans.edu/forums/diary/When+Google+isnt+Google/15968/
 #
 # Written in Python to reuse built-in Python batteries and not depend on
--- a/config/filter.d/mongodb-auth.conf
+++ b/config/filter.d/mongodb-auth.conf
@ -0,0 +1,49 @@
+# Fail2Ban filter for unsuccesfull MongoDB authentication attempts
+#
+# Logfile /var/log/mongodb/mongodb.log
+#
+# add setting in /etc/mongodb.conf
+# logpath=/var/log/mongodb/mongodb.log
+#
+# and use of the authentication
+# auth = true
+#
+
+[Definition]
+#failregex = ^\s+\[initandlisten\] connection accepted from <HOST>:\d+ \#(?P<__connid>\d+) \(1 connection now open\)<SKIPLINES>\s+\[conn(?P=__connid)\] Failed to authenticate\s+
+failregex = ^\s+\[conn(?P<__connid>\d+)\] Failed to authenticate [^\n]+<SKIPLINES>\s+\[conn(?P=__connid)\] end connection <HOST>
+
+ignoreregex =
+
+
+[Init]
+maxlines = 10
+
+# DEV Notes:
+#
+# Regarding the multiline regex:
+#
+# There can be a nunber of non-related lines between the first and second part
+# of this regex maxlines of 10 is quite generious.
+#
+# Note the capture __connid, includes the connection ID, used in second part of regex.
+#
+# The first regex is commented out (but will match also), because it is better to use
+# the host from "end connection" line (uncommented above):
+#  -  it has the same prefix, searching begins directly with failure message
+#     (so faster, because ignores success connections at all)
+#  -  it is not so vulnerable in case of possible race condition
+#
+# Log example:
+# 2016-10-20T09:54:27.108+0200 [initandlisten] connection accepted from 127.0.0.1:53276 #1 (1 connection now open)
+# 2016-10-20T09:54:27.109+0200 [conn1]  authenticate db: test { authenticate: 1, nonce: "xxx", user: "root", key: "xxx" }
+# 2016-10-20T09:54:27.110+0200 [conn1] Failed to authenticate root@test with mechanism MONGODB-CR: AuthenticationFailed UserNotFound Could not find user root@test
+# 2016-11-09T09:54:27.894+0100 [conn1] end connection 127.0.0.1:53276 (0 connections now open)
+# 2016-11-09T11:55:58.890+0100 [initandlisten] connection accepted from 127.0.0.1:54266 #1510 (1 connection now open)
+# 2016-11-09T11:55:58.892+0100 [conn1510]  authenticate db: admin { authenticate: 1, nonce: "xxx", user: "root", key: "xxx" }
+# 2016-11-09T11:55:58.892+0100 [conn1510] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed key mismatch
+# 2016-11-09T11:55:58.894+0100 [conn1510] end connection 127.0.0.1:54266 (0 connections now open)
+#
+# Authors: Alexander Finkhäuser
+#          Sergey G. Brester (sebres)
+
--- a/config/filter.d/postfix-sasl.conf
+++ b/config/filter.d/postfix-sasl.conf
@ -9,7 +9,7 @@ before = common.conf

 _daemon = postfix(-\w+)?/(?:submission/|smtps/)?smtp[ds]

-failregex = ^%(__prefix_line)swarning: [-._\w]+\[<HOST>\]: SASL ((?i)LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed(: [ A-Za-z0-9+/:]*={0,2})?\s*$
+failregex = ^%(__prefix_line)swarning: [-._\w]+\[<HOST>\]: SASL ((?i)LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed(:[ A-Za-z0-9+/:]*={0,2})?\s*$

 ignoreregex = authentication failed: Connection lost to authentication server$

--- a/config/filter.d/sendmail-reject.conf
+++ b/config/filter.d/sendmail-reject.conf
@ -23,7 +23,7 @@ _daemon = (?:(sm-(mta|acceptingconnections)|sendmail))

 failregex = ^%(__prefix_line)s\w{14}: ruleset=check_rcpt, arg1=(?P<email><\S+@\S+>), relay=(\S+ )?\[<HOST>\]( \(may be forged\))?, reject=(550 5\.7\.1 (?P=email)\.\.\. Relaying denied\. (IP name possibly forged \[(\d+\.){3}\d+\]|Proper authentication required\.|IP name lookup failed \[(\d+\.){3}\d+\])|553 5\.1\.8 (?P=email)\.\.\. Domain of sender address \S+ does not exist|550 5\.[71]\.1 (?P=email)\.\.\. (Rejected: .*|User unknown))$
            ^%(__prefix_line)sruleset=check_relay, arg1=(?P<dom>\S+), arg2=<HOST>, relay=((?P=dom) )?\[(\d+\.){3}\d+\]( \(may be forged\))?, reject=421 4\.3\.2 (Connection rate limit exceeded\.|Too many open connections\.)$
-            ^%(__prefix_line)s\w{14}: rejecting commands from  (\S+ )?\[<HOST>\] due to pre-greeting traffic after \d+ seconds$
+            ^%(__prefix_line)s\w{14}: rejecting commands from (\S* )?\[<HOST>\] due to pre-greeting traffic after \d+ seconds$
            ^%(__prefix_line)s\w{14}: (\S+ )?\[<HOST>\]: ((?i)expn|vrfy) \S+ \[rejected\]$
            ^(?P<__prefix>%(__prefix_line)s\w+: )<[^@]+@[^>]+>\.\.\. No such user here<SKIPLINES>(?P=__prefix)from=<[^@]+@[^>]+>, size=\d+, class=\d+, nrcpts=\d+, bodytype=\w+, proto=E?SMTP, daemon=MTA, relay=\S+ \[<HOST>\]$

--- a/config/filter.d/sshd.conf
+++ b/config/filter.d/sshd.conf
@ -20,9 +20,9 @@ _daemon = sshd

 failregex = ^%(__prefix_line)s(?:error: PAM: )?[aA]uthentication (?:failure|error|failed) for .* from <HOST>( via \S+)?\s*$
            ^%(__prefix_line)s(?:error: PAM: )?User not known to the underlying authentication module for .* from <HOST>\s*$
-            ^%(__prefix_line)sFailed \S+ for .*? from <HOST>(?: port \d*)?(?: ssh\d*)?(: (ruser .*|(\S+ ID \S+ \(serial \d+\) CA )?\S+ %(__md5hex)s(, client user ".*", client host ".*")?))?\s*$
+            ^%(__prefix_line)sFailed \S+ for (?P<cond_inv>invalid user )?(?P<user>(?P<cond_user>\S+)|(?(cond_inv)(?:(?! from ).)*?|[^:]+)) from <HOST>(?: port \d+)?(?: ssh\d*)?(?(cond_user):|(?:(?:(?! from ).)*)$)
            ^%(__prefix_line)sROOT LOGIN REFUSED.* FROM <HOST>\s*$
-            ^%(__prefix_line)s[iI](?:llegal|nvalid) user .* from <HOST>\s*$
+            ^%(__prefix_line)s[iI](?:llegal|nvalid) user .*? from <HOST>(?: port \d+)?\s*$
            ^%(__prefix_line)sUser .+ from <HOST> not allowed because not listed in AllowUsers\s*$
            ^%(__prefix_line)sUser .+ from <HOST> not allowed because listed in DenyUsers\s*$
            ^%(__prefix_line)sUser .+ from <HOST> not allowed because not in any group\s*$
--- a/config/filter.d/vsftpd.conf
+++ b/config/filter.d/vsftpd.conf
@ -14,7 +14,7 @@ __pam_re=\(?%(__pam_auth)s(?:\(\S+\))?\)?:?
 _daemon =  vsftpd

 failregex = ^%(__prefix_line)s%(__pam_re)s\s+authentication failure; logname=\S* uid=\S* euid=\S* tty=(ftp)? ruser=\S* rhost=<HOST>(?:\s+user=.*)?\s*$
-            ^ \[pid \d+\] \[.+\] FAIL LOGIN: Client "<HOST>"\s*$
+            ^ \[pid \d+\] \[[^\]]+\] FAIL LOGIN: Client "<HOST>"(?:\s*$|,)

 ignoreregex = 

--- a/config/jail.conf
+++ b/config/jail.conf
@ -731,6 +731,13 @@ logpath  = %(mysql_log)s
 backend  = %(mysql_backend)s


+# Log wrong MongoDB auth (for details see filter 'filter.d/mongodb-auth.conf')
+[mongodb-auth]
+# change port when running with "--shardsvr" or "--configsvr" runtime operation
+port     = 27017
+logpath  = /var/log/mongodb/mongodb.log
+
+
 # Jail for more extended banning of persistent abusers
 # !!! WARNINGS !!!
 # 1. Make sure that your loglevel specified in fail2ban.conf/.local
@ -810,8 +817,9 @@ maxretry = 1
 [pass2allow-ftp]
 # this pass2allow example allows FTP traffic after successful HTTP authentication
 port         = ftp,ftp-data,ftps,ftps-data
-# knocking_url variable must be overridden to some secret value in filter.d/apache-pass.local
-filter       = apache-pass
+# knocking_url variable must be overridden to some secret value in jail.local
+knocking_url = /knocking/
+filter       = apache-pass[knocking_url="%(knocking_url)s"]
 # access log of the website with HTTP auth
 logpath      = %(apache_access_log)s
 blocktype    = RETURN
--- a/config/paths-opensuse.conf
+++ b/config/paths-opensuse.conf
@ -36,3 +36,15 @@ mysql_log = /var/log/mysql/mysqld.log
 roundcube_errors_log = /srv/www/roundcubemail/logs/errors

 solidpop3d_log = %(syslog_mail)s
+
+# These services will log to the journal via syslog, so use the journal by
+# default.
+syslog_backend = systemd
+sshd_backend = systemd
+dropbear_backend = systemd
+proftpd_backend = systemd
+pureftpd_backend = systemd
+wuftpd_backend = systemd
+postfix_backend = systemd
+dovecot_backend = systemd
+mysql_backend = systemd
--- a/fail2ban/client/configparserinc.py
+++ b/fail2ban/client/configparserinc.py
@ -168,7 +168,7 @@ after = 1.conf
 			parser, i = self._getSharedSCPWI(resource)
 			if not i:
 				return []
-		except UnicodeDecodeError, e:
+		except UnicodeDecodeError as e:
 			logSys.error("Error decoding config file '%s': %s" % (resource, e))
 			return []
 		
--- a/fail2ban/client/configreader.py
+++ b/fail2ban/client/configreader.py
@ -221,7 +221,7 @@ class ConfigReaderUnshared(SafeConfigParserWithIncludes):
 				if not pOptions is None and option[1] in pOptions:
 					continue
 				values[option[1]] = v
-			except NoSectionError, e:
+			except NoSectionError as e:
 				# No "Definition" section or wrong basedir
 				logSys.error(e)
 				values[option[1]] = option[2]
--- a/fail2ban/client/fail2banregex.py
+++ b/fail2ban/client/fail2banregex.py
@ -329,7 +329,7 @@ class Fail2banRegex(object):
 			if ret is not None:
 				found = True
 				regex = self._ignoreregex[ret].inc()
-		except RegexException, e:
+		except RegexException as e:
 			output( e )
 			return False
 		return found
@ -346,7 +346,7 @@ class Fail2banRegex(object):
 				regex = self._failregex[match[0]]
 				regex.inc()
 				regex.appendIP(match)
-		except RegexException, e:
+		except RegexException as e:
 			output( e )
 			return False
 		except IndexError:
@ -510,7 +510,7 @@ class Fail2banRegex(object):
 				output( "Use         log file : %s" % cmd_log )
 				output( "Use         encoding : %s" % self.encoding )
 				test_lines = self.file_lines_gen(hdlr)
-			except IOError, e:
+			except IOError as e:
 				output( e )
 				return False
 		elif cmd_log == "systemd-journal": # pragma: no cover
--- a/fail2ban/client/jailreader.py
+++ b/fail2ban/client/jailreader.py
@ -171,7 +171,7 @@ class JailReader(ConfigReader):
 							self.__actions.append(action)
 						else:
 							raise AttributeError("Unable to read action")
-				except Exception, e:
+				except Exception as e:
 					logSys.error("Error in action definition " + act)
 					logSys.debug("Caught exception: %s" % (e,))
 					return False
@ -192,7 +192,7 @@ class JailReader(ConfigReader):
 		stream = []
 		for opt in self.__opts:
 			if opt == "logpath" and	\
-					self.__opts.get('backend', None) != "systemd":
+					not self.__opts.get('backend', None).startswith("systemd"):
 				found_files = 0
 				for path in self.__opts[opt].split("\n"):
 					path = path.rsplit(" ", 1)
--- a/fail2ban/server/database.py
+++ b/fail2ban/server/database.py
@ -42,7 +42,7 @@ if sys.version_info >= (3,):
 		try:
 			x = json.dumps(x, ensure_ascii=False).encode(
 				locale.getpreferredencoding(), 'replace')
-		except Exception, e: # pragma: no cover
+		except Exception as e: # pragma: no cover
 			logSys.error('json dumps failed: %s', e)
 			x = '{}'
 		return x
@ -51,7 +51,7 @@ if sys.version_info >= (3,):
 		try:
 			x = json.loads(x.decode(
 				locale.getpreferredencoding(), 'replace'))
-		except Exception, e: # pragma: no cover
+		except Exception as e: # pragma: no cover
 			logSys.error('json loads failed: %s', e)
 			x = {}
 		return x
@ -70,7 +70,7 @@ else:
 		try:
 			x = json.dumps(_normalize(x), ensure_ascii=False).decode(
 				locale.getpreferredencoding(), 'replace')
-		except Exception, e: # pragma: no cover
+		except Exception as e: # pragma: no cover
 			logSys.error('json dumps failed: %s', e)
 			x = '{}'
 		return x
@ -79,7 +79,7 @@ else:
 		try:
 			x = _normalize(json.loads(x.decode(
 				locale.getpreferredencoding(), 'replace')))
-		except Exception, e: # pragma: no cover
+		except Exception as e: # pragma: no cover
 			logSys.error('json loads failed: %s', e)
 			x = {}
 		return x
@ -175,7 +175,7 @@ class Fail2BanDb(object):

 			logSys.info(
 				"Connected to fail2ban persistent database '%s'", filename)
-		except sqlite3.OperationalError, e:
+		except sqlite3.OperationalError as e:
 			logSys.error(
 				"Error connecting to fail2ban persistent database '%s': %s",
 				filename, e.args[0])
@ -293,8 +293,12 @@ class Fail2BanDb(object):
 			Jail to be added to the database.
 		"""
 		cur.execute(
-			"INSERT OR REPLACE INTO jails(name, enabled) VALUES(?, 1)",
+			"INSERT OR IGNORE INTO jails(name, enabled) VALUES(?, 1)",
 			(jail.name,))
+		if cur.rowcount <= 0:
+			cur.execute(
+				"UPDATE jails SET enabled = 1 WHERE name = ? AND enabled != 1",
+				(jail.name,))

 	@commitandrollback
 	def delJail(self, cur, jail):
@ -317,7 +321,7 @@ class Fail2BanDb(object):
 		cur.execute("UPDATE jails SET enabled=0")

 	@commitandrollback
-	def getJailNames(self, cur):
+	def getJailNames(self, cur, enabled=None):
 		"""Get name of jails in database.

 		Currently only used for testing purposes.
@ -327,7 +331,11 @@ class Fail2BanDb(object):
 		set
 			Set of jail names.
 		"""
-		cur.execute("SELECT name FROM jails")
+		if enabled is None:
+			cur.execute("SELECT name FROM jails")
+		else:
+			cur.execute("SELECT name FROM jails WHERE enabled=%s" %
+				(int(enabled),))
 		return set(row[0] for row in cur.fetchmany())

 	@commitandrollback
--- a/fail2ban/server/datetemplate.py
+++ b/fail2ban/server/datetemplate.py
@ -64,7 +64,7 @@ class DateTemplate(object):
 	def getRegex(self):
 		return self._regex

-	def setRegex(self, regex, wordBegin=True):
+	def setRegex(self, regex, wordBegin=True, wordEnd=True):
 		"""Sets regex to use for searching for date in log line.

 		Parameters
@ -72,8 +72,12 @@ class DateTemplate(object):
 		regex : str
 			The regex the template will use for searching for a date.
 		wordBegin : bool
-			Defines whether the regex should be modified to search at
-			beginning of a word, by adding "\\b" to start of regex.
+			Defines whether the regex should be modified to search at beginning of a
+			word, by adding special boundary r'(?=^|\b|\W)' to start of regex.
+			Default True.
+		wordEnd : bool
+			Defines whether the regex should be modified to search at end of a word,
+			by adding special boundary r'(?=\b|\W|$)' to end of regex.
 			Default True.

 		Raises
@ -82,8 +86,10 @@ class DateTemplate(object):
 			If regular expression fails to compile
 		"""
 		regex = regex.strip()
-		if (wordBegin and not re.search(r'^\^', regex)):
-			regex = r'\b' + regex
+		if wordBegin and not re.search(r'^\^', regex):
+			regex = r'(?=^|\b|\W)' + regex
+		if wordEnd and not re.search(r'\$$', regex):
+			regex += r'(?=\b|\W|$)'
 		self._regex = regex
 		self._cRegex = re.compile(regex, re.UNICODE | re.IGNORECASE)

--- a/fail2ban/server/filter.py
+++ b/fail2ban/server/filter.py
@ -24,6 +24,7 @@ __license__ = "GPL"
 import codecs
 import fcntl
 import locale
+import logging
 import os
 import re
 import sys
@ -82,6 +83,8 @@ class Filter(JailThread):
 		self.__lastDate = None
 		## External command
 		self.__ignoreCommand = False
+		## Default or preferred encoding (to decode bytes from file or journal):
+		self.__encoding = locale.getpreferredencoding()

 		self.dateDetector = DateDetector()
 		self.dateDetector.addDefaultTemplate()
@ -105,7 +108,7 @@ class Filter(JailThread):
 				logSys.warning(
 					"Mutliline regex set for jail '%s' "
 					"but maxlines not greater than 1")
-		except RegexException, e:
+		except RegexException as e:
 			logSys.error(e)
 			raise e

@ -138,7 +141,7 @@ class Filter(JailThread):
 		try:
 			regex = Regex(value)
 			self.__ignoreRegex.append(regex)
-		except RegexException, e:
+		except RegexException as e:
 			logSys.error(e)
 			raise e 

@ -279,6 +282,27 @@ class Filter(JailThread):
 	def getMaxLines(self):
 		return self.__lineBufferSize

+	##
+	# Set the log file encoding
+	#
+	# @param encoding the encoding used with log files
+
+	def setLogEncoding(self, encoding):
+		if encoding.lower() == "auto":
+			encoding = locale.getpreferredencoding()
+		codecs.lookup(encoding) # Raise LookupError if invalid codec
+		self.__encoding = encoding
+		logSys.info("Set jail log file encoding to %s" % encoding)
+		return encoding
+
+	##
+	# Get the log file encoding
+	#
+	# @return log encoding value
+
+	def getLogEncoding(self):
+		return self.__encoding
+
 	##
 	# Main loop.
 	#
@ -394,8 +418,31 @@ class Filter(JailThread):

 		return False

+	if sys.version_info >= (3,):
+		@staticmethod
+		def uni_decode(x, enc, errors='strict'):
+			try:
+				if isinstance(x, bytes):
+					return x.decode(enc, errors)
+				return x
+			except (UnicodeDecodeError, UnicodeEncodeError): # pragma: no cover - unsure if reachable
+				if errors != 'strict': 
+					raise
+				return uni_decode(x, enc, 'replace')
+	else:
+		@staticmethod
+		def uni_decode(x, enc, errors='strict'):
+			try:
+				if isinstance(x, unicode):
+					return x.encode(enc, errors)
+				return x
+			except (UnicodeDecodeError, UnicodeEncodeError): # pragma: no cover - unsure if reachable
+				if errors != 'strict':
+					raise
+				return uni_decode(x, enc, 'replace')
+
 	def processLine(self, line, date=None, returnRawHost=False,
-		checkAllRegex=False):
+		checkAllRegex=False, checkFindTime=False):
 		"""Split the time portion from log msg and return findFailures on them
 		"""
 		if date:
@ -414,21 +461,17 @@ class Filter(JailThread):
 				tupleLine = (l, "", "")

 		return "".join(tupleLine[::2]), self.findFailure(
-			tupleLine, date, returnRawHost, checkAllRegex)
+			tupleLine, date, returnRawHost, checkAllRegex, checkFindTime)

 	def processLineAndAdd(self, line, date=None):
 		"""Processes the line for failures and populates failManager
 		"""
-		for element in self.processLine(line, date)[1]:
+		for element in self.processLine(line, date, checkFindTime=True)[1]:
 			ip = element[1]
 			unixTime = element[2]
 			lines = element[3]
 			logSys.debug("Processing line with time:%s and ip:%s"
 						 % (unixTime, ip))
-			if unixTime < MyTime.time() - self.getFindTime():
-				logSys.debug("Ignore line since time %s < %s - %s"
-							 % (unixTime, MyTime.time(), self.getFindTime()))
-				break
 			if self.inIgnoreIPList(ip, log_ignore=True):
 				continue
 			logSys.info("[%s] Found %s" % (self.jail.name, ip))
@ -457,7 +500,7 @@ class Filter(JailThread):
 	# @return a dict with IP and timestamp.

 	def findFailure(self, tupleLine, date=None, returnRawHost=False,
-		checkAllRegex=False):
+		checkAllRegex=False, checkFindTime=False):
 		failList = list()

 		# Checks if we must ignore this line.
@ -489,6 +532,11 @@ class Filter(JailThread):
 			timeText = self.__lastTimeText or "".join(tupleLine[::2])
 			date = self.__lastDate

+		if checkFindTime and date is not None and date < MyTime.time() - self.getFindTime():
+			logSys.log(5, "Ignore line since time %s < %s - %s", 
+				date, MyTime.time(), self.getFindTime())
+			return failList
+
 		self.__lineBuffer = (
 			self.__lineBuffer + [tupleLine])[-self.__lineBufferSize:]
 		logSys.log(5, "Looking for failregex match of %r" % self.__lineBuffer)
@ -536,7 +584,7 @@ class Filter(JailThread):
 										 failRegex.getMatchedLines()])
 								if not checkAllRegex:
 									break
-					except RegexException, e: # pragma: no cover - unsure if reachable
+					except RegexException as e: # pragma: no cover - unsure if reachable
 						logSys.error(e)
 		return failList

@ -554,7 +602,6 @@ class FileFilter(Filter):
 		Filter.__init__(self, jail, **kwargs)
 		## The log file path.
 		self.__logs = dict()
-		self.setLogEncoding("auto")

 	##
 	# Add a log file path
@ -625,21 +672,9 @@ class FileFilter(Filter):
 	# @param encoding the encoding used with log files

 	def setLogEncoding(self, encoding):
-		if encoding.lower() == "auto":
-			encoding = locale.getpreferredencoding()
-		codecs.lookup(encoding) # Raise LookupError if invalid codec
+		encoding = super(FileFilter, self).setLogEncoding(encoding)
 		for log in self.__logs.itervalues():
 			log.setEncoding(encoding)
-		self.__encoding = encoding
-		logSys.info("Set jail log file encoding to %s" % encoding)
-
-	##
-	# Get the log file encoding
-	#
-	# @return log encoding value
-
-	def getLogEncoding(self):
-		return self.__encoding

 	def getLog(self, path):
 		return self.__logs.get(path, None)
@ -660,15 +695,15 @@ class FileFilter(Filter):
 		try:
 			has_content = log.open()
 		# see http://python.org/dev/peps/pep-3151/
-		except IOError, e:
+		except IOError as e:
 			logSys.error("Unable to open %s" % filename)
 			logSys.exception(e)
 			return False
-		except OSError, e: # pragma: no cover - requires race condition to tigger this
+		except OSError as e: # pragma: no cover - requires race condition to tigger this
 			logSys.error("Error opening %s" % filename)
 			logSys.exception(e)
 			return False
-		except Exception, e: # pragma: no cover - Requires implemention error in FileContainer to generate
+		except Exception as e: # pragma: no cover - Requires implemention error in FileContainer to generate
 			logSys.error("Internal errror in FileContainer open method - please report as a bug to https://github.com/fail2ban/fail2ban/issues")
 			logSys.exception(e)
 			return False
@ -707,7 +742,12 @@ class FileFilter(Filter):

 try:
 	import hashlib
-	md5sum = hashlib.md5
+	try:
+		md5sum = hashlib.md5
+		# try to use it (several standards like FIPS forbid it):
+		md5sum(' ').hexdigest()
+	except: # pragma: no cover
+		md5sum = hashlib.sha1
 except ImportError: # pragma: no cover
 	# hashlib was introduced in Python 2.5.  For compatibility with those
 	# elderly Pythons, import from md5
@ -791,14 +831,19 @@ class FileContainer:
 	@staticmethod
 	def decode_line(filename, enc, line):
 		try:
-			line = line.decode(enc, 'strict')
-		except UnicodeDecodeError:
-			logSys.warning(
+			return line.decode(enc, 'strict')
+		except (UnicodeDecodeError, UnicodeEncodeError) as e:
+			global _decode_line_warn
+			lev = logging.DEBUG
+			if _decode_line_warn.get(filename, 0) <= MyTime.time():
+				lev = logging.WARNING
+				_decode_line_warn[filename] = MyTime.time() + 24*60*60
+			logSys.log(lev,
 				"Error decoding line from '%s' with '%s'."
 				" Consider setting logencoding=utf-8 (or another appropriate"
 				" encoding) for this jail. Continuing"
-				" to process line ignoring invalid characters: %r" %
-				(filename, enc, line))
+				" to process line ignoring invalid characters: %r",
+				filename, enc, line)
 			# decode with replacing error chars:
 			line = line.decode(enc, 'replace')
 		return line
@ -819,6 +864,8 @@ class FileContainer:
 		## print "D: Closed %s with pos %d" % (handler, self.__pos)
 		## sys.stdout.flush()

+_decode_line_warn = {}
+

 ##
 # JournalFilter class.
@ -858,11 +905,11 @@ class DNSUtils:
 		# retrieve ip (todo: use AF_INET6 for IPv6)
 		try:
 			return set([i[4][0] for i in socket.getaddrinfo(dns, None, socket.AF_INET, 0, socket.IPPROTO_TCP)])
-		except socket.error, e:
+		except socket.error as e:
 			logSys.warning("Unable to find a corresponding IP address for %s: %s"
 						% (dns, e))
 			return list()
-		except socket.error, e:
+		except socket.error as e:
 			logSys.warning("Socket error raised trying to resolve hostname %s: %s"
 						% (dns, e))
 			return list()
@ -871,7 +918,7 @@ class DNSUtils:
 	def ipToName(ip):
 		try:
 			return socket.gethostbyaddr(ip)[0]
-		except socket.error, e:
+		except socket.error as e:
 			logSys.debug("Unable to find a name for the IP %s: %s" % (ip, e))
 			return None

--- a/fail2ban/server/filterpoll.py
+++ b/fail2ban/server/filterpoll.py
@ -138,7 +138,7 @@ class FilterPoll(FileFilter):
 				logSys.debug("%s has been modified", filename)
 				self.__prevStats[filename] = stats
 				return True
-		except OSError, e:
+		except OSError as e:
 			logSys.error("Unable to get stat on %s because of: %s"
 						 % (filename, e))
 			self.__file404Cnt[filename] += 1
--- a/fail2ban/server/filterpyinotify.py
+++ b/fail2ban/server/filterpyinotify.py
@ -44,7 +44,7 @@ if not hasattr(pyinotify, '__version__') \
 try:
 	manager = pyinotify.WatchManager()
 	del manager
-except Exception, e:
+except Exception as e:
 	raise ImportError("Pyinotify is probably not functional on this system: %s"
 					  % str(e))

--- a/fail2ban/server/filtersystemd.py
+++ b/fail2ban/server/filtersystemd.py
@ -31,9 +31,9 @@ if LooseVersion(getattr(journal, '__version__', "0")) < '204':
 	raise ImportError("Fail2Ban requires systemd >= 204")

 from .failmanager import FailManagerEmpty
-from .filter import JournalFilter
+from .filter import JournalFilter, Filter
 from .mytime import MyTime
-from ..helpers import getLogger
+from ..helpers import getLogger, logging, splitwords

 # Gets the instance of the logger.
 logSys = getLogger(__name__)
@ -54,14 +54,45 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 	# @param jail the jail object

 	def __init__(self, jail, **kwargs):
+		jrnlargs = FilterSystemd._getJournalArgs(kwargs)
 		JournalFilter.__init__(self, jail, **kwargs)
-		self.__modified = False
+		self.__modified = 0
 		# Initialise systemd-journal connection
-		self.__journal = journal.Reader(converters={'__CURSOR': lambda x: x})
+		self.__journal = journal.Reader(**jrnlargs)
 		self.__matches = []
 		self.setDatePattern(None)
+		self.ticks = 0
 		logSys.debug("Created FilterSystemd")

+	@staticmethod
+	def _getJournalArgs(kwargs):
+		args = {'converters':{'__CURSOR': lambda x: x}}
+		try:
+			args['path'] = kwargs.pop('journalpath')
+		except KeyError:
+			pass
+
+		try:
+			args['files'] = kwargs.pop('journalfiles')
+		except KeyError:
+			pass
+		else:
+			import glob
+			p = args['files']
+			if not isinstance(p, (list, set, tuple)):
+				p = splitwords(p)
+			files = []
+			for p in p:
+				files.extend(glob.glob(p))
+			args['files'] = list(set(files))
+
+		try:
+			args['flags'] = kwargs.pop('journalflags')
+		except KeyError:
+			pass
+
+		return args
+
 	##
 	# Add a journal match filters from list structure
 	#
@ -139,21 +170,9 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 	def getJournalMatch(self):
 		return self.__matches

-    ##
-    # Join group of log elements which may be a mix of bytes and strings
-    #
-    # @param elements list of strings and bytes
-    # @return elements joined as string
-
-	@staticmethod
-	def _joinStrAndBytes(elements):
-		strElements = []
-		for element in elements:
-			if isinstance(element, str):
-				strElements.append(element)
-			else:
-				strElements.append(str(element, errors='ignore'))
-		return " ".join(strElements)
+	def uni_decode(self, x):
+		v = Filter.uni_decode(x, self.getLogEncoding())
+		return v

 	##
 	# Format journal log entry into syslog style
@ -161,52 +180,51 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 	# @param entry systemd journal entry dict
 	# @return format log line

-	@classmethod
-	def formatJournalEntry(cls, logentry):
-		logelements = [""]
-		if logentry.get('_HOSTNAME'):
-			logelements.append(logentry['_HOSTNAME'])
-		if logentry.get('SYSLOG_IDENTIFIER'):
-			logelements.append(logentry['SYSLOG_IDENTIFIER'])
-			if logentry.get('SYSLOG_PID'):
-				logelements[-1] += ("[%i]" % logentry['SYSLOG_PID'])
-			elif logentry.get('_PID'):
-				logelements[-1] += ("[%i]" % logentry['_PID'])
+	def formatJournalEntry(self, logentry):
+		# Be sure, all argument of line tuple should have the same type:
+		uni_decode = self.uni_decode
+		logelements = []
+		v = logentry.get('_HOSTNAME')
+		if v:
+			logelements.append(uni_decode(v))
+		v = logentry.get('SYSLOG_IDENTIFIER')
+		if not v:
+			v = logentry.get('_COMM')
+		if v:
+			logelements.append(uni_decode(v))
+			v = logentry.get('SYSLOG_PID')
+			if not v:
+				v = logentry.get('_PID')
+			if v:
+				logelements[-1] += ("[%i]" % v)
 			logelements[-1] += ":"
-		elif logentry.get('_COMM'):
-			logelements.append(logentry['_COMM'])
-			if logentry.get('_PID'):
-				logelements[-1] += ("[%i]" % logentry['_PID'])
-			logelements[-1] += ":"
-		if logelements[-1] == "kernel:":
-			if '_SOURCE_MONOTONIC_TIMESTAMP' in logentry:
-				monotonic = logentry.get('_SOURCE_MONOTONIC_TIMESTAMP')
-			else:
-				monotonic = logentry.get('__MONOTONIC_TIMESTAMP')[0]
-			logelements.append("[%12.6f]" % monotonic.total_seconds())
-		if isinstance(logentry.get('MESSAGE',''), list):
-			logelements.append(" ".join(logentry['MESSAGE']))
+			if logelements[-1] == "kernel:":
+				if '_SOURCE_MONOTONIC_TIMESTAMP' in logentry:
+					monotonic = logentry.get('_SOURCE_MONOTONIC_TIMESTAMP')
+				else:
+					monotonic = logentry.get('__MONOTONIC_TIMESTAMP')[0]
+				logelements.append("[%12.6f]" % monotonic.total_seconds())
+		msg = logentry.get('MESSAGE','')
+		if isinstance(msg, list):
+			logelements.append(" ".join(uni_decode(v) for v in msg))
 		else:
-			logelements.append(logentry.get('MESSAGE', ''))
+			logelements.append(uni_decode(msg))

-		try:
-			logline = u" ".join(logelements)
-		except UnicodeDecodeError:
-			# Python 2, so treat as string
-			logline = " ".join([str(logline) for logline in logelements])
-		except TypeError:
-			# Python 3, one or more elements bytes
-			logSys.warning("Error decoding log elements from journal: %s" %
-				repr(logelements))
-			logline =  cls._joinStrAndBytes(logelements)
+		logline = " ".join(logelements)

 		date = logentry.get('_SOURCE_REALTIME_TIMESTAMP',
 				logentry.get('__REALTIME_TIMESTAMP'))
 		logSys.debug("Read systemd journal entry: %r" %
 			"".join([date.isoformat(), logline]))
-		return (('', date.isoformat(), logline),
+		## use the same type for 1st argument:
+		return ((logline[:0], date.isoformat(), logline),
 			time.mktime(date.timetuple()) + date.microsecond/1.0E6)

+	def seekToTime(self, date):
+		if not isinstance(date, datetime.datetime):
+			date = datetime.datetime.fromtimestamp(date)
+		self.__journal.seek_realtime(date)
+
 	##
 	# Main loop.
 	#
@ -224,7 +242,7 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 		# Seek to now - findtime in journal
 		start_time = datetime.datetime.now() - \
 				datetime.timedelta(seconds=int(self.getFindTime()))
-		self.__journal.seek_realtime(start_time)
+		self.seekToTime(start_time)
 		# Move back one entry to ensure do not end up in dead space
 		# if start time beyond end of journal
 		try:
@ -233,29 +251,38 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 			pass # Reading failure, so safe to ignore

 		while self.active:
-			if not self.idle:
-				while self.active:
-					try:
-						logentry = self.__journal.get_next()
-					except OSError:
-						logSys.warning(
-							"Error reading line from systemd journal")
-						continue
-					if logentry:
-						self.processLineAndAdd(
-							*self.formatJournalEntry(logentry))
-						self.__modified = True
-					else:
-						break
-				if self.__modified:
-					try:
-						while True:
-							ticket = self.failManager.toBan()
-							self.jail.putFailTicket(ticket)
-					except FailManagerEmpty:
-						self.failManager.cleanup(MyTime.time())
-					self.__modified = False
+			# wait for records (or for timeout in sleeptime seconds):
 			self.__journal.wait(self.sleeptime)
+			if self.idle:
+				# because journal.wait will returns immediatelly if we have records in journal,
+				# just wait a little bit here for not idle, to prevent hi-load:
+				time.sleep(self.sleeptime)
+				continue
+			self.__modified = 0
+			while self.active:
+				logentry = None
+				try:
+					logentry = self.__journal.get_next()
+				except OSError as e:
+					logSys.error("Error reading line from systemd journal: %s",
+						e, exc_info=logSys.getEffectiveLevel() <= logging.DEBUG)
+				self.ticks += 1
+				if logentry:
+					self.processLineAndAdd(
+						*self.formatJournalEntry(logentry))
+					self.__modified += 1
+					if self.__modified >= 100: # todo: should be configurable
+						break
+				else:
+					break
+			if self.__modified:
+				try:
+					while True:
+						ticket = self.failManager.toBan()
+						self.jail.putFailTicket(ticket)
+				except FailManagerEmpty:
+					self.failManager.cleanup(MyTime.time())
+
 		logSys.debug((self.jail is not None and self.jail.name
                      or "jailless") +" filter terminated")
 		return True
--- a/fail2ban/server/jail.py
+++ b/fail2ban/server/jail.py
@ -27,6 +27,7 @@ import logging
 import Queue

 from .actions import Actions
+from ..client.jailreader import JailReader
 from ..helpers import getLogger

 # Gets the instance of the logger.
@ -82,6 +83,7 @@ class Jail:
 		return "%s(%r)" % (self.__class__.__name__, self.name)

 	def _setBackend(self, backend):
+		backend, beArgs = JailReader.extractOptions(backend)
 		backend = backend.lower()		# to assure consistent matching

 		backends = self._BACKENDS
@ -98,7 +100,7 @@ class Jail:
 		for b in backends:
 			initmethod = getattr(self, '_init%s' % b.capitalize())
 			try:
-				initmethod()
+				initmethod(**beArgs)
 				if backend != 'auto' and b != backend:
 					logSys.warning("Could only initiated %r backend whenever "
 								   "%r was requested" % (b, backend))
@ -106,7 +108,7 @@ class Jail:
 					logSys.info("Initiated %r backend" % b)
 				self.__actions = Actions(self)
 				return					# we are done
-			except ImportError, e:
+			except ImportError as e:
 				# Log debug if auto, but error if specific
 				logSys.log(
 					logging.DEBUG if backend == "auto" else logging.ERROR,
@ -117,28 +119,28 @@ class Jail:
 		raise RuntimeError(
 			"Failed to initialize any backend for Jail %r" % self.name)

-	def _initPolling(self):
+	def _initPolling(self, **kwargs):
 		from filterpoll import FilterPoll
-		logSys.info("Jail '%s' uses poller" % self.name)
-		self.__filter = FilterPoll(self)
+		logSys.info("Jail '%s' uses poller %r" % (self.name, kwargs))
+		self.__filter = FilterPoll(self, **kwargs)

-	def _initGamin(self):
+	def _initGamin(self, **kwargs):
 		# Try to import gamin
 		from filtergamin import FilterGamin
-		logSys.info("Jail '%s' uses Gamin" % self.name)
-		self.__filter = FilterGamin(self)
+		logSys.info("Jail '%s' uses Gamin %r" % (self.name, kwargs))
+		self.__filter = FilterGamin(self, **kwargs)

-	def _initPyinotify(self):
+	def _initPyinotify(self, **kwargs):
 		# Try to import pyinotify
 		from filterpyinotify import FilterPyinotify
-		logSys.info("Jail '%s' uses pyinotify" % self.name)
-		self.__filter = FilterPyinotify(self)
+		logSys.info("Jail '%s' uses pyinotify %r" % (self.name, kwargs))
+		self.__filter = FilterPyinotify(self, **kwargs)

-	def _initSystemd(self): # pragma: systemd no cover
+	def _initSystemd(self, **kwargs): # pragma: systemd no cover
 		# Try to import systemd
 		from filtersystemd import FilterSystemd
-		logSys.info("Jail '%s' uses systemd" % self.name)
-		self.__filter = FilterSystemd(self)
+		logSys.info("Jail '%s' uses systemd %r" % (self.name, kwargs))
+		self.__filter = FilterSystemd(self, **kwargs)

 	@property
 	def name(self):
--- a/fail2ban/server/server.py
+++ b/fail2ban/server/server.py
@ -108,20 +108,20 @@ class Server:
 			pidFile = open(pidfile, 'w')
 			pidFile.write("%s\n" % os.getpid())
 			pidFile.close()
-		except IOError, e:
+		except IOError as e:
 			logSys.error("Unable to create PID file: %s" % e)
 		
 		# Start the communication
 		logSys.debug("Starting communication")
 		try:
 			self.__asyncServer.start(sock, force)
-		except AsyncServerException, e:
+		except AsyncServerException as e:
 			logSys.error("Could not start server: %s", e)
 		# Removes the PID file.
 		try:
 			logSys.debug("Remove PID file %s" % pidfile)
 			os.remove(pidfile)
-		except OSError, e:
+		except OSError as e:
 			logSys.error("Unable to remove PID file: %s" % e)
 		logSys.info("Exiting Fail2ban")
 	
@ -237,13 +237,11 @@ class Server:
 	
 	def setLogEncoding(self, name, encoding):
 		filter_ = self.__jails[name].filter
-		if isinstance(filter_, FileFilter):
-			filter_.setLogEncoding(encoding)
+		filter_.setLogEncoding(encoding)
 	
 	def getLogEncoding(self, name):
 		filter_ = self.__jails[name].filter
-		if isinstance(filter_, FileFilter):
-			return filter_.getLogEncoding()
+		return filter_.getLogEncoding()
 	
 	def setFindTime(self, name, value):
 		self.__jails[name].filter.setFindTime(value)
@ -543,7 +541,7 @@ class Server:
 			# the child gets a new PID, making it impossible for its PID to equal its
 			# PGID.
 			pid = os.fork()
-		except OSError, e:
+		except OSError as e:
 			return((e.errno, e.strerror))	 # ERROR (return a tuple)
 		
 		if pid == 0:	   # The first child.
@ -564,7 +562,7 @@ class Server:
 				# fork guarantees that the child is no longer a session leader, thus
 				# preventing the daemon from ever acquiring a controlling terminal.
 				pid = os.fork()		# Fork a second child.
-			except OSError, e:
+			except OSError as e:
 				return((e.errno, e.strerror))  # ERROR (return a tuple)
 		
 			if (pid == 0):	  # The second child.
--- a/fail2ban/server/transmitter.py
+++ b/fail2ban/server/transmitter.py
@ -56,7 +56,7 @@ class Transmitter:
 		try:
 			ret = self.__commandHandler(command)
 			ack = 0, ret
-		except Exception, e:
+		except Exception as e:
 			logSys.warning("Command %r has failed. Received %r"
 						% (command, e))
 			ack = 1, e
--- a/fail2ban/setup.py
+++ b/fail2ban/setup.py
@ -0,0 +1,42 @@
+# emacs: -*- mode: python; py-indent-offset: 4; indent-tabs-mode: t -*-
+# vi: set ft=python sts=4 ts=4 sw=4 noet :
+
+# This file is part of Fail2Ban.
+#
+# Fail2Ban is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# Fail2Ban is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Fail2Ban; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+
+__author__ = "Serg G. Brester"
+__license__ = "GPL"
+
+import os
+import sys
+
+
+def updatePyExec(bindir, executable=None):
+	"""Update fail2ban-python link to current python version (where f2b-modules located/installed)
+	"""
+	bindir = os.path.realpath(bindir)
+	if executable is None:
+		executable = sys.executable
+	pypath = os.path.join(bindir, 'fail2ban-python')
+	# if not exists or point to another version - update link:
+	isfile = os.path.isfile(pypath)
+	if not isfile or os.path.realpath(pypath) != os.path.realpath(executable):
+		if isfile:
+			os.unlink(pypath)
+		os.symlink(executable, pypath)
+	# extend current environment path (e.g. if fail2ban not yet installed):
+	if bindir not in os.environ["PATH"].split(os.pathsep):
+		os.environ["PATH"] = os.environ["PATH"] + os.pathsep + bindir;
--- a/fail2ban/tests/action_d/test_badips.py
+++ b/fail2ban/tests/action_d/test_badips.py
@ -49,7 +49,7 @@ if sys.version_info >= (2,7):

 		def testCategory(self):
 			categories = self.action.getCategories()
-			self.assertTrue("ssh" in categories)
+			self.assertIn("ssh", categories)
 			self.assertTrue(len(categories) >= 10)

 			self.assertRaises(
--- a/fail2ban/tests/action_d/test_smtp.py
+++ b/fail2ban/tests/action_d/test_smtp.py
@ -101,21 +101,21 @@ class SMTPActionTest(unittest.TestCase):
 		self.assertEqual(self.smtpd.rcpttos, ["root"])
 		subject = "Subject: [Fail2Ban] %s: banned %s" % (
 			self.jail.name, aInfo['ip'])
-		self.assertTrue(subject in self.smtpd.data.replace("\n", ""))
+		self.assertIn(subject, self.smtpd.data.replace("\n", ""))
 		self.assertTrue(
 			"%i attempts" % aInfo['failures'] in self.smtpd.data)

 		self.action.matches = "matches"
 		self.action.ban(aInfo)
-		self.assertTrue(aInfo['matches'] in self.smtpd.data)
+		self.assertIn(aInfo['matches'], self.smtpd.data)

 		self.action.matches = "ipjailmatches"
 		self.action.ban(aInfo)
-		self.assertTrue(aInfo['ipjailmatches'] in self.smtpd.data)
+		self.assertIn(aInfo['ipjailmatches'], self.smtpd.data)

 		self.action.matches = "ipmatches"
 		self.action.ban(aInfo)
-		self.assertTrue(aInfo['ipmatches'] in self.smtpd.data)
+		self.assertIn(aInfo['ipmatches'], self.smtpd.data)

 	def testOptions(self):
 		self.action.start()
--- a/fail2ban/tests/actionstestcase.py
+++ b/fail2ban/tests/actionstestcase.py
@ -65,12 +65,12 @@ class ExecuteActions(LogCaptureTestCase):
 	def testActionsManipulation(self):
 		self.__actions.add('test')
 		self.assertTrue(self.__actions['test'])
-		self.assertTrue('test' in self.__actions)
-		self.assertFalse('nonexistant action' in self.__actions)
+		self.assertIn('test', self.__actions)
+		self.assertNotIn('nonexistant action', self.__actions)
 		self.__actions.add('test1')
 		del self.__actions['test']
 		del self.__actions['test1']
-		self.assertFalse('test' in self.__actions)
+		self.assertNotIn('test', self.__actions)
 		self.assertEqual(len(self.__actions), 0)

 		self.__actions.setBanTime(127)
--- a/fail2ban/tests/clientreadertestcase.py
+++ b/fail2ban/tests/clientreadertestcase.py
@ -36,7 +36,7 @@ from ..client.jailsreader import JailsReader
 from ..client.actionreader import ActionReader
 from ..client.configurator import Configurator
 from ..version import version
-from .utils import LogCaptureTestCase
+from .utils import LogCaptureTestCase, with_tmpdir

 TEST_FILES_DIR = os.path.join(os.path.dirname(__file__), "files")

@ -94,9 +94,8 @@ option = %s
 		if not os.access(f, os.R_OK):
 			self.assertFalse(self.c.read('d'))	# should not be readable BUT present
 		else:
-			# SkipTest introduced only in 2.7 thus can't yet use generally
-			# raise unittest.SkipTest("Skipping on %s -- access rights are not enforced" % platform)
-			pass
+			import platform
+			raise unittest.SkipTest("Skipping on %s -- access rights are not enforced" % platform.platform())

 	def testOptionalDotDDir(self):
 		self.assertFalse(self.c.read('c'))	# nothing is there yet
@ -281,8 +280,8 @@ class JailReaderTest(LogCaptureTestCase):
 		self.assertEqual(eval(act[2][5]).get('agent', '<wrong>'), useragent)
 		self.assertEqual(act[3], ['set', 'blocklisttest', 'action', 'mynetwatchman', 'agent', useragent])

-	def testGlob(self):
-		d = tempfile.mkdtemp(prefix="f2b-temp")
+	@with_tmpdir
+	def testGlob(self, d):
 		# Generate few files
 		# regular file
 		f1 = os.path.join(d, 'f1')
@ -297,9 +296,6 @@ class JailReaderTest(LogCaptureTestCase):
 		self.assertEqual(JailReader._glob(f2), [])
 		self.assertLogged('File %s is a dangling link, thus cannot be monitored' % f2)
 		self.assertEqual(JailReader._glob(os.path.join(d, 'nonexisting')), [])
-		os.remove(f1)
-		os.remove(f2)
-		os.rmdir(d)

 		
 class FilterReaderTest(unittest.TestCase):
@ -410,7 +406,7 @@ class FilterReaderTest(unittest.TestCase):
 			# from testcase01
 			filterReader.get('Definition', 'failregex')
 			filterReader.get('Definition', 'ignoreregex')
-		except Exception, e: # pragma: no cover - failed if reachable
+		except Exception as e: # pragma: no cover - failed if reachable
 			self.fail('unexpected options after readexplicit: %s' % (e))


@ -433,10 +429,10 @@ class JailsReaderTestCache(LogCaptureTestCase):
 				cnt += 1
 		return cnt

-	def testTestJailConfCache(self):
+	@with_tmpdir
+	def testTestJailConfCache(self, basedir):
 		saved_ll = configparserinc.logLevel
 		configparserinc.logLevel = logging.DEBUG
-		basedir = tempfile.mkdtemp("fail2ban_conf")
 		try:
 			shutil.rmtree(basedir)
 			shutil.copytree(CONFIG_DIR, basedir)
@ -468,7 +464,6 @@ class JailsReaderTestCache(LogCaptureTestCase):
 			cnt = self._getLoggedReadCount(r'action\.d/iptables-common\.conf')
 			self.assertTrue(cnt == 1, "Unexpected count by reading of action files, cnt = %s" % cnt)
 		finally:
-			shutil.rmtree(basedir)
 			configparserinc.logLevel = saved_ll


@ -525,12 +520,12 @@ class JailsReaderTest(LogCaptureTestCase):
 				self.assertTrue(actionReader.read())
 				actionReader.getOptions({})	  # populate _opts
 				if not actionName.endswith('-common'):
-					self.assertTrue('Definition' in actionReader.sections(),
+					self.assertIn('Definition', actionReader.sections(),
 						msg="Action file %r is lacking [Definition] section" % actionConfig)
 					# all must have some actionban defined
 					self.assertTrue(actionReader._opts.get('actionban', '').strip(),
 						msg="Action file %r is lacking actionban" % actionConfig)
-				self.assertTrue('Init' in actionReader.sections(),
+				self.assertIn('Init', actionReader.sections(),
 						msg="Action file %r is lacking [Init] section" % actionConfig)

 		def testReadStockJailConf(self):
@ -582,7 +577,7 @@ class JailsReaderTest(LogCaptureTestCase):
 					self.assertTrue(len(actName))
 					self.assertTrue(isinstance(actOpt, dict))
 					if actName == 'iptables-multiport':
-						self.assertTrue('port' in actOpt)
+						self.assertIn('port', actOpt)

 					actionReader = ActionReader(
 						actName, jail, {}, basedir=CONFIG_DIR)
@ -632,11 +627,13 @@ class JailsReaderTest(LogCaptureTestCase):

 			# and we know even some of them by heart
 			for j in ['sshd', 'recidive']:
-				# by default we have 'auto' backend ATM
-				self.assertTrue(['add', j, 'auto'] in comm_commands)
+				# by default we have 'auto' backend ATM, but some distributions can overwrite it, 
+				# (e.g. fedora default is 'systemd') therefore let check it without backend...
+				self.assertIn(['add', j], 
+					(cmd[:2] for cmd in comm_commands if len(cmd) == 3 and cmd[0] == 'add'))
 				# and warn on useDNS
-				self.assertTrue(['set', j, 'usedns', 'warn'] in comm_commands)
-				self.assertTrue(['start', j] in comm_commands)
+				self.assertIn(['set', j, 'usedns', 'warn'], comm_commands)
+				self.assertIn(['start', j], comm_commands)

 			# last commands should be the 'start' commands
 			self.assertEqual(comm_commands[-1][0], 'start')
@ -655,7 +652,7 @@ class JailsReaderTest(LogCaptureTestCase):
 					action_name = action.getName()
 					if '<blocktype>' in str(commands):
 						# Verify that it is among cInfo
-						self.assertTrue('blocktype' in action._initOpts)
+						self.assertIn('blocktype', action._initOpts)
 						# Verify that we have a call to set it up
 						blocktype_present = False
 						target_command = ['set', jail_name, 'action', action_name, 'blocktype']
@ -716,8 +713,8 @@ class JailsReaderTest(LogCaptureTestCase):
 			self.assertEqual(configurator._Configurator__jails.getBaseDir(), '/tmp')
 			self.assertEqual(configurator.getBaseDir(), CONFIG_DIR)

-	def testMultipleSameAction(self):
-		basedir = tempfile.mkdtemp("fail2ban_conf")
+	@with_tmpdir
+	def testMultipleSameAction(self, basedir):
 		os.mkdir(os.path.join(basedir, "filter.d"))
 		os.mkdir(os.path.join(basedir, "action.d"))
 		open(os.path.join(basedir, "action.d", "testaction1.conf"), 'w').close()
@ -746,4 +743,33 @@ filter = testfilter1
 		# Python actions should not be passed `actname`
 		self.assertEqual(add_actions[-1][-1], "{}")

-		shutil.rmtree(basedir)
+	def testLogPathFileFilterBackend(self):
+		self.assertRaisesRegexp(ValueError, r"Have not found any log file for .* jail", 
+			self._testLogPath, backend='polling')
+
+	def testLogPathSystemdBackend(self):
+		try: # pragma: systemd no cover
+			from ..server.filtersystemd import FilterSystemd
+		except Exception, e: # pragma: no cover
+			raise unittest.SkipTest("systemd python interface not available")
+		self._testLogPath(backend='systemd')
+		self._testLogPath(backend='systemd[journalflags=2]')
+	
+	@with_tmpdir
+	def _testLogPath(self, basedir, backend):
+		jailfd = open(os.path.join(basedir, "jail.conf"), 'w')
+		jailfd.write("""
+[testjail1]
+enabled = true
+backend = %s
+logpath = %s/not/exist.log
+          /this/path/should/not/exist.log
+action = 
+filter = 
+failregex = test <HOST>
+""" % (backend, basedir))
+		jailfd.close()
+		jails = JailsReader(basedir=basedir)
+		self.assertTrue(jails.read())
+		self.assertTrue(jails.getOptions())
+		jails.convert()
--- a/fail2ban/tests/databasetestcase.py
+++ b/fail2ban/tests/databasetestcase.py
@ -48,12 +48,10 @@ class DatabaseTest(LogCaptureTestCase):
 	def setUp(self):
 		"""Call before every test case."""
 		super(DatabaseTest, self).setUp()
-		if Fail2BanDb is None and sys.version_info >= (2,7): # pragma: no cover
+		if Fail2BanDb is None: # pragma: no cover
 			raise unittest.SkipTest(
 				"Unable to import fail2ban database module as sqlite is not "
 				"available.")
-		elif Fail2BanDb is None:
-			return
 		_, self.dbFilename = tempfile.mkstemp(".db", "fail2ban_")
 		self.db = Fail2BanDb(self.dbFilename)

@ -123,7 +121,7 @@ class DatabaseTest(LogCaptureTestCase):

 		self.db.addLog(self.jail, self.fileContainer)

-		self.assertTrue(filename in self.db.getLogPaths(self.jail))
+		self.assertIn(filename, self.db.getLogPaths(self.jail))
 		os.remove(filename)

 	def testUpdateLog(self):
@ -318,6 +316,25 @@ class DatabaseTest(LogCaptureTestCase):
 		actions._Actions__checkBan()
 		self.assertLogged("ban ainfo %s, %s, %s, %s" % (True, True, True, True))

+	def testDelAndAddJail(self):
+		self.testAddJail() # Add jail
+		# Delete jail (just disabled it):
+		self.db.delJail(self.jail)
+		jails = self.db.getJailNames()
+		self.assertIn(len(jails) == 1 and self.jail.name, jails)
+		jails = self.db.getJailNames(enabled=False)
+		self.assertIn(len(jails) == 1 and self.jail.name, jails)
+		jails = self.db.getJailNames(enabled=True)
+		self.assertTrue(len(jails) == 0)
+		# Add it again - should just enable it:
+		self.db.addJail(self.jail)
+		jails = self.db.getJailNames()
+		self.assertIn(len(jails) == 1 and self.jail.name, jails)
+		jails = self.db.getJailNames(enabled=True)
+		self.assertIn(len(jails) == 1 and self.jail.name, jails)
+		jails = self.db.getJailNames(enabled=False)
+		self.assertTrue(len(jails) == 0)
+
 	def testPurge(self):
 		if Fail2BanDb is None: # pragma: no cover
 			return
--- a/fail2ban/tests/fail2banregextestcase.py
+++ b/fail2ban/tests/fail2banregextestcase.py
@ -39,7 +39,7 @@ except ImportError:

 from ..client import fail2banregex
 from ..client.fail2banregex import Fail2banRegex, get_opt_parser, output
-from .utils import LogCaptureTestCase, logSys
+from .utils import setUpMyTime, tearDownMyTime, LogCaptureTestCase, logSys
 from .utils import CONFIG_DIR


@ -70,10 +70,12 @@ class Fail2banRegexTest(LogCaptureTestCase):
 	def setUp(self):
 		"""Call before every test case."""
 		LogCaptureTestCase.setUp(self)
+		setUpMyTime()

 	def tearDown(self):
 		"""Call after every test case."""
 		LogCaptureTestCase.tearDown(self)
+		tearDownMyTime()

 	def testWrongRE(self):
 		(opts, args, fail2banRegex) = _Fail2banRegex(
@ -159,8 +161,8 @@ class Fail2banRegexTest(LogCaptureTestCase):
 		self.assertTrue(fail2banRegex.start(opts, args))
 		self.assertLogged('Lines: 13 lines, 0 ignored, 5 matched, 8 missed')

-		self.assertLogged('141.3.81.106  Fri Aug 14 11:53:59 2015')
-		self.assertLogged('141.3.81.106  Fri Aug 14 11:54:59 2015')
+		self.assertLogged('141.3.81.106  Sun Aug 14 11:53:59 2005')
+		self.assertLogged('141.3.81.106  Sun Aug 14 11:54:59 2005')

 	def testWronChar(self):
 		(opts, args, fail2banRegex) = _Fail2banRegex(
@ -169,9 +171,8 @@ class Fail2banRegexTest(LogCaptureTestCase):
 		self.assertTrue(fail2banRegex.start(opts, args))
 		self.assertLogged('Lines: 4 lines, 0 ignored, 2 matched, 2 missed')

-		self.assertLogged('Error decoding line');
-		self.assertLogged('Continuing to process line ignoring invalid characters:', '2015-01-14 20:00:58 user ');
-		self.assertLogged('Continuing to process line ignoring invalid characters:', '2015-01-14 20:00:59 user ');
+		self.assertLogged('Error decoding line')
+		self.assertLogged('Continuing to process line ignoring invalid characters:')

 		self.assertLogged('Nov  8 00:16:12 main sshd[32548]: input_userauth_request: invalid user llinco')
 		self.assertLogged('Nov  8 00:16:12 main sshd[32547]: pam_succeed_if(sshd:auth): error retrieving information about user llinco')
--- a/fail2ban/tests/files/config/apache-auth/digest.py
+++ b/fail2ban/tests/files/config/apache-auth/digest.py
@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env fail2ban-python
 import requests

 try:
--- a/fail2ban/tests/files/ignorecommand.py
+++ b/fail2ban/tests/files/ignorecommand.py
@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env fail2ban-python
 import sys
 if sys.argv[1] == "10.0.0.1":
 	exit(0)
--- a/fail2ban/tests/files/logs/apache-modsecurity
+++ b/fail2ban/tests/files/logs/apache-modsecurity
@ -1,5 +1,5 @@
 # failJSON: { "time": "2013-12-23T13:12:31", "match": true , "host": "173.255.225.101" }
 [Mon Dec 23 13:12:31 2013] [error] [client 173.255.225.101] ModSecurity:  [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_21_protocol_anomalies.conf"] [line "47"] [id "960015"] [rev "1"] [msg "Request Missing an Accept Header"] [severity "NOTICE"] [ver "OWASP_CRS/2.2.8"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT"] [tag "WASCTC/WASC-21"][tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] Access denied with code 403 (phase 2). Operator EQ matched 0 at REQUEST_HEADERS. [hostname "www.mysite.net"] [uri "/"] [unique_id "Urf@f12qgHIAACrFOlgAAABA"]

-# failJSON: { "time": "2013-12-28T09:18:05", "match": true , "host": "32.65.254.69" }
-[Sat Dec 28 09:18:05 2013] [error] [client 32.65.254.69] ModSecurity:  [file "/etc/httpd/modsecurity.d/10_asl_rules.conf"] [line "635"] [id "340069"] [rev "4"] [msg "Atomicorp.com UNSUPPORTED DELAYED Rules: Web vulnerability scanner"] [severity "CRITICAL"] Access denied with code 403 (phase 2). Pattern match "(?:nessus(?:_is_probing_you_|test)|^/w00tw00t\\\\.at\\\\.)" at REQUEST_URI. [hostname "192.81.249.191"] [uri "/w00tw00t.at.blackhats.romanian.anti-sec:)"] [unique_id "4Q6RdsBR@b4AAA65LRUAAAAA"] 
+# failJSON: { "time": "2013-12-28T09:18:05", "match": true , "host": "32.65.254.69", "desc": "additional entry (and exact one space)" }
+[Sat Dec 28 09:18:05 2013] [error] [client 32.65.254.69] ModSecurity: [file "/etc/httpd/modsecurity.d/10_asl_rules.conf"] [line "635"] [id "340069"] [rev "4"] [msg "Atomicorp.com UNSUPPORTED DELAYED Rules: Web vulnerability scanner"] [severity "CRITICAL"] Access denied with code 403 (phase 2). Pattern match "(?:nessus(?:_is_probing_you_|test)|^/w00tw00t\\\\.at\\\\.)" at REQUEST_URI. [hostname "192.81.249.191"] [uri "/w00tw00t.at.blackhats.romanian.anti-sec:)"] [unique_id "4Q6RdsBR@b4AAA65LRUAAAAA"] 
--- a/fail2ban/tests/files/logs/assp
+++ b/fail2ban/tests/files/logs/assp
@ -22,4 +22,23 @@ Apr-27-13 02:25:10 [SSL-out] 217.194.197.97 max sender authentication errors (5)
 Apr-27-13 02:25:10 [SSL-out] 217.194.197.97 max sender authentication errors (5) exceeded -- dropping connection - after reply: 535 5.7.8 Error: authentication failed: UGFzc3dvcmQ6;
 # failJSON: { "time": "2013-04-27T02:25:11", "match": true , "host": "217.194.197.97" }
 Apr-27-13 02:25:11 [SSL-out] 217.194.197.97 max sender authentication errors (5) exceeded -- dropping connection - after reply: 535 5.7.8 Error: authentication failed: UGFzc3dvcmQ6;
-
+# failJSON: { "time": "2016-07-29T16:49:52", "match": true , "host": "0.0.0.0" }
+Jul-29-16 16:49:52 m1-25391-06124 [Worker_1] [TLS-out] [RelayAttempt] 0.0.0.0 <user@example.com> to: user@example.org relay attempt blocked for: someone@example.org
+# failJSON: { "time": "2016-07-30T17:07:25", "match": true , "host": "0.0.0.0" }
+Jul-30-16 17:07:25 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: UGFzc3dvcmQ6
+# failJSON: { "time": "2016-07-30T17:11:05", "match": true , "host": "0.0.0.0" }
+Jul-30-16 17:11:05 m1-13060-05386 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: UGFzc3dvcmQ6
+# failJSON: { "time": "2016-07-31T06:45:59", "match": true , "host": "0.0.0.0" }
+Jul-31-16 06:45:59 [Worker_1] [TLS-in] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: 
+# failJSON: { "time": "2016-01-05T08:38:49", "match": true , "host": "0.0.0.0" }
+Jan-05-16 08:38:49 m1-01129-09140 [Worker_1] [TLS-in] [TLS-out] [RelayAttempt] 0.0.0.0 <user@example.com> relay attempt blocked for (parsing): <user2@example>
+# failJSON: { "time": "2016-06-12T16:43:37", "match": true , "host": "0.0.0.0" }
+Jun-12-16 16:43:37 m1-64217-12013 [Worker_1] [TLS-in] [TLS-out] [RelayAttempt] 0.0.0.0 <user@example.com> to: user2@example.com relay attempt blocked for (parsing): <a.notheruser69@example.c>
+# failJSON: { "time": "2016-01-22T22:25:51", "match": true , "host": "0.0.0.0" }
+Jan-22-16 22:25:51 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: Invalid authentication mechanism
+# failJSON: { "time": "2016-03-19T13:42:20", "match": true , "host": "0.0.0.0" }
+Mar-19-16 13:42:20 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: Invalid base64 data in continued response
+# failJSON: { "time": "2016-07-18T16:54:21", "match": true , "host": "0.0.0.0" }
+Jul-18-16 16:54:21 [Worker_2] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: Connection lost to authentication server
+# failJSON: { "time": "2016-07-18T17:14:23", "match": true , "host": "0.0.0.0" }
+Jul-18-16 17:14:23 m1-76453-02949 [Worker_1] [TLS-out] 0.0.0.0 [SMTP Error] 535 5.7.8 Error: authentication failed: Connection lost to authentication server
--- a/fail2ban/tests/files/logs/asterisk
+++ b/fail2ban/tests/files/logs/asterisk
@ -43,6 +43,8 @@

 # failJSON: { "time": "2004-11-04T18:30:40", "match": true , "host": "192.168.200.100" }
 Nov 4 18:30:40 localhost asterisk[32229]: NOTICE[32257]: chan_sip.c:23417 in handle_request_register: Registration from '<sip:301@example.com>' failed for '192.168.200.100:36998' - Wrong password
+# failJSON: { "time": "2016-08-19T11:11:26", "match": true , "host": "192.0.2.1", "desc": "Another log_prefix used (` in` should be optional)" }
+[2016-08-19 11:11:26] NOTICE[12931]: chan_sip.c:28468 handle_request_register: Registration from 'sip:bob@192.0.2.1' failed for '192.0.2.1:42406' - Wrong password

 # failed authentication attempt on INVITE using PJSIP
 # failJSON: { "time": "2015-05-24T08:42:16", "match": true, "host": "10.250.251.252" }
--- a/fail2ban/tests/files/logs/dovecot
+++ b/fail2ban/tests/files/logs/dovecot
@ -73,3 +73,8 @@ Jul 02 13:49:32 hostname dovecot[442]: pop3-login: Disconnected (no auth attempt

 # failJSON: { "time": "2005-03-23T06:10:52", "match": true , "host": "52.37.139.121" }
 Mar 23 06:10:52 auth: Info: ldap(dog,52.37.139.121,): invalid credentials
+
+# failJSON: { "time": "2005-07-26T11:11:21", "match": true , "host": "192.0.2.1" }
+Jul 26 11:11:21 hostname dovecot: imap-login: Disconnected: Too many invalid commands (tried to use disallowed plaintext auth): user=<test>, rip=192.0.2.1, lip=192.168.1.1, session=<S5dIdTFCDKUWWMbU>
+# failJSON: { "time": "2005-07-26T11:12:19", "match": true , "host": "192.0.2.2" }
+Jul 26 11:12:19 hostname dovecot: imap-login: Disconnected: Too many invalid commands (auth failed, 1 attempts in 17 secs): user=<test>, method=PLAIN, rip=192.0.2.2, lip=192.168.1.1, TLS, session=<g3ZKeDECFqlWWMbU>
--- a/fail2ban/tests/files/logs/mongodb-auth
+++ b/fail2ban/tests/files/logs/mongodb-auth
@ -0,0 +1,30 @@
+# failJSON: { "match": false }
+2016-11-20T00:04:00.110+0100 [conn1] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed UserNotFound Could not find user root@admin
+# failJSON: { "time": "2016-11-20T00:04:00", "match": true , "host": "192.0.2.35" }
+2016-11-20T00:04:00.111+0100 [conn1] end connection 192.0.2.35:53276 (0 connections now open)
+
+# failJSON: { "match": false }
+2016-11-20T00:24:00.110+0100 [conn5] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed UserNotFound Could not find user root@admin
+# failJSON: { "time": "2016-11-20T00:24:00", "match": true , "host": "192.0.2.171" }
+2016-11-20T00:24:00.111+0100 [conn5] end connection 192.0.2.171:53276 (0 connections now open)
+
+# failJSON: { "match": false }
+2016-11-20T00:24:00.110+0100 [conn334] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed key mismatch
+# failJSON: { "time": "2016-11-20T00:24:00", "match": true , "host": "192.0.2.176" }
+2016-11-20T00:24:00.111+0100 [conn334] end connection 192.0.2.176:53276 (0 connections now open)
+
+# failJSON: { "match": false }
+2016-11-20T00:24:00.110+0100 [conn56] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed key mismatch
+# failJSON: { "time": "2016-11-20T00:24:00", "match": true , "host": "192.0.2.1" }
+2016-11-20T00:24:00.111+0100 [conn56] end connection 192.0.2.1:53276 (0 connections now open)
+
+# failJSON: { "match": false }
+2016-11-20T12:54:02.370+0100 [initandlisten] connection accepted from 127.0.0.1:58774 #2261 (1 connection now open)
+# failJSON: { "match": false }
+2016-11-20T12:54:02.370+0100 [conn2261] end connection 127.0.0.1:58774 (0 connections now open)
+
+# failJSON: { "match": false }
+2016-11-20T13:07:49.781+0100 [conn2271]  authenticate db: admin { authenticate: 1, nonce: "xxx", user: "root", key: "xxx" }
+# failJSON: { "time": "2016-11-20T13:07:49", "match": false , "host": "192.0.2.178" }
+2016-11-20T13:07:49.834+0100 [conn2271] end connection 192.0.2.178:60268 (3 connections now open)
+
--- a/fail2ban/tests/files/logs/postfix-sasl
+++ b/fail2ban/tests/files/logs/postfix-sasl
@ -26,3 +26,7 @@ Jan 29 08:11:45 mail postfix-incoming/smtpd[10752]: warning: unknown[1.1.1.1]: S

 # failJSON: { "time": "2005-04-12T02:24:11", "match": true , "host": "62.138.2.143" }
 Apr 12 02:24:11 xxx postfix/smtps/smtpd[42]: warning: astra4139.startdedicated.de[62.138.2.143]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
+
+# failJSON: { "time": "2005-08-03T15:30:49", "match": true , "host": "98.191.84.74" }
+Aug 3 15:30:49 ksusha postfix/smtpd[17041]: warning: mail.foldsandwalker.com[98.191.84.74]: SASL Plain authentication failed:
+
--- a/fail2ban/tests/files/logs/sendmail-reject
+++ b/fail2ban/tests/files/logs/sendmail-reject
@ -40,6 +40,8 @@ Feb 19 18:01:50 batman sm-mta[78152]: ruleset=check_relay, arg1=[196.213.73.146]

 # failJSON: { "time": "2005-02-27T10:53:06", "match": true , "host": "209.15.212.253" }
 Feb 27 10:53:06 batman sm-mta[44307]: s1R9r60D044307: rejecting commands from  [209.15.212.253] due to pre-greeting traffic after 0 seconds
+# failJSON: { "time": "2005-02-27T10:53:07", "match": true , "host": "1.2.3.4" }
+Feb 27 10:53:07 strange sm-mta[18001]: u9A0GtpL018001: rejecting commands from example.com [1.2.3.4] due to pre-greeting traffic after 6 seconds

 # failJSON: { "time": "2005-02-27T15:44:18", "match": true , "host": "41.204.78.137" }
 Feb 27 15:44:18 batman sm-mta[87838]: s1REiHdq087838: ruleset=check_rcpt, arg1=<gert-jan@t-online.ch>, relay=[41.204.78.137], reject=550 5.7.1 <gert-jan@t-online.ch>... Relaying denied. IP name lookup failed [41.204.78.137]
--- a/fail2ban/tests/files/logs/sshd
+++ b/fail2ban/tests/files/logs/sshd
@ -17,8 +17,10 @@ Jan  5 01:31:41 www sshd[1643]: ROOT LOGIN REFUSED FROM 1.2.3.4
 Jan  5 01:31:41 www sshd[1643]: ROOT LOGIN REFUSED FROM ::ffff:1.2.3.4

 #4
-# failJSON: { "time": "2005-07-20T14:42:11", "match": true , "host": "211.114.51.213" }
-Jul 20 14:42:11 localhost sshd[22708]: Invalid user ftp from 211.114.51.213
+# failJSON: { "time": "2005-07-20T14:42:11", "match": true , "host": "192.0.2.1", "desc": "Invalid user" }
+Jul 20 14:42:11 localhost sshd[22708]: Invalid user ftp from 192.0.2.1
+# failJSON: { "time": "2005-07-20T14:42:12", "match": true , "host": "192.0.2.2", "desc": "Invalid user with port" }
+Jul 20 14:42:12 localhost sshd[22708]: Invalid user ftp from 192.0.2.2 port 37220

 #5 new filter introduced after looking at 44087D8C.9090407@bluewin.ch
 # yoh: added ':' after [sshd] since the case without is not really common any more
@ -117,7 +119,13 @@ Sep 29 17:15:02 spaceman sshd[12946]: Failed password for user from 127.0.0.1 po

 # failJSON: { "time": "2004-11-11T08:04:51", "match": true , "host": "127.0.0.1", "desc": "Injecting on username ssh 'from 10.10.1.1'@localhost" }
 Nov 11 08:04:51 redbamboo sshd[2737]: Failed password for invalid user from 10.10.1.1 from 127.0.0.1 port 58946 ssh2
+# failJSON: { "time": "2004-11-11T08:04:52", "match": true , "host": "127.0.0.1", "desc": "More complex injecting on username ssh 'test from 10.10.1.2 port 55555 ssh2'@localhost" }
+Nov 11 08:04:52 redbamboo sshd[2737]: Failed password for invalid user test from 10.10.1.2 port 55555 ssh2 from 127.0.0.1 port 58946 ssh2
+# failJSON: { "time": "2004-11-11T08:04:52", "match": true , "host": "127.0.0.1", "desc": "More complex injecting on auth-info ssh test@localhost, auth-info: ' from 10.10.1.2 port 55555 ssh2'" }
+Nov 11 08:04:52 redbamboo sshd[2737]: Failed password for invalid user test from 127.0.0.1 port 58946 ssh2: from 10.10.1.2 port 55555 ssh2

+# failJSON: { "time": "2005-07-05T18:22:44", "match": true , "host": "127.0.0.1", "desc": "Failed publickey for ..." }
+Jul 05 18:22:44 mercury sshd[4669]: Failed publickey for graysky from 127.0.0.1 port 37954 ssh2: RSA SHA256:v3dpapGleDaUKf$4V1vKyR9ZyUgjaJAmoCTcb2PLljI

 # failJSON: { "match": false }
 Nov 23 21:50:19 sshd[8148]: Disconnecting: Too many authentication failures for root [preauth]
@ -161,4 +169,3 @@ Apr 27 13:02:04 host sshd[29116]: Received disconnect from 1.2.3.4: 11: Normal S
 # Match sshd auth errors on OpenSUSE systems
 # failJSON: { "time": "2015-04-16T20:02:50", "match": true , "host": "222.186.21.217", "desc": "Authentication for user failed" }
 2015-04-16T18:02:50.321974+00:00 host sshd[2716]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.186.21.217  user=root
-
--- a/fail2ban/tests/files/logs/vsftpd
+++ b/fail2ban/tests/files/logs/vsftpd
@ -12,3 +12,6 @@ Fri Jan 19 12:20:33 2007 [pid 27202] [anonymous] FAIL LOGIN: Client "64.106.46.9

 # failJSON: { "time": "2004-10-23T21:15:42", "match": true , "host": "58.254.172.161" }
 Oct 23 21:15:42 vps vsftpd: pam_unix(vsftpd:auth): authentication failure; logname= uid=0 euid=0 tty=ftp ruser=test rhost=58.254.172.161
+
+# failJSON: { "time": "2016-09-08T00:39:49", "match": true , "host": "192.0.2.1" }
+Thu Sep  8 00:39:49 2016 [pid 15019] [guest] FAIL LOGIN: Client "::ffff:192.0.2.1", "User is not in the allow user list."
--- a/fail2ban/tests/files/testcase-journal.log
+++ b/fail2ban/tests/files/testcase-journal.log
@ -13,7 +13,7 @@ error: PAM: Authentication failure for kevin from 193.168.0.128
 error: PAM: Authentication failure for kevin from 193.168.0.128
 error: PAM: Authentication failure for kevin from 193.168.0.128
 error: PAM: Authentication failure for kevin from 193.168.0.128
-error: PAM: Authentication failure for kevin from 87.142.124.10
-error: PAM: Authentication failure for kevin from 87.142.124.10
-error: PAM: Authentication failure for kevin from 87.142.124.10
-error: PAM: Authentication failure for kevin from 87.142.124.10
+error: PAM: Authentication failure for göran from 87.142.124.10
+error: PAM: Authentication failure for göran from 87.142.124.10
+error: PAM: Authentication failure for göran from 87.142.124.10
+error: PAM: Authentication failure for göran from 87.142.124.10
--- a/fail2ban/tests/filtertestcase.py
+++ b/fail2ban/tests/filtertestcase.py
@ -38,7 +38,7 @@ except ImportError:

 from ..server.jail import Jail
 from ..server.filterpoll import FilterPoll
-from ..server.filter import Filter, FileFilter, DNSUtils
+from ..server.filter import Filter, FileFilter, FileContainer, locale, DNSUtils
 from ..server.failmanager import FailManagerEmpty
 from ..server.mytime import MyTime
 from .utils import setUpMyTime, tearDownMyTime, mtimesleep, LogCaptureTestCase
@ -166,6 +166,10 @@ def _copy_lines_between_files(in_, fout, n=None, skip=0, mode='a', terminal_line
 	return fout


+TEST_JOURNAL_FIELDS = {
+  "SYSLOG_IDENTIFIER": "fail2ban-testcases",
+	"PRIORITY": "7",
+}
 def _copy_lines_to_journal(in_, fields={},n=None, skip=0, terminal_line=""): # pragma: systemd no cover
 	"""Copy lines from one file to systemd journal

@ -176,9 +180,7 @@ def _copy_lines_to_journal(in_, fields={},n=None, skip=0, terminal_line=""): # p
 	else:
 		fin = in_
 	# Required for filtering
-	fields.update({"SYSLOG_IDENTIFIER": "fail2ban-testcases",
-					"PRIORITY": "7",
-					})
+	fields.update(TEST_JOURNAL_FIELDS)
 	# Skip
 	for i in xrange(skip):
 		fin.readline()
@ -228,6 +230,19 @@ class BasicFilter(unittest.TestCase):
 			1)
 		)

+	def testWrongCharInTupleLine(self):
+		## line tuple has different types (ascii after ascii / unicode):
+		for a1 in ('', u'', b''):
+			for a2 in ('2016-09-05T20:18:56', u'2016-09-05T20:18:56', b'2016-09-05T20:18:56'):
+				for a3 in (
+					'Fail for "g\xc3\xb6ran" from 192.0.2.1', 
+					u'Fail for "g\xc3\xb6ran" from 192.0.2.1',
+					b'Fail for "g\xc3\xb6ran" from 192.0.2.1'
+				):
+					# join should work if all arguments have the same type:
+					enc = locale.getpreferredencoding()
+					"".join([Filter.uni_decode(v, enc) for v in (a1, a2, a3)])
+

 class IgnoreIP(LogCaptureTestCase):

@ -707,11 +722,16 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover
 			"""Call before every test case."""
 			self.test_file = os.path.join(TEST_FILES_DIR, "testcase-journal.log")
 			self.jail = DummyJail()
-			self.filter = Filter_(self.jail)
+			self.filter = None
 			# UUID used to ensure that only meeages generated
 			# as part of this test are picked up by the filter
 			self.test_uuid = str(uuid.uuid4())
 			self.name = "monitorjournalfailures-%s" % self.test_uuid
+			self.journal_fields = {
+				'TEST_FIELD': "1", 'TEST_UUID': self.test_uuid}
+
+		def _initFilter(self, **kwargs):
+			self.filter = Filter_(self.jail, **kwargs)
 			self.filter.addJournalMatch([
 				"SYSLOG_IDENTIFIER=fail2ban-testcases",
 				"TEST_FIELD=1",
@ -720,16 +740,16 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover
 				"SYSLOG_IDENTIFIER=fail2ban-testcases",
 				"TEST_FIELD=2",
 				"TEST_UUID=%s" % self.test_uuid])
-			self.journal_fields = {
-				'TEST_FIELD': "1", 'TEST_UUID': self.test_uuid}
-			self.filter.active = True
 			self.filter.addFailRegex("(?:(?:Authentication failure|Failed [-/\w+]+) for(?: [iI](?:llegal|nvalid) user)?|[Ii](?:llegal|nvalid) user|ROOT LOGIN REFUSED) .*(?: from|FROM) <HOST>")
-			self.filter.start()

 		def tearDown(self):
-			self.filter.stop()
-			self.filter.join()		  # wait for the thread to terminate
-			pass
+			if self.filter and self.filter.active:
+				self.filter.stop()
+				self.filter.join()		  # wait for the thread to terminate
+				pass
+
+		def testJournalFlagsArg(self):
+			self._initFilter(journalflags=2) # journal.RUNTIME_ONLY

 		def __str__(self):
 			return "MonitorJournalFailures%s(%s)" \
@ -761,6 +781,8 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover
 			self.assertEqual(attempts, test_attempts)

 		def test_grow_file(self):
+			self._initFilter()
+			self.filter.start()
 			self.assertRaises(FailManagerEmpty, self.filter.failManager.toBan)

 			# Now let's feed it with entries from the file
@ -790,6 +812,8 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover
 			self.assert_correct_ban("193.168.0.128", 3)

 		def test_delJournalMatch(self):
+			self._initFilter()
+			self.filter.start()
 			# Smoke test for removing of match

 			# basic full test
@ -819,6 +843,33 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover
 			# we should detect the failures
 			self.assertTrue(self.isFilled(6))

+		def test_WrongChar(self):
+			self._initFilter()
+			self.filter.start()
+			# Now let's feed it with entries from the file
+			_copy_lines_to_journal(
+				self.test_file, self.journal_fields, skip=15, n=4)
+			self.assertTrue(self.isFilled(10))
+			self.assert_correct_ban("87.142.124.10", 4)
+			# Add direct utf, unicode, blob:
+			for l in (
+		    "error: PAM: Authentication failure for \xe4\xf6\xfc\xdf from 192.0.2.1",
+		   u"error: PAM: Authentication failure for \xe4\xf6\xfc\xdf from 192.0.2.1",
+		   b"error: PAM: Authentication failure for \xe4\xf6\xfc\xdf from 192.0.2.1".decode('utf-8', 'replace'),
+		    "error: PAM: Authentication failure for \xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f from 192.0.2.2",
+		   u"error: PAM: Authentication failure for \xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f from 192.0.2.2",
+		   b"error: PAM: Authentication failure for \xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f from 192.0.2.2".decode('utf-8', 'replace')
+			):
+				fields = self.journal_fields
+				fields.update(TEST_JOURNAL_FIELDS)
+				journal.send(MESSAGE=l, **fields)
+			self.assertTrue(self.isFilled(10))
+			endtm = MyTime.time()+10
+			while len(self.jail) != 2 and MyTime.time() < endtm:
+				time.sleep(0.10)
+			self.assertEqual(sorted([self.jail.getFailTicket().getIP(), self.jail.getFailTicket().getIP()]), 
+				["192.0.2.1", "192.0.2.2"])
+
 	return MonitorJournalFailures


--- a/fail2ban/tests/misctestcase.py
+++ b/fail2ban/tests/misctestcase.py
@ -23,6 +23,7 @@ __license__ = "GPL"

 import logging
 import os
+import re
 import sys
 import unittest
 import tempfile
@ -32,8 +33,11 @@ import datetime
 from glob import glob
 from StringIO import StringIO

+from utils import LogCaptureTestCase, logSys as DefLogSys
+
 from ..helpers import formatExceptionInfo, mbasename, TraceBack, FormatterWithTraceBack, getLogger
 from ..helpers import splitwords
+from ..server.datedetector import DateDetector
 from ..server.datetemplate import DatePatternRegex


@ -67,6 +71,22 @@ class HelpersTest(unittest.TestCase):
 		self.assertEqual(splitwords(' 1\n  2, 3'), ['1', '2', '3'])


+if sys.version_info >= (2,7):
+	def _sh_call(cmd):
+		import subprocess, locale
+		ret = subprocess.check_output(cmd, shell=True)
+		if sys.version_info >= (3,):
+			ret = ret.decode(locale.getpreferredencoding(), 'replace')
+		return str(ret).rstrip()
+else:
+	def _sh_call(cmd):
+		import subprocess
+		ret = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout.read()
+		return str(ret).rstrip()
+
+def _getSysPythonVersion():
+	return _sh_call("fail2ban-python -c 'import sys; print(tuple(sys.version_info))'")
+
 class SetupTest(unittest.TestCase):

 	def setUp(self):
@ -76,6 +96,12 @@ class SetupTest(unittest.TestCase):
 			raise unittest.SkipTest(
 				"Seems to be running not out of source distribution"
 				" -- cannot locate setup.py")
+		# compare current version of python installed resp. active one:
+		sysVer = _getSysPythonVersion()
+		if sysVer != str(tuple(sys.version_info)):
+			raise unittest.SkipTest(
+				"Seems to be running with python distribution %s"
+				" -- install can be tested only with system distribution %s" % (str(tuple(sys.version_info)), sysVer))

 	def testSetupInstallRoot(self):
 		if not self.setup:
@ -122,6 +148,14 @@ class SetupTest(unittest.TestCase):
 					  'etc/fail2ban/jail.conf'):
 				self.assertTrue(os.path.exists(os.path.join(tmp, f)),
 								msg="Can't find %s" % f)
+			# Because the install (test) path in virtual-env differs from some development-env,
+			# it is not a `tmp + '/usr/local/bin/'`, so search for it:
+			installedPath = _sh_call('find ' + tmp+ ' -name fail2ban-python').split('\n')
+			self.assertTrue(len(installedPath) > 0)
+			for installedPath in installedPath:
+				self.assertEqual(
+					os.path.realpath(installedPath), os.path.realpath(sys.executable))
+
 		finally:
 			# clean up
 			shutil.rmtree(tmp)
@ -130,7 +164,7 @@ class SetupTest(unittest.TestCase):
 					  % (sys.executable, self.setup))


-class TestsUtilsTest(unittest.TestCase):
+class TestsUtilsTest(LogCaptureTestCase):

 	def testmbasename(self):
 		self.assertEqual(mbasename("sample.py"), 'sample')
@ -165,12 +199,88 @@ class TestsUtilsTest(unittest.TestCase):
 			if not ('fail2ban-testcases' in s):
 				# we must be calling it from setup or nosetests but using at least
 				# nose's core etc
-				self.assertTrue('>' in s, msg="no '>' in %r" % s)
+				self.assertIn('>', s)
 			elif not ('coverage' in s):
 				# There is only "fail2ban-testcases" in this case, no true traceback
-				self.assertFalse('>' in s, msg="'>' present in %r" % s)
+				self.assertNotIn('>', s)

-			self.assertTrue(':' in s, msg="no ':' in %r" % s)
+			self.assertIn(':', s)
+
+	def _testAssertionErrorRE(self, regexp, fun, *args, **kwargs):
+		self.assertRaisesRegexp(AssertionError, regexp, fun, *args, **kwargs)
+	
+	def testExtendedAssertRaisesRE(self):
+		## test _testAssertionErrorRE several fail cases:
+		def _key_err(msg):
+			raise KeyError(msg)			
+		self.assertRaises(KeyError,
+			self._testAssertionErrorRE, r"^failed$", 
+				_key_err, 'failed')
+		self.assertRaises(AssertionError,
+			self._testAssertionErrorRE, r"^failed$",
+				self.fail, '__failed__')
+		self._testAssertionErrorRE(r'failed.* does not match .*__failed__',
+			lambda: self._testAssertionErrorRE(r"^failed$",
+				self.fail, '__failed__')
+		)
+		## no exception in callable:
+		self.assertRaises(AssertionError,
+			self._testAssertionErrorRE, r"", int, 1)
+		self._testAssertionErrorRE(r'0 AssertionError not raised X.* does not match .*AssertionError not raised',
+			lambda: self._testAssertionErrorRE(r"^0 AssertionError not raised X$",
+				lambda: self._testAssertionErrorRE(r"", int, 1))
+		)
+
+	def testExtendedAssertMethods(self):
+		## assertIn, assertNotIn positive case:
+		self.assertIn('a', ['a', 'b', 'c', 'd'])
+		self.assertIn('a', ('a', 'b', 'c', 'd',))
+		self.assertIn('a', 'cba')
+		self.assertIn('a', (c for c in 'cba' if c != 'b'))
+		self.assertNotIn('a', ['b', 'c', 'd'])
+		self.assertNotIn('a', ('b', 'c', 'd',))
+		self.assertNotIn('a', 'cbd')
+		self.assertNotIn('a', (c.upper() for c in 'cba' if c != 'b'))
+		## assertIn, assertNotIn negative case:
+		self._testAssertionErrorRE(r"'a' unexpectedly found in 'cba'",
+			self.assertNotIn, 'a', 'cba')
+		self._testAssertionErrorRE(r"1 unexpectedly found in \[0, 1, 2\]",
+			self.assertNotIn, 1, xrange(3))
+		self._testAssertionErrorRE(r"'A' unexpectedly found in \['C', 'A'\]",
+			self.assertNotIn, 'A', (c.upper() for c in 'cba' if c != 'b'))
+		self._testAssertionErrorRE(r"'a' was not found in 'xyz'",
+			self.assertIn, 'a', 'xyz')
+		self._testAssertionErrorRE(r"5 was not found in \[0, 1, 2\]",
+			self.assertIn, 5, xrange(3))
+		self._testAssertionErrorRE(r"'A' was not found in \['C', 'B'\]",
+			self.assertIn, 'A', (c.upper() for c in 'cba' if c != 'a'))
+		## assertLogged, assertNotLogged positive case:
+		logSys = DefLogSys
+		self.pruneLog()
+		logSys.debug('test "xyz"')
+		self.assertLogged('test "xyz"')
+		self.assertLogged('test', 'xyz', all=True)
+		self.assertNotLogged('test', 'zyx', all=False)
+		self.assertNotLogged('test_zyx', 'zyx', all=True)
+		self.assertLogged('test', 'zyx', all=False)
+		self.pruneLog()
+		logSys.debug('xxxx "xxx"')
+		self.assertNotLogged('test "xyz"')
+		self.assertNotLogged('test', 'xyz', all=False)
+		self.assertNotLogged('test', 'xyz', 'zyx', all=True)
+		## assertLogged, assertNotLogged negative case:
+		self.pruneLog()
+		logSys.debug('test "xyz"')
+		self._testAssertionErrorRE(r"All of the .* were found present in the log",
+			self.assertNotLogged, 'test "xyz"')
+		self._testAssertionErrorRE(r"was found in the log",
+			self.assertNotLogged, 'test', 'xyz', all=True)
+		self._testAssertionErrorRE(r"was not found in the log",
+			self.assertLogged, 'test', 'zyx', all=True)
+		self._testAssertionErrorRE(r"None among .* was found in the log",
+			self.assertLogged, 'test_zyx', 'zyx', all=False)
+		self._testAssertionErrorRE(r"All of the .* were found present in the log",
+			self.assertNotLogged, 'test', 'xyz', all=False)

 	def testFormatterWithTraceBack(self):
 		strout = StringIO()
@ -231,3 +341,47 @@ class CustomDateFormatsTest(unittest.TestCase):
 		self.assertEqual(
 			date,
 			datetime.datetime(2007, 1, 25, 16, 0))
+
+	def testAmbiguousDatePattern(self):
+		defDD = DateDetector()
+		defDD.addDefaultTemplate()
+		logSys = DefLogSys
+		for (matched, dp, line) in (
+			# positive case:
+			('Jan 23 21:59:59',   None, 'Test failure Jan 23 21:59:59 for 192.0.2.1'),
+			# ambiguous "unbound" patterns (missed):
+			(False,               None, 'Test failure TestJan 23 21:59:59.011 2015 for 192.0.2.1'),
+			(False,               None, 'Test failure Jan 23 21:59:59123456789 for 192.0.2.1'),
+			# ambiguous "no optional year" patterns (matched):
+			('Aug 8 11:25:50',      None, 'Aug 8 11:25:50 14430f2329b8 Authentication failed from 192.0.2.1'),
+			('Aug 8 11:25:50',      None, '[Aug 8 11:25:50] 14430f2329b8 Authentication failed from 192.0.2.1'),
+			('Aug 8 11:25:50 2014', None, 'Aug 8 11:25:50 2014 14430f2329b8 Authentication failed from 192.0.2.1'),
+			# direct specified patterns:
+			('20:00:00 01.02.2003',    r'%H:%M:%S %d.%m.%Y$', '192.0.2.1 at 20:00:00 01.02.2003'),
+			('[20:00:00 01.02.2003]',  r'\[%H:%M:%S %d.%m.%Y\]', '192.0.2.1[20:00:00 01.02.2003]'),
+			('[20:00:00 01.02.2003]',  r'\[%H:%M:%S %d.%m.%Y\]', '[20:00:00 01.02.2003]192.0.2.1'),
+			('[20:00:00 01.02.2003]',  r'\[%H:%M:%S %d.%m.%Y\]$', '192.0.2.1[20:00:00 01.02.2003]'),
+			('[20:00:00 01.02.2003]',  r'^\[%H:%M:%S %d.%m.%Y\]', '[20:00:00 01.02.2003]192.0.2.1'),
+			('[17/Jun/2011 17:00:45]', r'^\[%d/%b/%Y %H:%M:%S\]', '[17/Jun/2011 17:00:45] Attempt, IP address 192.0.2.1'),
+			('[17/Jun/2011 17:00:45]', r'\[%d/%b/%Y %H:%M:%S\]', 'Attempt [17/Jun/2011 17:00:45] IP address 192.0.2.1'),
+			('[17/Jun/2011 17:00:45]', r'\[%d/%b/%Y %H:%M:%S\]', 'Attempt IP address 192.0.2.1, date: [17/Jun/2011 17:00:45]'),
+			# direct specified patterns (begin/end, missed):
+			(False,                 r'%H:%M:%S %d.%m.%Y', '192.0.2.1x20:00:00 01.02.2003'),
+			(False,                 r'%H:%M:%S %d.%m.%Y', '20:00:00 01.02.2003x192.0.2.1'),
+			# direct specified patterns (begin/end, matched):
+			('20:00:00 01.02.2003', r'%H:%M:%S %d.%m.%Y', '192.0.2.1 20:00:00 01.02.2003'),
+			('20:00:00 01.02.2003', r'%H:%M:%S %d.%m.%Y', '20:00:00 01.02.2003 192.0.2.1'),
+		):
+			logSys.debug('== test: %r', (matched, dp, line))
+			if dp is None:
+				dd = defDD
+			else:
+				dp = DatePatternRegex(dp)
+				dd = DateDetector()
+				dd.appendTemplate(dp)
+			date = dd.getTime(line)
+			if matched:
+				self.assertTrue(date)
+				self.assertEqual(matched, date[1].group())
+			else:
+				self.assertEqual(date, None)
--- a/fail2ban/tests/samplestestcase.py
+++ b/fail2ban/tests/samplestestcase.py
@ -94,7 +94,7 @@ def testSampleRegexsFactory(name, basedir):
 			if jsonREMatch:
 				try:
 					faildata = json.loads(jsonREMatch.group(1))
-				except ValueError, e:
+				except ValueError as e:
 					raise ValueError("%s: %s:%i" %
 						(e, logFile.filename(), logFile.filelineno()))
 				line = next(logFile)
--- a/fail2ban/tests/servertestcase.py
+++ b/fail2ban/tests/servertestcase.py
@ -228,7 +228,7 @@ class Transmitter(TransmitterBase):
 		time.sleep(1)
 		self.assertEqual(
 			self.transm.proceed(["stop", self.jailName]), (0, None))
-		self.assertTrue(self.jailName not in self.server._Server__jails)
+		self.assertNotIn(self.jailName, self.server._Server__jails)

 	def testStartStopAllJail(self):
 		self.server.addJail("TestJail2", "auto")
@ -242,8 +242,8 @@ class Transmitter(TransmitterBase):
 		time.sleep(0.1)
 		self.assertEqual(self.transm.proceed(["stop", "all"]), (0, None))
 		time.sleep(1)
-		self.assertTrue(self.jailName not in self.server._Server__jails)
-		self.assertTrue("TestJail2" not in self.server._Server__jails)
+		self.assertNotIn(self.jailName, self.server._Server__jails)
+		self.assertNotIn("TestJail2", self.server._Server__jails)

 	def testJailIdle(self):
 		self.assertEqual(
@ -688,10 +688,7 @@ class Transmitter(TransmitterBase):

 	def testJournalMatch(self):
 		if not filtersystemd: # pragma: no cover
-			if sys.version_info >= (2, 7):
-				raise unittest.SkipTest(
-					"systemd python interface not available")
-			return
+			raise unittest.SkipTest("systemd python interface not available")
 		jailName = "TestJail2"
 		self.server.addJail(jailName, "systemd")
 		values = [
@ -791,10 +788,8 @@ class TransmitterLogging(TransmitterBase):
 		self.setGetTest("logtarget", "STDERR")

 	def testLogTargetSYSLOG(self):
-		if not os.path.exists("/dev/log") and sys.version_info >= (2, 7):
+		if not os.path.exists("/dev/log"):
 			raise unittest.SkipTest("'/dev/log' not present")
-		elif not os.path.exists("/dev/log"):
-			return
 		self.assertTrue(self.server.getSyslogSocket(), "auto")
 		self.setGetTest("logtarget", "SYSLOG")
 		self.assertTrue(self.server.getSyslogSocket(), "/dev/log")
--- a/fail2ban/tests/utils.py
+++ b/fail2ban/tests/utils.py
@ -22,13 +22,17 @@ __author__ = "Yaroslav Halchenko"
 __copyright__ = "Copyright (c) 2013 Yaroslav Halchenko"
 __license__ = "GPL"

+import itertools
 import logging
 import os
 import re
+import tempfile
+import shutil
 import sys
 import time
 import unittest
 from StringIO import StringIO
+from functools import wraps

 from ..server.mytime import MyTime
 from ..helpers import getLogger
@ -45,6 +49,40 @@ if not CONFIG_DIR:
 		CONFIG_DIR = '/etc/fail2ban'


+def with_tmpdir(f):
+	"""Helper decorator to create a temporary directory
+
+	Directory gets removed after function returns, regardless
+	if exception was thrown of not
+	"""
+	@wraps(f)
+	def wrapper(self, *args, **kwargs):
+		tmp = tempfile.mkdtemp(prefix="f2b-temp")
+		try:
+			return f(self, tmp, *args, **kwargs)
+		finally:
+			# clean up
+			shutil.rmtree(tmp)
+	return wrapper
+
+
+# backwards compatibility to python 2.6:
+if not hasattr(unittest, 'SkipTest'): # pragma: no cover
+	class SkipTest(Exception):
+		pass
+	unittest.SkipTest = SkipTest
+	_org_AddError = unittest._TextTestResult.addError
+	def addError(self, test, err):
+		if err[0] is SkipTest: 
+			if self.showAll:
+				self.stream.writeln(str(err[1]))
+			elif self.dots:
+				self.stream.write('s')
+				self.stream.flush()
+			return
+		_org_AddError(self, test, err)
+	unittest._TextTestResult.addError = addError
+
 def mtimesleep():
 	# no sleep now should be necessary since polling tracks now not only
 	# mtime but also ino and size
@ -183,13 +221,13 @@ def gatherTests(regexps=None, no_network=False):
 	try:
 		from ..server.filtergamin import FilterGamin
 		filters.append(FilterGamin)
-	except Exception, e: # pragma: no cover
+	except Exception as e: # pragma: no cover
 		logSys.warning("Skipping gamin backend testing. Got exception '%s'" % e)

 	try:
 		from ..server.filterpyinotify import FilterPyinotify
 		filters.append(FilterPyinotify)
-	except Exception, e: # pragma: no cover
+	except Exception as e: # pragma: no cover
 		logSys.warning("I: Skipping pyinotify backend testing. Got exception '%s'" % e)

 	for Filter_ in filters:
@ -198,7 +236,7 @@ def gatherTests(regexps=None, no_network=False):
 	try: # pragma: systemd no cover
 		from ..server.filtersystemd import FilterSystemd
 		tests.addTest(unittest.makeSuite(filtertestcase.get_monitor_failures_journal_testcase(FilterSystemd)))
-	except Exception, e: # pragma: no cover
+	except Exception as e: # pragma: no cover
 		logSys.warning("I: Skipping systemd backend testing. Got exception '%s'" % e)

 	# Server test for logging elements which break logging used to support
@ -208,16 +246,45 @@ def gatherTests(regexps=None, no_network=False):
 	return tests


-# forwards compatibility of unittest.TestCase for some early python versions
-if not hasattr(unittest.TestCase, 'assertIn'):
-	def __assertIn(self, a, b, msg=None):
-		if a not in b: # pragma: no cover
-			self.fail(msg or "%r was not found in %r" % (a, b))
-	unittest.TestCase.assertIn = __assertIn
-	def __assertNotIn(self, a, b, msg=None):
-		if a in b: # pragma: no cover
-			self.fail(msg or "%r was found in %r" % (a, b))
-	unittest.TestCase.assertNotIn = __assertNotIn
+#
+# Forwards compatibility of unittest.TestCase for some early python versions
+#
+
+if not hasattr(unittest.TestCase, 'assertRaisesRegexp'):
+	def assertRaisesRegexp(self, exccls, regexp, fun, *args, **kwargs):
+		try:
+			fun(*args, **kwargs)
+		except exccls as e:
+			if re.search(regexp, str(e)) is None:
+				self.fail('\"%s\" does not match \"%s\"' % (regexp, e))
+		else:
+			self.fail('%s not raised' % getattr(exccls, '__name__'))
+	unittest.TestCase.assertRaisesRegexp = assertRaisesRegexp
+
+# always custom following methods, because we use atm better version of both (support generators)
+if True: ## if not hasattr(unittest.TestCase, 'assertIn'):
+	def assertIn(self, a, b, msg=None):
+		bb = b
+		wrap = False
+		if msg is None and hasattr(b, '__iter__') and not isinstance(b, basestring):
+			b, bb = itertools.tee(b)
+			wrap = True
+		if a not in b:
+			if wrap: bb = list(bb)
+			msg = msg or "%r was not found in %r" % (a, bb)
+			self.fail(msg)
+	unittest.TestCase.assertIn = assertIn
+	def assertNotIn(self, a, b, msg=None):
+		bb = b
+		wrap = False
+		if msg is None and hasattr(b, '__iter__') and not isinstance(b, basestring):
+			b, bb = itertools.tee(b)
+			wrap = True
+		if a in b:
+			if wrap: bb = list(bb)
+			msg = msg or "%r unexpectedly found in %r" % (a, bb)
+			self.fail(msg)
+	unittest.TestCase.assertNotIn = assertNotIn


 class LogCaptureTestCase(unittest.TestCase):
@ -241,6 +308,7 @@ class LogCaptureTestCase(unittest.TestCase):
 	def tearDown(self):
 		"""Call after every test case."""
 		# print "O: >>%s<<" % self._log.getvalue()
+		self.pruneLog()
 		logSys = getLogger("fail2ban")
 		logSys.handlers = self._old_handlers
 		logSys.level = self._old_level
@ -248,7 +316,7 @@ class LogCaptureTestCase(unittest.TestCase):
 	def _is_logged(self, s):
 		return s in self._log.getvalue()

-	def assertLogged(self, *s):
+	def assertLogged(self, *s, **kwargs):
 		"""Assert that one of the strings was logged

 		Preferable to assertTrue(self._is_logged(..)))
@ -258,14 +326,23 @@ class LogCaptureTestCase(unittest.TestCase):
 		----------
 		s : string or list/set/tuple of strings
 		  Test should succeed if string (or any of the listed) is present in the log
+		all : boolean (default False) if True should fail if any of s not logged
 		"""
 		logged = self._log.getvalue()
-		for s_ in s:
-			if s_ in logged:
-				return
-		raise AssertionError("None among %r was found in the log: %r" % (s, logged))
+		if not kwargs.get('all', False):
+			# at least one entry should be found:
+			for s_ in s:
+				if s_ in logged:
+					return
+			if True: # pragma: no cover
+				self.fail("None among %r was found in the log: ===\n%s===" % (s, logged))
+		else:
+			# each entry should be found:
+			for s_ in s:
+				if s_ not in logged: # pragma: no cover
+					self.fail("%r was not found in the log: ===\n%s===" % (s_, logged))

-	def assertNotLogged(self, *s):
+	def assertNotLogged(self, *s, **kwargs):
 		"""Assert that strings were not logged

 		Parameters
@ -273,13 +350,22 @@ class LogCaptureTestCase(unittest.TestCase):
 		s : string or list/set/tuple of strings
 		  Test should succeed if the string (or at least one of the listed) is not
 		  present in the log
+		all : boolean (default False) if True should fail if any of s logged
 		"""
 		logged = self._log.getvalue()
-		for s_ in s:
-			if s_ not in logged:
-				return
-		raise AssertionError("All of the %r were found present in the log: %r" % (s, logged))
+		if not kwargs.get('all', False):
+			for s_ in s:
+				if s_ not in logged:
+					return
+			if True: # pragma: no cover
+				self.fail("All of the %r were found present in the log: ===\n%s===" % (s, logged))
+		else:
+			for s_ in s:
+				if s_ in logged: # pragma: no cover
+					self.fail("%r was found in the log: ===\n%s===" % (s_, logged))

+	def pruneLog(self):
+		self._log.truncate(0)

 	def getLog(self):
 		return self._log.getvalue()
--- a/fail2ban/version.py
+++ b/fail2ban/version.py
@ -24,4 +24,4 @@ __author__ = "Cyril Jaquier, Yaroslav Halchenko, Steven Hiscocks, Daniel Black"
 __copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2005-2016 Yaroslav Halchenko, 2013-2014 Steven Hiscocks, Daniel Black"
 __license__ = "GPL-v2+"

-version = "0.9.5"
+version = "0.9.6"
--- a/files/monit/fail2ban
+++ b/files/monit/fail2ban
@ -1,7 +1,7 @@
 check process fail2ban with pidfile /var/run/fail2ban/fail2ban.pid
    group services
    start program = "/etc/init.d/fail2ban force-start"
-    stop  program = "/etc/init.d/fail2ban stop || :"
+    stop  program = "/etc/init.d/fail2ban stop"
    if failed unixsocket /var/run/fail2ban/fail2ban.sock then restart
    if 5 restarts within 5 cycles then timeout

--- a/man/fail2ban-client.1
+++ b/man/fail2ban-client.1
@ -1,12 +1,12 @@
-.\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.3.
-.TH FAIL2BAN-CLIENT "1" "July 2016" "fail2ban-client v0.9.5" "User Commands"
+.\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.4.
+.TH FAIL2BAN-CLIENT "1" "December 2016" "fail2ban-client v0.9.6" "User Commands"
 .SH NAME
 fail2ban-client \- configure and control the server
 .SH SYNOPSIS
 .B fail2ban-client
 [\fI\,OPTIONS\/\fR] \fI\,<COMMAND>\/\fR
 .SH DESCRIPTION
-Fail2Ban v0.9.5 reads log file that contains password failure report
+Fail2Ban v0.9.6 reads log file that contains password failure report
 and bans the corresponding IP addresses using firewall rules.
 .SH OPTIONS
 .TP
--- a/man/fail2ban-regex.1
+++ b/man/fail2ban-regex.1
@ -1,5 +1,5 @@
-.\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.3.
-.TH FAIL2BAN-REGEX "1" "July 2016" "fail2ban-regex 0.9.5" "User Commands"
+.\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.4.
+.TH FAIL2BAN-REGEX "1" "December 2016" "fail2ban-regex 0.9.6" "User Commands"
 .SH NAME
 fail2ban-regex \- test Fail2ban "failregex" option
 .SH SYNOPSIS
--- a/man/fail2ban-server.1
+++ b/man/fail2ban-server.1
@ -1,12 +1,12 @@
-.\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.3.
-.TH FAIL2BAN-SERVER "1" "July 2016" "fail2ban-server v0.9.5" "User Commands"
+.\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.4.
+.TH FAIL2BAN-SERVER "1" "December 2016" "fail2ban-server v0.9.6" "User Commands"
 .SH NAME
 fail2ban-server \- start the server
 .SH SYNOPSIS
 .B fail2ban-server
 [\fI\,OPTIONS\/\fR]
 .SH DESCRIPTION
-Fail2Ban v0.9.5 reads log file that contains password failure report
+Fail2Ban v0.9.6 reads log file that contains password failure report
 and bans the corresponding IP addresses using firewall rules.
 .PP
 Only use this command for debugging purpose. Start the server with
--- a/man/fail2ban-testcases.1
+++ b/man/fail2ban-testcases.1
@ -1,5 +1,5 @@
-.\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.3.
-.TH FAIL2BAN-TESTCASES "1" "July 2016" "fail2ban-testcases 0.9.5" "User Commands"
+.\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.4.
+.TH FAIL2BAN-TESTCASES "1" "December 2016" "fail2ban-testcases 0.9.6" "User Commands"
 .SH NAME
 fail2ban-testcases \- run Fail2Ban unit-tests
 .SH SYNOPSIS
--- a/setup.py
+++ b/setup.py
@ -19,9 +19,11 @@
 # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.

 __author__ = "Cyril Jaquier, Steven Hiscocks, Yaroslav Halchenko"
-__copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2008-2013 Fail2Ban Contributors"
+__copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2008-2016 Fail2Ban Contributors"
 __license__ = "GPL"

+import platform
+
 try:
 	import setuptools
 	from setuptools import setup
@ -38,12 +40,40 @@ except ImportError:
 	# python 2.x
 	from distutils.command.build_py import build_py
 	from distutils.command.build_scripts import build_scripts
+# all versions
+from distutils.command.install_scripts import install_scripts
+
 import os
 from os.path import isfile, join, isdir, realpath
 import sys
 import warnings
 from glob import glob

+from fail2ban.setup import updatePyExec
+
+
+# Wrapper to install python binding (to current python version):
+class install_scripts_f2b(install_scripts):
+
+	def get_outputs(self):
+		outputs = install_scripts.get_outputs(self)
+		fn = None
+		for fn in outputs:
+			if os.path.basename(fn) == 'fail2ban-server':
+				break
+		bindir = os.path.dirname(fn)
+		print('creating fail2ban-python binding -> %s' % (bindir,))
+		updatePyExec(bindir)
+		return outputs
+
+
+# Update fail2ban-python env to current python version (where f2b-modules located/installed)
+rootdir = os.path.realpath(os.path.dirname(
+	# __file__ seems to be overwritten sometimes on some python versions (e.g. bug of 2.6 by running under cProfile, etc.):
+	sys.argv[0] if os.path.basename(sys.argv[0]) == 'setup.py' else __file__
+))
+updatePyExec(os.path.join(rootdir, 'bin'))
+
 if setuptools and "test" in sys.argv:
 	import logging
 	logSys = logging.getLogger("fail2ban")
@ -85,6 +115,18 @@ if os.path.exists('/var/run'):
 	# realpath is used to possibly resolve /var/run -> /run symlink
 	data_files_extra += [(realpath('/var/run/fail2ban'), '')]

+# Installing documentation files only under Linux or other GNU/ systems
+# (e.g. GNU/kFreeBSD), since others might have protective mechanisms forbidding
+# installation there (see e.g. #1233)
+platform_system = platform.system().lower()
+doc_files = ['README.md', 'DEVELOP', 'FILTERS', 'doc/run-rootless.txt']
+if platform_system in ('solaris', 'sunos'):
+	doc_files.append('README.Solaris')
+if platform_system in ('linux', 'solaris', 'sunos') or platform_system.startswith('gnu'):
+	data_files_extra.append(
+		('/usr/share/doc/fail2ban', doc_files)
+	)
+
 # Get version number, avoiding importing fail2ban.
 # This is due to tests not functioning for python3 as 2to3 takes place later
 exec(open(join("fail2ban", "version.py")).read())
@ -99,12 +141,16 @@ setup(
 	url = "http://www.fail2ban.org",
 	license = "GPL",
 	platforms = "Posix",
-	cmdclass = {'build_py': build_py, 'build_scripts': build_scripts},
+	cmdclass = {
+		'build_py': build_py, 'build_scripts': build_scripts, 
+		'install_scripts': install_scripts_f2b
+	},
 	scripts = [
 		'bin/fail2ban-client',
 		'bin/fail2ban-server',
 		'bin/fail2ban-regex',
 		'bin/fail2ban-testcases',
+		# 'bin/fail2ban-python', -- link (binary), will be installed via install_scripts_f2b wrapper
 	],
 	packages = [
 		'fail2ban',
@ -148,10 +194,6 @@ setup(
 		('/var/lib/fail2ban',
 			''
 		),
-		('/usr/share/doc/fail2ban',
-			['README.md', 'README.Solaris', 'DEVELOP', 'FILTERS',
-			 'doc/run-rootless.txt']
-		)
 	] + data_files_extra,
 	**setup_extra
 )