Merge branch '0.10' into 0.10-full

2017-05-17 11:35:33 +02:00 · 2017-05-17 11:35:33 +02:00 · 6724de54e6
parent ef975307c4 17b0945a70
commit 6724de54e6
74 changed files with 1429 additions and 438 deletions
--- a/.travis.yml
+++ b/.travis.yml
@ -12,10 +12,13 @@ python:
  - 3.4
  - 3.5
  - 3.6
-  - pypy3
+  # disabled since setuptools dropped support for Python 3.0 - 3.2
+  # - pypy3
+  - pypy3.3-5.2-alpha1
 before_install:
-  - if [[ $TRAVIS_PYTHON_VERSION == 2* || $TRAVIS_PYTHON_VERSION == 'pypy' ]]; then export F2B_PY_2=true && echo "Set F2B_PY_2"; fi
-  - if [[ $TRAVIS_PYTHON_VERSION == 3* || $TRAVIS_PYTHON_VERSION == 'pypy3' ]]; then export F2B_PY_3=true && echo "Set F2B_PY_3"; fi
+  - echo "running under $TRAVIS_PYTHON_VERSION"
+  - if [[ $TRAVIS_PYTHON_VERSION == 2* || $TRAVIS_PYTHON_VERSION == pypy* && $TRAVIS_PYTHON_VERSION != pypy3* ]]; then export F2B_PY_2=true && echo "Set F2B_PY_2"; fi
+  - if [[ $TRAVIS_PYTHON_VERSION == 3* || $TRAVIS_PYTHON_VERSION == pypy3* ]]; then export F2B_PY_3=true && echo "Set F2B_PY_3"; fi
  - travis_retry sudo apt-get update -qq
  # Set this so sudo executes the correct python binary
  #   Anything not using sudo will already have the correct environment
--- a/57
+++ b/57
@ -13,6 +13,8 @@ TODO: implementing of options resp. other tasks from PR #1346
      documentation should be extended (new options, etc)
      
 ### Fixes
+* `filter.d/apache-auth.conf`:
+  - better failure recognition using short form of regex (url/referer are foreign inputs, see gh-1645)
 * `filter.d/pam-generic.conf`:
  - [grave] injection on user name to host fixed
 * `filter.d/sshd.conf`:
@ -23,10 +25,15 @@ TODO: implementing of options resp. other tasks from PR #1346
 * filter.d/sendmail-reject.conf:
  - rewritten using `prefregex` and used MLFID-related multi-line parsing;
  - optional parameter `mode` introduced: normal (default), extra or aggressive
+* filter.d/haproxy-http-auth: do not mistake client port for part of an IPv6 address (gh-1745)
 * `action.d/complain.conf`
  - fixed using new tag `<ip-rev>` (sh/dash compliant now)
 * `action.d/sendmail-geoip-lines.conf`
  - fixed using new tag `<ip-host>` (without external command execution)
+* fail2ban-regex: fixed matched output by multi-line (buffered) parsing
+* fail2ban-regex: support for multi-line debuggex URL implemented (gh-422)
+* fixed ipv6-action errors on systems not supporting ipv6 and vice versa (gh-1741)
+* fixed directory-based log-rotate for pyinotify-backend (gh-1778)

 ### New Features
 * New Actions:
@ -41,9 +48,13 @@ TODO: implementing of options resp. other tasks from PR #1346
  using single-line expressions:
  - tag `<F-MLFID>`: used to identify resp. store failure info for groups of log-lines with the same
    identifier (e. g. combined failure-info for the same conn-id by `<F-MLFID>(?:conn-id)</F-MLFID>`,
-    see sshd.conf for example)
+    see sshd.conf for example);
+  - tag `<F-MLFFORGET>`: can be used as mark to forget current multi-line MLFID (e. g. by connection 
+    closed, reset or disconnect etc);
  - tag `<F-NOFAIL>`: used as mark for no-failure (helper to accumulate common failure-info, 
    e. g. from lines that contain IP-address);
+  Opposite to obsolete multi-line parsing (using buffering with `maxlines`) it is more precise and 
+  can recognize multiple failure attempts within the same connection (MLFID).
 * Several filters optimized with pre-filtering using new option `prefregex`, and multiline filter 
  using `<F-MLFID>` + `<F-NOFAIL>` combination;
 * Exposes filter group captures in actions (non-recursive interpolation of tags `<F-...>`, 
@ -59,11 +70,34 @@ TODO: implementing of options resp. other tasks from PR #1346
  - `<ip-rev>` - PTR reversed representation of IP address
  - `<ip-host>` - host name of the IP address
  - `<F-...>` - interpolates to the corresponding filter group capture `...`
+  - `<fq-hostname>` - fully-qualified name of host (the same as `$(hostname -f)`)
+  - `<sh-hostname>` - short hostname (the same as `$(uname -n)`)
 * Allow to use filter options by `fail2ban-regex`, example:
  fail2ban-regex text.log "sshd[mode=aggressive]"
 * Samples test case factory extended with filter options - dict in JSON to control 
  filter options (e. g. mode, etc.):
  # filterOptions: {"mode": "aggressive"}
+* Introduced new jail option "ignoreself", specifies whether the local resp. own IP addresses 
+  should be ignored (default is true). Fail2ban will not ban a host which matches such addresses.
+  Option "ignoreip" affects additionally to "ignoreself" and don't need to include the DNS 
+  resp. IPs of the host self.
+* Regex will be compiled as MULTILINE only if needed (buffering with `maxlines` > 1), that enables:
+  - to improve performance by the single line parsing (see gh-1733);
+  - make regex more precise (because distinguish between anchors `^`/`$` for the begin/end of string
+    and the new-line character '\n', e. g. if coming from filters (like systemd journal) that allow
+    the parsing of log-entries contain new-line chars (as single entry);
+  - if multiline regex however expected (by single-line parsing without buffering) - prefix `(?m)` 
+    could be used in regex to enable it;
+* implemented execution of `actionstart` on demand (conditional), if action depends on `family` (gh-1742):
+  - new action parameter `actionstart_on_demand` (bool) can be set to prevent/allow starting action
+    on demand (default retrieved automatically, if some conditional parameter `param?family=...` 
+    presents in action properties), see `action.d/pf.conf` for example;
+  - additionally `actionstop` will be executed only for families previously executing `actionstart`
+    (starting on demand only)
+* introduced new command `actionflush`: executed in order to flush all bans at once 
+  e. g. by unban all, reload with removing action, stop, shutdown the system (gh-1743),
+  the actions having `actionflush` do not execute `actionunban` for each single ticket
+* add new command `actionflush` default for several iptables/iptables-ipset actions (and common include);


 ver. 0.10.0-alpha-1 (2016/07/14) - ipv6-support-etc
@ -265,13 +299,25 @@ fail2ban-client set loglevel INFO
  - new `with_foreground_server_thread` decorator to test several client/server commands


-ver. 0.9.x (2016/??/??) - wanna-be-released
+ver. 0.9.8 (2016/XX/XXX) - wanna-be-released
 -----------

 0.9.x line is no longer heavily developed.  If you are interested in
 new features (e.g. IPv6 support), please consider 0.10 branch and its
 releases.

+
+### Fixes
+
+### New Features
+
+### Enhancements
+
+
+
+ver. 0.9.7 (2017/05/11) - awaiting-victory
+-----------
+
 ### Fixes
 * Fixed a systemd-journal handling in fail2ban-regex (gh-1657)
 * filter.d/sshd.conf
@ -280,6 +326,10 @@ releases.
      (0.10th resp. IPv6 relevant only, amend for gh-1479)
 * config/pathes-freebsd.conf
    - Fixed filenames for apache and nginx log files (gh-1667)
+* filter.d/exim.conf
+    - optional part `(...)` after host-name before `[IP]` (gh-1751)
+    - new reason "Unrouteable address" for "rejected RCPT" regex (gh-1762)
+    - match of complex time like `D=2m42s` in regex "no MAIL in SMTP connection" (gh-1766)
 * filter.d/sshd.conf
    - new aggressive rules (gh-864):
      - Connection reset by peer (multi-line rule during authorization process)
@ -294,7 +344,7 @@ releases.
 * filter.d/cyrus-imap.conf
    - accept entries without login-info resp. hostname before IP address (gh-1707)
 * Filter tests extended with check of all config-regexp, that contains greedy catch-all
-  before `<HOST>`, that is hard-anchored at end or precise  sub expression after `<HOST>`
+  before `<HOST>`, that is hard-anchored at end or precise sub expression after `<HOST>`

 ### New Features
 * New Actions:
@ -304,6 +354,7 @@ releases.
    - filter.d/domino-smtp: IBM Domino SMTP task (gh-1603)

 ### Enhancements
+* Introduced new log-level `MSG` (as INFO-2, equivalent to 18)


 ver. 0.9.6 (2016/12/10) - stretch-is-coming
--- a/README.md
+++ b/README.md
@ -22,7 +22,8 @@ mechanisms if you really want to protect services.
 ------|------

 This README is a quick introduction to Fail2ban. More documentation, FAQ, HOWTOs
-are available in fail2ban(1) manpage and on the website http://www.fail2ban.org
+are available in fail2ban(1) manpage, [Wiki](https://github.com/fail2ban/fail2ban/wiki)
+and on the website http://www.fail2ban.org

 Installation:
 -------------
@ -89,7 +90,7 @@ Contact:
 See [CONTRIBUTING.md](https://github.com/fail2ban/fail2ban/blob/master/CONTRIBUTING.md)

 ### You just appreciate this program:
-send kudos to the original author ([Cyril Jaquier](mailto: Cyril Jaquier <cyril.jaquier@fail2ban.org>))
+send kudos to the original author ([Cyril Jaquier](mailto:cyril.jaquier@fail2ban.org))
 or *better* to the [mailing list](https://lists.sourceforge.net/lists/listinfo/fail2ban-users)
 since Fail2Ban is "community-driven" for years now.

--- a/config/action.d/dummy.conf
+++ b/config/action.d/dummy.conf
@ -10,14 +10,23 @@
 # Notes.:  command executed once at the start of Fail2Ban.
 # Values:  CMD
 #
-actionstart = touch /var/run/fail2ban/fail2ban.dummy
-              printf %%b "<init>\n" >> /var/run/fail2ban/fail2ban.dummy
+actionstart = if [ ! -z '<target>' ]; then touch <target>; fi;
+              printf %%b "<init>\n" <to_target>
+              echo "%(debug)s started"
+
+# Option:  actionflush
+# Notes.:  command executed once to flush (clear) all IPS, by shutdown (resp. by stop of the jail or this action)
+# Values:  CMD
+#
+actionflush = printf %%b "-*\n" <to_target>
+              echo "%(debug)s clear all"

 # Option:  actionstop
 # Notes.:  command executed once at the end of Fail2Ban
 # Values:  CMD
 #
-actionstop = rm -f /var/run/fail2ban/fail2ban.dummy
+actionstop = if [ ! -z '<target>' ]; then rm -f <target>; fi;
+             echo "%(debug)s stopped"

 # Option:  actioncheck
 # Notes.:  command executed once before each actionban command
@ -31,7 +40,8 @@ actioncheck =
 # Tags:    See jail.conf(5) man page
 # Values:  CMD
 #
-actionban = printf %%b "+<ip>\n" >> /var/run/fail2ban/fail2ban.dummy
+actionban = printf %%b "+<ip>\n" <to_target>
+            echo "%(debug)s banned <ip> (family: <family>)"

 # Option:  actionunban
 # Notes.:  command executed when unbanning an IP. Take care that the
@ -39,9 +49,15 @@ actionban = printf %%b "+<ip>\n" >> /var/run/fail2ban/fail2ban.dummy
 # Tags:    See jail.conf(5) man page
 # Values:  CMD
 #
-actionunban = printf %%b "-<ip>\n" >> /var/run/fail2ban/fail2ban.dummy
+actionunban = printf %%b "-<ip>\n" <to_target>
+              echo "%(debug)s unbanned <ip> (family: <family>)"
+
+
+debug = [<name>] <actname> <target> --

 [Init]

 init = 123

+target = /var/run/fail2ban/fail2ban.dummy
+to_target = >> <target>
--- a/config/action.d/iptables-allports.conf
+++ b/config/action.d/iptables-allports.conf
@ -26,7 +26,7 @@ actionstart = <iptables> -N f2b-<name>
 # Values:  CMD
 #
 actionstop = <iptables> -D <chain> -p <protocol> -j f2b-<name>
-             <iptables> -F f2b-<name>
+             <actionflush>
             <iptables> -X f2b-<name>

 # Option:  actioncheck
--- a/config/action.d/iptables-common.conf
+++ b/config/action.d/iptables-common.conf
@ -16,6 +16,14 @@ after = iptables-blocktype.local
        iptables-common.local
 # iptables-blocktype.local is obsolete

+[Definition]
+
+# Option:  actionflush
+# Notes.:  command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action)
+# Values:  CMD
+#
+actionflush = <iptables> -F f2b-<name>
+

 [Init]

--- a/config/action.d/iptables-ipset-proto4.conf
+++ b/config/action.d/iptables-ipset-proto4.conf
@ -30,12 +30,19 @@ before = iptables-common.conf
 actionstart = ipset --create f2b-<name> iphash
              <iptables> -I <chain> -p <protocol> -m multiport --dports <port> -m set --match-set f2b-<name> src -j <blocktype>

+
+# Option:  actionflush
+# Notes.:  command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action)
+# Values:  CMD
+#
+actionflush = ipset --flush f2b-<name>
+
 # Option:  actionstop
 # Notes.:  command executed once at the end of Fail2Ban
 # Values:  CMD
 #
 actionstop = <iptables> -D <chain> -p <protocol> -m multiport --dports <port> -m set --match-set f2b-<name> src -j <blocktype>
-             ipset --flush f2b-<name>
+             <actionflush>
             ipset --destroy f2b-<name>

 # Option:  actionban
--- a/config/action.d/iptables-ipset-proto6-allports.conf
+++ b/config/action.d/iptables-ipset-proto6-allports.conf
@ -29,12 +29,18 @@ before = iptables-common.conf
 actionstart = ipset create <ipmset> hash:ip timeout <bantime><familyopt>
              <iptables> -I <chain> -m set --match-set <ipmset> src -j <blocktype>

+# Option:  actionflush
+# Notes.:  command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action)
+# Values:  CMD
+#
+actionflush = ipset flush <ipmset>
+
 # Option:  actionstop
 # Notes.:  command executed once at the end of Fail2Ban
 # Values:  CMD
 #
 actionstop = <iptables> -D <chain> -m set --match-set <ipmset> src -j <blocktype>
-             ipset flush <ipmset>
+             <actionflush>
             ipset destroy <ipmset>

 # Option:  actionban
--- a/config/action.d/iptables-ipset-proto6.conf
+++ b/config/action.d/iptables-ipset-proto6.conf
@ -29,12 +29,18 @@ before = iptables-common.conf
 actionstart = ipset create <ipmset> hash:ip timeout <bantime><familyopt>
              <iptables> -I <chain> -p <protocol> -m multiport --dports <port> -m set --match-set <ipmset> src -j <blocktype>

+# Option:  actionflush
+# Notes.:  command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action)
+# Values:  CMD
+#
+actionflush = ipset flush <ipmset>
+
 # Option:  actionstop
 # Notes.:  command executed once at the end of Fail2Ban
 # Values:  CMD
 #
 actionstop = <iptables> -D <chain> -p <protocol> -m multiport --dports <port> -m set --match-set <ipmset> src -j <blocktype>
-             ipset flush <ipmset>
+             <actionflush>
             ipset destroy <ipmset>

 # Option:  actionban
--- a/config/action.d/iptables-multiport-log.conf
+++ b/config/action.d/iptables-multiport-log.conf
@ -26,13 +26,19 @@ actionstart = <iptables> -N f2b-<name>
              <iptables> -I f2b-<name>-log -j LOG --log-prefix "$(expr f2b-<name> : '\(.\{1,23\}\)'):DROP " --log-level warning -m limit --limit 6/m --limit-burst 2
              <iptables> -A f2b-<name>-log -j <blocktype>

+# Option:  actionflush
+# Notes.:  command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action)
+# Values:  CMD
+#
+actionflush = <iptables> -F f2b-<name>
+              <iptables> -F f2b-<name>-log
+
 # Option:  actionstop
 # Notes.:  command executed once at the end of Fail2Ban
 # Values:  CMD
 #
 actionstop = <iptables> -D <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name>
-             <iptables> -F f2b-<name>
-             <iptables> -F f2b-<name>-log
+             <actionflush>
             <iptables> -X f2b-<name>
             <iptables> -X f2b-<name>-log

--- a/config/action.d/iptables-multiport.conf
+++ b/config/action.d/iptables-multiport.conf
@ -23,7 +23,7 @@ actionstart = <iptables> -N f2b-<name>
 # Values:  CMD
 #
 actionstop = <iptables> -D <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name>
-             <iptables> -F f2b-<name>
+             <actionflush>
             <iptables> -X f2b-<name>

 # Option:  actioncheck
--- a/config/action.d/iptables-new.conf
+++ b/config/action.d/iptables-new.conf
@ -25,7 +25,7 @@ actionstart = <iptables> -N f2b-<name>
 # Values:  CMD
 #
 actionstop = <iptables> -D <chain> -m state --state NEW -p <protocol> --dport <port> -j f2b-<name>
-             <iptables> -F f2b-<name>
+             <actionflush>
             <iptables> -X f2b-<name>

 # Option:  actioncheck
--- a/config/action.d/iptables-xt_recent-echo.conf
+++ b/config/action.d/iptables-xt_recent-echo.conf
@ -35,6 +35,12 @@ before = iptables-common.conf
 #    shorter of the two timeouts actually matters.
 actionstart = if [ `id -u` -eq 0 ];then <iptables> -I <chain> -m recent --update --seconds 3600 --name <iptname> -j <blocktype>;fi

+# Option:  actionflush
+#
+# [TODO] Flushing is currently not implemented for xt_recent
+#
+actionflush = 
+
 # Option:  actionstop
 # Notes.:  command executed once at the end of Fail2Ban
 # Values:  CMD
--- a/config/action.d/iptables.conf
+++ b/config/action.d/iptables.conf
@ -23,7 +23,7 @@ actionstart = <iptables> -N f2b-<name>
 # Values:  CMD
 #
 actionstop = <iptables> -D <chain> -p <protocol> --dport <port> -j f2b-<name>
-             <iptables> -F f2b-<name>
+             <actionflush>
             <iptables> -X f2b-<name>

 # Option:  actioncheck
--- a/config/action.d/mail-buffered.conf
+++ b/config/action.d/mail-buffered.conf
@ -17,7 +17,7 @@ actionstart = printf %%b "Hi,\n
              The jail <name> has been started successfully.\n
              Output will be buffered until <lines> lines are available.\n
              Regards,\n
-              Fail2Ban"|mail -s "[Fail2Ban] <name>: started on `uname -n`" <dest>
+              Fail2Ban"|mail -s "[Fail2Ban] <name>: started on <fq-hostname>" <dest>

 # Option:  actionstop
 # Notes.:  command executed once at the end of Fail2Ban
@ -28,13 +28,13 @@ actionstop = if [ -f <tmpfile> ]; then
                 These hosts have been banned by Fail2Ban.\n
                 `cat <tmpfile>`
                 Regards,\n
-                 Fail2Ban"|mail -s "[Fail2Ban] <name>: Summary from `uname -n`" <dest>
+                 Fail2Ban"|mail -s "[Fail2Ban] <name>: Summary from <fq-hostname>" <dest>
                 rm <tmpfile>
             fi
             printf %%b "Hi,\n
             The jail <name> has been stopped.\n
             Regards,\n
-             Fail2Ban"|mail -s "[Fail2Ban] <name>: stopped on `uname -n`" <dest>
+             Fail2Ban"|mail -s "[Fail2Ban] <name>: stopped on <fq-hostname>" <dest>

 # Option:  actioncheck
 # Notes.:  command executed once before each actionban command
--- a/config/action.d/mail-whois-lines.conf
+++ b/config/action.d/mail-whois-lines.conf
@ -21,7 +21,7 @@ norestored = 1
 actionstart = printf %%b "Hi,\n
              The jail <name> has been started successfully.\n
              Regards,\n
-              Fail2Ban" | <mailcmd> -s "[Fail2Ban] <name>: started on `uname -n`" <dest>
+              Fail2Ban" | <mailcmd> "[Fail2Ban] <name>: started on <fq-hostname>" <dest>

 # Option:  actionstop
 # Notes.:  command executed once at the end of Fail2Ban
@ -30,7 +30,7 @@ actionstart = printf %%b "Hi,\n
 actionstop = printf %%b "Hi,\n
             The jail <name> has been stopped.\n
             Regards,\n
-             Fail2Ban" | <mailcmd> -s "[Fail2Ban] <name>: stopped on `uname -n`" <dest>
+             Fail2Ban" | <mailcmd> "[Fail2Ban] <name>: stopped on <fq-hostname>" <dest>

 # Option:  actioncheck
 # Notes.:  command executed once before each actionban command
@ -56,7 +56,7 @@ _ban_mail_content = ( printf %%b "Hi,\n
            Regards,\n
            Fail2Ban" )

-actionban = %(_ban_mail_content)s | <mailcmd> "[Fail2Ban] <name>: banned <ip> from  `uname -n`" <dest>
+actionban = %(_ban_mail_content)s | <mailcmd> "[Fail2Ban] <name>: banned <ip> from <fq-hostname>" <dest>

 # Option:  actionunban
 # Notes.:  command executed when unbanning an IP. Take care that the
--- a/config/action.d/mail-whois.conf
+++ b/config/action.d/mail-whois.conf
@ -20,7 +20,7 @@ norestored = 1
 actionstart = printf %%b "Hi,\n
              The jail <name> has been started successfully.\n
              Regards,\n
-              Fail2Ban"|mail -s "[Fail2Ban] <name>: started on `uname -n`" <dest>
+              Fail2Ban"|mail -s "[Fail2Ban] <name>: started on <fq-hostname>" <dest>

 # Option:  actionstop
 # Notes.:  command executed once at the end of Fail2Ban
@ -29,7 +29,7 @@ actionstart = printf %%b "Hi,\n
 actionstop = printf %%b "Hi,\n
             The jail <name> has been stopped.\n
             Regards,\n
-             Fail2Ban"|mail -s "[Fail2Ban] <name>: stopped on `uname -n`" <dest>
+             Fail2Ban"|mail -s "[Fail2Ban] <name>: stopped on <fq-hostname>" <dest>

 # Option:  actioncheck
 # Notes.:  command executed once before each actionban command
@ -49,7 +49,7 @@ actionban = printf %%b "Hi,\n
            Here is more information about <ip> :\n
            `%(_whois_command)s`\n
            Regards,\n
-            Fail2Ban"|mail -s "[Fail2Ban] <name>: banned <ip> from `uname -n`" <dest>
+            Fail2Ban"|mail -s "[Fail2Ban] <name>: banned <ip> from <fq-hostname>" <dest>

 # Option:  actionunban
 # Notes.:  command executed when unbanning an IP. Take care that the
--- a/config/action.d/mail.conf
+++ b/config/action.d/mail.conf
@ -16,7 +16,7 @@ norestored = 1
 actionstart = printf %%b "Hi,\n
              The jail <name> has been started successfully.\n
              Regards,\n
-              Fail2Ban"|mail -s "[Fail2Ban] <name>: started  on `uname -n`" <dest>
+              Fail2Ban"|mail -s "[Fail2Ban] <name>: started  on <fq-hostname>" <dest>

 # Option:  actionstop
 # Notes.:  command executed once at the end of Fail2Ban
@ -25,7 +25,7 @@ actionstart = printf %%b "Hi,\n
 actionstop = printf %%b "Hi,\n
             The jail <name> has been stopped.\n
             Regards,\n
-             Fail2Ban"|mail -s "[Fail2Ban] <name>: stopped on `uname -n`" <dest>
+             Fail2Ban"|mail -s "[Fail2Ban] <name>: stopped on <fq-hostname>" <dest>

 # Option:  actioncheck
 # Notes.:  command executed once before each actionban command
@ -43,7 +43,7 @@ actionban = printf %%b "Hi,\n
            The IP <ip> has just been banned by Fail2Ban after
            <failures> attempts against <name>.\n
            Regards,\n
-            Fail2Ban"|mail -s "[Fail2Ban] <name>: banned <ip> from `uname -n`" <dest>
+            Fail2Ban"|mail -s "[Fail2Ban] <name>: banned <ip> from <fq-hostname>" <dest>

 # Option:  actionunban
 # Notes.:  command executed when unbanning an IP. Take care that the
--- a/config/action.d/pf.conf
+++ b/config/action.d/pf.conf
@ -18,6 +18,9 @@
 actionstart = echo "table <<tablename>-<name>> persist counters" | pfctl -f- 
              echo "block proto <protocol> from <<tablename>-<name>> to <actiontype>" | pfctl -f-

+# Option:  start_on_demand - to start action on demand
+# Example: `action=pf[actionstart_on_demand=true]`
+actionstart_on_demand = false

 # Option:  actionstop
 # Notes.:  command executed once at the end of Fail2Ban
@ -71,8 +74,6 @@ tablename = f2b
 #
 protocol = tcp

-
-
 # Option: actiontype
 # Notes.: defines additions to the blocking rule
 # Values: leave empty to block all attempts from the host
--- a/config/action.d/sendmail-buffered.conf
+++ b/config/action.d/sendmail-buffered.conf
@ -17,7 +17,7 @@ norestored = 1
 # Notes.:  command executed once at the start of Fail2Ban.
 # Values:  CMD
 #
-actionstart = printf %%b "Subject: [Fail2Ban] <name>: started on `uname -n`
+actionstart = printf %%b "Subject: [Fail2Ban] <name>: started on <fq-hostname>
              From: <sendername> <<sender>>
              To: <dest>\n
              Hi,\n
@ -31,7 +31,7 @@ actionstart = printf %%b "Subject: [Fail2Ban] <name>: started on `uname -n`
 # Values:  CMD
 #
 actionstop = if [ -f <tmpfile> ]; then
-                 printf %%b "Subject: [Fail2Ban] <name>: summary from `uname -n`
+                 printf %%b "Subject: [Fail2Ban] <name>: summary from <fq-hostname>
                 From: <sendername> <<sender>>
                 To: <dest>\n
                 Hi,\n
@ -41,7 +41,7 @@ actionstop = if [ -f <tmpfile> ]; then
                 Fail2Ban" | /usr/sbin/sendmail -f <sender> <dest>
                 rm <tmpfile>
             fi
-             printf %%b "Subject: [Fail2Ban] <name>: stopped  on `uname -n`
+             printf %%b "Subject: [Fail2Ban] <name>: stopped  on <fq-hostname>
             From: Fail2Ban <<sender>>
             To: <dest>\n
             Hi,\n
@ -64,7 +64,7 @@ actioncheck =
 actionban = printf %%b "`date`: <ip> (<failures> failures)\n" >> <tmpfile>
            LINE=$( wc -l <tmpfile> | awk '{ print $1 }' )
            if [ $LINE -ge <lines> ]; then
-                printf %%b "Subject: [Fail2Ban] <name>: summary from `uname -n`
+                printf %%b "Subject: [Fail2Ban] <name>: summary from <fq-hostname>
                From: <sendername> <<sender>>
                To: <dest>\n
                Hi,\n
--- a/config/action.d/sendmail-common.conf
+++ b/config/action.d/sendmail-common.conf
@ -14,7 +14,7 @@ after = sendmail-common.local
 # Notes.:  command executed once at the start of Fail2Ban.
 # Values:  CMD
 #
-actionstart = printf %%b "Subject: [Fail2Ban] <name>: started on `uname -n`
+actionstart = printf %%b "Subject: [Fail2Ban] <name>: started on <fq-hostname>
              Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"`
              From: <sendername> <<sender>>
              To: <dest>\n
@ -27,7 +27,7 @@ actionstart = printf %%b "Subject: [Fail2Ban] <name>: started on `uname -n`
 # Notes.:  command executed once at the end of Fail2Ban
 # Values:  CMD
 #
-actionstop = printf %%b "Subject: [Fail2Ban] <name>: stopped on `uname -n`
+actionstop = printf %%b "Subject: [Fail2Ban] <name>: stopped on <fq-hostname>
             Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"`
             From: <sendername> <<sender>>
             To: <dest>\n
--- a/config/action.d/sendmail-geoip-lines.conf
+++ b/config/action.d/sendmail-geoip-lines.conf
@ -23,7 +23,7 @@ norestored = 1
 # Tags:    See jail.conf(5) man page
 # Values:  CMD
 #
-actionban = ( printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from `uname -n`
+actionban = ( printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from <fq-hostname>
            Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"`
            From: <sendername> <<sender>>
            To: <dest>\n
--- a/config/action.d/sendmail-whois-ipjailmatches.conf
+++ b/config/action.d/sendmail-whois-ipjailmatches.conf
@ -19,7 +19,7 @@ norestored = 1
 # Tags:    See jail.conf(5) man page
 # Values:  CMD
 #
-actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from `uname -n`
+actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from <fq-hostname>
            Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"`
            From: <sendername> <<sender>>
            To: <dest>\n
--- a/config/action.d/sendmail-whois-ipmatches.conf
+++ b/config/action.d/sendmail-whois-ipmatches.conf
@ -19,7 +19,7 @@ norestored = 1
 # Tags:    See jail.conf(5) man page
 # Values:  CMD
 #
-actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from `uname -n`
+actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from <fq-hostname>
            Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"`
            From: <sendername> <<sender>>
            To: <dest>\n
--- a/config/action.d/sendmail-whois-lines.conf
+++ b/config/action.d/sendmail-whois-lines.conf
@ -20,7 +20,7 @@ norestored = 1
 # Tags:    See jail.conf(5) man page
 # Values:  CMD
 #
-actionban = ( printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from `uname -n`
+actionban = ( printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from <fq-hostname>
            Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"`
            From: <sendername> <<sender>>
            To: <dest>\n
--- a/config/action.d/sendmail-whois-matches.conf
+++ b/config/action.d/sendmail-whois-matches.conf
@ -19,7 +19,7 @@ norestored = 1
 # Tags:    See jail.conf(5) man page
 # Values:  CMD
 #
-actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from `uname -n`
+actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from <fq-hostname>
            Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"`
            From: <sendername> <<sender>>
            To: <dest>\n
--- a/config/action.d/sendmail-whois.conf
+++ b/config/action.d/sendmail-whois.conf
@ -19,7 +19,7 @@ norestored = 1
 # Tags:    See jail.conf(5) man page
 # Values:  CMD
 #
-actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from `uname -n`
+actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from <fq-hostname>
            Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"`
            From: <sendername> <<sender>>
            To: <dest>\n
--- a/config/action.d/sendmail.conf
+++ b/config/action.d/sendmail.conf
@ -19,7 +19,7 @@ norestored = 1
 # Tags:    See jail.conf(5) man page
 # Values:  CMD
 #
-actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from `uname -n`
+actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from <fq-hostname>
            Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"`
            From: <sendername> <<sender>>
            To: <dest>\n
--- a/config/action.d/xarf-login-attack.conf
+++ b/config/action.d/xarf-login-attack.conf
@ -46,7 +46,7 @@ actionban = oifs=${IFS}; IFS=.;SEP_IP=( <ip> ); set -- ${SEP_IP}; ADDRESSES=$(di
            FROM=<sender>
            SERVICE=<service>
            FAILURES=<failures>
-            REPORTID=<time>@`uname -n`
+            REPORTID=<time>@<fq-hostname>
            TLP=<tlp>
            PORT=<port>
            DATE=`LC_ALL=C date --date=@<time> +"%%a, %%d %%h %%Y %%T %%z"`
@ -119,7 +119,7 @@ logpath = /dev/null

 # Option:  sender
 # Notes.:  This is the sender that is included in the XARF report
-sender = fail2ban@`uname -n`
+sender = fail2ban@<fq-hostname>

 # Option:  port
 # Notes.:  This is the port number that received the login-attack
--- a/config/filter.d/apache-auth.conf
+++ b/config/filter.d/apache-auth.conf
@ -14,19 +14,16 @@ prefregex = ^%(_apache_error_client)s (?:AH\d+: )?<F-CONTENT>.+</F-CONTENT>$
 # auth_type = ((?:Digest|Basic): )?
 auth_type = ([A-Z]\w+: )?

-failregex = ^client denied by server configuration: (uri )?\S*(, referer: \S+)?\s*$
-            ^user .*? authentication failure for "\S*": Password Mismatch(, referer: \S+)?$
-            ^user .*? not found(: )?\S*(, referer: \S+)?\s*$
-            ^client used wrong authentication scheme: \S*(, referer: \S+)?\s*$
-            ^Authorization of user \S+ to access \S* failed, reason: .*$
-            ^%(auth_type)suser .*?: password mismatch: \S*(, referer: \S+)?\s*$
-            ^%(auth_type)suser `.*?' in realm `.+' (not found|denied by provider): \S*(, referer: \S+)?\s*$
-            ^user .*?: authorization failure for "\S*":(, referer: \S+)?\s*$
-            ^%(auth_type)sinvalid nonce .* received - length is not \S+(, referer: \S+)?\s*$
-            ^%(auth_type)srealm mismatch - got `.*?' but expected `.+'(, referer: \S+)?\s*$
-            ^%(auth_type)sunknown algorithm `.*?' received: \S*(, referer: \S+)?\s*$
-            ^invalid qop `.*?' received: \S*(, referer: \S+)?\s*$
-            ^%(auth_type)sinvalid nonce .*? received - user attempted time travel(, referer: \S+)?\s*$
+failregex = ^client (?:denied by server configuration|used wrong authentication scheme)\b
+            ^user <F-USER>(?:\S*|.*?)</F-USER> (?:auth(?:oriz|entic)ation failure|not found|denied by provider)\b
+            ^Authorization of user <F-USER>(?:\S*|.*?)</F-USER> to access .*? failed\b
+            ^%(auth_type)suser <F-USER>(?:\S*|.*?)</F-USER>: password mismatch\b
+            ^%(auth_type)suser `<F-USER>(?:[^']*|.*?)</F-USER>' in realm `.+' (not found|denied by provider)\b
+            ^%(auth_type)sinvalid nonce .* received - length is not\b
+            ^%(auth_type)srealm mismatch - got `(?:[^']*|.*?)' but expected\b
+            ^%(auth_type)sunknown algorithm `(?:[^']*|.*?)' received\b
+            ^invalid qop `(?:[^']*|.*?)' received\b
+            ^%(auth_type)sinvalid nonce .*? received - user attempted time travel\b

 ignoreregex = 

@ -47,14 +44,17 @@ ignoreregex =
 # all of these expressions. Lots of submodules like mod_authz_* return back to mod_authz_core
 # to return the actual failure.
 #
+# Note that URI can contain spaces.
+#
 # See also: http://wiki.apache.org/httpd/ListOfErrors
 # Expressions that don't have tests and aren't common.
 # more be added with  https://issues.apache.org/bugzilla/show_bug.cgi?id=55284 
-#     ^%(_apache_error_client)s (AH01778: )?user .*: nonce expired \([\d.]+ seconds old - max lifetime [\d.]+\) - sending new nonce\s*$
-#     ^%(_apache_error_client)s (AH01779: )?user .*: one-time-nonce mismatch - sending new nonce\s*$
-#     ^%(_apache_error_client)s (AH02486: )?realm mismatch - got `.*' but no realm specified\s*$
+#     ^user .*: nonce expired \([\d.]+ seconds old - max lifetime [\d.]+\) - sending new nonce\s*$
+#     ^user .*: one-time-nonce mismatch - sending new nonce\s*$
+#     ^realm mismatch - got `(?:[^']*|.*?)' but no realm specified\s*$
 #
-# referer is always in error log messages if it exists added as per the log_error_core function in server/log.c
+# Because url/referer are foreign input, short form of regex used if long enough to idetify failure.
 # 
 # Author: Cyril Jaquier
-# Major edits by Daniel Black and Sergey Brester (sebres)
+# Major edits by Daniel Black and Ben Rubson.
+# Rewritten for v.0.10 by Sergey Brester (sebres).
--- a/config/filter.d/apache-modsecurity.conf
+++ b/config/filter.d/apache-modsecurity.conf
@ -16,4 +16,4 @@ ignoreregex =

 # https://github.com/SpiderLabs/ModSecurity/wiki/ModSecurity-2-Data-Formats
 # Author: Daniel Black
-#         Sergey G. Brester aka sebres (review, optimization)
+#         Sergey G. Brester aka sebres (review, optimization)
--- a/config/filter.d/botsearch-common.conf
+++ b/config/filter.d/botsearch-common.conf
@ -5,7 +5,7 @@
 # Block is the actual non-found directories to block
 block = \/?(<webmail>|<phpmyadmin>|<wordpress>|cgi-bin|mysqladmin)[^,]*

-# These are just convient definitions that assist the blocking of stuff that 
+# These are just convenient definitions that assist the blocking of stuff that 
 # isn't installed
 webmail = roundcube|(ext)?mail|horde|(v-?)?webmail

--- a/config/filter.d/exim.conf
+++ b/config/filter.d/exim.conf
@ -17,13 +17,13 @@ before = exim-common.conf
 #prefregex = ^%(pid)s <F-CONTENT>\b(?:\w+ authenticator failed|([\w\-]+ )?SMTP (?:(?:call|connection) from|protocol(?: synchronization)? error)|no MAIL in|(?:%(host_info_pre)s\[[^\]]+\]%(host_info_suf)s(?:sender verify fail|rejected RCPT|dropped|AUTH command))).+</F-CONTENT>$

 failregex = ^%(pid)s %(host_info)ssender verify fail for <\S+>: (?:Unknown user|Unrouteable address|all relevant MX records point to non-existent hosts)\s*$
-            ^%(pid)s \w+ authenticator failed for (\S+ )?\(\S+\) \[<HOST>\](?::\d+)?(?: I=\[\S+\](:\d+)?)?: 535 Incorrect authentication data( \(set_id=.*\)|: \d+ Time\(s\))?\s*$
-            ^%(pid)s %(host_info)srejected RCPT [^@]+@\S+: (?:relay not permitted|Sender verify failed|Unknown user)\s*$
+            ^%(pid)s \w+ authenticator failed for (?:[^\[\( ]* )?(?:\(\S*\) )?\[<HOST>\](?::\d+)?(?: I=\[\S+\](:\d+)?)?: 535 Incorrect authentication data( \(set_id=.*\)|: \d+ Time\(s\))?\s*$
+            ^%(pid)s %(host_info)srejected RCPT [^@]+@\S+: (?:relay not permitted|Sender verify failed|Unknown user|Unrouteable address)\s*$
            ^%(pid)s SMTP protocol synchronization error \([^)]*\): rejected (?:connection from|"\S+") %(host_info)s(?:next )?input=".*"\s*$
            ^%(pid)s SMTP call from \S+ %(host_info)sdropped: too many nonmail commands \(last was "\S+"\)\s*$
            ^%(pid)s SMTP protocol error in "AUTH \S*(?: \S*)?" %(host_info)sAUTH command used when not advertised\s*$
-            ^%(pid)s no MAIL in SMTP connection from (?:\S* )?(?:\(\S*\) )?%(host_info)sD=\d+s(?: C=\S*)?\s*$
-            ^%(pid)s ([\w\-]+ )?SMTP connection from (?:\S* )?(?:\(\S*\) )?%(host_info)sclosed by DROP in ACL\s*$
+            ^%(pid)s no MAIL in SMTP connection from (?:[^\[\( ]* )?(?:\(\S*\) )?%(host_info)sD=\d\S+s(?: C=\S*)?\s*$
+            ^%(pid)s (?:[\w\-]+ )?SMTP connection from (?:[^\[\( ]* )?(?:\(\S*\) )?%(host_info)sclosed by DROP in ACL\s*$

 ignoreregex = 

--- a/config/filter.d/haproxy-http-auth.conf
+++ b/config/filter.d/haproxy-http-auth.conf
@ -28,7 +28,7 @@ _daemon = haproxy
 #          (?:::f{4,6}:)?(?P<host>[\w\-.^_]+)
 # Values:  TEXT
 #
-failregex = ^%(__prefix_line)s<HOST>.*<NOSRV> -1/-1/-1/-1/\+*\d* 401
+failregex = ^%(__prefix_line)s<HOST>(?::\d+)?\s+.*<NOSRV> -1/-1/-1/-1/\+*\d* 401

 # Option:  ignoreregex
 # Notes.:  regex to ignore. If this regex matches, the line is ignored.
--- a/config/filter.d/mysqld-auth.conf
+++ b/config/filter.d/mysqld-auth.conf
@ -1,4 +1,4 @@
-# Fail2Ban filter for unsuccesfull MySQL authentication attempts
+# Fail2Ban filter for unsuccesful MySQL authentication attempts
 #
 #
 # To log wrong MySQL access attempts add to /etc/my.cnf in [mysqld]:
--- a/config/filter.d/sshd.conf
+++ b/config/filter.d/sshd.conf
@ -37,23 +37,24 @@ cmnfailre = ^[aA]uthentication (?:failure|error|failed) for <F-USER>.*</F-USER>
            ^User <F-USER>.+</F-USER> from <HOST> not allowed because listed in DenyUsers\s*%(__suff)s$
            ^User <F-USER>.+</F-USER> from <HOST> not allowed because not in any group\s*%(__suff)s$
            ^refused connect from \S+ \(<HOST>\)\s*%(__suff)s$
-            ^Received disconnect from <HOST>%(__on_port_opt)s:\s*3: .*: Auth fail%(__suff)s$
+            ^Received <F-MLFFORGET>disconnect</F-MLFFORGET> from <HOST>%(__on_port_opt)s:\s*3: .*: Auth fail%(__suff)s$
            ^User <F-USER>.+</F-USER> from <HOST> not allowed because a group is listed in DenyGroups\s*%(__suff)s$
            ^User <F-USER>.+</F-USER> from <HOST> not allowed because none of user's groups are listed in AllowGroups\s*%(__suff)s$
            ^pam_unix\(sshd:auth\):\s+authentication failure;\s*logname=\S*\s*uid=\d*\s*euid=\d*\s*tty=\S*\s*ruser=<F-USER>\S*</F-USER>\s*rhost=<HOST>\s.*%(__suff)s$
-            ^(error: )?maximum authentication attempts exceeded for <F-USER>.*</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)? \[preauth\]$
+            ^(error: )?maximum authentication attempts exceeded for <F-USER>.*</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)?%(__suff)s$
            ^User <F-USER>.+</F-USER> not allowed because account is locked%(__suff)s
-            ^Disconnecting: Too many authentication failures for <F-USER>.+?</F-USER>%(__suff)s
-            ^<F-NOFAIL>Received disconnect</F-NOFAIL> from <HOST>: 11:
-            ^<F-NOFAIL>Connection closed</F-NOFAIL> by <HOST>%(__suff)s$
+            ^<F-MLFFORGET>Disconnecting</F-MLFFORGET>: Too many authentication failures(?: for <F-USER>.+?</F-USER>)?%(__suff)s
+            ^<F-NOFAIL>Received <F-MLFFORGET>disconnect</F-MLFFORGET></F-NOFAIL> from <HOST>: 11:
+            ^<F-NOFAIL>Connection <F-MLFFORGET>closed</F-MLFFORGET></F-NOFAIL> by <HOST>%(__suff)s$

 mdre-normal =

 mdre-ddos = ^Did not receive identification string from <HOST>%(__suff)s$
+            ^Connection <F-MLFFORGET>reset</F-MLFFORGET> by <HOST>%(__on_port_opt)s%(__suff)s
            ^<F-NOFAIL>SSH: Server;Ltype:</F-NOFAIL> (?:Authname|Version|Kex);Remote: <HOST>-\d+;[A-Z]\w+:
-            ^Read from socket failed: Connection reset by peer \[preauth\]
+            ^Read from socket failed: Connection <F-MLFFORGET>reset</F-MLFFORGET> by peer%(__suff)s

-mdre-extra = ^Received disconnect from <HOST>%(__on_port_opt)s:\s*14: No supported authentication methods available%(__suff)s$
+mdre-extra = ^Received <F-MLFFORGET>disconnect</F-MLFFORGET> from <HOST>%(__on_port_opt)s:\s*14: No supported authentication methods available%(__suff)s$
            ^Unable to negotiate with <HOST>%(__on_port_opt)s: no matching (?:cipher|key exchange method) found.
            ^Unable to negotiate a (?:cipher|key exchange method)%(__suff)s$

--- a/config/jail.conf
+++ b/config/jail.conf
@ -82,10 +82,13 @@ before = paths-debian.conf

 # --------------------

-# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not
-# ban a host which matches an address in this list. Several addresses can be
-# defined using space (and/or comma) separator.
-ignoreip = 127.0.0.1/8 ::1
+# "ignorself" specifies whether the local resp. own IP addresses should be ignored
+# (default is true). Fail2ban will not ban a host which matches such addresses.
+#ignorself = true
+# "ignoreip" can be a list of IP addresses, CIDR masks or DNS hosts. Fail2ban
+# will not ban a host which matches an address in this list. Several addresses
+# can be defined using space (and/or comma) separator.
+#ignoreip = 127.0.0.1/8 ::1

 # External command that will take an tagged arguments to ignore, e.g. <ip>,
 # and return true if the IP is to be ignored. False otherwise.
@ -168,7 +171,7 @@ filter = %(__name__)s
 destemail = root@localhost

 # Sender email address used solely for some actions
-sender = root@localhost
+sender = root@<fq-hostname>

 # E-mail action. Since 0.8.1 Fail2Ban uses sendmail MTA for the
 # mailing. Change mta configuration parameter to mail if you want to
--- a/fail2ban/init.py
+++ b/fail2ban/init.py
@ -27,8 +27,10 @@ __license__ = "GPL"
 import logging.handlers

 # Custom debug levels
+logging.MSG = logging.INFO - 2
 logging.TRACEDEBUG = 7
 logging.HEAVYDEBUG = 5
+logging.addLevelName(logging.MSG, 'MSG')
 logging.addLevelName(logging.TRACEDEBUG, 'TRACE')
 logging.addLevelName(logging.HEAVYDEBUG, 'HEAVY')

--- a/fail2ban/client/actionreader.py
+++ b/fail2ban/client/actionreader.py
@ -38,7 +38,9 @@ class ActionReader(DefinitionInitConfigReader):

 	_configOpts = {
 		"actionstart": ["string", None],
+		"actionstart_on_demand": ["string", None],
 		"actionstop": ["string", None],
+		"actionflush": ["string", None],
 		"actionreload": ["string", None],
 		"actioncheck": ["string", None],
 		"actionrepair": ["string", None],
@ -73,8 +75,10 @@ class ActionReader(DefinitionInitConfigReader):
 		opts = self.getCombined(
 			ignore=CommandAction._escapedTags | set(('timeout', 'bantime')))
 		# type-convert only after combined (otherwise boolean converting prevents substitution):
-		if opts.get('norestored'):
-			opts['norestored'] = self._convert_to_boolean(opts['norestored'])
+		for o in ('norestored', 'actionstart_on_demand'):
+			if opts.get(o):
+				opts[o] = self._convert_to_boolean(opts[o])
+		
 		# stream-convert:
 		head = ["set", self._jailName]
 		stream = list()
--- a/fail2ban/client/beautifier.py
+++ b/fail2ban/client/beautifier.py
@ -89,6 +89,8 @@ class Beautifier:
 						val = " ".join(map(str, res1[1])) if isinstance(res1[1], list) else res1[1]
 						msg.append("%s %s:\t%s" % (prefix1, res1[0], val))
 				msg = "\n".join(msg)
+			elif len(inC) < 2:
+				pass # to few cmd args for below
 			elif inC[1] == "syslogsocket":
 				msg = "Current syslog socket is:\n"
 				msg += "`- " + response
@ -110,6 +112,8 @@ class Beautifier:
 				else:
 					msg = "Current database purge age is:\n"
 					msg += "`- %iseconds" % response
+			elif len(inC) < 3:
+				pass # to few cmd args for below
 			elif inC[2] in ("logpath", "addlogpath", "dellogpath"):
 				if len(response) == 0:
 					msg = "No file is currently monitored"
@ -178,7 +182,8 @@ class Beautifier:
 					msg += ", ".join(response)
 		except Exception:
 			logSys.warning("Beautifier error. Please report the error")
-			logSys.error("Beautify %r with %r failed", response, self.__inputCmd)
+			logSys.error("Beautify %r with %r failed", response, self.__inputCmd,
+				exc_info=logSys.getEffectiveLevel()<=logging.DEBUG)
 			msg = repr(msg) + repr(response)
 		return msg

--- a/fail2ban/client/configparserinc.py
+++ b/fail2ban/client/configparserinc.py
@ -32,7 +32,7 @@ from ..helpers import getLogger
 if sys.version_info >= (3,2):

 	# SafeConfigParser deprecated from Python 3.2 (renamed to ConfigParser)
-	from configparser import ConfigParser as SafeConfigParser, \
+	from configparser import ConfigParser as SafeConfigParser, NoSectionError, \
 		BasicInterpolation

 	# And interpolation of __name__ was simply removed, thus we need to
@ -60,7 +60,7 @@ if sys.version_info >= (3,2):
 				parser, option, accum, rest, section, map, depth)

 else: # pragma: no cover
-	from ConfigParser import SafeConfigParser
+	from ConfigParser import SafeConfigParser, NoSectionError

 # Gets the instance of the logger.
 logSys = getLogger(__name__)
@ -200,6 +200,21 @@ after = 1.conf
 	def get_sections(self):
 		return self._sections

+	def options(self, section, withDefault=True):
+		"""Return a list of option names for the given section name.
+
+		Parameter `withDefault` controls the include of names from section `[DEFAULT]`
+		"""
+		try:
+			opts = self._sections[section]
+		except KeyError:
+			raise NoSectionError(section)
+		if withDefault:
+			# mix it with defaults:
+			return set(opts.keys()) | set(self._defaults)
+		# only own option names:
+		return opts.keys()
+
 	def read(self, filenames, get_includes=True):
 		if not isinstance(filenames, list):
 			filenames = [ filenames ]
--- a/fail2ban/client/configreader.py
+++ b/fail2ban/client/configreader.py
@ -109,33 +109,44 @@ class ConfigReader():
 			self._cfg = ConfigReaderUnshared(**self._cfg_share_kwargs)

 	def sections(self):
-		if self._cfg is not None:
+		try:
 			return self._cfg.sections()
-		return []
+		except AttributeError:
+			return []

 	def has_section(self, sec):
-		if self._cfg is not None:
+		try:
 			return self._cfg.has_section(sec)
-		return False
+		except AttributeError:
+			return False

-	def merge_section(self, *args, **kwargs):
-		if self._cfg is not None:
-			return self._cfg.merge_section(*args, **kwargs)
+	def merge_section(self, section, *args, **kwargs):
+		try:
+			return self._cfg.merge_section(section, *args, **kwargs)
+		except AttributeError:
+			raise NoSectionError(section)
+	
+	def options(self, section, withDefault=False):
+		"""Return a list of option names for the given section name.

-	def options(self, *args):
-		if self._cfg is not None:
-			return self._cfg.options(*args)
-		return {}
+		Parameter `withDefault` controls the include of names from section `[DEFAULT]`
+		"""
+		try:
+			return self._cfg.options(section, withDefault)
+		except AttributeError:
+			raise NoSectionError(section)

 	def get(self, sec, opt, raw=False, vars={}):
-		if self._cfg is not None:
+		try:
 			return self._cfg.get(sec, opt, raw=raw, vars=vars)
-		return None
+		except AttributeError:
+			raise NoSectionError(sec)

-	def getOptions(self, *args, **kwargs):
-		if self._cfg is not None:
-			return self._cfg.getOptions(*args, **kwargs)
-		return {}
+	def getOptions(self, section, *args, **kwargs):
+		try:
+			return self._cfg.getOptions(section, *args, **kwargs)
+		except AttributeError:
+			raise NoSectionError(section)


 class ConfigReaderUnshared(SafeConfigParserWithIncludes):
@ -297,23 +308,35 @@ class DefinitionInitConfigReader(ConfigReader):
 			self._create_unshared(self._file)
 		return SafeConfigParserWithIncludes.read(self._cfg, self._file)
 	
-	def getOptions(self, pOpts):
+	def getOptions(self, pOpts, all=False):
 		# overwrite static definition options with init values, supplied as
 		# direct parameters from jail-config via action[xtra1="...", xtra2=...]:
+		if not pOpts:
+			pOpts = dict()
 		if self._initOpts:
-			if not pOpts:
-				pOpts = dict()
 			pOpts = _merge_dicts(pOpts, self._initOpts)
 		self._opts = ConfigReader.getOptions(
 			self, "Definition", self._configOpts, pOpts)
 		self._pOpts = pOpts
 		if self.has_section("Init"):
-			for opt in self.options("Init"):
-				v = self.get("Init", opt)
-				if not opt.startswith('known/') and opt != '__name__':
+			# get only own options (without options from default):
+			getopt = lambda opt: self.get("Init", opt)
+			for opt in self.options("Init", withDefault=False):
+				if opt == '__name__': continue
+				v = None
+				if not opt.startswith('known/'):
+					if v is None: v = getopt(opt)
 					self._initOpts['known/'+opt] = v
-				if not opt in self._initOpts:
+				if opt not in self._initOpts:
+					if v is None: v = getopt(opt)
 					self._initOpts[opt] = v
+		if all and self.has_section("Definition"):
+			# merge with all definition options (and options from default),
+			# bypass already converted option (so merge only new options):
+			for opt in self.options("Definition"):
+				if opt == '__name__' or opt in self._opts: continue
+				self._opts[opt] = self.get("Definition", opt)
+

 	def _convert_to_boolean(self, value):
 		return value.lower() in ("1", "yes", "true", "on")
@ -336,12 +359,12 @@ class DefinitionInitConfigReader(ConfigReader):

 	def getCombined(self, ignore=()):
 		combinedopts = self._opts
-		ignore = set(ignore).copy()
 		if self._initOpts:
-			combinedopts = _merge_dicts(self._opts, self._initOpts)
+			combinedopts = _merge_dicts(combinedopts, self._initOpts)
 		if not len(combinedopts):
 			return {}
 		# ignore conditional options:
+		ignore = set(ignore).copy()
 		for n in combinedopts:
 			cond = SafeConfigParserWithIncludes.CONDITIONAL_RE.match(n)
 			if cond:
--- a/fail2ban/client/fail2banregex.py
+++ b/fail2ban/client/fail2banregex.py
@ -55,11 +55,14 @@ from ..helpers import str2LogLevel, getVerbosityFormat, FormatterWithTraceBack,
 # Gets the instance of the logger.
 logSys = getLogger("fail2ban")

-def debuggexURL(sample, regex, useDns="yes"):
-	q = urllib.urlencode({ 're': Regex._resolveHostTag(regex, useDns=useDns),
-							'str': sample,
-							'flavor': 'python' })
-	return 'https://www.debuggex.com/?' + q
+def debuggexURL(sample, regex, multiline=False, useDns="yes"):
+	args = {
+		're': Regex._resolveHostTag(regex, useDns=useDns),
+		'str': sample,
+		'flavor': 'python'
+	}
+	if multiline: args['flags'] = 'm'
+	return 'https://www.debuggex.com/?' + urllib.urlencode(args)

 def output(args): # pragma: no cover (overriden in test-cases)
 	print(args)
@ -400,6 +403,7 @@ class Fail2banRegex(object):
 		fullBuffer = len(orgLineBuffer) >= self._filter.getMaxLines()
 		try:
 			ret = self._filter.processLine(line, date)
+			lines = []
 			line = self._filter.processedLine()
 			for match in ret:
 				# Append True/False flag depending if line was matched by
@ -422,9 +426,17 @@ class Fail2banRegex(object):
 								"".join(bufLine[::2])))
 				except ValueError:
 					pass
-				else:
-					self._line_stats.matched += 1
-					self._line_stats.missed -= 1
+				# if buffering - add also another lines from match:
+				if self._print_all_matched:
+					if not self._debuggex:
+						self._line_stats.matched_lines.append("".join(bufLine))
+					else:
+						lines.append(bufLine[0] + bufLine[2])
+				self._line_stats.matched += 1
+				self._line_stats.missed -= 1
+		if lines: # pre-lines parsed in multiline mode (buffering)
+			lines.append(line)
+			line = "\n".join(lines)
 		return line, ret

 	def process(self, test_lines):
@ -472,6 +484,7 @@ class Fail2banRegex(object):
 		assert(self._line_stats.missed == lstats.tested - (lstats.matched + lstats.ignored))
 		lines = lstats[ltype]
 		l = lstats[ltype + '_lines']
+		multiline = self._filter.getMaxLines() > 1
 		if lines:
 			header = "%s line(s):" % (ltype.capitalize(),)
 			if self._debuggex:
@ -485,7 +498,8 @@ class Fail2banRegex(object):
 					for arg in [l, regexlist]:
 						ans = [ x + [y] for x in ans for y in arg ]
 					b = map(lambda a: a[0] +  ' | ' + a[1].getFailRegex() + ' |  ' + 
-						debuggexURL(self.encode_line(a[0]), a[1].getFailRegex(), self._opts.usedns), ans)
+						debuggexURL(self.encode_line(a[0]), a[1].getFailRegex(), 
+							multiline, self._opts.usedns), ans)
 					pprint_list([x.rstrip() for x in b], header)
 				else:
 					output( "%s too many to print.  Use --print-all-%s " \
@ -599,8 +613,19 @@ class Fail2banRegex(object):
 			output( "Use    journal match : %s" % " ".join(journalmatch) )
 			test_lines = journal_lines_gen(flt, myjournal)
 		else:
-			output( "Use      single line : %s" % shortstr(cmd_log) )
-			test_lines = [ cmd_log ]
+			# if single line parsing (without buffering)
+			if self._filter.getMaxLines() <= 1:
+				output( "Use      single line : %s" % shortstr(cmd_log.replace("\n", r"\n")) )
+				test_lines = [ cmd_log ]
+			else: # multi line parsing (with buffering)
+				test_lines = cmd_log.split("\n")
+				output( "Use      multi line : %s line(s)" % len(test_lines) )
+				for i, l in enumerate(test_lines):
+					if i >= 5:
+						output( "| ..." ); break
+					output( "| %2.2s: %s" % (i+1, shortstr(l)) )
+				output( "`-" )
+			
 		output( "" )

 		self.process(test_lines)
--- a/fail2ban/client/jailreader.py
+++ b/fail2ban/client/jailreader.py
@ -117,6 +117,7 @@ class JailReader(ConfigReader):
 				["string", "failregex", None],
 				["string", "ignoreregex", None],
 				["string", "ignorecommand", None],
+				["bool",   "ignoreself", None],
 				["string", "ignoreip", None],
 				["string", "filter", ""],
 				["string", "datepattern", None],
@ -146,11 +147,11 @@ class JailReader(ConfigReader):
 					filterName, self.__name, filterOpt, 
 					share_config=self.share_config, basedir=self.getBaseDir())
 				ret = self.__filter.read()
-				# merge options from filter as 'known/...':
-				self.__filter.getOptions(self.__opts)
-				ConfigReader.merge_section(self, self.__name, self.__filter.getCombined(), 'known/')
 				if not ret:
 					raise JailDefError("Unable to read the filter %r" % filterName)
+				# merge options from filter as 'known/...' (all options unfiltered):
+				self.__filter.getOptions(self.__opts, all=True)
+				ConfigReader.merge_section(self, self.__name, self.__filter.getCombined(), 'known/')
 			else:
 				self.__filter = None
 				logSys.warning("No filter set for jail %s" % self.__name)
@ -227,8 +228,8 @@ class JailReader(ConfigReader):
 		if self.__filter:
 			stream.extend(self.__filter.convert())
 		for opt, value in self.__opts.iteritems():
-			if opt == "logpath" and	\
-					not self.__opts.get('backend', None).startswith("systemd"):
+			if opt == "logpath":
+				if self.__opts.get('backend', None).startswith("systemd"): continue
 				found_files = 0
 				for path in value.split("\n"):
 					path = path.rsplit(" ", 1)
--- a/fail2ban/protocol.py
+++ b/fail2ban/protocol.py
@ -81,6 +81,7 @@ protocol = [
 ["status <JAIL> [FLAVOR]", "gets the current status of <JAIL>, with optional flavor or extended info"],
 ['', "JAIL CONFIGURATION", ""],
 ["set <JAIL> idle on|off", "sets the idle state of <JAIL>"], 
+["set <JAIL> ignoreself true|false", "allows the ignoring of own IP addresses"], 
 ["set <JAIL> addignoreip <IP>", "adds <IP> to the ignore list of <JAIL>"], 
 ["set <JAIL> delignoreip <IP>", "removes <IP> from the ignore list of <JAIL>"], 
 ["set <JAIL> addlogpath <FILE> ['tail']", "adds <FILE> to the monitoring list of <JAIL>, optionally starting at the 'tail' of the file (default 'head')."], 
@ -117,6 +118,7 @@ protocol = [
 ["get <JAIL> logpath", "gets the list of the monitored files for <JAIL>"],
 ["get <JAIL> logencoding", "gets the encoding of the log files for <JAIL>"],
 ["get <JAIL> journalmatch", "gets the journal filter match for <JAIL>"],
+["get <JAIL> ignoreself", "gets the current value of the ignoring the own IP addresses"],
 ["get <JAIL> ignoreip", "gets the list of ignored IP addresses for <JAIL>"],
 ["get <JAIL> ignorecommand", "gets ignorecommand of <JAIL>"],
 ["get <JAIL> failregex", "gets the list of regular expressions which matches the failures for <JAIL>"],
--- a/fail2ban/server/action.py
+++ b/fail2ban/server/action.py
@ -50,6 +50,8 @@ allowed_ipv6 = True
 # capture groups from filter for map to ticket data:
 FCUSTAG_CRE = re.compile(r'<F-([A-Z0-9_\-]+)>'); # currently uppercase only

+CONDITIONAL_FAM_RE = re.compile(r"^(\w+)\?(family)=")
+
 # New line, space
 ADD_REPL_TAGS = {
  "br": "\n", 
@ -201,17 +203,17 @@ class ActionBase(object):
 		self._name = name
 		self._logSys = getLogger("fail2ban.%s" % self.__class__.__name__)

-	def start(self):
+	def start(self): # pragma: no cover - abstract
 		"""Executed when the jail/action is started.
 		"""
 		pass

-	def stop(self):
+	def stop(self): # pragma: no cover - abstract
 		"""Executed when the jail/action is stopped.
 		"""
 		pass

-	def ban(self, aInfo):
+	def ban(self, aInfo): # pragma: no cover - abstract
 		"""Executed when a ban occurs.

 		Parameters
@ -222,7 +224,7 @@ class ActionBase(object):
 		"""
 		pass

-	def unban(self, aInfo):
+	def unban(self, aInfo): # pragma: no cover - abstract
 		"""Executed when a ban expires.

 		Parameters
@ -279,6 +281,8 @@ class CommandAction(ActionBase):
 			self.actioncheck = ''
 			## Command executed in order to restore sane environment in error case.
 			self.actionrepair = ''
+			## Command executed in order to flush all bans at once (e. g. by stop/shutdown the system).
+			self.actionflush = ''
 			## Command executed in order to stop the system.
 			self.actionstop = ''
 			## Command executed in case of reloading action.
@ -290,6 +294,7 @@ class CommandAction(ActionBase):
 		super(CommandAction, self).__init__(jail, name)
 		self.__init = 1
 		self.__properties = None
+		self.__started = {}
 		self.__substCache = {}
 		self.clearAllParams()
 		self._logSys.debug("Created %s" % self.__class__)
@ -342,7 +347,11 @@ class CommandAction(ActionBase):
 	def _substCache(self):
 		return self.__substCache

-	def _executeOperation(self, tag, operation):
+	def _getOperation(self, tag, family):
+		return self.replaceTag(tag, self._properties,
+			conditional=('family=' + family), cache=self.__substCache)
+
+	def _executeOperation(self, tag, operation, family=[]):
 		"""Executes the operation commands (like "actionstart", "actionstop", etc).

 		Replace the tags in the action command with actions properties
@ -352,14 +361,14 @@ class CommandAction(ActionBase):
 		res = True
 		try:
 			# common (resp. ipv4):
-			startCmd = self.replaceTag(tag, self._properties, 
-				conditional='family=inet4', cache=self.__substCache)
-			if startCmd:
-				res &= self.executeCmd(startCmd, self.timeout)
+			startCmd = None
+			if not family or 'inet4' in family:
+				startCmd = self._getOperation(tag, 'inet4')
+				if startCmd:
+					res &= self.executeCmd(startCmd, self.timeout)
 			# start ipv6 actions if available:
-			if allowed_ipv6:
-				startCmd6 = self.replaceTag(tag, self._properties, 
-					conditional='family=inet6', cache=self.__substCache)
+			if allowed_ipv6 and (not family or 'inet6' in family):
+				startCmd6 = self._getOperation(tag, 'inet6')
 				if startCmd6 and startCmd6 != startCmd:
 					res &= self.executeCmd(startCmd6, self.timeout)
 			if not res:
@ -367,13 +376,34 @@ class CommandAction(ActionBase):
 		except ValueError as e:
 			raise RuntimeError("Error %s action %s/%s: %r" % (operation, self._jail, self._name, e))

-	def start(self):
+	COND_FAMILIES = {'inet4':1, 'inet6':1}
+
+	@property
+	def _startOnDemand(self):
+		"""Checks the action depends on family (conditional)"""
+		v = self._properties.get('actionstart_on_demand')
+		if v is None:
+			v = False
+			for n in self._properties:
+				if CONDITIONAL_FAM_RE.match(n):
+					v = True
+					break
+		self._properties['actionstart_on_demand'] = v
+		return v
+
+	def start(self, family=[]):
 		"""Executes the "actionstart" command.

 		Replace the tags in the action command with actions properties
 		and executes the resulting command.
 		"""
-		return self._executeOperation('<actionstart>', 'starting')
+		if not family:
+			# check the action depends on family (conditional):
+			if self._startOnDemand:
+				return True
+		elif self.__started.get(family): # pragma: no cover - normally unreachable
+			return True
+		return self._executeOperation('<actionstart>', 'starting', family=family)

 	def ban(self, aInfo):
 		"""Executes the "actionban" command.
@ -387,6 +417,20 @@ class CommandAction(ActionBase):
 			Dictionary which includes information in relation to
 			the ban.
 		"""
+		# if we should start the action on demand (conditional by family):
+		if self._startOnDemand:
+			family = aInfo.get('family')
+			if not self.__started.get(family):
+				self.start(family)
+				self.__started[family] = 1
+				# mark also another families as "started" (-1), if they are equal
+				# (on demand, but the same for ipv4 and ipv6):
+				cmd = self._getOperation('<actionstart>', family)
+				for f in CommandAction.COND_FAMILIES:
+					if f != family and not self.__started.get(f):
+						if cmd == self._getOperation('<actionstart>', f):
+							self.__started[f] = -1
+		# ban:
 		if not self._processCmd('<actionban>', aInfo):
 			raise RuntimeError("Error banning %(ip)s" % aInfo)

@ -405,13 +449,41 @@ class CommandAction(ActionBase):
 		if not self._processCmd('<actionunban>', aInfo):
 			raise RuntimeError("Error unbanning %(ip)s" % aInfo)

+	def flush(self):
+		"""Executes the "actionflush" command.
+		
+		Command executed in order to flush all bans at once (e. g. by stop/shutdown 
+		the system), instead of unbunning of each single ticket.
+
+		Replaces the tags in the action command with actions properties
+		and executes the resulting command.
+		"""
+		family = []
+		# cumulate started families, if started on demand (conditional):
+		if self._startOnDemand:
+			for f in CommandAction.COND_FAMILIES:
+				if self.__started.get(f) == 1: # only real started:
+					family.append(f)
+			# if no started (on demand) actions:
+			if not family: return True
+		return self._executeOperation('<actionflush>', 'flushing', family=family)
+
 	def stop(self):
 		"""Executes the "actionstop" command.

 		Replaces the tags in the action command with actions properties
 		and executes the resulting command.
 		"""
-		return self._executeOperation('<actionstop>', 'stopping')
+		family = []
+		# cumulate started families, if started on demand (conditional):
+		if self._startOnDemand:
+			for f in CommandAction.COND_FAMILIES:
+				if self.__started.get(f) == 1: # only real started:
+					family.append(f)
+					self.__started[f] = 0
+			# if no started (on demand) actions:
+			if not family: return True
+		return self._executeOperation('<actionstop>', 'stopping', family=family)

 	def reload(self, **kwargs):
 		"""Executes the "actionreload" command.
@ -453,7 +525,7 @@ class CommandAction(ActionBase):
 		return value

 	@classmethod
-	def replaceTag(cls, query, aInfo, conditional='', cache=None, substRec=True):
+	def replaceTag(cls, query, aInfo, conditional='', cache=None):
 		"""Replaces tags in `query` with property values.

 		Parameters
@ -481,9 +553,8 @@ class CommandAction(ActionBase):
 		# **Important**: don't replace if calling map - contains dynamic values only,
 		# no recursive tags, otherwise may be vulnerable on foreign user-input:
 		noRecRepl = isinstance(aInfo, CallingMap)
-		if noRecRepl:
-			subInfo = aInfo
-		else:
+		subInfo = aInfo
+		if not noRecRepl:
 			# substitute tags recursive (and cache if possible),
 			# first try get cached tags dictionary:
 			subInfo = csubkey = None
@ -534,13 +605,86 @@ class CommandAction(ActionBase):
 					"unexpected too long replacement interpolation, "
 					"possible self referencing definitions in query: %s" % (query,))

-
 		# cache if possible:
 		if cache is not None:
 			cache[ckey] = value
 		#
 		return value

+	ESCAPE_CRE = re.compile(r"""[\\#&;`|*?~<>\^\(\)\[\]{}$'"\n\r]""")
+	ESCAPE_VN_CRE = re.compile(r"\W")
+
+	@classmethod
+	def replaceDynamicTags(cls, realCmd, aInfo):
+		"""Replaces dynamical tags in `query` with property values.
+
+		**Important**
+		-------------
+		Because this tags are dynamic resp. foreign (user) input:
+		  - values should be escaped (using "escape" as shell variable)
+		  - no recursive substitution (no interpolation for <a<b>>)
+		  - don't use cache
+
+		Parameters
+		----------
+		query : str
+			String with tags.
+		aInfo : dict
+			Tags(keys) and associated values for substitution in query.
+
+		Returns
+		-------
+		str
+			shell script as string or array with tags replaced (direct or as variables).
+		"""
+		# array for escaped vars:
+		varsDict = dict()
+
+		def escapeVal(tag, value):
+			# if the value should be escaped:
+			if cls.ESCAPE_CRE.search(value):
+				# That one needs to be escaped since its content is
+				# out of our control
+				tag = 'f2bV_%s' % cls.ESCAPE_VN_CRE.sub('_', tag)
+				varsDict[tag] = value # add variable
+				value = '$'+tag	# replacement as variable
+			# replacement for tag:
+			return value
+
+		# substitution callable, used by interpolation of each tag
+		def substVal(m):
+			tag = m.group(1)			# tagname from match
+			try:
+				value = aInfo[tag]
+			except KeyError:
+				# fallback (no or default replacement)
+				return ADD_REPL_TAGS.get(tag, m.group())
+			value = str(value)		# assure string
+			# replacement for tag:
+			return escapeVal(tag, value)
+		
+		# Replace normally properties of aInfo non-recursive:
+		realCmd = TAG_CRE.sub(substVal, realCmd)
+
+		# Replace ticket options (filter capture groups) non-recursive:
+		if '<' in realCmd:
+			tickData = aInfo.get("F-*")
+			if not tickData: tickData = {}
+			def substTag(m):
+				tag = mapTag2Opt(m.groups()[0])
+				try:
+					value = str(tickData[tag])
+				except KeyError:
+					return ""
+				return escapeVal("F_"+tag, value)
+			
+			realCmd = FCUSTAG_CRE.sub(substTag, realCmd)
+
+		# build command corresponding "escaped" variables:
+		if varsDict:
+			realCmd = Utils.buildShellCmd(realCmd, varsDict)
+		return realCmd
+
 	def _processCmd(self, cmd, aInfo=None, conditional=''):
 		"""Executes a command with preliminary checks and substitutions.

@ -605,21 +749,9 @@ class CommandAction(ActionBase):
 		realCmd = self.replaceTag(cmd, self._properties, 
 			conditional=conditional, cache=self.__substCache)

-		# Replace dynamical tags (don't use cache here)
+		# Replace dynamical tags, important - don't cache, no recursion and auto-escape here
 		if aInfo is not None:
-			realCmd = self.replaceTag(realCmd, aInfo, conditional=conditional)
-			# Replace ticket options (filter capture groups) non-recursive:
-			if '<' in realCmd:
-				tickData = aInfo.get("F-*")
-				if not tickData: tickData = {}
-				def substTag(m):
-					tn = mapTag2Opt(m.groups()[0])
-					try:
-						return str(tickData[tn])
-					except KeyError:
-						return ""
-				
-				realCmd = FCUSTAG_CRE.sub(substTag, realCmd)
+			realCmd = self.replaceDynamicTags(realCmd, aInfo)
 		else:
 			realCmd = cmd

--- a/fail2ban/server/actions.py
+++ b/fail2ban/server/actions.py
@ -35,10 +35,11 @@ except ImportError:
 	OrderedDict = dict

 from .banmanager import BanManager
-from .observer import Observers
+from .ipdns import DNSUtils
 from .jailthread import JailThread
 from .action import ActionBase, CommandAction, CallingMap
 from .mytime import MyTime
+from .observer import Observers
 from .utils import Utils
 from ..helpers import getLogger

@ -291,6 +292,7 @@ class Actions(JailThread, Mapping):

 		AI_DICT = {
 			"ip":				lambda self: self.__ticket.getIP(),
+			"family":   lambda self: self['ip'].familyStr,
 			"ip-rev":		lambda self: self['ip'].getPTR(''),
 			"ip-host":	lambda self: self['ip'].getHost(),
 			"fid":			lambda self: self.__ticket.getID(),
@ -306,6 +308,9 @@ class Actions(JailThread, Mapping):
 			"ipjailmatches":	lambda self: "\n".join(self._mi4ip().getMatches()),
 			"ipfailures":			lambda self: self._mi4ip(True).getAttempt(),
 			"ipjailfailures":	lambda self: self._mi4ip().getAttempt(),
+			# system-information:
+			"fq-hostname":	lambda self: DNSUtils.getHostname(fqdn=True),
+			"sh-hostname":	lambda self: DNSUtils.getHostname(fqdn=False)
 		}

 		__slots__ = CallingMap.__slots__ + ('__ticket', '__jail', '__mi4ip')
@ -462,25 +467,37 @@ class Actions(JailThread, Mapping):
 		If actions specified, don't flush list - just execute unban for 
 		given actions (reload, obsolete resp. removed actions).
 		"""
+		log = True
 		if actions is None:
 			logSys.debug("Flush ban list")
 			lst = self.__banManager.flushBanList()
 		else:
+			log = False # don't log "[jail] Unban ..." if removing actions only.
 			lst = iter(self.__banManager)
 		cnt = 0
+		# first we'll execute flush for actions supporting this operation:
+		unbactions = {}
+		for name, action in (actions if actions is not None else self._actions).iteritems():
+			if hasattr(action, 'flush') and action.actionflush:
+				logSys.notice("[%s] Flush ticket(s) with %s", self._jail.name, name)
+				action.flush()
+			else:
+				unbactions[name] = action
+		actions = unbactions
+		# unban each ticket with non-flasheable actions:
 		for ticket in lst:
 			# delete ip from database also:
 			if db and self._jail.database is not None:
 				ip = str(ticket.getIP())
 				self._jail.database.delBan(self._jail, ip)
 			# unban ip:
-			self.__unBan(ticket, actions=actions)
+			self.__unBan(ticket, actions=actions, log=log)
 			cnt += 1
 		logSys.debug("Unbanned %s, %s ticket(s) in %r", 
 			cnt, self.__banManager.size(), self._jail.name)
 		return cnt

-	def __unBan(self, ticket, actions=None):
+	def __unBan(self, ticket, actions=None, log=True):
 		"""Unbans host corresponding to the ticket.

 		Executes the actions in order to unban the host given in the
@ -497,7 +514,7 @@ class Actions(JailThread, Mapping):
 			unbactions = actions
 		ip = ticket.getIP()
 		aInfo = self.__getActionInfo(ticket)
-		if actions is None:
+		if log:
 			logSys.notice("[%s] Unban %s", self._jail.name, aInfo["ip"])
 		for name, action in unbactions.iteritems():
 			try:
--- a/fail2ban/server/failregex.py
+++ b/fail2ban/server/failregex.py
@ -103,20 +103,16 @@ class Regex:
 	# avoid construction of invalid object.
 	# @param value the regular expression
 	
-	def __init__(self, regex, **kwargs):
+	def __init__(self, regex, multiline=False, **kwargs):
 		self._matchCache = None
 		# Perform shortcuts expansions.
-		# Resolve "<HOST>" tag using default regular expression for host:
+		# Replace standard f2b-tags (like "<HOST>", etc) using default regular expressions:
 		regex = Regex._resolveHostTag(regex, **kwargs)
-		# Replace "<SKIPLINES>" with regular expression for multiple lines.
-		regexSplit = regex.split("<SKIPLINES>")
-		regex = regexSplit[0]
-		for n, regexLine in enumerate(regexSplit[1:]):
-			regex += "\n(?P<skiplines%i>(?:(.*\n)*?))" % n + regexLine
+		#
 		if regex.lstrip() == '':
 			raise RegexException("Cannot add empty regex")
 		try:
-			self._regexObj = re.compile(regex, re.MULTILINE)
+			self._regexObj = re.compile(regex, re.MULTILINE if multiline else 0)
 			self._regex = regex
 		except sre_constants.error:
 			raise RegexException("Unable to compile regular expression '%s'" %
@ -135,6 +131,9 @@ class Regex:
 	def _resolveHostTag(regex, useDns="yes"):

 		openTags = dict()
+		props = {
+			'nl': 0, # new lines counter by <SKIPLINES> tag;
+		}
 		# tag interpolation callable:
 		def substTag(m):
 			tag = m.group()
@ -142,6 +141,11 @@ class Regex:
 			# 3 groups instead of <HOST> - separated ipv4, ipv6 and host (dns)
 			if tn == "HOST":
 				return R_HOST[RI_HOST if useDns not in ("no",) else RI_ADDR]
+			# replace "<SKIPLINES>" with regular expression for multiple lines (by buffering with maxlines)
+			if tn == "SKIPLINES":
+				nl = props['nl']
+				props['nl'] = nl + 1
+				return r"\n(?P<skiplines%i>(?:(?:.*\n)*?))" % (nl,)
 			# static replacement from RH4TAG:
 			try:
 				return RH4TAG[tn]
--- a/fail2ban/server/filter.py
+++ b/fail2ban/server/filter.py
@ -77,6 +77,8 @@ class Filter(JailThread):
 		self.setUseDns(useDns)
 		## The amount of time to look back.
 		self.__findTime = 600
+		## Ignore own IPs flag:
+		self.__ignoreSelf = True
 		## The ignore IP list.
 		self.__ignoreIpList = []
 		## Size of line buffer
@ -160,13 +162,11 @@ class Filter(JailThread):
 	# @param value the regular expression

 	def addFailRegex(self, value):
+		multiLine = self.getMaxLines() > 1
 		try:
-			regex = FailRegex(value, prefRegex=self.__prefRegex, useDns=self.__useDns)
+			regex = FailRegex(value, prefRegex=self.__prefRegex, multiline=multiLine,
+				useDns=self.__useDns)
 			self.__failRegex.append(regex)
-			if "\n" in regex.getRegex() and not self.getMaxLines() > 1:
-				logSys.warning(
-					"Mutliline regex set for jail %r "
-					"but maxlines not greater than 1", self.jailName)
 		except RegexException as e:
 			logSys.error(e)
 			raise e
@ -184,15 +184,12 @@ class Filter(JailThread):
 						 "valid", index)

 	##
-	# Get the regular expression which matches the failure.
+	# Get the regular expressions as list.
 	#
-	# @return the regular expression
+	# @return the regular expression list

 	def getFailRegex(self):
-		failRegex = list()
-		for regex in self.__failRegex:
-			failRegex.append(regex.getRegex())
-		return failRegex
+		return [regex.getRegex() for regex in self.__failRegex]

 	##
 	# Add the regular expression which matches the failure.
@ -417,6 +414,17 @@ class Filter(JailThread):

 		return ip

+	##
+	# Ignore own IP/DNS.
+	#
+	@property
+	def ignoreSelf(self):
+		return self.__ignoreSelf
+
+	@ignoreSelf.setter
+	def ignoreSelf(self, value):
+		self.__ignoreSelf = value
+
 	##
 	# Add an IP/DNS to the ignore list.
 	#
@ -462,6 +470,11 @@ class Filter(JailThread):
 	def inIgnoreIPList(self, ip, log_ignore=False):
 		if not isinstance(ip, IPAddr):
 			ip = IPAddr(ip)
+
+		# check own IPs should be ignored and 'ip' is self IP:
+		if self.__ignoreSelf and ip in DNSUtils.getSelfIPs():
+			return True
+
 		for net in self.__ignoreIpList:
 			# check if the IP is covered by ignore IP
 			if ip.isInNet(net):
@ -557,24 +570,29 @@ class Filter(JailThread):

 	def _mergeFailure(self, mlfid, fail, failRegex):
 		mlfidFail = self.mlfidCache.get(mlfid) if self.__mlfidCache else None
+		# if multi-line failure id (connection id) known:
 		if mlfidFail:
 			mlfidGroups = mlfidFail[1]
-			# if current line not failure, but previous was failure:
-			if fail.get('nofail') and not mlfidGroups.get('nofail'):
-				del fail['nofail'] # remove nofail flag - was already market as failure
-				self.mlfidCache.unset(mlfid) # remove cache entry
-			# if current line is failure, but previous was not:
-			elif not fail.get('nofail') and mlfidGroups.get('nofail'):
-				del mlfidGroups['nofail'] # remove nofail flag
-				self.mlfidCache.unset(mlfid) # remove cache entry
+			# update - if not forget (disconnect/reset):
+			if not fail.get('mlfforget'):
+				mlfidGroups.update(fail)
+			else:
+				self.mlfidCache.unset(mlfid) # remove cached entry
+			# merge with previous info:
 			fail2 = mlfidGroups.copy()
 			fail2.update(fail)
+			if not fail.get('nofail'): # be sure we've correct current state
+				try:
+					del fail2['nofail']
+				except KeyError:
+					pass
 			fail2["matches"] = fail.get("matches", []) + failRegex.getMatchedTupleLines()
 			fail = fail2
-		elif fail.get('nofail'):
-			fail["matches"] = failRegex.getMatchedTupleLines()
+		elif not fail.get('mlfforget'):
 			mlfidFail = [self.__lastDate, fail]
 			self.mlfidCache.set(mlfid, mlfidFail)
+			if fail.get('nofail'):
+				fail["matches"] = failRegex.getMatchedTupleLines()
 		return fail


@ -690,6 +708,11 @@ class Filter(JailThread):
 				mlfid = fail.get('mlfid')
 				if mlfid is not None:
 					fail = self._mergeFailure(mlfid, fail, failRegex)
+					# bypass if no-failure case:
+					if fail.get('nofail'):
+						logSys.log(7, "Nofail by mlfid %r in regex %s: %s",
+							mlfid, failRegexIndex, fail.get('mlfforget', "waiting for failure"))
+						if not self.checkAllRegex: return failList
 				else:
 					# matched lines:
 					fail["matches"] = fail.get("matches", []) + failRegex.getMatchedTupleLines()
@ -709,18 +732,16 @@ class Filter(JailThread):
 					host = fail.get('dns')
 					if host is None:
 						# first try to check we have mlfid case (cache connection id):
-						if fid is None:
-							if mlfid:
-								fail = self._mergeFailure(mlfid, fail, failRegex)
-							else:
+						if fid is None and mlfid is None:
 								# if no failure-id also (obscure case, wrong regex), throw error inside getFailID:
 								fid = failRegex.getFailID()
 						host = fid
 						cidr = IPAddr.CIDR_RAW
 				# if mlfid case (not failure):
 				if host is None:
-					if not self.checkAllRegex: # or fail.get('nofail'):
-					 	return failList
+					logSys.log(7, "No failure-id by mlfid %r in regex %s: %s",
+						mlfid, failRegexIndex, fail.get('mlfforget', "waiting for identifier"))
+					if not self.checkAllRegex: return failList
 					ips = [None]
 				# if raw - add single ip or failure-id,
 				# otherwise expand host to multiple ips using dns (or ignore it if not valid):
@ -878,7 +899,8 @@ class FileFilter(Filter):
 			# see http://python.org/dev/peps/pep-3151/
 			except IOError as e:
 				logSys.error("Unable to open %s", filename)
-				logSys.exception(e)
+				if e.errno != 2: # errno.ENOENT
+					logSys.exception(e)
 				return False
 			except OSError as e: # pragma: no cover - requires race condition to tigger this
 				logSys.error("Error opening %s", filename)
@ -1102,7 +1124,7 @@ class FileContainer:
 		## sys.stdout.flush()
 		# Compare hash and inode
 		if self.__hash != myHash or self.__ino != stats.st_ino:
-			logSys.info("Log rotation detected for %s", self.__filename)
+			logSys.log(logging.MSG, "Log rotation detected for %s", self.__filename)
 			self.__hash = myHash
 			self.__ino = stats.st_ino
 			self.__pos = 0
--- a/fail2ban/server/filtergamin.py
+++ b/fail2ban/server/filtergamin.py
@ -143,6 +143,8 @@ class FilterGamin(FileFilter):
 	# Desallocates the resources used by Gamin.

 	def __cleanup(self):
+		if not self.monitor:
+			return
 		for filename in self.getLogPaths():
 			self.monitor.stop_watch(filename)
 		self.monitor = None
--- a/fail2ban/server/filterpyinotify.py
+++ b/fail2ban/server/filterpyinotify.py
@ -25,19 +25,20 @@ __license__ = "GPL"

 import logging
 from distutils.version import LooseVersion
+import os
 from os.path import dirname, sep as pathsep

 import pyinotify

 from .failmanager import FailManagerEmpty
 from .filter import FileFilter
-from .mytime import MyTime
+from .mytime import MyTime, time
 from .utils import Utils
 from ..helpers import getLogger


 if not hasattr(pyinotify, '__version__') \
-  or LooseVersion(pyinotify.__version__) < '0.8.3':
+  or LooseVersion(pyinotify.__version__) < '0.8.3': # pragma: no cover
  raise ImportError("Fail2Ban requires pyinotify >= 0.8.3")

 # Verify that pyinotify is functional on this system
@ -45,13 +46,18 @@ if not hasattr(pyinotify, '__version__') \
 try:
 	manager = pyinotify.WatchManager()
 	del manager
-except Exception as e:
+except Exception as e: # pragma: no cover
 	raise ImportError("Pyinotify is probably not functional on this system: %s"
 					  % str(e))

 # Gets the instance of the logger.
 logSys = getLogger(__name__)

+# Override pyinotify default logger/init-handler:
+def _pyinotify_logger_init(): # pragma: no cover
+	return logSys
+pyinotify._logger_init = _pyinotify_logger_init
+pyinotify.log = logSys

 ##
 # Log reader class.
@ -72,30 +78,57 @@ class FilterPyinotify(FileFilter):
 		self.__modified = False
 		# Pyinotify watch manager
 		self.__monitor = pyinotify.WatchManager()
-		self.__watches = dict()
+		self.__watchFiles = dict()
+		self.__watchDirs = dict()
+		self.__pending = dict()
+		self.__pendingChkTime = 0
+		self.__pendingNextTime = 0
 		logSys.debug("Created FilterPyinotify")

 	def callback(self, event, origin=''):
 		logSys.log(7, "[%s] %sCallback for Event: %s", self.jailName, origin, event)
 		path = event.pathname
+		# check watching of this path:
+		isWF = False
+		isWD = path in self.__watchDirs
+		if not isWD and path in self.__watchFiles:
+			isWF = True
+		assumeNoDir = False
 		if event.mask & ( pyinotify.IN_CREATE | pyinotify.IN_MOVED_TO ):
 			# skip directories altogether
 			if event.mask & pyinotify.IN_ISDIR:
 				logSys.debug("Ignoring creation of directory %s", path)
 				return
 			# check if that is a file we care about
-			if not path in self.__watches:
+			if not isWF:
 				logSys.debug("Ignoring creation of %s we do not monitor", path)
 				return
-			else:
-				# we need to substitute the watcher with a new one, so first
-				# remove old one
-				self._delFileWatcher(path)
-				# place a new one
-				self._addFileWatcher(path)
+			self._refreshWatcher(path)
+		elif event.mask & (pyinotify.IN_IGNORED | pyinotify.IN_MOVE_SELF | pyinotify.IN_DELETE_SELF):
+			assumeNoDir = event.mask & (pyinotify.IN_MOVE_SELF | pyinotify.IN_DELETE_SELF)
+			# fix pyinotify behavior with '-unknown-path' (if target not watched also):
+			if (assumeNoDir and 
+					path.endswith('-unknown-path') and not isWF and not isWD
+			):
+				path = path[:-len('-unknown-path')]
+				isWD = path in self.__watchDirs
+			# watch was removed for some reasons (log-rotate?):
+			if isWD and (assumeNoDir or not os.path.isdir(path)):
+				self._addPending(path, event, isDir=True)
+			elif not isWF:
+				for logpath in self.__watchDirs:
+					if logpath.startswith(path + pathsep) and (assumeNoDir or not os.path.isdir(logpath)):
+						self._addPending(logpath, event, isDir=True)
+		if isWF and not os.path.isfile(path):
+			self._addPending(path, event)
+			return
 		# do nothing if idle:
 		if self.idle:
 			return
+		# be sure we process a file:
+		if not isWF:
+			logSys.debug("Ignoring event (%s) of %s we do not monitor", event.maskname, path)
+			return
 		self._process_file(path)

 	def _process_file(self, path):
@ -104,23 +137,97 @@ class FilterPyinotify(FileFilter):
 		TODO -- RF:
 		this is a common logic and must be shared/provided by FileFilter
 		"""
-		self.getFailures(path)
+		if not self.idle:
+			self.getFailures(path)
+			try:
+				while True:
+					ticket = self.failManager.toBan()
+					self.jail.putFailTicket(ticket)
+			except FailManagerEmpty:
+				self.failManager.cleanup(MyTime.time())
+			self.__modified = False
+
+	def _addPending(self, path, reason, isDir=False):
+		if path not in self.__pending:
+			self.__pending[path] = [Utils.DEFAULT_SLEEP_INTERVAL, isDir];
+			self.__pendingNextTime = 0
+			if isinstance(reason, pyinotify.Event):
+				reason = [reason.maskname, reason.pathname]
+			logSys.log(logging.MSG, "Log absence detected (possibly rotation) for %s, reason: %s of %s",
+				path, *reason)
+
+	def _delPending(self, path):
 		try:
-			while True:
-				ticket = self.failManager.toBan()
-				self.jail.putFailTicket(ticket)
-		except FailManagerEmpty:
-			self.failManager.cleanup(MyTime.time())
-		self.__modified = False
+			del self.__pending[path]
+		except KeyError: pass
+
+	def _checkPending(self):
+		if not self.__pending:
+			return
+		ntm = time.time()
+		if ntm < self.__pendingNextTime:
+			return
+		found = {}
+		minTime = 60
+		for path, (retardTM, isDir) in self.__pending.iteritems():
+			if ntm - self.__pendingChkTime < retardTM:
+				if minTime > retardTM: minTime = retardTM
+				continue
+			chkpath = os.path.isdir if isDir else os.path.isfile
+			if not chkpath(path): # not found - prolong for next time
+				if retardTM < 60: retardTM *= 2
+				if minTime > retardTM: minTime = retardTM
+				self.__pending[path][0] = retardTM
+				continue
+			logSys.log(logging.MSG, "Log presence detected for %s %s", 
+				"directory" if isDir else "file", path)
+			found[path] = isDir
+		for path in found:
+			try:
+				del self.__pending[path]
+			except KeyError: pass
+		self.__pendingChkTime = time.time()
+		self.__pendingNextTime = self.__pendingChkTime + minTime
+		# process now because we've missed it in monitoring:
+		for path, isDir in found.iteritems():
+			# refresh monitoring of this:
+			self._refreshWatcher(path, isDir=isDir)
+			if isDir:
+				# check all files belong to this dir:
+				for logpath in self.__watchFiles:
+					if logpath.startswith(path + pathsep):
+						# if still no file - add to pending, otherwise refresh and process:
+						if not os.path.isfile(logpath):
+							self._addPending(logpath, ('FROM_PARDIR', path))
+						else:
+							self._refreshWatcher(logpath)
+							self._process_file(logpath)
+			else:
+				# process (possibly no old events for it from watcher):
+				self._process_file(path)
+
+	def _refreshWatcher(self, oldPath, newPath=None, isDir=False):
+		if not newPath: newPath = oldPath
+		# we need to substitute the watcher with a new one, so first
+		# remove old one and then place a new one
+		if not isDir:
+			self._delFileWatcher(oldPath)
+			self._addFileWatcher(newPath)
+		else:
+			self._delDirWatcher(oldPath)
+			self._addDirWatcher(newPath)

 	def _addFileWatcher(self, path):
+		# we need to watch also the directory for IN_CREATE
+		self._addDirWatcher(dirname(path))
+		# add file watcher:
 		wd = self.__monitor.add_watch(path, pyinotify.IN_MODIFY)
-		self.__watches.update(wd)
+		self.__watchFiles.update(wd)
 		logSys.debug("Added file watcher for %s", path)

 	def _delFileWatcher(self, path):
 		try:
-			wdInt = self.__watches.pop(path)
+			wdInt = self.__watchFiles.pop(path)
 			wd = self.__monitor.rm_watch(wdInt)
 			if wd[wdInt]:
 				logSys.debug("Removed file watcher for %s", path)
@ -129,19 +236,30 @@ class FilterPyinotify(FileFilter):
 			pass
 		return False

+	def _addDirWatcher(self, path_dir):
+		# Add watch for the directory:
+		if path_dir not in self.__watchDirs:
+			self.__watchDirs.update(
+				self.__monitor.add_watch(path_dir, pyinotify.IN_CREATE | 
+					pyinotify.IN_MOVED_TO | pyinotify.IN_MOVE_SELF |
+					pyinotify.IN_DELETE_SELF | pyinotify.IN_ISDIR))
+			logSys.debug("Added monitor for the parent directory %s", path_dir)
+
+	def _delDirWatcher(self, path_dir):
+		# Remove watches for the directory:
+		try:
+			wdInt = self.__watchDirs.pop(path_dir)
+			self.__monitor.rm_watch(wdInt)
+		except KeyError: # pragma: no cover
+			pass
+		logSys.debug("Removed monitor for the parent directory %s", path_dir)
+
 	##
 	# Add a log file path
 	#
 	# @param path log file path

 	def _addLogPath(self, path):
-		path_dir = dirname(path)
-		if not (path_dir in self.__watches):
-			# we need to watch also  the directory for IN_CREATE
-			self.__watches.update(
-				self.__monitor.add_watch(path_dir, pyinotify.IN_CREATE | pyinotify.IN_MOVED_TO))
-			logSys.debug("Added monitor for the parent directory %s", path_dir)
-
 		self._addFileWatcher(path)
 		self._process_file(path)

@ -151,40 +269,32 @@ class FilterPyinotify(FileFilter):
 	# @param path the log file to delete

 	def _delLogPath(self, path):
-		if not self._delFileWatcher(path):
+		if not self._delFileWatcher(path): # pragma: no cover
 			logSys.error("Failed to remove watch on path: %s", path)
+		self._delPending(path)

 		path_dir = dirname(path)
-		if not len([k for k in self.__watches
-					if k.startswith(path_dir + pathsep)]):
+		for k in self.__watchFiles:
+			if k.startswith(path_dir + pathsep):
+				path_dir = None
+				break
+		if path_dir:
 			# Remove watches for the directory
 			# since there is no other monitored file under this directory
-			try:
-				wdInt = self.__watches.pop(path_dir)
-				self.__monitor.rm_watch(wdInt)
-			except KeyError: # pragma: no cover
-				pass
-			logSys.debug("Removed monitor for the parent directory %s", path_dir)
+			self._delDirWatcher(path_dir)
+			self._delPending(path_dir)

 	# pyinotify.ProcessEvent default handler:
 	def __process_default(self, event):
 		try:
 			self.callback(event, origin='Default ')
-		except Exception as e:
+		except Exception as e: # pragma: no cover
 			logSys.error("Error in FilterPyinotify callback: %s",
 				e, exc_info=logSys.getEffectiveLevel() <= logging.DEBUG)
+			# incr common error counter:
+			self.commonError()
 		self.ticks += 1

-	# slow check events while idle:
-	def __check_events(self, *args, **kwargs):
-		if self.idle:
-			if Utils.wait_for(lambda: not self.active or not self.idle,
-				self.sleeptime * 10, self.sleeptime
-			):
-				pass
-		self.ticks += 1
-		return pyinotify.ThreadedNotifier.check_events(self.__notifier, *args, **kwargs)
-
 	##
 	# Main loop.
 	#
@ -195,25 +305,59 @@ class FilterPyinotify(FileFilter):
 		prcevent = pyinotify.ProcessEvent()
 		prcevent.process_default = self.__process_default
 		## timeout for pyinotify must be set in milliseconds (our time values are floats contain seconds)
-		self.__notifier = pyinotify.ThreadedNotifier(self.__monitor,
+		self.__notifier = pyinotify.Notifier(self.__monitor,
 			prcevent, timeout=self.sleeptime * 1000)
-		self.__notifier.check_events = self.__check_events
-		self.__notifier.start()
 		logSys.debug("[%s] filter started (pyinotifier)", self.jailName)
+		while self.active:
+			try:
+
+				# slow check events while idle:
+				if self.idle:
+					if Utils.wait_for(lambda: not self.active or not self.idle,
+						self.sleeptime * 10, self.sleeptime
+					):
+						if not self.active: break
+
+				# default pyinotify handling using Notifier:
+				self.__notifier.process_events()
+				if Utils.wait_for(lambda: not self.active or self.__notifier.check_events(), self.sleeptime):
+					if not self.active: break
+					self.__notifier.read_events()
+
+				# check pending files/dirs (logrotate ready):
+				if not self.idle:
+					self._checkPending()
+
+			except Exception as e: # pragma: no cover
+				if not self.active: # if not active - error by stop...
+					break
+				logSys.error("Caught unhandled exception in main cycle: %r", e,
+					exc_info=logSys.getEffectiveLevel()<=logging.DEBUG)
+				# incr common error counter:
+				self.commonError()
+			
+			self.ticks += 1
+
+		logSys.debug("[%s] filter exited (pyinotifier)", self.jailName)
+		self.__notifier = None
+
 		return True

 	##
 	# Call super.stop() and then stop the 'Notifier'

 	def stop(self):
+		if self.__notifier: # stop the notifier
+			self.__notifier.stop()
+		# stop filter thread:
 		super(FilterPyinotify, self).stop()
-		# Stop the notifier thread
-		self.__notifier.stop()
+		self.join()

 	##
 	# Wait for exit with cleanup.

 	def join(self):
+		self.join = lambda *args: 0
 		self.__cleanup()
 		super(FilterPyinotify, self).join()
 		logSys.debug("[%s] filter terminated (pyinotifier)", self.jailName)
@ -223,6 +367,6 @@ class FilterPyinotify(FileFilter):

 	def __cleanup(self):
 		if self.__notifier:
-			self.__notifier.join()			# to not exit before notifier does
-			self.__notifier = None
-		self.__monitor = None
+			if Utils.wait_for(lambda: not self.__notifier, self.sleeptime * 10):
+				self.__notifier = None
+				self.__monitor = None
--- a/fail2ban/server/ipdns.py
+++ b/fail2ban/server/ipdns.py
@ -118,6 +118,60 @@ class DNSUtils:

 		return ipList

+	@staticmethod
+	def getHostname(fqdn=True):
+		"""Get short hostname or fully-qualified hostname of host self"""
+		# try find cached own hostnames (this tuple-key cannot be used elsewhere):
+		key = ('self','hostname', fqdn)
+		name = DNSUtils.CACHE_ipToName.get(key)
+		# get it using different ways (hostname, fully-qualified or vice versa):
+		if name is None:
+			name = ''
+			for hostname in (
+				(socket.getfqdn, socket.gethostname) if fqdn else (socket.gethostname, socket.getfqdn)
+			):
+				try:
+					name = hostname()
+					break
+				except Exception as e: # pragma: no cover
+					logSys.warning("Retrieving own hostnames failed: %s", e)
+		# cache and return :
+		DNSUtils.CACHE_ipToName.set(key, name)
+		return name
+
+	@staticmethod
+	def getSelfNames():
+		"""Get own host names of self"""
+		# try find cached own hostnames (this tuple-key cannot be used elsewhere):
+		key = ('self','dns')
+		names = DNSUtils.CACHE_ipToName.get(key)
+		# get it using different ways (a set with names of localhost, hostname, fully qualified):
+		if names is None:
+			names = set([
+				'localhost', DNSUtils.getHostname(False), DNSUtils.getHostname(True)
+			]) - set(['']) # getHostname can return ''
+		# cache and return :
+		DNSUtils.CACHE_ipToName.set(key, names)
+		return names
+
+	@staticmethod
+	def getSelfIPs():
+		"""Get own IP addresses of self"""
+		# try find cached own IPs (this tuple-key cannot be used elsewhere):
+		key = ('self','ips')
+		ips = DNSUtils.CACHE_nameToIp.get(key)
+		# get it using different ways (a set with IPs of localhost, hostname, fully qualified):
+		if ips is None:
+			ips = set()
+			for hostname in DNSUtils.getSelfNames():
+				try:
+					ips |= set(DNSUtils.textToIp(hostname, 'yes'))
+				except Exception as e: # pragma: no cover
+					logSys.warning("Retrieving own IPs of %s failed: %s", hostname, e)
+		# cache and return :
+		DNSUtils.CACHE_nameToIp.set(key, ips)
+		return ips
+

 ##
 # Class for IP address handling.
@ -261,6 +315,11 @@ class IPAddr(object):
 	def family(self):
 		return self._family

+	FAM2STR = {socket.AF_INET: 'inet4', socket.AF_INET6: 'inet6'}
+	@property
+	def familyStr(self):
+		return IPAddr.FAM2STR.get(self._family)
+
 	@property
 	def plen(self):
 		return self._plen
--- a/fail2ban/server/server.py
+++ b/fail2ban/server/server.py
@ -331,6 +331,12 @@ class Server:
 		return self.__jails[name].idle
 	
 	# Filter
+	def setIgnoreSelf(self, name, value):
+		self.__jails[name].filter.ignoreSelf = value
+	
+	def getIgnoreSelf(self, name):
+		return self.__jails[name].filter.ignoreSelf
+
 	def addIgnoreIP(self, name, ip):
 		self.__jails[name].filter.addIgnoreIP(ip)
 	
@ -661,9 +667,9 @@ class Server:
 			if self.__syslogSocket == syslogsocket:
 				return True
 			self.__syslogSocket = syslogsocket
-			# Conditionally reload, logtarget depends on socket path when SYSLOG
-			return self.__logTarget != "SYSLOG"\
-				   or self.setLogTarget(self.__logTarget)
+		# Conditionally reload, logtarget depends on socket path when SYSLOG
+		return self.__logTarget != "SYSLOG"\
+			   or self.setLogTarget(self.__logTarget)

 	def getLogTarget(self):
 		with self.__loggingLock:
--- a/fail2ban/server/strptime.py
+++ b/fail2ban/server/strptime.py
@ -95,18 +95,10 @@ def reGroupDictStrptime(found_dict, msec=False):
 		Unix time stamp.
 	"""

-	now = MyTime.now()
-	year = month = day = hour = minute = None
-	hour = minute = None
+	now = \
+	year = month = day = hour = minute = tzoffset = \
+	weekday = julian = week_of_year = None
 	second = fraction = 0
-	tzoffset = None
-	# Default to -1 to signify that values not known; not critical to have,
-	# though
-	week_of_year = -1
-	week_of_year_start = -1
-	# weekday and julian defaulted to -1 so as to signal need to calculate
-	# values
-	weekday = julian = -1
 	for key, val in found_dict.iteritems():
 		if val is None: continue
 		# Directives not explicitly handled below:
@ -116,13 +108,9 @@ def reGroupDictStrptime(found_dict, msec=False):
 		#	  worthless without day of the week
 		if key == 'y':
 			year = int(val)
-			# Open Group specification for strptime() states that a %y
-			#value in the range of [00, 68] is in the century 2000, while
-			#[69,99] is in the century 1900
-			if year <= 68:
+			# Fail2ban year should be always in the current century (>= 2000)
+			if year <= 2000:
 				year += 2000
-			else:
-				year += 1900
 		elif key == 'Y':
 			year = int(val)
 		elif key == 'm':
@ -156,7 +144,7 @@ def reGroupDictStrptime(found_dict, msec=False):
 		elif key == 'S':
 			second = int(val)
 		elif key == 'f':
-			if msec:
+			if msec: # pragma: no cover - currently unused
 				s = val
 				# Pad to always return microseconds.
 				s += "0" * (6 - len(s))
@ -166,21 +154,14 @@ def reGroupDictStrptime(found_dict, msec=False):
 		elif key == 'a':
 			weekday = locale_time.a_weekday.index(val.lower())
 		elif key == 'w':
-			weekday = int(val)
-			if weekday == 0:
-				weekday = 6
-			else:
-				weekday -= 1
+			weekday = int(val) - 1
+			if weekday < 0: weekday = 6
 		elif key == 'j':
 			julian = int(val)
 		elif key in ('U', 'W'):
 			week_of_year = int(val)
-			if key == 'U':
-				# U starts week on Sunday.
-				week_of_year_start = 6
-			else:
-				# W starts week on Monday.
-				week_of_year_start = 0
+			# U starts week on Sunday, W - on Monday
+			week_of_year_start = 6 if key == 'U' else 0
 		elif key == 'z':
 			z = val
 			if z in ("Z", "UTC", "GMT"):
@ -199,31 +180,28 @@ def reGroupDictStrptime(found_dict, msec=False):
 	# Fail2Ban will assume it's this year
 	assume_year = False
 	if year is None:
+		if not now: now = MyTime.now()
 		year = now.year
 		assume_year = True
-	# If we know the week of the year and what day of that week, we can figure
-	# out the Julian day of the year.
-	if julian == -1 and week_of_year != -1 and weekday != -1:
-		week_starts_Mon = True if week_of_year_start == 0 else False
-		julian = _calc_julian_from_U_or_W(year, week_of_year, weekday,
-											week_starts_Mon)
-	# Cannot pre-calculate datetime.datetime() since can change in Julian
-	# calculation and thus could have different value for the day of the week
-	# calculation.
-	if julian != -1 and (month is None or day is None):
-		datetime_result = datetime.datetime.fromordinal((julian - 1) + datetime.datetime(year, 1, 1).toordinal())
-		year = datetime_result.year
-		month = datetime_result.month
-		day = datetime_result.day
-	# Add timezone info
-	if tzoffset is not None:
-		gmtoff = tzoffset * 60
-	else:
-		gmtoff = None
+	if month is None or day is None:
+		# If we know the week of the year and what day of that week, we can figure
+		# out the Julian day of the year.
+		if julian is None and week_of_year is not None and weekday is not None:
+			julian = _calc_julian_from_U_or_W(year, week_of_year, weekday,
+												(week_of_year_start == 0))
+		# Cannot pre-calculate datetime.datetime() since can change in Julian
+		# calculation and thus could have different value for the day of the week
+		# calculation.
+		if julian is not None:
+			datetime_result = datetime.datetime.fromordinal((julian - 1) + datetime.datetime(year, 1, 1).toordinal())
+			year = datetime_result.year
+			month = datetime_result.month
+			day = datetime_result.day

 	# Fail2Ban assume today
 	assume_today = False
 	if month is None and day is None:
+		if not now: now = MyTime.now()
 		month = now.month
 		day = now.day
 		assume_today = True
@ -231,22 +209,28 @@ def reGroupDictStrptime(found_dict, msec=False):
 	# Actully create date
 	date_result =  datetime.datetime(
 		year, month, day, hour, minute, second, fraction)
-	if gmtoff is not None:
-		date_result = date_result - datetime.timedelta(seconds=gmtoff)
+	# Add timezone info
+	if tzoffset is not None:
+		date_result -= datetime.timedelta(seconds=tzoffset * 60)

-	if date_result > now and assume_today:
-		# Rollover at midnight, could mean it's yesterday...
-		date_result = date_result - datetime.timedelta(days=1)
-	if date_result > now and assume_year:
-		# Could be last year?
-		# also reset month and day as it's not yesterday...
-		date_result = date_result.replace(
-			year=year-1, month=month, day=day)
+	if assume_today:
+		if not now: now = MyTime.now()
+		if date_result > now:
+			# Rollover at midnight, could mean it's yesterday...
+			date_result -= datetime.timedelta(days=1)
+	if assume_year:
+		if not now: now = MyTime.now()
+		if date_result > now:
+			# Could be last year?
+			# also reset month and day as it's not yesterday...
+			date_result = date_result.replace(
+				year=year-1, month=month, day=day)

-	if gmtoff is not None:
+	# make time:
+	if tzoffset is not None:
 		tm = calendar.timegm(date_result.utctimetuple())
 	else:
 		tm = time.mktime(date_result.timetuple())
-	if msec:
+	if msec: # pragma: no cover - currently unused
 		tm += fraction/1000000.0
 	return tm
--- a/fail2ban/server/transmitter.py
+++ b/fail2ban/server/transmitter.py
@ -108,11 +108,11 @@ class Transmitter:
 			value = command[1:]
 			# if all ips:
 			if len(value) == 1 and value[0] == "--all":
-				self.__server.setUnbanIP()
-				return
+				return self.__server.setUnbanIP()
+			cnt = 0
 			for value in value:
-				self.__server.setUnbanIP(None, value)
-			return None
+				cnt += self.__server.setUnbanIP(None, value)
+			return cnt
 		elif command[0] == "echo":
 			return command[1:]
 		elif command[0] == "sleep":
@ -181,6 +181,10 @@ class Transmitter:
 				raise Exception("Invalid idle option, must be 'on' or 'off'")
 			return self.__server.getIdleJail(name)
 		# Filter
+		elif command[1] == "ignoreself":
+			value = command[2]
+			self.__server.setIgnoreSelf(name, value)
+			return self.__server.getIgnoreSelf(name)
 		elif command[1] == "addignoreip":
 			value = command[2]
 			self.__server.addIgnoreIP(name, value)
@ -346,6 +350,8 @@ class Transmitter:
 			return self.__server.getLogEncoding(name)
 		elif command[1] == "journalmatch": # pragma: systemd no cover
 			return self.__server.getJournalMatch(name)
+		elif command[1] == "ignoreself":
+			return self.__server.getIgnoreSelf(name)
 		elif command[1] == "ignoreip":
 			return self.__server.getIgnoreIP(name)
 		elif command[1] == "ignorecommand":
--- a/fail2ban/server/utils.py
+++ b/fail2ban/server/utils.py
@ -28,7 +28,7 @@ import signal
 import subprocess
 import sys
 import time
-from ..helpers import getLogger, uni_decode
+from ..helpers import getLogger, _merge_dicts, uni_decode

 if sys.version_info >= (3, 3):
 	import importlib.machinery
@ -60,6 +60,7 @@ class Utils():
 	DEFAULT_SLEEP_TIME = 2
 	DEFAULT_SLEEP_INTERVAL = 0.2
 	DEFAULT_SHORT_INTERVAL = 0.001
+	DEFAULT_SHORTEST_INTERVAL = DEFAULT_SHORT_INTERVAL / 100


 	class Cache(object):
@ -116,7 +117,31 @@ class Utils():
 		return flags

 	@staticmethod
-	def executeCmd(realCmd, timeout=60, shell=True, output=False, tout_kill_tree=True, success_codes=(0,)):
+	def buildShellCmd(realCmd, varsDict):
+		"""Generates new shell command as array, contains map as variables to
+		arguments statement (varsStat), the command (realCmd) used this variables and
+		the list of the arguments, mapped from varsDict
+
+		Example:
+			buildShellCmd('echo "V2: $v2, V1: $v1"', {"v1": "val 1", "v2": "val 2", "vUnused": "unused var"})
+		returns:
+			['v1=$0 v2=$1 vUnused=$2 \necho "V2: $v2, V1: $v1"', 'val 1', 'val 2', 'unused var']
+		"""
+		# build map as array of vars and command line array:
+		varsStat = ""
+		if not isinstance(realCmd, list):
+			realCmd = [realCmd]
+		i = len(realCmd)-1
+		for k, v in varsDict.iteritems():
+			varsStat += "%s=$%s " % (k, i)
+			realCmd.append(v)
+			i += 1
+		realCmd[0] = varsStat + "\n" + realCmd[0]
+		return realCmd
+
+	@staticmethod
+	def executeCmd(realCmd, timeout=60, shell=True, output=False, tout_kill_tree=True, 
+		success_codes=(0,), varsDict=None):
 		"""Executes a command.

 		Parameters
@ -131,6 +156,8 @@ class Utils():
 		output : bool
 			If output is True, the function returns tuple (success, stdoutdata, stderrdata, returncode).
 			If False, just indication of success is returned
+		varsDict: dict
+			variables supplied to the command (or to the shell script)

 		Returns
 		-------
@ -146,10 +173,18 @@ class Utils():
 		"""
 		stdout = stderr = None
 		retcode = None
-		popen = None
+		popen = env = None
+		if varsDict:
+			if shell:
+				# build map as array of vars and command line array:
+				realCmd = Utils.buildShellCmd(realCmd, varsDict)
+			else: # pragma: no cover - currently unused
+				env = _merge_dicts(os.environ, varsDict)
+		realCmdId = id(realCmd)
+		logCmd = lambda level: logSys.log(level, "%x -- exec: %s", realCmdId, realCmd)
 		try:
 			popen = subprocess.Popen(
-				realCmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=shell,
+				realCmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=shell, env=env,
 				preexec_fn=os.setsid  # so that killpg does not kill our process
 			)
 			# wait with timeout for process has terminated:
@ -158,13 +193,15 @@ class Utils():
 				def _popen_wait_end():
 					retcode = popen.poll()
 					return (True, retcode) if retcode is not None else None
-				retcode = Utils.wait_for(_popen_wait_end, timeout, Utils.DEFAULT_SHORT_INTERVAL)
+				# popen.poll is fast operation so we can use the shortest sleep interval:
+				retcode = Utils.wait_for(_popen_wait_end, timeout, Utils.DEFAULT_SHORTEST_INTERVAL)
 				if retcode:
 					retcode = retcode[1]
 			# if timeout:
 			if retcode is None:
-				logSys.error("%s -- timed out after %s seconds." %
-					(realCmd, timeout))
+				if logCmd: logCmd(logging.ERROR); logCmd = None
+				logSys.error("%x -- timed out after %s seconds." %
+					(realCmdId, timeout))
 				pgid = os.getpgid(popen.pid)
 				# if not tree - first try to terminate and then kill, otherwise - kill (-9) only:
 				os.killpg(pgid, signal.SIGTERM) # Terminate the process
@ -174,59 +211,62 @@ class Utils():
 				if retcode is None or tout_kill_tree: # Still going...
 					os.killpg(pgid, signal.SIGKILL) # Kill the process
 					time.sleep(Utils.DEFAULT_SLEEP_INTERVAL)
-					retcode = popen.poll()
+					if retcode is None: # pragma: no cover - too sporadic
+						retcode = popen.poll()
 					#logSys.debug("%s -- killed %s ", realCmd, retcode)
 				if retcode is None and not Utils.pid_exists(pgid): # pragma: no cover
 					retcode = signal.SIGKILL
 		except OSError as e:
+			if logCmd: logCmd(logging.ERROR); logCmd = None
 			stderr = "%s -- failed with %s" % (realCmd, e)
 			logSys.error(stderr)
 			if not popen:
 				return False if not output else (False, stdout, stderr, retcode)

 		std_level = logging.DEBUG if retcode in success_codes else logging.ERROR
+		if std_level > logSys.getEffectiveLevel():
+			if logCmd: logCmd(std_level-1); logCmd = None
 		# if we need output (to return or to log it): 
 		if output or std_level >= logSys.getEffectiveLevel():
+
 			# if was timeouted (killed/terminated) - to prevent waiting, set std handles to non-blocking mode.
 			if popen.stdout:
 				try:
 					if retcode is None or retcode < 0:
 						Utils.setFBlockMode(popen.stdout, False)
 					stdout = popen.stdout.read()
-				except IOError as e:
+				except IOError as e: # pragma: no cover
 					logSys.error(" ... -- failed to read stdout %s", e)
 				if stdout is not None and stdout != '' and std_level >= logSys.getEffectiveLevel():
-					logSys.log(std_level, "%s -- stdout:", realCmd)
 					for l in stdout.splitlines():
-						logSys.log(std_level, " -- stdout: %r", uni_decode(l))
+						logSys.log(std_level, "%x -- stdout: %r", realCmdId, uni_decode(l))
 				popen.stdout.close()
 			if popen.stderr:
 				try:
 					if retcode is None or retcode < 0:
 						Utils.setFBlockMode(popen.stderr, False)
 					stderr = popen.stderr.read()
-				except IOError as e:
+				except IOError as e: # pragma: no cover
 					logSys.error(" ... -- failed to read stderr %s", e)
 				if stderr is not None and stderr != '' and std_level >= logSys.getEffectiveLevel():
-					logSys.log(std_level, "%s -- stderr:", realCmd)
 					for l in stderr.splitlines():
-						logSys.log(std_level, " -- stderr: %r", uni_decode(l))
+						logSys.log(std_level, "%x -- stderr: %r", realCmdId, uni_decode(l))
 				popen.stderr.close()

 		success = False
 		if retcode in success_codes:
-			logSys.debug("%-.40s -- returned successfully %i", realCmd, retcode)
+			logSys.debug("%x -- returned successfully %i", realCmdId, retcode)
 			success = True
 		elif retcode is None:
-			logSys.error("%-.40s -- unable to kill PID %i", realCmd, popen.pid)
+			logSys.error("%x -- unable to kill PID %i", realCmdId, popen.pid)
 		elif retcode < 0 or retcode > 128:
 			# dash would return negative while bash 128 + n
 			sigcode = -retcode if retcode < 0 else retcode - 128
-			logSys.error("%-.40s -- killed with %s (return code: %s)",
-				realCmd, signame.get(sigcode, "signal %i" % sigcode), retcode)
+			logSys.error("%x -- killed with %s (return code: %s)",
+				realCmdId, signame.get(sigcode, "signal %i" % sigcode), retcode)
 		else:
 			msg = _RETCODE_HINTS.get(retcode, None)
-			logSys.error("%-.40s -- returned %i", realCmd, retcode)
+			logSys.error("%x -- returned %i", realCmdId, retcode)
 			if msg:
 				logSys.info("HINT on %i: %s", retcode, msg % locals())
 		if output:
@ -290,7 +330,7 @@ class Utils():
 				return e.errno == errno.EPERM
 			else:
 				return True
-	else:
+	else: # pragma : no cover (no windows currently supported)
 		@staticmethod
 		def pid_exists(pid):
 			import ctypes
--- a/fail2ban/tests/action_d/test_smtp.py
+++ b/fail2ban/tests/action_d/test_smtp.py
@ -67,7 +67,7 @@ class SMTPActionTest(unittest.TestCase):
 		port = self.smtpd.socket.getsockname()[1]

 		self.action = customActionModule.Action(
-			self.jail, "test", host="127.0.0.1:%i" % port)
+			self.jail, "test", host="localhost:%i" % port)

 		## because of bug in loop (see loop in asyncserver.py) use it's loop instead of asyncore.loop:
 		self._active = True
--- a/fail2ban/tests/actiontestcase.py
+++ b/fail2ban/tests/actiontestcase.py
@ -389,6 +389,51 @@ class CommandActionTest(LogCaptureTestCase):
 		self.assertLogged('Nothing to do')
 		self.pruneLog()

+	def testExecuteWithVars(self):
+		self.assertTrue(self.__action.executeCmd(
+			r'''printf %b "foreign input:\n'''
+			r''' -- $f2bV_A --\n'''
+			r''' -- $f2bV_B --\n'''
+			r''' -- $(echo -n $f2bV_C) --''' # echo just replaces \n to test it as single line
+			r'''"''', 
+			varsDict={
+			'f2bV_A': 'I\'m a hacker; && $(echo $f2bV_B)', 
+			'f2bV_B': 'I"m very bad hacker', 
+			'f2bV_C': '`Very | very\n$(bad & worst hacker)`'
+		}))
+		self.assertLogged(r"""foreign input:""",
+			' -- I\'m a hacker; && $(echo $f2bV_B) --',
+			' -- I"m very bad hacker --',
+			' -- `Very | very $(bad & worst hacker)` --', all=True)
+
+	def testExecuteReplaceEscapeWithVars(self):
+		self.__action.actionban = 'echo "** ban <ip>, reason: <reason> ...\\n<matches>"'
+		self.__action.actionunban = 'echo "** unban <ip>"'
+		self.__action.actionstop = 'echo "** stop monitoring"'
+		matches = [
+			'<actionunban>',
+			'" Hooray! #',
+			'`I\'m cool script kiddy',
+			'`I`m very cool > /here-is-the-path/to/bin/.x-attempt.sh',
+			'<actionstop>',
+		]
+		aInfo = {
+			'ip': '192.0.2.1',
+			'reason': 'hacking attempt ( he thought he knows how f2b internally works ;)',
+			'matches': '\n'.join(matches)
+		}
+		self.pruneLog()
+		self.__action.ban(aInfo)
+		self.assertLogged(
+			'** ban %s' % aInfo['ip'], aInfo['reason'], *matches, all=True)
+		self.assertNotLogged(
+			'** unban %s' % aInfo['ip'], '** stop monitoring', all=True)
+		self.pruneLog()
+		self.__action.unban(aInfo)
+		self.__action.stop()
+		self.assertLogged(
+			'** unban %s' % aInfo['ip'], '** stop monitoring', all=True)
+
 	def testExecuteIncorrectCmd(self):
 		CommandAction.executeCmd('/bin/ls >/dev/null\nbogusXXX now 2>/dev/null')
 		self.assertLogged('HINT on 127: "Command not found"')
@ -400,8 +445,9 @@ class CommandActionTest(LogCaptureTestCase):
 		self.assertFalse(CommandAction.executeCmd('sleep 30', timeout=timeout))
 		# give a test still 1 second, because system could be too busy
 		self.assertTrue(time.time() >= stime + timeout and time.time() <= stime + timeout + 1)
-		self.assertLogged('sleep 30 -- timed out after')
-		self.assertLogged('sleep 30 -- killed with SIGTERM')
+		self.assertLogged('sleep 30', ' -- timed out after', all=True)
+		self.assertLogged(' -- killed with SIGTERM', 
+		                  ' -- killed with SIGKILL')

 	def testExecuteTimeoutWithNastyChildren(self):
 		# temporary file for a nasty kid shell script
@ -457,9 +503,9 @@ class CommandActionTest(LogCaptureTestCase):
 		# Verify that the process itself got killed
 		self.assertTrue(Utils.wait_for(lambda: not pid_exists(cpid), 3))
 		self.assertLogged('my pid ', 'Resource temporarily unavailable')
-		self.assertLogged('timed out')
-		self.assertLogged('killed with SIGTERM', 
-		                  'killed with SIGKILL')
+		self.assertLogged(' -- timed out')
+		self.assertLogged(' -- killed with SIGTERM', 
+		                  ' -- killed with SIGKILL')
 		os.unlink(tmpFilename)
 		os.unlink(tmpFilename + '.pid')

--- a/fail2ban/tests/clientreadertestcase.py
+++ b/fail2ban/tests/clientreadertestcase.py
@ -28,7 +28,7 @@ import re
 import shutil
 import tempfile
 import unittest
-from ..client.configreader import ConfigReader, ConfigReaderUnshared
+from ..client.configreader import ConfigReader, ConfigReaderUnshared, NoSectionError
 from ..client import configparserinc
 from ..client.jailreader import JailReader
 from ..client.filterreader import FilterReader
@ -317,7 +317,17 @@ class JailReaderTest(LogCaptureTestCase):
 		self.assertLogged('File %s is a dangling link, thus cannot be monitored' % f2)
 		self.assertEqual(JailReader._glob(os.path.join(d, 'nonexisting')), [])

-		
+	def testCommonFunction(self):
+		c = ConfigReader(share_config={})
+		# test common functionalities (no shared, without read of config):
+		self.assertEqual(c.sections(), [])
+		self.assertFalse(c.has_section('test'))
+		self.assertRaises(NoSectionError, c.merge_section, 'test', {})
+		self.assertRaises(NoSectionError, c.options, 'test')
+		self.assertRaises(NoSectionError, c.get, 'test', 'any')
+		self.assertRaises(NoSectionError, c.getOptions, 'test', {})
+
+
 class FilterReaderTest(unittest.TestCase):

 	def __init__(self, *args, **kwargs):
@ -712,6 +722,7 @@ class JailsReaderTest(LogCaptureTestCase):
 			self.assertEqual(opts['socket'], '/var/run/fail2ban/fail2ban.sock')
 			self.assertEqual(opts['pidfile'], '/var/run/fail2ban/fail2ban.pid')

+			configurator.readAll()
 			configurator.getOptions()
 			configurator.convertToProtocol()
 			commands = configurator.getConfigStream()
--- a/fail2ban/tests/config/filter.d/test.conf
+++ b/fail2ban/tests/config/filter.d/test.conf
@ -1,6 +1,13 @@
 #[INCLUDES]
 #before = common.conf

-[Definition]
-failregex = failure test 1 (filter.d/test.conf) <HOST>
+[DEFAULT]
+_daemon = default

+[Definition]
+where = conf
+failregex = failure <_daemon> <one> (filter.d/test.%(where)s) <HOST>
+
+[Init]
+# test parameter, should be overriden in jail by "filter=test[one=1,...]"
+one = *1*
--- a/fail2ban/tests/config/filter.d/test.local
+++ b/fail2ban/tests/config/filter.d/test.local
@ -2,6 +2,15 @@
 #before = common.conf

 [Definition]
+# overwrite default daemon, additionally it should be accessible in jail with "%(known/_daemon)s":
+_daemon = test
+# interpolate previous regex (from test.conf) + new 2nd + dynamical substitution) of "two" an "where":
 failregex = %(known/failregex)s
-            failure test 2 (filter.d/test.local) <HOST>
+            failure %(_daemon)s <two> (filter.d/test.<where>) <HOST>
+# parameter "two" should be specified in jail by "filter=test[..., two=2]"

+[Init]
+# this parameter can be used in jail with "%(known/three)s":
+three = 3
+# this parameter "where" does not overwrite "where" in definition of test.conf (dynamical values only):
+where = local
--- a/fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf
+++ b/fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf
@ -40,12 +40,13 @@ cmnfailre = ^%(__prefix_line_sl)s[aA]uthentication (?:failure|error|failed) for
         ^%(__prefix_line_sl)spam_unix\(sshd:auth\):\s+authentication failure;\s*logname=\S*\s*uid=\d*\s*euid=\d*\s*tty=\S*\s*ruser=\S*\s*rhost=<HOST>\s.*%(__suff)s$
         ^%(__prefix_line_sl)s(error: )?maximum authentication attempts exceeded for .* from <HOST>%(__on_port_opt)s(?: ssh\d*)? \[preauth\]$
         ^%(__prefix_line_ml1)sUser .+ not allowed because account is locked%(__prefix_line_ml2)sReceived disconnect from <HOST>: 11: .+%(__suff)s$
-         ^%(__prefix_line_ml1)sDisconnecting: Too many authentication failures for .+?%(__prefix_line_ml2)sConnection closed by <HOST>%(__suff)s$
-         ^%(__prefix_line_ml1)sConnection from <HOST>%(__on_port_opt)s%(__prefix_line_ml2)sDisconnecting: Too many authentication failures for .+%(__suff)s$
+         ^%(__prefix_line_ml1)sDisconnecting: Too many authentication failures(?: for .+?)?%(__suff)s%(__prefix_line_ml2)sConnection closed by <HOST>%(__suff)s$
+         ^%(__prefix_line_ml1)sConnection from <HOST>%(__on_port_opt)s%(__prefix_line_ml2)sDisconnecting: Too many authentication failures(?: for .+?)?%(__suff)s$

 mdre-normal =

 mdre-ddos =  ^%(__prefix_line_sl)sDid not receive identification string from <HOST>%(__suff)s$
+             ^%(__prefix_line_sl)sConnection reset by <HOST>%(__on_port_opt)s%(__suff)s
             ^%(__prefix_line_ml1)sSSH: Server;Ltype: (?:Authname|Version|Kex);Remote: <HOST>-\d+;[A-Z]\w+:.*%(__prefix_line_ml2)sRead from socket failed: Connection reset by peer%(__suff)s$

 mdre-extra = ^%(__prefix_line_sl)sReceived disconnect from <HOST>%(__on_port_opt)s:\s*14: No supported authentication methods available%(__suff)s$
--- a/fail2ban/tests/config/jail.conf
+++ b/fail2ban/tests/config/jail.conf
@ -15,9 +15,9 @@ ignoreip =

 [test-known-interp]
 enabled = true
-filter = test
+filter = test[one=1,two=2]
 failregex = %(known/failregex)s
-            failure test 3 (jail.local) <HOST>
+            failure %(known/_daemon)s %(known/three)s (jail.local) <HOST>

 [missinglogfiles]
 enabled = true
--- a/fail2ban/tests/datedetectortestcase.py
+++ b/fail2ban/tests/datedetectortestcase.py
@ -298,6 +298,16 @@ iso8601 = DatePatternRegex("%Y-%m-%d[T ]%H:%M:%S(?:\.%f)?%z")

 class CustomDateFormatsTest(unittest.TestCase):

+	def setUp(self):
+		"""Call before every test case."""
+		unittest.TestCase.setUp(self)
+		setUpMyTime()
+
+	def tearDown(self):
+		"""Call after every test case."""
+		unittest.TestCase.tearDown(self)
+		tearDownMyTime()
+
 	def testIso8601(self):
 		date = datetime.datetime.utcfromtimestamp(
 			iso8601.getDate("2007-01-25T12:00:00Z")[0])
@ -411,6 +421,37 @@ class CustomDateFormatsTest(unittest.TestCase):
 			else:
 				self.assertEqual(date, None)

+	def testVariousFormatSpecs(self):
+		for (matched, dp, line) in (
+			# cover %B (full-month-name) and %I (as 12 == 0):
+			(1106438399.0, "^%B %Exd %I:%ExM:%ExS**", 'January 23 12:59:59'),
+			# cover %U (week of year starts on sunday) and %A (weekday):
+			(985208399.0,  "^%y %U %A %ExH:%ExM:%ExS**", '01 11 Wednesday 21:59:59'),
+			# cover %W (week of year starts on monday) and %A (weekday):
+			(984603599.0,  "^%y %W %A %ExH:%ExM:%ExS**", '01 11 Wednesday 21:59:59'),
+			# cover %W (week of year starts on monday) and %w (weekday, 0 - sunday):
+			(984949199.0,  "^%y %W %w %ExH:%ExM:%ExS**", '01 11 0 21:59:59'),
+			# cover %W (week of year starts on monday) and %w (weekday, 6 - saturday):
+			(984862799.0,  "^%y %W %w %ExH:%ExM:%ExS**", '01 11 6 21:59:59'),
+			# cover time only, current date, in test cases now == 14 Aug 2005 12:00 -> back to yesterday (13 Aug):
+			(1123963199.0,  "^%ExH:%ExM:%ExS**", '21:59:59'),
+			# cover time only, current date, in test cases now == 14 Aug 2005 12:00 -> today (14 Aug):
+			(1123970401.0,  "^%ExH:%ExM:%ExS**", '00:00:01'),
+			# cover date with current year, in test cases now == Aug 2005 -> back to last year (Sep 2004):
+			(1094068799.0,  "^%m/%d %ExH:%ExM:%ExS**", '09/01 21:59:59'),
+		):
+			logSys.debug('== test: %r', (matched, dp, line))
+			dd = DateDetector()
+			dd.appendTemplate(dp)
+			date = dd.getTime(line)
+			if matched:
+				self.assertTrue(date)
+				if isinstance(matched, basestring): # pragma: no cover
+					self.assertEqual(matched, date[1].group(1))
+				else:
+					self.assertEqual(matched, date[0])
+			else: # pragma: no cover
+				self.assertEqual(date, None)

 #	def testDefaultTempate(self):
 #		self.__datedetector.setDefaultRegex("^\S{3}\s{1,2}\d{1,2} \d{2}:\d{2}:\d{2}")
--- a/fail2ban/tests/fail2banclienttestcase.py
+++ b/fail2ban/tests/fail2banclienttestcase.py
@ -770,6 +770,7 @@ class Fail2banServerTest(Fail2banClientServerBase):
 				"norestored = %(_exec_once)s",
 				"restore = ",
 				"info = ",
+				"_use_flush_ = echo [<name>] <actname>: -- flushing IPs",
 				"actionstart =  echo '[%(name)s] %(actname)s: ** start'", start,
 				"actionreload = echo '[%(name)s] %(actname)s: .. reload'", reload,
 				"actionban =    echo '[%(name)s] %(actname)s: ++ ban <ip> %(restore)s%(info)s'", ban,
@ -788,6 +789,7 @@ class Fail2banServerTest(Fail2banClientServerBase):
 				"findtime = 10m",
 				"failregex = ^\s*failure <F-ERRCODE>401|403</F-ERRCODE> from <HOST>",
 				"datepattern = {^LN-BEG}EPOCH",
+				"ignoreip = 127.0.0.1/8 ::1", # just to cover ignoreip in jailreader/transmitter
 				"",
 				"[test-jail1]", "backend = " + backend, "filter =", 
 				"action = ",
@ -795,7 +797,8 @@ class Fail2banServerTest(Fail2banClientServerBase):
 					if 1 in actions else "",
 				"         test-action2[name='%(__name__)s', restore='restored: <restored>', info=', err-code: <F-ERRCODE>']" \
 					if 2 in actions else "",
-				"         test-action2[name='%(__name__)s', actname=test-action3, _exec_once=1, restore='restored: <restored>']" \
+				"         test-action2[name='%(__name__)s', actname=test-action3, _exec_once=1, restore='restored: <restored>',"
+										" actionflush=<_use_flush_>]" \
 					if 3 in actions else "",
 				"logpath = " + test1log,
 				"          " + test2log if 2 in enabled else "",
@ -809,7 +812,8 @@ class Fail2banServerTest(Fail2banClientServerBase):
 				"action = ",
 				"         test-action2[name='%(__name__)s', restore='restored: <restored>', info=', err-code: <F-ERRCODE>']" \
 					if 2 in actions else "",
-				"         test-action2[name='%(__name__)s', actname=test-action3, _exec_once=1, restore='restored: <restored>']" \
+				"         test-action2[name='%(__name__)s', actname=test-action3, _exec_once=1, restore='restored: <restored>']"
+										" actionflush=<_use_flush_>]" \
 					if 3 in actions else "",
 				"logpath = " + test2log,
 				"enabled = true" if 2 in enabled else "",
@ -881,6 +885,12 @@ class Fail2banServerTest(Fail2banClientServerBase):
 		self.assertLogged(
 			"Creating new jail 'test-jail2'",
 			"Jail 'test-jail2' started", all=True)
+		# test action3 removed, test flushing successful (and no single unban occurred):
+		self.assertLogged(
+			"stdout: '[test-jail1] test-action3: -- flushing IPs'",
+			"stdout: '[test-jail1] test-action3: __ stop'", all=True)
+		self.assertNotLogged(
+			"stdout: '[test-jail1] test-action3: -- unban 192.0.2.1'")
 		
 		# update action1, delete action2 (should be stopped via configuration)...
 		self.pruneLog("[test-phase 2a]")
@ -978,12 +988,18 @@ class Fail2banServerTest(Fail2banClientServerBase):
 			"stdout: '[test-jail2] test-action3: ++ ban 192.0.2.8 restored: 1'",
 			all=True)

-		# don't need actions anymore:
-		_write_action_cfg(actname="test-action2", allow=False)
-		_write_jail_cfg(actions=[])
+		# ban manually to test later flush by unban all:
+		self.pruneLog("[test-phase 2d]")
+		self.execSuccess(startparams,
+			"set", "test-jail2", "banip", "192.0.2.21")
+		self.execSuccess(startparams,
+			"set", "test-jail2", "banip", "192.0.2.22")
+		self.assertLogged(
+			"stdout: '[test-jail2] test-action3: ++ ban 192.0.2.22",
+			"stdout: '[test-jail2] test-action3: ++ ban 192.0.2.22 ", all=True, wait=MID_WAITTIME)

 		# restart jail with unban all:
-		self.pruneLog("[test-phase 2d]")
+		self.pruneLog("[test-phase 2e]")
 		self.execSuccess(startparams,
 			"restart", "--unban", "test-jail2")
 		self.assertLogged(
@ -995,12 +1011,26 @@ class Fail2banServerTest(Fail2banClientServerBase):
 			"[test-jail2] Unban 192.0.2.4",
 			"[test-jail2] Unban 192.0.2.8", all=True
 		)
+		# test unban (action2):
+		self.assertLogged(
+			"stdout: '[test-jail2] test-action2: -- unban 192.0.2.21",
+			"stdout: '[test-jail2] test-action2: -- unban 192.0.2.22'", all=True)
+		# test flush (action3, and no single unban via action3 occurred):
+		self.assertLogged(
+			"stdout: '[test-jail2] test-action3: -- flushing IPs'")
+		self.assertNotLogged(
+			"stdout: '[test-jail2] test-action3: -- unban 192.0.2.21'",
+			"stdout: '[test-jail2] test-action3: -- unban 192.0.2.22'", all=True)
 		# no more ban (unbanned all):
 		self.assertNotLogged(
 			"[test-jail2] Ban 192.0.2.4",
 			"[test-jail2] Ban 192.0.2.8", all=True
 		)

+		# don't need actions anymore:
+		_write_action_cfg(actname="test-action2", allow=False)
+		_write_jail_cfg(actions=[])
+
 		# reload jail1 without restart (without ban/unban):
 		self.pruneLog("[test-phase 3]")
 		self.execSuccess(startparams, "reload", "test-jail1")
@ -1085,6 +1115,14 @@ class Fail2banServerTest(Fail2banClientServerBase):
 			"[test-jail1] Ban 192.0.2.4", all=True
 		)

+		# unban all (just to test command, already empty - nothing to unban):
+		self.pruneLog("[test-phase 7b]")
+		self.execSuccess(startparams,
+			"--async", "unban", "--all")
+		self.assertLogged(
+			"Flush ban list",
+			"Unbanned 0, 0 ticket(s) in 'test-jail1'", all=True)
+
 		# backend-switch (restart instead of reload):
 		self.pruneLog("[test-phase 8a]")
 		_write_jail_cfg(enabled=[1], backend="xxx-unknown-backend-zzz")
--- a/fail2ban/tests/fail2banregextestcase.py
+++ b/fail2ban/tests/fail2banregextestcase.py
@ -252,6 +252,44 @@ class Fail2banRegexTest(LogCaptureTestCase):
 		)
 		self.assertTrue(fail2banRegex.start(args))

+	def testDirectMultilineBuf(self):
+		# test it with some pre-lines also to cover correct buffer scrolling (all multi-lines printed):
+		for preLines in (0, 20):
+			self.pruneLog("[test-phase %s]" % preLines)
+			(opts, args, fail2banRegex) = _Fail2banRegex(
+				"--usedns", "no", "-d", "^Epoch", "--print-all-matched", "--maxlines", "5", 
+				("1490349000 TEST-NL\n"*preLines) + 
+				"1490349000 FAIL\n1490349000 TEST1\n1490349001 TEST2\n1490349001 HOST 192.0.2.34",
+				r"^\s*FAIL\s*$<SKIPLINES>^\s*HOST <HOST>\s*$"
+			)
+			self.assertTrue(fail2banRegex.start(args))
+			self.assertLogged('Lines: %s lines, 0 ignored, 2 matched, %s missed' % (preLines+4, preLines+2))
+			# both matched lines were printed:
+			self.assertLogged("|  1490349000 FAIL", "|  1490349001 HOST 192.0.2.34", all=True)
+
+
+	def testDirectMultilineBufDebuggex(self):
+		(opts, args, fail2banRegex) = _Fail2banRegex(
+			"--usedns", "no", "-d", "^Epoch", "--debuggex", "--print-all-matched", "--maxlines", "5",
+			"1490349000 FAIL\n1490349000 TEST1\n1490349001 TEST2\n1490349001 HOST 192.0.2.34",
+			r"^\s*FAIL\s*$<SKIPLINES>^\s*HOST <HOST>\s*$"
+		)
+		self.assertTrue(fail2banRegex.start(args))
+		self.assertLogged('Lines: 4 lines, 0 ignored, 2 matched, 2 missed')
+		# the sequence in args-dict is currently undefined (so can be 1st argument)
+		self.assertLogged("&flags=m", "?flags=m")
+
+	def testSinglelineWithNLinContent(self):
+		# 
+		(opts, args, fail2banRegex) = _Fail2banRegex(
+			"--usedns", "no", "-d", "^Epoch", "--print-all-matched",
+			"1490349000 FAIL: failure\nhost: 192.0.2.35",
+			r"^\s*FAIL:\s*.*\nhost:\s+<HOST>$"
+		)
+		self.assertTrue(fail2banRegex.start(args))
+		self.assertLogged('Lines: 1 lines, 0 ignored, 1 matched, 0 missed')
+
+
 	def testWrongFilterFile(self):
 		# use test log as filter file to cover eror cases...
 		(opts, args, fail2banRegex) = _Fail2banRegex(
--- a/fail2ban/tests/files/logs/exim
+++ b/fail2ban/tests/files/logs/exim
@ -60,10 +60,19 @@
 2016-03-21 04:07:49 [25874] 1ahr79-0006jK-G9 SMTP connection from (voyeur.webair.com) [174.137.147.204]:44884 I=[172.89.0.6]:25 closed by DROP in ACL
 # failJSON: { "time": "2016-03-21T04:33:13", "match": true , "host": "206.214.71.53" }
 2016-03-21 04:33:13 [26074] 1ahrVl-0006mY-79 SMTP connection from riveruse.com [206.214.71.53]:39865 I=[172.89.0.6]:25 closed by DROP in ACL
+# failJSON: { "time": "2016-03-21T04:33:14", "match": true , "host": "192.0.2.33", "desc": "short form without optional session-id" }
+2016-03-21 04:33:14 SMTP connection from (some.domain) [192.0.2.33] closed by DROP in ACL

 # failJSON: { "time": "2016-04-01T11:08:39", "match": true , "host": "192.0.2.1" }
 2016-04-01 11:08:39 [18643] no MAIL in SMTP connection from host.example.com (SERVER) [192.0.2.1]:1418 I=[172.89.0.6]:25 D=34s C=EHLO,AUTH
+# failJSON: { "time": "2016-04-01T11:08:40", "match": true , "host": "192.0.2.2" }
+2016-04-01 11:08:40 [18643] no MAIL in SMTP connection from host.example.com (SERVER) [192.0.2.2]:1418 I=[172.89.0.6]:25 D=2m42s C=QUIT
 # failJSON: { "time": "2016-04-01T11:09:21", "match": true , "host": "192.0.2.1" }
 2016-04-01 11:09:21 [18648] SMTP protocol error in "AUTH LOGIN" H=host.example.com (SERVER) [192.0.2.1]:4692 I=[172.89.0.6]:25 AUTH command used when not advertised
 # failJSON: { "time": "2016-03-27T16:48:48", "match": true , "host": "192.0.2.1" }
 2016-03-27 16:48:48 [21478] 1akDqs-0005aQ-9b SMTP connection from host.example.com (SERVER) [192.0.2.1]:47714 I=[172.89.0.6]:25 closed by DROP in ACL
+
+# failJSON: { "time": "2017-04-23T22:45:59", "match": true , "host": "192.0.2.2", "desc": "optional part (...)" }
+2017-04-23 22:45:59 fixed_login authenticator failed for bad.host.example.com [192.0.2.2]:54412 I=[172.89.0.6]:587: 535 Incorrect authentication data (set_id=user@example.com)
+# failJSON: { "time": "2017-05-01T07:42:42", "match": true , "host": "192.0.2.3", "desc": "rejected RCPT - Unrouteable address" }
+2017-05-01 07:42:42 H=some.rev.dns.if.found (the.connector.reports.this.name) [192.0.2.3] F=<some.name@some.domain> rejected RCPT <some.invalid.name@a.domain>: Unrouteable address
--- a/fail2ban/tests/files/logs/haproxy-http-auth
+++ b/fail2ban/tests/files/logs/haproxy-http-auth
@ -2,3 +2,7 @@
 Nov 14 22:45:27 test haproxy[760]: 192.168.33.1:58444 [14/Nov/2015:22:45:25.439] main app/app1 1939/0/1/0/1940 403 5168 - - ---- 3/3/0/0/0 0/0 "GET / HTTP/1.1"
 # failJSON: { "time": "2004-11-14T22:45:11", "match": true , "host": "192.168.33.1" }
 Nov 14 22:45:11 test haproxy[760]: 192.168.33.1:58430 [14/Nov/2015:22:45:11.608] main main/<NOSRV> -1/-1/-1/-1/0 401 248 - - PR-- 0/0/0/0/0 0/0 "GET / HTTP/1.1"
+# failJSON: { "time": "2004-11-14T22:45:11", "match": true , "host": "2001:db8::1234" }
+Nov 14 22:45:11 test haproxy[760]: 2001:db8::1234:58430 [14/Nov/2015:22:45:11.608] main main/<NOSRV> -1/-1/-1/-1/0 401 248 - - PR-- 0/0/0/0/0 0/0 "GET / HTTP/1.1"
+# failJSON: { "time": "2004-11-14T22:45:11", "match": true , "host": "192.168.33.1" }
+Nov 14 22:45:11 test haproxy[760]: ::ffff:192.168.33.1:58430 [14/Nov/2015:22:45:11.608] main main/<NOSRV> -1/-1/-1/-1/0 401 248 - - PR-- 0/0/0/0/0 0/0 "GET / HTTP/1.1"
--- a/fail2ban/tests/files/logs/sshd
+++ b/fail2ban/tests/files/logs/sshd
@ -113,6 +113,11 @@ May 27 00:16:33 host sshd[2364]: Received disconnect from 198.51.100.76: 11: Bye
 # failJSON: { "time": "2004-09-29T16:28:02", "match": true , "host": "127.0.0.1" }
 Sep 29 16:28:02 spaceman sshd[16699]: Failed password for dan from 127.0.0.1 port 45416 ssh1

+# failJSON: { "match": false, "desc": "no failure, just cache mlfid (conn-id)" }
+Sep 29 16:28:05 localhost sshd[16700]: Connection from 192.0.2.5
+# failJSON: { "match": false, "desc": "no failure, just covering mlfid (conn-id) forget" }
+Sep 29 16:28:05 localhost sshd[16700]: Connection closed by 192.0.2.5 [preauth]
+
 # failJSON: { "time": "2004-09-29T17:15:02", "match": true , "host": "127.0.0.1" }
 Sep 29 17:15:02 spaceman sshd[12946]: Failed hostbased for dan from 127.0.0.1 port 45785 ssh2: RSA 8c:e3:aa:0f:64:51:02:f7:14:79:89:3f:65:84:7c:30, client user "dan", client host "localhost.localdomain"

@ -168,7 +173,7 @@ Feb 12 04:09:21 localhost sshd[26713]: Disconnecting: Too many authentication fa
 # failJSON: { "match": false }
 Feb 12 04:09:18 localhost sshd[26713]: Connection from 115.249.163.77 port 51353 on 127.0.0.1 port 22
 # failJSON: { "time": "2005-02-12T04:09:21", "match": true , "host": "115.249.163.77", "desc": "Multiline match with interface address" }
-Feb 12 04:09:21 localhost sshd[26713]: Disconnecting: Too many authentication failures for root [preauth]
+Feb 12 04:09:21 localhost sshd[26713]: Disconnecting: Too many authentication failures [preauth]

 # failJSON: { "time": "2004-11-23T21:50:37", "match": true , "host": "61.0.0.1", "desc": "New logline format as openssh 6.8 to replace prev multiline version" }
 Nov 23 21:50:37 myhost sshd[21810]: error: maximum authentication attempts exceeded for root from 61.0.0.1 port 49940 ssh2 [preauth]
@ -208,6 +213,11 @@ Nov 24 23:46:41 host sshd[32686]: SSH: Server;Ltype: Authname;Remote: 127.0.0.1-
 # failJSON: { "time": "2004-11-24T23:46:43", "match": true , "host": "127.0.0.1", "desc": "Multiline for connection reset by peer (3)" }
 Nov 24 23:46:43 host sshd[32686]: fatal: Read from socket failed: Connection reset by peer [preauth]

+# gh-1719:
+# failJSON: { "time": "2005-03-15T09:20:57", "match": true , "host": "192.0.2.39", "desc": "Singleline for connection reset by" }
+Mar 15 09:20:57 host sshd[28972]: Connection reset by 192.0.2.39 port 14282 [preauth]
+
+
 # filterOptions: {"mode": "extra"}

 # several other cases from gh-864:
@ -228,4 +238,6 @@ Nov 26 13:03:30 srv sshd[45]: fatal: Unable to negotiate with 192.0.2.2 port 554
 # failJSON: { "match": false }
 Nov 26 15:03:30 host sshd[22440]: Connection from 192.0.2.3 port 39678 on 192.168.1.9 port 22
 # failJSON: { "time": "2004-11-26T15:03:31", "match": true , "host": "192.0.2.3", "desc": "Multiline - no matching key exchange method" }
-Nov 26 15:03:31 host sshd[22440]: fatal: Unable to negotiate a key exchange method [preauth]
+Nov 26 15:03:31 host sshd[22440]: fatal: Unable to negotiate a key exchange method [preauth]
+# failJSON: { "time": "2004-11-26T15:03:32", "match": true , "host": "192.0.2.3", "filter": "sshd", "desc": "Second attempt within the same connect" }
+Nov 26 15:03:32 host sshd[22440]: fatal: Unable to negotiate a key exchange method [preauth]
--- a/fail2ban/tests/filtertestcase.py
+++ b/fail2ban/tests/filtertestcase.py
@ -43,7 +43,7 @@ from ..server.failmanager import FailManagerEmpty
 from ..server.ipdns import DNSUtils, IPAddr
 from ..server.mytime import MyTime
 from ..server.utils import Utils, uni_decode
-from .utils import setUpMyTime, tearDownMyTime, mtimesleep, LogCaptureTestCase
+from .utils import setUpMyTime, tearDownMyTime, mtimesleep, with_tmpdir, LogCaptureTestCase
 from .dummyjail import DummyJail

 TEST_FILES_DIR = os.path.join(os.path.dirname(__file__), "files")
@ -325,6 +325,17 @@ class IgnoreIP(LogCaptureTestCase):
 		LogCaptureTestCase.setUp(self)
 		self.jail = DummyJail()
 		self.filter = FileFilter(self.jail)
+		self.filter.ignoreSelf = False
+
+	def testIgnoreSelfIP(self):
+		ipList = ("127.0.0.1",)
+		# test ignoreSelf is false:
+		for ip in ipList:
+			self.assertFalse(self.filter.inIgnoreIPList(ip))
+		# test ignoreSelf with true:
+		self.filter.ignoreSelf = True
+		for ip in ipList:
+			self.assertTrue(self.filter.inIgnoreIPList(ip))

 	def testIgnoreIPOK(self):
 		ipList = "127.0.0.1", "192.168.0.1", "255.255.255.255", "99.99.99.99"
@ -931,18 +942,21 @@ def get_monitor_failures_testcase(Filter_):
 												  skip=3, mode='w')
 			self.assert_correct_last_attempt(GetFailures.FAILURES_01)

-		def test_move_file(self):
-			# if we move file into a new location while it has been open already
-			self.file.close()
-			self.file = _copy_lines_between_files(GetFailures.FILENAME_01, self.name,
-												  n=14, mode='w')
+		def _wait4failures(self, count=2):
 			# Poll might need more time
 			self.assertTrue(self.isEmpty(_maxWaitTime(5)),
 							"Queue must be empty but it is not: %s."
 							% (', '.join([str(x) for x in self.jail.queue])))
 			self.assertRaises(FailManagerEmpty, self.filter.failManager.toBan)
-			Utils.wait_for(lambda: self.filter.failManager.getFailTotal() == 2, _maxWaitTime(10))
-			self.assertEqual(self.filter.failManager.getFailTotal(), 2)
+			Utils.wait_for(lambda: self.filter.failManager.getFailTotal() >= count, _maxWaitTime(10))
+			self.assertEqual(self.filter.failManager.getFailTotal(), count)
+
+		def test_move_file(self):
+			# if we move file into a new location while it has been open already
+			self.file.close()
+			self.file = _copy_lines_between_files(GetFailures.FILENAME_01, self.name,
+												  n=14, mode='w')
+			self._wait4failures()

 			# move aside, but leaving the handle still open...
 			os.rename(self.name, self.name + '.bak')
@ -956,6 +970,34 @@ def get_monitor_failures_testcase(Filter_):
 			self.assert_correct_last_attempt(GetFailures.FAILURES_01)
 			self.assertEqual(self.filter.failManager.getFailTotal(), 6)

+		@with_tmpdir
+		def test_move_dir(self, tmp):
+			self.file.close()
+			self.filter.delLogPath(self.name)
+			# if we rename parent dir into a new location (simulate directory-base log rotation)
+			tmpsub1 = os.path.join(tmp, "1")
+			tmpsub2 = os.path.join(tmp, "2")
+			os.mkdir(tmpsub1)
+			self.name = os.path.join(tmpsub1, os.path.basename(self.name))
+			os.close(os.open(self.name, os.O_CREAT|os.O_APPEND)); # create empty file
+			self.filter.addLogPath(self.name, autoSeek=False)
+			
+			self.file = _copy_lines_between_files(GetFailures.FILENAME_01, self.name,
+												  skip=12, n=1, mode='w')
+			self.file.close()
+			self._wait4failures(1)
+
+			# rotate whole directory: rename directory 1 as 2:
+			os.rename(tmpsub1, tmpsub2)
+			os.mkdir(tmpsub1)
+			self.file = _copy_lines_between_files(GetFailures.FILENAME_01, self.name,
+												  skip=12, n=1, mode='w')
+			self.file.close()
+			self._wait4failures(2)
+			# stop before tmpdir deleted (just prevents many monitor events)
+			self.filter.stop()
+
+
 		def _test_move_into_file(self, interim_kill=False):
 			# if we move a new file into the location of an old (monitored) file
 			_copy_lines_between_files(GetFailures.FILENAME_01, self.name,
@ -1468,8 +1510,8 @@ class GetFailures(LogCaptureTestCase):
 		output = [("192.0.43.10", 2, 1124013599.0),
 			("192.0.43.11", 1, 1124013598.0)]
 		self.filter.addLogPath(GetFailures.FILENAME_MULTILINE, autoSeek=False)
-		self.filter.addFailRegex("^.*rsyncd\[(?P<pid>\d+)\]: connect from .+ \(<HOST>\)$<SKIPLINES>^.+ rsyncd\[(?P=pid)\]: rsync error: .*$")
 		self.filter.setMaxLines(100)
+		self.filter.addFailRegex("^.*rsyncd\[(?P<pid>\d+)\]: connect from .+ \(<HOST>\)$<SKIPLINES>^.+ rsyncd\[(?P=pid)\]: rsync error: .*$")
 		self.filter.setMaxRetry(1)

 		self.filter.getFailures(GetFailures.FILENAME_MULTILINE)
@ -1486,9 +1528,9 @@ class GetFailures(LogCaptureTestCase):
 	def testGetFailuresMultiLineIgnoreRegex(self):
 		output = [("192.0.43.10", 2, 1124013599.0)]
 		self.filter.addLogPath(GetFailures.FILENAME_MULTILINE, autoSeek=False)
+		self.filter.setMaxLines(100)
 		self.filter.addFailRegex("^.*rsyncd\[(?P<pid>\d+)\]: connect from .+ \(<HOST>\)$<SKIPLINES>^.+ rsyncd\[(?P=pid)\]: rsync error: .*$")
 		self.filter.addIgnoreRegex("rsync error: Received SIGINT")
-		self.filter.setMaxLines(100)
 		self.filter.setMaxRetry(1)

 		self.filter.getFailures(GetFailures.FILENAME_MULTILINE)
@ -1502,9 +1544,9 @@ class GetFailures(LogCaptureTestCase):
 			("192.0.43.11", 1, 1124013598.0),
 			("192.0.43.15", 1, 1124013598.0)]
 		self.filter.addLogPath(GetFailures.FILENAME_MULTILINE, autoSeek=False)
+		self.filter.setMaxLines(100)
 		self.filter.addFailRegex("^.*rsyncd\[(?P<pid>\d+)\]: connect from .+ \(<HOST>\)$<SKIPLINES>^.+ rsyncd\[(?P=pid)\]: rsync error: .*$")
 		self.filter.addFailRegex("^.* sendmail\[.*, msgid=<(?P<msgid>[^>]+).*relay=\[<HOST>\].*$<SKIPLINES>^.+ spamd: result: Y \d+ .*,mid=<(?P=msgid)>(,bayes=[.\d]+)?(,autolearn=\S+)?\s*$")
-		self.filter.setMaxLines(100)
 		self.filter.setMaxRetry(1)

 		self.filter.getFailures(GetFailures.FILENAME_MULTILINE)
--- a/fail2ban/tests/samplestestcase.py
+++ b/fail2ban/tests/samplestestcase.py
@ -40,8 +40,8 @@ TEST_CONFIG_DIR = os.path.join(os.path.dirname(__file__), "config")
 TEST_FILES_DIR = os.path.join(os.path.dirname(__file__), "files")

 # regexp to test greedy catch-all should be not-greedy:
-RE_HOST = Regex('<HOST>').getRegex()
-RE_WRONG_GREED = re.compile(r'\.[+\*](?!\?).*' + re.escape(RE_HOST) + r'.*(?:\.[+\*].*|[^\$])$')
+RE_HOST = Regex._resolveHostTag('<HOST>')
+RE_WRONG_GREED = re.compile(r'\.[+\*](?!\?)[^\$\^]*' + re.escape(RE_HOST) + r'.*(?:\.[+\*].*|[^\$])$')


 class FilterSamplesRegex(unittest.TestCase):
@ -99,8 +99,8 @@ class FilterSamplesRegex(unittest.TestCase):
 				optval = opt[3]
 			elif opt[0] == 'set':
 				optval = [opt[3]]
-			else:
-				continue
+			else: # pragma: no cover - unexpected
+				self.fail('Unexpected config-token %r in stream' % (opt,))
 			for optval in optval:
 				if opt[2] == "prefregex":
 					self.filter.prefRegex = optval
@ -115,11 +115,13 @@ class FilterSamplesRegex(unittest.TestCase):

 		# test regexp contains greedy catch-all before <HOST>, that is
 		# not hard-anchored at end or has not precise sub expression after <HOST>:
-		for fr in self.filter.getFailRegex():
+		regexList = self.filter.getFailRegex()
+		for fr in regexList:
 			if RE_WRONG_GREED.search(fr): # pragma: no cover
 				raise AssertionError("Following regexp of \"%s\" contains greedy catch-all before <HOST>, "
 					"that is not hard-anchored at end or has not precise sub expression after <HOST>:\n%s" %
 					(name, str(fr).replace(RE_HOST, '<HOST>')))
+		return regexList

 def testSampleRegexsFactory(name, basedir):
 	def testFilter(self):
@ -128,8 +130,17 @@ def testSampleRegexsFactory(name, basedir):
 			os.path.isfile(os.path.join(TEST_FILES_DIR, "logs", name)),
 			"No sample log file available for '%s' filter" % name)

-		regexsUsed = set()
+		regexList = None
+		regexsUsedIdx = set()
+		regexsUsedRe = set()
 		filenames = [name]
+
+		def _testMissingSamples():
+			for failRegexIndex, failRegex in enumerate(regexList):
+				self.assertTrue(
+					failRegexIndex in regexsUsedIdx or failRegex in regexsUsedRe,
+					"Regex for filter '%s' has no samples: %i: %r" %
+						(name, failRegexIndex, failRegex))
 		i = 0
 		while i < len(filenames):
 			filename = filenames[i]; i += 1;
@ -143,29 +154,37 @@ def testSampleRegexsFactory(name, basedir):
 						faildata = json.loads(jsonREMatch.group(2))
 						# filterOptions - dict in JSON to control filter options (e. g. mode, etc.):
 						if jsonREMatch.group(1) == 'filterOptions':
+							# another filter mode - we should check previous also:
+							if self.filter is not None:
+								_testMissingSamples()
+								regexsUsedIdx = set() # clear used indices (possible overlapping by mode change)
+							# read filter with another setting:
 							self.filter = None
-							self._readFilter(name, basedir, opts=faildata)
+							regexList = self._readFilter(name, basedir, opts=faildata)
 							continue
 						# addFILE - filename to "include" test-files should be additionally parsed:
 						if jsonREMatch.group(1) == 'addFILE':
 							filenames.append(faildata)
 							continue
 						# failJSON - faildata contains info of the failure to check it.
-					except ValueError as e:
+					except ValueError as e: # pragma: no cover - we've valid json's
 						raise ValueError("%s: %s:%i" %
 							(e, logFile.filename(), logFile.filelineno()))
 					line = next(logFile)
 				elif line.startswith("#") or not line.strip():
 					continue
-				else:
+				else: # pragma: no cover - normally unreachable
 					faildata = {}

 				if self.filter is None:
-					self._readFilter(name, basedir, opts=None)
+					regexList = self._readFilter(name, basedir, opts=None)

 				try:
 					ret = self.filter.processLine(line)
 					if not ret:
+						# Bypass if filter constraint specified:
+						if faildata.get('filter') and name != faildata.get('filter'):
+							continue
 						# Check line is flagged as none match
 						self.assertFalse(faildata.get('match', True),
 							"Line not matched when should have")
@ -174,7 +193,8 @@ def testSampleRegexsFactory(name, basedir):
 					failregex, fid, fail2banTime, fail = ret[0]
 					# Bypass no failure helpers-regexp:
 					if not faildata.get('match', False) and (fid is None or fail.get('nofail')):
-						regexsUsed.add(failregex)
+						regexsUsedIdx.add(failregex)
+						regexsUsedRe.add(regexList[failregex])
 						continue

 					# Check line is flagged to match
@ -183,13 +203,13 @@ def testSampleRegexsFactory(name, basedir):
 					self.assertEqual(len(ret), 1,
 						"Multiple regexs matched %r" % (map(lambda x: x[0], ret)))

-					# Fallback for backwards compatibility (previously no fid, was host only):
-					if faildata.get("host", None) is not None and fail.get("host", None) is None:
-						fail["host"] = fid
 					# Verify match captures (at least fid/host) and timestamp as expected
 					for k, v in faildata.iteritems():
-						if k not in ("time", "match", "desc"):
+						if k not in ("time", "match", "desc", "filter"):
 							fv = fail.get(k, None)
+							# Fallback for backwards compatibility (previously no fid, was host only):
+							if k == "host" and fv is None:
+								fv = fid
 							self.assertEqual(fv, v)

 					t = faildata.get("time", None)
@ -208,16 +228,13 @@ def testSampleRegexsFactory(name, basedir):
 						jsonTime, time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(jsonTime)),
 						fail2banTime - jsonTime) )

-					regexsUsed.add(failregex)
+					regexsUsedIdx.add(failregex)
+					regexsUsedRe.add(regexList[failregex])
 				except AssertionError as e: # pragma: no cover
 					raise AssertionError("%s on: %s:%i, line:\n%s" % (
 								e, logFile.filename(), logFile.filelineno(), line))

-		for failRegexIndex, failRegex in enumerate(self.filter.getFailRegex()):
-			self.assertTrue(
-				failRegexIndex in regexsUsed,
-				"Regex for filter '%s' has no samples: %i: %r" %
-					(name, failRegexIndex, failRegex))
+		_testMissingSamples()

 	return testFilter

--- a/fail2ban/tests/servertestcase.py
+++ b/fail2ban/tests/servertestcase.py
@ -449,6 +449,16 @@ class Transmitter(TransmitterBase):
 			self.transm.proceed(["set", self.jailName, "delignoreip", value]),
 			(0, [value]))

+		self.assertEqual(
+			self.transm.proceed(["get", self.jailName, "ignoreself"]),
+			(0, True))
+		self.assertEqual(
+			self.transm.proceed(["set", self.jailName, "ignoreself", False]),
+			(0, False))
+		self.assertEqual(
+			self.transm.proceed(["get", self.jailName, "ignoreself"]),
+			(0, False))
+
 	def testJailIgnoreCommand(self):
 		self.setGetTest("ignorecommand", "bin ", jail=self.jailName)

@ -1073,16 +1083,16 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 				action.start()
 				# test ban ip4 :
 				logSys.debug('# === ban-ipv4 ==='); self.pruneLog()
-				action.ban({'ip': IPAddr('192.0.2.1')})
+				action.ban({'ip': IPAddr('192.0.2.1'), 'family': 'inet4'})
 				# test unban ip4 :
 				logSys.debug('# === unban ipv4 ==='); self.pruneLog()
-				action.unban({'ip': IPAddr('192.0.2.1')})
+				action.unban({'ip': IPAddr('192.0.2.1'), 'family': 'inet4'})
 				# test ban ip6 :
 				logSys.debug('# === ban ipv6 ==='); self.pruneLog()
-				action.ban({'ip': IPAddr('2001:DB8::')})
+				action.ban({'ip': IPAddr('2001:DB8::'), 'family': 'inet6'})
 				# test unban ip6 :
 				logSys.debug('# === unban ipv6 ==='); self.pruneLog()
-				action.unban({'ip': IPAddr('2001:DB8::')})
+				action.unban({'ip': IPAddr('2001:DB8::'), 'family': 'inet6'})
 				# test stop :
 				logSys.debug('# === stop ==='); self.pruneLog()
 				action.stop()
@ -1181,17 +1191,50 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 			#   'start', 'stop' - should be found (logged) on action start/stop,
 			#   etc.
 			testJailsActions = (
+				# dummy --
+				('j-dummy', 'dummy[name=%(__name__)s, init="==", target="/tmp/fail2ban.dummy"]', {
+					'ip4': ('family: inet4',), 'ip6': ('family: inet6',),
+					'start': (
+						'`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- started"`',
+					), 
+					'flush': (
+						'`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- clear all"`',
+					),
+					'stop': (
+						'`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- stopped"`',
+					),
+					'ip4-check': (),
+					'ip6-check': (),
+					'ip4-ban': (
+						'`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- banned 192.0.2.1 (family: inet4)"`',
+					),
+					'ip4-unban': (
+						'`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- unbanned 192.0.2.1 (family: inet4)"`',
+					),
+					'ip6-ban': (
+						'`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- banned 2001:db8:: (family: inet6)"`',
+					),
+					'ip6-unban': (
+						'`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- unbanned 2001:db8:: (family: inet6)"`',
+					),					
+				}),
 				# iptables-multiport --
 				('j-w-iptables-mp', 'iptables-multiport[name=%(__name__)s, bantime="10m", port="http,https", protocol="tcp", chain="INPUT"]', {
 					'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'),
-					'start': (
+					'ip4-start': (
 						"`iptables -w -N f2b-j-w-iptables-mp`",
 						"`iptables -w -A f2b-j-w-iptables-mp -j RETURN`",
 						"`iptables -w -I INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`",
+					), 
+					'ip6-start': (
 						"`ip6tables -w -N f2b-j-w-iptables-mp`",
 						"`ip6tables -w -A f2b-j-w-iptables-mp -j RETURN`",
 						"`ip6tables -w -I INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`",
 					),
+					'flush': (
+						"`iptables -w -F f2b-j-w-iptables-mp`",
+						"`ip6tables -w -F f2b-j-w-iptables-mp`",
+					),
 					'stop': (
 						"`iptables -w -D INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`",
 						"`iptables -w -F f2b-j-w-iptables-mp`",
@ -1222,14 +1265,20 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 				# iptables-allports --
 				('j-w-iptables-ap', 'iptables-allports[name=%(__name__)s, bantime="10m", protocol="tcp", chain="INPUT"]', {
 					'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'),
-					'start': (
+					'ip4-start': (
 						"`iptables -w -N f2b-j-w-iptables-ap`",
 						"`iptables -w -A f2b-j-w-iptables-ap -j RETURN`",
 						"`iptables -w -I INPUT -p tcp -j f2b-j-w-iptables-ap`",
+					), 
+					'ip6-start': (
 						"`ip6tables -w -N f2b-j-w-iptables-ap`",
 						"`ip6tables -w -A f2b-j-w-iptables-ap -j RETURN`",
 						"`ip6tables -w -I INPUT -p tcp -j f2b-j-w-iptables-ap`",
 					),
+					'flush': (
+						"`iptables -w -F f2b-j-w-iptables-ap`",
+						"`ip6tables -w -F f2b-j-w-iptables-ap`",
+					),
 					'stop': (
 						"`iptables -w -D INPUT -p tcp -j f2b-j-w-iptables-ap`",
 						"`iptables -w -F f2b-j-w-iptables-ap`",
@ -1260,12 +1309,18 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 				# iptables-ipset-proto6 --
 				('j-w-iptables-ipset', 'iptables-ipset-proto6[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain="INPUT"]', {
 					'ip4': (' f2b-j-w-iptables-ipset ',), 'ip6': (' f2b-j-w-iptables-ipset6 ',),
-					'start': (
+					'ip4-start': (
 						"`ipset create f2b-j-w-iptables-ipset hash:ip timeout 600`",
 						"`iptables -w -I INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable`",
+					), 
+					'ip6-start': (
 						"`ipset create f2b-j-w-iptables-ipset6 hash:ip timeout 600 family inet6`",
 						"`ip6tables -w -I INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`",
 					),
+					'flush': (
+						"`ipset flush f2b-j-w-iptables-ipset`",
+						"`ipset flush f2b-j-w-iptables-ipset6`",
+					),
 					'stop': (
 						"`iptables -w -D INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable`",
 						"`ipset flush f2b-j-w-iptables-ipset`",
@ -1292,12 +1347,18 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 				# iptables-ipset-proto6-allports --
 				('j-w-iptables-ipset-ap', 'iptables-ipset-proto6-allports[name=%(__name__)s, bantime="10m", chain="INPUT"]', {
 					'ip4': (' f2b-j-w-iptables-ipset-ap ',), 'ip6': (' f2b-j-w-iptables-ipset-ap6 ',),
-					'start': (
+					'ip4-start': (
 						"`ipset create f2b-j-w-iptables-ipset-ap hash:ip timeout 600`",
 						"`iptables -w -I INPUT -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`",
+					), 
+					'ip6-start': (
 						"`ipset create f2b-j-w-iptables-ipset-ap6 hash:ip timeout 600 family inet6`",
 						"`ip6tables -w -I INPUT -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`",
 					),
+					'flush': (
+						"`ipset flush f2b-j-w-iptables-ipset-ap`",
+						"`ipset flush f2b-j-w-iptables-ipset-ap6`",
+					),
 					'stop': (
 						"`iptables -w -D INPUT -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`",
 						"`ipset flush f2b-j-w-iptables-ipset-ap`",
@ -1324,14 +1385,20 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 				# iptables --
 				('j-w-iptables', 'iptables[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain="INPUT"]', {
 					'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'),
-					'start': (
+					'ip4-start': (
 						"`iptables -w -N f2b-j-w-iptables`",
 						"`iptables -w -A f2b-j-w-iptables -j RETURN`",
 						"`iptables -w -I INPUT -p tcp --dport http -j f2b-j-w-iptables`",
+					), 
+					'ip6-start': (
 						"`ip6tables -w -N f2b-j-w-iptables`",
 						"`ip6tables -w -A f2b-j-w-iptables -j RETURN`",
 						"`ip6tables -w -I INPUT -p tcp --dport http -j f2b-j-w-iptables`",
 					),
+					'flush': (
+						"`iptables -w -F f2b-j-w-iptables`",
+						"`ip6tables -w -F f2b-j-w-iptables`",
+					),
 					'stop': (
 						"`iptables -w -D INPUT -p tcp --dport http -j f2b-j-w-iptables`",
 						"`iptables -w -F f2b-j-w-iptables`",
@ -1362,14 +1429,20 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 				# iptables-new --
 				('j-w-iptables-new', 'iptables-new[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain="INPUT"]', {
 					'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'),
-					'start': (
+					'ip4-start': (
 						"`iptables -w -N f2b-j-w-iptables-new`",
 						"`iptables -w -A f2b-j-w-iptables-new -j RETURN`",
 						"`iptables -w -I INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`",
+					), 
+					'ip6-start': (
 						"`ip6tables -w -N f2b-j-w-iptables-new`",
 						"`ip6tables -w -A f2b-j-w-iptables-new -j RETURN`",
 						"`ip6tables -w -I INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`",
 					),
+					'flush': (
+						"`iptables -w -F f2b-j-w-iptables-new`",
+						"`ip6tables -w -F f2b-j-w-iptables-new`",
+					),
 					'stop': (
 						"`iptables -w -D INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`",
 						"`iptables -w -F f2b-j-w-iptables-new`",
@ -1400,8 +1473,10 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 				# iptables-xt_recent-echo --
 				('j-w-iptables-xtre', 'iptables-xt_recent-echo[name=%(__name__)s, bantime="10m", chain="INPUT"]', {
 					'ip4': ('`iptables ', '/f2b-j-w-iptables-xtre`'), 'ip6': ('`ip6tables ', '/f2b-j-w-iptables-xtre6`'),
-					'start': (
+					'ip4-start': (
 						"`if [ `id -u` -eq 0 ];then iptables -w -I INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre -j REJECT --reject-with icmp-port-unreachable;fi`",
+					), 
+					'ip6-start': (
 						"`if [ `id -u` -eq 0 ];then ip6tables -w -I INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre6 -j REJECT --reject-with icmp6-port-unreachable;fi`",
 					),
 					'stop': (
@ -1430,7 +1505,7 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 					),
 				}),
 				# pf default -- multiport on default port (tag <port> set in jail.conf, but not in this test case)
-				('j-w-pf', 'pf[name=%(__name__)s]', {
+				('j-w-pf', 'pf[name=%(__name__)s, actionstart_on_demand=false]', {
 					'ip4': (), 'ip6': (),
 					'start': (
 						'`echo "table <f2b-j-w-pf> persist counters" | pfctl -f-`',
@ -1467,13 +1542,14 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 					'ip6-ban':   ("`pfctl -t f2b-j-w-pf-mp -T add 2001:db8::`",),
 					'ip6-unban': ("`pfctl -t f2b-j-w-pf-mp -T delete 2001:db8::`",),
 				}),
-				# pf allports --
-				('j-w-pf-ap', 'pf[actiontype=<allports>][name=%(__name__)s]', {
+				# pf allports -- test additionally "actionstart_on_demand" was set to true
+				('j-w-pf-ap', 'pf[actiontype=<allports>, actionstart_on_demand=true][name=%(__name__)s]', {
 					'ip4': (), 'ip6': (),
-					'start': (
+					'ip4-start': (
 						'`echo "table <f2b-j-w-pf-ap> persist counters" | pfctl -f-`',
 						'`echo "block proto tcp from <f2b-j-w-pf-ap> to any" | pfctl -f-`',
 					),
+					'ip6-start': (), # the same as ipv4
 					'stop': (
 						'`pfctl -sr 2>/dev/null | grep -v f2b-j-w-pf-ap | pfctl -f-`',
 						'`pfctl -t f2b-j-w-pf-ap -T flush`',
@ -1489,10 +1565,12 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 				# firewallcmd-multiport --
 				('j-w-fwcmd-mp', 'firewallcmd-multiport[name=%(__name__)s, bantime="10m", port="http,https", protocol="tcp", chain="INPUT"]', {
 					'ip4': (' ipv4 ', 'icmp-port-unreachable'), 'ip6': (' ipv6 ', 'icmp6-port-unreachable'),
-					'start': (
+					'ip4-start': (
 						"`firewall-cmd --direct --add-chain ipv4 filter f2b-j-w-fwcmd-mp`",
 						"`firewall-cmd --direct --add-rule ipv4 filter f2b-j-w-fwcmd-mp 1000 -j RETURN`",
 						"`firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports http,https -j f2b-j-w-fwcmd-mp`",
+					), 
+					'ip6-start': (
 						"`firewall-cmd --direct --add-chain ipv6 filter f2b-j-w-fwcmd-mp`",
 						"`firewall-cmd --direct --add-rule ipv6 filter f2b-j-w-fwcmd-mp 1000 -j RETURN`",
 						"`firewall-cmd --direct --add-rule ipv6 filter INPUT 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports http,https -j f2b-j-w-fwcmd-mp`",
@ -1527,10 +1605,12 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 				# firewallcmd-allports --
 				('j-w-fwcmd-ap', 'firewallcmd-allports[name=%(__name__)s, bantime="10m", protocol="tcp", chain="INPUT"]', {
 					'ip4': (' ipv4 ', 'icmp-port-unreachable'), 'ip6': (' ipv6 ', 'icmp6-port-unreachable'),
-					'start': (
+					'ip4-start': (
 						"`firewall-cmd --direct --add-chain ipv4 filter f2b-j-w-fwcmd-ap`",
 						"`firewall-cmd --direct --add-rule ipv4 filter f2b-j-w-fwcmd-ap 1000 -j RETURN`",
 						"`firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -j f2b-j-w-fwcmd-ap`",
+					), 
+					'ip6-start': (
 						"`firewall-cmd --direct --add-chain ipv6 filter f2b-j-w-fwcmd-ap`",
 						"`firewall-cmd --direct --add-rule ipv6 filter f2b-j-w-fwcmd-ap 1000 -j RETURN`",
 						"`firewall-cmd --direct --add-rule ipv6 filter INPUT 0 -j f2b-j-w-fwcmd-ap`",
@ -1565,9 +1645,11 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 				# firewallcmd-ipset --
 				('j-w-fwcmd-ipset', 'firewallcmd-ipset[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain="INPUT"]', {
 					'ip4': (' f2b-j-w-fwcmd-ipset ',), 'ip6': (' f2b-j-w-fwcmd-ipset6 ',),
-					'start': (
+					'ip4-start': (
 						"`ipset create f2b-j-w-fwcmd-ipset hash:ip timeout 600`",
 						"`firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset src -j REJECT --reject-with icmp-port-unreachable`",
+					), 
+					'ip6-start': (
 						"`ipset create f2b-j-w-fwcmd-ipset6 hash:ip timeout 600`",
 						"`firewall-cmd --direct --add-rule ipv6 filter INPUT 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`",
 					),
@ -1613,6 +1695,10 @@ class ServerConfigReaderTests(LogCaptureTestCase):

 			jails = server._Server__jails

+			tickets = {
+				'ip4': BanTicket('192.0.2.1'),
+				'ip6': BanTicket('2001:DB8::'),
+			}
 			for jail, act, tests in testJailsActions:
 				# print(jail, jails[jail])
 				for a in jails[jail].actions:
@ -1626,27 +1712,43 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 					# test start :
 					self.pruneLog('# === start ===')
 					action.start()
-					self.assertLogged(*tests['start'], all=True)
+					if tests.get('start'):
+						self.assertLogged(*tests['start'], all=True)
+					else:
+						self.assertNotLogged(*tests['ip4-start']+tests['ip6-start'], all=True)
+					ainfo = {
+						'ip4': _actions.Actions.ActionInfo(tickets['ip4'], jails[jail]),
+						'ip6': _actions.Actions.ActionInfo(tickets['ip6'], jails[jail]),
+					}
 					# test ban ip4 :
 					self.pruneLog('# === ban-ipv4 ===')
-					action.ban({'ip': IPAddr('192.0.2.1')})
+					action.ban(ainfo['ip4'])
+					if tests.get('ip4-start'): self.assertLogged(*tests['ip4-start'], all=True)
+					if tests.get('ip6-start'): self.assertNotLogged(*tests['ip6-start'], all=True)
 					self.assertLogged(*tests['ip4-check']+tests['ip4-ban'], all=True)
 					self.assertNotLogged(*tests['ip6'], all=True)
 					# test unban ip4 :
 					self.pruneLog('# === unban ipv4 ===')
-					action.unban({'ip': IPAddr('192.0.2.1')})
+					action.unban(ainfo['ip4'])
 					self.assertLogged(*tests['ip4-check']+tests['ip4-unban'], all=True)
 					self.assertNotLogged(*tests['ip6'], all=True)
 					# test ban ip6 :
 					self.pruneLog('# === ban ipv6 ===')
-					action.ban({'ip': IPAddr('2001:DB8::')})
+					action.ban(ainfo['ip6'])
+					if tests.get('ip6-start'): self.assertLogged(*tests['ip6-start'], all=True)
+					if tests.get('ip4-start'): self.assertNotLogged(*tests['ip4-start'], all=True)
 					self.assertLogged(*tests['ip6-check']+tests['ip6-ban'], all=True)
 					self.assertNotLogged(*tests['ip4'], all=True)
 					# test unban ip6 :
 					self.pruneLog('# === unban ipv6 ===')
-					action.unban({'ip': IPAddr('2001:DB8::')})
+					action.unban(ainfo['ip6'])
 					self.assertLogged(*tests['ip6-check']+tests['ip6-unban'], all=True)
 					self.assertNotLogged(*tests['ip4'], all=True)
+					# test flush for actions should supported this:
+					if tests.get('flush'):
+						self.pruneLog('# === flush ===')
+						action.flush()
+						self.assertLogged(*tests['flush'], all=True)
 					# test stop :
 					self.pruneLog('# === stop ===')
 					action.stop()
@ -1655,7 +1757,7 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 		def _executeMailCmd(self, realCmd, timeout=60):
 			# replace pipe to mail with pipe to cat:
 			realCmd = re.sub(r'\)\s*\|\s*mail\b([^\n]*)',
-				r' echo mail \1 ) | cat', realCmd)
+				r') | cat; printf "\\n... | "; echo mail \1', realCmd)
 			# replace abuse retrieving (possible no-network), just replace first occurrence of 'dig...':
 			realCmd = re.sub(r'\bADDRESSES=\$\(dig\s[^\n]+',
 				lambda m: 'ADDRESSES="abuse-1@abuse-test-server, abuse-2@abuse-test-server"',
@ -1688,7 +1790,7 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 				# complain --
 				('j-complain-abuse', 
 					'complain['
-					  'name=%(__name__)s, grepopts="-m 1", grepmax=2, mailcmd="mail -s Hostname: <ip-host> - ",' +
+					  'name=%(__name__)s, grepopts="-m 1", grepmax=2, mailcmd="mail -s \'Hostname: <ip-host>, family: <family>\' - ",' +
 					  # test reverse ip:
 					  'debug=1,' +
 						# 2 logs to test grep from multiple logs:
@ -1703,14 +1805,14 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 						'testcase01.log:Dec 31 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 87.142.124.10',
 						'testcase01a.log:Dec 31 11:55:01 [sshd] error: PAM: Authentication failure for test from 87.142.124.10',
 						# both abuse mails should be separated with space:
-						'mail -s Hostname: test-host - Abuse from 87.142.124.10 abuse-1@abuse-test-server abuse-2@abuse-test-server',
+						'mail -s Hostname: test-host, family: inet4 - Abuse from 87.142.124.10 abuse-1@abuse-test-server abuse-2@abuse-test-server',
 					),
 					'ip6-ban': (
 						# test reverse ip:
 						'try to resolve 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.abuse-contacts.abusix.org',
 						'Lines containing failures of 2001:db8::1 (max 2)',
 						# both abuse mails should be separated with space:
-						'mail -s Hostname: test-host - Abuse from 2001:db8::1 abuse-1@abuse-test-server abuse-2@abuse-test-server',
+						'mail -s Hostname: test-host, family: inet6 - Abuse from 2001:db8::1 abuse-1@abuse-test-server abuse-2@abuse-test-server',
 					),
 				}),
 			)
--- a/fail2ban/tests/utils.py
+++ b/fail2ban/tests/utils.py
@ -262,7 +262,10 @@ def initTests(opts):

 	# persistently set time zone to CET (used in zone-related test-cases),
 	# yoh: we need to adjust TZ to match the one used by Cyril so all the timestamps match
-	os.environ['TZ'] = 'Europe/Zurich'
+	# This offset corresponds to Europe/Zurich timezone.  Specifying it
+	# explicitly allows to avoid requiring tzdata package to be installed during
+	# testing.   See https://bugs.debian.org/855920 for more information
+	os.environ['TZ'] = 'CET-01CEST-02,M3.5.0,M10.5.0'
 	time.tzset()
 	# set alternate now for time related test cases:
 	MyTime.setAlternateNow(TEST_NOW)
--- a/man/jail.conf.5
+++ b/man/jail.conf.5
@ -199,11 +199,14 @@ Arguments can be passed to actions to override the default values from the [Init
 Values can also be quoted (required when value includes a ","). More that one action can be specified (in separate lines).
 .RE
 .TP
+.B ignoreself
+boolean value (default true) indicates the banning of own IP addresses should be prevented
+.TP
 .B ignoreip
-list of IPs not to ban. They can include a CIDR mask too.
+list of IPs not to ban. They can include a DNS resp. CIDR mask too. The option affects additionally to \fBignoreself\fR (if true) and don't need to contain own DNS resp. IPs of the running host.
 .TP
 .B ignorecommand
-command that is executed to determine if the current candidate IP for banning should not be banned.
+command that is executed to determine if the current candidate IP for banning (or failure-ID for raw IDs) should not be banned. The option affects additionally to \fBignoreself\fR and \fBignoreip\fR and will be first executed if both don't hit.
 .br
 IP will not be banned if command returns successfully (exit code 0).
 Like ACTION FILES, tags like <ip> are can be included in the ignorecommand value and will be substituted before execution. Currently only <ip> is supported however more will be added later.