Merge branch 'upstream'

.gitattributes

@@ -0,0 +1 @@
+ChangeLog linguist-language=Markdown

.github/FUNDING.yml

@@ -0,0 +1,4 @@
+# These are supported funding model platforms
+
+github: [sebres]
+custom: [paypal.me/sebres]

.github/ISSUE_TEMPLATE.md

@@ -1,49 +0,0 @@
-_We will be very grateful, if your problem was described as completely as possible,
-enclosing excerpts from logs (if possible within DEBUG mode, if no errors evident
-within INFO mode), and configuration in particular of effected relevant settings
-(e.g., with ` fail2ban-client -d | grep 'affected-jail-name' ` for a particular
-jail troubleshooting).
-Thank you in advance for the details, because such issues like "It does not work" 
-alone could not help to resolve anything!
-Thanks! (remove this paragraph and other comments upon reading)_
-
-### Environment:
-
-_Fill out and check (`[x]`) the boxes which apply. If your Fail2Ban version is outdated, 
-and you can't verify that the issue persists in the recent release, better seek support 
-from the distribution you obtained Fail2Ban from_
-
-- Fail2Ban version (including any possible distribution suffixes):
-- OS, including release name/version: 
-- [ ] Fail2Ban installed via OS/distribution mechanisms
-- [ ] You have not applied any additional foreign patches to the codebase
-- [ ] Some customizations were done to the configuration (provide details below is so)
-
-### The issue:
-
-_Summary here_
-
-#### Steps to reproduce
-
-#### Expected behavior
-
-#### Observed behavior
-
-#### Any additional information
-
-### Configuration, dump and another helpful excerpts
-
-#### Any customizations done to /etc/fail2ban/ configuration
-```
-```
-
-#### Relevant parts of /var/log/fail2ban.log file:
-_preferably obtained while running fail2ban with `loglevel = 4`_
-
-```
-```
-
-#### Relevant lines from monitored log files in question:
-
-```
-```

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,70 @@
+---
+name: Bug report
+about: Report a bug within the fail2ban engines (not filters or jails)
+title: '[BR]: '
+labels: bug
+assignees: ''
+
+---
+
+<!--
+  - Before reporting, please make sure to search the open and closed issues for any reports in the past.
+  - Use this issue template to report a bug in the fail2ban engine (not in a filter or jail).
+  - If you want to request a feature or a new filter, please use "Feature request" or "Filter request" instead.
+  - If you have rather some question, please open or join to some discussion.
+
+  We will be very grateful, if your problem was described as completely as possible,
+  enclosing excerpts from logs (if possible within DEBUG mode, if no errors evident
+  within INFO mode), and configuration in particular of effected relevant settings
+  (e.g., with ` fail2ban-client -d | grep 'affected-jail-name' ` for a particular
+  jail troubleshooting).
+  Thank you in advance for the details, because such issues like "It does not work" 
+  alone could not help to resolve anything!
+  Thanks! 
+  (you can remove this paragraph and other comments upon reading)
+-->
+
+### Environment:
+
+<!--
+  Fill out and check (`[x]`) the boxes which apply. If your Fail2Ban version is outdated, 
+  and you can't verify that the issue persists in the recent release, better seek support 
+  from the distribution you obtained Fail2Ban from
+-->
+
+- Fail2Ban version <!-- including any possible distribution suffixes --> : 
+- OS, including release name/version : 
+- [ ] Fail2Ban installed via OS/distribution mechanisms
+- [ ] You have not applied any additional foreign patches to the codebase
+- [ ] Some customizations were done to the configuration (provide details below is so)
+
+### The issue:
+
+<!-- summary here -->
+
+#### Steps to reproduce
+
+#### Expected behavior
+
+#### Observed behavior
+
+#### Any additional information
+
+
+### Configuration, dump and another helpful excerpts
+
+#### Any customizations done to /etc/fail2ban/ configuration
+<!-- put your configuration excerpts between next 2 lines -->
+```
+```
+
+#### Relevant parts of /var/log/fail2ban.log file:
+<!-- preferably obtained while running fail2ban with `loglevel = 4` -->
+<!-- put your log excerpt between next 2 lines -->
+```
+```
+
+#### Relevant lines from monitored log files:
+<!-- put your log excerpt between next 2 lines -->
+```
+```

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,35 @@
+---
+name: Feature request
+about: Suggest an idea or an enhancement for this project
+title: '[RFE]: '
+labels: enhancement
+assignees: ''
+
+---
+
+<!--
+  - Before requesting, please make sure to search the open and closed issues for any requests in the past.
+  - Use this issue template to request a feature in the fail2ban engine (not a new filter or jail).
+  - If you want to request a new filter or failregex, please use "Filter request" instead.
+  - If you have rather some question, please open or join to some discussion.
+-->
+
+#### Feature request type
+<!--
+  Please provide a summary description of the feature request.
+-->
+
+#### Description
+<!--
+  Please describe the feature in more detail.
+-->
+
+#### Considered alternatives
+<!--
+  A clear and concise description of any alternative solutions or features you've considered.
+-->
+
+#### Any additional information
+<!--
+  Add any other context or screenshots about the feature request here.
+-->

.github/ISSUE_TEMPLATE/filter_request.md

@@ -0,0 +1,59 @@
+---
+name: Filter request
+about: Request a new jail or filter to be supported or existing filter extended with new failregex
+title: '[FR]: '
+labels: filter-request
+assignees: ''
+
+---
+
+<!--
+  - Before requesting, please make sure to search the open and closed issues for any requests in the past.
+  - Sometimes failregex have been already requested before but are not implemented yet due to various reasons.
+  - If there are no hits for your concerns, please proceed otherwise add a comment to the related issue (also if it is closed).
+  - If you want to request a new feature, please use "Feature request" instead.
+  - If you have rather some question, please open or join to some discussion.
+-->
+
+### Environment:
+
+<!--
+  Fill out and check (`[x]`) the boxes which apply.
+-->
+
+- Fail2Ban version <!-- including any possible distribution suffixes --> : 
+- OS, including release name/version : 
+
+#### Service, project or product which log or journal should be monitored
+
+- Name of filter or jail in Fail2Ban (if already exists) : 
+- Service, project or product name, including release name/version : 
+- Repository or URL (if known) : 
+- Service type : 
+- Ports and protocols the service is listening : 
+
+#### Log or journal information
+<!-- Delete unrelated group -->
+
+<!-- Log file -->
+
+- Log file name(s) : 
+
+<!-- Systemd journal -->
+
+- Journal identifier or unit name : 
+
+#### Any additional information
+
+
+### Relevant lines from monitored log files:
+
+#### failures in sense of fail2ban filter (fail2ban must match):
+<!-- put your log excerpt between next 2 lines -->
+```
+```
+
+#### legitimate messages (fail2ban should not consider as failures):
+<!-- put your log excerpt between next 2 lines -->
+```
+```

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,7 +1,8 @@
 Before submitting your PR, please review the following checklist:
 
 - [ ] **CHOOSE CORRECT BRANCH**: if filing a bugfix/enhancement
-      against 0.9.x series, choose `master` branch
+      against certain release version, choose `0.9`, `0.10` or `0.11` branch,
+      for dev-edition use `master` branch
 - [ ] **CONSIDER adding a unit test** if your PR resolves an issue
 - [ ] **LIST ISSUES** this PR resolves
 - [ ] **MAKE SURE** this PR doesn't break existing tests

.github/workflows/main.yml

@@ -22,7 +22,7 @@ jobs:
     runs-on: ubuntu-20.04
     strategy:
       matrix:
-        python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, pypy2, pypy3]
+        python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, '3.10', '3.11.0-beta.3', pypy2, pypy3]
       fail-fast: false
     # Steps represent a sequence of tasks that will be executed as part of the job
     steps:
@@ -34,33 +34,67 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
 
+      - name: Grant systemd-journal access
+        run: sudo usermod -a -G systemd-journal "$USER" || echo 'no systemd-journal access'
+
       - name: Python version
         run: |
           F2B_PY=$(python -c "import sys; print(sys.version)")
-          echo "Python: ${{ matrix.python-version }} -- $F2B_PY"
+          echo "Python: ${{ matrix.python-version }} -- ${F2B_PY/$'\n'/ }"
+          F2B_PYV=$(echo "${F2B_PY}" | grep -oP '^\d+(?:\.\d+)')
           F2B_PY=${F2B_PY:0:1}
-          echo "Set F2B_PY=$F2B_PY"
+          echo "Set F2B_PY=$F2B_PY, F2B_PYV=$F2B_PYV"
           echo "F2B_PY=$F2B_PY" >> $GITHUB_ENV
+          echo "F2B_PYV=$F2B_PYV" >> $GITHUB_ENV
+          # for GHA we need to monitor all journals, since it cannot be found using SYSTEM_ONLY(4):
+          echo "F2B_SYSTEMD_DEFAULT_FLAGS=0" >> $GITHUB_ENV
 
       - name: Install dependencies
         run: |
-          python -m pip install --upgrade pip
+          if [[ "$F2B_PY" = 3 ]]; then python -m pip install --upgrade pip || echo "can't upgrade pip"; fi
           if [[ "$F2B_PY" = 3 ]] && ! command -v 2to3x -v 2to3 > /dev/null; then
-            pip install 2to3
+            #pip install 2to3
+            sudo apt-get -y install 2to3
           fi
-          pip install systemd-python || echo 'systemd not available'
+          #sudo apt-get -y install python${F2B_PY/2/}-pyinotify || echo 'inotify not available'
-          pip install pyinotify || echo 'inotify not available'
+          python -m pip install pyinotify || echo 'inotify not available'
+          #sudo apt-get -y install python${F2B_PY/2/}-systemd || echo 'systemd not available'
+          sudo apt-get -y install libsystemd-dev || echo 'systemd dependencies seems to be unavailable'
+          python -m pip install systemd-python || echo 'systemd not available'
+          #readline if available as module:
+          python -c 'import readline' 2> /dev/null || python -m pip install readline || echo 'readline not available'
 
       - name: Before scripts
         run: |
           cd "$GITHUB_WORKSPACE"
           # Manually execute 2to3 for now
           if [[ "$F2B_PY" = 3 ]]; then echo "2to3 ..." && ./fail2ban-2to3; fi
+          _debug() { echo -n "$1 "; err=$("${@:2}" 2>&1) && echo 'OK' || echo -e "FAIL\n$err"; }
           # (debug) output current preferred encoding:
-          python -c 'import locale, sys; from fail2ban.helpers import PREFER_ENC; print(PREFER_ENC, locale.getpreferredencoding(), (sys.stdout and sys.stdout.encoding))'
+          _debug 'Encodings:' python -c 'import locale, sys; from fail2ban.helpers import PREFER_ENC; print(PREFER_ENC, locale.getpreferredencoding(), (sys.stdout and sys.stdout.encoding))'
+          # (debug) backend availabilities:
+          echo 'Backends:'
+          _debug '- systemd:' python -c 'from fail2ban.server.filtersystemd import FilterSystemd'
+          #_debug '- systemd (root): ' sudo python -c 'from fail2ban.server.filtersystemd import FilterSystemd'
+          _debug '- pyinotify:' python -c 'from fail2ban.server.filterpyinotify import FilterPyinotify'
 
       - name: Test suite
-        run: if [[ "$F2B_PY" = 2 ]]; then python setup.py test; else python bin/fail2ban-testcases --verbosity=2; fi
+        run: |
+          if [[ "$F2B_PY" = 2 ]]; then
+            python setup.py test
+          elif dpkg --compare-versions "$F2B_PYV" lt 3.10; then 
+            python bin/fail2ban-testcases --verbosity=2
+          else
+            echo "Skip systemd backend since systemd-python module must be fixed for python >= v.3.10 in GHA ..."
+            python bin/fail2ban-testcases --verbosity=2 -i "[sS]ystemd|[jJ]ournal"
+          fi
+
+      #- name: Test suite (debug some systemd tests only)
+         #run: python bin/fail2ban-testcases --verbosity=2 "[sS]ystemd|[jJ]ournal"
+         #run: python bin/fail2ban-testcases --verbosity=2 -l 5 "test_WrongChar"
+
+      - name: Build
+        run: python setup.py build
 
       #- name: Test initd scripts
       #  run: shellcheck -s bash -e SC1090,SC1091 files/debian-initd

.travis.yml

@@ -10,16 +10,8 @@ dist: xenial
 matrix:
   fast_finish: true
   include:
-  - python: 2.6
-    dist: trusty   # required for Python 2.6
   - python: 2.7
-    dist: trusty   # required for packages like gamin
+  #- python: pypy
-    name: 2.7 (trusty)
-  - python: 2.7
-    name: 2.7 (xenial)
-  - python: pypy
-  - python: 3.3
-    dist: trusty
   - python:  3.4
   - python:  3.5
   - python:  3.6
@@ -41,7 +33,8 @@ install:
   #   coverage
   - travis_retry pip install coverage
   #   coveralls (note coveralls doesn't support 2.6 now):
-  - if [[ $TRAVIS_PYTHON_VERSION != 2.6* ]]; then F2B_COV=1; else F2B_COV=0; fi
+  #- if [[ $TRAVIS_PYTHON_VERSION != 2.6* ]]; then F2B_COV=1; else F2B_COV=0; fi
+  - F2B_COV=1
   - if [[ "$F2B_COV" = 1 ]]; then travis_retry pip install coveralls; fi
   #   codecov:
   - travis_retry pip install codecov

ChangeLog

@@ -1,3 +1,4 @@
+<!-- vim: syntax=Markdown -->
                          __      _ _ ___ _
                         / _|__ _(_) |_  ) |__  __ _ _ _
                        |  _/ _` | | |/ /| '_ \/ _` | ' \
@@ -6,10 +7,171 @@
 Fail2Ban: Changelog
 ===================
 
+ver. 1.0.1 (2022/09/27) - energy-equals-mass-times-the-speed-of-light-squared
+-----------
+
+### Compatibility
+* the minimum supported python version is now 2.7, if you have previous python version
+  you can use the 0.11 version of fail2ban or upgrade python (or even build it from source).
+* potential incompatibility by parsing of options of `backend`, `filter` and `action` parameters (if they
+  are partially incorrect), because fail2ban could throw an error now (doesn't silently bypass it anymore).
+* due to fix for CVE-2021-32749 (GHSA-m985-3f3v-cwmm) the mailing action using mailutils may require extra configuration,
+  if it is not compatible or doesn't support `-E 'set escape'` (e. g. with `mailcmd` parameter), see gh-3059
+* automatic invocation of 2to3 is removed in setup now (gh-3098), there is also no option `--disable-2to3` anymore,
+  `./fail2ban-2to3` should be called outside before setup
+* to v.0.11:
+  - due to change of `actioncheck` behavior (gh-488), some actions can be incompatible as regards
+    the invariant check, if `actionban` or `actionunban` would not throw an error (exit code 
+    different from 0) in case of unsane environment.
+  - actions that have used tag `<ip>` (instead of `<fid>` or `<F-ID>`) to get failure-ID may become
+    incompatible, if filter uses IP-related tags (like `<ADDR>` or `<HOST>`) additionally to `<F-ID>`
+    and the values are different (gh-3217)
+
+### Fixes
+* theoretical RCE vulnerability in mailing action using mailutils (mail-whois), CVE-2021-32749, GHSA-m985-3f3v-cwmm
+* readline fixed to consider interim new-line character as part of code point in multi-byte logs
+  (e. g. unicode encoding like utf-16be, utf-16le);
+* [stability] solves race condition with uncontrolled growth of failure list (jail with too many matches,
+  that did not cause ban), behavior changed to ban ASAP, gh-2945
+* fixes search for the best datepattern - e. g. if line is too short, boundaries check for previously known
+  unprecise pattern may fail on incomplete lines (logging break-off, no flush, etc), gh-3020
+* [stability, performance] backend `systemd`:
+  - fixes error "local variable 'line' referenced before assignment", introduced in 55d7d9e2, gh-3097
+  - don't update database too often (every 10 ticks or ~ 10 seconds in production)
+  - fixes wrong time point of "in operation" mode, gh-2882
+  - better avoidance of landing in dead space by seeks over journals (improved seek to time)
+  - fixes missing space in message (tag `<matches>`) between timestamp and host if the message read from systemd journal, gh-3293
+* [stability] backend `pyinotify`: fixes sporadic runtime error "dictionary changed size during iteration"
+* several backends optimizations (in file and journal filters):
+  - don't need to wait if we still had log-entries from last iteration (which got interrupted for servicing)
+  - rewritten update log/journal position, it is more stable and faster now (fewer DB access and surely up-to-date at end)
+* `paths-debian.conf`:
+  - add debian path to roundcube error logs
+* `action.d/firewallcmd-*.conf` (multiport only): fixed port range selector, replacing `:` with `-`;"
+  reverted the incompatibility gh-3047 introduced in a038fd5, gh-2821, because this depends now on firewalld backend
+  (e. g. `-` vs. `:` related to `iptables` vs. `nftables`)
+* `action.d/nginx-block-map.conf`: reload nginx only if it is running (also avoid error in nginx-errorlog, gh-2949)
+* `action.d/ufw.conf`:
+  - fixed handling on IPv6 (using prepend, gh-2331, gh-3018)
+  - application names containing spaces can be used now (gh-656, gh-1532, gh-3018)
+* `filter.d/apache-fakegooglebot.conf`:
+  - better, more precise regex and datepattern (closes possible weakness like gh-3013)
+  - `filter.d/ignorecommands/apache-fakegooglebot` - added timeout parameter (default 55 seconds), avoid fail with timeout
+    (default 1 minute) by reverse lookup on some slow DNS services (googlebots must be resolved fast), gh-2951
+* `filter.d/apache-overflows.conf` - extended to match AH00126 error (Invalid URI ...), gh-2908
+* `filter.d/asterisk.conf` - add transport to asterisk RE: call rejection messages can have the transport prefixed to the IP address, gh-2913
+* `filter.d/courier-auth.conf`:
+  - consider optional port after IP, gh-3211
+  - regex is rewritten without catch-all's and right anchor, so it is more stable against further modifications now
+* `filter.d/dovecot.conf`:
+  - adjusted for updated dovecot log format with `read(size=...)` in message (gh-3210)
+  - parse everything in parenthesis by auth-worker info, e. g. can match (pid=...,uid=...) too (amend to gh-2553)
+  - extended to match prefix like `conn unix:auth-worker (uid=143): auth-worker<13247>:`
+    (authenticate from external service like exim), gh-2553
+  - fixed "Authentication failure" regex, matches "Password mismatch" in title case (gh-2880)
+* `filter.d/drupal-auth.conf` - more strict regex, extended to match "Login attempt failed from" (gh-2742)
+* `filter.d/exim-common.conf` - pid-prefix extended to match `mx1 exim[...]:` (gh-2553)
+* `filter.d/lighttpd-auth.conf` - adjusted to the current source code + avoiding catch-all's, etc (gh-3116)
+* `filter.d/named-refused.conf`:
+  - added support for alternate names (suffix), FreeIPA renames the BIND9 named daemon to named-pkcs11, gh-2636
+  - fixes prefix for messages from systemd journal (no mandatory space ahead, because don't have timestamp), gh-2899
+* `filter.d/nginx-*.conf` - added journalmatch to nginx filters, gh-2935
+* `filter.d/nsd.conf` - support for current log format, gh-2965
+* `filter.d/postfix.conf`: fixes and new vectors, review and combining several regex to single RE:
+    - mode `ddos` (and `aggressive`) extended:
+      * to consider abusive handling of clients hitting command limit, gh-3040
+      * to handle postscreen's PREGREET and HANGUP messages, gh-2898
+    - matches rejects with "undeliverable address" (sender/recipient verification) additionally to "Unknown user", gh-3039
+      both are configurable now via extended parameter and can be disabled using `exre-user=` supplied in filter parameters
+    - reject: BDAT/DATA from, gh-2927
+    - (since regex is more precise now) token selector changed to `[A-Z]{4}`, e. g. no matter what a command is supplied now 
+      (RCPT, EHLO, VRFY, DATA, BDAT or something else)
+    - matches "Command rejected" and "Data command rejected" now
+    - matches RCPT from unknown, 504 5.5.2, need fully-qualified hostname, gh-2995
+    - matches 550 5.7.25 Client host rejected, gh-2996
+* `filter.d/sendmail-auth.conf`:
+  - detect several "authentication failure" messages, sendmail 8.16.1, gh-2757
+  - detect user not found, gh-3030
+  - detect failures without user part, gh-3324
+* `filter.d/sendmail-reject.conf`:
+  - fix reverse DNS for ... (gh-3012)
+  - fixed regex to consider "Connection rate limit exceeded" with different combination of arguments
+* `filter.d/sshd.conf`:
+  - mode `ddos` extended - recognizes messages "kex_exchange_identification: Connection closed / reset by pear", gh-3086
+    (fixed possible regression of f77398c)
+  - mode `ddos` extended - recognizes new message "banner exchange: invalid format" generated by port scanner
+    (https payload on ssh port), gh-3169
+* `filter.d/zoneminder.conf` - support new log format (ERR instead of WAR), add detection of non-existent user login attempts, gh-2984
+* amend to gh-980 fixing several actions (correctly supporting new enhancements now)
+* fixed typo by `--dump-pretty` option which did never work (only `--dp` was working)
+* fixes start of fail2ban-client in docker: speedup daemonization process by huge open files limit, gh-3334
+* provides details of failed regex compilation in the error message we throw in Regex-constructor 
+  (it's good to know what exactly is wrong)
+* fixed failed update of database didn't signal with an error, gh-3352:
+  - client and server exit with error code by failure during start process (in foreground mode)
+  - added fallback to repair if database cannot be upgraded
+
+### New Features and Enhancements
+* python 3.10 and 3.11 compatibility (and GHA-CI support)
+* `actioncheck` behavior is changed now (gh-488), so invariant check as well as restore or repair
+   of sane environment (in case of recognized unsane state) would only occur on action errors (e. g.
+   if ban or unban operations are exiting with other code as 0)
+* better recognition of log rotation, better performance by reopen: avoid unnecessary seek to begin of file
+  (and hash calculation)
+* file filter reads only complete lines (ended with new-line) now, so waits for end of line (for its completion)
+* datedetector:
+  - token `%Z` must recognize zone abbreviation `Z` (GMT/UTC) also (similar to `%z`)
+  - token `%Z` recognizes all known zone abbreviation besides Z, GMT, UTC correctly, if it is matching
+    (`%z` remains unchanged for backwards-compatibility, see comment in code)
+  - date patterns `%ExY` and `%Exy` accept every year from 19xx up to current century (+3 years) in `fail2ban-regex`
+  - better grouping algorithm for resulting century RE for `%ExY` and `%Exy`
+* actions differentiate tags `<ip>` and `<fid>` (`<F-ID>`), if IP-address deviates from ID then the value
+  of `<ip>` is not equal `<fid>` anymore (gh-3217)
+* action info extended with new members for jail info (usable as tags in command actions), gh-10:
+  - `<jail.found>`, `<jail.found_total>` - current and total found failures
+  - `<jail.banned>`, `<jail.banned_total>` - current and total bans
+* `filter.d/monitorix.conf` - added new filter and jail for Monitorix, gh-2679
+* `filter.d/mssql-auth.conf` - new filter and jail for Microsoft SQL Server, gh-2642
+* `filter.d/nginx-bad-request.conf` - added filter to find bad requests (400), gh-2750
+* `filter.d/nginx-http-auth.conf` - extended with parameter mode, so additionally to `auth` (or `normal`) 
+   mode `fallback` (or combined as `aggressive`) can find SSL errors while SSL handshaking, gh-2881
+* `filter.d/scanlogd.conf` - new filter and jail, add support for filtering out detected port scans via scanlogd, gh-2950
+* `action.d/apprise.conf` - added Apprise support (50+ Notifications), gh-2565
+* `action.d/badips.*` - removed actions, badips.com is no longer active, gh-2889
+* `action.d/cloudflare.conf` - better IPv6 capability, gh-2891
+* `action.d/cloudflare-token.conf` - added support for Cloudflare Token APIs.  This method is more restrictive and therefore safter than using API Keys.
+* `action.d/ipthreat.conf` - new action for IPThreat integration, gh-3349
+* `action.d/ufw.conf` (gh-3018):
+  - new option `add` (default `prepend`), can be supplied as `insert 1` for ufw versions before v.0.36 (gh-2331, gh-3018)
+  - new options `kill-mode` and `kill` to drop established connections of intruder (see action for details, gh-3018)
+* `iptables` and `iptables-ipset` actions extended to support multiple protocols with single action 
+  for multiport or oneport type (back-ported from nftables action);
+* `iptables` actions are more breakdown-safe: start wouldn't fail if chain or rule already exists
+  (e. g. created by previous instance and doesn't get purged properly); ultimately closes gh-980
+* `ipset` actions are more breakdown-safe: start wouldn't fail if set with this name already exists
+  (e. g. created by previous instance and don't deleted properly)
+* replace internals of several `iptables` and `iptables-ipset` actions using internals of iptables include:
+  - better check mechanism (using `-C`, option `--check` is available long time);
+  - additionally iptables-ipset is a common action for `iptables-ipset-proto6-*` now (which become obsolete now);
+  - many features of different iptables actions are combinable as single chain/rule (can be supplied to action as parameters);
+  - iptables is a replacement for iptables-common now, several actions using this as include now become obsolete;
+* new logtarget SYSTEMD-JOURNAL, gh-1403
+* fail2ban.conf: new fail2ban configuration option `allowipv6` (default `auto`), can be used to allow or disallow IPv6 
+  interface in fail2ban immediately by start (e. g. if fail2ban starts before network interfaces), gh-2804
+* invalidate IP/DNS caches by reload, so inter alia would allow to recognize IPv6IsAllowed immediately, previously
+  retarded up to cache max-time (5m), gh-2804
+* OpenRC (Gentoo, mainly) service script improvements, gh-2182
+* suppress unneeded info "Jail is not a JournalFilter instance" (moved to debug level), gh-3186
+* implements new interpolation variable `%(fail2ban_confpath)s` (automatically substituted from config-reader path, 
+  default `/etc/fail2ban` or `/usr/local/etc/fail2ban` depending on distribution); `ignorecommands_dir` is unneeded anymore,
+  thus removed from `paths-common.conf`, fixes gh-3005
+* `fail2ban-regex`: accepts filter parameters containing new-line
+
+
 ver. 0.11.2 (2020/11/23) - heal-the-world-with-security-tools
 -----------
 
-### Compatibility:
+### Compatibility
 * to v.0.10:
   - 0.11 is totally compatible to 0.10 (configuration- and API-related stuff), but the database
     got some new tables and fields (auto-converted during the first start), so once updated to 0.11, you
@@ -189,7 +351,7 @@ Yes, Hrrrm...
 ### New Features
 * new replacement tags for failregex to match subnets in form of IP-addresses with CIDR mask (gh-2559):
   - `<CIDR>` - helper regex to match CIDR (simple integer form of net-mask);
-  - `<SUBNET>` - regex to match sub-net adresses (in form of IP/CIDR, also single IP is matched, so part /CIDR is optional);
+  - `<SUBNET>` - regex to match sub-net addresses (in form of IP/CIDR, also single IP is matched, so part /CIDR is optional);
 * grouped tags (`<ADDR>`, `<HOST>`, `<SUBNET>`) recognize IP addresses enclosed in square brackets
 * new failregex-flag tag `<F-MLFGAINED>` for failregex, signaled that the access to service was gained
   (ATM used similar to tag `<F-NOFAIL>`, but it does not add the log-line to matches, gh-2279)

FILTERS

@@ -278,6 +278,7 @@ to tune it.  fail2ban-regex -D ...  will present Debuggex URLs for the regexs
 and sample log files that you pass into it.
 
 In general use when using regex debuggers for generating fail2ban filters:
+
 * use regex from the ./fail2ban-regex output (to ensure all substitutions are
 done)
 * replace <HOST> with (?&.ipv4)

MANIFEST

@@ -5,11 +5,11 @@ bin/fail2ban-testcases
 ChangeLog
 config/action.d/abuseipdb.conf
 config/action.d/apf.conf
-config/action.d/badips.conf
+config/action.d/apprise.conf
-config/action.d/badips.py
 config/action.d/blocklist_de.conf
 config/action.d/bsd-ipfw.conf
 config/action.d/cloudflare.conf
+config/action.d/cloudflare-token.conf
 config/action.d/complain.conf
 config/action.d/dshield.conf
 config/action.d/dummy.conf
@@ -25,8 +25,8 @@ config/action.d/hostsdeny.conf
 config/action.d/ipfilter.conf
 config/action.d/ipfw.conf
 config/action.d/iptables-allports.conf
-config/action.d/iptables-common.conf
 config/action.d/iptables.conf
+config/action.d/iptables-ipset.conf
 config/action.d/iptables-ipset-proto4.conf
 config/action.d/iptables-ipset-proto6-allports.conf
 config/action.d/iptables-ipset-proto6.conf
@@ -34,6 +34,7 @@ config/action.d/iptables-multiport.conf
 config/action.d/iptables-multiport-log.conf
 config/action.d/iptables-new.conf
 config/action.d/iptables-xt_recent-echo.conf
+config/action.d/ipthreat.conf
 config/action.d/mail-buffered.conf
 config/action.d/mail.conf
 config/action.d/mail-whois-common.conf
@@ -112,10 +113,13 @@ config/filter.d/kerio.conf
 config/filter.d/lighttpd-auth.conf
 config/filter.d/mongodb-auth.conf
 config/filter.d/monit.conf
+config/filter.d/monitorix.conf
+config/filter.d/mssql-auth.conf
 config/filter.d/murmur.conf
 config/filter.d/mysqld-auth.conf
 config/filter.d/nagios.conf
 config/filter.d/named-refused.conf
+config/filter.d/nginx-bad-request.conf
 config/filter.d/nginx-botsearch.conf
 config/filter.d/nginx-http-auth.conf
 config/filter.d/nginx-limit-req.conf
@@ -134,6 +138,7 @@ config/filter.d/pure-ftpd.conf
 config/filter.d/qmail.conf
 config/filter.d/recidive.conf
 config/filter.d/roundcube-auth.conf
+config/filter.d/scanlogd.conf
 config/filter.d/screensharingd.conf
 config/filter.d/selinux-common.conf
 config/filter.d/selinux-ssh.conf
@@ -220,7 +225,6 @@ fail2ban/setup.py
 fail2ban-testcases-all
 fail2ban-testcases-all-python3
 fail2ban/tests/action_d/__init__.py
-fail2ban/tests/action_d/test_badips.py
 fail2ban/tests/action_d/test_smtp.py
 fail2ban/tests/actionstestcase.py
 fail2ban/tests/actiontestcase.py
@@ -317,10 +321,13 @@ fail2ban/tests/files/logs/kerio
 fail2ban/tests/files/logs/lighttpd-auth
 fail2ban/tests/files/logs/mongodb-auth
 fail2ban/tests/files/logs/monit
+fail2ban/tests/files/logs/monitorix
+fail2ban/tests/files/logs/mssql-auth
 fail2ban/tests/files/logs/murmur
 fail2ban/tests/files/logs/mysqld-auth
 fail2ban/tests/files/logs/nagios
 fail2ban/tests/files/logs/named-refused
+fail2ban/tests/files/logs/nginx-bad-request
 fail2ban/tests/files/logs/nginx-botsearch
 fail2ban/tests/files/logs/nginx-http-auth
 fail2ban/tests/files/logs/nginx-limit-req
@@ -339,6 +346,7 @@ fail2ban/tests/files/logs/pure-ftpd
 fail2ban/tests/files/logs/qmail
 fail2ban/tests/files/logs/recidive
 fail2ban/tests/files/logs/roundcube-auth
+fail2ban/tests/files/logs/scanlogd
 fail2ban/tests/files/logs/screensharingd
 fail2ban/tests/files/logs/selinux-ssh
 fail2ban/tests/files/logs/sendmail-auth
@@ -391,12 +399,12 @@ files/cacti/fail2ban_stats.sh
 files/cacti/README
 files/debian-initd
 files/fail2ban-logrotate
+files/fail2ban-openrc.conf
+files/fail2ban-openrc.init.in
 files/fail2ban.service.in
 files/fail2ban-tmpfiles.conf
 files/fail2ban.upstart
 files/gen_badbots
-files/gentoo-confd
-files/gentoo-initd
 files/ipmasq-ZZZzzz_fail2ban.rul
 files/logwatch/fail2ban
 files/logwatch/fail2ban-0.8.log

README.md

@@ -2,7 +2,7 @@
                         / _|__ _(_) |_  ) |__  __ _ _ _  
                        |  _/ _` | | |/ /| '_ \/ _` | ' \ 
                        |_| \__,_|_|_/___|_.__/\__,_|_||_|
-                       v0.11.0.dev1            20??/??/??
+                       v1.0.1.dev1            20??/??/??
 
 ## Fail2Ban: ban hosts that cause multiple authentication errors
 
@@ -33,7 +33,8 @@ Installation:
 this case, you should use that instead.**
 
 Required:
-- [Python2 >= 2.6 or Python >= 3.2](https://www.python.org) or [PyPy](https://pypy.org)
+- [Python2 >= 2.7 or Python >= 3.2](https://www.python.org) or [PyPy](https://pypy.org)
+- python-setuptools, python-distutils or python3-setuptools for installation from source
 
 Optional:
 - [pyinotify >= 0.8.3](https://github.com/seb-m/pyinotify), may require:
@@ -46,11 +47,11 @@ Optional:
 
 To install:
 
-    tar xvfj fail2ban-0.11.0.tar.bz2
+    tar xvfj fail2ban-1.0.1.tar.bz2
-    cd fail2ban-0.11.0
+    cd fail2ban-1.0.1
     sudo python setup.py install
    
-Alternatively, you can clone the source from GitHub to a directory of Your choice, and do the install from there. Pick the correct branch, for example, 0.11
+Alternatively, you can clone the source from GitHub to a directory of Your choice, and do the install from there. Pick the correct branch, for example, master or 0.11
 
     git clone https://github.com/fail2ban/fail2ban.git
     cd fail2ban
@@ -89,11 +90,11 @@ fail2ban(1) and jail.conf(5)  manpages for further references.
 Code status:
 ------------
 
-* travis-ci.org: [![tests status](https://secure.travis-ci.org/fail2ban/fail2ban.svg?branch=0.11)](https://travis-ci.org/fail2ban/fail2ban?branch=0.11) (0.11 branch) / [![tests status](https://secure.travis-ci.org/fail2ban/fail2ban.svg?branch=0.10)](https://travis-ci.org/fail2ban/fail2ban?branch=0.10) (0.10 branch) 
+* travis-ci.org: [![tests status](https://secure.travis-ci.org/fail2ban/fail2ban.svg?branch=master)](https://travis-ci.org/fail2ban/fail2ban?branch=master) / [![tests status](https://secure.travis-ci.org/fail2ban/fail2ban.svg?branch=0.11)](https://travis-ci.org/fail2ban/fail2ban?branch=0.11) (0.11 branch) / [![tests status](https://secure.travis-ci.org/fail2ban/fail2ban.svg?branch=0.10)](https://travis-ci.org/fail2ban/fail2ban?branch=0.10) (0.10 branch) 
 
-* coveralls.io: [![Coverage Status](https://coveralls.io/repos/fail2ban/fail2ban/badge.svg?branch=0.11)](https://coveralls.io/github/fail2ban/fail2ban?branch=0.11) (0.11 branch) / [![Coverage Status](https://coveralls.io/repos/fail2ban/fail2ban/badge.svg?branch=0.10)](https://coveralls.io/github/fail2ban/fail2ban?branch=0.10) / (0.10 branch)
+* coveralls.io: [![Coverage Status](https://coveralls.io/repos/fail2ban/fail2ban/badge.svg?branch=master)](https://coveralls.io/github/fail2ban/fail2ban?branch=master) / [![Coverage Status](https://coveralls.io/repos/fail2ban/fail2ban/badge.svg?branch=0.11)](https://coveralls.io/github/fail2ban/fail2ban?branch=0.11) (0.11 branch) / [![Coverage Status](https://coveralls.io/repos/fail2ban/fail2ban/badge.svg?branch=0.10)](https://coveralls.io/github/fail2ban/fail2ban?branch=0.10) / (0.10 branch)
 
-* codecov.io: [![codecov.io](https://codecov.io/gh/fail2ban/fail2ban/coverage.svg?branch=0.11)](https://codecov.io/gh/fail2ban/fail2ban/branch/0.11) (0.11 branch) / [![codecov.io](https://codecov.io/gh/fail2ban/fail2ban/coverage.svg?branch=0.10)](https://codecov.io/gh/fail2ban/fail2ban/branch/0.10) (0.10 branch)
+* codecov.io: [![codecov.io](https://codecov.io/gh/fail2ban/fail2ban/coverage.svg?branch=master)](https://codecov.io/gh/fail2ban/fail2ban/branch/master) / [![codecov.io](https://codecov.io/gh/fail2ban/fail2ban/coverage.svg?branch=0.11)](https://codecov.io/gh/fail2ban/fail2ban/branch/0.11) (0.11 branch) / [![codecov.io](https://codecov.io/gh/fail2ban/fail2ban/coverage.svg?branch=0.10)](https://codecov.io/gh/fail2ban/fail2ban/branch/0.10) (0.10 branch)
 
 Contact:
 --------

THANKS

@@ -33,6 +33,7 @@ Christoph Haas
 Christos Psonis
 craneworks
 Cyril Jaquier
+Daniel Aleksandersen
 Daniel B. Cid
 Daniel B.
 Daniel Black

config/action.d/apprise.conf

@@ -0,0 +1,49 @@
+# Fail2Ban configuration file
+#
+# Author: Chris Caron <lead2gold@gmail.com>
+#
+#
+
+[Definition]
+
+# Option:  actionstart
+# Notes.:  command executed once at the start of Fail2Ban.
+# Values:  CMD
+#
+actionstart = printf %%b "The jail <name> as been started successfully." | <apprise> -t "[Fail2Ban] <name>: started on `uname -n`"
+
+# Option:  actionstop
+# Notes.:  command executed once at the end of Fail2Ban
+# Values:  CMD
+#
+actionstop = printf %%b "The jail <name> has been stopped." | <apprise> -t "[Fail2Ban] <name>: stopped on `uname -n`"
+
+# Option:  actioncheck
+# Notes.:  command executed once before each actionban command
+# Values:  CMD
+#
+actioncheck =
+
+# Option:  actionban
+# Notes.:  command executed when banning an IP. Take care that the
+#          command is executed with Fail2Ban user rights.
+# Tags:    See jail.conf(5) man page
+# Values:  CMD
+#
+actionban = printf %%b "The IP <ip> has just been banned by Fail2Ban after <failures> attempts against <name>" | <apprise> -n "warning" -t "[Fail2Ban] <name>: banned <ip> from `uname -n`"
+
+# Option:  actionunban
+# Notes.:  command executed when unbanning an IP. Take care that the
+#          command is executed with Fail2Ban user rights.
+# Tags:    See jail.conf(5) man page
+# Values:  CMD
+#
+actionunban =
+
+[Init]
+
+# Define location of the default apprise configuration file to use
+#
+config = /etc/fail2ban/apprise.conf
+#
+apprise = apprise -c "<config>"

config/action.d/badips.conf

@@ -1,19 +0,0 @@
-# Fail2ban reporting to badips.com
-#
-# Note: This reports an IP only and does not actually ban traffic. Use
-# another action in the same jail if you want bans to occur.
-#
-# Set the category to the appropriate value before use.
-#
-# To get see register and optional key to get personalised graphs see:
-# http://www.badips.com/blog/personalized-statistics-track-the-attackers-of-all-your-servers-with-one-key
-
-[Definition]
-
-actionban = curl --fail  --user-agent "<agent>" http://www.badips.com/add/<category>/<ip>
-
-[Init]
-
-# Option: category
-# Notes.: Values are from the list here: http://www.badips.com/get/categories
-category = 

config/action.d/badips.py

@@ -1,391 +0,0 @@
-# emacs: -*- mode: python; py-indent-offset: 4; indent-tabs-mode: t -*-
-# vi: set ft=python sts=4 ts=4 sw=4 noet :
-
-# This file is part of Fail2Ban.
-#
-# Fail2Ban is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2 of the License, or
-# (at your option) any later version.
-#
-# Fail2Ban is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Fail2Ban; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
-
-import sys
-if sys.version_info < (2, 7): # pragma: no cover
-	raise ImportError("badips.py action requires Python >= 2.7")
-import json
-import threading
-import logging
-if sys.version_info >= (3, ): # pragma: 2.x no cover
-	from urllib.request import Request, urlopen
-	from urllib.parse import urlencode
-	from urllib.error import HTTPError
-else: # pragma: 3.x no cover
-	from urllib2 import Request, urlopen, HTTPError
-	from urllib import urlencode
-
-from fail2ban.server.actions import Actions, ActionBase, BanTicket
-from fail2ban.helpers import splitwords, str2LogLevel
-
-
-
-class BadIPsAction(ActionBase): # pragma: no cover - may be unavailable
-	"""Fail2Ban action which reports bans to badips.com, and also
-	blacklist bad IPs listed on badips.com by using another action's
-	ban method.
-
-	Parameters
-	----------
-	jail : Jail
-		The jail which the action belongs to.
-	name : str
-		Name assigned to the action.
-	category : str
-		Valid badips.com category for reporting failures.
-	score : int, optional
-		Minimum score for bad IPs. Default 3.
-	age : str, optional
-		Age of last report for bad IPs, per badips.com syntax.
-		Default "24h" (24 hours)
-	banaction : str, optional
-		Name of banaction to use for blacklisting bad IPs. If `None`,
-		no blacklist of IPs will take place.
-		Default `None`.
-	bancategory : str, optional
-		Name of category to use for blacklisting, which can differ
-		from category used for reporting. e.g. may want to report
-		"postfix", but want to use whole "mail" category for blacklist.
-		Default `category`.
-	bankey : str, optional
-		Key issued by badips.com to retrieve personal list
-		of blacklist IPs.
-	updateperiod : int, optional
-		Time in seconds between updating bad IPs blacklist.
-		Default 900 (15 minutes)
-	loglevel : int/str, optional
-		Log level of the message when an IP is (un)banned.
-		Default `DEBUG`.
-		Can be also supplied as two-value list (comma- or space separated) to
-		provide level of the summary message when a group of IPs is (un)banned.
-		Example `DEBUG,INFO`.
-	agent : str, optional
-		User agent transmitted to server.
-		Default `Fail2Ban/ver.`
-
-	Raises
-	------
-	ValueError
-		If invalid `category`, `score`, `banaction` or `updateperiod`.
-	"""
-
-	TIMEOUT = 10
-	_badips = "https://www.badips.com"
-	def _Request(self, url, **argv):
-		return Request(url, headers={'User-Agent': self.agent}, **argv)
-
-	def __init__(self, jail, name, category, score=3, age="24h",
-		banaction=None, bancategory=None, bankey=None, updateperiod=900, 
-		loglevel='DEBUG', agent="Fail2Ban", timeout=TIMEOUT):
-		super(BadIPsAction, self).__init__(jail, name)
-
-		self.timeout = timeout
-		self.agent = agent
-		self.category = category
-		self.score = score
-		self.age = age
-		self.banaction = banaction
-		self.bancategory = bancategory or category
-		self.bankey = bankey
-		loglevel = splitwords(loglevel)
-		self.sumloglevel = str2LogLevel(loglevel[-1])
-		self.loglevel = str2LogLevel(loglevel[0])
-		self.updateperiod = updateperiod
-
-		self._bannedips = set()
-		# Used later for threading.Timer for updating badips
-		self._timer = None
-
-	@staticmethod
-	def isAvailable(timeout=1):
-		try:
-			response = urlopen(Request("/".join([BadIPsAction._badips]),
-					headers={'User-Agent': "Fail2Ban"}), timeout=timeout)
-			return True, ''
-		except Exception as e: # pragma: no cover
-			return False, e
-
-	def logError(self, response, what=''): # pragma: no cover - sporadical (502: Bad Gateway, etc)
-		messages = {}
-		try:
-			messages = json.loads(response.read().decode('utf-8'))
-		except:
-			pass
-		self._logSys.error(
-			"%s. badips.com response: '%s'", what,
-				messages.get('err', 'Unknown'))
-
-	def getCategories(self, incParents=False):
-		"""Get badips.com categories.
-
-		Returns
-		-------
-		set
-			Set of categories.
-
-		Raises
-		------
-		HTTPError
-			Any issues with badips.com request.
-		ValueError
-			If badips.com response didn't contain necessary information
-		"""
-		try:
-			response = urlopen(
-				self._Request("/".join([self._badips, "get", "categories"])), timeout=self.timeout)
-		except HTTPError as response: # pragma: no cover
-			self.logError(response, "Failed to fetch categories")
-			raise
-		else:
-			response_json = json.loads(response.read().decode('utf-8'))
-			if not 'categories' in response_json:
-				err = "badips.com response lacked categories specification. Response was: %s" \
-				  % (response_json,)
-				self._logSys.error(err)
-				raise ValueError(err)
-			categories = response_json['categories']
-			categories_names = set(
-				value['Name'] for value in categories)
-			if incParents:
-				categories_names.update(set(
-					value['Parent'] for value in categories
-					if "Parent" in value))
-			return categories_names
-
-	def getList(self, category, score, age, key=None):
-		"""Get badips.com list of bad IPs.
-
-		Parameters
-		----------
-		category : str
-			Valid badips.com category.
-		score : int
-			Minimum score for bad IPs.
-		age : str
-			Age of last report for bad IPs, per badips.com syntax.
-		key : str, optional
-			Key issued by badips.com to fetch IPs reported with the
-			associated key.
-
-		Returns
-		-------
-		set
-			Set of bad IPs.
-
-		Raises
-		------
-		HTTPError
-			Any issues with badips.com request.
-		"""
-		try:
-			url = "?".join([
-				"/".join([self._badips, "get", "list", category, str(score)]),
-				urlencode({'age': age})])
-			if key:
-				url = "&".join([url, urlencode({'key': key})])
-			self._logSys.debug('badips.com: get list, url: %r', url)
-			response = urlopen(self._Request(url), timeout=self.timeout)
-		except HTTPError as response: # pragma: no cover
-			self.logError(response, "Failed to fetch bad IP list")
-			raise
-		else:
-			return set(response.read().decode('utf-8').split())
-
-	@property
-	def category(self):
-		"""badips.com category for reporting IPs.
-		"""
-		return self._category
-
-	@category.setter
-	def category(self, category):
-		if category not in self.getCategories():
-			self._logSys.error("Category name '%s' not valid. "
-				"see badips.com for list of valid categories",
-				category)
-			raise ValueError("Invalid category: %s" % category)
-		self._category = category
-
-	@property
-	def bancategory(self):
-		"""badips.com bancategory for fetching IPs.
-		"""
-		return self._bancategory
-
-	@bancategory.setter
-	def bancategory(self, bancategory):
-		if bancategory != "any" and bancategory not in self.getCategories(incParents=True):
-			self._logSys.error("Category name '%s' not valid. "
-				"see badips.com for list of valid categories",
-				bancategory)
-			raise ValueError("Invalid bancategory: %s" % bancategory)
-		self._bancategory = bancategory
-
-	@property
-	def score(self):
-		"""badips.com minimum score for fetching IPs.
-		"""
-		return self._score
-
-	@score.setter
-	def score(self, score):
-		score = int(score)
-		if 0 <= score <= 5:
-			self._score = score
-		else:
-			raise ValueError("Score must be 0-5")
-
-	@property
-	def banaction(self):
-		"""Jail action to use for banning/unbanning.
-		"""
-		return self._banaction
-
-	@banaction.setter
-	def banaction(self, banaction):
-		if banaction is not None and banaction not in self._jail.actions:
-			self._logSys.error("Action name '%s' not in jail '%s'",
-				banaction, self._jail.name)
-			raise ValueError("Invalid banaction")
-		self._banaction = banaction
-
-	@property
-	def updateperiod(self):
-		"""Period in seconds between banned bad IPs will be updated.
-		"""
-		return self._updateperiod
-
-	@updateperiod.setter
-	def updateperiod(self, updateperiod):
-		updateperiod = int(updateperiod)
-		if updateperiod > 0:
-			self._updateperiod = updateperiod
-		else:
-			raise ValueError("Update period must be integer greater than 0")
-
-	def _banIPs(self, ips):
-		for ip in ips:
-			try:
-				ai = Actions.ActionInfo(BanTicket(ip), self._jail)
-				self._jail.actions[self.banaction].ban(ai)
-			except Exception as e:
-				self._logSys.error(
-					"Error banning IP %s for jail '%s' with action '%s': %s",
-					ip, self._jail.name, self.banaction, e,
-					exc_info=self._logSys.getEffectiveLevel()<=logging.DEBUG)
-			else:
-				self._bannedips.add(ip)
-				self._logSys.log(self.loglevel,
-					"Banned IP %s for jail '%s' with action '%s'",
-					ip, self._jail.name, self.banaction)
-
-	def _unbanIPs(self, ips):
-		for ip in ips:
-			try:
-				ai = Actions.ActionInfo(BanTicket(ip), self._jail)
-				self._jail.actions[self.banaction].unban(ai)
-			except Exception as e:
-				self._logSys.error(
-					"Error unbanning IP %s for jail '%s' with action '%s': %s",
-					ip, self._jail.name, self.banaction, e,
-					exc_info=self._logSys.getEffectiveLevel()<=logging.DEBUG)
-			else:
-				self._logSys.log(self.loglevel,
-					"Unbanned IP %s for jail '%s' with action '%s'",
-					ip, self._jail.name, self.banaction)
-			finally:
-				self._bannedips.remove(ip)
-
-	def start(self):
-		"""If `banaction` set, blacklists bad IPs.
-		"""
-		if self.banaction is not None:
-			self.update()
-
-	def update(self):
-		"""If `banaction` set, updates blacklisted IPs.
-
-		Queries badips.com for list of bad IPs, removing IPs from the
-		blacklist if no longer present, and adds new bad IPs to the
-		blacklist.
-		"""
-		if self.banaction is not None:
-			if self._timer:
-				self._timer.cancel()
-				self._timer = None
-
-			try:
-				ips = self.getList(
-					self.bancategory, self.score, self.age, self.bankey)
-				# Remove old IPs no longer listed
-				s = self._bannedips - ips
-				m = len(s)
-				self._unbanIPs(s)
-				# Add new IPs which are now listed
-				s = ips - self._bannedips
-				p = len(s)
-				self._banIPs(s)
-				if m != 0 or p != 0:
-					self._logSys.log(self.sumloglevel,
-						"Updated IPs for jail '%s' (-%d/+%d)",
-						self._jail.name, m, p)
-				self._logSys.debug(
-					"Next update for jail '%' in %i seconds",
-					self._jail.name, self.updateperiod)
-			finally:
-				self._timer = threading.Timer(self.updateperiod, self.update)
-				self._timer.start()
-
-	def stop(self):
-		"""If `banaction` set, clears blacklisted IPs.
-		"""
-		if self.banaction is not None:
-			if self._timer:
-				self._timer.cancel()
-				self._timer = None
-			self._unbanIPs(self._bannedips.copy())
-
-	def ban(self, aInfo):
-		"""Reports banned IP to badips.com.
-
-		Parameters
-		----------
-		aInfo : dict
-			Dictionary which includes information in relation to
-			the ban.
-
-		Raises
-		------
-		HTTPError
-			Any issues with badips.com request.
-		"""
-		try:
-			url = "/".join([self._badips, "add", self.category, str(aInfo['ip'])])
-			self._logSys.debug('badips.com: ban, url: %r', url)
-			response = urlopen(self._Request(url), timeout=self.timeout)
-		except HTTPError as response: # pragma: no cover
-			self.logError(response, "Failed to ban")
-			raise
-		else:
-			messages = json.loads(response.read().decode('utf-8'))
-			self._logSys.debug(
-				"Response from badips.com report: '%s'",
-				messages['suc'])
-
-Action = BadIPsAction

config/action.d/cloudflare-token.conf

@@ -0,0 +1,92 @@
+#
+# Author: Logic-32
+#
+# IMPORTANT
+#
+# Please set jail.local's permission to 640 because it contains your CF API token.
+#
+# This action depends on curl.
+#
+# To get your Cloudflare API token: https://developers.cloudflare.com/api/tokens/create/
+#
+# Cloudflare Firewall API: https://developers.cloudflare.com/firewall/api/cf-firewall-rules/endpoints/
+
+[Definition]
+
+# Option:  actionstart
+# Notes.:  command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false).
+# Values:  CMD
+#
+actionstart =
+
+# Option:  actionstop
+# Notes.:  command executed at the stop of jail (or at the end of Fail2Ban)
+# Values:  CMD
+#
+actionstop =
+
+# Option:  actioncheck
+# Notes.:  command executed once before each actionban command
+# Values:  CMD
+#
+actioncheck =
+
+# Option:  actionban
+# Notes.:  command executed when banning an IP. Take care that the
+#          command is executed with Fail2Ban user rights.
+# Tags:    <ip>  IP address
+#          <failures>  number of failures
+#          <time>  unix timestamp of the ban time
+# Values:  CMD
+actionban = curl -s -X POST "<_cf_api_url>" \
+              <_cf_api_prms> \
+              --data '{"mode":"<cfmode>","configuration":{"target":"<cftarget>","value":"<ip>"},"notes":"<notes>"}'
+
+# Option:  actionunban
+# Notes.:  command executed when unbanning an IP. Take care that the
+#          command is executed with Fail2Ban user rights.
+# Tags:    <ip>  IP address
+#          <failures>  number of failures
+#          <time>  unix timestamp of the ban time
+# Values:  CMD
+#
+actionunban = id=$(curl -s -X GET "<_cf_api_url>?mode=<cfmode>&notes=<notes>&configuration.target=<cftarget>&configuration.value=<ip>" \
+                       <_cf_api_prms> \
+                       | awk -F"[,:}]" '{for(i=1;i<=NF;i++){if($i~/'id'\042/){print $(i+1)}}}' \
+                       | tr -d ' "' \
+                       | head -n 1)
+              if [ -z "$id" ]; then echo "<name>: id for <ip> cannot be found using target <cftarget>"; exit 0; fi; \
+              curl -s -X DELETE "<_cf_api_url>/$id" \
+                  <_cf_api_prms> \
+                  --data '{"cascade": "none"}'
+
+_cf_api_url = https://api.cloudflare.com/client/v4/zones/<cfzone>/firewall/access_rules/rules
+_cf_api_prms = -H "Authorization: Bearer <cftoken>" -H "Content-Type: application/json"
+
+[Init]
+
+# Declare your Cloudflare Authorization Bearer Token in the [DEFAULT] section of your jail.local file.
+
+# The Cloudflare <ZONE_ID> of hte domain you want to manage.
+#
+# cfzone =
+
+# Your personal Cloudflare token.  Ideally restricted to just have "Zone.Firewall Services" permissions.
+#
+# cftoken =
+
+# Target of the firewall rule.  Default is "ip" (v4).
+#
+cftarget = ip
+
+# The firewall mode Cloudflare should use.  Default is "block" (deny access).
+# Consider also "js_challenge" or other "allowed_modes" if you want.
+#
+cfmode = block
+
+# The message to include in the firewall IP banning rule.
+#
+notes = Fail2Ban <name>
+
+[Init?family=inet6]
+cftarget = ip6

config/action.d/cloudflare.conf

@@ -44,7 +44,7 @@ actioncheck =
 #actionban = curl -s -o /dev/null https://www.cloudflare.com/api_json.html -d 'a=ban' -d 'tkn=<cftoken>' -d 'email=<cfuser>' -d 'key=<ip>'
 # API v4
 actionban = curl -s -o /dev/null -X POST <_cf_api_prms> \
-            -d '{"mode":"block","configuration":{"target":"ip","value":"<ip>"},"notes":"Fail2Ban <name>"}' \
+            -d '{"mode":"block","configuration":{"target":"<cftarget>","value":"<ip>"},"notes":"Fail2Ban <name>"}' \
             <_cf_api_url>
 
 # Option:  actionunban
@@ -59,7 +59,7 @@ actionban = curl -s -o /dev/null -X POST <_cf_api_prms> \
 #actionunban = curl -s -o /dev/null https://www.cloudflare.com/api_json.html -d 'a=nul' -d 'tkn=<cftoken>' -d 'email=<cfuser>' -d 'key=<ip>'
 # API v4
 actionunban = id=$(curl -s -X GET <_cf_api_prms> \
-                   "<_cf_api_url>?mode=block&configuration_target=ip&configuration_value=<ip>&page=1&per_page=1&notes=Fail2Ban%%20<name>" \
+                   "<_cf_api_url>?mode=block&configuration_target=<cftarget>&configuration_value=<ip>&page=1&per_page=1&notes=Fail2Ban%%20<name>" \
                    | { jq -r '.result[0].id' 2>/dev/null || tr -d '\n' | sed -nE 's/^.*"result"\s*:\s*\[\s*\{\s*"id"\s*:\s*"([^"]+)".*$/\1/p'; })
               if [ -z "$id" ]; then echo "<name>: id for <ip> cannot be found"; exit 0; fi;
               curl -s -o /dev/null -X DELETE <_cf_api_prms> "<_cf_api_url>/$id"
@@ -81,3 +81,8 @@ _cf_api_prms = -H 'X-Auth-Email: <cfuser>' -H 'X-Auth-Key: <cftoken>' -H 'Conten
 cftoken =
 
 cfuser =
+
+cftarget = ip
+
+[Init?family=inet6]
+cftarget = ip6

config/action.d/complain.conf

@@ -102,7 +102,7 @@ logpath = /dev/null
 # Notes.:  Your system mail command. Is passed 2 args: subject and recipient
 # Values:  CMD
 #
-mailcmd = mail -s
+mailcmd = mail -E 'set escape' -s
 
 # Option:  mailargs
 # Notes.:  Additional arguments to mail command. e.g. for standard Unix mail:

config/action.d/dshield.conf

@@ -179,7 +179,7 @@ tcpflags =
 # Notes.:  Your system mail command. Is passed 2 args: subject and recipient
 # Values:  CMD
 #
-mailcmd = mail -s
+mailcmd = mail -E 'set escape' -s
 
 # Option:  mailargs
 # Notes.:  Additional arguments to mail command. e.g. for standard Unix mail:

config/action.d/firewallcmd-ipset.conf

@@ -18,20 +18,45 @@ before = firewallcmd-common.conf
 
 [Definition]
 
-actionstart = ipset create <ipmset> hash:ip timeout <default-ipsettime> <familyopt>
+actionstart = <ipstype_<ipsettype>/actionstart>
               firewall-cmd --direct --add-rule <family> filter <chain> 0 <actiontype> -m set --match-set <ipmset> src -j <blocktype>
 
-actionflush = ipset flush <ipmset>
+actionflush = <ipstype_<ipsettype>/actionflush>
 
 actionstop = firewall-cmd --direct --remove-rule <family> filter <chain> 0 <actiontype> -m set --match-set <ipmset> src -j <blocktype>
              <actionflush>
-             ipset destroy <ipmset>
+             <ipstype_<ipsettype>/actionstop>
 
-actionban = ipset add <ipmset> <ip> timeout <ipsettime> -exist
+actionban = <ipstype_<ipsettype>/actionban>
 
 # actionprolong = %(actionban)s
 
-actionunban = ipset del <ipmset> <ip> -exist
+actionunban = <ipstype_<ipsettype>/actionunban>
+
+[ipstype_ipset]
+
+actionstart = ipset -exist create <ipmset> hash:ip timeout <default-ipsettime> <familyopt>
+
+actionflush = ipset flush <ipmset>
+
+actionstop = ipset destroy <ipmset>
+
+actionban = ipset -exist add <ipmset> <ip> timeout <ipsettime>
+
+actionunban = ipset -exist del <ipmset> <ip>
+
+[ipstype_firewalld]
+
+actionstart = firewall-cmd --direct --new-ipset=<ipmset> --type=hash:ip --option=timeout=<default-ipsettime> <firewalld_familyopt>
+
+# TODO: there doesn't seem to be an explicit way to invoke the ipset flush function using firewall-cmd
+actionflush = 
+
+actionstop = firewall-cmd --direct --delete-ipset=<ipmset>
+
+actionban = firewall-cmd --ipset=<ipmset> --add-entry=<ip>
+
+actionunban = firewall-cmd --ipset=<ipmset> --remove-entry=<ip>
 
 [Init]
 
@@ -56,6 +81,12 @@ ipsettime = 0
 # banaction = %(known/banaction)s[ipsettime='<timeout-bantime>']
 timeout-bantime = $([ "<bantime>" -le 2147483 ] && echo "<bantime>" || echo 0)
 
+# Option: ipsettype
+# Notes.: defines type of ipset used for match-set (firewalld or ipset)
+# Values: firewalld or ipset
+# Default: ipset
+ipsettype = ipset
+
 # Option: actiontype
 # Notes.: defines additions to the blocking rule
 # Values: leave empty to block all attempts from the host
@@ -71,18 +102,20 @@ allports = -p <protocol>
 # Option: multiport
 # Notes.: addition to block access only to specific ports
 # Usage.: use in jail config:  banaction = firewallcmd-ipset[actiontype=<multiport>]
-multiport = -p <protocol> -m multiport --dports "$(echo '<port>' | sed s/:/-/g)"
+multiport = -p <protocol> -m multiport --dports <port>
 
 ipmset = f2b-<name>
 familyopt =
+firewalld_familyopt =
 
 [Init?family=inet6]
 
 ipmset = f2b-<name>6
 familyopt = family inet6
+firewalld_familyopt = --option=family=inet6
 
 
 # DEV NOTES:
 #
-# Author: Edgar Hoch and Daniel Black
+# Author: Edgar Hoch, Daniel Black, Sergey Brester and Mihail Politaev
 # firewallcmd-new / iptables-ipset-proto6 combined for maximium goodness

config/action.d/firewallcmd-multiport.conf

@@ -11,9 +11,9 @@ before = firewallcmd-common.conf
 
 actionstart = firewall-cmd --direct --add-chain <family> filter f2b-<name>
               firewall-cmd --direct --add-rule <family> filter f2b-<name> 1000 -j RETURN
-              firewall-cmd --direct --add-rule <family> filter <chain> 0 -m conntrack --ctstate NEW -p <protocol> -m multiport --dports "$(echo '<port>' | sed s/:/-/g)" -j f2b-<name>
+              firewall-cmd --direct --add-rule <family> filter <chain> 0 -m conntrack --ctstate NEW -p <protocol> -m multiport --dports <port> -j f2b-<name>
 
-actionstop = firewall-cmd --direct --remove-rule <family> filter <chain> 0 -m conntrack --ctstate NEW -p <protocol> -m multiport --dports "$(echo '<port>' | sed s/:/-/g)" -j f2b-<name>
+actionstop = firewall-cmd --direct --remove-rule <family> filter <chain> 0 -m conntrack --ctstate NEW -p <protocol> -m multiport --dports <port> -j f2b-<name>
              firewall-cmd --direct --remove-rules <family> filter f2b-<name>
              firewall-cmd --direct --remove-chain <family> filter f2b-<name>
 

config/action.d/firewallcmd-new.conf

@@ -10,9 +10,9 @@ before = firewallcmd-common.conf
 
 actionstart = firewall-cmd --direct --add-chain <family> filter f2b-<name>
               firewall-cmd --direct --add-rule <family> filter f2b-<name> 1000 -j RETURN
-              firewall-cmd --direct --add-rule <family> filter <chain> 0 -m state --state NEW -p <protocol> -m multiport --dports "$(echo '<port>' | sed s/:/-/g)" -j f2b-<name>
+              firewall-cmd --direct --add-rule <family> filter <chain> 0 -m state --state NEW -p <protocol> -m multiport --dports <port> -j f2b-<name>
 
-actionstop = firewall-cmd --direct --remove-rule <family> filter <chain> 0 -m state --state NEW -p <protocol> -m multiport --dports "$(echo '<port>' | sed s/:/-/g)" -j f2b-<name>
+actionstop = firewall-cmd --direct --remove-rule <family> filter <chain> 0 -m state --state NEW -p <protocol> -m multiport --dports <port> -j f2b-<name>
              firewall-cmd --direct --remove-rules <family> filter f2b-<name>
              firewall-cmd --direct --remove-chain <family> filter f2b-<name>
 

config/action.d/firewallcmd-rich-rules.conf

@@ -37,8 +37,8 @@ actioncheck =
 
 fwcmd_rich_rule = rule family='<family>' source address='<ip>' port port='$p' protocol='<protocol>' %(rich-suffix)s
 
-actionban = ports="$(echo '<port>' | sed s/:/-/g)"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="%(fwcmd_rich_rule)s"; done
+actionban = ports="<port>"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="%(fwcmd_rich_rule)s"; done
 	   
-actionunban = ports="$(echo '<port>' | sed s/:/-/g)"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --remove-rich-rule="%(fwcmd_rich_rule)s"; done
+actionunban = ports="<port>"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --remove-rich-rule="%(fwcmd_rich_rule)s"; done
 
 rich-suffix = <rich-blocktype>

config/action.d/iptables-allports.conf

@@ -4,52 +4,12 @@
 # Modified: Yaroslav O. Halchenko <debian@onerussian.com>
 # 			made active on all ports from original iptables.conf
 #
-#
+# Obsolete: superseded by iptables[type=allports]
 
 [INCLUDES]
 
-before = iptables-common.conf
+before = iptables.conf
-
 
 [Definition]
 
-# Option:  actionstart
+type = allports
-# Notes.:  command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false).
-# Values:  CMD
-#
-actionstart = <iptables> -N f2b-<name>
-              <iptables> -A f2b-<name> -j <returntype>
-              <iptables> -I <chain> -p <protocol> -j f2b-<name>
-
-# Option:  actionstop
-# Notes.:  command executed at the stop of jail (or at the end of Fail2Ban)
-# Values:  CMD
-#
-actionstop = <iptables> -D <chain> -p <protocol> -j f2b-<name>
-             <actionflush>
-             <iptables> -X f2b-<name>
-
-# Option:  actioncheck
-# Notes.:  command executed once before each actionban command
-# Values:  CMD
-#
-actioncheck = <iptables> -n -L <chain> | grep -q 'f2b-<name>[ \t]'
-
-# Option:  actionban
-# Notes.:  command executed when banning an IP. Take care that the
-#          command is executed with Fail2Ban user rights.
-# Tags:    See jail.conf(5) man page
-# Values:  CMD
-#
-actionban = <iptables> -I f2b-<name> 1 -s <ip> -j <blocktype>
-
-# Option:  actionunban
-# Notes.:  command executed when unbanning an IP. Take care that the
-#          command is executed with Fail2Ban user rights.
-# Tags:    See jail.conf(5) man page
-# Values:  CMD
-#
-actionunban = <iptables> -D f2b-<name> -s <ip> -j <blocktype>
-
-[Init]
-

config/action.d/iptables-common.conf

@@ -1,92 +0,0 @@
-# Fail2Ban configuration file
-#
-# Author: Daniel Black
-#
-# This is a included configuration file and includes the definitions for the iptables
-# used in all iptables based actions by default.
-#
-# The user can override the defaults in iptables-common.local
-#
-# Modified: Alexander Koeppe <format_c@online.de>, Serg G. Brester <serg.brester@sebres.de>
-#       made config file IPv6 capable (see new section Init?family=inet6)
-
-[INCLUDES]
-
-after = iptables-blocktype.local
-        iptables-common.local
-# iptables-blocktype.local is obsolete
-
-[Definition]
-
-# Option:  actionflush
-# Notes.:  command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action)
-# Values:  CMD
-#
-actionflush = <iptables> -F f2b-<name>
-
-
-[Init]
-
-# Option:  chain
-# Notes    specifies the iptables chain to which the Fail2Ban rules should be
-#          added
-# Values:  STRING  Default: INPUT
-chain = INPUT
-
-# Default name of the chain
-#
-name = default
-
-# Option:  port
-# Notes.:  specifies port to monitor
-# Values:  [ NUM | STRING ]  Default:
-#
-port = ssh
-
-# Option:  protocol
-# Notes.:  internally used by config reader for interpolations.
-# Values:  [ tcp | udp | icmp | all ] Default: tcp
-#
-protocol = tcp
-
-# Option:  blocktype
-# Note:    This is what the action does with rules. This can be any jump target
-#          as per the iptables man page (section 8). Common values are DROP
-#          REJECT, REJECT --reject-with icmp-port-unreachable
-# Values:  STRING
-blocktype = REJECT --reject-with icmp-port-unreachable
-
-# Option:  returntype
-# Note:    This is the default rule on "actionstart". This should be RETURN
-#          in all (blocking) actions, except REJECT in allowing actions.
-# Values:  STRING
-returntype = RETURN
-
-# Option:  lockingopt
-# Notes.:  Option was introduced to iptables to prevent multiple instances from
-#          running concurrently and causing irratic behavior.  -w was introduced
-#          in iptables 1.4.20, so might be absent on older systems
-#          See https://github.com/fail2ban/fail2ban/issues/1122
-# Values:  STRING
-lockingopt = -w
-
-# Option:  iptables
-# Notes.:  Actual command to be executed, including common to all calls options
-# Values:  STRING
-iptables = iptables <lockingopt>
-
-
-[Init?family=inet6]
-
-# Option:  blocktype (ipv6)
-# Note:    This is what the action does with rules. This can be any jump target
-#          as per the iptables man page (section 8). Common values are DROP
-#          REJECT, REJECT --reject-with icmp6-port-unreachable
-# Values:  STRING
-blocktype = REJECT --reject-with icmp6-port-unreachable
-
-# Option:  iptables (ipv6)
-# Notes.:  Actual command to be executed, including common to all calls options
-# Values:  STRING
-iptables = ip6tables <lockingopt>
-

config/action.d/iptables-ipset-proto4.conf

@@ -19,7 +19,7 @@
 
 [INCLUDES]
 
-before = iptables-common.conf
+before = iptables.conf
 
 [Definition]
 
@@ -28,7 +28,7 @@ before = iptables-common.conf
 # Values:  CMD
 #
 actionstart = ipset --create f2b-<name> iphash
-              <iptables> -I <chain> -p <protocol> -m multiport --dports <port> -m set --match-set f2b-<name> src -j <blocktype>
+              <_ipt_add_rules>
 
 
 # Option:  actionflush
@@ -41,7 +41,7 @@ actionflush = ipset --flush f2b-<name>
 # Notes.:  command executed at the stop of jail (or at the end of Fail2Ban)
 # Values:  CMD
 #
-actionstop = <iptables> -D <chain> -p <protocol> -m multiport --dports <port> -m set --match-set f2b-<name> src -j <blocktype>
+actionstop = <_ipt_del_rules>
              <actionflush>
              ipset --destroy f2b-<name>
 
@@ -61,5 +61,6 @@ actionban = ipset --test f2b-<name> <ip> ||  ipset --add f2b-<name> <ip>
 #
 actionunban = ipset --test f2b-<name> <ip> && ipset --del f2b-<name> <ip>
 
-[Init]
+# Several capabilities used internaly:
 
+rule-jump = -m set --match-set f2b-<name> src -j <blocktype>

config/action.d/iptables-ipset-proto6-allports.conf

@@ -15,73 +15,13 @@
 #
 # Modified: Alexander Koeppe <format_c@online.de>, Serg G. Brester <serg.brester@sebres.de>
 #       made config file IPv6 capable (see new section Init?family=inet6)
+#
+# Obsolete: superseded by iptables-ipset[type=allports]
 
 [INCLUDES]
 
-before = iptables-common.conf
+before = iptables-ipset.conf
 
 [Definition]
 
-# Option:  actionstart
+type = allports
-# Notes.:  command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false).
-# Values:  CMD
-#
-actionstart = ipset create <ipmset> hash:ip timeout <default-ipsettime> <familyopt>
-              <iptables> -I <chain> -m set --match-set <ipmset> src -j <blocktype>
-
-# Option:  actionflush
-# Notes.:  command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action)
-# Values:  CMD
-#
-actionflush = ipset flush <ipmset>
-
-# Option:  actionstop
-# Notes.:  command executed at the stop of jail (or at the end of Fail2Ban)
-# Values:  CMD
-#
-actionstop = <iptables> -D <chain> -m set --match-set <ipmset> src -j <blocktype>
-             <actionflush>
-             ipset destroy <ipmset>
-
-# Option:  actionban
-# Notes.:  command executed when banning an IP. Take care that the
-#          command is executed with Fail2Ban user rights.
-# Tags:    See jail.conf(5) man page
-# Values:  CMD
-#
-actionban = ipset add <ipmset> <ip> timeout <ipsettime> -exist
-
-# actionprolong = %(actionban)s
-
-# Option:  actionunban
-# Notes.:  command executed when unbanning an IP. Take care that the
-#          command is executed with Fail2Ban user rights.
-# Tags:    See jail.conf(5) man page
-# Values:  CMD
-#
-actionunban = ipset del <ipmset> <ip> -exist
-
-[Init]
-
-# Option: default-ipsettime
-# Notes:  specifies default timeout in seconds (handled default ipset timeout only)
-# Values:  [ NUM ]  Default: 0 (no timeout, managed by fail2ban by unban)
-default-ipsettime = 0
-
-# Option: ipsettime
-# Notes:  specifies ticket timeout (handled ipset timeout only)
-# Values:  [ NUM ]  Default: 0 (managed by fail2ban by unban)
-ipsettime = 0
-
-# expresion to caclulate timeout from bantime, example:
-# banaction = %(known/banaction)s[ipsettime='<timeout-bantime>']
-timeout-bantime = $([ "<bantime>" -le 2147483 ] && echo "<bantime>" || echo 0)
-
-ipmset = f2b-<name>
-familyopt =
-
-
-[Init?family=inet6]
-
-ipmset = f2b-<name>6
-familyopt = family inet6

config/action.d/iptables-ipset-proto6.conf

@@ -15,73 +15,13 @@
 #
 # Modified: Alexander Koeppe <format_c@online.de>, Serg G. Brester <serg.brester@sebres.de>
 #       made config file IPv6 capable (see new section Init?family=inet6)
+#
+# Obsolete: superseded by iptables-ipset[type=multiport]
 
 [INCLUDES]
 
-before = iptables-common.conf
+before = iptables-ipset.conf
 
 [Definition]
 
-# Option:  actionstart
+type = multiport
-# Notes.:  command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false).
-# Values:  CMD
-#
-actionstart = ipset create <ipmset> hash:ip timeout <default-ipsettime> <familyopt>
-              <iptables> -I <chain> -p <protocol> -m multiport --dports <port> -m set --match-set <ipmset> src -j <blocktype>
-
-# Option:  actionflush
-# Notes.:  command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action)
-# Values:  CMD
-#
-actionflush = ipset flush <ipmset>
-
-# Option:  actionstop
-# Notes.:  command executed at the stop of jail (or at the end of Fail2Ban)
-# Values:  CMD
-#
-actionstop = <iptables> -D <chain> -p <protocol> -m multiport --dports <port> -m set --match-set <ipmset> src -j <blocktype>
-             <actionflush>
-             ipset destroy <ipmset>
-
-# Option:  actionban
-# Notes.:  command executed when banning an IP. Take care that the
-#          command is executed with Fail2Ban user rights.
-# Tags:    See jail.conf(5) man page
-# Values:  CMD
-#
-actionban = ipset add <ipmset> <ip> timeout <ipsettime> -exist
-
-# actionprolong = %(actionban)s
-
-# Option:  actionunban
-# Notes.:  command executed when unbanning an IP. Take care that the
-#          command is executed with Fail2Ban user rights.
-# Tags:    See jail.conf(5) man page
-# Values:  CMD
-#
-actionunban = ipset del <ipmset> <ip> -exist
-
-[Init]
-
-# Option: default-ipsettime
-# Notes:  specifies default timeout in seconds (handled default ipset timeout only)
-# Values:  [ NUM ]  Default: 0 (no timeout, managed by fail2ban by unban)
-default-ipsettime = 0
-
-# Option: ipsettime
-# Notes:  specifies ticket timeout (handled ipset timeout only)
-# Values:  [ NUM ]  Default: 0 (managed by fail2ban by unban)
-ipsettime = 0
-
-# expresion to caclulate timeout from bantime, example:
-# banaction = %(known/banaction)s[ipsettime='<timeout-bantime>']
-timeout-bantime = $([ "<bantime>" -le 2147483 ] && echo "<bantime>" || echo 0)
-
-ipmset = f2b-<name>
-familyopt =
-
-
-[Init?family=inet6]
-
-ipmset = f2b-<name>6
-familyopt = family inet6

config/action.d/iptables-ipset.conf

@@ -0,0 +1,90 @@
+# Fail2Ban configuration file
+#
+# Authors: Sergey G Brester (sebres), Daniel Black, Alexander Koeppe
+#
+# This is for ipset protocol 6 (and hopefully later) (ipset v6.14).
+# Use ipset -V to see the protocol and version. Version 4 should use
+# iptables-ipset-proto4.conf.
+#
+# This requires the program ipset which is normally in package called ipset.
+#
+# IPset was a feature introduced in the linux kernel 2.6.39 and 3.0.0 kernels.
+#
+# If you are running on an older kernel you make need to patch in external
+# modules.
+#
+
+[INCLUDES]
+
+before = iptables.conf
+
+[Definition]
+
+# Option:  actionstart
+# Notes.:  command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false).
+# Values:  CMD
+#
+actionstart = ipset -exist create <ipmset> hash:ip timeout <default-ipsettime> <familyopt>
+              <_ipt_add_rules>
+
+# Option:  actionflush
+# Notes.:  command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action)
+# Values:  CMD
+#
+actionflush = ipset flush <ipmset>
+
+# Option:  actionstop
+# Notes.:  command executed at the stop of jail (or at the end of Fail2Ban)
+# Values:  CMD
+#
+actionstop = <_ipt_del_rules>
+             <actionflush>
+             ipset destroy <ipmset>
+
+# Option:  actionban
+# Notes.:  command executed when banning an IP. Take care that the
+#          command is executed with Fail2Ban user rights.
+# Tags:    See jail.conf(5) man page
+# Values:  CMD
+#
+actionban = ipset -exist add <ipmset> <ip> timeout <ipsettime>
+
+# actionprolong = %(actionban)s
+
+# Option:  actionunban
+# Notes.:  command executed when unbanning an IP. Take care that the
+#          command is executed with Fail2Ban user rights.
+# Tags:    See jail.conf(5) man page
+# Values:  CMD
+#
+actionunban = ipset -exist del <ipmset> <ip>
+
+# Several capabilities used internaly:
+
+rule-jump = -m set --match-set <ipmset> src -j <blocktype>
+
+
+[Init]
+
+# Option: default-ipsettime
+# Notes:  specifies default timeout in seconds (handled default ipset timeout only)
+# Values:  [ NUM ]  Default: 0 (no timeout, managed by fail2ban by unban)
+default-ipsettime = 0
+
+# Option: ipsettime
+# Notes:  specifies ticket timeout (handled ipset timeout only)
+# Values:  [ NUM ]  Default: 0 (managed by fail2ban by unban)
+ipsettime = 0
+
+# expresion to caclulate timeout from bantime, example:
+# banaction = %(known/banaction)s[ipsettime='<timeout-bantime>']
+timeout-bantime = $([ "<bantime>" -le 2147483 ] && echo "<bantime>" || echo 0)
+
+ipmset = f2b-<name>
+familyopt =
+
+
+[Init?family=inet6]
+
+ipmset = f2b-<name>6
+familyopt = family inet6

config/action.d/iptables-multiport-log.conf

@@ -11,7 +11,7 @@
 
 [INCLUDES]
 
-before = iptables-common.conf
+before = iptables.conf
 
 [Definition]
 

config/action.d/iptables-multiport.conf

@@ -3,50 +3,12 @@
 # Author: Cyril Jaquier
 # Modified by Yaroslav Halchenko for multiport banning
 #
+# Obsolete: superseded by iptables[type=multiport]
 
 [INCLUDES]
 
-before = iptables-common.conf
+before = iptables.conf
 
 [Definition]
 
-# Option:  actionstart
+type = multiport
-# Notes.:  command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false).
-# Values:  CMD
-#
-actionstart = <iptables> -N f2b-<name>
-              <iptables> -A f2b-<name> -j <returntype>
-              <iptables> -I <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name>
-
-# Option:  actionstop
-# Notes.:  command executed at the stop of jail (or at the end of Fail2Ban)
-# Values:  CMD
-#
-actionstop = <iptables> -D <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name>
-             <actionflush>
-             <iptables> -X f2b-<name>
-
-# Option:  actioncheck
-# Notes.:  command executed once before each actionban command
-# Values:  CMD
-#
-actioncheck = <iptables> -n -L <chain> | grep -q 'f2b-<name>[ \t]'
-
-# Option:  actionban
-# Notes.:  command executed when banning an IP. Take care that the
-#          command is executed with Fail2Ban user rights.
-# Tags:    See jail.conf(5) man page
-# Values:  CMD
-#
-actionban = <iptables> -I f2b-<name> 1 -s <ip> -j <blocktype>
-
-# Option:  actionunban
-# Notes.:  command executed when unbanning an IP. Take care that the
-#          command is executed with Fail2Ban user rights.
-# Tags:    See jail.conf(5) man page
-# Values:  CMD
-#
-actionunban = <iptables> -D f2b-<name> -s <ip> -j <blocktype>
-
-[Init]
-

config/action.d/iptables-new.conf

@@ -4,51 +4,12 @@
 # Copied from iptables.conf and modified by Yaroslav Halchenko 
 #  to fulfill the needs of bugreporter dbts#350746.
 #
-#
+# Obsolete: superseded by iptables[pre-rule='-m state --state NEW<sp>']
 
 [INCLUDES]
 
-before = iptables-common.conf
+before = iptables.conf
 
 [Definition]
 
-# Option:  actionstart
+pre-rule = -m state --state NEW<sp>
-# Notes.:  command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false).
-# Values:  CMD
-#
-actionstart = <iptables> -N f2b-<name>
-              <iptables> -A f2b-<name> -j <returntype>
-              <iptables> -I <chain> -m state --state NEW -p <protocol> --dport <port> -j f2b-<name>
-
-# Option:  actionstop
-# Notes.:  command executed at the stop of jail (or at the end of Fail2Ban)
-# Values:  CMD
-#
-actionstop = <iptables> -D <chain> -m state --state NEW -p <protocol> --dport <port> -j f2b-<name>
-             <actionflush>
-             <iptables> -X f2b-<name>
-
-# Option:  actioncheck
-# Notes.:  command executed once before each actionban command
-# Values:  CMD
-#
-actioncheck = <iptables> -n -L <chain> | grep -q 'f2b-<name>[ \t]'
-
-# Option:  actionban
-# Notes.:  command executed when banning an IP. Take care that the
-#          command is executed with Fail2Ban user rights.
-# Tags:    See jail.conf(5) man page
-# Values:  CMD
-#
-actionban = <iptables> -I f2b-<name> 1 -s <ip> -j <blocktype>
-
-# Option:  actionunban
-# Notes.:  command executed when unbanning an IP. Take care that the
-#          command is executed with Fail2Ban user rights.
-# Tags:    See jail.conf(5) man page
-# Values:  CMD
-#
-actionunban = <iptables> -D f2b-<name> -s <ip> -j <blocktype>
-
-[Init]
-

config/action.d/iptables-xt_recent-echo.conf

@@ -7,10 +7,14 @@
 
 [INCLUDES]
 
-before = iptables-common.conf
+before = iptables.conf
 
 [Definition]
 
+_ipt_chain_rule = -m recent --update --seconds 3600 --name <iptname> -j <blocktype>
+_ipt_for_proto-iter =
+_ipt_for_proto-done =
+
 # Option:  actionstart
 # Notes.:  command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false).
 # Values:  CMD
@@ -33,7 +37,9 @@ before = iptables-common.conf
 #    own rules. The 3600 second timeout is independent and acts as a
 #    safeguard in case the fail2ban process dies unexpectedly. The
 #    shorter of the two timeouts actually matters.
-actionstart = if [ `id -u` -eq 0 ];then <iptables> -I <chain> -m recent --update --seconds 3600 --name <iptname> -j <blocktype>;fi
+actionstart = if [ `id -u` -eq 0 ];then
+              { %(_ipt_check_rule)s >/dev/null 2>&1; } || { <iptables> -I <chain> %(_ipt_chain_rule)s; }
+              fi
 
 # Option:  actionflush
 #
@@ -46,13 +52,15 @@ actionflush =
 # Values:  CMD
 #
 actionstop = echo / > /proc/net/xt_recent/<iptname>
-             if [ `id -u` -eq 0 ];then <iptables> -D <chain> -m recent --update --seconds 3600 --name <iptname> -j <blocktype>;fi
+             if [ `id -u` -eq 0 ];then
+             <iptables> -D <chain> %(_ipt_chain_rule)s;
+             fi
 
 # Option:  actioncheck
-# Notes.:  command executed once before each actionban command
+# Notes.:  command executed as invariant check (error by ban)
 # Values:  CMD
 #
-actioncheck = test -e /proc/net/xt_recent/<iptname>
+actioncheck = { <iptables> -C <chain> %(_ipt_chain_rule)s; } && test -e /proc/net/xt_recent/<iptname>
 
 # Option:  actionban
 # Notes.:  command executed when banning an IP. Take care that the

config/action.d/iptables.conf

@@ -1,28 +1,35 @@
 # Fail2Ban configuration file
 #
-# Author: Cyril Jaquier
+# Authors: Sergey G. Brester (sebres), Cyril Jaquier, Daniel Black, 
+#          Yaroslav O. Halchenko, Alexander Koeppe et al.
 #
-#
-
-[INCLUDES]
-
-before = iptables-common.conf
 
 [Definition]
 
+# Option:  type
+# Notes.:  type of the action.
+# Values:  [ oneport | multiport | allports ]  Default: oneport
+#
+type = oneport
+
+# Option:  actionflush
+# Notes.:  command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action)
+# Values:  CMD
+#
+actionflush = <iptables> -F f2b-<name>
+
 # Option:  actionstart
 # Notes.:  command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false).
 # Values:  CMD
 #
-actionstart = <iptables> -N f2b-<name>
+actionstart = { <iptables> -C f2b-<name> -j <returntype> >/dev/null 2>&1; } || { <iptables> -N f2b-<name> || true; <iptables> -A f2b-<name> -j <returntype>; }
-              <iptables> -A f2b-<name> -j <returntype>
+              <_ipt_add_rules>
-              <iptables> -I <chain> -p <protocol> --dport <port> -j f2b-<name>
 
 # Option:  actionstop
 # Notes.:  command executed at the stop of jail (or at the end of Fail2Ban)
 # Values:  CMD
 #
-actionstop = <iptables> -D <chain> -p <protocol> --dport <port> -j f2b-<name>
+actionstop = <_ipt_del_rules>
              <actionflush>
              <iptables> -X f2b-<name>
 
@@ -30,7 +37,7 @@ actionstop = <iptables> -D <chain> -p <protocol> --dport <port> -j f2b-<name>
 # Notes.:  command executed once before each actionban command
 # Values:  CMD
 #
-actioncheck = <iptables> -n -L <chain> | grep -q 'f2b-<name>[ \t]'
+actioncheck = <_ipt_check_rules>
 
 # Option:  actionban
 # Notes.:  command executed when banning an IP. Take care that the
@@ -48,5 +55,108 @@ actionban = <iptables> -I f2b-<name> 1 -s <ip> -j <blocktype>
 #
 actionunban = <iptables> -D f2b-<name> -s <ip> -j <blocktype>
 
+# Option:  pre-rule
+# Notes.:  prefix parameter(s) inserted to the begin of rule. No default (empty)
+#
+pre-rule =
+
+rule-jump = -j <_ipt_rule_target>
+
+# Several capabilities used internaly:
+
+_ipt_for_proto-iter = for proto in $(echo '<protocol>' | sed 's/,/ /g'); do
+_ipt_for_proto-done = done
+
+_ipt_add_rules = <_ipt_for_proto-iter>
+              { %(_ipt_check_rule)s >/dev/null 2>&1; } || { <iptables> -I <chain> %(_ipt_chain_rule)s; }
+              <_ipt_for_proto-done>
+
+_ipt_del_rules = <_ipt_for_proto-iter>
+              <iptables> -D <chain> %(_ipt_chain_rule)s
+              <_ipt_for_proto-done>
+
+_ipt_check_rules = <_ipt_for_proto-iter>
+              %(_ipt_check_rule)s
+              <_ipt_for_proto-done>
+
+_ipt_chain_rule = <pre-rule><ipt_<type>/_chain_rule>
+_ipt_check_rule = <iptables> -C <chain> %(_ipt_chain_rule)s
+_ipt_rule_target = f2b-<name>
+
+[ipt_oneport]
+
+_chain_rule = -p $proto --dport <port> <rule-jump>
+
+[ipt_multiport]
+
+_chain_rule = -p $proto -m multiport --dports <port> <rule-jump>
+
+[ipt_allports]
+
+_chain_rule = -p $proto <rule-jump>
+
+
 [Init]
 
+# Option:  chain
+# Notes    specifies the iptables chain to which the Fail2Ban rules should be
+#          added
+# Values:  STRING  Default: INPUT
+chain = INPUT
+
+# Default name of the chain
+#
+name = default
+
+# Option:  port
+# Notes.:  specifies port to monitor
+# Values:  [ NUM | STRING ]  Default:
+#
+port = ssh
+
+# Option:  protocol
+# Notes.:  internally used by config reader for interpolations.
+# Values:  [ tcp | udp | icmp | all ] Default: tcp
+#
+protocol = tcp
+
+# Option:  blocktype
+# Note:    This is what the action does with rules. This can be any jump target
+#          as per the iptables man page (section 8). Common values are DROP
+#          REJECT, REJECT --reject-with icmp-port-unreachable
+# Values:  STRING
+blocktype = REJECT --reject-with icmp-port-unreachable
+
+# Option:  returntype
+# Note:    This is the default rule on "actionstart". This should be RETURN
+#          in all (blocking) actions, except REJECT in allowing actions.
+# Values:  STRING
+returntype = RETURN
+
+# Option:  lockingopt
+# Notes.:  Option was introduced to iptables to prevent multiple instances from
+#          running concurrently and causing irratic behavior.  -w was introduced
+#          in iptables 1.4.20, so might be absent on older systems
+#          See https://github.com/fail2ban/fail2ban/issues/1122
+# Values:  STRING
+lockingopt = -w
+
+# Option:  iptables
+# Notes.:  Actual command to be executed, including common to all calls options
+# Values:  STRING
+iptables = iptables <lockingopt>
+
+
+[Init?family=inet6]
+
+# Option:  blocktype (ipv6)
+# Note:    This is what the action does with rules. This can be any jump target
+#          as per the iptables man page (section 8). Common values are DROP
+#          REJECT, REJECT --reject-with icmp6-port-unreachable
+# Values:  STRING
+blocktype = REJECT --reject-with icmp6-port-unreachable
+
+# Option:  iptables (ipv6)
+# Notes.:  Actual command to be executed, including common to all calls options
+# Values:  STRING
+iptables = ip6tables <lockingopt>

config/action.d/ipthreat.conf

@@ -0,0 +1,107 @@
+# IPThreat configuration file
+#
+# Added to fail2ban by Jeff Johnson (jjxtra)
+#
+# Action to report IP address to ipthreat.net
+#
+# You must sign up to obtain an API key from ipthreat.net and request bulk report permissions
+# https://ipthreat.net/integrations
+#
+# IPThreat is a 100% free site and service, all data is licensed under a creative commons by attribution license
+# Please do not integrate if you do not agree to the license
+#
+# IMPORTANT:
+#
+# Reporting an IP is a serious action. Make sure that it is legit.
+# Consider using this action only for:
+#   * IP that has been banned more than once
+#   * High max retry to avoid user mis-typing password
+#   * Filters that are unlikely to be human error
+#
+# Example:
+# ```
+#   action = %(known/action)s
+#            ipthreat[]
+# ```
+#
+# The action accepts the following arguments: ipthreat[ipthreat_flags="8",ipthreat_system="SSH", ipthreat_apikey=...]
+# In most cases your action could be as simple as: ipthreat[], since the default flags and system are set to the most correct default values.
+# You can optionally override ipthreat_system and ipthreat_flags if desired.
+# The ipthreat_apikey must be set at the bottom of this configuration file.
+#
+# `ipthreat_system` is a short name of the system attacked, i.e. SSH, SMTP, MYSQL, PHP, etc.
+#
+# For `ipthreat_flags`, most cases will use 8 (BruteForce) which is the default, but you could use others.
+# You can use the name or the ordinal.
+# Multiple values are comma separated.
+# ```
+# Name           Ordinal Description
+# Dns            1       Abuse/attack of dns (domain name server)
+# Fraud          2       General fraud, whether orders, misuse of payment info, etc
+# DDos           4       Distributed denial of service attack, whether through http requests, large ping attack, etc
+# BruteForce     8       Brute force login attack
+# Proxy          16      IP is a proxy like TOR or other proxy server
+# Spam           32      Email, comment or other type of spam
+# Vpn            64      IP is part of a VPN
+# Hacking        128     General hacking outside of brute force attack (includes vulnerability scans, sql injection, etc.). Use port scan flag instead if it's just probe on ports.
+# BadBot         256     Bad bot that is not honoring robots.txt or just flooding with too many requests, etc
+# Compromised    512     The ip has been taken over by malware or botnet
+# Phishing       1024    The ip is involved in phishing or spoofing
+# Iot            2048    The ip has targetted an iot (Internet of Things) device
+# PortScan       4096    Port scan
+# See https://ipthreat.net/bulkreportformat for more information
+# ```
+
+[Definition]
+
+# bypass action for restored tickets
+norestored = 1
+
+# Option:  actionstart
+# Notes.:  command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false).
+# Values:  CMD
+#
+actionstart =
+
+# Option:  actionstop
+# Notes.:  command executed at the stop of jail (or at the end of Fail2Ban)
+# Values:  CMD
+#
+actionstop =
+
+# Option:  actioncheck
+# Notes.:  command executed once before each actionban command
+# Values:  CMD
+#
+actioncheck =
+
+# Option:  actionban
+# Notes.:  command executed when banning an IP. Take care that the
+#          command is executed with Fail2Ban user rights.
+#
+# Tags:    See jail.conf(5) man page
+# Values:  CMD
+#
+actionban = curl -sSf "https://api.ipthreat.net/api/report" -X POST -H "Content-Type: application/json" -H "X-API-KEY: <ipthreat_apikey>" -d "{\"ip\":\"<ip>\",\"flags\":\"<ipthreat_flags>\",\"system\":\"<ipthreat_system>\",\"notes\":\"fail2ban\"}"
+
+# Option:  actionunban
+# Notes.:  command executed when unbanning an IP. Take care that the
+#          command is executed with Fail2Ban user rights.
+# Tags:    See jail.conf(5) man page
+# Values:  CMD
+#
+actionunban =
+
+[Init]
+# Option:  ipthreat_apikey
+# Notes    Your API key from ipthreat.net
+# Values:  STRING Default: None
+# Register for ipthreat [https://ipthreat.net], get api key and set below.
+# You will need to set the flags and system in the action call in jail.conf
+ipthreat_apikey =
+
+# By default, the ipthreat system is the name of the fail2ban jail
+ipthreat_system = <name>
+
+# By default the ip threat flags is 8 (brute force), but you can override this per jail if desired
+ipthreat_flags = 8

config/action.d/mail-buffered.conf

@@ -17,7 +17,7 @@ actionstart = printf %%b "Hi,\n
               The jail <name> has been started successfully.\n
               Output will be buffered until <lines> lines are available.\n
               Regards,\n
-              Fail2Ban"|mail -s "[Fail2Ban] <name>: started on <fq-hostname>" <dest>
+              Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] <name>: started on <fq-hostname>" <dest>
 
 # Option:  actionstop
 # Notes.:  command executed at the stop of jail (or at the end of Fail2Ban)
@@ -28,13 +28,13 @@ actionstop = if [ -f <tmpfile> ]; then
                  These hosts have been banned by Fail2Ban.\n
                  `cat <tmpfile>`
                  Regards,\n
-                 Fail2Ban"|mail -s "[Fail2Ban] <name>: Summary from <fq-hostname>" <dest>
+                 Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] <name>: Summary from <fq-hostname>" <dest>
                  rm <tmpfile>
              fi
              printf %%b "Hi,\n
              The jail <name> has been stopped.\n
              Regards,\n
-             Fail2Ban"|mail -s "[Fail2Ban] <name>: stopped on <fq-hostname>" <dest>
+             Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] <name>: stopped on <fq-hostname>" <dest>
 
 # Option:  actioncheck
 # Notes.:  command executed once before each actionban command
@@ -55,7 +55,7 @@ actionban = printf %%b "`date`: <ip> (<failures> failures)\n" >> <tmpfile>
                 These hosts have been banned by Fail2Ban.\n
                 `cat <tmpfile>`
                 \nRegards,\n
-                Fail2Ban"|mail -s "[Fail2Ban] <name>: Summary" <dest>
+                Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] <name>: Summary" <dest>
                 rm <tmpfile>
             fi
 

config/action.d/mail-whois-lines.conf

@@ -72,7 +72,7 @@ actionunban =
 # Notes.:  Your system mail command. Is passed 2 args: subject and recipient
 # Values:  CMD
 #
-mailcmd = mail -s
+mailcmd = mail -E 'set escape' -s
 
 # Default name of the chain
 #

config/action.d/mail-whois.conf

@@ -20,7 +20,7 @@ norestored = 1
 actionstart = printf %%b "Hi,\n
               The jail <name> has been started successfully.\n
               Regards,\n
-              Fail2Ban"|mail -s "[Fail2Ban] <name>: started on <fq-hostname>" <dest>
+              Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] <name>: started on <fq-hostname>" <dest>
 
 # Option:  actionstop
 # Notes.:  command executed at the stop of jail (or at the end of Fail2Ban)
@@ -29,7 +29,7 @@ actionstart = printf %%b "Hi,\n
 actionstop = printf %%b "Hi,\n
              The jail <name> has been stopped.\n
              Regards,\n
-             Fail2Ban"|mail -s "[Fail2Ban] <name>: stopped on <fq-hostname>" <dest>
+             Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] <name>: stopped on <fq-hostname>" <dest>
 
 # Option:  actioncheck
 # Notes.:  command executed once before each actionban command
@@ -49,7 +49,7 @@ actionban = printf %%b "Hi,\n
             Here is more information about <ip> :\n
             `%(_whois_command)s`\n
             Regards,\n
-            Fail2Ban"|mail -s "[Fail2Ban] <name>: banned <ip> from <fq-hostname>" <dest>
+            Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] <name>: banned <ip> from <fq-hostname>" <dest>
 
 # Option:  actionunban
 # Notes.:  command executed when unbanning an IP. Take care that the

config/action.d/mail.conf

@@ -16,7 +16,7 @@ norestored = 1
 actionstart = printf %%b "Hi,\n
               The jail <name> has been started successfully.\n
               Regards,\n
-              Fail2Ban"|mail -s "[Fail2Ban] <name>: started  on <fq-hostname>" <dest>
+              Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] <name>: started  on <fq-hostname>" <dest>
 
 # Option:  actionstop
 # Notes.:  command executed at the stop of jail (or at the end of Fail2Ban)
@@ -25,7 +25,7 @@ actionstart = printf %%b "Hi,\n
 actionstop = printf %%b "Hi,\n
              The jail <name> has been stopped.\n
              Regards,\n
-             Fail2Ban"|mail -s "[Fail2Ban] <name>: stopped on <fq-hostname>" <dest>
+             Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] <name>: stopped on <fq-hostname>" <dest>
 
 # Option:  actioncheck
 # Notes.:  command executed once before each actionban command
@@ -43,7 +43,7 @@ actionban = printf %%b "Hi,\n
             The IP <ip> has just been banned by Fail2Ban after
             <failures> attempts against <name>.\n
             Regards,\n
-            Fail2Ban"|mail -s "[Fail2Ban] <name>: banned <ip> from <fq-hostname>" <dest>
+            Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] <name>: banned <ip> from <fq-hostname>" <dest>
 
 # Option:  actionunban
 # Notes.:  command executed when unbanning an IP. Take care that the

config/action.d/nginx-block-map.conf

@@ -84,8 +84,15 @@ srv_cfg_path = /etc/nginx/
 #srv_cmd = nginx -c %(srv_cfg_path)s/nginx.conf
 srv_cmd = nginx
 
-# first test configuration is correct, hereafter send reload signal:
+# pid file (used to check nginx is running):
-blck_lst_reload = %(srv_cmd)s -qt; if [ $? -eq 0 ]; then
+srv_pid = /run/nginx.pid
+
+# command used to check whether nginx is running and configuration is valid:
+srv_is_running = [ -f "%(srv_pid)s" ]
+srv_check_cmd = %(srv_is_running)s && %(srv_cmd)s -qt
+
+# first test nginx is running and configuration is correct, hereafter send reload signal:
+blck_lst_reload = %(srv_check_cmd)s; if [ $? -eq 0 ]; then
                     %(srv_cmd)s -s reload; if [ $? -ne 0 ]; then echo 'reload failed.'; fi;
                   fi;
 

config/action.d/symbiosis-blacklist-allports.conf

@@ -5,7 +5,7 @@
 
 [INCLUDES]
 
-before = iptables-common.conf
+before = iptables.conf
 
 [Definition]
 
@@ -41,6 +41,11 @@ actionban = echo 'all' >| /etc/symbiosis/firewall/blacklist.d/<ip>.auto
 actionunban = rm -f /etc/symbiosis/firewall/blacklist.d/<ip>.auto
               <iptables> -D <chain> -s <ip> -j <blocktype> || :
 
+# [TODO] Flushing is currently not implemented for symbiosis blacklist.d
+#
+actionflush = 
+
+
 [Init]
 
 # Option:  chain

config/action.d/ufw.conf

@@ -13,16 +13,45 @@ actionstop =
 
 actioncheck = 
 
-actionban = [ -n "<application>" ] && app="app <application>"
+# ufw does "quickly process packets for which we already have a connection" in before.rules,
-            ufw insert <insertpos> <blocktype> from <ip> to <destination> $app
+# therefore all related sockets should be closed
+# actionban is using `ss` to do so, this only handles IPv4 and IPv6.
 
-actionunban = [ -n "<application>" ] && app="app <application>"
+actionban = if [ -n "<application>" ] && ufw app info "<application>"
-              ufw delete <blocktype> from <ip> to <destination> $app
+            then
+              ufw <add> <blocktype> from <ip> to <destination> app "<application>" comment "<comment>"
+            else
+              ufw <add> <blocktype> from <ip> to <destination> comment "<comment>"
+            fi
+            <kill>
+
+actionunban = if [ -n "<application>" ] && ufw app info "<application>"
+              then
+                ufw delete <blocktype> from <ip> to <destination> app "<application>"
+              else
+                ufw delete <blocktype> from <ip> to <destination>
+              fi
+
+# Option: kill-mode
+# Notes.: can be set to ss or conntrack (may be extended later with other modes) to immediately drop all connections from banned IP, default empty (no kill)
+# Example: banaction = ufw[kill-mode=ss]
+kill-mode =
+
+# intern conditional parameter used to provide killing mode after ban:
+_kill_ =
+_kill_ss = ss -K dst "[<ip>]"
+_kill_conntrack = conntrack -D -s "<ip>"
+
+# Option: kill
+# Notes.: can be used to specify custom killing feature, by default depending on option kill-mode
+# Examples: banaction = ufw[kill='ss -K "( sport = :http || sport = :https )" dst "[<ip>]"']
+#           banaction = ufw[kill='cutter "<ip>"']
+kill = <_kill_<kill-mode>>
 
 [Init]
-# Option: insertpos
+# Option: add
-# Notes.:  The position number in the firewall list to insert the block rule
+# Notes.: can be set to "insert 1" to insert a rule at certain position (here 1):
-insertpos = 1
+add = prepend
 
 # Option: blocktype
 # Notes.: reject or deny
@@ -36,6 +65,10 @@ destination = any
 # Notes.: application from sudo ufw app list
 application = 
 
+# Option: comment
+# Notes.: comment for rule added by fail2ban
+comment = by Fail2Ban after <failures> attempts against <name>
+
 # DEV NOTES:
 # 
 # Author: Guilhem Lettron

config/fail2ban.conf

@@ -24,13 +24,13 @@
 loglevel = INFO
 
 # Option: logtarget
-# Notes.: Set the log target. This could be a file, SYSLOG, STDERR or STDOUT.
+# Notes.: Set the log target. This could be a file, SYSTEMD-JOURNAL, SYSLOG, STDERR or STDOUT.
 #         Only one log target can be specified.
 #         If you change logtarget from the default value and you are
 #         using logrotate -- also adjust or disable rotation in the
 #         corresponding configuration file
 #         (e.g. /etc/logrotate.d/fail2ban on Debian systems)
-# Values: [ STDOUT | STDERR | SYSLOG | SYSOUT | FILE ]  Default: STDERR
+# Values: [ STDOUT | STDERR | SYSLOG | SYSOUT | SYSTEMD-JOURNAL | FILE ]  Default: STDERR
 #
 logtarget = /var/log/fail2ban.log
 
@@ -55,6 +55,12 @@ socket = /var/run/fail2ban/fail2ban.sock
 #
 pidfile = /var/run/fail2ban/fail2ban.pid
 
+# Option: allowipv6
+# Notes.: Allows IPv6 interface:
+#         Default: auto
+# Values: [ auto yes (on, true, 1) no (off, false, 0) ] Default: auto
+#allowipv6 = auto
+
 # Options: dbfile
 # Notes.: Set the file for the fail2ban persistent data to be stored.
 #         A value of ":memory:" means database is only stored in memory 

config/filter.d/apache-fakegooglebot.conf

@@ -2,11 +2,11 @@
 
 [Definition]
 
-failregex = ^<HOST> .*Googlebot.*$
+failregex = ^\s*<HOST> \S+ \S+(?: \S+)?\s+\S+ "[A-Z]+ /\S* [^"]*" \d+ \d+ \"[^"]*\" "[^"]*\bGooglebot/[^"]*"
 
 ignoreregex =
 
-datepattern = ^[^\[]*\[({DATE})
+datepattern = ^[^\[]*(\[{DATE}\s*\])
               {^LN-BEG}
 
 # DEV Notes:

config/filter.d/apache-overflows.conf

@@ -8,7 +8,7 @@ before = apache-common.conf
 
 [Definition]
 
-failregex = ^%(_apache_error_client)s (?:(?:AH0013[456]: )?Invalid (method|URI) in request\b|(?:AH00565: )?request failed: URI too long \(longer than \d+\)|request failed: erroneous characters after protocol string:|(?:AH00566: )?request failed: invalid characters in URI\b)
+failregex = ^%(_apache_error_client)s (?:(?:AH001[23][456]: )?Invalid (method|URI) in request\b|(?:AH00565: )?request failed: URI too long \(longer than \d+\)|request failed: erroneous characters after protocol string:|(?:AH00566: )?request failed: invalid characters in URI\b)
 
 ignoreregex =
 

config/filter.d/asterisk.conf

@@ -21,7 +21,7 @@ log_prefix= (?:NOTICE|SECURITY|WARNING)%(__pid_re)s:?(?:\[C-[\da-f]*\])?:? [^:]+
 prefregex = ^%(__prefix_line)s%(log_prefix)s <F-CONTENT>.+</F-CONTENT>$
 
 failregex = ^Registration from '[^']*' failed for '<HOST>(:\d+)?' - (?:Wrong password|Username/auth name mismatch|No matching peer found|Not a local domain|Device does not match ACL|Peer is not supposed to register|ACL error \(permit/deny\)|Not a local domain)$
-            ^Call from '[^']*' \(<HOST>:\d+\) to extension '[^']*' rejected because extension not found in context
+            ^Call from '[^']*' \((?:(?:TCP|UDP):)?<HOST>:\d+\) to extension '[^']*' rejected because extension not found in context
             ^(?:Host )?<HOST> (?:failed (?:to authenticate\b|MD5 authentication\b)|tried to authenticate with nonexistent user\b)
             ^No registration for peer '[^']*' \(from <HOST>\)$
             ^hacking attempt detected '<HOST>'$

config/filter.d/common.conf

@@ -10,7 +10,7 @@ after = common.local
 
 [DEFAULT]
 
-# Type of log-file resp. log-format (file, short, journal, rfc542):
+# Type of log-file resp. log-format (file, short, journal, rfc5424):
 logtype = file
 
 # Daemon definition is to be specialized (if needed) in .conf file

config/filter.d/courier-auth.conf

@@ -11,7 +11,7 @@ before = common.conf
 
 _daemon = (?:courier)?(?:imapd?|pop3d?)(?:login)?(?:-ssl)?
 
-failregex = ^%(__prefix_line)sLOGIN FAILED, (?:user|method)=.*, ip=\[<HOST>\]$
+failregex = ^%(__prefix_line)sLOGIN FAILED, (?:(?!ip=)(?:user=<F-USER>[^,]*</F-USER>|\w+=[^,]*), )*ip=\[<HOST>\]
 
 ignoreregex = 
 

config/filter.d/dovecot.conf

@@ -8,14 +8,15 @@ before = common.conf
 [Definition]
 
 _auth_worker = (?:dovecot: )?auth(?:-worker)?
+_auth_worker_info = (?:conn \w+:auth(?:-worker)? \([^\)]+\): auth(?:-worker)?<\d+>: )?
 _daemon = (?:dovecot(?:-auth)?|auth)
 
-prefregex = ^%(__prefix_line)s(?:%(_auth_worker)s(?:\([^\)]+\))?: )?(?:%(__pam_auth)s(?:\(dovecot:auth\))?: |(?:pop3|imap|managesieve|submission)-login: )?(?:Info: )?<F-CONTENT>.+</F-CONTENT>$
+prefregex = ^%(__prefix_line)s(?:%(_auth_worker)s(?:\([^\)]+\))?: )?(?:%(__pam_auth)s(?:\(dovecot:auth\))?: |(?:pop3|imap|managesieve|submission)-login: )?(?:Info: )?%(_auth_worker_info)s<F-CONTENT>.+</F-CONTENT>$
 
 failregex = ^authentication failure; logname=<F-ALT_USER1>\S*</F-ALT_USER1> uid=\S* euid=\S* tty=dovecot ruser=<F-USER>\S*</F-USER> rhost=<HOST>(?:\s+user=<F-ALT_USER>\S*</F-ALT_USER>)?\s*$
-            ^(?:Aborted login|Disconnected|Remote closed connection|Client has quit the connection)(?::(?: [^ \(]+)+)? \((?:auth failed, \d+ attempts(?: in \d+ secs)?|tried to use (?:disabled|disallowed) \S+ auth|proxy dest auth failed)\):(?: user=<<F-USER>[^>]*</F-USER>>,)?(?: method=\S+,)? rip=<HOST>(?:[^>]*(?:, session=<\S+>)?)\s*$
+            ^(?:Aborted login|Disconnected|Remote closed connection|Client has quit the connection)(?:: (?:[^\(]+|\w+\([^\)]*\))+)? \((?:auth failed, \d+ attempts(?: in \d+ secs)?|tried to use (?:disabled|disallowed) \S+ auth|proxy dest auth failed)\):(?: user=<<F-USER>[^>]*</F-USER>>,)?(?: method=\S+,)? rip=<HOST>(?:[^>]*(?:, session=<\S+>)?)\s*$
-            ^pam\(\S+,<HOST>(?:,\S*)?\): pam_authenticate\(\) failed: (?:User not known to the underlying authentication module: \d+ Time\(s\)|Authentication failure \(password mismatch\?\)|Permission denied)\s*$
+            ^pam\(\S+,<HOST>(?:,\S*)?\): pam_authenticate\(\) failed: (?:User not known to the underlying authentication module: \d+ Time\(s\)|Authentication failure \([Pp]assword mismatch\?\)|Permission denied)\s*$
-            ^[a-z\-]{3,15}\(\S*,<HOST>(?:,\S*)?\): (?:unknown user|invalid credentials|Password mismatch)
+            ^[a-z\-]{3,15}\(\S*,<HOST>(?:,\S*)?\): (?:[Uu]nknown user|[Ii]nvalid credentials|[Pp]assword mismatch)
             <mdre-<mode>>
 
 mdre-aggressive = ^(?:Aborted login|Disconnected|Remote closed connection|Client has quit the connection)(?::(?: [^ \(]+)+)? \((?:no auth attempts|disconnected before auth was ready,|client didn't finish \S+ auth,)(?: (?:in|waited) \d+ secs)?\):(?: user=<[^>]*>,)?(?: method=\S+,)? rip=<HOST>(?:[^>]*(?:, session=<\S+>)?)\s*$

config/filter.d/drupal-auth.conf

@@ -14,7 +14,7 @@ before = common.conf
 
 [Definition]
 
-failregex = ^%(__prefix_line)s(https?:\/\/)([\da-z\.-]+)\.([a-z\.]{2,6})(\/[\w\.-]+)*\|\d{10}\|user\|<HOST>\|.+\|.+\|\d\|.*\|Login attempt failed for .+\.$
+failregex = ^%(__prefix_line)s(?:https?:\/\/)[^|]+\|[^|]+\|[^|]+\|<ADDR>\|(?:[^|]*\|)*Login attempt failed (?:for|from) <F-USER>[^|]+</F-USER>\.$
 
 ignoreregex =
 

config/filter.d/exim-common.conf

@@ -12,7 +12,7 @@ after = exim-common.local
 host_info_pre = (?:H=([\w.-]+ )?(?:\(\S+\) )?)?
 host_info_suf = (?::\d+)?(?: I=\[\S+\](:\d+)?)?(?: U=\S+)?(?: P=e?smtp)?(?: F=(?:<>|[^@]+@\S+))?\s
 host_info = %(host_info_pre)s\[<HOST>\]%(host_info_suf)s
-pid = (?: \[\d+\])?
+pid = (?: \[\d+\]| \w+ exim\[\d+\]:)?
 
 # DEV Notes:
 # From exim source code: ./src/receive.c:add_host_info_for_log

config/filter.d/ignorecommands/apache-fakegooglebot

@@ -6,24 +6,35 @@
 #
 import sys
 from fail2ban.server.ipdns import DNSUtils, IPAddr
+from threading import Thread
 
 def process_args(argv):
-    if len(argv) != 2:
+    if len(argv) - 1 not in (1, 2):
-       raise ValueError("Please provide a single IP as an argument. Got: %s\n"
+       raise ValueError("Usage %s ip ?timeout?. Got: %s\n"
-                        % (argv[1:]))
+                        % (argv[0], argv[1:]))
     ip = argv[1]
 
     if not IPAddr(ip).isValid:
        raise ValueError("Argument must be a single valid IP. Got: %s\n"
                         % ip)
-    return ip
+    return argv[1:]
 
 google_ips = None
 
-def is_googlebot(ip):
+def is_googlebot(ip, timeout=55):
     import re
 
+    timeout = float(timeout or 0)
+    if timeout:
+      def ipToNameTO(host, ip, timeout):
+        host[0] = DNSUtils.ipToName(ip)
+      host = [None]
+      th = Thread(target=ipToNameTO, args=(host, ip, timeout)); th.daemon=True; th.start()
+      th.join(timeout)
+      host = host[0]
+    else:
       host = DNSUtils.ipToName(ip)
+
     if not host or not re.match(r'.*\.google(bot)?\.com$', host):
        return False
     host_ips = DNSUtils.dnsToIp(host)
@@ -31,7 +42,7 @@ def is_googlebot(ip):
 
 if __name__ == '__main__': # pragma: no cover
     try:
-      ret = is_googlebot(process_args(sys.argv))
+      ret = is_googlebot(*process_args(sys.argv))
     except ValueError as e:
       sys.stderr.write(str(e))
       sys.exit(2)

config/filter.d/lighttpd-auth.conf

@@ -3,7 +3,7 @@
 
 [Definition]
 
-failregex = ^: \((?:http|mod)_auth\.c\.\d+\) (?:password doesn\'t match .* username: .*|digest: auth failed for .*: wrong password|get_password failed), IP: <HOST>\s*$
+failregex = ^\s*(?:: )?\(?(?:http|mod)_auth\.c\.\d+\) (?:password doesn\'t match for (?:\S+|.*?) username:\s+<F-USER>(?:\S+|.*?)</F-USER>\s*|digest: auth failed(?: for\s+<F-ALT_USER>(?:\S+|.*?)</F-ALT_USER>\s*)?: (?:wrong password|uri mismatch \([^\)]*\))|get_password failed),? IP: <HOST>\s*$
 
 ignoreregex = 
 

config/filter.d/monitorix.conf

@@ -0,0 +1,25 @@
+# Fail2Ban filter for Monitorix (HTTP built-in server)
+#
+
+[INCLUDES]
+
+before = common.conf
+
+[Definition]
+
+_daemon = monitorix-httpd
+
+# Option:  failregex
+# Notes.:  regex to match the password failures messages in the logfile. The
+#          host must be matched by a group named "host". The tag "<HOST>" can
+#          be used for standard IP/hostname matching and is only an alias for
+#          (?:::f{4,6}:)?(?P<host>\S+)
+# Values:  TEXT
+#
+failregex = ^(?:\s+-)?\s*(?:NOTEXIST|AUTHERR|NOTALLOWED) - <ADDR>\b
+
+# Option:  ignoreregex
+# Notes.:  regex to ignore. If this regex matches, the line is ignored.
+# Values:  TEXT
+#
+ignoreregex =

config/filter.d/mssql-auth.conf

@@ -0,0 +1,15 @@
+# Fail2Ban filter for failed MSSQL Server authentication attempts
+
+[Definition]
+
+failregex = ^\s*Logon\s+Login failed for user '<F-USER>(?:[^']*|.*)</F-USER>'\. [^'\[]+\[CLIENT: <ADDR>\]$
+
+
+# DEV Notes:
+#	Tested with SQL Server 2019 on Ubuntu 18.04
+#
+# Example:
+#	2020-02-24 14:48:55.12 Logon       Login failed for user 'root'. Reason: Could not find a login matching the name provided. [CLIENT: 127.0.0.1]
+#
+# Author: Rüdiger Olschewsky
+#

config/filter.d/named-refused.conf

@@ -22,7 +22,7 @@
 [Definition]
 
 # Daemon name
-_daemon=named
+_daemon=named(?:-\w+)?
 
 # Shortcuts for easier comprehension of the failregex
 
@@ -32,7 +32,7 @@ __daemon_combs_re=(?:%(__pid_re)s?:\s+%(__daemon_re)s|%(__daemon_re)s%(__pid_re)
 
 #       hostname       daemon_id         spaces
 # this can be optional (for instance if we match named native log files)
-__line_prefix=(?:\s\S+ %(__daemon_combs_re)s\s+)?
+__line_prefix=(?:\s*\S+ %(__daemon_combs_re)s\s+)?
 
 prefregex = ^%(__line_prefix)s(?: error:)?\s*client(?: @\S*)? <HOST>#\S+(?: \([\S.]+\))?: <F-CONTENT>.+</F-CONTENT>\s(?:denied|\(NOTAUTH\))\s*$
 

config/filter.d/nginx-bad-request.conf

@@ -0,0 +1,16 @@
+# Fail2Ban filter to match bad requests to nginx
+#
+
+[Definition]
+
+# The request often doesn't contain a method, only some encoded garbage
+# This will also match requests that are entirely empty
+failregex = ^<HOST> - \S+ \[\] "[^"]*" 400
+
+datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]%%f)?(?:\s*%%z)?
+              ^[^\[]*\[({DATE})
+              {^LN-BEG}
+
+journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx
+
+# Author: Jan Przybylak

config/filter.d/nginx-botsearch.conf

@@ -17,6 +17,8 @@ datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]
               ^[^\[]*\[({DATE})
               {^LN-BEG}
 
+journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx
+
 # DEV Notes:
 # Based on apache-botsearch filter
 # 

config/filter.d/nginx-http-auth.conf

@@ -3,15 +3,32 @@
 
 [Definition]
 
+mode = normal
 
-failregex = ^ \[error\] \d+#\d+: \*\d+ user "(?:[^"]+|.*?)":? (?:password mismatch|was not found in "[^\"]*"), client: <HOST>, server: \S*, request: "\S+ \S+ HTTP/\d+\.\d+", host: "\S+"(?:, referrer: "\S+")?\s*$
+mdre-auth = ^\s*\[error\] \d+#\d+: \*\d+ user "(?:[^"]+|.*?)":? (?:password mismatch|was not found in "[^\"]*"), client: <HOST>, server: \S*, request: "\S+ \S+ HTTP/\d+\.\d+", host: "\S+"(?:, referrer: "\S+")?\s*$
+mdre-fallback = ^\s*\[crit\] \d+#\d+: \*\d+ SSL_do_handshake\(\) failed \(SSL: error:\S+(?: \S+){1,3} too (?:long|short)\)[^,]*, client: <HOST>
+
+mdre-normal = %(mdre-auth)s
+mdre-aggressive = %(mdre-auth)s
+                  %(mdre-fallback)s
+
+failregex = <mdre-<mode>>
 
 ignoreregex = 
 
 datepattern = {^LN-BEG}
 
+journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx
+
 # DEV NOTES:
+# mdre-auth:
 # Based on samples in https://github.com/fail2ban/fail2ban/pull/43/files
 # Extensive search of all nginx auth failures not done yet.
 # 
 # Author: Daniel Black
+
+# mdre-fallback:
+# Ban people checking for TLS_FALLBACK_SCSV repeatedly
+# https://stackoverflow.com/questions/28010492/nginx-critical-error-with-ssl-handshaking/28010608#28010608
+# Author: Stephan Orlowsky
+

config/filter.d/nginx-limit-req.conf

@@ -44,3 +44,6 @@ failregex = ^\s*\[[a-z]+\] \d+#\d+: \*\d+ limiting requests, excess: [\d\.]+ by
 ignoreregex = 
 
 datepattern = {^LN-BEG}
+
+journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx
+

config/filter.d/nsd.conf

@@ -22,8 +22,8 @@ _daemon = nsd
 #          (?:::f{4,6}:)?(?P<host>[\w\-.^_]+)
 # Values:  TEXT
 
-failregex =  ^%(__prefix_line)sinfo: ratelimit block .* query <HOST> TYPE255$
+failregex =  ^%(__prefix_line)sinfo: ratelimit block .* query <ADDR> TYPE255$
-             ^%(__prefix_line)sinfo: .* <HOST> refused, no acl matches\.$
+             ^%(__prefix_line)sinfo: .* from(?: client)? <ADDR> refused, no acl matches\.?$
 
 ignoreregex =
 

config/filter.d/postfix.conf

@@ -12,16 +12,15 @@ before = common.conf
 
 _daemon = postfix(-\w+)?/\w+(?:/smtp[ds])?
 _port = (?::\d+)?
+_pref = [A-Z]{4}
 
 prefregex = ^%(__prefix_line)s<mdpr-<mode>> <F-CONTENT>.+</F-CONTENT>$
 
-mdpr-normal = (?:\w+: reject:|(?:improper command pipelining|too many errors) after \S+)
+# Extended RE for normal mode to match reject by unknown users or undeliverable address, can be set to empty to avoid this:
-mdre-normal=^RCPT from [^[]*\[<HOST>\]%(_port)s: 55[04] 5\.7\.1\s
+exre-user = |[Uu](?:ser unknown|ndeliverable address)
-            ^RCPT from [^[]*\[<HOST>\]%(_port)s: 45[04] 4\.7\.\d+ (?:Service unavailable\b|Client host rejected: cannot find your (reverse )?hostname\b)
+
-            ^RCPT from [^[]*\[<HOST>\]%(_port)s: 450 4\.7\.\d+ (<[^>]*>)?: Helo command rejected: Host not found\b
+mdpr-normal = (?:\w+: (?:milter-)?reject:|(?:improper command pipelining|too many errors) after \S+)
-            ^EHLO from [^[]*\[<HOST>\]%(_port)s: 504 5\.5\.\d+ (<[^>]*>)?: Helo command rejected: need fully-qualified hostname\b
+mdre-normal=^%(_pref)s from [^[]*\[<HOST>\]%(_port)s: [45][50][04] [45]\.\d\.\d+ (?:(?:<[^>]*>)?: )?(?:(?:Helo command|(?:Sender|Recipient) address) rejected: )?(?:Service unavailable|(?:Client host|Command|Data command) rejected|Relay access denied|(?:Host|Domain) not found|need fully-qualified hostname|match%(exre-user)s)\b
-            ^(RCPT|VRFY) from [^[]*\[<HOST>\]%(_port)s: 550 5\.1\.1\s
-            ^RCPT from [^[]*\[<HOST>\]%(_port)s: 450 4\.1\.\d+ (<[^>]*>)?: Sender address rejected: Domain not found\b
             ^from [^[]*\[<HOST>\]%(_port)s:?
 
 mdpr-auth = warning:
@@ -31,13 +30,15 @@ mdre-auth2= ^[^[]*\[<HOST>\]%(_port)s: SASL ((?i)LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5
 
 # Mode "rbl" currently included in mode "normal", but if needed for jail "postfix-rbl" only:
 mdpr-rbl = %(mdpr-normal)s
-mdre-rbl  = ^RCPT from [^[]*\[<HOST>\]%(_port)s: [45]54 [45]\.7\.1 Service unavailable; Client host \[\S+\] blocked\b
+mdre-rbl  = ^%(_pref)s from [^[]*\[<HOST>\]%(_port)s: [45]54 [45]\.7\.1 Service unavailable; Client host \[\S+\] blocked\b
 
 # Mode "rbl" currently included in mode "normal" (within 1st rule)
 mdpr-more = %(mdpr-normal)s
 mdre-more = %(mdre-normal)s
 
-mdpr-ddos = (?:lost connection after(?! DATA) [A-Z]+|disconnect(?= from \S+(?: \S+=\d+)* auth=0/(?:[1-9]|\d\d+)))
+# Includes some of the log messages described in
+# <http://www.postfix.org/POSTSCREEN_README.html>.
+mdpr-ddos = (?:lost connection after(?! DATA) [A-Z]+|disconnect(?= from \S+(?: \S+=\d+)* auth=0/(?:[1-9]|\d\d+))|(?:PREGREET \d+|HANGUP) after \S+|COMMAND (?:TIME|COUNT|LENGTH) LIMIT)
 mdre-ddos = ^from [^[]*\[<HOST>\]%(_port)s:?
 
 mdpr-extra = (?:%(mdpr-auth)s|%(mdpr-normal)s)

config/filter.d/scanlogd.conf

@@ -0,0 +1,17 @@
+# Fail2Ban filter for port scans detected by scanlogd
+
+[INCLUDES]
+
+# Read common prefixes. If any customizations available -- read them from
+# common.local
+before = common.conf
+
+[Definition]
+
+_daemon = scanlogd
+
+failregex =  ^%(__prefix_line)s<ADDR>(?::<F-PORT/>)? to \S+ ports\b
+
+ignoreregex =
+
+# Author: Mike Gabriel <mike.gabriel@das-netzwerkteam.de>

config/filter.d/sendmail-auth.conf

@@ -15,7 +15,7 @@ addr = (?:IPv6:<IP6>|<IP4>)
 prefregex = ^<F-MLFID>%(__prefix_line)s</F-MLFID><F-CONTENT>.+</F-CONTENT>$
 
 failregex = ^(\S+ )?\[%(addr)s\]( \(may be forged\))?: possible SMTP attack: command=AUTH, count=\d+$
-            ^AUTH failure \(LOGIN\):(?: [^:]+:)? authentication failure: checkpass failed, user=<F-USER>(?:\S+|.*?)</F-USER>, relay=(?:\S+ )?\[%(addr)s\](?: \(may be forged\))?$
+            ^AUTH failure \([^\)]+\):(?: [^:]+:)? (?:authentication failure|user not found): [^,]*, (?:user=<F-USER>(?:\S+|.*?)</F-USER>, )?relay=(?:\S+ )?\[%(addr)s\](?: \(may be forged\))?$
 ignoreregex =
 
 journalmatch = _SYSTEMD_UNIT=sendmail.service

config/filter.d/sendmail-reject.conf

@@ -21,12 +21,12 @@ before = common.conf
 
 _daemon = (?:(sm-(mta|acceptingconnections)|sendmail))
 __prefix_line = %(known/__prefix_line)s(?:\w{14,20}: )?
-addr = (?:IPv6:<IP6>|<IP4>)
+addr = (?:(?:IPv6:)?<IP6>|<IP4>)
 
 prefregex = ^<F-MLFID>%(__prefix_line)s</F-MLFID><F-CONTENT>.+</F-CONTENT>$
 
-cmnfailre = ^ruleset=check_rcpt, arg1=(?P<email><\S+@\S+>), relay=(\S+ )?\[%(addr)s\](?: \(may be forged\))?, reject=(550 5\.7\.1 (?P=email)\.\.\. Relaying denied\. (IP name possibly forged \[(\d+\.){3}\d+\]|Proper authentication required\.|IP name lookup failed \[(\d+\.){3}\d+\])|553 5\.1\.8 (?P=email)\.\.\. Domain of sender address \S+ does not exist|550 5\.[71]\.1 (?P=email)\.\.\. (Rejected: .*|User unknown))$
+cmnfailre = ^ruleset=check_rcpt, arg1=(?P<email><\S+@\S+>), relay=(\S+ )?\[%(addr)s\](?: \(may be forged\))?, reject=(?:550 5\.7\.1(?: (?P=email)\.\.\.)?(?: Relaying denied\.)? (?:IP name possibly forged \[(\d+\.){3}\d+\]|Proper authentication required\.|IP name lookup failed \[(\d+\.){3}\d+\]|Fix reverse DNS for \S+)|553 5\.1\.8(?: (?P=email)\.\.\.)? Domain of sender address \S+ does not exist|550 5\.[71]\.1 (?P=email)\.\.\. (Rejected: .*|User unknown))$
-            ^ruleset=check_relay, arg1=(?P<dom>\S+), arg2=%(addr)s, relay=((?P=dom) )?\[(\d+\.){3}\d+\](?: \(may be forged\))?, reject=421 4\.3\.2 (Connection rate limit exceeded\.|Too many open connections\.)$
+            ^ruleset=check_relay(?:, arg\d+=\S*)*, relay=(\S+ )?\[%(addr)s\](?: \(may be forged\))?, reject=421 4\.3\.2 (Connection rate limit exceeded\.|Too many open connections\.)$
             ^rejecting commands from (\S* )?\[%(addr)s\] due to pre-greeting traffic after \d+ seconds$
             ^(?:\S+ )?\[%(addr)s\]: (?:(?i)expn|vrfy) \S+ \[rejected\]$
             ^<[^@]+@[^>]+>\.\.\. No such user here$

config/filter.d/sshd.conf

@@ -68,15 +68,17 @@ cmnfailed = <cmnfailed-<publickey>>
 
 mdre-normal =
 # used to differentiate "connection closed" with and without `[preauth]` (fail/nofail cases in ddos mode)
-mdre-normal-other = ^<F-NOFAIL><F-MLFFORGET>(Connection closed|Disconnected)</F-MLFFORGET></F-NOFAIL> (?:by|from)%(__authng_user)s <HOST>(?:%(__suff)s|\s*)$
+mdre-normal-other = ^<F-NOFAIL><F-MLFFORGET>(Connection (?:closed|reset)|Disconnected)</F-MLFFORGET></F-NOFAIL> (?:by|from)%(__authng_user)s <HOST>(?:%(__suff)s|\s*)$
 
 mdre-ddos = ^Did not receive identification string from <HOST>
-            ^kex_exchange_identification: (?:[Cc]lient sent invalid protocol identifier|[Cc]onnection closed by remote host)
+            ^kex_exchange_identification: (?:read: )?(?:[Cc]lient sent invalid protocol identifier|[Cc]onnection (?:closed by remote host|reset by peer))
             ^Bad protocol version identification '.*' from <HOST>
             ^<F-NOFAIL>SSH: Server;Ltype:</F-NOFAIL> (?:Authname|Version|Kex);Remote: <HOST>-\d+;[A-Z]\w+:
             ^Read from socket failed: Connection <F-MLFFORGET>reset</F-MLFFORGET> by peer
-# same as mdre-normal-other, but as failure (without <F-NOFAIL>) and [preauth] only:
+            ^banner exchange: Connection from <HOST><__on_port_opt>: invalid format
+# same as mdre-normal-other, but as failure (without <F-NOFAIL> with [preauth] and with <F-NOFAIL> on no preauth phase as helper to identify address):
 mdre-ddos-other = ^<F-MLFFORGET>(Connection (?:closed|reset)|Disconnected)</F-MLFFORGET> (?:by|from)%(__authng_user)s <HOST>%(__on_port_opt)s\s+\[preauth\]\s*$
+                  ^<F-NOFAIL><F-MLFFORGET>(Connection (?:closed|reset)|Disconnected)</F-MLFFORGET></F-NOFAIL> (?:by|from)%(__authng_user)s <HOST>(?:%(__on_port_opt)s|\s*)$
 
 mdre-extra = ^Received <F-MLFFORGET>disconnect</F-MLFFORGET> from <HOST>%(__on_port_opt)s:\s*14: No(?: supported)? authentication methods available
             ^Unable to negotiate with <HOST>%(__on_port_opt)s: no matching <__alg_match> found.

config/filter.d/zoneminder.conf

@@ -5,17 +5,23 @@ before = apache-common.conf
 
 [Definition]
 
-# pattern: [Wed Apr 27 23:12:07.736196 2016] [:error] [pid 2460] [client 10.1.1.1:47296] WAR [Login denied for user "test"], referer: https://zoneminderurl/index.php
+# patterns: [Mon Mar 28 16:50:49.522240 2016] [:error] [pid 1795] [client 10.1.1.1:50700] WAR [Login denied for user "username1"], referer: https://zoneminder/
-#
+#           [Sun Mar 28 16:53:00.472693 2021] [php7:notice] [pid 11328] [client 10.1.1.1:39568] ERR [Could not retrieve user test details], referer: https://zm/
+#           [Sun Mar 28 16:59:14.150625 2021] [php7:notice] [pid 11336] [client 10.1.1.1:39654] ERR [Login denied for user "john"], referer: https://zm/
 #
 # Option:  failregex
-# Notes.:  regex to match the password failure messages in the logfile.
+# Notes.:  regex to match the login failure and non-existent user error messages in the logfile.
 
-failregex = ^%(_apache_error_client)s WAR \[Login denied for user "[^"]*"\]
+prefregex = ^%(_apache_error_client)s (?:ERR|WAR) <F-CONTENT>\[(?:Login denied|Could not retrieve).*</F-CONTENT>$
+
+failregex = ^\[Login denied for user "<F-USER>[^"]*</F-USER>"\]
+            ^\[Could not retrieve user <F-USER>\S*</F-USER>
 
 ignoreregex =
 
 # Notes:
-#	Tested on Zoneminder 1.29.0
+#	Tested on Zoneminder 1.29 and 1.35.21
+#
+#	Zoneminder versions > 1.3x use "ERR" and < 1.3x use "WAR" level logs, so i've kept both for compatibility reasons
 #
 # Author: John Marzella

config/jail.conf

@@ -67,7 +67,7 @@ before = paths-debian.conf
 # more aggressive example of formula has the same values only for factor "2.0 / 2.885385" :
 #bantime.formula = ban.Time * math.exp(float(ban.Count+1)*banFactor)/math.exp(1*banFactor)
 
-# "bantime.multipliers" used to calculate next value of ban time instead of formula, coresponding 
+# "bantime.multipliers" used to calculate next value of ban time instead of formula, corresponding
 # previously ban count and given "bantime.factor" (for multipliers default is 1);
 # following example grows ban time by 1, 2, 4, 8, 16 ... and if last ban count greater as multipliers count, 
 # always used last multiplier (64 in example), for factor '1' and original ban time 600 - 10.6 hours
@@ -77,7 +77,7 @@ before = paths-debian.conf
 #bantime.multipliers = 1 5 30 60 300 720 1440 2880
 
 # "bantime.overalljails" (if true) specifies the search of IP in the database will be executed 
-# cross over all jails, if false (dafault), only current jail of the ban IP will be searched
+# cross over all jails, if false (default), only current jail of the ban IP will be searched
 #bantime.overalljails = false
 
 # --------------------
@@ -227,6 +227,15 @@ action_mwl = %(action_)s
 action_xarf = %(action_)s
              xarf-login-attack[service=%(__name__)s, sender="%(sender)s", logpath="%(logpath)s", port="%(port)s"]
 
+# ban & send a notification to one or more of the 50+ services supported by Apprise.
+# See https://github.com/caronc/apprise/wiki for details on what is supported.
+#
+# You may optionally over-ride the default configuration line (containing the Apprise URLs)
+# by using 'apprise[config="/alternate/path/to/apprise.cfg"]' otherwise
+# /etc/fail2ban/apprise.conf is sourced for your supported notification configuration.
+# action = %(action_)s
+#          apprise
+
 # ban IP on CloudFlare & send an e-mail with whois report and relevant log lines
 # to the destemail.
 action_cf_mwl = cloudflare[cfuser="%(cfemail)s", cftoken="%(cfapikey)s"]
@@ -242,20 +251,6 @@ action_cf_mwl = cloudflare[cfuser="%(cfemail)s", cftoken="%(cfapikey)s"]
 #
 action_blocklist_de  = blocklist_de[email="%(sender)s", service="%(__name__)s", apikey="%(blocklist_de_apikey)s", agent="%(fail2ban_agent)s"]
 
-# Report ban via badips.com, and use as blacklist
-#
-# See BadIPsAction docstring in config/action.d/badips.py for
-# documentation for this action.
-#
-# NOTE: This action relies on banaction being present on start and therefore
-# should be last action defined for a jail.
-#
-action_badips = badips.py[category="%(__name__)s", banaction="%(banaction)s", agent="%(fail2ban_agent)s"]
-#
-# Report ban via badips.com (uses action.d/badips.conf for reporting only)
-#
-action_badips_report = badips[category="%(__name__)s", agent="%(fail2ban_agent)s"]
-
 # Report ban via abuseipdb.com.
 #
 # See action.d/abuseipdb.conf for usage example and details.
@@ -351,7 +346,7 @@ maxretry = 2
 port     = http,https
 logpath  = %(apache_access_log)s
 maxretry = 1
-ignorecommand = %(ignorecommands_dir)s/apache-fakegooglebot <ip>
+ignorecommand = %(fail2ban_confpath)s/filter.d/ignorecommands/apache-fakegooglebot <ip>
 
 
 [apache-modsecurity]
@@ -375,8 +370,11 @@ banaction = %(banaction_allports)s
 logpath = /opt/openhab/logs/request.log
 
 
+# To use more aggressive http-auth modes set filter parameter "mode" in jail.local:
+# normal (default), aggressive (combines all), auth or fallback
+# See "tests/files/logs/nginx-http-auth" or "filter.d/nginx-http-auth.conf" for usage example and details.
 [nginx-http-auth]
-
+# mode = normal
 port    = http,https
 logpath = %(nginx_error_log)s
 
@@ -392,8 +390,10 @@ logpath = %(nginx_error_log)s
 
 port     = http,https
 logpath  = %(nginx_error_log)s
-maxretry = 2
 
+[nginx-bad-request]
+port    = http,https
+logpath = %(nginx_access_log)s
 
 # Ban attackers that try to use PHP's URL-fopen() functionality
 # through GET/POST variables. - Experimental, with more than a year
@@ -797,6 +797,14 @@ logpath  = %(mysql_log)s
 backend  = %(mysql_backend)s
 
 
+[mssql-auth]
+# Default configuration for Microsoft SQL Server for Linux
+# See the 'mssql-conf' manpage how to change logpath or port
+logpath = /var/opt/mssql/log/errorlog
+port = 1433
+filter = mssql-auth
+
+
 # Log wrong MongoDB auth (for details see filter 'filter.d/mongodb-auth.conf')
 [mongodb-auth]
 # change port when running with "--shardsvr" or "--configsvr" runtime operation
@@ -962,3 +970,11 @@ logpath = %(apache_error_log)s
 # see `filter.d/traefik-auth.conf` for details and service example.
 port    = http,https
 logpath = /var/log/traefik/access.log
+
+[scanlogd]
+logpath = %(syslog_local0)s
+banaction = %(banaction_allports)s
+
+[monitorix]
+port	= 8080
+logpath = /var/log/monitorix-httpd

config/paths-common.conf

@@ -91,6 +91,3 @@ mysql_log = %(syslog_daemon)s
 mysql_backend = %(default_backend)s
 
 roundcube_errors_log = /var/log/roundcube/errors
-
-# Directory with ignorecommand scripts
-ignorecommands_dir = /etc/fail2ban/filter.d/ignorecommands

config/paths-debian.conf

@@ -26,3 +26,5 @@ exim_main_log = /var/log/exim4/mainlog
 # was in debian squeezy but not in wheezy
 # /etc/proftpd/proftpd.conf (SystemLog)
 proftpd_log = /var/log/proftpd/proftpd.log
+
+roundcube_errors_log = /var/log/roundcube/errors.log

fail2ban/client/fail2banclient.py

@@ -175,9 +175,13 @@ class Fail2banClient(Fail2banCmdLine, Thread):
 
 		return [["server-stream", stream], ['server-status']]
 
+	def _set_server(self, s):
+		self._server = s
+
 	##
 	def __startServer(self, background=True):
 		from .fail2banserver import Fail2banServer
+		# read configuration here (in client only, in server we do that in the config-thread):
 		stream = self.__prepareStartServer()
 		self._alive = True
 		if not stream:
@@ -192,16 +196,19 @@ class Fail2banClient(Fail2banCmdLine, Thread):
 					return False
 			else:
 				# In foreground mode we should make server/client communication in different threads:
-				th = Thread(target=Fail2banClient.__processStartStreamAfterWait, args=(self, stream, False))
+				phase = dict()
-				th.daemon = True
+				self.configureServer(phase=phase, stream=stream)
-				th.start()
 				# Mark current (main) thread as daemon:
-				self.setDaemon(True)
+				self.daemon = True
 				# Start server direct here in main thread (not fork):
-				self._server = Fail2banServer.startServerDirect(self._conf, False)
+				self._server = Fail2banServer.startServerDirect(self._conf, False, self._set_server)
-
+				if not phase.get('done', False):
+					if self._server: # pragma: no cover
+						self._server.quit()
+						self._server = None
+					exit(255)
 		except ExitException: # pragma: no cover
-			pass
+			raise
 		except Exception as e: # pragma: no cover
 			output("")
 			logSys.error("Exception while starting server " + ("background" if background else "foreground"))
@@ -214,23 +221,39 @@ class Fail2banClient(Fail2banCmdLine, Thread):
 		return True
 
 	##
-	def configureServer(self, nonsync=True, phase=None):
+	def configureServer(self, nonsync=True, phase=None, stream=None):
 		# if asynchronous start this operation in the new thread:
 		if nonsync:
-			th = Thread(target=Fail2banClient.configureServer, args=(self, False, phase))
+			if phase is not None:
+				# event for server ready flag:
+				def _server_ready():
+					phase['start-ready'] = True
+					logSys.log(5, '  server phase %s', phase)
+				# notify waiting thread if server really ready
+				self._conf['onstart'] = _server_ready
+			th = Thread(target=Fail2banClient.configureServer, args=(self, False, phase, stream))
 			th.daemon = True
-			return th.start()
+			th.start()
+			# if we need to read configuration stream:
+			if stream is None and phase is not None:
+				# wait, do not continue if configuration is not 100% valid:
+				Utils.wait_for(lambda: phase.get('ready', None) is not None, self._conf["timeout"], 0.001)
+				logSys.log(5, '  server phase %s', phase)
+				if not phase.get('start', False):
+					raise ServerExecutionException('Async configuration of server failed')
+			return True
 		# prepare: read config, check configuration is valid, etc.:
 		if phase is not None:
 			phase['start'] = True
 			logSys.log(5, '  client phase %s', phase)
+		if stream is None:
 			stream = self.__prepareStartServer()
 		if phase is not None:
 			phase['ready'] = phase['start'] = (True if stream else False)
 			logSys.log(5, '  client phase %s', phase)
 		if not stream:
 			return False
-		# wait a litle bit for phase "start-ready" before enter active waiting:
+		# wait a little bit for phase "start-ready" before enter active waiting:
 		if phase is not None:
 			Utils.wait_for(lambda: phase.get('start-ready', None) is not None, 0.5, 0.001)
 			phase['configure'] = (True if stream else False)
@@ -321,13 +344,14 @@ class Fail2banClient(Fail2banCmdLine, Thread):
 
 
 	def __processStartStreamAfterWait(self, *args):
+		ret = False
 		try:
 			# Wait for the server to start
 			if not self.__waitOnServer(): # pragma: no cover
 				logSys.error("Could not find server, waiting failed")
 				return False
 				# Configure the server
-			self.__processCmd(*args)
+			ret = self.__processCmd(*args)
 		except ServerExecutionException as e: # pragma: no cover
 			if self._conf["verbose"] > 1:
 				logSys.exception(e)
@@ -336,10 +360,11 @@ class Fail2banClient(Fail2banCmdLine, Thread):
 						 "remove " + self._conf["socket"] + ". If "
 						 "you used fail2ban-client to start the "
 						 "server, adding the -x option will do it")
-			if self._server:
+
+		if not ret and self._server: # stop on error (foreground, config read in another thread):
 			self._server.quit()
-			return False
+			self._server = None
-		return True
+		return ret
 
 	def __waitOnServer(self, alive=True, maxtime=None):
 		if maxtime is None:

fail2ban/client/fail2bancmdline.py

@@ -192,7 +192,7 @@ class Fail2banCmdLine():
 				cmdOpts = 'hc:s:p:xfbdtviqV'
 				cmdLongOpts = ['loglevel=', 'logtarget=', 'syslogsocket=', 'test', 'async',
 					'conf=', 'pidfile=', 'pname=', 'socket=',
-					'timeout=', 'str2sec=', 'help', 'version', 'dp', '--dump-pretty']
+					'timeout=', 'str2sec=', 'help', 'version', 'dp', 'dump-pretty']
 				optList, self._args = getopt.getopt(self._argv[1:], cmdOpts, cmdLongOpts)
 			except getopt.GetoptError:
 				self.dispUsage()

fail2ban/client/fail2banreader.py

@@ -53,6 +53,7 @@ class Fail2banReader(ConfigReader):
 		opts = [["string", "loglevel", "INFO" ],
 				["string", "logtarget", "STDERR"],
 				["string", "syslogsocket", "auto"],
+				["string", "allowipv6", "auto"],
 				["string", "dbfile", "/var/lib/fail2ban/fail2ban.sqlite3"],
 				["int",    "dbmaxmatches", None],
 				["string", "dbpurgeage", "1d"]]
@@ -74,6 +75,7 @@ class Fail2banReader(ConfigReader):
 		# Also dbfile should be set before all other database options.
 		# So adding order indices into items, to be stripped after sorting, upon return
 		order = {"thread":0, "syslogsocket":11, "loglevel":12, "logtarget":13,
+			"allowipv6": 14,
 			"dbfile":50, "dbmaxmatches":51, "dbpurgeage":51}
 		stream = list()
 		for opt in self.__opts:

fail2ban/client/fail2banregex.py

@@ -35,11 +35,11 @@ __license__ = "GPL"
 
 import getopt
 import logging
+import re
 import os
 import shlex
 import sys
 import time
-import time
 import urllib
 from optparse import OptionParser, Option
 
@@ -52,7 +52,7 @@ except ImportError:
 
 from ..version import version, normVersion
 from .filterreader import FilterReader
-from ..server.filter import Filter, FileContainer
+from ..server.filter import Filter, FileContainer, MyTime
 from ..server.failregex import Regex, RegexException
 
 from ..helpers import str2LogLevel, getVerbosityFormat, FormatterWithTraceBack, getLogger, \
@@ -269,15 +269,19 @@ class Fail2banRegex(object):
 			self.setJournalMatch(shlex.split(opts.journalmatch))
 		if opts.timezone:
 			self._filter.setLogTimeZone(opts.timezone)
+		self._filter.checkFindTime = False
+		if True: # not opts.out:
+			MyTime.setAlternateNow(0); # accept every date (years from 19xx up to end of current century, '%ExY' and 'Exy' patterns)
+			from ..server.strptime import _updateTimeRE
+			_updateTimeRE()
 		if opts.datepattern:
 			self.setDatePattern(opts.datepattern)
 		if opts.usedns:
 			self._filter.setUseDns(opts.usedns)
 		self._filter.returnRawHost = opts.raw
-		self._filter.checkFindTime = False
 		self._filter.checkAllRegex = opts.checkAllRegex and not opts.out
 		# ignore pending (without ID/IP), added to matches if it hits later (if ID/IP can be retreved)
-		self._filter.ignorePending = opts.out
+		self._filter.ignorePending = bool(opts.out)
 		# callback to increment ignored RE's by index (during process):
 		self._filter.onIgnoreRegex = self._onIgnoreRegex
 		self._backend = 'auto'
@@ -285,9 +289,6 @@ class Fail2banRegex(object):
 	def output(self, line):
 		if not self._opts.out: output(line)
 
-	def decode_line(self, line):
-		return FileContainer.decode_line('<LOG>', self._encoding, line)
-
 	def encode_line(self, line):
 		return line.encode(self._encoding, 'ignore')
 
@@ -326,11 +327,13 @@ class Fail2banRegex(object):
 		regex = regextype + 'regex'
 		# try to check - we've case filter?[options...]?:
 		basedir = self._opts.config
+		fltName = value
 		fltFile = None
 		fltOpt = {}
 		if regextype == 'fail':
+			if re.search(r'(?ms)^/{0,3}[\w/_\-.]+(?:\[.*\])?$', value):
+				try:
 					fltName, fltOpt = extractOptions(value)
-			if fltName is not None:
 					if "." in fltName[~5:]:
 						tryNames = (fltName,)
 					else:
@@ -346,6 +349,11 @@ class Fail2banRegex(object):
 						if os.path.isfile(fltFile):
 							break
 						fltFile = None
+				except Exception as e:
+					output("ERROR: Wrong filter name or options: %s" % (str(e),))
+					output("       while parsing: %s" % (value,))
+					if self._verbose: raise(e)
+					return False
 		# if it is filter file:
 		if fltFile is not None:
 			if (basedir == self._opts.config
@@ -512,10 +520,14 @@ class Fail2banRegex(object):
 	def _prepaireOutput(self):
 		"""Prepares output- and fetch-function corresponding given '--out' option (format)"""
 		ofmt = self._opts.out
-		if ofmt in ('id', 'ip'):
+		if ofmt in ('id', 'fid'):
 			def _out(ret):
 				for r in ret:
 					output(r[1])
+		elif ofmt == 'ip':
+			def _out(ret):
+				for r in ret:
+					output(r[3].get('ip', r[1]))
 		elif ofmt == 'msg':
 			def _out(ret):
 				for r in ret:
@@ -712,10 +724,6 @@ class Fail2banRegex(object):
 
 		return True
 
-	def file_lines_gen(self, hdlr):
-		for line in hdlr:
-			yield self.decode_line(line)
-
 	def start(self, args):
 
 		cmd_log, cmd_regex = args[:2]
@@ -734,10 +742,10 @@ class Fail2banRegex(object):
 
 		if os.path.isfile(cmd_log):
 			try:
-				hdlr = open(cmd_log, 'rb')
+				test_lines = FileContainer(cmd_log, self._encoding, doOpen=True)
+
 				self.output( "Use         log file : %s" % cmd_log )
 				self.output( "Use         encoding : %s" % self._encoding )
-				test_lines = self.file_lines_gen(hdlr)
 			except IOError as e: # pragma: no cover
 				output( e )
 				return False

fail2ban/client/fail2banserver.py

@@ -44,7 +44,7 @@ class Fail2banServer(Fail2banCmdLine):
 	# Start the Fail2ban server in background/foreground (daemon mode or not).
 
 	@staticmethod
-	def startServerDirect(conf, daemon=True):
+	def startServerDirect(conf, daemon=True, setServer=None):
 		logSys.debug("  direct starting of server in %s, deamon: %s", os.getpid(), daemon)
 		from ..server.server import Server
 		server = None
@@ -52,6 +52,10 @@ class Fail2banServer(Fail2banCmdLine):
 			# Start it in foreground (current thread, not new process),
 			# server object will internally fork self if daemon is True
 			server = Server(daemon)
+			# notify caller - set server handle:
+			if setServer:
+				setServer(server)
+			# run:
 			server.start(conf["socket"],
 							conf["pidfile"], conf["force"],
 							conf=conf)
@@ -63,6 +67,10 @@ class Fail2banServer(Fail2banCmdLine):
 				if conf["verbose"] > 1:
 					logSys.exception(e2)
 			raise
+		finally:
+			# notify waiting thread server ready resp. done (background execution, error case, etc):
+			if conf.get('onstart'):
+				conf['onstart']()
 
 		return server
 
@@ -179,27 +187,15 @@ class Fail2banServer(Fail2banCmdLine):
 				# Start new thread with client to read configuration and
 				# transfer it to the server:
 				cli = self._Fail2banClient()
+				cli._conf = self._conf
 				phase = dict()
 				logSys.debug('Configure via async client thread')
 				cli.configureServer(phase=phase)
-				# wait, do not continue if configuration is not 100% valid:
-				Utils.wait_for(lambda: phase.get('ready', None) is not None, self._conf["timeout"], 0.001)
-				logSys.log(5, '  server phase %s', phase)
-				if not phase.get('start', False):
-					raise ServerExecutionException('Async configuration of server failed')
-				# event for server ready flag:
-				def _server_ready():
-					phase['start-ready'] = True
-					logSys.log(5, '  server phase %s', phase)
-				# notify waiting thread if server really ready
-				self._conf['onstart'] = _server_ready
 
 			# Start server, daemonize it, etc.
 			pid = os.getpid()
-			server = Fail2banServer.startServerDirect(self._conf, background)
+			server = Fail2banServer.startServerDirect(self._conf, background,
-			# notify waiting thread server ready resp. done (background execution, error case, etc):
+				 cli._set_server if cli else None)
-			if not nonsync:
-				_server_ready()
 			# If forked - just exit other processes
 			if pid != os.getpid(): # pragma: no cover
 				os._exit(0)

fail2ban/client/filterreader.py

@@ -72,8 +72,9 @@ class FilterReader(DefinitionInitConfigReader):
 	def _fillStream(stream, opts, jailName):
 		prio0idx = 0
 		for opt, value in opts.iteritems():
-			if opt in ("failregex", "ignoreregex"):
+			# Do not send a command if the value is not set (empty).
 			if value is None: continue
+			if opt in ("failregex", "ignoreregex"):
 				multi = []
 				for regex in value.split('\n'):
 					# Do not send a command if the rule is empty.
@@ -91,8 +92,6 @@ class FilterReader(DefinitionInitConfigReader):
 			elif opt in ('datepattern'):
 				stream.append(["set", jailName, opt, value])
 			elif opt == 'journalmatch':
-				# Do not send a command if the match is empty.
-				if value is None: continue
 				for match in value.split("\n"):
 					if match == '': continue
 					stream.append(

fail2ban/client/jailreader.py

@@ -121,9 +121,12 @@ class JailReader(ConfigReader):
 
 	def getOptions(self):
 
+		basedir = self.getBaseDir()
+
 		# Before interpolation (substitution) add static options always available as default:
 		self.merge_defaults({
-			"fail2ban_version": version
+			"fail2ban_version": version,
+			"fail2ban_confpath": basedir
 		})
 
 		try:
@@ -140,12 +143,13 @@ class JailReader(ConfigReader):
 			# Read filter
 			flt = self.__opts["filter"]
 			if flt:
+				try:
 					filterName, filterOpt = extractOptions(flt)
-				if not filterName:
+				except ValueError as e:
-					raise JailDefError("Invalid filter definition %r" % flt)
+					raise JailDefError("Invalid filter definition %r: %s" % (flt, e))
 				self.__filter = FilterReader(
 					filterName, self.__name, filterOpt, 
-					share_config=self.share_config, basedir=self.getBaseDir())
+					share_config=self.share_config, basedir=basedir)
 				ret = self.__filter.read()
 				if not ret:
 					raise JailDefError("Unable to read the filter %r" % filterName)
@@ -174,10 +178,10 @@ class JailReader(ConfigReader):
 					if not act:			  # skip empty actions
 						continue
 					# join with previous line if needed (consider possible new-line):
+					try:
 						actName, actOpt = extractOptions(act)
-					prevln = ''
+					except ValueError as e:
-					if not actName:
+						raise JailDefError("Invalid action definition %r: %s" % (act, e))
-						raise JailDefError("Invalid action definition %r" % act)
 					if actName.endswith(".py"):
 						self.__actions.append([
 							"set",
@@ -185,13 +189,13 @@ class JailReader(ConfigReader):
 							"addaction",
 							actOpt.pop("actname", os.path.splitext(actName)[0]),
 							os.path.join(
-								self.getBaseDir(), "action.d", actName),
+								basedir, "action.d", actName),
 							json.dumps(actOpt),
 							])
 					else:
 						action = ActionReader(
 							actName, self.__name, actOpt,
-							share_config=self.share_config, basedir=self.getBaseDir())
+							share_config=self.share_config, basedir=basedir)
 						ret = action.read()
 						if ret:
 							action.getOptions(self.__opts)

fail2ban/helpers.py

@@ -371,7 +371,7 @@ OPTION_CRE = re.compile(r"^([^\[]+)(?:\[(.*)\])?\s*$", re.DOTALL)
 # since v0.10 separator extended with `]\s*[` for support of multiple option groups, syntax 
 # `action = act[p1=...][p2=...]`
 OPTION_EXTRACT_CRE = re.compile(
-	r'([\w\-_\.]+)=(?:"([^"]*)"|\'([^\']*)\'|([^,\]]*))(?:,|\]\s*\[|$)', re.DOTALL)
+	r'\s*([\w\-_\.]+)=(?:"([^"]*)"|\'([^\']*)\'|([^,\]]*))(?:,|\]\s*\[|$|(?P<wrngA>.+))|,?\s*$|(?P<wrngB>.+)', re.DOTALL)
 # split by new-line considering possible new-lines within options [...]:
 OPTION_SPLIT_CRE = re.compile(
 	r'(?:[^\[\s]+(?:\s*\[\s*(?:[\w\-_\.]+=(?:"[^"]*"|\'[^\']*\'|[^,\]]*)\s*(?:,|\]\s*\[)?\s*)*\])?\s*|\S+)(?=\n\s*|\s+|$)', re.DOTALL)
@@ -379,13 +379,19 @@ OPTION_SPLIT_CRE = re.compile(
 def extractOptions(option):
 	match = OPTION_CRE.match(option)
 	if not match:
-		# TODO proper error handling
+		raise ValueError("unexpected option syntax")
-		return None, None
 	option_name, optstr = match.groups()
 	option_opts = dict()
 	if optstr:
 		for optmatch in OPTION_EXTRACT_CRE.finditer(optstr):
+			if optmatch.group("wrngA"):
+				raise ValueError("unexpected syntax at %d after option %r: %s" % (
+					optmatch.start("wrngA"), optmatch.group(1), optmatch.group("wrngA")[0:25]))
+			if optmatch.group("wrngB"):
+				raise ValueError("expected option, wrong syntax at %d: %s" % (
+					optmatch.start("wrngB"), optmatch.group("wrngB")[0:25]))
 			opt = optmatch.group(1)
+			if not opt: continue
 			value = [
 				val for val in optmatch.group(2,3,4) if val is not None][0]
 			option_opts[opt.strip()] = value.strip()

fail2ban/protocol.py

@@ -66,7 +66,7 @@ protocol = [
 ["set loglevel <LEVEL>", "sets logging level to <LEVEL>. Levels: CRITICAL, ERROR, WARNING, NOTICE, INFO, "
 	"DEBUG, TRACEDEBUG, HEAVYDEBUG or corresponding numeric value (50-5)"], 
 ["get loglevel", "gets the logging level"], 
-["set logtarget <TARGET>", "sets logging target to <TARGET>. Can be STDOUT, STDERR, SYSLOG or a file"], 
+["set logtarget <TARGET>", "sets logging target to <TARGET>. Can be STDOUT, STDERR, SYSLOG, SYSTEMD-JOURNAL or a file"], 
 ["get logtarget", "gets logging target"], 
 ["set syslogsocket auto|<SOCKET>", "sets the syslog socket path to auto or <SOCKET>. Only used if logtarget is SYSLOG"],
 ["get syslogsocket", "gets syslog socket path"],
@@ -134,7 +134,7 @@ protocol = [
 ["get <JAIL> ignoreregex", "gets the list of regular expressions which matches patterns to ignore for <JAIL>"],
 ["get <JAIL> findtime", "gets the time for which the filter will look back for failures for <JAIL>"],
 ["get <JAIL> bantime", "gets the time a host is banned for <JAIL>"],
-["get <JAIL> datepattern", "gets the patern used to match date/times for <JAIL>"],
+["get <JAIL> datepattern", "gets the pattern used to match date/times for <JAIL>"],
 ["get <JAIL> usedns", "gets the usedns setting for <JAIL>"],
 ["get <JAIL> banip [<SEP>|--with-time]", "gets the list of of banned IP addresses for <JAIL>. Optionally the separator character ('<SEP>', default is space) or the option '--with-time' (printing the times of ban) may be specified. The IPs are ordered by end of ban."],
 ["get <JAIL> maxretry", "gets the number of failures allowed for <JAIL>"],

fail2ban/server/action.py

@@ -30,6 +30,9 @@ import tempfile
 import threading
 import time
 from abc import ABCMeta
+try:
+	from collections.abc import MutableMapping
+except ImportError:
 	from collections import MutableMapping
 
 from .failregex import mapTag2Opt
@@ -407,7 +410,7 @@ class CommandAction(ActionBase):
 		cmd = self.replaceTag(tag, self._properties,
 			conditional=('family='+family if family else ''),
 			cache=self.__substCache)
-		if '<' not in cmd or not family: return cmd
+		if not family or '<' not in cmd: return cmd
 		# replace family as dynamic tags, important - don't cache, no recursion and auto-escape here:
 		cmd = self.replaceDynamicTags(cmd, {'family':family})
 		return cmd
@@ -974,8 +977,11 @@ class CommandAction(ActionBase):
 		except (KeyError, TypeError):
 			family = ''
 
-		# invariant check:
+		repcnt = 0
-		if self.actioncheck:
+		while True:
+
+			# got some error, do invariant check:
+			if repcnt and self.actioncheck:
 				# don't repair/restore if unban (no matter):
 				def _beforeRepair():
 					if cmd == '<actionunban>' and not self._properties.get('actionrepair_on_unban'):
@@ -998,7 +1004,11 @@ class CommandAction(ActionBase):
 			else:
 				realCmd = cmd
 
-		return self.executeCmd(realCmd, self.timeout)
+			# try execute command:
+			ret = self.executeCmd(realCmd, self.timeout)
+			repcnt += 1
+			if ret or repcnt > 1:
+				return ret
 
 	@staticmethod
 	def executeCmd(realCmd, timeout=60, **kwargs):

fail2ban/server/actions.py

@@ -28,11 +28,11 @@ import logging
 import os
 import sys
 import time
-from collections import Mapping
 try:
-	from collections import OrderedDict
+	from collections.abc import Mapping
 except ImportError:
-	OrderedDict = dict
+	from collections import Mapping
+from collections import OrderedDict
 
 from .banmanager import BanManager, BanTicket
 from .ipdns import IPAddr
@@ -81,7 +81,7 @@ class Actions(JailThread, Mapping):
 		self._jail = jail
 		self._actions = OrderedDict()
 		## The ban manager.
-		self.__banManager = BanManager()
+		self.banManager = BanManager()
 		self.banEpoch = 0
 		self.__lastConsistencyCheckTM = 0
 		## Precedence of ban (over unban), so max number of tickets banned (to call an unban check):
@@ -200,7 +200,7 @@ class Actions(JailThread, Mapping):
 	
 	def setBanTime(self, value):
 		value = MyTime.str2seconds(value)
-		self.__banManager.setBanTime(value)
+		self.banManager.setBanTime(value)
 		logSys.info("  banTime: %s" % value)
 	
 	##
@@ -209,10 +209,10 @@ class Actions(JailThread, Mapping):
 	# @return the time
 	
 	def getBanTime(self):
-		return self.__banManager.getBanTime()
+		return self.banManager.getBanTime()
 
 	def getBanned(self, ids):
-		lst = self.__banManager.getBanList()
+		lst = self.banManager.getBanList()
 		if not ids:
 			return lst
 		if len(ids) == 1:
@@ -227,7 +227,7 @@ class Actions(JailThread, Mapping):
 		list
 			The list of banned IP addresses.
 		"""
-		return self.__banManager.getBanList(ordered=True, withTime=withTime)
+		return self.banManager.getBanList(ordered=True, withTime=withTime)
 
 	def addBannedIP(self, ip):
 		"""Ban an IP or list of IPs."""
@@ -279,7 +279,7 @@ class Actions(JailThread, Mapping):
 		if db and self._jail.database is not None:
 			self._jail.database.delBan(self._jail, ip)
 		# Find the ticket with the IP.
-		ticket = self.__banManager.getTicketByID(ip)
+		ticket = self.banManager.getTicketByID(ip)
 		if ticket is not None:
 			# Unban the IP.
 			self.__unBan(ticket)
@@ -288,7 +288,7 @@ class Actions(JailThread, Mapping):
 			if not isinstance(ip, IPAddr):
 				ipa = IPAddr(ip)
 				if not ipa.isSingle: # subnet (mask/cidr) or raw (may be dns/hostname):
-					ips = filter(ipa.contains, self.__banManager.getBanList())
+					ips = filter(ipa.contains, self.banManager.getBanList())
 					if ips:
 						return self.removeBannedIP(ips, db, ifexists)
 			# not found:
@@ -305,9 +305,7 @@ class Actions(JailThread, Mapping):
 		"""
 		if actions is None:
 			actions = self._actions
-		revactions = actions.items()
+		for name, action in reversed(actions.items()):
-		revactions.reverse()
-		for name, action in revactions:
 			try:
 				action.stop()
 			except Exception as e:
@@ -347,7 +345,7 @@ class Actions(JailThread, Mapping):
 					continue
 				# wait for ban (stop if gets inactive, pending ban or unban):
 				bancnt = 0
-				wt = min(self.sleeptime, self.__banManager._nextUnbanTime - MyTime.time())
+				wt = min(self.sleeptime, self.banManager._nextUnbanTime - MyTime.time())
 				logSys.log(5, "Actions: wait for pending tickets %s (default %s)", wt, self.sleeptime)
 				if Utils.wait_for(lambda: not self.active or self._jail.hasFailTickets, wt):
 					bancnt = self.__checkBan()
@@ -394,7 +392,12 @@ class Actions(JailThread, Mapping):
 			"ipfailures":			lambda self: self._mi4ip(True).getAttempt(),
 			"ipjailfailures":	lambda self: self._mi4ip().getAttempt(),
 			# raw ticket info:
-			"raw-ticket":			lambda self: repr(self.__ticket)
+			"raw-ticket":			lambda self: repr(self.__ticket),
+			# jail info:
+			"jail.banned":       lambda self: self.__jail.actions.banManager.size(),
+			"jail.banned_total": lambda self: self.__jail.actions.banManager.getBanTotal(),
+			"jail.found":        lambda self: self.__jail.filter.failManager.size(),
+			"jail.found_total":  lambda self: self.__jail.filter.failManager.getFailTotal()
 		}
 
 		__slots__ = CallingMap.__slots__ + ('__ticket', '__jail', '__mi4ip')
@@ -491,11 +494,11 @@ class Actions(JailThread, Mapping):
 		for ticket in tickets:
 
 			bTicket = BanTicket.wrap(ticket)
-			btime = ticket.getBanTime(self.__banManager.getBanTime())
+			btime = ticket.getBanTime(self.banManager.getBanTime())
-			ip = bTicket.getIP()
+			ip = bTicket.getID()
 			aInfo = self._getActionInfo(bTicket)
 			reason = {}
-			if self.__banManager.addBanTicket(bTicket, reason=reason):
+			if self.banManager.addBanTicket(bTicket, reason=reason):
 				cnt += 1
 				# report ticket to observer, to check time should be increased and hereafter observer writes ban to database (asynchronous)
 				if Observers.Main is not None and not bTicket.restored:
@@ -539,9 +542,10 @@ class Actions(JailThread, Mapping):
 					if bTicket.banEpoch == self.banEpoch and diftm > 3:
 						# avoid too often checks:
 						if not rebanacts and MyTime.time() > self.__lastConsistencyCheckTM + 3:
-							for action in self._actions.itervalues():
-								action.consistencyCheck()
 							self.__lastConsistencyCheckTM = MyTime.time()
+							for action in self._actions.itervalues():
+								if hasattr(action, 'consistencyCheck'):
+									action.consistencyCheck()
 					# check epoch in order to reban it:
 					if bTicket.banEpoch < self.banEpoch:
 						if not rebanacts: rebanacts = dict(
@@ -554,7 +558,7 @@ class Actions(JailThread, Mapping):
 			# and increase ticket time if "bantime.increment" set)
 		if cnt:
 			logSys.debug("Banned %s / %s, %s ticket(s) in %r", cnt, 
-				self.__banManager.getBanTotal(), self.__banManager.size(), self._jail.name)
+				self.banManager.getBanTotal(), self.banManager.size(), self._jail.name)
 		return cnt
 
 	def __reBan(self, ticket, actions=None, log=True):
@@ -569,10 +573,10 @@ class Actions(JailThread, Mapping):
 			Ticket to reban
 		"""
 		actions = actions or self._actions
-		ip = ticket.getIP()
+		ip = ticket.getID()
 		aInfo = self._getActionInfo(ticket)
 		if log:
-			logSys.notice("[%s] Reban %s%s", self._jail.name, aInfo["ip"], (', action %r' % actions.keys()[0] if len(actions) == 1 else ''))
+			logSys.notice("[%s] Reban %s%s", self._jail.name, ip, (', action %r' % actions.keys()[0] if len(actions) == 1 else ''))
 		for name, action in actions.iteritems():
 			try:
 				logSys.debug("[%s] action %r: reban %s", self._jail.name, name, ip)
@@ -594,7 +598,7 @@ class Actions(JailThread, Mapping):
 	def _prolongBan(self, ticket):
 		# prevent to prolong ticket that was removed in-between,
 		# if it in ban list - ban time already prolonged (and it stays there):
-		if not self.__banManager._inBanList(ticket): return
+		if not self.banManager._inBanList(ticket): return
 		# do actions :
 		aInfo = None
 		for name, action in self._actions.iteritems():
@@ -619,13 +623,13 @@ class Actions(JailThread, Mapping):
 
 		Unban IP addresses which are outdated.
 		"""
-		lst = self.__banManager.unBanList(MyTime.time(), maxCount)
+		lst = self.banManager.unBanList(MyTime.time(), maxCount)
 		for ticket in lst:
 			self.__unBan(ticket)
 		cnt = len(lst)
 		if cnt:
 			logSys.debug("Unbanned %s, %s ticket(s) in %r", 
-				cnt, self.__banManager.size(), self._jail.name)
+				cnt, self.banManager.size(), self._jail.name)
 		return cnt
 
 	def __flushBan(self, db=False, actions=None, stop=False):
@@ -639,10 +643,10 @@ class Actions(JailThread, Mapping):
 		log = True
 		if actions is None:
 			logSys.debug("  Flush ban list")
-			lst = self.__banManager.flushBanList()
+			lst = self.banManager.flushBanList()
 		else:
 			log = False # don't log "[jail] Unban ..." if removing actions only.
-			lst = iter(self.__banManager)
+			lst = iter(self.banManager)
 		cnt = 0
 		# first we'll execute flush for actions supporting this operation:
 		unbactions = {}
@@ -660,7 +664,7 @@ class Actions(JailThread, Mapping):
 				if hasattr(action, 'consistencyCheck'):
 					def _beforeRepair():
 						if stop and not getattr(action, 'actionrepair_on_unban', None): # don't need repair on stop
-							self._logSys.error("Invariant check failed. Flush is impossible.")
+							logSys.error("Invariant check failed. Flush is impossible.")
 							return False
 						return True
 					action.consistencyCheck(_beforeRepair)
@@ -679,7 +683,7 @@ class Actions(JailThread, Mapping):
 			self.__unBan(ticket, actions=actions, log=log)
 			cnt += 1
 		logSys.debug("  Unbanned %s, %s ticket(s) in %r", 
-			cnt, self.__banManager.size(), self._jail.name)
+			cnt, self.banManager.size(), self._jail.name)
 		return cnt
 
 	def __unBan(self, ticket, actions=None, log=True):
@@ -697,10 +701,10 @@ class Actions(JailThread, Mapping):
 			unbactions = self._actions
 		else:
 			unbactions = actions
-		ip = ticket.getIP()
+		ip = ticket.getID()
 		aInfo = self._getActionInfo(ticket)
 		if log:
-			logSys.notice("[%s] Unban %s", self._jail.name, aInfo["ip"])
+			logSys.notice("[%s] Unban %s", self._jail.name, ip)
 		for name, action in unbactions.iteritems():
 			try:
 				logSys.debug("[%s] action %r: unban %s", self._jail.name, name, ip)
@@ -722,18 +726,18 @@ class Actions(JailThread, Mapping):
 			logSys.warning("Unsupported extended jail status flavor %r. Supported: %s" % (flavor, supported_flavors))
 		# Always print this information (basic)
 		if flavor != "short":
-			banned = self.__banManager.getBanList()
+			banned = self.banManager.getBanList()
 			cnt = len(banned)
 		else:
-			cnt = self.__banManager.size()
+			cnt = self.banManager.size()
 		ret = [("Currently banned", cnt),
-			   ("Total banned", self.__banManager.getBanTotal())]
+			   ("Total banned", self.banManager.getBanTotal())]
 		if flavor != "short":
 			ret += [("Banned IP list", banned)]
 		if flavor == "cymru":
-			cymru_info = self.__banManager.getBanListExtendedCymruInfo()
+			cymru_info = self.banManager.getBanListExtendedCymruInfo()
 			ret += \
-				[("Banned ASN list", self.__banManager.geBanListExtendedASN(cymru_info)),
+				[("Banned ASN list", self.banManager.geBanListExtendedASN(cymru_info)),
-				 ("Banned Country list", self.__banManager.geBanListExtendedCountry(cymru_info)),
+				 ("Banned Country list", self.banManager.geBanListExtendedCountry(cymru_info)),
-				 ("Banned RIR list", self.__banManager.geBanListExtendedRIR(cymru_info))]
+				 ("Banned RIR list", self.banManager.geBanListExtendedRIR(cymru_info))]
 		return ret

fail2ban/server/database.py

@@ -104,7 +104,11 @@ def commitandrollback(f):
 	def wrapper(self, *args, **kwargs):
 		with self._lock: # Threading lock
 			with self._db: # Auto commit and rollback on exception
-				return f(self, self._db.cursor(), *args, **kwargs)
+				cur = self._db.cursor()
+				try:
+					return f(self, cur, *args, **kwargs)
+				finally:
+					cur.close()
 	return wrapper
 
 
@@ -253,7 +257,7 @@ class Fail2BanDb(object):
 			self.repairDB()
 		else:
 			version = cur.fetchone()[0]
-			if version < Fail2BanDb.__version__:
+			if version != Fail2BanDb.__version__:
 				newversion = self.updateDb(version)
 				if newversion == Fail2BanDb.__version__:
 					logSys.warning( "Database updated from '%r' to '%r'",
@@ -301,9 +305,11 @@ class Fail2BanDb(object):
 		try:
 			# backup
 			logSys.info("Trying to repair database %s", self._dbFilename)
+			if not os.path.isfile(self._dbBackupFilename):
 				shutil.move(self._dbFilename, self._dbBackupFilename)
 				logSys.info("  Database backup created: %s", self._dbBackupFilename)
-
+			elif os.path.isfile(self._dbFilename):
+				os.remove(self._dbFilename)
 			# first try to repair using dump/restore in order 
 			Utils.executeCmd((r"""f2b_db=$0; f2b_dbbk=$1; sqlite3 "$f2b_dbbk" ".dump" | sqlite3 "$f2b_db" """,
 				self._dbFilename, self._dbBackupFilename))
@@ -415,7 +421,7 @@ class Fail2BanDb(object):
 			logSys.error("Failed to upgrade database '%s': %s",
 				self._dbFilename, e.args[0], 
 				exc_info=logSys.getEffectiveLevel() <= 10)
-			raise
+			self.repairDB()
 
 	@commitandrollback
 	def addJail(self, cur, jail):
@@ -502,7 +508,7 @@ class Fail2BanDb(object):
 		except TypeError:
 			firstLineMD5 = None
 
-		if not firstLineMD5 and (pos or md5):
+		if firstLineMD5 is None and (pos or md5 is not None):
 			cur.execute(
 					"INSERT OR REPLACE INTO logs(jail, path, firstlinemd5, lastfilepos) "
 						"VALUES(?, ?, ?, ?)", (jail.name, name, md5, pos))
@@ -599,7 +605,7 @@ class Fail2BanDb(object):
 		ticket : BanTicket
 			Ticket of the ban to be added.
 		"""
-		ip = str(ticket.getIP())
+		ip = str(ticket.getID())
 		try:
 			del self._bansMergedCache[(ip, jail)]
 		except KeyError:
@@ -789,7 +795,6 @@ class Fail2BanDb(object):
 			queryArgs.append(fromtime)
 		if overalljails or jail is None:
 			query += " GROUP BY ip ORDER BY timeofban DESC LIMIT 1"
-		cur = self._db.cursor()
 		# repack iterator as long as in lock:
 		return list(cur.execute(query, queryArgs))
 
@@ -812,11 +817,9 @@ class Fail2BanDb(object):
 			query += " GROUP BY ip ORDER BY ip, timeofban DESC"
 		else:
 			query += " ORDER BY timeofban DESC LIMIT 1"
-		cur = self._db.cursor()
 		return cur.execute(query, queryArgs)
 
-	@commitandrollback
+	def getCurrentBans(self, jail=None, ip=None, forbantime=None, fromtime=None,
-	def getCurrentBans(self, cur, jail=None, ip=None, forbantime=None, fromtime=None,
 		correctBanTime=True, maxmatches=None
 	):
 		"""Reads tickets (with merged info) currently affected from ban from the database.
@@ -828,6 +831,8 @@ class Fail2BanDb(object):
 		(and therefore endOfBan) of the ticket (normally it is ban-time of jail as maximum)
 		for all tickets with ban-time greater (or persistent).
 		"""
+		cur = self._db.cursor()
+		try:
 			if fromtime is None:
 				fromtime = MyTime.time()
 			tickets = []
@@ -837,9 +842,11 @@ class Fail2BanDb(object):
 				# don't change if persistent allowed:
 				if correctBanTime == -1: correctBanTime = None
 
-		for ticket in self._getCurrentBans(cur, jail=jail, ip=ip, 
+			with self._lock:
+				bans = self._getCurrentBans(cur, jail=jail, ip=ip, 
 					forbantime=forbantime, fromtime=fromtime
-		):
+				)
+			for ticket in bans:
 				# can produce unpack error (database may return sporadical wrong-empty row):
 				try:
 					banip, timeofban, bantime, bancount, data = ticket
@@ -879,6 +886,8 @@ class Fail2BanDb(object):
 				ticket.setBanCount(bancount)
 				if ip is not None: return ticket
 				tickets.append(ticket)
+		finally:
+			cur.close() 
 
 		return tickets
 

fail2ban/server/datedetector.py

@@ -35,7 +35,7 @@ from ..helpers import getLogger
 # Gets the instance of the logger.
 logSys = getLogger(__name__)
 
-logLevel = 6
+logLevel = 5
 
 RE_DATE_PREMATCH = re.compile(r"(?<!\\)\{DATE\}", re.IGNORECASE)
 DD_patternCache = Utils.Cache(maxCount=1000, maxTime=60*60)
@@ -365,10 +365,10 @@ class DateDetector(object):
 				# with space or some special char), otherwise possible collision/pattern switch:
 				if ((
 					line[distance-1:distance] == self.__lastPos[1] or
-						(line[distance] == self.__lastPos[2] and not self.__lastPos[2].isalnum())
+						(line[distance:distance+1] == self.__lastPos[2] and not self.__lastPos[2].isalnum())
 					) and (
 					line[endpos:endpos+1] == self.__lastEndPos[2] or
-						(line[endpos-1] == self.__lastEndPos[1] and not self.__lastEndPos[1].isalnum())
+						(line[endpos-1:endpos] == self.__lastEndPos[1] and not self.__lastEndPos[1].isalnum())
 				)):
 					# search in line part only:
 					log(logLevel-1, "  boundaries are correct, search in part %r", line[distance:endpos])

fail2ban/server/datetemplate.py

@@ -35,6 +35,7 @@ logSys = getLogger(__name__)
 # check already grouped contains "(", but ignores char "\(" and conditional "(?(id)...)":
 RE_GROUPED = re.compile(r'(?<!(?:\(\?))(?<!\\)\((?!\?)')
 RE_GROUP = ( re.compile(r'^((?:\(\?\w+\))?\^?(?:\(\?\w+\))?)(.*?)(\$?)$'), r"\1(\2)\3" )
+RE_GLOBALFLAGS = re.compile(r'((?:^|(?!<\\))\(\?[a-z]+\))')
 
 RE_EXLINE_NO_BOUNDS = re.compile(r'^\{UNB\}')
 RE_EXLINE_BOUND_BEG = re.compile(r'^\{\^LN-BEG\}')
@@ -110,6 +111,11 @@ class DateTemplate(object):
 		# because it may be very slow in negative case (by long log-lines not matching pattern)
 
 		regex = regex.strip()
+		# cut global flags like (?iu) from RE in order to pre-set it after processing:
+		gf = RE_GLOBALFLAGS.search(regex)
+		if gf:
+			regex = RE_GLOBALFLAGS.sub('', regex, count=1)
+		# check word boundaries needed:
 		boundBegin = wordBegin and not RE_NO_WRD_BOUND_BEG.search(regex)
 		boundEnd = wordEnd and not RE_NO_WRD_BOUND_END.search(regex)
 		# if no group add it now, should always have a group(1):
@@ -135,8 +141,10 @@ class DateTemplate(object):
 			self.flags |= DateTemplate.LINE_END
 		# remove possible special pattern "**" in front and end of regex:
 		regex = RE_DEL_WRD_BOUNDS[0].sub(RE_DEL_WRD_BOUNDS[1], regex)
+		if gf: # restore global flags:
+			regex = gf.group(1) + regex
 		self._regex = regex
-		logSys.log(7, '  constructed regex %s', regex)
+		logSys.log(4, '  constructed regex %s', regex)
 		self._cRegex = None
 
 	regex = property(getRegex, setRegex, doc=
@@ -159,6 +167,7 @@ class DateTemplate(object):
 		"""
 		if not self._cRegex:
 			self._compileRegex()
+		logSys.log(4, "   search %s", self.regex)
 		dateMatch = self._cRegex.search(line, *args); # pos, endpos
 		if dateMatch:
 			self.hits += 1

fail2ban/server/failmanager.py

@@ -127,9 +127,10 @@ class FailManager:
 		return len(self.__failList)
 	
 	def cleanup(self, time):
+		time -= self.__maxTime
 		with self.__lock:
 			todelete = [fid for fid,item in self.__failList.iteritems() \
-				if item.getTime() + self.__maxTime <= time]
+				if item.getTime() <= time]
 			if len(todelete) == len(self.__failList):
 				# remove all:
 				self.__failList = dict()
@@ -143,7 +144,7 @@ class FailManager:
 			else:
 				# create new dictionary without items to be deleted:
 				self.__failList = dict((fid,item) for fid,item in self.__failList.iteritems() \
-					if item.getTime() + self.__maxTime > time)
+					if item.getTime() > time)
 		self.__bgSvc.service()
 	
 	def delFailure(self, fid):

fail2ban/server/failregex.py

@@ -91,6 +91,13 @@ R_MAP = {
 	"port": "fport",
 }
 
+# map global flags like ((?i)xxx) or (?:(?i)xxx) to local flags (?i:xxx) if supported by RE-engine in this python version:
+try:
+	re.search("^re(?i:val)$", "reVAL")
+	R_GLOB2LOCFLAGS = ( re.compile(r"(?<!\\)\((?:\?:)?(\(\?[a-z]+)\)"), r"\1:" )
+except:
+	R_GLOB2LOCFLAGS = ()
+
 def mapTag2Opt(tag):
 	tag = tag.lower()
 	return R_MAP.get(tag, tag)
@@ -128,6 +135,9 @@ class Regex:
 		#
 		if regex.lstrip() == '':
 			raise RegexException("Cannot add empty regex")
+		# special handling wrapping global flags to local flags:
+		if R_GLOB2LOCFLAGS:
+			regex = R_GLOB2LOCFLAGS[0].sub(R_GLOB2LOCFLAGS[1], regex)
 		try:
 			self._regexObj = re.compile(regex, re.MULTILINE if multiline else 0)
 			self._regex = regex
@@ -147,9 +157,9 @@ class Regex:
 			self._tupleValues.sort()
 			self._altValues = self._altValues if len(self._altValues) else None
 			self._tupleValues = self._tupleValues if len(self._tupleValues) else None
-		except sre_constants.error:
+		except sre_constants.error as e:
-			raise RegexException("Unable to compile regular expression '%s'" %
+			raise RegexException("Unable to compile regular expression '%s':\n%s" %
-								 regex)
+								 (regex, e))
 		# set fetch handler depending on presence of alternate (or tuple) tags:
 		self.getGroups = self._getGroupsWithAlt if (self._altValues or self._tupleValues) else self._getGroups
 

fail2ban/server/filter.py

@@ -94,6 +94,8 @@ class Filter(JailThread):
 		## Store last time stamp, applicable for multi-line
 		self.__lastTimeText = ""
 		self.__lastDate = None
+		## Next service (cleanup) time
+		self.__nextSvcTime = -(1<<63)
 		## if set, treat log lines without explicit time zone to be in this time zone
 		self.__logtimezone = None
 		## Default or preferred encoding (to decode bytes from file or journal):
@@ -103,6 +105,10 @@ class Filter(JailThread):
 		## Error counter (protected, so can be used in filter implementations)
 		## if it reached 100 (at once), run-cycle will go idle
 		self._errors = 0
+		## Next time to update log or journal position in database:
+		self._nextUpdateTM = 0
+		## Pending updates (must be executed at next update time or during stop):
+		self._pendDBUpdates = {}
 		## return raw host (host is not dns):
 		self.returnRawHost = False
 		## check each regex (used for test purposes):
@@ -115,10 +121,10 @@ class Filter(JailThread):
 		self.checkFindTime = True
 		## shows that filter is in operation mode (processing new messages):
 		self.inOperation = True
-		## if true prevents against retarded banning in case of RC by too many failures (disabled only for test purposes):
-		self.banASAP = True
 		## Ticks counter
 		self.ticks = 0
+		## Processed lines counter
+		self.procLines = 0
 		## Thread name:
 		self.name="f2b/f."+self.jailName
 
@@ -442,12 +448,23 @@ class Filter(JailThread):
 
 	def performBan(self, ip=None):
 		"""Performs a ban for IPs (or given ip) that are reached maxretry of the jail."""
-		try: # pragma: no branch - exception is the only way out
 		while True:
+			try:
 				ticket = self.failManager.toBan(ip)
-				self.jail.putFailTicket(ticket)
 			except FailManagerEmpty:
-			self.failManager.cleanup(MyTime.time())
+				break
+			self.jail.putFailTicket(ticket)
+			if ip: break
+		self.performSvc()
+
+	def performSvc(self, force=False):
+		"""Performs a service tasks (clean failure list)."""
+		tm = MyTime.time()
+		# avoid too early clean up:
+		if force or tm >= self.__nextSvcTime:
+			self.__nextSvcTime = tm + 5
+			# clean up failure list:
+			self.failManager.cleanup(tm)
 
 	def addAttempt(self, ip, *matches):
 		"""Generate a failed attempt for ip"""
@@ -536,7 +553,7 @@ class Filter(JailThread):
 		ticket = None
 		if isinstance(ip, FailTicket):
 			ticket = ip
-			ip = ticket.getIP()
+			ip = ticket.getID()
 		elif not isinstance(ip, IPAddr):
 			ip = IPAddr(ip)
 		return self._inIgnoreIPList(ip, ticket, log_ignore)
@@ -606,6 +623,7 @@ class Filter(JailThread):
 		noDate = False
 		if date:
 			tupleLine = line
+			line = "".join(line)
 			self.__lastTimeText = tupleLine[1]
 			self.__lastDate = date
 		else:
@@ -642,29 +660,36 @@ class Filter(JailThread):
 				if self.__lastDate and self.__lastDate > MyTime.time() - 60:
 					tupleLine = ("", self.__lastTimeText, line)
 					date = self.__lastDate
+				elif self.checkFindTime and self.inOperation:
+					date = MyTime.time()
 		
-		if self.checkFindTime:
+		if self.checkFindTime and date is not None:
 			# if in operation (modifications have been really found):
 			if self.inOperation:
 				# if weird date - we'd simulate now for timeing issue (too large deviation from now):
-				if (date is None or date < MyTime.time() - 60 or date > MyTime.time() + 60):
+				delta = int(date - MyTime.time())
-					# log time zone issue as warning once per day:
+				if abs(delta) > 60:
+					# log timing issue as warning once per day:
 					self._logWarnOnce("_next_simByTimeWarn",
-						("Simulate NOW in operation since found time has too large deviation %s ~ %s +/- %s",
+						("Detected a log entry %s %s the current time in operation mode. "
-							date, MyTime.time(), 60),
+						 "This looks like a %s problem. Treating such entries as if they just happened.",
-						("Please check jail has possibly a timezone issue. Line with odd timestamp: %s", 
+						 MyTime.seconds2str(abs(delta)), "before" if delta < 0 else "after",
+						 "latency" if -3300 <= delta < 0 else "timezone"
+						 ),
+						("Please check a jail for a timing issue. Line with odd timestamp: %s",
 						 line))
 					# simulate now as date:
 					date = MyTime.time()
 					self.__lastDate = date
 			else:
 				# in initialization (restore) phase, if too old - ignore:
-				if date is not None and date < MyTime.time() - self.getFindTime():
+				if date < MyTime.time() - self.getFindTime():
 					# log time zone issue as warning once per day:
 					self._logWarnOnce("_next_ignByTimeWarn",
-						("Ignore line since time %s < %s - %s",
+						("Ignoring all log entries older than %ss; these are probably" +
-							date, MyTime.time(), self.getFindTime()),
+						 " messages generated while fail2ban was not running.",
-						("Please check jail has possibly a timezone issue. Line with odd timestamp: %s", 
+							self.getFindTime()),
+						("Please check a jail for a timing issue. Line with odd timestamp: %s",
 						 line))
 					# ignore - too old (obsolete) entry:
 					return []
@@ -677,10 +702,7 @@ class Filter(JailThread):
 		"""Processes the line for failures and populates failManager
 		"""
 		try:
-			for element in self.processLine(line, date):
+			for (_, ip, unixTime, fail) in self.processLine(line, date):
-				ip = element[1]
-				unixTime = element[2]
-				fail = element[3]
 				logSys.debug("Processing line with time:%s and ip:%s", 
 						unixTime, ip)
 				# ensure the time is not in the future, e. g. by some estimated (assumed) time:
@@ -695,11 +717,15 @@ class Filter(JailThread):
 				attempts = self.failManager.addFailure(tick)
 				# avoid RC on busy filter (too many failures) - if attempts for IP/ID reached maxretry,
 				# we can speedup ban, so do it as soon as possible:
-				if self.banASAP and attempts >= self.failManager.getMaxRetry():
+				if attempts >= self.failManager.getMaxRetry():
 					self.performBan(ip)
 				# report to observer - failure was found, for possibly increasing of it retry counter (asynchronous)
 				if Observers.Main is not None:
-					Observers.Main.add('failureFound', self.failManager, self.jail, tick)
+					Observers.Main.add('failureFound', self.jail, tick)
+			self.procLines += 1
+			# every 100 lines check need to perform service tasks:
+			if self.procLines % 100 == 0:
+				self.performSvc()
 			# reset (halve) error counter (successfully processed line):
 			if self._errors:
 				self._errors //= 2
@@ -709,7 +735,7 @@ class Filter(JailThread):
 			# incr common error counter:
 			self.commonError()
 
-	def commonError(self):
+	def commonError(self, reason="common", exc=None):
 		# incr error counter, stop processing (going idle) after 100th error :
 		self._errors += 1
 		# sleep a little bit (to get around time-related errors):
@@ -768,6 +794,8 @@ class Filter(JailThread):
 			# be sure we've correct current state ('nofail' and 'mlfgained' only from last failure)
 			if mlfidGroups.pop('nofail', None): nfflgs |= 4
 			if mlfidGroups.pop('mlfgained', None): nfflgs |= 4
+			# gained resets all pending failures (retaining users to check it later)
+			if nfflgs & 8: mlfidGroups.pop('mlfpending', None)
 			# if we had no pending failures then clear the matches (they are already provided):
 			if (nfflgs & 4) == 0 and not mlfidGroups.get('mlfpending', 0):
 				mlfidGroups.pop("matches", None)
@@ -812,11 +840,9 @@ class Filter(JailThread):
 		failList = list()
 
 		ll = logSys.getEffectiveLevel()
-		returnRawHost = self.returnRawHost
+		defcidr = IPAddr.CIDR_UNSPEC
-		cidr = IPAddr.CIDR_UNSPEC
+		if self.__useDns == "raw" or self.returnRawHost:
-		if self.__useDns == "raw":
+			defcidr = IPAddr.CIDR_RAW
-			returnRawHost = True
-			cidr = IPAddr.CIDR_RAW
 
 		if self.__lineBufferSize > 1:
 			self.__lineBuffer.append(tupleLine)
@@ -879,7 +905,8 @@ class Filter(JailThread):
 				if not self.checkAllRegex or self.__lineBufferSize > 1:
 					self.__lineBuffer, buf = failRegex.getUnmatchedTupleLines(), None
 				# merge data if multi-line failure:
-				raw = returnRawHost
+				cidr = defcidr
+				raw = (defcidr == IPAddr.CIDR_RAW)
 				if preGroups:
 					currFail, fail = fail, preGroups.copy()
 					fail.update(currFail)
@@ -898,49 +925,50 @@ class Filter(JailThread):
 				# failure-id:
 				fid = fail.get('fid')
 				# ip-address or host:
-				host = fail.get('ip4')
+				ip = fail.get('ip4')
-				if host is not None:
+				if ip is not None:
 					cidr = int(fail.get('cidr') or IPAddr.FAM_IPv4)
 					raw = True
 				else:
-					host = fail.get('ip6')
+					ip = fail.get('ip6')
-					if host is not None:
+					if ip is not None:
 						cidr = int(fail.get('cidr') or IPAddr.FAM_IPv6)
 						raw = True
-				if host is None:
+					else:
-					host = fail.get('dns')
+						ip = fail.get('dns')
-					if host is None:
+						if ip is None:
 							# first try to check we have mlfid case (cache connection id):
 							if fid is None and mlfid is None:
 									# if no failure-id also (obscure case, wrong regex), throw error inside getFailID:
 									fid = failRegex.getFailID()
-						host = fid
+							ip = fid
-						cidr = IPAddr.CIDR_RAW
 							raw = True
 				# if mlfid case (not failure):
-				if host is None:
+				if ip is None:
 					if ll <= 7: logSys.log(7, "No failure-id by mlfid %r in regex %s: %s",
 						mlfid, failRegexIndex, fail.get('mlfforget', "waiting for identifier"))
 					fail['mlfpending'] = 1; # mark failure is pending
 					if not self.checkAllRegex and self.ignorePending: return failList
-					ips = [None]
+					fids = [None]
 				# if raw - add single ip or failure-id,
 				# otherwise expand host to multiple ips using dns (or ignore it if not valid):
 				elif raw:
-					ip = IPAddr(host, cidr)
+					# check ip/host equal failure-id, if not - failure with complex id:
-					# check host equal failure-id, if not - failure with complex id:
+					if fid is None or fid == ip:
-					if fid is not None and fid != host:
+						fid = IPAddr(ip, cidr)
-						ip = IPAddr(fid, IPAddr.CIDR_RAW)
+					else:
-					ips = [ip]
+						fail['ip'] = IPAddr(ip, cidr)
+						fid = IPAddr(fid, defcidr)
+					fids = [fid]
 				# otherwise, try to use dns conversion:
 				else:
-					ips = DNSUtils.textToIp(host, self.__useDns)
+					fids = DNSUtils.textToIp(ip, self.__useDns)
 				# if checkAllRegex we must make a copy (to be sure next RE doesn't change merged/cached failure):
 				if self.checkAllRegex and mlfid is not None:
 					fail = fail.copy()
 				# append failure with match to the list:
-				for ip in ips:
+				for fid in fids:
-					failList.append([failRegexIndex, ip, date, fail])
+					failList.append([failRegexIndex, fid, date, fail])
 				if not self.checkAllRegex:
 					break
 			except RegexException as e: # pragma: no cover - unsure if reachable
@@ -1002,9 +1030,6 @@ class FileFilter(Filter):
 			log = self.__logs.pop(path)
 		except KeyError:
 			return
-		db = self.jail.database
-		if db is not None:
-			db.updateLog(self.jail, log)
 		logSys.info("Removed logfile: %r", path)
 		self._delLogPath(path)
 		return
@@ -1068,6 +1093,7 @@ class FileFilter(Filter):
 	# is created and is added to the FailManager.
 
 	def getFailures(self, filename, inOperation=None):
+		if self.idle: return False
 		log = self.getLog(filename)
 		if log is None:
 			logSys.error("Unable to get failures in %s", filename)
@@ -1113,19 +1139,25 @@ class FileFilter(Filter):
 				while not self.idle:
 					line = log.readline()
 					if not self.active: break; # jail has been stopped
-					if not line:
+					if line is None:
 						# The jail reached the bottom, simply set in operation for this log
 						# (since we are first time at end of file, growing is only possible after modifications):
 						log.inOperation = True
 						break
 					# acquire in operation from log and process:
 					self.inOperation = inOperation if inOperation is not None else log.inOperation
-					self.processLineAndAdd(line.rstrip('\r\n'))
+					self.processLineAndAdd(line)
 		finally:
 			log.close()
-		db = self.jail.database
+		if self.jail.database is not None:
-		if db is not None:
+			self._pendDBUpdates[log] = 1
-			db.updateLog(self.jail, log)
+			if (
+				self.ticks % 100 == 0
+				or MyTime.time() >= self._nextUpdateTM
+				or not self.active
+			):
+				self._updateDBPending()
+				self._nextUpdateTM = MyTime.time() + Utils.DEFAULT_SLEEP_TIME * 5
 		return True
 
 	##
@@ -1137,6 +1169,8 @@ class FileFilter(Filter):
 		if logSys.getEffectiveLevel() <= logging.DEBUG:
 			logSys.debug("Seek to find time %s (%s), file size %s", date, 
 				MyTime.time2str(date), fs)
+		if not fs:
+			return
 		minp = container.getPos()
 		maxp = fs
 		tryPos = minp
@@ -1160,8 +1194,8 @@ class FileFilter(Filter):
 			dateTimeMatch = None
 			nextp = None
 			while True:
-				line = container.readline()
+				line = container.readline(False)
-				if not line:
+				if line is None:
 					break
 				(timeMatch, template) = self.dateDetector.matchTime(line)
 				if timeMatch:
@@ -1225,12 +1259,33 @@ class FileFilter(Filter):
 		ret.append(("File list", path))
 		return ret
 
-	def stop(self):
+	def _updateDBPending(self):
-		"""Stop monitoring of log-file(s)
+		"""Apply pending updates (log position) to database.
 		"""
+		db = self.jail.database
+		while True:
+			try:
+				log, args = self._pendDBUpdates.popitem()
+			except KeyError:
+				break
+			db.updateLog(self.jail, log)
+			
+	def onStop(self):
+		"""Stop monitoring of log-file(s). Invoked after run method.
+		"""
+		# ensure positions of pending logs are up-to-date:
+		if self._pendDBUpdates and self.jail.database:
+			self._updateDBPending()
 		# stop files monitoring:
 		for path in self.__logs.keys():
 			self.delLogPath(path)
+
+	def stop(self):
+		"""Stop filter
+		"""
+		# normally onStop will be called automatically in thread after its run ends, 
+		# but for backwards compatibilities we'll invoke it in caller of stop method.
+		self.onStop()
 		# stop thread:
 		super(Filter, self).stop()
 
@@ -1258,34 +1313,56 @@ except ImportError: # pragma: no cover
 
 class FileContainer:
 
-	def __init__(self, filename, encoding, tail=False):
+	def __init__(self, filename, encoding, tail=False, doOpen=False):
 		self.__filename = filename
+		self.waitForLineEnd = True
 		self.setEncoding(encoding)
 		self.__tail = tail
 		self.__handler = None
+		self.__pos = 0
+		self.__pos4hash = 0
+		self.__hash = ''
+		self.__hashNextTime = time.time() + 30
 		# Try to open the file. Raises an exception if an error occurred.
 		handler = open(filename, 'rb')
+		if doOpen: # fail2ban-regex only (don't need to reopen it and check for rotation)
+			self.__handler = handler
+			return
+		try:
 			stats = os.fstat(handler.fileno())
 			self.__ino = stats.st_ino
-		try:
+			if stats.st_size:
 				firstLine = handler.readline()
+				# first line available and contains new-line:
+				if firstLine != firstLine.rstrip(b'\r\n'):
 					# Computes the MD5 of the first line.
 					self.__hash = md5sum(firstLine).hexdigest()
-			# Start at the beginning of file if tail mode is off.
+				# if tail mode scroll to the end of file
 				if tail:
 					handler.seek(0, 2)
 					self.__pos = handler.tell()
-			else:
-				self.__pos = 0
 		finally:
 			handler.close()
 		## shows that log is in operation mode (expecting new messages only from here):
 		self.inOperation = tail
 
+	def __hash__(self):
+		return hash(self.__filename)
+	def __eq__(self, other):
+		return (id(self) == id(other) or
+			self.__filename == (other.__filename if isinstance(other, FileContainer) else other)
+		)
+	def __repr__(self):
+		return 'file-log:'+self.__filename
+
 	def getFileName(self):
 		return self.__filename
 
 	def getFileSize(self):
+		h = self.__handler
+		if h is not None:
+			stats = os.fstat(h.fileno())
+			return stats.st_size
 		return os.path.getsize(self.__filename);
 
 	def setEncoding(self, encoding):
@@ -1304,38 +1381,54 @@ class FileContainer:
 	def setPos(self, value):
 		self.__pos = value
 
-	def open(self):
+	def open(self, forcePos=None):
-		self.__handler = open(self.__filename, 'rb')
+		h = open(self.__filename, 'rb')
+		try:
 			# Set the file descriptor to be FD_CLOEXEC
-		fd = self.__handler.fileno()
+			fd = h.fileno()
 			flags = fcntl.fcntl(fd, fcntl.F_GETFD)
 			fcntl.fcntl(fd, fcntl.F_SETFD, flags | fcntl.FD_CLOEXEC)
+			myHash = self.__hash
 			# Stat the file before even attempting to read it
-		stats = os.fstat(self.__handler.fileno())
+			stats = os.fstat(h.fileno())
-		if not stats.st_size:
+			rotflg = stats.st_size < self.__pos or stats.st_ino != self.__ino
-			# yoh: so it is still an empty file -- nothing should be
+			if rotflg or not len(myHash) or time.time() > self.__hashNextTime:
-			#      read from it yet
+				myHash = ''
-			# print "D: no content -- return"
+				firstLine = h.readline()
-			return False
+				# Computes the MD5 of the first line (if it is complete)
-		firstLine = self.__handler.readline()
+				if firstLine != firstLine.rstrip(b'\r\n'):
-		# Computes the MD5 of the first line.
 					myHash = md5sum(firstLine).hexdigest()
-		## print "D: fn=%s hashes=%s/%s inos=%s/%s pos=%s rotate=%s" % (
+					self.__hashNextTime = time.time() + 30
-		## 	self.__filename, self.__hash, myHash, stats.st_ino, self.__ino, self.__pos,
+			elif stats.st_size == self.__pos:
-		## 	self.__hash != myHash or self.__ino != stats.st_ino)
+				myHash = self.__hash
-		## sys.stdout.flush()
+			# Compare size, hash and inode
-		# Compare hash and inode
+			if rotflg or myHash != self.__hash:
-		if self.__hash != myHash or self.__ino != stats.st_ino:
+				if self.__hash != '':
-			logSys.log(logging.MSG, "Log rotation detected for %s", self.__filename)
+					logSys.log(logging.MSG, "Log rotation detected for %s, reason: %r", self.__filename,
-			self.__hash = myHash
+						(stats.st_size, self.__pos, stats.st_ino, self.__ino, myHash, self.__hash))
 					self.__ino = stats.st_ino
 					self.__pos = 0
+				self.__hash = myHash
+			# if nothing to read from file yet (empty or no new data):
+			if forcePos is not None:
+				self.__pos = forcePos
+			elif stats.st_size <= self.__pos:
+				return False
 			# Sets the file pointer to the last position.
-		self.__handler.seek(self.__pos)
+			h.seek(self.__pos)
+			# leave file open (to read content):
+			self.__handler = h; h = None
+		finally:
+			# close (no content or error only)
+			if h:
+				h.close(); h = None
 		return True
 
 	def seek(self, offs, endLine=True):
 		h = self.__handler
+		if h is None:
+			self.open(offs)
+			h = self.__handler
 		# seek to given position
 		h.seek(offs, 0)
 		# goto end of next line
@@ -1353,6 +1446,9 @@ class FileContainer:
 		try:
 			return line.decode(enc, 'strict')
 		except (UnicodeDecodeError, UnicodeEncodeError) as e:
+			# avoid warning if got incomplete end of line (e. g. '\n' in "...[0A" followed by "00]..." for utf-16le:
+			if (e.end == len(line) and line[e.start] in b'\r\n'):
+				return line[0:e.start].decode(enc, 'replace')
 			global _decode_line_warn
 			lev = 7
 			if not _decode_line_warn.get(filename, 0):
@@ -1361,29 +1457,85 @@ class FileContainer:
 			logSys.log(lev,
 				"Error decoding line from '%s' with '%s'.", filename, enc)
 			if logSys.getEffectiveLevel() <= lev:
-				logSys.log(lev, "Consider setting logencoding=utf-8 (or another appropriate"
+				logSys.log(lev,
-					" encoding) for this jail. Continuing"
+					"Consider setting logencoding to appropriate encoding for this jail. "
-					" to process line ignoring invalid characters: %r",
+					"Continuing to process line ignoring invalid characters: %r",
 					line)
 			# decode with replacing error chars:
 			line = line.decode(enc, 'replace')
 		return line
 
-	def readline(self):
+	def readline(self, complete=True):
+		"""Read line from file
+
+		In opposite to pythons readline it doesn't return new-line, 
+		so returns either the line if line is complete (and complete=True) or None
+		if line is not complete (and complete=True) or there is no content to read.
+		If line is complete (and complete is True), it also shift current known 
+		position to begin of next line.
+
+		Also it is safe against interim new-line bytes (e. g. part of multi-byte char)
+		in given encoding.
+		"""
 		if self.__handler is None:
 			return ""
-		return FileContainer.decode_line(
+		# read raw bytes up to \n char:
-			self.getFileName(), self.getEncoding(), self.__handler.readline())
+		b = self.__handler.readline()
+		if not b:
+			return None
+		bl = len(b)
+		# convert to log-encoding (new-line char could disappear if it is part of multi-byte sequence):
+		r = FileContainer.decode_line(
+			self.getFileName(), self.getEncoding(), b)
+		# trim new-line at end and check the line was written complete (contains a new-line):
+		l = r.rstrip('\r\n')
+		if complete:
+			if l == r:
+				# try to fill buffer in order to find line-end in log encoding:
+				fnd = 0
+				while 1:
+					r = self.__handler.readline()
+					if not r:
+						break
+					b += r
+					bl += len(r)
+					# convert to log-encoding:
+					r = FileContainer.decode_line(
+						self.getFileName(), self.getEncoding(), b)
+					# ensure new-line is not in the middle (buffered 2 strings, e. g. in utf-16le it is "...[0A"+"00]..."):
+					e = r.find('\n')
+					if e >= 0 and e != len(r)-1:
+						l, r = r[0:e], r[0:e+1]
+						# back to bytes and get offset to seek after NL:
+						r = r.encode(self.getEncoding(), 'replace')
+						self.__handler.seek(-bl+len(r), 1)
+						return l
+					# trim new-line at end and check the line was written complete (contains a new-line):
+					l = r.rstrip('\r\n')
+					if l != r:
+						return l
+				if self.waitForLineEnd:
+					# not fulfilled - seek back and return:
+					self.__handler.seek(-bl, 1)
+					return None
+		return l
 
 	def close(self):
-		if not self.__handler is None:
+		if self.__handler is not None:
-			# Saves the last position.
+			# Saves the last real position.
 			self.__pos = self.__handler.tell()
 			# Closes the file.
 			self.__handler.close()
 			self.__handler = None
-		## print "D: Closed %s with pos %d" % (handler, self.__pos)
+
-		## sys.stdout.flush()
+	def __iter__(self):
+		return self
+	def next(self):
+		line = self.readline()
+		if line is None:
+			self.close()
+			raise StopIteration
+		return line
 
 _decode_line_warn = Utils.Cache(maxCount=1000, maxTime=24*60*60);
 

fail2ban/server/filtergamin.py

@@ -55,7 +55,6 @@ class FilterGamin(FileFilter):
 
 	def __init__(self, jail):
 		FileFilter.__init__(self, jail)
-		self.__modified = False
 		# Gamin monitor
 		self.monitor = gamin.WatchMonitor()
 		fd = self.monitor.get_fd()
@@ -67,21 +66,9 @@ class FilterGamin(FileFilter):
 		logSys.log(4, "Got event: " + repr(event) + " for " + path)
 		if event in (gamin.GAMCreated, gamin.GAMChanged, gamin.GAMExists):
 			logSys.debug("File changed: " + path)
-			self.__modified = True
 
 		self.ticks += 1
-		self._process_file(path)
-
-	def _process_file(self, path):
-		"""Process a given file
-
-		TODO -- RF:
-		this is a common logic and must be shared/provided by FileFilter
-		"""
 		self.getFailures(path)
-		if not self.banASAP: # pragma: no cover
-			self.performBan()
-		self.__modified = False
 
 	##
 	# Add a log file path
@@ -128,6 +115,9 @@ class FilterGamin(FileFilter):
 			Utils.wait_for(lambda: not self.active or self._handleEvents(),
 				self.sleeptime)
 			self.ticks += 1
+			if self.ticks % 10 == 0:
+				self.performSvc()
+
 		logSys.debug("[%s] filter terminated", self.jailName)
 		return True
 

fail2ban/server/filterpoll.py

@@ -27,9 +27,7 @@ __license__ = "GPL"
 import os
 import time
 
-from .failmanager import FailManagerEmpty
 from .filter import FileFilter
-from .mytime import MyTime
 from .utils import Utils
 from ..helpers import getLogger, logging
 
@@ -55,7 +53,6 @@ class FilterPoll(FileFilter):
 
 	def __init__(self, jail):
 		FileFilter.__init__(self, jail)
-		self.__modified = False
 		## The time of the last modification of the file.
 		self.__prevStats = dict()
 		self.__file404Cnt = dict()
@@ -115,20 +112,17 @@ class FilterPoll(FileFilter):
 					break
 				for filename in modlst:
 					self.getFailures(filename)
-					self.__modified = True
 
 				self.ticks += 1
-				if self.__modified:
+				if self.ticks % 10 == 0:
-					if not self.banASAP: # pragma: no cover
+					self.performSvc()
-						self.performBan()
-					self.__modified = False
 			except Exception as e: # pragma: no cover
 				if not self.active: # if not active - error by stop...
 					break
 				logSys.error("Caught unhandled exception in main cycle: %r", e,
 					exc_info=logSys.getEffectiveLevel()<=logging.DEBUG)
 				# incr common error counter:
-				self.commonError()
+				self.commonError("unhandled", e)
 		logSys.debug("[%s] filter terminated", self.jailName)
 		return True
 

fail2ban/server/filterpyinotify.py

@@ -75,7 +75,6 @@ class FilterPyinotify(FileFilter):
 
 	def __init__(self, jail):
 		FileFilter.__init__(self, jail)
-		self.__modified = False
 		# Pyinotify watch manager
 		self.__monitor = pyinotify.WatchManager()
 		self.__notifier = None
@@ -140,9 +139,6 @@ class FilterPyinotify(FileFilter):
 		"""
 		if not self.idle:
 			self.getFailures(path)
-			if not self.banASAP: # pragma: no cover
-				self.performBan()
-			self.__modified = False
 
 	def _addPending(self, path, reason, isDir=False):
 		if path not in self.__pending:
@@ -169,7 +165,7 @@ class FilterPyinotify(FileFilter):
 			return
 		found = {}
 		minTime = 60
-		for path, (retardTM, isDir) in self.__pending.iteritems():
+		for path, (retardTM, isDir) in list(self.__pending.items()):
 			if ntm - self.__pendingChkTime < retardTM:
 				if minTime > retardTM: minTime = retardTM
 				continue
@@ -192,7 +188,7 @@ class FilterPyinotify(FileFilter):
 				self._refreshWatcher(path, isDir=isDir)
 			if isDir:
 				# check all files belong to this dir:
-				for logpath in self.__watchFiles:
+				for logpath in list(self.__watchFiles):
 					if logpath.startswith(path + pathsep):
 						# if still no file - add to pending, otherwise refresh and process:
 						if not os.path.isfile(logpath):
@@ -272,11 +268,9 @@ class FilterPyinotify(FileFilter):
 
 	def _addLogPath(self, path):
 		self._addFileWatcher(path)
-		# initial scan:
+		# notify (wake up if in waiting):
 		if self.active:
-			# we can execute it right now:
+			self.__pendingMinTime = 0
-			self._process_file(path)
-		else:
 		# retard until filter gets started, isDir=None signals special case: process file only (don't need to refresh monitor):
 		self._addPending(path, ('INITIAL', path), isDir=None)
 
@@ -291,7 +285,7 @@ class FilterPyinotify(FileFilter):
 			logSys.error("Failed to remove watch on path: %s", path)
 
 		path_dir = dirname(path)
-		for k in self.__watchFiles:
+		for k in list(self.__watchFiles):
 			if k.startswith(path_dir + pathsep):
 				path_dir = None
 				break
@@ -345,16 +339,26 @@ class FilterPyinotify(FileFilter):
 				self.__notifier.process_events()
 
 				# wait for events / timeout:
-				notify_maxtout = self.__notify_maxtout
 				def __check_events():
-					return not self.active or self.__notifier.check_events(timeout=notify_maxtout)
+					return (
-				if Utils.wait_for(__check_events, min(self.sleeptime, self.__pendingMinTime)):
+						not self.active
+						or bool(self.__notifier.check_events(timeout=self.__notify_maxtout))
+						or (self.__pendingMinTime and self.__pending)
+					)
+				wres = Utils.wait_for(__check_events, min(self.sleeptime, self.__pendingMinTime))
+				if wres:
 					if not self.active: break
+					if not isinstance(wres, dict):
 						self.__notifier.read_events()
 
+				self.ticks += 1
+
 				# check pending files/dirs (logrotate ready):
-				if not self.idle:
+				if self.idle:
+					continue
 				self._checkPending()
+				if self.ticks % 10 == 0:
+					self.performSvc()
 
 			except Exception as e: # pragma: no cover
 				if not self.active: # if not active - error by stop...
@@ -362,9 +366,7 @@ class FilterPyinotify(FileFilter):
 				logSys.error("Caught unhandled exception in main cycle: %r", e,
 					exc_info=logSys.getEffectiveLevel()<=logging.DEBUG)
 				# incr common error counter:
-				self.commonError()
+				self.commonError("unhandled", e)
-			
-			self.ticks += 1
 			
 		logSys.debug("[%s] filter exited (pyinotifier)", self.jailName)
 		self.__notifier = None

fail2ban/server/filtersystemd.py

@@ -22,7 +22,7 @@ __author__ = "Steven Hiscocks"
 __copyright__ = "Copyright (c) 2013 Steven Hiscocks"
 __license__ = "GPL"
 
-import datetime
+import os
 import time
 from distutils.version import LooseVersion
 
@@ -91,9 +91,14 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 		try:
 			args['flags'] = int(kwargs.pop('journalflags'))
 		except KeyError:
-			# be sure all journal types will be opened if files specified (don't set flags):
+			# be sure all journal types will be opened if files/path specified (don't set flags):
-			if 'files' not in args or not len(args['files']):
+			if ('files' not in args or not len(args['files'])) and ('path' not in args or not args['path']):
-				args['flags'] = 4
+				args['flags'] = int(os.getenv("F2B_SYSTEMD_DEFAULT_FLAGS", 4))
+				
+		try:
+			args['namespace'] = kwargs.pop('namespace')
+		except KeyError:
+			pass
 
 		return args
 
@@ -245,13 +250,17 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 		logSys.log(5, "[%s] Read systemd journal entry: %s %s", self.jailName,
 			date[0], logline)
 		## use the same type for 1st argument:
-		return ((logline[:0], date[0], logline.replace('\n', '\\n')), date[1])
+		return ((logline[:0], date[0] + ' ', logline.replace('\n', '\\n')), date[1])
 
 	def seekToTime(self, date):
-		if not isinstance(date, datetime.datetime):
+		if isinstance(date, (int, long)):
-			date = datetime.datetime.fromtimestamp(date)
+			date = float(date)
 		self.__journal.seek_realtime(date)
 
+	def inOperationMode(self):
+		self.inOperation = True
+		logSys.info("[%s] Jail is in operation now (process new journal entries)", self.jailName)
+
 	##
 	# Main loop.
 	#
@@ -262,17 +271,40 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 
 		if not self.getJournalMatch():
 			logSys.notice(
-				"Jail started without 'journalmatch' set. "
+				"[%s] Jail started without 'journalmatch' set. "
 				"Jail regexs will be checked against all journal entries, "
-				"which is not advised for performance reasons.")
+				"which is not advised for performance reasons.", self.jailName)
 
+		# Save current cursor position (to recognize in operation mode):
+		logentry = None
+		try:
+			self.__journal.seek_tail()
+			logentry = self.__journal.get_previous()
+			if logentry:
+				self.__journal.get_next()
+		except OSError:
+			logentry = None # Reading failure, so safe to ignore
+		if logentry:
 			# Try to obtain the last known time (position of journal)
-		start_time = 0
+			startTime = 0
 			if self.jail.database is not None:
-			start_time = self.jail.database.getJournalPos(self.jail, 'systemd-journal') or 0
+				startTime = self.jail.database.getJournalPos(self.jail, 'systemd-journal') or 0
 			# Seek to max(last_known_time, now - findtime) in journal
-		start_time = max( start_time, MyTime.time() - int(self.getFindTime()) )
+			startTime = max( startTime, MyTime.time() - int(self.getFindTime()) )
-		self.seekToTime(start_time)
+			self.seekToTime(startTime)
+			# Not in operation while we'll read old messages ...
+			self.inOperation = False
+			# Save current time in order to check time to switch "in operation" mode
+			startTime = (1, MyTime.time(), logentry.get('__CURSOR'))
+		else:
+			# empty journal or no entries for current filter:
+			self.inOperationMode()
+			# seek_tail() seems to have a bug by no entries (could bypass some entries hereafter), so seek to now instead:
+			startTime = MyTime.time()
+			self.seekToTime(startTime)
+			# for possible future switches of in-operation mode:
+			startTime = (0, startTime)
+
 		# Move back one entry to ensure do not end up in dead space
 		# if start time beyond end of journal
 		try:
@@ -280,6 +312,7 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 		except OSError:
 			pass # Reading failure, so safe to ignore
 
+		line = None
 		while self.active:
 			# wait for records (or for timeout in sleeptime seconds):
 			try:
@@ -289,6 +322,7 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 				#self.__journal.wait(self.sleeptime) != journal.NOP
 				## 
 				## wait for entries without sleep in intervals, because "sleeping" in journal.wait:
+				if not logentry:
 					Utils.wait_for(lambda: not self.active or \
 						self.__journal.wait(Utils.DEFAULT_SLEEP_INTERVAL) != journal.NOP,
 						self.sleeptime, 0.00001)
@@ -310,27 +344,50 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 							e, exc_info=logSys.getEffectiveLevel() <= logging.DEBUG)
 					self.ticks += 1
 					if logentry:
-						line = self.formatJournalEntry(logentry)
+						line, tm = self.formatJournalEntry(logentry)
-						self.processLineAndAdd(*line)
+						# switch "in operation" mode if we'll find start entry (+ some delta):
+						if not self.inOperation:
+							if tm >= MyTime.time() - 1: # reached now (approximated):
+								self.inOperationMode()
+							elif startTime[0] == 1:
+								# if it reached start entry (or get read time larger than start time)
+								if logentry.get('__CURSOR') == startTime[2] or tm > startTime[1]:
+									# give the filter same time it needed to reach the start entry:
+									startTime = (0, MyTime.time()*2 - startTime[1])
+							elif tm > startTime[1]: # reached start time (approximated):
+								self.inOperationMode()
+						# process line
+						self.processLineAndAdd(line, tm)
 						self.__modified += 1
 						if self.__modified >= 100: # todo: should be configurable
 							break
 					else:
+						# "in operation" mode since we don't have messages anymore (reached end of journal):
+						if not self.inOperation:
+							self.inOperationMode()
 						break
-				if self.__modified:
-					if not self.banASAP: # pragma: no cover
-						self.performBan()
 				self.__modified = 0
+				if self.ticks % 10 == 0:
+					self.performSvc()
 				# update position in log (time and iso string):
-					if self.jail.database is not None:
+				if self.jail.database:
-						self.jail.database.updateJournal(self.jail, 'systemd-journal', line[1], line[0][1])
+					if line:
+						self._pendDBUpdates['systemd-journal'] = (tm, line[1])
+						line = None
+					if self._pendDBUpdates and (
+				    self.ticks % 100 == 0
+				    or MyTime.time() >= self._nextUpdateTM
+				    or not self.active
+				  ):
+						self._updateDBPending()
+						self._nextUpdateTM = MyTime.time() + Utils.DEFAULT_SLEEP_TIME * 5
 			except Exception as e: # pragma: no cover
 				if not self.active: # if not active - error by stop...
 					break
 				logSys.error("Caught unhandled exception in main cycle: %r", e,
 					exc_info=logSys.getEffectiveLevel()<=logging.DEBUG)
 				# incr common error counter:
-				self.commonError()
+				self.commonError("unhandled", e)
 
 		logSys.debug("[%s] filter terminated", self.jailName)
 
@@ -350,3 +407,22 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 		ret.append(("Journal matches",
 			[" + ".join(" ".join(match) for match in self.__matches)]))
 		return ret
+
+	def _updateDBPending(self):
+		"""Apply pending updates (jornal position) to database.
+		"""
+		db = self.jail.database
+		while True:
+			try:
+				log, args = self._pendDBUpdates.popitem()
+			except KeyError:
+				break
+			db.updateJournal(self.jail, log, *args)
+
+	def onStop(self):
+		"""Stop monitoring of journal. Invoked after run method.
+		"""
+		# ensure positions of pending logs are up-to-date:
+		if self._pendDBUpdates and self.jail.database:
+			self._updateDBPending()
+

fail2ban/server/ipdns.py

@@ -169,27 +169,31 @@ class DNSUtils:
 		DNSUtils.CACHE_ipToName.set(key, name)
 		return name
 
+	# key find cached own hostnames (this tuple-key cannot be used elsewhere):
+	_getSelfNames_key = ('self','dns')
+
 	@staticmethod
 	def getSelfNames():
 		"""Get own host names of self"""
-		# try find cached own hostnames (this tuple-key cannot be used elsewhere):
+		# try find cached own hostnames:
-		key = ('self','dns')
+		names = DNSUtils.CACHE_ipToName.get(DNSUtils._getSelfNames_key)
-		names = DNSUtils.CACHE_ipToName.get(key)
 		# get it using different ways (a set with names of localhost, hostname, fully qualified):
 		if names is None:
 			names = set([
 				'localhost', DNSUtils.getHostname(False), DNSUtils.getHostname(True)
 			]) - set(['']) # getHostname can return ''
 		# cache and return :
-		DNSUtils.CACHE_ipToName.set(key, names)
+		DNSUtils.CACHE_ipToName.set(DNSUtils._getSelfNames_key, names)
 		return names
 
+	# key to find cached own IPs (this tuple-key cannot be used elsewhere):
+	_getSelfIPs_key = ('self','ips')
+
 	@staticmethod
 	def getSelfIPs():
 		"""Get own IP addresses of self"""
-		# try find cached own IPs (this tuple-key cannot be used elsewhere):
+		# to find cached own IPs:
-		key = ('self','ips')
+		ips = DNSUtils.CACHE_nameToIp.get(DNSUtils._getSelfIPs_key)
-		ips = DNSUtils.CACHE_nameToIp.get(key)
 		# get it using different ways (a set with IPs of localhost, hostname, fully qualified):
 		if ips is None:
 			ips = set()
@@ -199,13 +203,30 @@ class DNSUtils:
 				except Exception as e: # pragma: no cover
 					logSys.warning("Retrieving own IPs of %s failed: %s", hostname, e)
 		# cache and return :
-		DNSUtils.CACHE_nameToIp.set(key, ips)
+		DNSUtils.CACHE_nameToIp.set(DNSUtils._getSelfIPs_key, ips)
 		return ips
 
+	_IPv6IsAllowed = None
+
+	@staticmethod
+	def setIPv6IsAllowed(value):
+		DNSUtils._IPv6IsAllowed = value
+		logSys.debug("IPv6 is %s", ('on' if value else 'off') if value is not None else 'auto')
+		return value
+
+	# key to find cached value of IPv6 allowance (this tuple-key cannot be used elsewhere):
+	_IPv6IsAllowed_key = ('self','ipv6-allowed')
+
 	@staticmethod
 	def IPv6IsAllowed():
-		# return os.path.exists("/proc/net/if_inet6") || any((':' in ip) for ip in DNSUtils.getSelfIPs())
+		if DNSUtils._IPv6IsAllowed is not None:
-		return any((':' in ip.ntoa) for ip in DNSUtils.getSelfIPs())
+			return DNSUtils._IPv6IsAllowed
+		v = DNSUtils.CACHE_nameToIp.get(DNSUtils._IPv6IsAllowed_key)
+		if v is not None:
+			return v
+		v = any((':' in ip.ntoa) for ip in DNSUtils.getSelfIPs())
+		DNSUtils.CACHE_nameToIp.set(DNSUtils._IPv6IsAllowed_key, v)
+		return v
 
 
 ##
@@ -236,6 +257,8 @@ class IPAddr(object):
 	FAM_IPv6 = CIDR_RAW - socket.AF_INET6
 
 	def __new__(cls, ipstr, cidr=CIDR_UNSPEC):
+		if cidr == IPAddr.CIDR_UNSPEC and isinstance(ipstr, (tuple, list)):
+			cidr = IPAddr.CIDR_RAW
 		if cidr == IPAddr.CIDR_RAW: # don't cache raw
 			ip = super(IPAddr, cls).__new__(cls)
 			ip.__init(ipstr, cidr)

fail2ban/server/jail.py

@@ -295,7 +295,7 @@ class Jail(object):
 				):
 					try:
 						#logSys.debug('restored ticket: %s', ticket)
-						if self.filter.inIgnoreIPList(ticket.getIP(), log_ignore=True): continue
+						if self.filter.inIgnoreIPList(ticket.getID(), log_ignore=True): continue
 						# mark ticked was restored from database - does not put it again into db:
 						ticket.restored = True
 						# correct start time / ban time (by the same end of ban):

fail2ban/server/jails.py

@@ -22,6 +22,9 @@ __copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2013- Yaroslav Halchenko"
 __license__ = "GPL"
 
 from threading import Lock
+try:
+	from collections.abc import Mapping
+except ImportError:
 	from collections import Mapping
 
 from ..exceptions import DuplicateJailException, UnknownJailException

fail2ban/server/jailthread.py

@@ -67,6 +67,8 @@ class JailThread(Thread):
 		def run_with_except_hook(*args, **kwargs):
 			try:
 				run(*args, **kwargs)
+				# call on stop callback to do some finalizations:
+				self.onStop()
 			except Exception as e:
 				# avoid very sporadic error "'NoneType' object has no attribute 'exc_info'" (https://bugs.python.org/issue7336)
 				# only extremely fast systems are affected ATM (2.7 / 3.x), if thread ends nothing is available here.
@@ -97,6 +99,12 @@ class JailThread(Thread):
 		self.active = True
 		super(JailThread, self).start()
 
+	@abstractmethod
+	def onStop(self): # pragma: no cover - absract
+		"""Abstract - Called when thread ends (after run).
+		"""
+		pass
+
 	def stop(self):
 		"""Sets `active` property to False, to flag run method to return.
 		"""

fail2ban/server/mytime.py

@@ -174,3 +174,62 @@ class MyTime:
 			val = rexp.sub(rpl, val)
 		val = MyTime._str2sec_fini.sub(r"\1+\2", val)
 		return eval(val)
+
+	class seconds2str():
+		"""Converts seconds to string on demand (if string representation needed).
+		Ex: seconds2str(86400*390)            = 1y 3w 4d
+		    seconds2str(86400*368)            = 1y 3d
+		    seconds2str(86400*365.5)          = 1y
+		    seconds2str(86400*2+3600*7+60*15) = 2d 7h 15m
+		    seconds2str(86400*2+3599)         = 2d 1h
+		    seconds2str(3600-5)               = 1h
+		    seconds2str(3600-10)              = 59m 50s
+		    seconds2str(59)                   = 59s
+		"""
+		def __init__(self, sec):
+			self.sec = sec
+		def __str__(self):
+			# s = str(datetime.timedelta(seconds=int(self.sec)))
+			# return s if s[-3:] != ":00" else s[:-3]
+			s = self.sec; c = 3
+			# automatic accuracy: round by large values (and maximally 3 groups)
+			if s >= 31536000: # a year as 365*24*60*60 (don't need to consider leap year by this accuracy)
+				s = int(round(float(s)/86400)) # round by a day
+				r = str(s//365) + 'y '; s %= 365
+				if s >= 7:
+					r += str(s//7) + 'w '; s %= 7
+				if s:
+					r += str(s) + 'd '
+				return r[:-1]
+			if s >= 604800: # a week as 24*60*60*7
+				s = int(round(float(s)/3600)) # round by a hour
+				r = str(s//168) + 'w '; s %= 168
+				if s >= 24:
+					r += str(s//24) + 'd '; s %= 24
+				if s:
+					r += str(s) + 'h '
+				return r[:-1]
+			if s >= 86400: # a day as 24*60*60
+				s = int(round(float(s)/60)) # round by a minute
+				r = str(s//1440) + 'd '; s %= 1440
+				if s >= 60:
+					r += str(s//60) + 'h '; s %= 60
+				if s:
+					r += str(s) + 'm '
+				return r[:-1]
+			if s >= 3595: # a hour as 60*60 (- 5 seconds)
+				s = int(round(float(s)/10)) # round by 10 seconds
+				r = str(s//360) + 'h '; s %= 360
+				if s >= 6: # a minute
+					r += str(s//6) + 'm '; s %= 6
+				return r[:-1]
+			r = ''
+			if s >= 60: # a minute
+				r += str(s//60) + 'm '; s %= 60
+			if s: # remaining seconds
+				r += str(s) + 's '
+			elif not self.sec: # 0s
+				r = '0 '
+			return r[:-1]
+		def __repr__(self):
+			return self.__str__()

fail2ban/server/observer.py

@@ -232,7 +232,7 @@ class ObserverThread(JailThread):
 					if self._paused:
 						continue
 				else:
-					## notify event deleted (shutdown) - just sleep a litle bit (waiting for shutdown events, prevent high cpu usage)
+					## notify event deleted (shutdown) - just sleep a little bit (waiting for shutdown events, prevent high cpu usage)
 					time.sleep(ObserverThread.DEFAULT_SLEEP_INTERVAL)
 					## stop by shutdown and empty queue :
 					if not self.is_full:
@@ -364,7 +364,7 @@ class ObserverThread(JailThread):
 	## [Async] ban time increment functionality ...
 	## -----------------------------------------
 
-	def failureFound(self, failManager, jail, ticket):
+	def failureFound(self, jail, ticket):
 		""" Notify observer a failure for ip was found
 
 		Observer will check ip was known (bad) and possibly increase an retry count
@@ -372,7 +372,7 @@ class ObserverThread(JailThread):
 		# check jail active :
 		if not jail.isAlive() or not jail.getBanTimeExtra("increment"):
 			return
-		ip = ticket.getIP()
+		ip = ticket.getID()
 		unixTime = ticket.getTime()
 		logSys.debug("[%s] Observer: failure found %s", jail.name, ip)
 		# increase retry count for known (bad) ip, corresponding banCount of it (one try will count than 2, 3, 5, 9 ...)  :
@@ -380,7 +380,7 @@ class ObserverThread(JailThread):
 		retryCount = 1
 		timeOfBan = None
 		try:
-			maxRetry = failManager.getMaxRetry()
+			maxRetry = jail.filter.failManager.getMaxRetry()
 			db = jail.database
 			if db is not None:
 				for banCount, timeOfBan, lastBanTime in db.getBan(ip, jail):
@@ -403,18 +403,12 @@ class ObserverThread(JailThread):
 				MyTime.time2str(unixTime), banCount, retryCount,
 				(', Ban' if retryCount >= maxRetry else ''))
 			# retryCount-1, because a ticket was already once incremented by filter self
-			retryCount = failManager.addFailure(ticket, retryCount - 1, True)
+			retryCount = jail.filter.failManager.addFailure(ticket, retryCount - 1, True)
 			ticket.setBanCount(banCount)
 			# after observe we have increased attempt count, compare it >= maxretry ...
 			if retryCount >= maxRetry:
 				# perform the banning of the IP now (again)
-				# [todo]: this code part will be used multiple times - optimize it later.
+				jail.filter.performBan(ip)
-				try: # pragma: no branch - exception is the only way out
-					while True:
-						ticket = failManager.toBan(ip)
-						jail.putFailTicket(ticket)
-				except FailManagerEmpty:
-					failManager.cleanup(MyTime.time())
 
 		except Exception as e:
 			logSys.error('%s', e, exc_info=logSys.getEffectiveLevel()<=logging.DEBUG)
@@ -441,7 +435,7 @@ class ObserverThread(JailThread):
 		if not jail.isAlive() or not jail.database:
 			return banTime
 		be = jail.getBanTimeExtra()
-		ip = ticket.getIP()
+		ip = ticket.getID()
 		orgBanTime = banTime
 		# check ip was already banned (increment time of ban):
 		try:
@@ -462,7 +456,7 @@ class ObserverThread(JailThread):
 					if ticket.getTime() > timeOfBan:
 						logSys.info('[%s] IP %s is bad: %s # last %s - incr %s to %s' % (jail.name, ip, banCount, 
 							MyTime.time2str(timeOfBan), 
-							datetime.timedelta(seconds=int(orgBanTime)), datetime.timedelta(seconds=int(banTime))));
+							MyTime.seconds2str(orgBanTime), MyTime.seconds2str(banTime)))
 					else:
 						ticket.restored = True
 					break
@@ -480,7 +474,7 @@ class ObserverThread(JailThread):
 			return
 		try:
 			oldbtime = btime
-			ip = ticket.getIP()
+			ip = ticket.getID()
 			logSys.debug("[%s] Observer: ban found %s, %s", jail.name, ip, btime)
 			# if not permanent and ban time was not set - check time should be increased:
 			if btime != -1 and ticket.getBanTime() is None:
@@ -491,8 +485,7 @@ class ObserverThread(JailThread):
 			# if not permanent
 			if btime != -1:
 				bendtime = ticket.getTime() + btime
-				logtime = (datetime.timedelta(seconds=int(btime)),
+				logtime = (MyTime.seconds2str(btime), MyTime.time2str(bendtime))
-					MyTime.time2str(bendtime))
 				# check ban is not too old :
 				if bendtime < MyTime.time():
 					logSys.debug('Ignore old bantime %s', logtime[1])
@@ -521,7 +514,7 @@ class ObserverThread(JailThread):
 		"""
 		try:
 			btime = ticket.getBanTime()
-			ip = ticket.getIP()
+			ip = ticket.getID()
 			logSys.debug("[%s] Observer: prolong %s, %s", jail.name, ip, btime)
 			# prolong ticket via actions that expected this:
 			jail.actions._prolongBan(ticket)

fail2ban/server/server.py

@@ -34,7 +34,7 @@ import sys
 
 from .observer import Observers, ObserverThread
 from .jails import Jails
-from .filter import FileFilter, JournalFilter
+from .filter import DNSUtils, FileFilter, JournalFilter
 from .transmitter import Transmitter
 from .asyncserver import AsyncServer, AsyncServerException
 from .. import version
@@ -293,6 +293,11 @@ class Server:
 			for name in self.__jails.keys():
 				self.delJail(name, stop=False, join=True)
 
+	def clearCaches(self):
+		# we need to clear caches, to be able to recognize new IPs/families etc:
+		DNSUtils.CACHE_nameToIp.clear()
+		DNSUtils.CACHE_ipToName.clear()	
+
 	def reloadJails(self, name, opts, begin):
 		if begin:
 			# begin reload:
@@ -314,6 +319,8 @@ class Server:
 						if "--restart" in opts:
 							self.stopJail(name)
 				else:
+					# invalidate caches by reload
+					self.clearCaches()
 					# first unban all ips (will be not restored after (re)start):
 					if "--unban" in opts:
 						self.setUnbanIP()
@@ -384,7 +391,7 @@ class Server:
 		if isinstance(filter_, FileFilter):
 			return filter_.getLogPaths()
 		else: # pragma: systemd no cover
-			logSys.info("Jail %s is not a FileFilter instance" % name)
+			logSys.debug("Jail %s is not a FileFilter instance" % name)
 			return []
 	
 	def addJournalMatch(self, name, match): # pragma: systemd no cover
@@ -402,7 +409,7 @@ class Server:
 		if isinstance(filter_, JournalFilter):
 			return filter_.getJournalMatch()
 		else:
-			logSys.info("Jail %s is not a JournalFilter instance" % name)
+			logSys.debug("Jail %s is not a JournalFilter instance" % name)
 			return []
 	
 	def setLogEncoding(self, name, encoding):
@@ -671,7 +678,10 @@ class Server:
 				return True
 			padding = logOptions.get('padding')
 			# set a format which is simpler for console use
-			if systarget == "SYSLOG":
+			if systarget == "SYSTEMD-JOURNAL":
+				from systemd.journal import JournalHandler
+				hdlr = JournalHandler(SYSLOG_IDENTIFIER='fail2ban')
+			elif systarget == "SYSLOG":
 				facility = logOptions.get('facility', 'DAEMON').upper()
 				# backwards compatibility - default no padding for syslog handler:
 				if padding is None: padding = '0'
@@ -721,9 +731,7 @@ class Server:
 				except (ValueError, KeyError): # pragma: no cover
 					# Is known to be thrown after logging was shutdown once
 					# with older Pythons -- seems to be safe to ignore there
-					# At least it was still failing on 2.6.2-0ubuntu1 (jaunty)
+					if sys.version_info < (3,) or sys.version_info >= (3, 2):
-					if (2, 6, 3) <= sys.version_info < (3,) or \
-							(3, 2) <= sys.version_info:
 						raise
 			# detailed format by deep log levels (as DEBUG=10):
 			if logger.getEffectiveLevel() <= logging.DEBUG: # pragma: no cover
@@ -749,6 +757,7 @@ class Server:
 					verbose = self.__verbose-1
 				fmt = getVerbosityFormat(verbose, addtime=addtime, padding=padding)
 			# tell the handler to use this format
+			if target != "SYSTEMD-JOURNAL":
 				hdlr.setFormatter(logging.Formatter(fmt))
 			logger.addHandler(hdlr)
 			# Does not display this message at startup.
@@ -788,7 +797,7 @@ class Server:
 			return self.__syslogSocket
 
 	def flushLogs(self):
-		if self.__logTarget not in ['STDERR', 'STDOUT', 'SYSLOG']:
+		if self.__logTarget not in ['STDERR', 'STDOUT', 'SYSLOG', 'SYSTEMD-JOURNAL']:
 			for handler in getLogger("fail2ban").handlers:
 				try:
 					handler.doRollover()
@@ -803,6 +812,11 @@ class Server:
 				logSys.info("flush performed on %s" % self.__logTarget)
 			return "flushed"
 			
+	@staticmethod
+	def setIPv6IsAllowed(value):
+		value = _as_bool(value) if value != 'auto' else None
+		return DNSUtils.setIPv6IsAllowed(value)
+
 	def setThreadOptions(self, value):
 		for o, v in value.iteritems():
 			if o == 'stacksize':
@@ -839,6 +853,26 @@ class Server:
 	def getDatabase(self):
 		return self.__db
 
+	@staticmethod
+	def __get_fdlist():
+		"""Generate a list of open file descriptors.
+		
+		This wouldn't work on some platforms, or if proc/fdescfs not mounted, or a chroot environment,
+		then it'd raise a FileExistsError.
+		"""
+		for path in (
+			'/proc/self/fd', # Linux, Cygwin and NetBSD
+			'/proc/fd',      # MacOS and FreeBSD
+		):
+			if os.path.exists(path):
+				def fdlist():
+					for name in os.listdir(path):
+						if name.isdigit():
+							yield int(name)
+				return fdlist()
+		# other platform or unmounted, chroot etc:
+		raise FileExistsError("fd-list not found")
+
 	def __createDaemon(self): # pragma: no cover
 		""" Detach a process from the controlling terminal and run it in the
 			background as a daemon.
@@ -896,25 +930,37 @@ class Server:
 			# Signal to exit, parent of the first child.
 			return None
 	
-		# Close all open files.  Try the system configuration variable, SC_OPEN_MAX,
+		# Close all open files. Try to obtain the range of open descriptors directly.
+		# As a fallback try the system configuration variable, SC_OPEN_MAX,
 		# for the maximum number of open files to close.  If it doesn't exist, use
 		# the default value (configurable).
+		try:
+			fdlist = self.__get_fdlist()
+			maxfd = -1
+		except:
 			try:
 				maxfd = os.sysconf("SC_OPEN_MAX")
 			except (AttributeError, ValueError):
 				maxfd = 256	   # default maximum
+			fdlist = xrange(maxfd+1)
 	
 		# urandom should not be closed in Python 3.4.0. Fixed in 3.4.1
 		# http://bugs.python.org/issue21207
 		if sys.version_info[0:3] == (3, 4, 0): # pragma: no cover
 			urandom_fd = os.open("/dev/urandom", os.O_RDONLY)
-			for fd in range(0, maxfd):
+			for fd in fdlist:
 				try:
 					if not os.path.sameopenfile(urandom_fd, fd):
 						os.close(fd)
 				except OSError:   # ERROR (ignore)
 					pass
 			os.close(urandom_fd)
+		elif maxfd == -1:
+			for fd in fdlist:
+				try:
+					os.close(fd)
+				except OSError:   # ERROR (ignore)
+					pass
 		else:
 			os.closerange(0, maxfd)