Merge branch 'master' into updated-to-latest-jail.conf

.github/ISSUE_TEMPLATE.md

@@ -1,49 +0,0 @@
-_We will be very grateful, if your problem was described as completely as possible,
-enclosing excerpts from logs (if possible within DEBUG mode, if no errors evident
-within INFO mode), and configuration in particular of effected relevant settings
-(e.g., with ` fail2ban-client -d | grep 'affected-jail-name' ` for a particular
-jail troubleshooting).
-Thank you in advance for the details, because such issues like "It does not work" 
-alone could not help to resolve anything!
-Thanks! (remove this paragraph and other comments upon reading)_
-
-### Environment:
-
-_Fill out and check (`[x]`) the boxes which apply. If your Fail2Ban version is outdated, 
-and you can't verify that the issue persists in the recent release, better seek support 
-from the distribution you obtained Fail2Ban from_
-
-- Fail2Ban version (including any possible distribution suffixes):
-- OS, including release name/version: 
-- [ ] Fail2Ban installed via OS/distribution mechanisms
-- [ ] You have not applied any additional foreign patches to the codebase
-- [ ] Some customizations were done to the configuration (provide details below is so)
-
-### The issue:
-
-_Summary here_
-
-#### Steps to reproduce
-
-#### Expected behavior
-
-#### Observed behavior
-
-#### Any additional information
-
-### Configuration, dump and another helpful excerpts
-
-#### Any customizations done to /etc/fail2ban/ configuration
-```
-```
-
-#### Relevant parts of /var/log/fail2ban.log file:
-_preferably obtained while running fail2ban with `loglevel = 4`_
-
-```
-```
-
-#### Relevant lines from monitored log files in question:
-
-```
-```

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,70 @@
+---
+name: Bug report
+about: Report a bug within the fail2ban engines (not filters or jails)
+title: '[BR]: '
+labels: bug
+assignees: ''
+
+---
+
+<!--
+  - Before reporting, please make sure to search the open and closed issues for any reports in the past.
+  - Use this issue template to report a bug in the fail2ban engine (not in a filter or jail).
+  - If you want to request a feature or a new filter, please use "Feature request" or "Filter request" instead.
+  - If you have rather some question, please open or join to some discussion.
+
+  We will be very grateful, if your problem was described as completely as possible,
+  enclosing excerpts from logs (if possible within DEBUG mode, if no errors evident
+  within INFO mode), and configuration in particular of effected relevant settings
+  (e.g., with ` fail2ban-client -d | grep 'affected-jail-name' ` for a particular
+  jail troubleshooting).
+  Thank you in advance for the details, because such issues like "It does not work" 
+  alone could not help to resolve anything!
+  Thanks! 
+  (you can remove this paragraph and other comments upon reading)
+-->
+
+### Environment:
+
+<!--
+  Fill out and check (`[x]`) the boxes which apply. If your Fail2Ban version is outdated, 
+  and you can't verify that the issue persists in the recent release, better seek support 
+  from the distribution you obtained Fail2Ban from
+-->
+
+- Fail2Ban version <!-- including any possible distribution suffixes --> : 
+- OS, including release name/version : 
+- [ ] Fail2Ban installed via OS/distribution mechanisms
+- [ ] You have not applied any additional foreign patches to the codebase
+- [ ] Some customizations were done to the configuration (provide details below is so)
+
+### The issue:
+
+<!-- summary here -->
+
+#### Steps to reproduce
+
+#### Expected behavior
+
+#### Observed behavior
+
+#### Any additional information
+
+
+### Configuration, dump and another helpful excerpts
+
+#### Any customizations done to /etc/fail2ban/ configuration
+<!-- put your configuration excerpts between next 2 lines -->
+```
+```
+
+#### Relevant parts of /var/log/fail2ban.log file:
+<!-- preferably obtained while running fail2ban with `loglevel = 4` -->
+<!-- put your log excerpt between next 2 lines -->
+```
+```
+
+#### Relevant lines from monitored log files:
+<!-- put your log excerpt between next 2 lines -->
+```
+```

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,35 @@
+---
+name: Feature request
+about: Suggest an idea or an enhancement for this project
+title: '[RFE]: '
+labels: enhancement
+assignees: ''
+
+---
+
+<!--
+  - Before requesting, please make sure to search the open and closed issues for any requests in the past.
+  - Use this issue template to request a feature in the fail2ban engine (not a new filter or jail).
+  - If you want to request a new filter or failregex, please use "Filter request" instead.
+  - If you have rather some question, please open or join to some discussion.
+-->
+
+#### Feature request type
+<!--
+  Please provide a summary description of the feature request.
+-->
+
+#### Description
+<!--
+  Please describe the feature in more detail.
+-->
+
+#### Considered alternatives
+<!--
+  A clear and concise description of any alternative solutions or features you've considered.
+-->
+
+#### Any additional information
+<!--
+  Add any other context or screenshots about the feature request here.
+-->

.github/ISSUE_TEMPLATE/filter_request.md

@@ -0,0 +1,59 @@
+---
+name: Filter request
+about: Request a new jail or filter to be supported or existing filter extended with new failregex
+title: '[FR]: '
+labels: filter-request
+assignees: ''
+
+---
+
+<!--
+  - Before requesting, please make sure to search the open and closed issues for any requests in the past.
+  - Sometimes failregex have been already requested before but are not implemented yet due to various reasons.
+  - If there are no hits for your concerns, please proceed otherwise add a comment to the related issue (also if it is closed).
+  - If you want to request a new feature, please use "Feature request" instead.
+  - If you have rather some question, please open or join to some discussion.
+-->
+
+### Environment:
+
+<!--
+  Fill out and check (`[x]`) the boxes which apply.
+-->
+
+- Fail2Ban version <!-- including any possible distribution suffixes --> : 
+- OS, including release name/version : 
+
+#### Service, project or product which log or journal should be monitored
+
+- Name of filter or jail in Fail2Ban (if already exists) : 
+- Service, project or product name, including release name/version : 
+- Repository or URL (if known) : 
+- Service type : 
+- Ports and protocols the service is listening : 
+
+#### Log or journal information
+<!-- Delete unrelated group -->
+
+<!-- Log file -->
+
+- Log file name(s) : 
+
+<!-- Systemd journal -->
+
+- Journal identifier or unit name : 
+
+#### Any additional information
+
+
+### Relevant lines from monitored log files:
+
+#### failures in sense of fail2ban filter (fail2ban must match):
+<!-- put your log excerpt between next 2 lines -->
+```
+```
+
+#### legitimate messages (fail2ban should not consider as failures):
+<!-- put your log excerpt between next 2 lines -->
+```
+```

.github/workflows/main.yml

@@ -0,0 +1,66 @@
+name: CI
+
+# Controls when the action will run. Triggers the workflow on push or pull request
+# events but only for the master branch
+on:
+  push:
+    paths-ignore:
+    - 'doc/**'
+    - 'files/**'
+    - 'man/**'
+  pull_request:
+    paths-ignore:
+    - 'doc/**'
+    - 'files/**'
+    - 'man/**'
+
+# A workflow run is made up of one or more jobs that can run sequentially or in parallel
+jobs:
+  # This workflow contains a single job called "build"
+  build:
+    # The type of runner that the job will run on
+    runs-on: ubuntu-20.04
+    strategy:
+      matrix:
+        python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, '3.10.0-alpha.5', pypy2, pypy3]
+      fail-fast: false
+    # Steps represent a sequence of tasks that will be executed as part of the job
+    steps:
+      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
+      - uses: actions/checkout@v2
+
+      - name: Set up Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+      
+      - name: Python version
+        run: |
+          F2B_PY=$(python -c "import sys; print(sys.version)")
+          echo "Python: ${{ matrix.python-version }} -- $F2B_PY"
+          F2B_PY=${F2B_PY:0:1}
+          echo "Set F2B_PY=$F2B_PY"
+          echo "F2B_PY=$F2B_PY" >> $GITHUB_ENV
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          if [[ "$F2B_PY" = 3 ]] && ! command -v 2to3x -v 2to3 > /dev/null; then
+            pip install 2to3
+          fi
+          pip install systemd-python || echo 'systemd not available'
+          pip install pyinotify || echo 'inotify not available'
+
+      - name: Before scripts
+        run: |
+          cd "$GITHUB_WORKSPACE"
+          # Manually execute 2to3 for now
+          if [[ "$F2B_PY" = 3 ]]; then echo "2to3 ..." && ./fail2ban-2to3; fi
+          # (debug) output current preferred encoding:
+          python -c 'import locale, sys; from fail2ban.helpers import PREFER_ENC; print(PREFER_ENC, locale.getpreferredencoding(), (sys.stdout and sys.stdout.encoding))'
+      
+      - name: Test suite
+        run: if [[ "$F2B_PY" = 2 ]]; then python setup.py test; else python bin/fail2ban-testcases --verbosity=2; fi
+        
+      #- name: Test initd scripts
+      #  run: shellcheck -s bash -e SC1090,SC1091 files/debian-initd

.travis.yml

@@ -18,7 +18,6 @@ matrix:
   - python: 2.7
     name: 2.7 (xenial)
   - python: pypy
-    dist: trusty
   - python: 3.3
     dist: trusty
   - python:  3.4
@@ -70,8 +69,8 @@ script:
   - if [[ "$F2B_PY" = 3 ]]; then coverage run bin/fail2ban-testcases --verbosity=2; fi
   # Use $VENV_BIN (not python) or else sudo will always run the system's python (2.7)
   - sudo $VENV_BIN/pip install .
-  # Doc files should get installed on Travis under Linux (python >= 3.8 seem to use another path segment)
+  # Doc files should get installed on Travis under Linux (some builds/python's seem to use another path segment)
-  - if [[ $TRAVIS_PYTHON_VERSION < 3.8 ]]; then test -e /usr/share/doc/fail2ban/FILTERS; fi
+  - test -e /usr/share/doc/fail2ban/FILTERS && echo 'found' || echo 'not found'
   # Test initd script
   - shellcheck -s bash -e SC1090,SC1091 files/debian-initd
 after_success:

ChangeLog

@@ -10,33 +10,30 @@ ver. 1.0.1-dev-1 (20??/??/??) - development nightly edition
 -----------
 
 ### Compatibility:
+* potential incompatibility by parsing of options of `backend`, `filter` and `action` parameters (if they
+  are partially incorrect), because fail2ban could throw an error now (doesn't silently bypass it anymore).
 * to v.0.11:
   - due to change of `actioncheck` behavior (gh-488), some actions can be incompatible as regards
     the invariant check, if `actionban` or `actionunban` would not throw an error (exit code 
     different from 0) in case of unsane environment.
 
 ### Fixes
-* [stability] prevent race condition - no ban if filter (backend) is continuously busy if
+* readline fixed to consider interim new-line character as part of code point in multi-byte logs
-  too many messages will be found in log, e. g. initial scan of large log-file or journal (gh-2660)
+  (e. g. unicode encoding like utf-16be, utf-16le);
-* pyinotify-backend sporadically avoided initial scanning of log-file by start
+* `filter.d/drupal-auth.conf` more strict regex, extended to match "Login attempt failed from" (gh-2742)
-* python 3.9 compatibility (and Travis CI support)
-* restoring a large number (500+ depending on files ulimit) of current bans when using PyPy fixed
-* manual ban is written to database, so can be restored by restart (gh-2647)
-* `filter.d/common.conf`: avoid substitute of default values in related `lt_*` section, `__prefix_line`
-  should be interpolated in definition section (inside the filter-config, gh-2650)
 
-### New Features
+### New Features and Enhancements
-
-### Enhancements
-* introduced new prefix `{UNB}` for `datepattern` to disable word boundaries in regex;
-* datetemplate: improved anchor detection for capturing groups `(^...)`;
-* performance optimization of `datepattern` (better search algorithm in datedetector, especially for single template);
 * `actioncheck` behavior is changed now (gh-488), so invariant check as well as restore or repair
    of sane environment (in case of recognized unsane state) would only occur on action errors (e. g.
    if ban or unban operations are exiting with other code as 0)
+* better recognition of log rotation, better performance by reopen: avoid unnecessary seek to begin of file
+  (and hash calculation)
+* file filter reads only complete lines (ended with new-line) now, so waits for end of line (for its completion)
+* `filter.d/nginx-http-auth.conf` - extended with parameter mode, so additionally to `auth` (or `normal`) 
+   mode `fallback` (or combined as `aggressive`) can find SSL errors while SSL handshaking, gh-2881
 
 
-ver. 0.11.1 (2020/01/11) - this-is-the-way
+ver. 0.11.2 (2020/11/23) - heal-the-world-with-security-tools
 -----------
 
 ### Compatibility:
@@ -67,6 +64,73 @@ ver. 0.11.1 (2020/01/11) - this-is-the-way
   - Since v0.10 fail2ban supports the matching of IPv6 addresses, but not all ban actions are
     IPv6-capable now.
 
+### Fixes
+* [stability] prevent race condition - no ban if filter (backend) is continuously busy if
+  too many messages will be found in log, e. g. initial scan of large log-file or journal (gh-2660)
+* pyinotify-backend sporadically avoided initial scanning of log-file by start
+* python 3.9 compatibility (and Travis CI support)
+* restoring a large number (500+ depending on files ulimit) of current bans when using PyPy fixed
+* manual ban is written to database, so can be restored by restart (gh-2647)
+* `jail.conf`: don't specify `action` directly in jails (use `action_` or `banaction` instead)
+* no mails-action added per default anymore (e. g. to allow that `action = %(action_mw)s` should be specified
+  per jail or in default section in jail.local), closes gh-2357
+* ensure we've unique action name per jail (also if parameter `actname` is not set but name deviates from standard name, gh-2686)
+* don't use `%(banaction)s` interpolation because it can be complex value (containing `[...]` and/or quotes), 
+  so would bother the action interpolation
+* fixed type conversion in config readers (take place after all interpolations get ready), that allows to 
+  specify typed parameters variable (as substitutions) as well as to supply it in other sections or as init parameters.
+* `action.d/*-ipset*.conf`: several ipset actions fixed (no timeout per default anymore), so no discrepancy
+  between ipset and fail2ban (removal from ipset will be managed by fail2ban only, gh-2703)
+* `action.d/cloudflare.conf`: fixed `actionunban` (considering new-line chars and optionally real json-parsing
+   with `jq`, gh-2140, gh-2656)
+* `action.d/nftables.conf` (type=multiport only): fixed port range selector, replacing `:` with `-` (gh-2763)
+* `action.d/firewallcmd-*.conf` (multiport only): fixed port range selector, replacing `:` with `-` (gh-2821)
+* `action.d/bsd-ipfw.conf`: fixed selection of rule-no by large list or initial `lowest_rule_num` (gh-2836)
+* `filter.d/common.conf`: avoid substitute of default values in related `lt_*` section, `__prefix_line`
+  should be interpolated in definition section (inside the filter-config, gh-2650)
+* `filter.d/dovecot.conf`: 
+  - add managesieve and submission support (gh-2795);
+  - accept messages with more verbose logging (gh-2573);
+* `filter.d/courier-smtp.conf`: prefregex extended to consider port in log-message (gh-2697)
+* `filter.d/traefik-auth.conf`: filter extended with parameter mode (`normal`, `ddos`, `aggressive`) to handle 
+  the match of username differently (gh-2693):
+  - `normal`: matches 401 with supplied username only
+  - `ddos`: matches 401 without supplied username only
+  - `aggressive`: matches 401 and any variant (with and without username)
+* `filter.d/sshd.conf`: normalizing of user pattern in all RE's, allowing empty user (gh-2749)
+
+### New Features and Enhancements
+* fail2ban-regex:
+  - speedup formatted output (bypass unneeded stats creation)
+  - extended with prefregex statistic
+  - more informative output for `datepattern` (e. g. set from filter) - pattern : description
+* parsing of action in jail-configs considers space between action-names as separator also
+  (previously only new-line was allowed), for example `action = a b` would specify 2 actions `a` and `b`
+* new filter and jail for GitLab recognizing failed application logins (gh-2689)
+* new filter and jail for Grafana recognizing failed application logins (gh-2855)
+* new filter and jail for SoftEtherVPN recognizing failed application logins (gh-2723)
+* `filter.d/guacamole.conf` extended with `logging` parameter to follow webapp-logging if it's configured (gh-2631)
+* `filter.d/bitwarden.conf` enhanced to support syslog (gh-2778)
+* introduced new prefix `{UNB}` for `datepattern` to disable word boundaries in regex;
+* datetemplate: improved anchor detection for capturing groups `(^...)`;
+* datepattern: improved handling with wrong recognized timestamps (timezones, no datepattern, etc)
+  as well as some warnings signaling user about invalid pattern or zone (gh-2814):
+  - filter gets mode in-operation, which gets activated if filter starts processing of new messages;
+    in this mode a timestamp read from log-line that appeared recently (not an old line), deviating too much
+    from now (up too 24h), will be considered as now (assuming a timezone issue), so could avoid unexpected 
+    bypass of failure (previously exceeding `findtime`);
+  - better interaction with non-matching optional datepattern or invalid timestamps;
+  - implements special datepattern `{NONE}` - allow to find failures totally without date-time in log messages,
+    whereas filter will use now as timestamp (gh-2802)
+* performance optimization of `datepattern` (better search algorithm in datedetector, especially for single template);
+* fail2ban-client: extended to unban IP range(s) by subnet (CIDR/mask) or hostname (DNS), gh-2791;
+* extended capturing of alternate tags in filter, allowing combine of multiple groups to single tuple token with new tag
+  prefix `<F-TUPLE_`, that would combine value of `<F-V>` with all value of `<F-TUPLE_V?_n?>` tags (gh-2755)
+
+
+ver. 0.11.1 (2020/01/11) - this-is-the-way
+-----------
+
 ### Fixes
 * purge database will be executed now (within observer).
 * restoring currently banned ip after service restart fixed 

FILTERS

@@ -278,6 +278,7 @@ to tune it.  fail2ban-regex -D ...  will present Debuggex URLs for the regexs
 and sample log files that you pass into it.
 
 In general use when using regex debuggers for generating fail2ban filters:
+
 * use regex from the ./fail2ban-regex output (to ensure all substitutions are
 done)
 * replace <HOST> with (?&.ipv4)

MANIFEST

@@ -3,10 +3,9 @@ bin/fail2ban-regex
 bin/fail2ban-server
 bin/fail2ban-testcases
 ChangeLog
+config/action.d/apprise.conf
 config/action.d/abuseipdb.conf
 config/action.d/apf.conf
-config/action.d/badips.conf
-config/action.d/badips.py
 config/action.d/blocklist_de.conf
 config/action.d/bsd-ipfw.conf
 config/action.d/cloudflare.conf
@@ -100,6 +99,8 @@ config/filter.d/exim.conf
 config/filter.d/exim-spam.conf
 config/filter.d/freeswitch.conf
 config/filter.d/froxlor-auth.conf
+config/filter.d/gitlab.conf
+config/filter.d/grafana.conf
 config/filter.d/groupoffice.conf
 config/filter.d/gssftpd.conf
 config/filter.d/guacamole.conf
@@ -139,6 +140,7 @@ config/filter.d/sendmail-auth.conf
 config/filter.d/sendmail-reject.conf
 config/filter.d/sieve.conf
 config/filter.d/slapd.conf
+config/filter.d/softethervpn.conf
 config/filter.d/sogo-auth.conf
 config/filter.d/solid-pop3d.conf
 config/filter.d/squid.conf
@@ -217,7 +219,6 @@ fail2ban/setup.py
 fail2ban-testcases-all
 fail2ban-testcases-all-python3
 fail2ban/tests/action_d/__init__.py
-fail2ban/tests/action_d/test_badips.py
 fail2ban/tests/action_d/test_smtp.py
 fail2ban/tests/actionstestcase.py
 fail2ban/tests/actiontestcase.py
@@ -267,6 +268,8 @@ fail2ban/tests/files/database_v1.db
 fail2ban/tests/files/database_v2.db
 fail2ban/tests/files/filter.d/substition.conf
 fail2ban/tests/files/filter.d/testcase01.conf
+fail2ban/tests/files/filter.d/testcase02.conf
+fail2ban/tests/files/filter.d/testcase02.local
 fail2ban/tests/files/filter.d/testcase-common.conf
 fail2ban/tests/files/ignorecommand.py
 fail2ban/tests/files/logs/3proxy
@@ -301,6 +304,8 @@ fail2ban/tests/files/logs/exim
 fail2ban/tests/files/logs/exim-spam
 fail2ban/tests/files/logs/freeswitch
 fail2ban/tests/files/logs/froxlor-auth
+fail2ban/tests/files/logs/gitlab
+fail2ban/tests/files/logs/grafana
 fail2ban/tests/files/logs/groupoffice
 fail2ban/tests/files/logs/gssftpd
 fail2ban/tests/files/logs/guacamole
@@ -338,6 +343,7 @@ fail2ban/tests/files/logs/sendmail-auth
 fail2ban/tests/files/logs/sendmail-reject
 fail2ban/tests/files/logs/sieve
 fail2ban/tests/files/logs/slapd
+fail2ban/tests/files/logs/softethervpn
 fail2ban/tests/files/logs/sogo-auth
 fail2ban/tests/files/logs/solid-pop3d
 fail2ban/tests/files/logs/squid

config/action.d/abuseipdb.conf

@@ -21,14 +21,13 @@
 #
 # Example, for ssh bruteforce (in section [sshd] of `jail.local`): 
 #   action = %(known/action)s
-#            %(action_abuseipdb)s[abuseipdb_apikey="my-api-key", abuseipdb_category="18,22"]
+#            abuseipdb[abuseipdb_apikey="my-api-key", abuseipdb_category="18,22"]
 #
-# See below for catagories.
+# See below for categories.
 #
-# Original Ref: https://wiki.shaunc.com/wikka.php?wakka=ReportingToAbuseIPDBWithFail2Ban
 # Added to fail2ban by Andrew James Collett (ajcollett)
 
-## abuseIPDB Catagories, `the abuseipdb_category` MUST be set in the jail.conf action call.
+## abuseIPDB Categories, `the abuseipdb_category` MUST be set in the jail.conf action call.
 # Example, for ssh bruteforce: action = %(action_abuseipdb)s[abuseipdb_category="18,22"]
 # ID	Title	Description
 # 3	  Fraud Orders

config/action.d/apprise.conf

@@ -0,0 +1,49 @@
+# Fail2Ban configuration file
+#
+# Author: Chris Caron <lead2gold@gmail.com>
+#
+#
+
+[Definition]
+
+# Option:  actionstart
+# Notes.:  command executed once at the start of Fail2Ban.
+# Values:  CMD
+#
+actionstart = printf %%b "The jail <name> as been started successfully." | <apprise> -t "[Fail2Ban] <name>: started on `uname -n`"
+
+# Option:  actionstop
+# Notes.:  command executed once at the end of Fail2Ban
+# Values:  CMD
+#
+actionstop = printf %%b "The jail <name> has been stopped." | <apprise> -t "[Fail2Ban] <name>: stopped on `uname -n`"
+
+# Option:  actioncheck
+# Notes.:  command executed once before each actionban command
+# Values:  CMD
+#
+actioncheck =
+
+# Option:  actionban
+# Notes.:  command executed when banning an IP. Take care that the
+#          command is executed with Fail2Ban user rights.
+# Tags:    See jail.conf(5) man page
+# Values:  CMD
+#
+actionban = printf %%b "The IP <ip> has just been banned by Fail2Ban after <failures> attempts against <name>" | <apprise> -n "warning" -t "[Fail2Ban] <name>: banned <ip> from `uname -n`"
+
+# Option:  actionunban
+# Notes.:  command executed when unbanning an IP. Take care that the
+#          command is executed with Fail2Ban user rights.
+# Tags:    See jail.conf(5) man page
+# Values:  CMD
+#
+actionunban =
+
+[Init]
+
+# Define location of the default apprise configuration file to use
+#
+config = /etc/fail2ban/apprise.conf
+#
+apprise = apprise -c "<config>"

config/action.d/badips.conf

@@ -1,19 +0,0 @@
-# Fail2ban reporting to badips.com
-#
-# Note: This reports an IP only and does not actually ban traffic. Use
-# another action in the same jail if you want bans to occur.
-#
-# Set the category to the appropriate value before use.
-#
-# To get see register and optional key to get personalised graphs see:
-# http://www.badips.com/blog/personalized-statistics-track-the-attackers-of-all-your-servers-with-one-key
-
-[Definition]
-
-actionban = curl --fail  --user-agent "<agent>" http://www.badips.com/add/<category>/<ip>
-
-[Init]
-
-# Option: category
-# Notes.: Values are from the list here: http://www.badips.com/get/categories
-category = 

config/action.d/badips.py

@@ -1,391 +0,0 @@
-# emacs: -*- mode: python; py-indent-offset: 4; indent-tabs-mode: t -*-
-# vi: set ft=python sts=4 ts=4 sw=4 noet :
-
-# This file is part of Fail2Ban.
-#
-# Fail2Ban is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2 of the License, or
-# (at your option) any later version.
-#
-# Fail2Ban is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Fail2Ban; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
-
-import sys
-if sys.version_info < (2, 7): # pragma: no cover
-	raise ImportError("badips.py action requires Python >= 2.7")
-import json
-import threading
-import logging
-if sys.version_info >= (3, ): # pragma: 2.x no cover
-	from urllib.request import Request, urlopen
-	from urllib.parse import urlencode
-	from urllib.error import HTTPError
-else: # pragma: 3.x no cover
-	from urllib2 import Request, urlopen, HTTPError
-	from urllib import urlencode
-
-from fail2ban.server.actions import Actions, ActionBase, BanTicket
-from fail2ban.helpers import splitwords, str2LogLevel
-
-
-
-class BadIPsAction(ActionBase): # pragma: no cover - may be unavailable
-	"""Fail2Ban action which reports bans to badips.com, and also
-	blacklist bad IPs listed on badips.com by using another action's
-	ban method.
-
-	Parameters
-	----------
-	jail : Jail
-		The jail which the action belongs to.
-	name : str
-		Name assigned to the action.
-	category : str
-		Valid badips.com category for reporting failures.
-	score : int, optional
-		Minimum score for bad IPs. Default 3.
-	age : str, optional
-		Age of last report for bad IPs, per badips.com syntax.
-		Default "24h" (24 hours)
-	banaction : str, optional
-		Name of banaction to use for blacklisting bad IPs. If `None`,
-		no blacklist of IPs will take place.
-		Default `None`.
-	bancategory : str, optional
-		Name of category to use for blacklisting, which can differ
-		from category used for reporting. e.g. may want to report
-		"postfix", but want to use whole "mail" category for blacklist.
-		Default `category`.
-	bankey : str, optional
-		Key issued by badips.com to retrieve personal list
-		of blacklist IPs.
-	updateperiod : int, optional
-		Time in seconds between updating bad IPs blacklist.
-		Default 900 (15 minutes)
-	loglevel : int/str, optional
-		Log level of the message when an IP is (un)banned.
-		Default `DEBUG`.
-		Can be also supplied as two-value list (comma- or space separated) to
-		provide level of the summary message when a group of IPs is (un)banned.
-		Example `DEBUG,INFO`.
-	agent : str, optional
-		User agent transmitted to server.
-		Default `Fail2Ban/ver.`
-
-	Raises
-	------
-	ValueError
-		If invalid `category`, `score`, `banaction` or `updateperiod`.
-	"""
-
-	TIMEOUT = 10
-	_badips = "https://www.badips.com"
-	def _Request(self, url, **argv):
-		return Request(url, headers={'User-Agent': self.agent}, **argv)
-
-	def __init__(self, jail, name, category, score=3, age="24h",
-		banaction=None, bancategory=None, bankey=None, updateperiod=900, 
-		loglevel='DEBUG', agent="Fail2Ban", timeout=TIMEOUT):
-		super(BadIPsAction, self).__init__(jail, name)
-
-		self.timeout = timeout
-		self.agent = agent
-		self.category = category
-		self.score = score
-		self.age = age
-		self.banaction = banaction
-		self.bancategory = bancategory or category
-		self.bankey = bankey
-		loglevel = splitwords(loglevel)
-		self.sumloglevel = str2LogLevel(loglevel[-1])
-		self.loglevel = str2LogLevel(loglevel[0])
-		self.updateperiod = updateperiod
-
-		self._bannedips = set()
-		# Used later for threading.Timer for updating badips
-		self._timer = None
-
-	@staticmethod
-	def isAvailable(timeout=1):
-		try:
-			response = urlopen(Request("/".join([BadIPsAction._badips]),
-					headers={'User-Agent': "Fail2Ban"}), timeout=timeout)
-			return True, ''
-		except Exception as e: # pragma: no cover
-			return False, e
-
-	def logError(self, response, what=''): # pragma: no cover - sporadical (502: Bad Gateway, etc)
-		messages = {}
-		try:
-			messages = json.loads(response.read().decode('utf-8'))
-		except:
-			pass
-		self._logSys.error(
-			"%s. badips.com response: '%s'", what,
-				messages.get('err', 'Unknown'))
-
-	def getCategories(self, incParents=False):
-		"""Get badips.com categories.
-
-		Returns
-		-------
-		set
-			Set of categories.
-
-		Raises
-		------
-		HTTPError
-			Any issues with badips.com request.
-		ValueError
-			If badips.com response didn't contain necessary information
-		"""
-		try:
-			response = urlopen(
-				self._Request("/".join([self._badips, "get", "categories"])), timeout=self.timeout)
-		except HTTPError as response: # pragma: no cover
-			self.logError(response, "Failed to fetch categories")
-			raise
-		else:
-			response_json = json.loads(response.read().decode('utf-8'))
-			if not 'categories' in response_json:
-				err = "badips.com response lacked categories specification. Response was: %s" \
-				  % (response_json,)
-				self._logSys.error(err)
-				raise ValueError(err)
-			categories = response_json['categories']
-			categories_names = set(
-				value['Name'] for value in categories)
-			if incParents:
-				categories_names.update(set(
-					value['Parent'] for value in categories
-					if "Parent" in value))
-			return categories_names
-
-	def getList(self, category, score, age, key=None):
-		"""Get badips.com list of bad IPs.
-
-		Parameters
-		----------
-		category : str
-			Valid badips.com category.
-		score : int
-			Minimum score for bad IPs.
-		age : str
-			Age of last report for bad IPs, per badips.com syntax.
-		key : str, optional
-			Key issued by badips.com to fetch IPs reported with the
-			associated key.
-
-		Returns
-		-------
-		set
-			Set of bad IPs.
-
-		Raises
-		------
-		HTTPError
-			Any issues with badips.com request.
-		"""
-		try:
-			url = "?".join([
-				"/".join([self._badips, "get", "list", category, str(score)]),
-				urlencode({'age': age})])
-			if key:
-				url = "&".join([url, urlencode({'key': key})])
-			self._logSys.debug('badips.com: get list, url: %r', url)
-			response = urlopen(self._Request(url), timeout=self.timeout)
-		except HTTPError as response: # pragma: no cover
-			self.logError(response, "Failed to fetch bad IP list")
-			raise
-		else:
-			return set(response.read().decode('utf-8').split())
-
-	@property
-	def category(self):
-		"""badips.com category for reporting IPs.
-		"""
-		return self._category
-
-	@category.setter
-	def category(self, category):
-		if category not in self.getCategories():
-			self._logSys.error("Category name '%s' not valid. "
-				"see badips.com for list of valid categories",
-				category)
-			raise ValueError("Invalid category: %s" % category)
-		self._category = category
-
-	@property
-	def bancategory(self):
-		"""badips.com bancategory for fetching IPs.
-		"""
-		return self._bancategory
-
-	@bancategory.setter
-	def bancategory(self, bancategory):
-		if bancategory != "any" and bancategory not in self.getCategories(incParents=True):
-			self._logSys.error("Category name '%s' not valid. "
-				"see badips.com for list of valid categories",
-				bancategory)
-			raise ValueError("Invalid bancategory: %s" % bancategory)
-		self._bancategory = bancategory
-
-	@property
-	def score(self):
-		"""badips.com minimum score for fetching IPs.
-		"""
-		return self._score
-
-	@score.setter
-	def score(self, score):
-		score = int(score)
-		if 0 <= score <= 5:
-			self._score = score
-		else:
-			raise ValueError("Score must be 0-5")
-
-	@property
-	def banaction(self):
-		"""Jail action to use for banning/unbanning.
-		"""
-		return self._banaction
-
-	@banaction.setter
-	def banaction(self, banaction):
-		if banaction is not None and banaction not in self._jail.actions:
-			self._logSys.error("Action name '%s' not in jail '%s'",
-				banaction, self._jail.name)
-			raise ValueError("Invalid banaction")
-		self._banaction = banaction
-
-	@property
-	def updateperiod(self):
-		"""Period in seconds between banned bad IPs will be updated.
-		"""
-		return self._updateperiod
-
-	@updateperiod.setter
-	def updateperiod(self, updateperiod):
-		updateperiod = int(updateperiod)
-		if updateperiod > 0:
-			self._updateperiod = updateperiod
-		else:
-			raise ValueError("Update period must be integer greater than 0")
-
-	def _banIPs(self, ips):
-		for ip in ips:
-			try:
-				ai = Actions.ActionInfo(BanTicket(ip), self._jail)
-				self._jail.actions[self.banaction].ban(ai)
-			except Exception as e:
-				self._logSys.error(
-					"Error banning IP %s for jail '%s' with action '%s': %s",
-					ip, self._jail.name, self.banaction, e,
-					exc_info=self._logSys.getEffectiveLevel()<=logging.DEBUG)
-			else:
-				self._bannedips.add(ip)
-				self._logSys.log(self.loglevel,
-					"Banned IP %s for jail '%s' with action '%s'",
-					ip, self._jail.name, self.banaction)
-
-	def _unbanIPs(self, ips):
-		for ip in ips:
-			try:
-				ai = Actions.ActionInfo(BanTicket(ip), self._jail)
-				self._jail.actions[self.banaction].unban(ai)
-			except Exception as e:
-				self._logSys.error(
-					"Error unbanning IP %s for jail '%s' with action '%s': %s",
-					ip, self._jail.name, self.banaction, e,
-					exc_info=self._logSys.getEffectiveLevel()<=logging.DEBUG)
-			else:
-				self._logSys.log(self.loglevel,
-					"Unbanned IP %s for jail '%s' with action '%s'",
-					ip, self._jail.name, self.banaction)
-			finally:
-				self._bannedips.remove(ip)
-
-	def start(self):
-		"""If `banaction` set, blacklists bad IPs.
-		"""
-		if self.banaction is not None:
-			self.update()
-
-	def update(self):
-		"""If `banaction` set, updates blacklisted IPs.
-
-		Queries badips.com for list of bad IPs, removing IPs from the
-		blacklist if no longer present, and adds new bad IPs to the
-		blacklist.
-		"""
-		if self.banaction is not None:
-			if self._timer:
-				self._timer.cancel()
-				self._timer = None
-
-			try:
-				ips = self.getList(
-					self.bancategory, self.score, self.age, self.bankey)
-				# Remove old IPs no longer listed
-				s = self._bannedips - ips
-				m = len(s)
-				self._unbanIPs(s)
-				# Add new IPs which are now listed
-				s = ips - self._bannedips
-				p = len(s)
-				self._banIPs(s)
-				if m != 0 or p != 0:
-					self._logSys.log(self.sumloglevel,
-						"Updated IPs for jail '%s' (-%d/+%d)",
-						self._jail.name, m, p)
-				self._logSys.debug(
-					"Next update for jail '%' in %i seconds",
-					self._jail.name, self.updateperiod)
-			finally:
-				self._timer = threading.Timer(self.updateperiod, self.update)
-				self._timer.start()
-
-	def stop(self):
-		"""If `banaction` set, clears blacklisted IPs.
-		"""
-		if self.banaction is not None:
-			if self._timer:
-				self._timer.cancel()
-				self._timer = None
-			self._unbanIPs(self._bannedips.copy())
-
-	def ban(self, aInfo):
-		"""Reports banned IP to badips.com.
-
-		Parameters
-		----------
-		aInfo : dict
-			Dictionary which includes information in relation to
-			the ban.
-
-		Raises
-		------
-		HTTPError
-			Any issues with badips.com request.
-		"""
-		try:
-			url = "/".join([self._badips, "add", self.category, str(aInfo['ip'])])
-			self._logSys.debug('badips.com: ban, url: %r', url)
-			response = urlopen(self._Request(url), timeout=self.timeout)
-		except HTTPError as response: # pragma: no cover
-			self.logError(response, "Failed to ban")
-			raise
-		else:
-			messages = json.loads(response.read().decode('utf-8'))
-			self._logSys.debug(
-				"Response from badips.com report: '%s'",
-				messages['suc'])
-
-Action = BadIPsAction

config/action.d/bsd-ipfw.conf

@@ -14,7 +14,10 @@
 # Notes.:  command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false).
 # Values:  CMD
 #
-actionstart = ipfw show | fgrep -c -m 1 -s 'table(<table>)' > /dev/null 2>&1 || ( ipfw show | awk 'BEGIN { b = <lowest_rule_num> } { if ($1 < b) {} else if ($1 == b) { b = $1 + 1 } else { e = b } } END { if (e) exit e <br> else exit b }'; num=$?; ipfw -q add $num <blocktype> <block> from table\(<table>\) to me <port>; echo $num > "<startstatefile>" )
+actionstart = ipfw show | fgrep -c -m 1 -s 'table(<table>)' > /dev/null 2>&1 || (
+    num=$(ipfw show | awk 'BEGIN { b = <lowest_rule_num> } { if ($1 == b) { b = $1 + 1 } } END { print b }');
+    ipfw -q add "$num" <blocktype> <block> from table\(<table>\) to me <port>; echo "$num" > "<startstatefile>"
+  )
 
 
 # Option:  actionstop

config/action.d/cloudflare.conf

@@ -5,7 +5,7 @@
 #
 # Please set jail.local's permission to 640 because it contains your CF API key.
 #
-# This action depends on curl.
+# This action depends on curl (and optionally jq).
 # Referenced from http://www.normyee.net/blog/2012/02/02/adding-cloudflare-support-to-fail2ban by NORM YEE
 #
 # To get your CloudFlare API Key: https://www.cloudflare.com/a/account/my-account
@@ -43,9 +43,9 @@ actioncheck =
 # API v1
 #actionban = curl -s -o /dev/null https://www.cloudflare.com/api_json.html -d 'a=ban' -d 'tkn=<cftoken>' -d 'email=<cfuser>' -d 'key=<ip>'
 # API v4
-actionban = curl -s -o /dev/null -X POST -H 'X-Auth-Email: <cfuser>' -H 'X-Auth-Key: <cftoken>' \
+actionban = curl -s -o /dev/null -X POST <_cf_api_prms> \
-            -H 'Content-Type: application/json' -d '{ "mode": "block", "configuration": { "target": "ip", "value": "<ip>" } }' \
+            -d '{"mode":"block","configuration":{"target":"<cftarget>","value":"<ip>"},"notes":"Fail2Ban <name>"}' \
-            https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules
+            <_cf_api_url>
 
 # Option:  actionunban
 # Notes.:  command executed when unbanning an IP. Take care that the
@@ -58,9 +58,14 @@ actionban = curl -s -o /dev/null -X POST -H 'X-Auth-Email: <cfuser>' -H 'X-Auth-
 # API v1
 #actionunban = curl -s -o /dev/null https://www.cloudflare.com/api_json.html -d 'a=nul' -d 'tkn=<cftoken>' -d 'email=<cfuser>' -d 'key=<ip>'
 # API v4
-actionunban = curl -s -o /dev/null -X DELETE -H 'X-Auth-Email: <cfuser>' -H 'X-Auth-Key: <cftoken>' \
+actionunban = id=$(curl -s -X GET <_cf_api_prms> \
-              https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules/$(curl -s -X GET -H 'X-Auth-Email: <cfuser>' -H 'X-Auth-Key: <cftoken>' \
+                   "<_cf_api_url>?mode=block&configuration_target=<cftarget>&configuration_value=<ip>&page=1&per_page=1&notes=Fail2Ban%%20<name>" \
-              'https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules?mode=block&configuration_target=ip&configuration_value=<ip>&page=1&per_page=1' | cut -d'"' -f6)
+                   | { jq -r '.result[0].id' 2>/dev/null || tr -d '\n' | sed -nE 's/^.*"result"\s*:\s*\[\s*\{\s*"id"\s*:\s*"([^"]+)".*$/\1/p'; })
+              if [ -z "$id" ]; then echo "<name>: id for <ip> cannot be found"; exit 0; fi;
+              curl -s -o /dev/null -X DELETE <_cf_api_prms> "<_cf_api_url>/$id"
+
+_cf_api_url = https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules
+_cf_api_prms = -H 'X-Auth-Email: <cfuser>' -H 'X-Auth-Key: <cftoken>' -H 'Content-Type: application/json'
 
 [Init]
 
@@ -76,3 +81,8 @@ actionunban = curl -s -o /dev/null -X DELETE -H 'X-Auth-Email: <cfuser>' -H 'X-A
 cftoken =
 
 cfuser =
+
+cftarget = ip
+
+[Init?family=inet6]
+cftarget = ip6

config/action.d/firewallcmd-ipset.conf

@@ -18,7 +18,7 @@ before = firewallcmd-common.conf
 
 [Definition]
 
-actionstart = ipset create <ipmset> hash:ip timeout <default-timeout><familyopt>
+actionstart = ipset create <ipmset> hash:ip timeout <default-ipsettime> <familyopt>
               firewall-cmd --direct --add-rule <family> filter <chain> 0 <actiontype> -m set --match-set <ipmset> src -j <blocktype>
 
 actionflush = ipset flush <ipmset>
@@ -27,9 +27,9 @@ actionstop = firewall-cmd --direct --remove-rule <family> filter <chain> 0 <acti
              <actionflush>
              ipset destroy <ipmset>
 
-actionban = ipset add <ipmset> <ip> timeout <bantime> -exist
+actionban = ipset add <ipmset> <ip> timeout <ipsettime> -exist
 
-actionprolong = %(actionban)s
+# actionprolong = %(actionban)s
 
 actionunban = ipset del <ipmset> <ip> -exist
 
@@ -42,11 +42,19 @@ actionunban = ipset del <ipmset> <ip> -exist
 #
 chain = INPUT_direct
 
-# Option: default-timeout
+# Option: default-ipsettime
 # Notes:  specifies default timeout in seconds (handled default ipset timeout only)
-# Values:  [ NUM ]  Default: 600
+# Values:  [ NUM ]  Default: 0 (no timeout, managed by fail2ban by unban)
+default-ipsettime = 0
 
-default-timeout = 600
+# Option: ipsettime
+# Notes:  specifies ticket timeout (handled ipset timeout only)
+# Values:  [ NUM ]  Default: 0 (managed by fail2ban by unban)
+ipsettime = 0
+
+# expresion to caclulate timeout from bantime, example:
+# banaction = %(known/banaction)s[ipsettime='<timeout-bantime>']
+timeout-bantime = $([ "<bantime>" -le 2147483 ] && echo "<bantime>" || echo 0)
 
 # Option: actiontype
 # Notes.: defines additions to the blocking rule
@@ -63,7 +71,7 @@ allports = -p <protocol>
 # Option: multiport
 # Notes.: addition to block access only to specific ports
 # Usage.: use in jail config:  banaction = firewallcmd-ipset[actiontype=<multiport>]
-multiport = -p <protocol> -m multiport --dports <port>
+multiport = -p <protocol> -m multiport --dports "$(echo '<port>' | sed s/:/-/g)"
 
 ipmset = f2b-<name>
 familyopt =
@@ -71,7 +79,7 @@ familyopt =
 [Init?family=inet6]
 
 ipmset = f2b-<name>6
-familyopt = <sp>family inet6
+familyopt = family inet6
 
 
 # DEV NOTES:

config/action.d/firewallcmd-multiport.conf

@@ -11,9 +11,9 @@ before = firewallcmd-common.conf
 
 actionstart = firewall-cmd --direct --add-chain <family> filter f2b-<name>
               firewall-cmd --direct --add-rule <family> filter f2b-<name> 1000 -j RETURN
-              firewall-cmd --direct --add-rule <family> filter <chain> 0 -m conntrack --ctstate NEW -p <protocol> -m multiport --dports <port> -j f2b-<name>
+              firewall-cmd --direct --add-rule <family> filter <chain> 0 -m conntrack --ctstate NEW -p <protocol> -m multiport --dports "$(echo '<port>' | sed s/:/-/g)" -j f2b-<name>
 
-actionstop = firewall-cmd --direct --remove-rule <family> filter <chain> 0 -m conntrack --ctstate NEW -p <protocol> -m multiport --dports <port> -j f2b-<name>
+actionstop = firewall-cmd --direct --remove-rule <family> filter <chain> 0 -m conntrack --ctstate NEW -p <protocol> -m multiport --dports "$(echo '<port>' | sed s/:/-/g)" -j f2b-<name>
              firewall-cmd --direct --remove-rules <family> filter f2b-<name>
              firewall-cmd --direct --remove-chain <family> filter f2b-<name>
 

config/action.d/firewallcmd-new.conf

@@ -10,9 +10,9 @@ before = firewallcmd-common.conf
 
 actionstart = firewall-cmd --direct --add-chain <family> filter f2b-<name>
               firewall-cmd --direct --add-rule <family> filter f2b-<name> 1000 -j RETURN
-              firewall-cmd --direct --add-rule <family> filter <chain> 0 -m state --state NEW -p <protocol> -m multiport --dports <port> -j f2b-<name>
+              firewall-cmd --direct --add-rule <family> filter <chain> 0 -m state --state NEW -p <protocol> -m multiport --dports "$(echo '<port>' | sed s/:/-/g)" -j f2b-<name>
 
-actionstop = firewall-cmd --direct --remove-rule <family> filter <chain> 0 -m state --state NEW -p <protocol> -m multiport --dports <port> -j f2b-<name>
+actionstop = firewall-cmd --direct --remove-rule <family> filter <chain> 0 -m state --state NEW -p <protocol> -m multiport --dports "$(echo '<port>' | sed s/:/-/g)" -j f2b-<name>
              firewall-cmd --direct --remove-rules <family> filter f2b-<name>
              firewall-cmd --direct --remove-chain <family> filter f2b-<name>
 

config/action.d/firewallcmd-rich-logging.conf

@@ -1,6 +1,6 @@
 # Fail2Ban configuration file
 #
-# Author: Donald Yandt 
+# Authors: Donald Yandt, Sergey G. Brester
 # 
 # Because of the rich rule commands requires firewalld-0.3.1+
 # This action uses firewalld rich-rules which gives you a cleaner iptables since it stores rules according to zones and not
@@ -10,36 +10,15 @@
 #
 # If you use the --permanent rule you get a xml file in /etc/firewalld/zones/<zone>.xml that can be shared and parsed easliy
 #
-# Example commands to view rules:
+# This is an derivative of firewallcmd-rich-rules.conf, see there for details and other parameters.
-# firewall-cmd [--zone=<zone>] --list-rich-rules
-# firewall-cmd [--zone=<zone>] --list-all
-# firewall-cmd [--zone=zone] --query-rich-rule='rule'
 
 [INCLUDES]
 
-before = firewallcmd-common.conf
+before = firewallcmd-rich-rules.conf
 
 [Definition]
 
-actionstart = 
+rich-suffix = log prefix='f2b-<name>' level='<level>' limit value='<rate>/m' <rich-blocktype>
-
-actionstop = 
-
-actioncheck = 
-
-# you can also use zones and/or service names. 
-#
-# zone example: 
-# firewall-cmd --zone=<zone> --add-rich-rule="rule family='<family>' source address='<ip>' port port='<port>' protocol='<protocol>' log prefix='f2b-<name>' level='<level>' limit value='<rate>/m' <rich-blocktype>"
-#
-# service name example:
-# firewall-cmd --zone=<zone> --add-rich-rule="rule family='<family>' source address='<ip>' service name='<service>' log prefix='f2b-<name>' level='<level>' limit value='<rate>/m' <rich-blocktype>"
-#
-# Because rich rules can only handle single or a range of ports we must split ports and execute the command for each port. Ports can be single and ranges separated by a comma or space for an example: http, https, 22-60, 18 smtp 
-
-actionban = ports="<port>"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="rule family='<family>' source address='<ip>' port port='$p' protocol='<protocol>' log prefix='f2b-<name>' level='<level>' limit value='<rate>/m' <rich-blocktype>"; done
-	   
-actionunban = ports="<port>"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --remove-rich-rule="rule family='<family>' source address='<ip>' port port='$p' protocol='<protocol>' log prefix='f2b-<name>' level='<level>' limit value='<rate>/m' <rich-blocktype>"; done
 
 [Init]
 
@@ -48,4 +27,3 @@ level = info
 
 # log rate per minute
 rate = 1
-

config/action.d/firewallcmd-rich-rules.conf

@@ -35,8 +35,10 @@ actioncheck =
 #
 # Because rich rules can only handle single or a range of ports we must split ports and execute the command for each port. Ports can be single and ranges separated by a comma or space for an example: http, https, 22-60, 18 smtp 
 
-actionban = ports="<port>"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="rule family='<family>' source address='<ip>' port port='$p' protocol='<protocol>' <rich-blocktype>"; done
+fwcmd_rich_rule = rule family='<family>' source address='<ip>' port port='$p' protocol='<protocol>' %(rich-suffix)s
+
+actionban = ports="$(echo '<port>' | sed s/:/-/g)"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="%(fwcmd_rich_rule)s"; done
 	   
-actionunban = ports="<port>"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --remove-rich-rule="rule family='<family>' source address='<ip>' port port='$p' protocol='<protocol>' <rich-blocktype>"; done
+actionunban = ports="$(echo '<port>' | sed s/:/-/g)"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --remove-rich-rule="%(fwcmd_rich_rule)s"; done
-
 
+rich-suffix = <rich-blocktype>

config/action.d/iptables-ipset-proto6-allports.conf

@@ -26,7 +26,7 @@ before = iptables-common.conf
 # Notes.:  command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false).
 # Values:  CMD
 #
-actionstart = ipset create <ipmset> hash:ip timeout <default-timeout><familyopt>
+actionstart = ipset create <ipmset> hash:ip timeout <default-ipsettime> <familyopt>
               <iptables> -I <chain> -m set --match-set <ipmset> src -j <blocktype>
 
 # Option:  actionflush
@@ -49,9 +49,9 @@ actionstop = <iptables> -D <chain> -m set --match-set <ipmset> src -j <blocktype
 # Tags:    See jail.conf(5) man page
 # Values:  CMD
 #
-actionban = ipset add <ipmset> <ip> timeout <bantime> -exist
+actionban = ipset add <ipmset> <ip> timeout <ipsettime> -exist
 
-actionprolong = %(actionban)s
+# actionprolong = %(actionban)s
 
 # Option:  actionunban
 # Notes.:  command executed when unbanning an IP. Take care that the
@@ -63,11 +63,19 @@ actionunban = ipset del <ipmset> <ip> -exist
 
 [Init]
 
-# Option: default-timeout
+# Option: default-ipsettime
 # Notes:  specifies default timeout in seconds (handled default ipset timeout only)
-# Values:  [ NUM ]  Default: 600
+# Values:  [ NUM ]  Default: 0 (no timeout, managed by fail2ban by unban)
+default-ipsettime = 0
 
-default-timeout = 600
+# Option: ipsettime
+# Notes:  specifies ticket timeout (handled ipset timeout only)
+# Values:  [ NUM ]  Default: 0 (managed by fail2ban by unban)
+ipsettime = 0
+
+# expresion to caclulate timeout from bantime, example:
+# banaction = %(known/banaction)s[ipsettime='<timeout-bantime>']
+timeout-bantime = $([ "<bantime>" -le 2147483 ] && echo "<bantime>" || echo 0)
 
 ipmset = f2b-<name>
 familyopt =
@@ -76,4 +84,4 @@ familyopt =
 [Init?family=inet6]
 
 ipmset = f2b-<name>6
-familyopt = <sp>family inet6
+familyopt = family inet6

config/action.d/iptables-ipset-proto6.conf

@@ -26,7 +26,7 @@ before = iptables-common.conf
 # Notes.:  command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false).
 # Values:  CMD
 #
-actionstart = ipset create <ipmset> hash:ip timeout <default-timeout><familyopt>
+actionstart = ipset create <ipmset> hash:ip timeout <default-ipsettime> <familyopt>
               <iptables> -I <chain> -p <protocol> -m multiport --dports <port> -m set --match-set <ipmset> src -j <blocktype>
 
 # Option:  actionflush
@@ -49,9 +49,9 @@ actionstop = <iptables> -D <chain> -p <protocol> -m multiport --dports <port> -m
 # Tags:    See jail.conf(5) man page
 # Values:  CMD
 #
-actionban = ipset add <ipmset> <ip> timeout <bantime> -exist
+actionban = ipset add <ipmset> <ip> timeout <ipsettime> -exist
 
-actionprolong = %(actionban)s
+# actionprolong = %(actionban)s
 
 # Option:  actionunban
 # Notes.:  command executed when unbanning an IP. Take care that the
@@ -63,11 +63,19 @@ actionunban = ipset del <ipmset> <ip> -exist
 
 [Init]
 
-# Option: default-timeout
+# Option: default-ipsettime
 # Notes:  specifies default timeout in seconds (handled default ipset timeout only)
-# Values:  [ NUM ]  Default: 600
+# Values:  [ NUM ]  Default: 0 (no timeout, managed by fail2ban by unban)
+default-ipsettime = 0
 
-default-timeout = 600
+# Option: ipsettime
+# Notes:  specifies ticket timeout (handled ipset timeout only)
+# Values:  [ NUM ]  Default: 0 (managed by fail2ban by unban)
+ipsettime = 0
+
+# expresion to caclulate timeout from bantime, example:
+# banaction = %(known/banaction)s[ipsettime='<timeout-bantime>']
+timeout-bantime = $([ "<bantime>" -le 2147483 ] && echo "<bantime>" || echo 0)
 
 ipmset = f2b-<name>
 familyopt =
@@ -76,4 +84,4 @@ familyopt =
 [Init?family=inet6]
 
 ipmset = f2b-<name>6
-familyopt = <sp>family inet6
+familyopt = family inet6

config/action.d/nftables.conf

@@ -34,7 +34,7 @@ type = multiport
 
 rule_match-custom =
 rule_match-allports = meta l4proto \{ <protocol> \}
-rule_match-multiport = $proto dport \{ <port> \}
+rule_match-multiport = $proto dport \{ $(echo '<port>' | sed s/:/-/g) \}
 match = <rule_match-<type>>
 
 # Option:  rule_stat

config/action.d/nginx-block-map.conf

@@ -84,8 +84,15 @@ srv_cfg_path = /etc/nginx/
 #srv_cmd = nginx -c %(srv_cfg_path)s/nginx.conf
 srv_cmd = nginx
 
-# first test configuration is correct, hereafter send reload signal:
+# pid file (used to check nginx is running):
-blck_lst_reload = %(srv_cmd)s -qt; if [ $? -eq 0 ]; then
+srv_pid = /run/nginx.pid
+
+# command used to check whether nginx is running and configuration is valid:
+srv_is_running = [ -f "%(srv_pid)s" ]
+srv_check_cmd = %(srv_is_running)s && %(srv_cmd)s -qt
+
+# first test nginx is running and configuration is correct, hereafter send reload signal:
+blck_lst_reload = %(srv_check_cmd)s; if [ $? -eq 0 ]; then
                     %(srv_cmd)s -s reload; if [ $? -ne 0 ]; then echo 'reload failed.'; fi;
                   fi;
 

config/action.d/shorewall-ipset-proto6.conf

@@ -51,7 +51,7 @@
 # Values:  CMD
 #
 actionstart = if ! ipset -quiet -name list f2b-<name> >/dev/null;
-              then ipset -quiet -exist create f2b-<name> hash:ip timeout <default-timeout>;
+              then ipset -quiet -exist create f2b-<name> hash:ip timeout <default-ipsettime>;
               fi
 
 # Option:  actionstop
@@ -66,9 +66,9 @@ actionstop = ipset flush f2b-<name>
 # Tags:    See jail.conf(5) man page
 # Values:  CMD
 #
-actionban = ipset add f2b-<name> <ip> timeout <bantime> -exist
+actionban = ipset add f2b-<name> <ip> timeout <ipsettime> -exist
 
-actionprolong = %(actionban)s
+# actionprolong = %(actionban)s
 
 # Option:  actionunban
 # Notes.:  command executed when unbanning an IP. Take care that the
@@ -78,8 +78,16 @@ actionprolong = %(actionban)s
 #
 actionunban = ipset del f2b-<name> <ip> -exist
 
-# Option: default-timeout
+# Option: default-ipsettime
 # Notes:  specifies default timeout in seconds (handled default ipset timeout only)
-# Values:  [ NUM ]  Default: 600
+# Values:  [ NUM ]  Default: 0 (no timeout, managed by fail2ban by unban)
+default-ipsettime = 0
 
-default-timeout = 600
+# Option: ipsettime
+# Notes:  specifies ticket timeout (handled ipset timeout only)
+# Values:  [ NUM ]  Default: 0 (managed by fail2ban by unban)
+ipsettime = 0
+
+# expresion to caclulate timeout from bantime, example:
+# banaction = %(known/banaction)s[ipsettime='<timeout-bantime>']
+timeout-bantime = $([ "<bantime>" -le 2147483 ] && echo "<bantime>" || echo 0)

config/fail2ban.conf

@@ -19,7 +19,7 @@
 #         NOTICE
 #         INFO
 #         DEBUG
-# Values: [ LEVEL ]  Default: ERROR
+# Values: [ LEVEL ]  Default: INFO
 #
 loglevel = INFO
 
@@ -55,6 +55,12 @@ socket = /var/run/fail2ban/fail2ban.sock
 #
 pidfile = /var/run/fail2ban/fail2ban.pid
 
+# Option: allowipv6
+# Notes.: Allows IPv6 interface:
+#         Default: auto
+# Values: [ auto yes (on, true, 1) no (off, false, 0) ] Default: auto
+#allowipv6 = auto
+
 # Options: dbfile
 # Notes.: Set the file for the fail2ban persistent data to be stored.
 #         A value of ":memory:" means database is only stored in memory 

config/filter.d/apache-noscript.conf

@@ -17,9 +17,9 @@ before = apache-common.conf
 
 [Definition]
 
-script = /\S*(?:php(?:[45]|[.-]cgi)?|\.asp|\.exe|\.pl)
+script = /\S*(?:php(?:[45]|[.-]cgi)?|\.asp|\.exe|\.pl|\bcgi-bin/)
 
-prefregex = ^%(_apache_error_client)s (?:AH0(?:01(?:28|30)|1(?:264|071)): )?(?:(?:[Ff]ile|script|[Gg]ot) )<F-CONTENT>.+</F-CONTENT>$
+prefregex = ^%(_apache_error_client)s (?:AH0(?:01(?:28|30)|1(?:264|071)|2811): )?(?:(?:[Ff]ile|script|[Gg]ot) )<F-CONTENT>.+</F-CONTENT>$
 
 failregex = ^(?:does not exist|not found or unable to stat): <script>\b
             ^'<script>\S*' not found or unable to stat

config/filter.d/apache-overflows.conf

@@ -8,7 +8,7 @@ before = apache-common.conf
 
 [Definition]
 
-failregex = ^%(_apache_error_client)s (?:(?:AH0013[456]: )?Invalid (method|URI) in request\b|(?:AH00565: )?request failed: URI too long \(longer than \d+\)|request failed: erroneous characters after protocol string:|(?:AH00566: )?request failed: invalid characters in URI\b)
+failregex = ^%(_apache_error_client)s (?:(?:AH001[23][456]: )?Invalid (method|URI) in request\b|(?:AH00565: )?request failed: URI too long \(longer than \d+\)|request failed: erroneous characters after protocol string:|(?:AH00566: )?request failed: invalid characters in URI\b)
 
 ignoreregex =
 

config/filter.d/asterisk.conf

@@ -21,7 +21,7 @@ log_prefix= (?:NOTICE|SECURITY|WARNING)%(__pid_re)s:?(?:\[C-[\da-f]*\])?:? [^:]+
 prefregex = ^%(__prefix_line)s%(log_prefix)s <F-CONTENT>.+</F-CONTENT>$
 
 failregex = ^Registration from '[^']*' failed for '<HOST>(:\d+)?' - (?:Wrong password|Username/auth name mismatch|No matching peer found|Not a local domain|Device does not match ACL|Peer is not supposed to register|ACL error \(permit/deny\)|Not a local domain)$
-            ^Call from '[^']*' \(<HOST>:\d+\) to extension '[^']*' rejected because extension not found in context
+            ^Call from '[^']*' \((?:(?:TCP|UDP):)?<HOST>:\d+\) to extension '[^']*' rejected because extension not found in context
             ^(?:Host )?<HOST> (?:failed (?:to authenticate\b|MD5 authentication\b)|tried to authenticate with nonexistent user\b)
             ^No registration for peer '[^']*' \(from <HOST>\)$
             ^hacking attempt detected '<HOST>'$

config/filter.d/bitwarden.conf

@@ -2,5 +2,12 @@
 # Detecting failed login attempts
 # Logged in bwdata/logs/identity/Identity/log.txt
 
+[INCLUDES]
+before = common.conf
+
 [Definition]
-failregex = ^\s*\[WRN\]\s+Failed login attempt(?:, 2FA invalid)?\. <HOST>$
+_daemon = Bitwarden-Identity
+failregex = ^%(__prefix_line)s\s*\[(?:W(?:RN|arning)|Bit\.Core\.[^\]]+)\]\s+Failed login attempt(?:, 2FA invalid)?\. <ADDR>$
+
+# DEV Notes:
+# __prefix_line can result to an empty string, so it can support syslog and non-syslog at once.

config/filter.d/courier-smtp.conf

@@ -12,7 +12,7 @@ before = common.conf
 
 _daemon = courieresmtpd
 
-prefregex = ^%(__prefix_line)serror,relay=<HOST>,<F-CONTENT>.+</F-CONTENT>$
+prefregex = ^%(__prefix_line)serror,relay=<HOST>,(?:port=\d+,)?<F-CONTENT>.+</F-CONTENT>$
 
 failregex = ^[^:]*: 550 User (<.*> )?unknown\.?$
             ^msg="535 Authentication failed\.",cmd:( AUTH \S+)?( [0-9a-zA-Z\+/=]+)?(?: \S+)$

config/filter.d/dovecot.conf

@@ -10,15 +10,15 @@ before = common.conf
 _auth_worker = (?:dovecot: )?auth(?:-worker)?
 _daemon = (?:dovecot(?:-auth)?|auth)
 
-prefregex = ^%(__prefix_line)s(?:%(_auth_worker)s(?:\([^\)]+\))?: )?(?:%(__pam_auth)s(?:\(dovecot:auth\))?: |(?:pop3|imap)-login: )?(?:Info: )?<F-CONTENT>.+</F-CONTENT>$
+prefregex = ^%(__prefix_line)s(?:%(_auth_worker)s(?:\([^\)]+\))?: )?(?:%(__pam_auth)s(?:\(dovecot:auth\))?: |(?:pop3|imap|managesieve|submission)-login: )?(?:Info: )?<F-CONTENT>.+</F-CONTENT>$
 
 failregex = ^authentication failure; logname=<F-ALT_USER1>\S*</F-ALT_USER1> uid=\S* euid=\S* tty=dovecot ruser=<F-USER>\S*</F-USER> rhost=<HOST>(?:\s+user=<F-ALT_USER>\S*</F-ALT_USER>)?\s*$
-            ^(?:Aborted login|Disconnected)(?::(?: [^ \(]+)+)? \((?:auth failed, \d+ attempts(?: in \d+ secs)?|tried to use (?:disabled|disallowed) \S+ auth|proxy dest auth failed)\):(?: user=<<F-USER>[^>]*</F-USER>>,)?(?: method=\S+,)? rip=<HOST>(?:[^>]*(?:, session=<\S+>)?)\s*$
+            ^(?:Aborted login|Disconnected|Remote closed connection|Client has quit the connection)(?::(?: [^ \(]+)+)? \((?:auth failed, \d+ attempts(?: in \d+ secs)?|tried to use (?:disabled|disallowed) \S+ auth|proxy dest auth failed)\):(?: user=<<F-USER>[^>]*</F-USER>>,)?(?: method=\S+,)? rip=<HOST>(?:[^>]*(?:, session=<\S+>)?)\s*$
             ^pam\(\S+,<HOST>(?:,\S*)?\): pam_authenticate\(\) failed: (?:User not known to the underlying authentication module: \d+ Time\(s\)|Authentication failure \(password mismatch\?\)|Permission denied)\s*$
-            ^[a-z\-]{3,15}\(\S*,<HOST>(?:,\S*)?\): (?:unknown user|invalid credentials|Password mismatch)\s*$
+            ^[a-z\-]{3,15}\(\S*,<HOST>(?:,\S*)?\): (?:unknown user|invalid credentials|Password mismatch)
             <mdre-<mode>>
 
-mdre-aggressive = ^(?:Aborted login|Disconnected)(?::(?: [^ \(]+)+)? \((?:no auth attempts|disconnected before auth was ready,|client didn't finish \S+ auth,)(?: (?:in|waited) \d+ secs)?\):(?: user=<[^>]*>,)?(?: method=\S+,)? rip=<HOST>(?:[^>]*(?:, session=<\S+>)?)\s*$
+mdre-aggressive = ^(?:Aborted login|Disconnected|Remote closed connection|Client has quit the connection)(?::(?: [^ \(]+)+)? \((?:no auth attempts|disconnected before auth was ready,|client didn't finish \S+ auth,)(?: (?:in|waited) \d+ secs)?\):(?: user=<[^>]*>,)?(?: method=\S+,)? rip=<HOST>(?:[^>]*(?:, session=<\S+>)?)\s*$
 
 mdre-normal = 
 

config/filter.d/drupal-auth.conf

@@ -14,7 +14,7 @@ before = common.conf
 
 [Definition]
 
-failregex = ^%(__prefix_line)s(https?:\/\/)([\da-z\.-]+)\.([a-z\.]{2,6})(\/[\w\.-]+)*\|\d{10}\|user\|<HOST>\|.+\|.+\|\d\|.*\|Login attempt failed for .+\.$
+failregex = ^%(__prefix_line)s(?:https?:\/\/)[^|]+\|[^|]+\|[^|]+\|<ADDR>\|(?:[^|]*\|)*Login attempt failed (?:for|from) <F-USER>[^|]+</F-USER>\.$
 
 ignoreregex =
 

config/filter.d/gitlab.conf

@@ -0,0 +1,6 @@
+# Fail2Ban filter for Gitlab
+# Detecting unauthorized access to the Gitlab Web portal
+# typically logged in /var/log/gitlab/gitlab-rails/application.log
+
+[Definition]
+failregex = ^: Failed Login: username=<F-USER>.+</F-USER> ip=<HOST>$

config/filter.d/grafana.conf

@@ -0,0 +1,9 @@
+# Fail2Ban filter for Grafana
+# Detecting unauthorized access
+# Typically logged in /var/log/grafana/grafana.log
+
+[Init]
+datepattern = ^t=%%Y-%%m-%%dT%%H:%%M:%%S%%z
+
+[Definition]
+failregex = ^(?: lvl=err?or)? msg="Invalid username or password"(?: uname=(?:"<F-ALT_USER>[^"]+</F-ALT_USER>"|<F-USER>\S+</F-USER>)| error="<F-ERROR>[^"]+</F-ERROR>"| \S+=(?:\S*|"[^"]+"))* remote_addr=<ADDR>$

config/filter.d/guacamole.conf

@@ -5,21 +5,47 @@
 
 [Definition]
 
-# Option: failregex
+logging = catalina
-# Notes.: regex to match the password failures messages in the logfile.
+failregex = <L_<logging>/failregex>
-# Values: TEXT
+maxlines = <L_<logging>/maxlines>
-#
+datepattern = <L_<logging>/datepattern>
+
+[L_catalina]
+
 failregex = ^.*\nWARNING: Authentication attempt from <HOST> for user "[^"]*" failed\.$
 
-# Option:  ignoreregex
-# Notes.:  regex to ignore. If this regex matches, the line is ignored.
-# Values:  TEXT
-#
-ignoreregex = 
-
-# "maxlines" is number of log lines to buffer for multi-line regex searches
 maxlines = 2
 
 datepattern = ^%%b %%d, %%ExY %%I:%%M:%%S %%p
               ^WARNING:()**
-              {^LN-BEG}
+              {^LN-BEG}
+
+[L_webapp]
+
+failregex = ^ \[\S+\] WARN  \S+ - Authentication attempt from <HOST> for user "<F-USER>[^"]+</F-USER>" failed.
+
+maxlines = 1
+
+datepattern = ^%%H:%%M:%%S.%%f
+
+# DEV Notes:
+#
+# failregex is based on the default pattern given in Guacamole documentation :
+# https://guacamole.apache.org/doc/gug/configuring-guacamole.html#webapp-logging
+#
+# The following logback.xml Guacamole configuration file can then be used accordingly :
+# <configuration>
+#   <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
+#     <file>/var/log/guacamole.log</file>
+#     <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
+#       <fileNamePattern>/var/log/guacamole.%d.log.gz</fileNamePattern>
+#       <maxHistory>32</maxHistory>
+#     </rollingPolicy>
+#     <encoder>
+#       <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
+#     </encoder>
+#   </appender>
+#   <root level="info">
+#     <appender-ref ref="FILE" />
+#   </root>
+# </configuration>

config/filter.d/ignorecommands/apache-fakegooglebot

@@ -6,24 +6,35 @@
 #
 import sys
 from fail2ban.server.ipdns import DNSUtils, IPAddr
+from threading import Thread
 
 def process_args(argv):
-    if len(argv) != 2:
+    if len(argv) - 1 not in (1, 2):
-       raise ValueError("Please provide a single IP as an argument. Got: %s\n"
+       raise ValueError("Usage %s ip ?timeout?. Got: %s\n"
-                        % (argv[1:]))
+                        % (argv[0], argv[1:]))
     ip = argv[1]
 
     if not IPAddr(ip).isValid:
        raise ValueError("Argument must be a single valid IP. Got: %s\n"
                         % ip)
-    return ip
+    return argv[1:]
 
 google_ips = None
 
-def is_googlebot(ip):
+def is_googlebot(ip, timeout=55):
     import re
 
-    host = DNSUtils.ipToName(ip)
+    timeout = float(timeout or 0)
+    if timeout:
+      def ipToNameTO(host, ip, timeout):
+        host[0] = DNSUtils.ipToName(ip)
+      host = [None]
+      th = Thread(target=ipToNameTO, args=(host, ip, timeout)); th.daemon=True; th.start()
+      th.join(timeout)
+      host = host[0]
+    else:
+      host = DNSUtils.ipToName(ip)
+
     if not host or not re.match(r'.*\.google(bot)?\.com$', host):
        return False
     host_ips = DNSUtils.dnsToIp(host)
@@ -31,7 +42,7 @@ def is_googlebot(ip):
 
 if __name__ == '__main__': # pragma: no cover
     try:
-      ret = is_googlebot(process_args(sys.argv))
+      ret = is_googlebot(*process_args(sys.argv))
     except ValueError as e:
       sys.stderr.write(str(e))
       sys.exit(2)

config/filter.d/mssql-auth.conf

@@ -0,0 +1,15 @@
+# Fail2Ban filter for failed MSSQL Server authentication attempts
+
+[Definition]
+
+failregex = ^\s*Logon\s+Login failed for user '<F-USER>(?:[^']*|.*)</F-USER>'\. [^'\[]+\[CLIENT: <ADDR>\]$
+
+
+# DEV Notes:
+#	Tested with SQL Server 2019 on Ubuntu 18.04
+#
+# Example:
+#	2020-02-24 14:48:55.12 Logon       Login failed for user 'root'. Reason: Could not find a login matching the name provided. [CLIENT: 127.0.0.1]
+#
+# Author: Rüdiger Olschewsky
+#

config/filter.d/named-refused.conf

@@ -32,7 +32,7 @@ __daemon_combs_re=(?:%(__pid_re)s?:\s+%(__daemon_re)s|%(__daemon_re)s%(__pid_re)
 
 #       hostname       daemon_id         spaces
 # this can be optional (for instance if we match named native log files)
-__line_prefix=(?:\s\S+ %(__daemon_combs_re)s\s+)?
+__line_prefix=(?:\s*\S+ %(__daemon_combs_re)s\s+)?
 
 prefregex = ^%(__line_prefix)s(?: error:)?\s*client(?: @\S*)? <HOST>#\S+(?: \([\S.]+\))?: <F-CONTENT>.+</F-CONTENT>\s(?:denied|\(NOTAUTH\))\s*$
 

config/filter.d/nginx-bad-request.conf

@@ -0,0 +1,16 @@
+# Fail2Ban filter to match bad requests to nginx
+#
+
+[Definition]
+
+# The request often doesn't contain a method, only some encoded garbage
+# This will also match requests that are entirely empty
+failregex = ^<HOST> - \S+ \[\] "[^"]*" 400
+
+datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]%%f)?(?:\s*%%z)?
+              ^[^\[]*\[({DATE})
+              {^LN-BEG}
+
+journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx
+
+# Author: Jan Przybylak

config/filter.d/nginx-botsearch.conf

@@ -17,7 +17,9 @@ datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]
               ^[^\[]*\[({DATE})
               {^LN-BEG}
 
+journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx
+
 # DEV Notes:
 # Based on apache-botsearch filter
 # 
-# Author: Frantisek Sumsal
+# Author: Frantisek Sumsal

config/filter.d/nginx-http-auth.conf

@@ -3,15 +3,32 @@
 
 [Definition]
 
+mode = normal
 
-failregex = ^ \[error\] \d+#\d+: \*\d+ user "(?:[^"]+|.*?)":? (?:password mismatch|was not found in "[^\"]*"), client: <HOST>, server: \S*, request: "\S+ \S+ HTTP/\d+\.\d+", host: "\S+"(?:, referrer: "\S+")?\s*$
+mdre-auth = ^\s*\[error\] \d+#\d+: \*\d+ user "(?:[^"]+|.*?)":? (?:password mismatch|was not found in "[^\"]*"), client: <HOST>, server: \S*, request: "\S+ \S+ HTTP/\d+\.\d+", host: "\S+"(?:, referrer: "\S+")?\s*$
+mdre-fallback = ^\s*\[crit\] \d+#\d+: \*\d+ SSL_do_handshake\(\) failed \(SSL: error:\S+(?: \S+){1,3} too (?:long|short)\)[^,]*, client: <HOST>
+
+mdre-normal = %(mdre-auth)s
+mdre-aggressive = %(mdre-auth)s
+                  %(mdre-fallback)s
+
+failregex = <mdre-<mode>>
 
 ignoreregex = 
 
 datepattern = {^LN-BEG}
 
+journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx
+
 # DEV NOTES:
+# mdre-auth:
 # Based on samples in https://github.com/fail2ban/fail2ban/pull/43/files
 # Extensive search of all nginx auth failures not done yet.
 # 
 # Author: Daniel Black
+
+# mdre-fallback:
+# Ban people checking for TLS_FALLBACK_SCSV repeatedly
+# https://stackoverflow.com/questions/28010492/nginx-critical-error-with-ssl-handshaking/28010608#28010608
+# Author: Stephan Orlowsky
+

config/filter.d/nginx-limit-req.conf

@@ -44,3 +44,6 @@ failregex = ^\s*\[[a-z]+\] \d+#\d+: \*\d+ limiting requests, excess: [\d\.]+ by
 ignoreregex = 
 
 datepattern = {^LN-BEG}
+
+journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx
+

config/filter.d/nsd.conf

@@ -22,10 +22,10 @@ _daemon = nsd
 #          (?:::f{4,6}:)?(?P<host>[\w\-.^_]+)
 # Values:  TEXT
 
-failregex =  ^%(__prefix_line)sinfo: ratelimit block .* query <HOST> TYPE255$
+failregex =  ^%(__prefix_line)sinfo: ratelimit block .* query <ADDR> TYPE255$
-             ^%(__prefix_line)sinfo: .* <HOST> refused, no acl matches\.$
+             ^%(__prefix_line)sinfo: .* from(?: client)? <ADDR> refused, no acl matches\.?$
 
 ignoreregex =
 
 datepattern = {^LN-BEG}Epoch
-              {^LN-BEG}
+              {^LN-BEG}

config/filter.d/phpmyadmin-syslog.conf

@@ -1,4 +1,4 @@
-# Fail2Ban fitler for the phpMyAdmin-syslog
+# Fail2Ban filter for the phpMyAdmin-syslog
 #
 
 [INCLUDES]

config/filter.d/postfix.conf

@@ -37,7 +37,9 @@ mdre-rbl  = ^RCPT from [^[]*\[<HOST>\]%(_port)s: [45]54 [45]\.7\.1 Service unava
 mdpr-more = %(mdpr-normal)s
 mdre-more = %(mdre-normal)s
 
-mdpr-ddos = (?:lost connection after(?! DATA) [A-Z]+|disconnect(?= from \S+(?: \S+=\d+)* auth=0/(?:[1-9]|\d\d+)))
+# Includes some of the log messages described in
+# <http://www.postfix.org/POSTSCREEN_README.html>.
+mdpr-ddos = (?:lost connection after(?! DATA) [A-Z]+|disconnect(?= from \S+(?: \S+=\d+)* auth=0/(?:[1-9]|\d\d+))|(?:PREGREET \d+|HANGUP) after \S+)
 mdre-ddos = ^from [^[]*\[<HOST>\]%(_port)s:?
 
 mdpr-extra = (?:%(mdpr-auth)s|%(mdpr-normal)s)

config/filter.d/proftpd.conf

@@ -1,4 +1,4 @@
-# Fail2Ban fitler for the Proftpd FTP daemon
+# Fail2Ban filter for the Proftpd FTP daemon
 #
 # Set "UseReverseDNS off" in proftpd.conf to avoid the need for DNS.
 # See: http://www.proftpd.org/docs/howto/DNS.html
@@ -14,16 +14,15 @@ before = common.conf
 
 _daemon = proftpd
 
-__suffix_failed_login = (User not authorized for login|No such user found|Incorrect password|Password expired|Account disabled|Invalid shell: '\S+'|User in \S+|Limit (access|configuration) denies login|Not a UserAlias|maximum login length exceeded).?
+__suffix_failed_login = ([uU]ser not authorized for login|[nN]o such user found|[iI]ncorrect password|[pP]assword expired|[aA]ccount disabled|[iI]nvalid shell: '\S+'|[uU]ser in \S+|[lL]imit (access|configuration) denies login|[nN]ot a UserAlias|[mM]aximum login length exceeded)
 
 
-prefregex = ^%(__prefix_line)s%(__hostname)s \(\S+\[<HOST>\]\)[: -]+ <F-CONTENT>(?:USER|SECURITY|Maximum).+</F-CONTENT>$
+prefregex = ^%(__prefix_line)s%(__hostname)s \(\S+\[<HOST>\]\)[: -]+ <F-CONTENT>(?:USER|SECURITY|Maximum) .+</F-CONTENT>$
 
 
-failregex = ^USER .*: no such user found from \S+ \[\S+\] to \S+:\S+ *$
+failregex = ^USER <F-USER>\S+|.*?</F-USER>(?: \(Login failed\))?: %(__suffix_failed_login)s
-            ^USER .* \(Login failed\): %(__suffix_failed_login)s\s*$
+            ^SECURITY VIOLATION: <F-USER>\S+|.*?</F-USER> login attempted
-            ^SECURITY VIOLATION: .* login attempted\. *$
+            ^Maximum login attempts \(\d+\) exceeded
-            ^Maximum login attempts \(\d+\) exceeded *$
 
 ignoreregex = 
 

config/filter.d/scanlogd.conf

@@ -0,0 +1,17 @@
+# Fail2Ban filter for port scans detected by scanlogd
+
+[INCLUDES]
+
+# Read common prefixes. If any customizations available -- read them from
+# common.local
+before = common.conf
+
+[Definition]
+
+_daemon = scanlogd
+
+failregex =  ^%(__prefix_line)s<ADDR>(?::<F-PORT/>)? to \S+ ports\b
+
+ignoreregex =
+
+# Author: Mike Gabriel <mike.gabriel@das-netzwerkteam.de>

config/filter.d/sendmail-auth.conf

@@ -8,11 +8,14 @@ before = common.conf
 [Definition]
 
 _daemon = (?:sendmail|sm-(?:mta|acceptingconnections))
+# "\w{14,20}" will give support for IDs from 14 up to 20 characters long
 __prefix_line = %(known/__prefix_line)s(?:\w{14,20}: )?
+addr = (?:IPv6:<IP6>|<IP4>)
 
-# "w{14,20}" will give support for IDs from 14 up to 20 characters long
+prefregex = ^<F-MLFID>%(__prefix_line)s</F-MLFID><F-CONTENT>.+</F-CONTENT>$
-failregex = ^%(__prefix_line)s(\S+ )?\[(?:IPv6:<IP6>|<IP4>)\]( \(may be forged\))?: possible SMTP attack: command=AUTH, count=\d+$
 
+failregex = ^(\S+ )?\[%(addr)s\]( \(may be forged\))?: possible SMTP attack: command=AUTH, count=\d+$
+            ^AUTH failure \(LOGIN\):(?: [^:]+:)? authentication failure: checkpass failed, user=<F-USER>(?:\S+|.*?)</F-USER>, relay=(?:\S+ )?\[%(addr)s\](?: \(may be forged\))?$
 ignoreregex =
 
 journalmatch = _SYSTEMD_UNIT=sendmail.service

config/filter.d/sendmail-reject.conf

@@ -21,19 +21,20 @@ before = common.conf
 
 _daemon = (?:(sm-(mta|acceptingconnections)|sendmail))
 __prefix_line = %(known/__prefix_line)s(?:\w{14,20}: )?
+addr = (?:IPv6:<IP6>|<IP4>)
 
 prefregex = ^<F-MLFID>%(__prefix_line)s</F-MLFID><F-CONTENT>.+</F-CONTENT>$
 
-cmnfailre = ^ruleset=check_rcpt, arg1=(?P<email><\S+@\S+>), relay=(\S+ )?\[(?:IPv6:<IP6>|<IP4>)\](?: \(may be forged\))?, reject=(550 5\.7\.1 (?P=email)\.\.\. Relaying denied\. (IP name possibly forged \[(\d+\.){3}\d+\]|Proper authentication required\.|IP name lookup failed \[(\d+\.){3}\d+\])|553 5\.1\.8 (?P=email)\.\.\. Domain of sender address \S+ does not exist|550 5\.[71]\.1 (?P=email)\.\.\. (Rejected: .*|User unknown))$
+cmnfailre = ^ruleset=check_rcpt, arg1=(?P<email><\S+@\S+>), relay=(\S+ )?\[%(addr)s\](?: \(may be forged\))?, reject=(550 5\.7\.1 (?P=email)\.\.\. Relaying denied\. (IP name possibly forged \[(\d+\.){3}\d+\]|Proper authentication required\.|IP name lookup failed \[(\d+\.){3}\d+\])|553 5\.1\.8 (?P=email)\.\.\. Domain of sender address \S+ does not exist|550 5\.[71]\.1 (?P=email)\.\.\. (Rejected: .*|User unknown))$
-            ^ruleset=check_relay, arg1=(?P<dom>\S+), arg2=(?:IPv6:<IP6>|<IP4>), relay=((?P=dom) )?\[(\d+\.){3}\d+\](?: \(may be forged\))?, reject=421 4\.3\.2 (Connection rate limit exceeded\.|Too many open connections\.)$
+            ^ruleset=check_relay, arg1=(?P<dom>\S+), arg2=%(addr)s, relay=((?P=dom) )?\[(\d+\.){3}\d+\](?: \(may be forged\))?, reject=421 4\.3\.2 (Connection rate limit exceeded\.|Too many open connections\.)$
-            ^rejecting commands from (\S* )?\[(?:IPv6:<IP6>|<IP4>)\] due to pre-greeting traffic after \d+ seconds$
+            ^rejecting commands from (\S* )?\[%(addr)s\] due to pre-greeting traffic after \d+ seconds$
-            ^(?:\S+ )?\[(?:IPv6:<IP6>|<IP4>)\]: (?:(?i)expn|vrfy) \S+ \[rejected\]$
+            ^(?:\S+ )?\[%(addr)s\]: (?:(?i)expn|vrfy) \S+ \[rejected\]$
             ^<[^@]+@[^>]+>\.\.\. No such user here$
-            ^<F-NOFAIL>from=<[^@]+@[^>]+></F-NOFAIL>, size=\d+, class=\d+, nrcpts=\d+, bodytype=\w+, proto=E?SMTP, daemon=MTA, relay=\S+ \[(?:IPv6:<IP6>|<IP4>)\]$
+            ^<F-NOFAIL>from=<[^@]+@[^>]+></F-NOFAIL>, size=\d+, class=\d+, nrcpts=\d+, bodytype=\w+, proto=E?SMTP, daemon=MTA, relay=\S+ \[%(addr)s\]$
 
 mdre-normal =
 
-mdre-extra = ^(?:\S+ )?\[(?:IPv6:<IP6>|<IP4>)\](?: \(may be forged\))? did not issue (?:[A-Z]{4}[/ ]?)+during connection to (?:TLS)?M(?:TA|S[PA])(?:-\w+)?$
+mdre-extra = ^(?:\S+ )?\[%(addr)s\](?: \(may be forged\))? did not issue \S+ during connection
 
 mdre-aggressive = %(mdre-extra)s
 

config/filter.d/softethervpn.conf

@@ -0,0 +1,9 @@
+# Fail2Ban filter for SoftEtherVPN
+# Detecting unauthorized access to SoftEtherVPN
+# typically logged in /usr/local/vpnserver/security_log/*/sec.log, or in syslog, depending on configuration
+
+[INCLUDES]
+before = common.conf
+
+[Definition]
+failregex = ^%(__prefix_line)s(?:(?:\([\d\-]+ [\d:.]+\) )?<SECURITY_LOG>: )?Connection "[^"]+": User authentication failed. The user name that has been provided was "<F-USER>(?:[^"]+|.+)</F-USER>", from <ADDR>\.$

config/filter.d/sshd.conf

@@ -25,7 +25,7 @@ __pref = (?:(?:error|fatal): (?:PAM: )?)?
 __suff = (?: (?:port \d+|on \S+|\[preauth\])){0,3}\s*
 __on_port_opt = (?: (?:port \d+|on \S+)){0,2}
 # close by authenticating user:
-__authng_user = (?: (?:invalid|authenticating) user <F-USER>\S+|.+?</F-USER>)?
+__authng_user = (?: (?:invalid|authenticating) user <F-USER>\S+|.*?</F-USER>)?
 
 # for all possible (also future) forms of "no matching (cipher|mac|MAC|compression method|key exchange method|host key type) found",
 # see ssherr.c for all possible SSH_ERR_..._ALG_MATCH errors.
@@ -44,18 +44,18 @@ cmnfailre = ^[aA]uthentication (?:failure|error|failed) for <F-USER>.*</F-USER>
             ^Failed <cmnfailed> for (?P<cond_inv>invalid user )?<F-USER>(?P<cond_user>\S+)|(?(cond_inv)(?:(?! from ).)*?|[^:]+)</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$)
             ^<F-USER>ROOT</F-USER> LOGIN REFUSED FROM <HOST>
             ^[iI](?:llegal|nvalid) user <F-USER>.*?</F-USER> from <HOST>%(__suff)s$
-            ^User <F-USER>.+</F-USER> from <HOST> not allowed because not listed in AllowUsers%(__suff)s$
+            ^User <F-USER>\S+|.*?</F-USER> from <HOST> not allowed because not listed in AllowUsers%(__suff)s$
-            ^User <F-USER>.+</F-USER> from <HOST> not allowed because listed in DenyUsers%(__suff)s$
+            ^User <F-USER>\S+|.*?</F-USER> from <HOST> not allowed because listed in DenyUsers%(__suff)s$
-            ^User <F-USER>.+</F-USER> from <HOST> not allowed because not in any group%(__suff)s$
+            ^User <F-USER>\S+|.*?</F-USER> from <HOST> not allowed because not in any group%(__suff)s$
             ^refused connect from \S+ \(<HOST>\)
             ^Received <F-MLFFORGET>disconnect</F-MLFFORGET> from <HOST>%(__on_port_opt)s:\s*3: .*: Auth fail%(__suff)s$
-            ^User <F-USER>.+</F-USER> from <HOST> not allowed because a group is listed in DenyGroups%(__suff)s$
+            ^User <F-USER>\S+|.*?</F-USER> from <HOST> not allowed because a group is listed in DenyGroups%(__suff)s$
-            ^User <F-USER>.+</F-USER> from <HOST> not allowed because none of user's groups are listed in AllowGroups%(__suff)s$
+            ^User <F-USER>\S+|.*?</F-USER> from <HOST> not allowed because none of user's groups are listed in AllowGroups%(__suff)s$
             ^<F-NOFAIL>%(__pam_auth)s\(sshd:auth\):\s+authentication failure;</F-NOFAIL>(?:\s+(?:(?:logname|e?uid|tty)=\S*)){0,4}\s+ruser=<F-ALT_USER>\S*</F-ALT_USER>\s+rhost=<HOST>(?:\s+user=<F-USER>\S*</F-USER>)?%(__suff)s$
             ^maximum authentication attempts exceeded for <F-USER>.*</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)?%(__suff)s$
-            ^User <F-USER>.+</F-USER> not allowed because account is locked%(__suff)s
+            ^User <F-USER>\S+|.*?</F-USER> not allowed because account is locked%(__suff)s
             ^<F-MLFFORGET>Disconnecting</F-MLFFORGET>(?: from)?(?: (?:invalid|authenticating)) user <F-USER>\S+</F-USER> <HOST>%(__on_port_opt)s:\s*Change of username or service not allowed:\s*.*\[preauth\]\s*$
-            ^Disconnecting: Too many authentication failures(?: for <F-USER>.+?</F-USER>)?%(__suff)s$
+            ^Disconnecting: Too many authentication failures(?: for <F-USER>\S+|.*?</F-USER>)?%(__suff)s$
             ^<F-NOFAIL>Received <F-MLFFORGET>disconnect</F-MLFFORGET></F-NOFAIL> from <HOST>%(__on_port_opt)s:\s*11:
             <mdre-<mode>-other>
             ^<F-MLFFORGET><F-MLFGAINED>Accepted \w+</F-MLFGAINED></F-MLFFORGET> for <F-USER>\S+</F-USER> from <HOST>(?:\s|$)
@@ -71,15 +71,14 @@ mdre-normal =
 mdre-normal-other = ^<F-NOFAIL><F-MLFFORGET>(Connection closed|Disconnected)</F-MLFFORGET></F-NOFAIL> (?:by|from)%(__authng_user)s <HOST>(?:%(__suff)s|\s*)$
 
 mdre-ddos = ^Did not receive identification string from <HOST>
-            ^kex_exchange_identification: client sent invalid protocol identifier
+            ^kex_exchange_identification: (?:[Cc]lient sent invalid protocol identifier|[Cc]onnection closed by remote host)
             ^Bad protocol version identification '.*' from <HOST>
-            ^Connection <F-MLFFORGET>reset</F-MLFFORGET> by <HOST>
             ^<F-NOFAIL>SSH: Server;Ltype:</F-NOFAIL> (?:Authname|Version|Kex);Remote: <HOST>-\d+;[A-Z]\w+:
             ^Read from socket failed: Connection <F-MLFFORGET>reset</F-MLFFORGET> by peer
 # same as mdre-normal-other, but as failure (without <F-NOFAIL>) and [preauth] only:
-mdre-ddos-other = ^<F-MLFFORGET>(Connection closed|Disconnected)</F-MLFFORGET> (?:by|from)%(__authng_user)s <HOST>%(__on_port_opt)s\s+\[preauth\]\s*$
+mdre-ddos-other = ^<F-MLFFORGET>(Connection (?:closed|reset)|Disconnected)</F-MLFFORGET> (?:by|from)%(__authng_user)s <HOST>%(__on_port_opt)s\s+\[preauth\]\s*$
 
-mdre-extra = ^Received <F-MLFFORGET>disconnect</F-MLFFORGET> from <HOST>%(__on_port_opt)s:\s*14: No supported authentication methods available
+mdre-extra = ^Received <F-MLFFORGET>disconnect</F-MLFFORGET> from <HOST>%(__on_port_opt)s:\s*14: No(?: supported)? authentication methods available
             ^Unable to negotiate with <HOST>%(__on_port_opt)s: no matching <__alg_match> found.
             ^Unable to negotiate a <__alg_match>
             ^no matching <__alg_match> found:

config/filter.d/traefik-auth.conf

@@ -51,6 +51,26 @@
 
 [Definition]
 
-failregex = ^<HOST> \- (?!- )\S+ \[\] \"(GET|POST|HEAD) [^\"]+\" 401\b
+# Parameter "method" can be used to specifiy request method
+req-method = \S+
+# Usage example (for jail.local):
+#   filter = traefik-auth[req-method="GET|POST|HEAD"]
+
+failregex = ^<HOST> \- <usrre-<mode>> \[\] \"(?:<req-method>) [^\"]+\" 401\b
 
 ignoreregex =
+
+# Parameter "mode": normal (default), ddos or aggressive
+# Usage example (for jail.local):
+#   [traefik-auth]
+#   mode = aggressive
+#   # or another jail (rewrite filter parameters of jail):
+#   [traefik-auth-ddos]
+#   filter = traefik-auth[mode=ddos]
+#
+mode = normal
+
+# part of failregex matches user name (must be available in normal mode, must be empty in ddos mode, and both for aggressive mode):
+usrre-normal = (?!- )<F-USER>\S+</F-USER>
+usrre-ddos = -
+usrre-aggressive = <F-USER>\S+</F-USER>

config/jail.conf

@@ -52,7 +52,7 @@ before = paths-debian.conf
 # to prevent "clever" botnets calculate exact time IP can be unbanned again:
 #bantime.rndtime = 
 
-# "bantime.maxtime" is the max number of seconds using the ban time can reach (don't grows further)
+# "bantime.maxtime" is the max number of seconds using the ban time can reach (doesn't grow further)
 #bantime.maxtime = 
 
 # "bantime.factor" is a coefficient to calculate exponent growing of the formula or common multiplier,
@@ -60,14 +60,14 @@ before = paths-debian.conf
 # grows by 1, 2, 4, 8, 16 ...
 #bantime.factor = 1
 
-# "bantime.formula" used by default to calculate next value of ban time, default value bellow,
+# "bantime.formula" used by default to calculate next value of ban time, default value below,
 # the same ban time growing will be reached by multipliers 1, 2, 4, 8, 16, 32...
 #bantime.formula = ban.Time * (1<<(ban.Count if ban.Count<20 else 20)) * banFactor
 #
 # more aggressive example of formula has the same values only for factor "2.0 / 2.885385" :
 #bantime.formula = ban.Time * math.exp(float(ban.Count+1)*banFactor)/math.exp(1*banFactor)
 
-# "bantime.multipliers" used to calculate next value of ban time instead of formula, coresponding 
+# "bantime.multipliers" used to calculate next value of ban time instead of formula, corresponding
 # previously ban count and given "bantime.factor" (for multipliers default is 1);
 # following example grows ban time by 1, 2, 4, 8, 16 ... and if last ban count greater as multipliers count, 
 # always used last multiplier (64 in example), for factor '1' and original ban time 600 - 10.6 hours
@@ -77,7 +77,7 @@ before = paths-debian.conf
 #bantime.multipliers = 1 5 30 60 300 720 1440 2880
 
 # "bantime.overalljails" (if true) specifies the search of IP in the database will be executed 
-# cross over all jails, if false (dafault), only current jail of the ban IP will be searched
+# cross over all jails, if false (default), only current jail of the ban IP will be searched
 #bantime.overalljails = false
 
 # --------------------
@@ -209,28 +209,37 @@ banaction = iptables-multiport
 banaction_allports = iptables-allports
 
 # The simplest action to take: ban only
-action_ = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
+action_ = %(banaction)s[port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
 
 # ban & send an e-mail with whois report to the destemail.
-action_mw = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
+action_mw = %(action_)s
-            %(mta)s-whois[name=%(__name__)s, sender="%(sender)s", dest="%(destemail)s", protocol="%(protocol)s", chain="%(chain)s"]
+            %(mta)s-whois[sender="%(sender)s", dest="%(destemail)s", protocol="%(protocol)s", chain="%(chain)s"]
 
 # ban & send an e-mail with whois report and relevant log lines
 # to the destemail.
-action_mwl = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
+action_mwl = %(action_)s
-             %(mta)s-whois-lines[name=%(__name__)s, sender="%(sender)s", dest="%(destemail)s", logpath="%(logpath)s", chain="%(chain)s"]
+             %(mta)s-whois-lines[sender="%(sender)s", dest="%(destemail)s", logpath="%(logpath)s", chain="%(chain)s"]
 
 # See the IMPORTANT note in action.d/xarf-login-attack for when to use this action
 #
 # ban & send a xarf e-mail to abuse contact of IP address and include relevant log lines
 # to the destemail.
-action_xarf = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
+action_xarf = %(action_)s
              xarf-login-attack[service=%(__name__)s, sender="%(sender)s", logpath="%(logpath)s", port="%(port)s"]
 
+# ban & send a notification to one or more of the 50+ services supported by Apprise.
+# See https://github.com/caronc/apprise/wiki for details on what is supported.
+#
+# You may optionally over-ride the default configuration line (containing the Apprise URLs)
+# by using 'apprise[config="/alternate/path/to/apprise.cfg"]' otherwise
+# /etc/fail2ban/apprise.conf is sourced for your supported notification configuration.
+# action = %(action_)s
+#          apprise
+
 # ban IP on CloudFlare & send an e-mail with whois report and relevant log lines
 # to the destemail.
 action_cf_mwl = cloudflare[cfuser="%(cfemail)s", cftoken="%(cfapikey)s"]
-                %(mta)s-whois-lines[name=%(__name__)s, sender="%(sender)s", dest="%(destemail)s", logpath="%(logpath)s", chain="%(chain)s"]
+                %(mta)s-whois-lines[sender="%(sender)s", dest="%(destemail)s", logpath="%(logpath)s", chain="%(chain)s"]
 
 # Report block via blocklist.de fail2ban reporting service API
 # 
@@ -242,20 +251,6 @@ action_cf_mwl = cloudflare[cfuser="%(cfemail)s", cftoken="%(cfapikey)s"]
 #
 action_blocklist_de  = blocklist_de[email="%(sender)s", service="%(__name__)s", apikey="%(blocklist_de_apikey)s", agent="%(fail2ban_agent)s"]
 
-# Report ban via badips.com, and use as blacklist
-#
-# See BadIPsAction docstring in config/action.d/badips.py for
-# documentation for this action.
-#
-# NOTE: This action relies on banaction being present on start and therefore
-# should be last action defined for a jail.
-#
-action_badips = badips.py[category="%(__name__)s", banaction="%(banaction)s", agent="%(fail2ban_agent)s"]
-#
-# Report ban via badips.com (uses action.d/badips.conf for reporting only)
-#
-action_badips_report = badips[category="%(__name__)s", agent="%(fail2ban_agent)s"]
-
 # Report ban via abuseipdb.com.
 #
 # See action.d/abuseipdb.conf for usage example and details.
@@ -371,12 +366,15 @@ maxretry = 1
 [openhab-auth]
 
 filter = openhab
-action = iptables-allports[name=NoAuthFailures]
+banaction = %(banaction_allports)s
 logpath = /opt/openhab/logs/request.log
 
 
+# To use more aggressive http-auth modes set filter parameter "mode" in jail.local:
+# normal (default), aggressive (combines all), auth or fallback
+# See "tests/files/logs/nginx-http-auth" or "filter.d/nginx-http-auth.conf" for usage example and details.
 [nginx-http-auth]
-
+# mode = normal
 port    = http,https
 logpath = %(nginx_error_log)s
 
@@ -392,8 +390,10 @@ logpath = %(nginx_error_log)s
 
 port     = http,https
 logpath  = %(nginx_error_log)s
-maxretry = 2
 
+[nginx-bad-request]
+port    = http,https
+logpath = %(nginx_access_log)s
 
 # Ban attackers that try to use PHP's URL-fopen() functionality
 # through GET/POST variables. - Experimental, with more than a year
@@ -478,6 +478,7 @@ backend  = %(syslog_backend)s
 
 port     = http,https
 logpath  = /var/log/tomcat*/catalina.out
+#logpath  = /var/log/guacamole.log
 
 [monit]
 #Ban clients brute-forcing the monit gui login
@@ -744,8 +745,8 @@ logpath  = /var/log/named/security.log
 [nsd]
 
 port     = 53
-action   = %(banaction)s[name=%(__name__)s-tcp, port="%(port)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
+action_  = %(default/action_)s[name=%(__name__)s-tcp, protocol="tcp"]
-           %(banaction)s[name=%(__name__)s-udp, port="%(port)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]
+           %(default/action_)s[name=%(__name__)s-udp, protocol="udp"]
 logpath = /var/log/nsd.log
 
 
@@ -756,9 +757,8 @@ logpath = /var/log/nsd.log
 [asterisk]
 
 port     = 5060,5061
-action   = %(banaction)s[name=%(__name__)s-tcp, port="%(port)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
+action_  = %(default/action_)s[name=%(__name__)s-tcp, protocol="tcp"]
-           %(banaction)s[name=%(__name__)s-udp, port="%(port)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]
+           %(default/action_)s[name=%(__name__)s-udp, protocol="udp"]
-           %(mta)s-whois[name=%(__name__)s, dest="%(destemail)s"]
 logpath  = /var/log/asterisk/messages
 maxretry = 10
 
@@ -766,9 +766,8 @@ maxretry = 10
 [freeswitch]
 
 port     = 5060,5061
-action   = %(banaction)s[name=%(__name__)s-tcp, port="%(port)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
+action_  = %(default/action_)s[name=%(__name__)s-tcp, protocol="tcp"]
-           %(banaction)s[name=%(__name__)s-udp, port="%(port)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]
+           %(default/action_)s[name=%(__name__)s-udp, protocol="udp"]
-           %(mta)s-whois[name=%(__name__)s, dest="%(destemail)s"]
 logpath  = /var/log/freeswitch.log
 maxretry = 10
 
@@ -798,6 +797,14 @@ logpath  = %(mysql_log)s
 backend  = %(mysql_backend)s
 
 
+[mssql-auth]
+# Default configuration for Microsoft SQL Server for Linux
+# See the 'mssql-conf' manpage how to change logpath or port
+logpath = /var/opt/mssql/log/errorlog
+port = 1433
+filter = mssql-auth
+
+
 # Log wrong MongoDB auth (for details see filter 'filter.d/mongodb-auth.conf')
 [mongodb-auth]
 # change port when running with "--shardsvr" or "--configsvr" runtime operation
@@ -853,11 +860,23 @@ logpath = /var/log/ejabberd/ejabberd.log
 [counter-strike]
 
 logpath = /opt/cstrike/logs/L[0-9]*.log
-# Firewall: http://www.cstrike-planet.com/faq/6
 tcpport = 27030,27031,27032,27033,27034,27035,27036,27037,27038,27039
 udpport = 1200,27000,27001,27002,27003,27004,27005,27006,27007,27008,27009,27010,27011,27012,27013,27014,27015
-action  = %(banaction)s[name=%(__name__)s-tcp, port="%(tcpport)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
+action_  = %(default/action_)s[name=%(__name__)s-tcp, port="%(tcpport)s", protocol="tcp"]
-           %(banaction)s[name=%(__name__)s-udp, port="%(udpport)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]
+           %(default/action_)s[name=%(__name__)s-udp, port="%(udpport)s", protocol="udp"]
+
+[softethervpn]
+port     = 500,4500
+protocol = udp
+logpath  = /usr/local/vpnserver/security_log/*/sec.log
+
+[gitlab]
+port    = http,https
+logpath = /var/log/gitlab/gitlab-rails/application.log
+
+[grafana]
+port    = http,https
+logpath = /var/log/grafana/grafana.log
 
 [bitwarden]
 port    = http,https
@@ -909,8 +928,8 @@ findtime     = 1
 [murmur]
 # AKA mumble-server
 port     = 64738
-action   = %(banaction)s[name=%(__name__)s-tcp, port="%(port)s", protocol=tcp, chain="%(chain)s", actname=%(banaction)s-tcp]
+action_  = %(default/action_)s[name=%(__name__)s-tcp, protocol="tcp"]
-           %(banaction)s[name=%(__name__)s-udp, port="%(port)s", protocol=udp, chain="%(chain)s", actname=%(banaction)s-udp]
+           %(default/action_)s[name=%(__name__)s-udp, protocol="udp"]
 logpath  = /var/log/mumble-server/mumble-server.log
 
 
@@ -952,6 +971,9 @@ logpath = %(apache_error_log)s
 port    = http,https
 logpath = /var/log/traefik/access.log
 
+[scanlogd]
+logpath = %(syslog_local0)s
+banaction = %(banaction_allports)s
 
 [monitorix]
 port	= 8080

fail2ban/client/actionreader.py

@@ -38,28 +38,32 @@ class ActionReader(DefinitionInitConfigReader):
 
 	_configOpts = {
 		"actionstart": ["string", None],
-		"actionstart_on_demand": ["string", None],
+		"actionstart_on_demand": ["bool", None],
 		"actionstop": ["string", None],
 		"actionflush": ["string", None],
 		"actionreload": ["string", None],
 		"actioncheck": ["string", None],
 		"actionrepair": ["string", None],
-		"actionrepair_on_unban": ["string", None],
+		"actionrepair_on_unban": ["bool", None],
 		"actionban": ["string", None],
 		"actionprolong": ["string", None],
 		"actionreban": ["string", None],
 		"actionunban": ["string", None],
-		"norestored": ["string", None],
+		"norestored": ["bool", None],
 	}
 
 	def __init__(self, file_, jailName, initOpts, **kwargs):
+		# always supply jail name as name parameter if not specified in options:
+		n = initOpts.get("name")
+		if n is None:
+			initOpts["name"] = n = jailName
 		actname = initOpts.get("actname")
 		if actname is None:
 			actname = file_
+			# ensure we've unique action name per jail:
+			if n != jailName:
+				actname += n[len(jailName):] if n.startswith(jailName) else '-' + n
 			initOpts["actname"] = actname
-		# always supply jail name as name parameter if not specified in options:
-		if initOpts.get("name") is None:
-			initOpts["name"] = jailName
 		self._name = actname
 		DefinitionInitConfigReader.__init__(
 			self, file_, jailName, initOpts, **kwargs)
@@ -80,11 +84,6 @@ class ActionReader(DefinitionInitConfigReader):
 	def convert(self):
 		opts = self.getCombined(
 			ignore=CommandAction._escapedTags | set(('timeout', 'bantime')))
-		# type-convert only after combined (otherwise boolean converting prevents substitution):
-		for o in ('norestored', 'actionstart_on_demand', 'actionrepair_on_unban'):
-			if opts.get(o):
-				opts[o] = self._convert_to_boolean(opts[o])
-		
 		# stream-convert:
 		head = ["set", self._jailName]
 		stream = list()

fail2ban/client/configparserinc.py

@@ -29,7 +29,7 @@ import re
 import sys
 from ..helpers import getLogger
 
-if sys.version_info >= (3,2):
+if sys.version_info >= (3,): # pragma: 2.x no cover
 
 	# SafeConfigParser deprecated from Python 3.2 (renamed to ConfigParser)
 	from configparser import ConfigParser as SafeConfigParser, BasicInterpolation, \
@@ -61,7 +61,7 @@ if sys.version_info >= (3,2):
 				return super(BasicInterpolationWithName, self)._interpolate_some(
 					parser, option, accum, rest, section, map, *args, **kwargs)
 
-else: # pragma: no cover
+else: # pragma: 3.x no cover
 	from ConfigParser import SafeConfigParser, \
 		InterpolationMissingOptionError, NoOptionError, NoSectionError
 
@@ -372,7 +372,8 @@ after = 1.conf
 						s2 = alls.get(n)
 						if isinstance(s2, dict):
 							# save previous known values, for possible using in local interpolations later:
-							self.merge_section('KNOWN/'+n, s2, '')
+							self.merge_section('KNOWN/'+n, 
+								dict(filter(lambda i: i[0] in s, s2.iteritems())), '')
 							# merge section
 							s2.update(s)
 						else:

fail2ban/client/configreader.py

@@ -34,6 +34,30 @@ from ..helpers import getLogger, _as_bool, _merge_dicts, substituteRecursiveTags
 # Gets the instance of the logger.
 logSys = getLogger(__name__)
 
+CONVERTER = {
+	"bool": _as_bool,
+	"int": int,
+}
+def _OptionsTemplateGen(options):
+	"""Iterator over the options template with default options.
+	
+	Each options entry is composed of an array or tuple with:
+		[[type, name, ?default?], ...]
+	Or it is a dict:
+		{name: [type, default], ...}
+	"""
+	if isinstance(options, (list,tuple)):
+		for optname in options:
+			if len(optname) > 2:
+				opttype, optname, optvalue = optname
+			else:
+				(opttype, optname), optvalue = optname, None
+			yield opttype, optname, optvalue
+	else:
+		for optname in options:
+			opttype, optvalue = options[optname]
+			yield opttype, optname, optvalue
+
 
 class ConfigReader():
 	"""Generic config reader class.
@@ -228,31 +252,22 @@ class ConfigReaderUnshared(SafeConfigParserWithIncludes):
 	# Or it is a dict:
 	#  {name: [type, default], ...}
 	
-	def getOptions(self, sec, options, pOptions=None, shouldExist=False):
+	def getOptions(self, sec, options, pOptions=None, shouldExist=False, convert=True):
 		values = dict()
 		if pOptions is None:
 			pOptions = {}
 		# Get only specified options:
-		for optname in options:
+		for opttype, optname, optvalue in _OptionsTemplateGen(options):
-			if isinstance(options, (list,tuple)):
-				if len(optname) > 2:
-					opttype, optname, optvalue = optname
-				else:
-					(opttype, optname), optvalue = optname, None
-			else:
-				opttype, optvalue = options[optname]
 			if optname in pOptions:
 				continue
 			try:
-				if opttype == "bool":
+				v = self.get(sec, optname, vars=pOptions)
-					v = self.getboolean(sec, optname)
-					if v is None: continue
-				elif opttype == "int":
-					v = self.getint(sec, optname)
-					if v is None: continue
-				else:
-					v = self.get(sec, optname, vars=pOptions)
 				values[optname] = v
+				if convert:
+					conv = CONVERTER.get(opttype)
+					if conv:
+						if v is None: continue
+						values[optname] = conv(v)
 			except NoSectionError as e:
 				if shouldExist:
 					raise
@@ -324,8 +339,9 @@ class DefinitionInitConfigReader(ConfigReader):
 			pOpts = dict()
 		if self._initOpts:
 			pOpts = _merge_dicts(pOpts, self._initOpts)
+		# type-convert only in combined (otherwise int/bool converting prevents substitution):
 		self._opts = ConfigReader.getOptions(
-			self, "Definition", self._configOpts, pOpts)
+			self, "Definition", self._configOpts, pOpts, convert=False)
 		self._pOpts = pOpts
 		if self.has_section("Init"):
 			# get only own options (without options from default):
@@ -346,10 +362,21 @@ class DefinitionInitConfigReader(ConfigReader):
 				if opt == '__name__' or opt in self._opts: continue
 				self._opts[opt] = self.get("Definition", opt)
 
+	def convertOptions(self, opts, configOpts):
+		"""Convert interpolated combined options to expected type.
+		"""
+		for opttype, optname, optvalue in _OptionsTemplateGen(configOpts):
+			conv = CONVERTER.get(opttype)
+			if conv:
+				v = opts.get(optname)
+				if v is None: continue
+				try:
+					opts[optname] = conv(v)
+				except ValueError:
+					logSys.warning("Wrong %s value %r for %r. Using default one: %r",
+						opttype, v, optname, optvalue)
+					opts[optname] = optvalue
 
-	def _convert_to_boolean(self, value):
-		return _as_bool(value)
-	
 	def getCombOption(self, optname):
 		"""Get combined definition option (as string) using pre-set and init
 		options as preselection (values with higher precedence as specified in section).
@@ -384,6 +411,8 @@ class DefinitionInitConfigReader(ConfigReader):
 			ignore=ignore, addrepl=self.getCombOption)
 		if not opts:
 			raise ValueError('recursive tag definitions unable to be resolved')
+		# convert options after all interpolations:
+		self.convertOptions(opts, self._configOpts)
 		return opts
 	
 	def convert(self):

fail2ban/client/csocket.py

@@ -48,7 +48,8 @@ class CSocket:
 	def send(self, msg, nonblocking=False, timeout=None):
 		# Convert every list member to string
 		obj = dumps(map(CSocket.convert, msg), HIGHEST_PROTOCOL)
-		self.__csock.send(obj + CSPROTO.END)
+		self.__csock.send(obj)
+		self.__csock.send(CSPROTO.END)
 		return self.receive(self.__csock, nonblocking, timeout)
 
 	def settimeout(self, timeout):
@@ -81,9 +82,12 @@ class CSocket:
 		msg = CSPROTO.EMPTY
 		if nonblocking: sock.setblocking(0)
 		if timeout: sock.settimeout(timeout)
-		while msg.rfind(CSPROTO.END) == -1:
+		bufsize = 1024
-			chunk = sock.recv(512)
+		while msg.rfind(CSPROTO.END, -32) == -1:
-			if chunk in ('', b''): # python 3.x may return b'' instead of ''
+			chunk = sock.recv(bufsize)
-				raise RuntimeError("socket connection broken")
+			if not len(chunk):
+				raise socket.error(104, 'Connection reset by peer')
+			if chunk == CSPROTO.END: break
 			msg = msg + chunk
+			if bufsize < 32768: bufsize <<= 1
 		return loads(msg)

fail2ban/client/fail2banclient.py

@@ -168,19 +168,6 @@ class Fail2banClient(Fail2banCmdLine, Thread):
 		if not ret:
 			return None
 
-		# verify that directory for the socket file exists
-		socket_dir = os.path.dirname(self._conf["socket"])
-		if not os.path.exists(socket_dir):
-			logSys.error(
-				"There is no directory %s to contain the socket file %s."
-				% (socket_dir, self._conf["socket"]))
-			return None
-		if not os.access(socket_dir, os.W_OK | os.X_OK): # pragma: no cover
-			logSys.error(
-				"Directory %s exists but not accessible for writing"
-				% (socket_dir,))
-			return None
-
 		# Check already running
 		if not self._conf["force"] and os.path.exists(self._conf["socket"]):
 			logSys.error("Fail2ban seems to be in unexpected state (not running but the socket exists)")
@@ -243,7 +230,7 @@ class Fail2banClient(Fail2banCmdLine, Thread):
 			logSys.log(5, '  client phase %s', phase)
 		if not stream:
 			return False
-		# wait a litle bit for phase "start-ready" before enter active waiting:
+		# wait a little bit for phase "start-ready" before enter active waiting:
 		if phase is not None:
 			Utils.wait_for(lambda: phase.get('start-ready', None) is not None, 0.5, 0.001)
 			phase['configure'] = (True if stream else False)

fail2ban/client/fail2bancmdline.py

@@ -27,13 +27,17 @@ import sys
 
 from ..version import version, normVersion
 from ..protocol import printFormatted
-from ..helpers import getLogger, str2LogLevel, getVerbosityFormat
+from ..helpers import getLogger, str2LogLevel, getVerbosityFormat, BrokenPipeError
 
 # Gets the instance of the logger.
 logSys = getLogger("fail2ban")
 
 def output(s): # pragma: no cover
-	print(s)
+	try:
+		print(s)
+	except (BrokenPipeError, IOError) as e: # pragma: no cover
+		if e.errno != 32: # closed / broken pipe
+			raise
 
 # Config parameters required to start fail2ban which can be also set via command line (overwrite fail2ban.conf),
 CONFIG_PARAMS = ("socket", "pidfile", "logtarget", "loglevel", "syslogsocket")
@@ -188,7 +192,7 @@ class Fail2banCmdLine():
 				cmdOpts = 'hc:s:p:xfbdtviqV'
 				cmdLongOpts = ['loglevel=', 'logtarget=', 'syslogsocket=', 'test', 'async',
 					'conf=', 'pidfile=', 'pname=', 'socket=',
-					'timeout=', 'str2sec=', 'help', 'version', 'dp', '--dump-pretty']
+					'timeout=', 'str2sec=', 'help', 'version', 'dp', 'dump-pretty']
 				optList, self._args = getopt.getopt(self._argv[1:], cmdOpts, cmdLongOpts)
 			except getopt.GetoptError:
 				self.dispUsage()
@@ -308,18 +312,24 @@ class Fail2banCmdLine():
 	# since method is also exposed in API via globally bound variable
 	@staticmethod
 	def _exit(code=0):
-		if hasattr(os, '_exit') and os._exit:
+		# implicit flush without to produce broken pipe error (32):
-			os._exit(code)
+		sys.stderr.close()
-		else:
+		try:
-			sys.exit(code)
+			sys.stdout.flush()
+			# exit:
+			if hasattr(sys, 'exit') and sys.exit:
+				sys.exit(code)
+			else:
+				os._exit(code)
+		except (BrokenPipeError, IOError) as e: # pragma: no cover
+			if e.errno != 32: # closed / broken pipe
+				raise
 
 	@staticmethod
 	def exit(code=0):
 		logSys.debug("Exit with code %s", code)
 		# because of possible buffered output in python, we should flush it before exit:
 		logging.shutdown()
-		sys.stdout.flush()
-		sys.stderr.flush()
 		# exit
 		Fail2banCmdLine._exit(code)
 

fail2ban/client/fail2banreader.py

@@ -53,6 +53,7 @@ class Fail2banReader(ConfigReader):
 		opts = [["string", "loglevel", "INFO" ],
 				["string", "logtarget", "STDERR"],
 				["string", "syslogsocket", "auto"],
+				["string", "allowipv6", "auto"],
 				["string", "dbfile", "/var/lib/fail2ban/fail2ban.sqlite3"],
 				["int",    "dbmaxmatches", None],
 				["string", "dbpurgeage", "1d"]]
@@ -74,6 +75,7 @@ class Fail2banReader(ConfigReader):
 		# Also dbfile should be set before all other database options.
 		# So adding order indices into items, to be stripped after sorting, upon return
 		order = {"thread":0, "syslogsocket":11, "loglevel":12, "logtarget":13,
+			"allowipv6": 14,
 			"dbfile":50, "dbmaxmatches":51, "dbpurgeage":51}
 		stream = list()
 		for opt in self.__opts:

fail2ban/client/fail2banregex.py

@@ -21,7 +21,6 @@ Fail2Ban  reads log file that contains password failure report
 and bans the corresponding IP addresses using firewall rules.
 
 This tools can test regular expressions for "fail2ban".
-
 """
 
 __author__ = "Fail2Ban Developers"
@@ -36,11 +35,11 @@ __license__ = "GPL"
 
 import getopt
 import logging
+import re
 import os
 import shlex
 import sys
 import time
-import time
 import urllib
 from optparse import OptionParser, Option
 
@@ -53,7 +52,7 @@ except ImportError:
 
 from ..version import version, normVersion
 from .filterreader import FilterReader
-from ..server.filter import Filter, FileContainer
+from ..server.filter import Filter, FileContainer, MyTime
 from ..server.failregex import Regex, RegexException
 
 from ..helpers import str2LogLevel, getVerbosityFormat, FormatterWithTraceBack, getLogger, \
@@ -109,19 +108,22 @@ class _f2bOptParser(OptionParser):
 	def format_help(self, *args, **kwargs):
 		""" Overwritten format helper with full ussage."""
 		self.usage = ''
-		return "Usage: " + usage() + __doc__ + """
+		return "Usage: " + usage() + "\n" + __doc__ + """
 LOG:
-    string                  a string representing a log line
+  string                a string representing a log line
-    filename                path to a log file (/var/log/auth.log)
+  filename              path to a log file (/var/log/auth.log)
-    "systemd-journal"       search systemd journal (systemd-python required)
+  systemd-journal       search systemd journal (systemd-python required),
+                        optionally with backend parameters, see `man jail.conf`
+                        for usage and examples (systemd-journal[journalflags=1]).
 
 REGEX:
-    string                  a string representing a 'failregex'
+  string                a string representing a 'failregex'
-    filename                path to a filter file (filter.d/sshd.conf)
+  filter                name of filter, optionally with options (sshd[mode=aggressive])
+  filename              path to a filter file (filter.d/sshd.conf)
 
 IGNOREREGEX:
-    string                  a string representing an 'ignoreregex'
+  string                a string representing an 'ignoreregex'
-    filename                path to a filter file (filter.d/sshd.conf)
+  filename              path to a filter file (filter.d/sshd.conf)
 \n""" + OptionParser.format_help(self, *args, **kwargs) + """\n
 Report bugs to https://github.com/fail2ban/fail2ban/issues\n
 """ + __copyright__ + "\n"
@@ -252,6 +254,8 @@ class Fail2banRegex(object):
 
 		self.share_config=dict()
 		self._filter = Filter(None)
+		self._prefREMatched = 0
+		self._prefREGroups = list()
 		self._ignoreregex = list()
 		self._failregex = list()
 		self._time_elapsed = None
@@ -265,15 +269,19 @@ class Fail2banRegex(object):
 			self.setJournalMatch(shlex.split(opts.journalmatch))
 		if opts.timezone:
 			self._filter.setLogTimeZone(opts.timezone)
+		self._filter.checkFindTime = False
+		if True: # not opts.out:
+			MyTime.setAlternateNow(0); # accept every date (years from 19xx up to end of current century, '%ExY' and 'Exy' patterns)
+			from ..server.strptime import _updateTimeRE
+			_updateTimeRE()
 		if opts.datepattern:
 			self.setDatePattern(opts.datepattern)
 		if opts.usedns:
 			self._filter.setUseDns(opts.usedns)
 		self._filter.returnRawHost = opts.raw
-		self._filter.checkFindTime = False
 		self._filter.checkAllRegex = opts.checkAllRegex and not opts.out
 		# ignore pending (without ID/IP), added to matches if it hits later (if ID/IP can be retreved)
-		self._filter.ignorePending = opts.out
+		self._filter.ignorePending = bool(opts.out)
 		# callback to increment ignored RE's by index (during process):
 		self._filter.onIgnoreRegex = self._onIgnoreRegex
 		self._backend = 'auto'
@@ -281,9 +289,6 @@ class Fail2banRegex(object):
 	def output(self, line):
 		if not self._opts.out: output(line)
 
-	def decode_line(self, line):
-		return FileContainer.decode_line('<LOG>', self._encoding, line)
-
 	def encode_line(self, line):
 		return line.encode(self._encoding, 'ignore')
 
@@ -292,8 +297,8 @@ class Fail2banRegex(object):
 			self._filter.setDatePattern(pattern)
 			self._datepattern_set = True
 			if pattern is not None:
-				self.output( "Use      datepattern : %s" % (
+				self.output( "Use      datepattern : %s : %s" % (
-					self._filter.getDatePattern()[1], ) )
+					pattern, self._filter.getDatePattern()[1], ) )
 
 	def setMaxLines(self, v):
 		if not self._maxlines_set:
@@ -322,26 +327,33 @@ class Fail2banRegex(object):
 		regex = regextype + 'regex'
 		# try to check - we've case filter?[options...]?:
 		basedir = self._opts.config
+		fltName = value
 		fltFile = None
 		fltOpt = {}
 		if regextype == 'fail':
-			fltName, fltOpt = extractOptions(value)
+			if re.search(r'^/{0,3}[\w/_\-.]+(?:\[.*\])?$', value):
-			if fltName is not None:
+				try:
-				if "." in fltName[~5:]:
+					fltName, fltOpt = extractOptions(value)
-					tryNames = (fltName,)
+					if "." in fltName[~5:]:
-				else:
+						tryNames = (fltName,)
-					tryNames = (fltName, fltName + '.conf', fltName + '.local')
-				for fltFile in tryNames:
-					if not "/" in fltFile:
-						if os.path.basename(basedir) == 'filter.d':
-							fltFile = os.path.join(basedir, fltFile)
-						else:
-							fltFile = os.path.join(basedir, 'filter.d', fltFile)
 					else:
-						basedir = os.path.dirname(fltFile)
+						tryNames = (fltName, fltName + '.conf', fltName + '.local')
-					if os.path.isfile(fltFile):
+					for fltFile in tryNames:
-						break
+						if not "/" in fltFile:
-					fltFile = None
+							if os.path.basename(basedir) == 'filter.d':
+								fltFile = os.path.join(basedir, fltFile)
+							else:
+								fltFile = os.path.join(basedir, 'filter.d', fltFile)
+						else:
+							basedir = os.path.dirname(fltFile)
+						if os.path.isfile(fltFile):
+							break
+						fltFile = None
+				except Exception as e:
+					output("ERROR: Wrong filter name or options: %s" % (str(e),))
+					output("       while parsing: %s" % (value,))
+					if self._verbose: raise(e)
+					return False
 		# if it is filter file:
 		if fltFile is not None:
 			if (basedir == self._opts.config
@@ -453,19 +465,33 @@ class Fail2banRegex(object):
 			lines = []
 			ret = []
 			for match in found:
-				# Append True/False flag depending if line was matched by
+				if not self._opts.out:
-				# more than one regex
+					# Append True/False flag depending if line was matched by
-				match.append(len(ret)>1)
+					# more than one regex
-				regex = self._failregex[match[0]]
+					match.append(len(ret)>1)
-				regex.inc()
+					regex = self._failregex[match[0]]
-				regex.appendIP(match)
+					regex.inc()
+					regex.appendIP(match)
 				if not match[3].get('nofail'):
 					ret.append(match)
 				else:
 					is_ignored = True
+			if self._opts.out: # (formated) output - don't need stats:
+				return None, ret, None
+			# prefregex stats:
+			if self._filter.prefRegex:
+				pre = self._filter.prefRegex
+				if pre.hasMatched():
+					self._prefREMatched += 1
+					if self._verbose:
+						if len(self._prefREGroups) < self._maxlines:
+							self._prefREGroups.append(pre.getGroups())
+						else:
+							if len(self._prefREGroups) == self._maxlines:
+								self._prefREGroups.append('...')
 		except RegexException as e: # pragma: no cover
 			output( 'ERROR: %s' % e )
-			return False
+			return None, 0, None
 		if self._filter.getMaxLines() > 1:
 			for bufLine in orgLineBuffer[int(fullBuffer):]:
 				if bufLine not in self._filter._Filter__lineBuffer:
@@ -651,7 +677,18 @@ class Fail2banRegex(object):
 			pprint_list(out, " #) [# of hits] regular expression")
 			return total
 
-		# Print title
+		# Print prefregex:
+		if self._filter.prefRegex:
+			#self._filter.prefRegex.hasMatched()
+			pre = self._filter.prefRegex 
+			out = [pre.getRegex()]
+			if self._verbose:
+				for grp in self._prefREGroups:
+					out.append("    %s" % (grp,))
+			output( "\n%s: %d total" % ("Prefregex", self._prefREMatched) )
+			pprint_list(out)
+
+		# Print regex's:
 		total = print_failregexes("Failregex", self._failregex)
 		_ = print_failregexes("Ignoreregex", self._ignoreregex)
 
@@ -683,10 +720,6 @@ class Fail2banRegex(object):
 
 		return True
 
-	def file_lines_gen(self, hdlr):
-		for line in hdlr:
-			yield self.decode_line(line)
-
 	def start(self, args):
 
 		cmd_log, cmd_regex = args[:2]
@@ -705,10 +738,10 @@ class Fail2banRegex(object):
 
 		if os.path.isfile(cmd_log):
 			try:
-				hdlr = open(cmd_log, 'rb')
+				test_lines = FileContainer(cmd_log, self._encoding, doOpen=True)
+
 				self.output( "Use         log file : %s" % cmd_log )
 				self.output( "Use         encoding : %s" % self._encoding )
-				test_lines = self.file_lines_gen(hdlr)
 			except IOError as e: # pragma: no cover
 				output( e )
 				return False

fail2ban/client/jailreader.py

@@ -140,9 +140,10 @@ class JailReader(ConfigReader):
 			# Read filter
 			flt = self.__opts["filter"]
 			if flt:
-				filterName, filterOpt = extractOptions(flt)
+				try:
-				if not filterName:
+					filterName, filterOpt = extractOptions(flt)
-					raise JailDefError("Invalid filter definition %r" % flt)
+				except ValueError as e:
+					raise JailDefError("Invalid filter definition %r: %s" % (flt, e))
 				self.__filter = FilterReader(
 					filterName, self.__name, filterOpt, 
 					share_config=self.share_config, basedir=self.getBaseDir())
@@ -174,10 +175,10 @@ class JailReader(ConfigReader):
 					if not act:			  # skip empty actions
 						continue
 					# join with previous line if needed (consider possible new-line):
-					actName, actOpt = extractOptions(act)
+					try:
-					prevln = ''
+						actName, actOpt = extractOptions(act)
-					if not actName:
+					except ValueError as e:
-						raise JailDefError("Invalid action definition %r" % act)
+						raise JailDefError("Invalid action definition %r: %s" % (act, e))
 					if actName.endswith(".py"):
 						self.__actions.append([
 							"set",

fail2ban/helpers.py

@@ -224,9 +224,10 @@ def __stopOnIOError(logSys=None, logHndlr=None): # pragma: no cover
 		sys.exit(0)
 
 try:
-    BrokenPipeError
+	BrokenPipeError = BrokenPipeError
 except NameError: # pragma: 3.x no cover
-    BrokenPipeError = IOError
+	BrokenPipeError = IOError
+
 __origLog = logging.Logger._log
 def __safeLog(self, level, msg, args, **kwargs):
 	"""Safe log inject to avoid possible errors by unsafe log-handlers, 
@@ -302,7 +303,7 @@ def getVerbosityFormat(verbosity, fmt=' %(message)s', addtime=True, padding=True
 			if addtime:
 				fmt = ' %(asctime)-15s' + fmt
 	else: # default (not verbose):
-		fmt = "%(name)-23.23s [%(process)d]: %(levelname)-7s" + fmt
+		fmt = "%(name)-24s[%(process)d]: %(levelname)-7s" + fmt
 		if addtime:
 			fmt = "%(asctime)s " + fmt
 	# remove padding if not needed:
@@ -370,21 +371,27 @@ OPTION_CRE = re.compile(r"^([^\[]+)(?:\[(.*)\])?\s*$", re.DOTALL)
 # since v0.10 separator extended with `]\s*[` for support of multiple option groups, syntax 
 # `action = act[p1=...][p2=...]`
 OPTION_EXTRACT_CRE = re.compile(
-	r'([\w\-_\.]+)=(?:"([^"]*)"|\'([^\']*)\'|([^,\]]*))(?:,|\]\s*\[|$)', re.DOTALL)
+	r'\s*([\w\-_\.]+)=(?:"([^"]*)"|\'([^\']*)\'|([^,\]]*))(?:,|\]\s*\[|$|(?P<wrngA>.+))|,?\s*$|(?P<wrngB>.+)', re.DOTALL)
 # split by new-line considering possible new-lines within options [...]:
 OPTION_SPLIT_CRE = re.compile(
-	r'(?:[^\[\n]+(?:\s*\[\s*(?:[\w\-_\.]+=(?:"[^"]*"|\'[^\']*\'|[^,\]]*)\s*(?:,|\]\s*\[)?\s*)*\])?\s*|[^\n]+)(?=\n\s*|$)', re.DOTALL)
+	r'(?:[^\[\s]+(?:\s*\[\s*(?:[\w\-_\.]+=(?:"[^"]*"|\'[^\']*\'|[^,\]]*)\s*(?:,|\]\s*\[)?\s*)*\])?\s*|\S+)(?=\n\s*|\s+|$)', re.DOTALL)
 
 def extractOptions(option):
 	match = OPTION_CRE.match(option)
 	if not match:
-		# TODO proper error handling
+		raise ValueError("unexpected option syntax")
-		return None, None
 	option_name, optstr = match.groups()
 	option_opts = dict()
 	if optstr:
 		for optmatch in OPTION_EXTRACT_CRE.finditer(optstr):
+			if optmatch.group("wrngA"):
+				raise ValueError("unexpected syntax at %d after option %r: %s" % (
+					optmatch.start("wrngA"), optmatch.group(1), optmatch.group("wrngA")[0:25]))
+			if optmatch.group("wrngB"):
+				raise ValueError("expected option, wrong syntax at %d: %s" % (
+					optmatch.start("wrngB"), optmatch.group("wrngB")[0:25]))
 			opt = optmatch.group(1)
+			if not opt: continue
 			value = [
 				val for val in optmatch.group(2,3,4) if val is not None][0]
 			option_opts[opt.strip()] = value.strip()
@@ -398,8 +405,8 @@ def splitWithOptions(option):
 # tags (<tag>) in tagged options.
 #
 
-# max tag replacement count:
+# max tag replacement count (considering tag X in tag Y repeat):
-MAX_TAG_REPLACE_COUNT = 10
+MAX_TAG_REPLACE_COUNT = 25
 
 # compiled RE for tag name (replacement name) 
 TAG_CRE = re.compile(r'<([^ <>]+)>')
@@ -433,6 +440,7 @@ def substituteRecursiveTags(inptags, conditional='',
 	done = set()
 	noRecRepl = hasattr(tags, "getRawItem")
 	# repeat substitution while embedded-recursive (repFlag is True)
+	repCounts = {}
 	while True:
 		repFlag = False
 		# substitute each value:
@@ -444,7 +452,7 @@ def substituteRecursiveTags(inptags, conditional='',
 			value = orgval = uni_string(tags[tag])
 			# search and replace all tags within value, that can be interpolated using other tags:
 			m = tre_search(value)
-			refCounts = {}
+			rplc = repCounts.get(tag, {})
 			#logSys.log(5, 'TAG: %s, value: %s' % (tag, value))
 			while m:
 				# found replacement tag:
@@ -454,13 +462,13 @@ def substituteRecursiveTags(inptags, conditional='',
 					m = tre_search(value, m.end())
 					continue
 				#logSys.log(5, 'found: %s' % rtag)
-				if rtag == tag or refCounts.get(rtag, 1) > MAX_TAG_REPLACE_COUNT:
+				if rtag == tag or rplc.get(rtag, 1) > MAX_TAG_REPLACE_COUNT:
 					# recursive definitions are bad
 					#logSys.log(5, 'recursion fail tag: %s value: %s' % (tag, value) )
 					raise ValueError(
 						"properties contain self referencing definitions "
 						"and cannot be resolved, fail tag: %s, found: %s in %s, value: %s" % 
-						(tag, rtag, refCounts, value))
+						(tag, rtag, rplc, value))
 				repl = None
 				if conditional:
 					repl = tags.get(rtag + '?' + conditional)
@@ -480,7 +488,7 @@ def substituteRecursiveTags(inptags, conditional='',
 				value = value.replace('<%s>' % rtag, repl)
 				#logSys.log(5, 'value now: %s' % value)
 				# increment reference count:
-				refCounts[rtag] = refCounts.get(rtag, 0) + 1
+				rplc[rtag] = rplc.get(rtag, 0) + 1
 				# the next match for replace:
 				m = tre_search(value, m.start())
 			#logSys.log(5, 'TAG: %s, newvalue: %s' % (tag, value))
@@ -488,6 +496,7 @@ def substituteRecursiveTags(inptags, conditional='',
 			if orgval != value:
 				# check still contains any tag - should be repeated (possible embedded-recursive substitution):
 				if tre_search(value):
+					repCounts[tag] = rplc
 					repFlag = True
 				# copy return tags dict to prevent modifying of inptags:
 				if id(tags) == id(inptags):

fail2ban/protocol.py

@@ -55,6 +55,8 @@ protocol = [
 ["stop", "stops all jails and terminate the server"], 
 ["unban --all", "unbans all IP addresses (in all jails and database)"],
 ["unban <IP> ... <IP>", "unbans <IP> (in all jails and database)"],
+["banned", "return jails with banned IPs as dictionary"],
+["banned <IP> ... <IP>]", "return list(s) of jails where given IP(s) are banned"],
 ["status", "gets the current status of the server"], 
 ["ping", "tests if the server is alive"],
 ["echo", "for internal usage, returns back and outputs a given string"],
@@ -120,6 +122,8 @@ protocol = [
 ["set <JAIL> action <ACT> <PROPERTY> <VALUE>", "sets the <VALUE> of <PROPERTY> for the action <ACT> for <JAIL>"],
 ["set <JAIL> action <ACT> <METHOD>[ <JSONKWARGS>]", "calls the <METHOD> with <JSONKWARGS> for the action <ACT> for <JAIL>"],
 ['', "JAIL INFORMATION", ""],
+["get <JAIL> banned", "return banned IPs of <JAIL>"],
+["get <JAIL> banned <IP> ... <IP>]", "return 1 if IP is banned in <JAIL> otherwise 0, or a list of 1/0 for multiple IPs"],
 ["get <JAIL> logpath", "gets the list of the monitored files for <JAIL>"],
 ["get <JAIL> logencoding", "gets the encoding of the log files for <JAIL>"],
 ["get <JAIL> journalmatch", "gets the journal filter match for <JAIL>"],

fail2ban/server/action.py

@@ -30,7 +30,10 @@ import tempfile
 import threading
 import time
 from abc import ABCMeta
-from collections import MutableMapping
+try:
+	from collections.abc import MutableMapping
+except ImportError:
+	from collections import MutableMapping
 
 from .failregex import mapTag2Opt
 from .ipdns import DNSUtils
@@ -455,7 +458,18 @@ class CommandAction(ActionBase):
 				ret = True
 				# avoid double execution of same command for both families:
 				if cmd and cmd not in self._operationExecuted(tag, lambda f: f != famoper):
-					ret = self.executeCmd(cmd, self.timeout)
+					realCmd = cmd
+					if self._jail:
+						# simulate action info with "empty" ticket:
+						aInfo = getattr(self._jail.actions, 'actionInfo', None)
+						if not aInfo:
+							aInfo = self._jail.actions._getActionInfo(None)
+							setattr(self._jail.actions, 'actionInfo', aInfo)
+						aInfo['time'] = MyTime.time()
+						aInfo['family'] = famoper
+						# replace dynamical tags, important - don't cache, no recursion and auto-escape here
+						realCmd = self.replaceDynamicTags(cmd, aInfo)
+					ret = self.executeCmd(realCmd, self.timeout)
 					res &= ret
 				if afterExec: afterExec(famoper, ret)
 				self._operationExecuted(tag, famoper, cmd if ret else None)
@@ -868,7 +882,7 @@ class CommandAction(ActionBase):
 			tickData = aInfo.get("F-*")
 			if not tickData: tickData = {}
 			def substTag(m):
-				tag = mapTag2Opt(m.groups()[0])
+				tag = mapTag2Opt(m.group(1))
 				try:
 					value = uni_string(tickData[tag])
 				except KeyError:

fail2ban/server/actions.py

@@ -28,7 +28,10 @@ import logging
 import os
 import sys
 import time
-from collections import Mapping
+try:
+	from collections.abc import Mapping
+except ImportError:
+	from collections import Mapping
 try:
 	from collections import OrderedDict
 except ImportError:
@@ -81,7 +84,7 @@ class Actions(JailThread, Mapping):
 		self._jail = jail
 		self._actions = OrderedDict()
 		## The ban manager.
-		self.__banManager = BanManager()
+		self.banManager = BanManager()
 		self.banEpoch = 0
 		self.__lastConsistencyCheckTM = 0
 		## Precedence of ban (over unban), so max number of tickets banned (to call an unban check):
@@ -200,7 +203,7 @@ class Actions(JailThread, Mapping):
 	
 	def setBanTime(self, value):
 		value = MyTime.str2seconds(value)
-		self.__banManager.setBanTime(value)
+		self.banManager.setBanTime(value)
 		logSys.info("  banTime: %s" % value)
 	
 	##
@@ -209,7 +212,15 @@ class Actions(JailThread, Mapping):
 	# @return the time
 	
 	def getBanTime(self):
-		return self.__banManager.getBanTime()
+		return self.banManager.getBanTime()
+
+	def getBanned(self, ids):
+		lst = self.banManager.getBanList()
+		if not ids:
+			return lst
+		if len(ids) == 1:
+			return 1 if ids[0] in lst else 0
+		return map(lambda ip: 1 if ip in lst else 0, ids)
 
 	def getBanList(self, withTime=False):
 		"""Returns the list of banned IP addresses.
@@ -219,7 +230,7 @@ class Actions(JailThread, Mapping):
 		list
 			The list of banned IP addresses.
 		"""
-		return self.__banManager.getBanList(ordered=True, withTime=withTime)
+		return self.banManager.getBanList(ordered=True, withTime=withTime)
 
 	def addBannedIP(self, ip):
 		"""Ban an IP or list of IPs."""
@@ -254,7 +265,7 @@ class Actions(JailThread, Mapping):
 		if ip is None:
 			return self.__flushBan(db)
 		# Multiple IPs:
-		if isinstance(ip, list):
+		if isinstance(ip, (list, tuple)):
 			missed = []
 			cnt = 0
 			for i in ip:
@@ -271,11 +282,19 @@ class Actions(JailThread, Mapping):
 		if db and self._jail.database is not None:
 			self._jail.database.delBan(self._jail, ip)
 		# Find the ticket with the IP.
-		ticket = self.__banManager.getTicketByID(ip)
+		ticket = self.banManager.getTicketByID(ip)
 		if ticket is not None:
 			# Unban the IP.
 			self.__unBan(ticket)
 		else:
+			# Multiple IPs by subnet or dns:
+			if not isinstance(ip, IPAddr):
+				ipa = IPAddr(ip)
+				if not ipa.isSingle: # subnet (mask/cidr) or raw (may be dns/hostname):
+					ips = filter(ipa.contains, self.banManager.getBanList())
+					if ips:
+						return self.removeBannedIP(ips, db, ifexists)
+			# not found:
 			msg = "%s is not banned" % ip
 			logSys.log(logging.MSG, msg)
 			if ifexists:
@@ -322,25 +341,33 @@ class Actions(JailThread, Mapping):
 					self._jail.name, name, e,
 					exc_info=logSys.getEffectiveLevel()<=logging.DEBUG)
 		while self.active:
-			if self.idle:
+			try:
-				logSys.debug("Actions: enter idle mode")
+				if self.idle:
-				Utils.wait_for(lambda: not self.active or not self.idle,
+					logSys.debug("Actions: enter idle mode")
-					lambda: False, self.sleeptime)
+					Utils.wait_for(lambda: not self.active or not self.idle,
-				logSys.debug("Actions: leave idle mode")
+						lambda: False, self.sleeptime)
-				continue
+					logSys.debug("Actions: leave idle mode")
-			# wait for ban (stop if gets inactive):
+					continue
-			bancnt = 0
+				# wait for ban (stop if gets inactive, pending ban or unban):
-			if Utils.wait_for(lambda: not self.active or self._jail.hasFailTickets, self.sleeptime):
+				bancnt = 0
-				bancnt = self.__checkBan()
+				wt = min(self.sleeptime, self.banManager._nextUnbanTime - MyTime.time())
-				cnt += bancnt
+				logSys.log(5, "Actions: wait for pending tickets %s (default %s)", wt, self.sleeptime)
-			# unban if nothing is banned not later than banned tickets >= banPrecedence
+				if Utils.wait_for(lambda: not self.active or self._jail.hasFailTickets, wt):
-			if not bancnt or cnt >= self.banPrecedence:
+					bancnt = self.__checkBan()
-				if self.active:
+					cnt += bancnt
-					# let shrink the ban list faster
+				# unban if nothing is banned not later than banned tickets >= banPrecedence
-					bancnt *= 2
+				if not bancnt or cnt >= self.banPrecedence:
-					self.__checkUnBan(bancnt if bancnt and bancnt < self.unbanMaxCount else self.unbanMaxCount)
+					if self.active:
-				cnt = 0
+						# let shrink the ban list faster
-		
+						bancnt *= 2
+						logSys.log(5, "Actions: check-unban %s, bancnt %s, max: %s", bancnt if bancnt and bancnt < self.unbanMaxCount else self.unbanMaxCount, bancnt, self.unbanMaxCount)
+						self.__checkUnBan(bancnt if bancnt and bancnt < self.unbanMaxCount else self.unbanMaxCount)
+					cnt = 0
+			except Exception as e: # pragma: no cover
+				logSys.error("[%s] unhandled error in actions thread: %s",
+					self._jail.name, e,
+					exc_info=logSys.getEffectiveLevel()<=logging.DEBUG)
+
 		self.__flushBan(stop=True)
 		self.stopActions()
 		return True
@@ -370,7 +397,12 @@ class Actions(JailThread, Mapping):
 			"ipfailures":			lambda self: self._mi4ip(True).getAttempt(),
 			"ipjailfailures":	lambda self: self._mi4ip().getAttempt(),
 			# raw ticket info:
-			"raw-ticket":			lambda self: repr(self.__ticket)
+			"raw-ticket":			lambda self: repr(self.__ticket),
+			# jail info:
+			"jail.banned":       lambda self: self.__jail.actions.banManager.size(),
+			"jail.banned_total": lambda self: self.__jail.actions.banManager.getBanTotal(),
+			"jail.found":        lambda self: self.__jail.filter.failManager.size(),
+			"jail.found_total":  lambda self: self.__jail.filter.failManager.getFailTotal()
 		}
 
 		__slots__ = CallingMap.__slots__ + ('__ticket', '__jail', '__mi4ip')
@@ -433,7 +465,9 @@ class Actions(JailThread, Mapping):
 			return mi[idx] if mi[idx] is not None else self.__ticket
 
 
-	def __getActionInfo(self, ticket):
+	def _getActionInfo(self, ticket):
+		if not ticket:
+			ticket = BanTicket("", MyTime.time())
 		aInfo = Actions.ActionInfo(ticket, self._jail)
 		return aInfo
 
@@ -465,11 +499,11 @@ class Actions(JailThread, Mapping):
 		for ticket in tickets:
 
 			bTicket = BanTicket.wrap(ticket)
-			btime = ticket.getBanTime(self.__banManager.getBanTime())
+			btime = ticket.getBanTime(self.banManager.getBanTime())
 			ip = bTicket.getIP()
-			aInfo = self.__getActionInfo(bTicket)
+			aInfo = self._getActionInfo(bTicket)
 			reason = {}
-			if self.__banManager.addBanTicket(bTicket, reason=reason):
+			if self.banManager.addBanTicket(bTicket, reason=reason):
 				cnt += 1
 				# report ticket to observer, to check time should be increased and hereafter observer writes ban to database (asynchronous)
 				if Observers.Main is not None and not bTicket.restored:
@@ -528,7 +562,7 @@ class Actions(JailThread, Mapping):
 			# and increase ticket time if "bantime.increment" set)
 		if cnt:
 			logSys.debug("Banned %s / %s, %s ticket(s) in %r", cnt, 
-				self.__banManager.getBanTotal(), self.__banManager.size(), self._jail.name)
+				self.banManager.getBanTotal(), self.banManager.size(), self._jail.name)
 		return cnt
 
 	def __reBan(self, ticket, actions=None, log=True):
@@ -544,7 +578,7 @@ class Actions(JailThread, Mapping):
 		"""
 		actions = actions or self._actions
 		ip = ticket.getIP()
-		aInfo = self.__getActionInfo(ticket)
+		aInfo = self._getActionInfo(ticket)
 		if log:
 			logSys.notice("[%s] Reban %s%s", self._jail.name, aInfo["ip"], (', action %r' % actions.keys()[0] if len(actions) == 1 else ''))
 		for name, action in actions.iteritems():
@@ -568,7 +602,7 @@ class Actions(JailThread, Mapping):
 	def _prolongBan(self, ticket):
 		# prevent to prolong ticket that was removed in-between,
 		# if it in ban list - ban time already prolonged (and it stays there):
-		if not self.__banManager._inBanList(ticket): return
+		if not self.banManager._inBanList(ticket): return
 		# do actions :
 		aInfo = None
 		for name, action in self._actions.iteritems():
@@ -578,7 +612,7 @@ class Actions(JailThread, Mapping):
 				if not action._prolongable:
 					continue
 				if aInfo is None:
-					aInfo = self.__getActionInfo(ticket)
+					aInfo = self._getActionInfo(ticket)
 				if not aInfo.immutable: aInfo.reset()
 				action.prolong(aInfo)
 			except Exception as e:
@@ -593,13 +627,13 @@ class Actions(JailThread, Mapping):
 
 		Unban IP addresses which are outdated.
 		"""
-		lst = self.__banManager.unBanList(MyTime.time(), maxCount)
+		lst = self.banManager.unBanList(MyTime.time(), maxCount)
 		for ticket in lst:
 			self.__unBan(ticket)
 		cnt = len(lst)
 		if cnt:
 			logSys.debug("Unbanned %s, %s ticket(s) in %r", 
-				cnt, self.__banManager.size(), self._jail.name)
+				cnt, self.banManager.size(), self._jail.name)
 		return cnt
 
 	def __flushBan(self, db=False, actions=None, stop=False):
@@ -613,10 +647,10 @@ class Actions(JailThread, Mapping):
 		log = True
 		if actions is None:
 			logSys.debug("  Flush ban list")
-			lst = self.__banManager.flushBanList()
+			lst = self.banManager.flushBanList()
 		else:
 			log = False # don't log "[jail] Unban ..." if removing actions only.
-			lst = iter(self.__banManager)
+			lst = iter(self.banManager)
 		cnt = 0
 		# first we'll execute flush for actions supporting this operation:
 		unbactions = {}
@@ -653,7 +687,7 @@ class Actions(JailThread, Mapping):
 			self.__unBan(ticket, actions=actions, log=log)
 			cnt += 1
 		logSys.debug("  Unbanned %s, %s ticket(s) in %r", 
-			cnt, self.__banManager.size(), self._jail.name)
+			cnt, self.banManager.size(), self._jail.name)
 		return cnt
 
 	def __unBan(self, ticket, actions=None, log=True):
@@ -672,7 +706,7 @@ class Actions(JailThread, Mapping):
 		else:
 			unbactions = actions
 		ip = ticket.getIP()
-		aInfo = self.__getActionInfo(ticket)
+		aInfo = self._getActionInfo(ticket)
 		if log:
 			logSys.notice("[%s] Unban %s", self._jail.name, aInfo["ip"])
 		for name, action in unbactions.iteritems():
@@ -691,17 +725,23 @@ class Actions(JailThread, Mapping):
 		"""Status of current and total ban counts and current banned IP list.
 		"""
 		# TODO: Allow this list to be printed as 'status' output
-		supported_flavors = ["basic", "cymru"]
+		supported_flavors = ["short", "basic", "cymru"]
 		if flavor is None or flavor not in supported_flavors:
 			logSys.warning("Unsupported extended jail status flavor %r. Supported: %s" % (flavor, supported_flavors))
 		# Always print this information (basic)
-		ret = [("Currently banned", self.__banManager.size()),
+		if flavor != "short":
-			   ("Total banned", self.__banManager.getBanTotal()),
+			banned = self.banManager.getBanList()
-			   ("Banned IP list", self.__banManager.getBanList())]
+			cnt = len(banned)
+		else:
+			cnt = self.banManager.size()
+		ret = [("Currently banned", cnt),
+			   ("Total banned", self.banManager.getBanTotal())]
+		if flavor != "short":
+			ret += [("Banned IP list", banned)]
 		if flavor == "cymru":
-			cymru_info = self.__banManager.getBanListExtendedCymruInfo()
+			cymru_info = self.banManager.getBanListExtendedCymruInfo()
 			ret += \
-				[("Banned ASN list", self.__banManager.geBanListExtendedASN(cymru_info)),
+				[("Banned ASN list", self.banManager.geBanListExtendedASN(cymru_info)),
-				 ("Banned Country list", self.__banManager.geBanListExtendedCountry(cymru_info)),
+				 ("Banned Country list", self.banManager.geBanListExtendedCountry(cymru_info)),
-				 ("Banned RIR list", self.__banManager.geBanListExtendedRIR(cymru_info))]
+				 ("Banned RIR list", self.banManager.geBanListExtendedRIR(cymru_info))]
 		return ret

fail2ban/server/banmanager.py

@@ -57,7 +57,7 @@ class BanManager:
 		## Total number of banned IP address
 		self.__banTotal = 0
 		## The time for next unban process (for performance and load reasons):
-		self.__nextUnbanTime = BanTicket.MAX_TIME
+		self._nextUnbanTime = BanTicket.MAX_TIME
 	
 	##
 	# Set the ban time.
@@ -66,7 +66,6 @@ class BanManager:
 	# @param value the time
 	
 	def setBanTime(self, value):
-		with self.__lock:
 			self.__banTime = int(value)
 	
 	##
@@ -76,7 +75,6 @@ class BanManager:
 	# @return the time
 	
 	def getBanTime(self):
-		with self.__lock:
 			return self.__banTime
 	
 	##
@@ -85,7 +83,6 @@ class BanManager:
 	# @param value total number
 	
 	def setBanTotal(self, value):
-		with self.__lock:
 			self.__banTotal = value
 	
 	##
@@ -94,7 +91,6 @@ class BanManager:
 	# @return the total number
 	
 	def getBanTotal(self):
-		with self.__lock:
 			return self.__banTotal
 
 	##
@@ -103,21 +99,21 @@ class BanManager:
 	# @return IP list
 	
 	def getBanList(self, ordered=False, withTime=False):
+		if not ordered:
+			return list(self.__banList.keys())
 		with self.__lock:
-			if not ordered:
-				return self.__banList.keys()
 			lst = []
 			for ticket in self.__banList.itervalues():
 				eob = ticket.getEndOfBanTime(self.__banTime)
 				lst.append((ticket,eob))
-			lst.sort(key=lambda t: t[1])
+		lst.sort(key=lambda t: t[1])
-			t2s = MyTime.time2str
+		t2s = MyTime.time2str
-			if withTime:
+		if withTime:
-				return ['%s \t%s + %d = %s' % (
+			return ['%s \t%s + %d = %s' % (
-						t[0].getID(), 
+					t[0].getID(), 
-						t2s(t[0].getTime()), t[0].getBanTime(self.__banTime), t2s(t[1])
+					t2s(t[0].getTime()), t[0].getBanTime(self.__banTime), t2s(t[1])
-					) for t in lst]
+				) for t in lst]
-			return [t[0].getID() for t in lst]
+		return [t[0].getID() for t in lst]
 
 	##
 	# Returns a iterator to ban list (used in reload, so idle).
@@ -125,8 +121,8 @@ class BanManager:
 	# @return ban list iterator
 	
 	def __iter__(self):
-		with self.__lock:
+		# ensure iterator is safe - traverse over the list in snapshot created within lock (GIL):
-			return self.__banList.itervalues()
+			return iter(list(self.__banList.values()))
 
 	##
 	# Returns normalized value
@@ -297,8 +293,8 @@ class BanManager:
 			self.__banTotal += 1
 			ticket.incrBanCount()
 			# correct next unban time:
-			if self.__nextUnbanTime > eob:
+			if self._nextUnbanTime > eob:
-				self.__nextUnbanTime = eob
+				self._nextUnbanTime = eob
 			return True
 
 	##
@@ -329,12 +325,8 @@ class BanManager:
 	
 	def unBanList(self, time, maxCount=0x7fffffff):
 		with self.__lock:
-			# Permanent banning
-			if self.__banTime < 0:
-				return list()
-
 			# Check next unban time:
-			nextUnbanTime = self.__nextUnbanTime
+			nextUnbanTime = self._nextUnbanTime
 			if nextUnbanTime > time:
 				return list()
 
@@ -347,12 +339,12 @@ class BanManager:
 				if time > eob:
 					unBanList[fid] = ticket
 					if len(unBanList) >= maxCount: # stop search cycle, so reset back the next check time
-						nextUnbanTime = self.__nextUnbanTime
+						nextUnbanTime = self._nextUnbanTime
 						break
 				elif nextUnbanTime > eob:
 					nextUnbanTime = eob
 
-			self.__nextUnbanTime = nextUnbanTime
+			self._nextUnbanTime = nextUnbanTime
 			# Removes tickets.
 			if len(unBanList):
 				if len(unBanList) / 2.0 <= len(self.__banList) / 3.0:

fail2ban/server/database.py

@@ -502,7 +502,7 @@ class Fail2BanDb(object):
 		except TypeError:
 			firstLineMD5 = None
 
-		if not firstLineMD5 and (pos or md5):
+		if firstLineMD5 is None and (pos or md5 is not None):
 			cur.execute(
 					"INSERT OR REPLACE INTO logs(jail, path, firstlinemd5, lastfilepos) "
 						"VALUES(?, ?, ?, ?)", (jail.name, name, md5, pos))

fail2ban/server/datedetector.py

@@ -35,7 +35,7 @@ from ..helpers import getLogger
 # Gets the instance of the logger.
 logSys = getLogger(__name__)
 
-logLevel = 6
+logLevel = 5
 
 RE_DATE_PREMATCH = re.compile(r"(?<!\\)\{DATE\}", re.IGNORECASE)
 DD_patternCache = Utils.Cache(maxCount=1000, maxTime=60*60)
@@ -282,6 +282,8 @@ class DateDetector(object):
 				elif "{DATE}" in key:
 					self.addDefaultTemplate(preMatch=pattern, allDefaults=False)
 					return
+				elif key == "{NONE}":
+					template = _getPatternTemplate('{UNB}^', key)
 				else:
 					template = _getPatternTemplate(pattern, key)
 

fail2ban/server/datetemplate.py

@@ -136,7 +136,7 @@ class DateTemplate(object):
 		# remove possible special pattern "**" in front and end of regex:
 		regex = RE_DEL_WRD_BOUNDS[0].sub(RE_DEL_WRD_BOUNDS[1], regex)
 		self._regex = regex
-		logSys.log(7, '  constructed regex %s', regex)
+		logSys.log(4, '  constructed regex %s', regex)
 		self._cRegex = None
 
 	regex = property(getRegex, setRegex, doc=
@@ -159,6 +159,7 @@ class DateTemplate(object):
 		"""
 		if not self._cRegex:
 			self._compileRegex()
+		logSys.log(4, "   search %s", self.regex)
 		dateMatch = self._cRegex.search(line, *args); # pos, endpos
 		if dateMatch:
 			self.hits += 1

fail2ban/server/failmanager.py

@@ -43,26 +43,20 @@ class FailManager:
 		self.__maxRetry = 3
 		self.__maxTime = 600
 		self.__failTotal = 0
-		self.maxMatches = 50
+		self.maxMatches = 5
 		self.__bgSvc = BgService()
 	
 	def setFailTotal(self, value):
-		with self.__lock:
+		self.__failTotal = value
-			self.__failTotal = value
 		
 	def getFailTotal(self):
-		with self.__lock:
+		return self.__failTotal
-			return self.__failTotal
 	
 	def getFailCount(self):
 		# may be slow on large list of failures, should be used for test purposes only...
 		with self.__lock:
 			return len(self.__failList), sum([f.getRetry() for f in self.__failList.values()])
 
-	def getFailTotal(self):
-		with self.__lock:
-			return self.__failTotal
-
 	def setMaxRetry(self, value):
 		self.__maxRetry = value
 	
@@ -130,13 +124,13 @@ class FailManager:
 		return attempts
 	
 	def size(self):
-		with self.__lock:
+		return len(self.__failList)
-			return len(self.__failList)
 	
 	def cleanup(self, time):
+		time -= self.__maxTime
 		with self.__lock:
 			todelete = [fid for fid,item in self.__failList.iteritems() \
-				if item.getTime() + self.__maxTime <= time]
+				if item.getTime() <= time]
 			if len(todelete) == len(self.__failList):
 				# remove all:
 				self.__failList = dict()
@@ -150,7 +144,7 @@ class FailManager:
 			else:
 				# create new dictionary without items to be deleted:
 				self.__failList = dict((fid,item) for fid,item in self.__failList.iteritems() \
-					if item.getTime() + self.__maxTime > time)
+					if item.getTime() > time)
 		self.__bgSvc.service()
 	
 	def delFailure(self, fid):

fail2ban/server/failregex.py

@@ -87,20 +87,24 @@ RH4TAG = {
 
 # default failure groups map for customizable expressions (with different group-id):
 R_MAP = {
-	"ID": "fid",
+	"id": "fid",
-	"PORT": "fport",
+	"port": "fport",
 }
 
 def mapTag2Opt(tag):
-	try: # if should be mapped:
+	tag = tag.lower()
-		return R_MAP[tag]
+	return R_MAP.get(tag, tag)
-	except KeyError:
-		return tag.lower()
 
 
-# alternate names to be merged, e. g. alt_user_1 -> user ...
+# complex names:
+# ALT_   - alternate names to be merged, e. g. alt_user_1 -> user ...
 ALTNAME_PRE = 'alt_'
-ALTNAME_CRE = re.compile(r'^' + ALTNAME_PRE + r'(.*)(?:_\d+)?$')
+# TUPLE_ - names of parts to be combined to single value as tuple
+TUPNAME_PRE = 'tuple_'
+
+COMPLNAME_PRE = (ALTNAME_PRE, TUPNAME_PRE)
+COMPLNAME_CRE = re.compile(r'^(' + '|'.join(COMPLNAME_PRE) + r')(.*?)(?:_\d+)?$')
+
 
 ##
 # Regular expression class.
@@ -127,19 +131,27 @@ class Regex:
 		try:
 			self._regexObj = re.compile(regex, re.MULTILINE if multiline else 0)
 			self._regex = regex
-			self._altValues = {}
+			self._altValues = []
+			self._tupleValues = []
 			for k in filter(
-				lambda k: len(k) > len(ALTNAME_PRE) and k.startswith(ALTNAME_PRE),
+				lambda k: len(k) > len(COMPLNAME_PRE[0]), self._regexObj.groupindex
-				self._regexObj.groupindex
 			):
-				n = ALTNAME_CRE.match(k).group(1)
+				n = COMPLNAME_CRE.match(k)
-				self._altValues[k] = n
+				if n:
-			self._altValues = list(self._altValues.items()) if len(self._altValues) else None
+					g, n = n.group(1), mapTag2Opt(n.group(2))
+					if g == ALTNAME_PRE:
+						self._altValues.append((k,n))
+					else:
+						self._tupleValues.append((k,n))
+			self._altValues.sort()
+			self._tupleValues.sort()
+			self._altValues = self._altValues if len(self._altValues) else None
+			self._tupleValues = self._tupleValues if len(self._tupleValues) else None
 		except sre_constants.error:
 			raise RegexException("Unable to compile regular expression '%s'" %
 								 regex)
-		# set fetch handler depending on presence of alternate tags:
+		# set fetch handler depending on presence of alternate (or tuple) tags:
-		self.getGroups = self._getGroupsWithAlt if self._altValues else self._getGroups
+		self.getGroups = self._getGroupsWithAlt if (self._altValues or self._tupleValues) else self._getGroups
 
 	def __str__(self):
 		return "%s(%r)" % (self.__class__.__name__, self._regex)
@@ -284,12 +296,23 @@ class Regex:
 
 	def _getGroupsWithAlt(self):
 		fail = self._matchCache.groupdict()
-		# merge alternate values (e. g. 'alt_user_1' -> 'user' or 'alt_host' -> 'host'):
 		#fail = fail.copy()
-		for k,n in self._altValues:
+		# merge alternate values (e. g. 'alt_user_1' -> 'user' or 'alt_host' -> 'host'):
-			v = fail.get(k)
+		if self._altValues:
-			if v and not fail.get(n):
+			for k,n in self._altValues:
-				fail[n] = v
+				v = fail.get(k)
+				if v and not fail.get(n):
+					fail[n] = v
+		# combine tuple values (e. g. 'id', 'tuple_id' ... 'tuple_id_N' -> 'id'):
+		if self._tupleValues:
+			for k,n in self._tupleValues:
+				v = fail.get(k)
+				t = fail.get(n)
+				if isinstance(t, tuple):
+					t += (v,)
+				else:
+					t = (t,v,)
+				fail[n] = t
 		return fail
 
 	def getGroups(self): # pragma: no cover - abstract function (replaced in __init__)

fail2ban/server/filter.py

@@ -81,6 +81,7 @@ class Filter(JailThread):
 		## Ignore own IPs flag:
 		self.__ignoreSelf = True
 		## The ignore IP list.
+		self.__ignoreIpSet = set()
 		self.__ignoreIpList = []
 		## External command
 		self.__ignoreCommand = False
@@ -93,6 +94,8 @@ class Filter(JailThread):
 		## Store last time stamp, applicable for multi-line
 		self.__lastTimeText = ""
 		self.__lastDate = None
+		## Next service (cleanup) time
+		self.__nextSvcTime = -(1<<63)
 		## if set, treat log lines without explicit time zone to be in this time zone
 		self.__logtimezone = None
 		## Default or preferred encoding (to decode bytes from file or journal):
@@ -112,10 +115,12 @@ class Filter(JailThread):
 		self.onIgnoreRegex = None
 		## if true ignores obsolete failures (failure time < now - findTime):
 		self.checkFindTime = True
-		## if true prevents against retarded banning in case of RC by too many failures (disabled only for test purposes):
+		## shows that filter is in operation mode (processing new messages):
-		self.banASAP = True
+		self.inOperation = True
 		## Ticks counter
 		self.ticks = 0
+		## Processed lines counter
+		self.procLines = 0
 		## Thread name:
 		self.name="f2b/f."+self.jailName
 
@@ -439,12 +444,23 @@ class Filter(JailThread):
 
 	def performBan(self, ip=None):
 		"""Performs a ban for IPs (or given ip) that are reached maxretry of the jail."""
-		try: # pragma: no branch - exception is the only way out
+		while True:
-			while True:
+			try:
 				ticket = self.failManager.toBan(ip)
-				self.jail.putFailTicket(ticket)
+			except FailManagerEmpty:
-		except FailManagerEmpty:
+				break
-			self.failManager.cleanup(MyTime.time())
+			self.jail.putFailTicket(ticket)
+			if ip: break
+		self.performSvc()
+
+	def performSvc(self, force=False):
+		"""Performs a service tasks (clean failure list)."""
+		tm = MyTime.time()
+		# avoid too early clean up:
+		if force or tm >= self.__nextSvcTime:
+			self.__nextSvcTime = tm + 5
+			# clean up failure list:
+			self.failManager.cleanup(tm)
 
 	def addAttempt(self, ip, *matches):
 		"""Generate a failed attempt for ip"""
@@ -490,28 +506,36 @@ class Filter(JailThread):
 		# Create IP address object
 		ip = IPAddr(ipstr)
 		# Avoid exact duplicates
-		if ip in self.__ignoreIpList:
+		if ip in self.__ignoreIpSet or ip in self.__ignoreIpList:
-			logSys.warn("  Ignore duplicate %r (%r), already in ignore list", ip, ipstr)
+			logSys.log(logging.MSG, "  Ignore duplicate %r (%r), already in ignore list", ip, ipstr)
 			return
 		# log and append to ignore list
 		logSys.debug("  Add %r to ignore list (%r)", ip, ipstr)
-		self.__ignoreIpList.append(ip)
+		# if single IP (not DNS or a subnet) add to set, otherwise to list:
+		if ip.isSingle:
+			self.__ignoreIpSet.add(ip)
+		else:
+			self.__ignoreIpList.append(ip)
 
 	def delIgnoreIP(self, ip=None):
 		# clear all:
 		if ip is None:
+			self.__ignoreIpSet.clear()
 			del self.__ignoreIpList[:]
 			return
 		# delete by ip:
 		logSys.debug("  Remove %r from ignore list", ip)
-		self.__ignoreIpList.remove(ip)
+		if ip in self.__ignoreIpSet:
+			self.__ignoreIpSet.remove(ip)
+		else:
+			self.__ignoreIpList.remove(ip)
 
 	def logIgnoreIp(self, ip, log_ignore, ignore_source="unknown source"):
 		if log_ignore:
 			logSys.info("[%s] Ignore %s by %s", self.jailName, ip, ignore_source)
 
 	def getIgnoreIP(self):
-		return self.__ignoreIpList
+		return self.__ignoreIpList + list(self.__ignoreIpSet)
 
 	##
 	# Check if IP address/DNS is in the ignore list.
@@ -551,8 +575,11 @@ class Filter(JailThread):
 			if self.__ignoreCache: c.set(key, True)
 			return True
 
+		# check if the IP is covered by ignore IP (in set or in subnet/dns):
+		if ip in self.__ignoreIpSet:
+			self.logIgnoreIp(ip, log_ignore, ignore_source="ip")
+			return True
 		for net in self.__ignoreIpList:
-			# check if the IP is covered by ignore IP
 			if ip.isInNet(net):
 				self.logIgnoreIp(ip, log_ignore, ignore_source=("ip" if net.isValid else "dns"))
 				if self.__ignoreCache: c.set(key, True)
@@ -575,16 +602,26 @@ class Filter(JailThread):
 		if self.__ignoreCache: c.set(key, False)
 		return False
 
+	def _logWarnOnce(self, nextLTM, *args):
+		"""Log some issue as warning once per day, otherwise level 7"""
+		if MyTime.time() < getattr(self, nextLTM, 0):
+			if logSys.getEffectiveLevel() <= 7: logSys.log(7, *(args[0]))
+		else:
+			setattr(self, nextLTM, MyTime.time() + 24*60*60)
+			for args in args:
+				logSys.warning('[%s] ' + args[0], self.jailName, *args[1:])
+
 	def processLine(self, line, date=None):
 		"""Split the time portion from log msg and return findFailures on them
 		"""
+		logSys.log(7, "Working on line %r", line)
+
+		noDate = False
 		if date:
 			tupleLine = line
 			self.__lastTimeText = tupleLine[1]
 			self.__lastDate = date
 		else:
-			logSys.log(7, "Working on line %r", line)
-
 			# try to parse date:
 			timeMatch = self.dateDetector.matchTime(line)
 			m = timeMatch[0]
@@ -595,22 +632,59 @@ class Filter(JailThread):
 				tupleLine = (line[:s], m, line[e:])
 				if m: # found and not empty - retrive date:
 					date = self.dateDetector.getTime(m, timeMatch)
-
+					if date is not None:
-				if date is None:
+						# Lets get the time part
-					if m: logSys.error("findFailure failed to parse timeText: %s", m)
+						date = date[0]
+						self.__lastTimeText = m
+						self.__lastDate = date
+					else:
+						logSys.error("findFailure failed to parse timeText: %s", m)
+				# matched empty value - date is optional or not available - set it to last known or now:
+				elif self.__lastDate and self.__lastDate > MyTime.time() - 60:
+					# set it to last known:
+					tupleLine = ("", self.__lastTimeText, line)
 					date = self.__lastDate
 				else:
-					# Lets get the time part
+					# set it to now:
-					date = date[0]
+					date = MyTime.time()
-					self.__lastTimeText = m
+			else:
+				tupleLine = ("", "", line)
+			# still no date - try to use last known:
+			if date is None:
+				noDate = True
+				if self.__lastDate and self.__lastDate > MyTime.time() - 60:
+					tupleLine = ("", self.__lastTimeText, line)
+					date = self.__lastDate
+		
+		if self.checkFindTime:
+			# if in operation (modifications have been really found):
+			if self.inOperation:
+				# if weird date - we'd simulate now for timeing issue (too large deviation from now):
+				if (date is None or date < MyTime.time() - 60 or date > MyTime.time() + 60):
+					# log time zone issue as warning once per day:
+					self._logWarnOnce("_next_simByTimeWarn",
+						("Simulate NOW in operation since found time has too large deviation %s ~ %s +/- %s",
+							date, MyTime.time(), 60),
+						("Please check jail has possibly a timezone issue. Line with odd timestamp: %s", 
+							line))
+					# simulate now as date:
+					date = MyTime.time()
 					self.__lastDate = date
 			else:
-				tupleLine = (line, self.__lastTimeText, "")
+				# in initialization (restore) phase, if too old - ignore:
-				date = self.__lastDate
+				if date is not None and date < MyTime.time() - self.getFindTime():
+					# log time zone issue as warning once per day:
+					self._logWarnOnce("_next_ignByTimeWarn",
+						("Ignore line since time %s < %s - %s",
+							date, MyTime.time(), self.getFindTime()),
+						("Please check jail has possibly a timezone issue. Line with odd timestamp: %s", 
+							line))
+					# ignore - too old (obsolete) entry:
+					return []
 
 		# save last line (lazy convert of process line tuple to string on demand):
 		self.processedLine = lambda: "".join(tupleLine[::2])
-		return self.findFailure(tupleLine, date)
+		return self.findFailure(tupleLine, date, noDate=noDate)
 
 	def processLineAndAdd(self, line, date=None):
 		"""Processes the line for failures and populates failManager
@@ -622,6 +696,9 @@ class Filter(JailThread):
 				fail = element[3]
 				logSys.debug("Processing line with time:%s and ip:%s", 
 						unixTime, ip)
+				# ensure the time is not in the future, e. g. by some estimated (assumed) time:
+				if self.checkFindTime and unixTime > MyTime.time():
+					unixTime = MyTime.time()
 				tick = FailTicket(ip, unixTime, data=fail)
 				if self._inIgnoreIPList(ip, tick):
 					continue
@@ -631,11 +708,15 @@ class Filter(JailThread):
 				attempts = self.failManager.addFailure(tick)
 				# avoid RC on busy filter (too many failures) - if attempts for IP/ID reached maxretry,
 				# we can speedup ban, so do it as soon as possible:
-				if self.banASAP and attempts >= self.failManager.getMaxRetry():
+				if attempts >= self.failManager.getMaxRetry():
 					self.performBan(ip)
 				# report to observer - failure was found, for possibly increasing of it retry counter (asynchronous)
 				if Observers.Main is not None:
 					Observers.Main.add('failureFound', self.failManager, self.jail, tick)
+			self.procLines += 1
+			# every 100 lines check need to perform service tasks:
+			if self.procLines % 100 == 0:
+				self.performSvc()
 			# reset (halve) error counter (successfully processed line):
 			if self._errors:
 				self._errors //= 2
@@ -744,7 +825,7 @@ class Filter(JailThread):
 	# to find the logging time.
 	# @return a dict with IP and timestamp.
 
-	def findFailure(self, tupleLine, date):
+	def findFailure(self, tupleLine, date, noDate=False):
 		failList = list()
 
 		ll = logSys.getEffectiveLevel()
@@ -754,11 +835,6 @@ class Filter(JailThread):
 			returnRawHost = True
 			cidr = IPAddr.CIDR_RAW
 
-		if self.checkFindTime and date is not None and date < MyTime.time() - self.getFindTime():
-			if ll <= 5: logSys.log(5, "Ignore line since time %s < %s - %s", 
-				date, MyTime.time(), self.getFindTime())
-			return failList
-
 		if self.__lineBufferSize > 1:
 			self.__lineBuffer.append(tupleLine)
 			orgBuffer = self.__lineBuffer = self.__lineBuffer[-self.__lineBufferSize:]
@@ -809,17 +885,13 @@ class Filter(JailThread):
 					if not self.checkAllRegex:
 						break
 					continue
-				if date is None:
+				if noDate:
-					logSys.warning(
+					self._logWarnOnce("_next_noTimeWarn",
-						"Found a match for %r but no valid date/time "
+						("Found a match but no valid date/time found for %r.", tupleLine[1]),
-						"found for %r. Please try setting a custom "
+						("Match without a timestamp: %s", "\n".join(failRegex.getMatchedLines())),
-						"date pattern (see man page jail.conf(5)). "
+						("Please try setting a custom date pattern (see man page jail.conf(5)).",)
-						"If format is complex, please "
+					)
-						"file a detailed issue on"
+					if date is None and self.checkFindTime: continue
-						" https://github.com/fail2ban/fail2ban/issues "
-						"in order to get support for this format.",
-						 "\n".join(failRegex.getMatchedLines()), tupleLine[1])
-					continue
 				# we should check all regex (bypass on multi-line, otherwise too complex):
 				if not self.checkAllRegex or self.__lineBufferSize > 1:
 					self.__lineBuffer, buf = failRegex.getUnmatchedTupleLines(), None
@@ -928,7 +1000,7 @@ class FileFilter(Filter):
 					log.setPos(lastpos)
 			self.__logs[path] = log
 			logSys.info("Added logfile: %r (pos = %s, hash = %s)" , path, log.getPos(), log.getHash())
-			if autoSeek:
+			if autoSeek and not tail:
 				self.__autoSeek[path] = autoSeek
 			self._addLogPath(path)			# backend specific
 
@@ -1012,7 +1084,8 @@ class FileFilter(Filter):
 	# MyTime.time()-self.findTime. When a failure is detected, a FailTicket
 	# is created and is added to the FailManager.
 
-	def getFailures(self, filename):
+	def getFailures(self, filename, inOperation=None):
+		if self.idle: return False
 		log = self.getLog(filename)
 		if log is None:
 			logSys.error("Unable to get failures in %s", filename)
@@ -1057,10 +1130,15 @@ class FileFilter(Filter):
 			if has_content:
 				while not self.idle:
 					line = log.readline()
-					if not line or not self.active:
+					if not self.active: break; # jail has been stopped
-						# The jail reached the bottom or has been stopped
+					if line is None:
+						# The jail reached the bottom, simply set in operation for this log
+						# (since we are first time at end of file, growing is only possible after modifications):
+						log.inOperation = True
 						break
-					self.processLineAndAdd(line.rstrip('\r\n'))
+					# acquire in operation from log and process:
+					self.inOperation = inOperation if inOperation is not None else log.inOperation
+					self.processLineAndAdd(line)
 		finally:
 			log.close()
 		db = self.jail.database
@@ -1077,6 +1155,8 @@ class FileFilter(Filter):
 		if logSys.getEffectiveLevel() <= logging.DEBUG:
 			logSys.debug("Seek to find time %s (%s), file size %s", date, 
 				MyTime.time2str(date), fs)
+		if not fs:
+			return
 		minp = container.getPos()
 		maxp = fs
 		tryPos = minp
@@ -1100,8 +1180,8 @@ class FileFilter(Filter):
 			dateTimeMatch = None
 			nextp = None
 			while True:
-				line = container.readline()
+				line = container.readline(False)
-				if not line:
+				if line is None:
 					break
 				(timeMatch, template) = self.dateDetector.matchTime(line)
 				if timeMatch:
@@ -1198,32 +1278,47 @@ except ImportError: # pragma: no cover
 
 class FileContainer:
 
-	def __init__(self, filename, encoding, tail = False):
+	def __init__(self, filename, encoding, tail=False, doOpen=False):
 		self.__filename = filename
+		self.waitForLineEnd = True
 		self.setEncoding(encoding)
 		self.__tail = tail
 		self.__handler = None
+		self.__pos = 0
+		self.__pos4hash = 0
+		self.__hash = ''
+		self.__hashNextTime = time.time() + 30
 		# Try to open the file. Raises an exception if an error occurred.
 		handler = open(filename, 'rb')
-		stats = os.fstat(handler.fileno())
+		if doOpen: # fail2ban-regex only (don't need to reopen it and check for rotation)
-		self.__ino = stats.st_ino
+			self.__handler = handler
+			return
 		try:
-			firstLine = handler.readline()
+			stats = os.fstat(handler.fileno())
-			# Computes the MD5 of the first line.
+			self.__ino = stats.st_ino
-			self.__hash = md5sum(firstLine).hexdigest()
+			if stats.st_size:
-			# Start at the beginning of file if tail mode is off.
+				firstLine = handler.readline()
-			if tail:
+				# first line available and contains new-line:
-				handler.seek(0, 2)
+				if firstLine != firstLine.rstrip(b'\r\n'):
-				self.__pos = handler.tell()
+					# Computes the MD5 of the first line.
-			else:
+					self.__hash = md5sum(firstLine).hexdigest()
-				self.__pos = 0
+				# if tail mode scroll to the end of file
+				if tail:
+					handler.seek(0, 2)
+					self.__pos = handler.tell()
 		finally:
 			handler.close()
+		## shows that log is in operation mode (expecting new messages only from here):
+		self.inOperation = tail
 
 	def getFileName(self):
 		return self.__filename
 
 	def getFileSize(self):
+		h = self.__handler
+		if h is not None:
+			stats = os.fstat(h.fileno())
+			return stats.st_size
 		return os.path.getsize(self.__filename);
 
 	def setEncoding(self, encoding):
@@ -1242,38 +1337,54 @@ class FileContainer:
 	def setPos(self, value):
 		self.__pos = value
 
-	def open(self):
+	def open(self, forcePos=None):
-		self.__handler = open(self.__filename, 'rb')
+		h = open(self.__filename, 'rb')
-		# Set the file descriptor to be FD_CLOEXEC
+		try:
-		fd = self.__handler.fileno()
+			# Set the file descriptor to be FD_CLOEXEC
-		flags = fcntl.fcntl(fd, fcntl.F_GETFD)
+			fd = h.fileno()
-		fcntl.fcntl(fd, fcntl.F_SETFD, flags | fcntl.FD_CLOEXEC)
+			flags = fcntl.fcntl(fd, fcntl.F_GETFD)
-		# Stat the file before even attempting to read it
+			fcntl.fcntl(fd, fcntl.F_SETFD, flags | fcntl.FD_CLOEXEC)
-		stats = os.fstat(self.__handler.fileno())
+			myHash = self.__hash
-		if not stats.st_size:
+			# Stat the file before even attempting to read it
-			# yoh: so it is still an empty file -- nothing should be
+			stats = os.fstat(h.fileno())
-			#      read from it yet
+			rotflg = stats.st_size < self.__pos or stats.st_ino != self.__ino
-			# print "D: no content -- return"
+			if rotflg or not len(myHash) or time.time() > self.__hashNextTime:
-			return False
+				myHash = ''
-		firstLine = self.__handler.readline()
+				firstLine = h.readline()
-		# Computes the MD5 of the first line.
+				# Computes the MD5 of the first line (if it is complete)
-		myHash = md5sum(firstLine).hexdigest()
+				if firstLine != firstLine.rstrip(b'\r\n'):
-		## print "D: fn=%s hashes=%s/%s inos=%s/%s pos=%s rotate=%s" % (
+					myHash = md5sum(firstLine).hexdigest()
-		## 	self.__filename, self.__hash, myHash, stats.st_ino, self.__ino, self.__pos,
+					self.__hashNextTime = time.time() + 30
-		## 	self.__hash != myHash or self.__ino != stats.st_ino)
+			elif stats.st_size == self.__pos:
-		## sys.stdout.flush()
+				myHash = self.__hash
-		# Compare hash and inode
+			# Compare size, hash and inode
-		if self.__hash != myHash or self.__ino != stats.st_ino:
+			if rotflg or myHash != self.__hash:
-			logSys.log(logging.MSG, "Log rotation detected for %s", self.__filename)
+				if self.__hash != '':
-			self.__hash = myHash
+					logSys.log(logging.MSG, "Log rotation detected for %s, reason: %r", self.__filename,
-			self.__ino = stats.st_ino
+						(stats.st_size, self.__pos, stats.st_ino, self.__ino, myHash, self.__hash))
-			self.__pos = 0
+					self.__ino = stats.st_ino
-		# Sets the file pointer to the last position.
+					self.__pos = 0
-		self.__handler.seek(self.__pos)
+				self.__hash = myHash
+			# if nothing to read from file yet (empty or no new data):
+			if forcePos is not None:
+				self.__pos = forcePos
+			elif stats.st_size <= self.__pos:
+				return False
+			# Sets the file pointer to the last position.
+			h.seek(self.__pos)
+			# leave file open (to read content):
+			self.__handler = h; h = None
+		finally:
+			# close (no content or error only)
+			if h:
+				h.close(); h = None
 		return True
 
 	def seek(self, offs, endLine=True):
 		h = self.__handler
+		if h is None:
+			self.open(offs)
+			h = self.__handler
 		# seek to given position
 		h.seek(offs, 0)
 		# goto end of next line
@@ -1291,38 +1402,98 @@ class FileContainer:
 		try:
 			return line.decode(enc, 'strict')
 		except (UnicodeDecodeError, UnicodeEncodeError) as e:
+			# avoid warning if got incomplete end of line (e. g. '\n' in "...[0A" followed by "00]..." for utf-16le:
+			if (e.end == len(line) and line[e.start] in b'\r\n'):
+				return line[0:e.start].decode(enc, 'replace')
 			global _decode_line_warn
-			lev = logging.DEBUG
+			lev = 7
-			if _decode_line_warn.get(filename, 0) <= MyTime.time():
+			if not _decode_line_warn.get(filename, 0):
 				lev = logging.WARNING
-				_decode_line_warn[filename] = MyTime.time() + 24*60*60
+				_decode_line_warn.set(filename, 1)
 			logSys.log(lev,
-				"Error decoding line from '%s' with '%s'."
+				"Error decoding line from '%s' with '%s'.", filename, enc)
-				" Consider setting logencoding=utf-8 (or another appropriate"
+			if logSys.getEffectiveLevel() <= lev:
-				" encoding) for this jail. Continuing"
+				logSys.log(lev,
-				" to process line ignoring invalid characters: %r",
+					"Consider setting logencoding to appropriate encoding for this jail. "
-				filename, enc, line)
+					"Continuing to process line ignoring invalid characters: %r",
+					line)
 			# decode with replacing error chars:
 			line = line.decode(enc, 'replace')
 		return line
 
-	def readline(self):
+	def readline(self, complete=True):
+		"""Read line from file
+
+		In opposite to pythons readline it doesn't return new-line, 
+		so returns either the line if line is complete (and complete=True) or None
+		if line is not complete (and complete=True) or there is no content to read.
+		If line is complete (and complete is True), it also shift current known 
+		position to begin of next line.
+
+		Also it is safe against interim new-line bytes (e. g. part of multi-byte char)
+		in given encoding.
+		"""
 		if self.__handler is None:
 			return ""
-		return FileContainer.decode_line(
+		# read raw bytes up to \n char:
-			self.getFileName(), self.getEncoding(), self.__handler.readline())
+		b = self.__handler.readline()
+		if not b:
+			return None
+		bl = len(b)
+		# convert to log-encoding (new-line char could disappear if it is part of multi-byte sequence):
+		r = FileContainer.decode_line(
+			self.getFileName(), self.getEncoding(), b)
+		# trim new-line at end and check the line was written complete (contains a new-line):
+		l = r.rstrip('\r\n')
+		if complete:
+			if l == r:
+				# try to fill buffer in order to find line-end in log encoding:
+				fnd = 0
+				while 1:
+					r = self.__handler.readline()
+					if not r:
+						break
+					b += r
+					bl += len(r)
+					# convert to log-encoding:
+					r = FileContainer.decode_line(
+						self.getFileName(), self.getEncoding(), b)
+					# ensure new-line is not in the middle (buffered 2 strings, e. g. in utf-16le it is "...[0A"+"00]..."):
+					e = r.find('\n')
+					if e >= 0 and e != len(r)-1:
+						l, r = r[0:e], r[0:e+1]
+						# back to bytes and get offset to seek after NL:
+						r = r.encode(self.getEncoding(), 'replace')
+						self.__handler.seek(-bl+len(r), 1)
+						return l
+					# trim new-line at end and check the line was written complete (contains a new-line):
+					l = r.rstrip('\r\n')
+					if l != r:
+						return l
+				if self.waitForLineEnd:
+					# not fulfilled - seek back and return:
+					self.__handler.seek(-bl, 1)
+					return None
+		return l
 
 	def close(self):
-		if not self.__handler is None:
+		if self.__handler is not None:
-			# Saves the last position.
+			# Saves the last real position.
 			self.__pos = self.__handler.tell()
 			# Closes the file.
 			self.__handler.close()
 			self.__handler = None
-		## print "D: Closed %s with pos %d" % (handler, self.__pos)
-		## sys.stdout.flush()
 
-_decode_line_warn = {}
+	def __iter__(self):
+		return self
+	def next(self):
+		line = self.readline()
+		if line is None:
+			self.close()
+			raise StopIteration
+		return line
+
+_decode_line_warn = Utils.Cache(maxCount=1000, maxTime=24*60*60);
 
 
 ##

fail2ban/server/filtergamin.py

@@ -55,7 +55,6 @@ class FilterGamin(FileFilter):
 
 	def __init__(self, jail):
 		FileFilter.__init__(self, jail)
-		self.__modified = False
 		# Gamin monitor
 		self.monitor = gamin.WatchMonitor()
 		fd = self.monitor.get_fd()
@@ -67,21 +66,9 @@ class FilterGamin(FileFilter):
 		logSys.log(4, "Got event: " + repr(event) + " for " + path)
 		if event in (gamin.GAMCreated, gamin.GAMChanged, gamin.GAMExists):
 			logSys.debug("File changed: " + path)
-			self.__modified = True
 
 		self.ticks += 1
-		self._process_file(path)
-
-	def _process_file(self, path):
-		"""Process a given file
-
-		TODO -- RF:
-		this is a common logic and must be shared/provided by FileFilter
-		"""
 		self.getFailures(path)
-		if not self.banASAP: # pragma: no cover
-			self.performBan()
-		self.__modified = False
 
 	##
 	# Add a log file path
@@ -128,6 +115,9 @@ class FilterGamin(FileFilter):
 			Utils.wait_for(lambda: not self.active or self._handleEvents(),
 				self.sleeptime)
 			self.ticks += 1
+			if self.ticks % 10 == 0:
+				self.performSvc()
+
 		logSys.debug("[%s] filter terminated", self.jailName)
 		return True
 

fail2ban/server/filterpoll.py

@@ -27,9 +27,7 @@ __license__ = "GPL"
 import os
 import time
 
-from .failmanager import FailManagerEmpty
 from .filter import FileFilter
-from .mytime import MyTime
 from .utils import Utils
 from ..helpers import getLogger, logging
 
@@ -55,7 +53,6 @@ class FilterPoll(FileFilter):
 
 	def __init__(self, jail):
 		FileFilter.__init__(self, jail)
-		self.__modified = False
 		## The time of the last modification of the file.
 		self.__prevStats = dict()
 		self.__file404Cnt = dict()
@@ -111,15 +108,14 @@ class FilterPoll(FileFilter):
 				modlst = []
 				Utils.wait_for(lambda: not self.active or self.getModified(modlst),
 					self.sleeptime)
+				if not self.active: # pragma: no cover - timing
+					break
 				for filename in modlst:
 					self.getFailures(filename)
-					self.__modified = True
 
 				self.ticks += 1
-				if self.__modified:
+				if self.ticks % 10 == 0:
-					if not self.banASAP: # pragma: no cover
+					self.performSvc()
-						self.performBan()
-					self.__modified = False
 			except Exception as e: # pragma: no cover
 				if not self.active: # if not active - error by stop...
 					break
@@ -140,7 +136,7 @@ class FilterPoll(FileFilter):
 		try:
 			logStats = os.stat(filename)
 			stats = logStats.st_mtime, logStats.st_ino, logStats.st_size
-			pstats = self.__prevStats.get(filename, (0))
+			pstats = self.__prevStats.get(filename, (0,))
 			if logSys.getEffectiveLevel() <= 4:
 				# we do not want to waste time on strftime etc if not necessary
 				dt = logStats.st_mtime - pstats[0]

fail2ban/server/filterpyinotify.py

@@ -75,7 +75,6 @@ class FilterPyinotify(FileFilter):
 
 	def __init__(self, jail):
 		FileFilter.__init__(self, jail)
-		self.__modified = False
 		# Pyinotify watch manager
 		self.__monitor = pyinotify.WatchManager()
 		self.__notifier = None
@@ -140,9 +139,6 @@ class FilterPyinotify(FileFilter):
 		"""
 		if not self.idle:
 			self.getFailures(path)
-			if not self.banASAP: # pragma: no cover
-				self.performBan()
-			self.__modified = False
 
 	def _addPending(self, path, reason, isDir=False):
 		if path not in self.__pending:
@@ -352,9 +348,14 @@ class FilterPyinotify(FileFilter):
 					if not self.active: break
 					self.__notifier.read_events()
 
+				self.ticks += 1
+
 				# check pending files/dirs (logrotate ready):
-				if not self.idle:
+				if self.idle:
-					self._checkPending()
+					continue
+				self._checkPending()
+				if self.ticks % 10 == 0:
+					self.performSvc()
 
 			except Exception as e: # pragma: no cover
 				if not self.active: # if not active - error by stop...
@@ -364,8 +365,6 @@ class FilterPyinotify(FileFilter):
 				# incr common error counter:
 				self.commonError()
 			
-			self.ticks += 1
-
 		logSys.debug("[%s] filter exited (pyinotifier)", self.jailName)
 		self.__notifier = None
 

fail2ban/server/filtersystemd.py

@@ -94,6 +94,11 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 			# be sure all journal types will be opened if files specified (don't set flags):
 			if 'files' not in args or not len(args['files']):
 				args['flags'] = 4
+				
+		try:
+			args['namespace'] = kwargs.pop('namespace')
+		except KeyError:
+			pass
 
 		return args
 
@@ -317,13 +322,12 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 							break
 					else:
 						break
-				if self.__modified:
+				self.__modified = 0
-					if not self.banASAP: # pragma: no cover
+				if self.ticks % 10 == 0:
-						self.performBan()
+					self.performSvc()
-					self.__modified = 0
+				# update position in log (time and iso string):
-					# update position in log (time and iso string):
+				if self.jail.database is not None:
-					if self.jail.database is not None:
+					self.jail.database.updateJournal(self.jail, 'systemd-journal', line[1], line[0][1])
-						self.jail.database.updateJournal(self.jail, 'systemd-journal', line[1], line[0][1])
 			except Exception as e: # pragma: no cover
 				if not self.active: # if not active - error by stop...
 					break

fail2ban/server/ipdns.py

@@ -169,27 +169,31 @@ class DNSUtils:
 		DNSUtils.CACHE_ipToName.set(key, name)
 		return name
 
+	# key find cached own hostnames (this tuple-key cannot be used elsewhere):
+	_getSelfNames_key = ('self','dns')
+
 	@staticmethod
 	def getSelfNames():
 		"""Get own host names of self"""
-		# try find cached own hostnames (this tuple-key cannot be used elsewhere):
+		# try find cached own hostnames:
-		key = ('self','dns')
+		names = DNSUtils.CACHE_ipToName.get(DNSUtils._getSelfNames_key)
-		names = DNSUtils.CACHE_ipToName.get(key)
 		# get it using different ways (a set with names of localhost, hostname, fully qualified):
 		if names is None:
 			names = set([
 				'localhost', DNSUtils.getHostname(False), DNSUtils.getHostname(True)
 			]) - set(['']) # getHostname can return ''
 		# cache and return :
-		DNSUtils.CACHE_ipToName.set(key, names)
+		DNSUtils.CACHE_ipToName.set(DNSUtils._getSelfNames_key, names)
 		return names
 
+	# key to find cached own IPs (this tuple-key cannot be used elsewhere):
+	_getSelfIPs_key = ('self','ips')
+
 	@staticmethod
 	def getSelfIPs():
 		"""Get own IP addresses of self"""
-		# try find cached own IPs (this tuple-key cannot be used elsewhere):
+		# to find cached own IPs:
-		key = ('self','ips')
+		ips = DNSUtils.CACHE_nameToIp.get(DNSUtils._getSelfIPs_key)
-		ips = DNSUtils.CACHE_nameToIp.get(key)
 		# get it using different ways (a set with IPs of localhost, hostname, fully qualified):
 		if ips is None:
 			ips = set()
@@ -199,13 +203,30 @@ class DNSUtils:
 				except Exception as e: # pragma: no cover
 					logSys.warning("Retrieving own IPs of %s failed: %s", hostname, e)
 		# cache and return :
-		DNSUtils.CACHE_nameToIp.set(key, ips)
+		DNSUtils.CACHE_nameToIp.set(DNSUtils._getSelfIPs_key, ips)
 		return ips
 
+	_IPv6IsAllowed = None
+
+	@staticmethod
+	def setIPv6IsAllowed(value):
+		DNSUtils._IPv6IsAllowed = value
+		logSys.debug("IPv6 is %s", ('on' if value else 'off') if value is not None else 'auto')
+		return value
+
+	# key to find cached value of IPv6 allowance (this tuple-key cannot be used elsewhere):
+	_IPv6IsAllowed_key = ('self','ipv6-allowed')
+
 	@staticmethod
 	def IPv6IsAllowed():
-		# return os.path.exists("/proc/net/if_inet6") || any((':' in ip) for ip in DNSUtils.getSelfIPs())
+		if DNSUtils._IPv6IsAllowed is not None:
-		return any((':' in ip.ntoa) for ip in DNSUtils.getSelfIPs())
+			return DNSUtils._IPv6IsAllowed
+		v = DNSUtils.CACHE_nameToIp.get(DNSUtils._IPv6IsAllowed_key)
+		if v is not None:
+			return v
+		v = any((':' in ip.ntoa) for ip in DNSUtils.getSelfIPs())
+		DNSUtils.CACHE_nameToIp.set(DNSUtils._IPv6IsAllowed_key, v)
+		return v
 
 
 ##
@@ -337,7 +358,7 @@ class IPAddr(object):
 		return repr(self.ntoa)
 
 	def __str__(self):
-		return self.ntoa
+		return self.ntoa if isinstance(self.ntoa, basestring) else str(self.ntoa)
 
 	def __reduce__(self):
 		"""IPAddr pickle-handler, that simply wraps IPAddr to the str
@@ -379,6 +400,12 @@ class IPAddr(object):
 		"""
 		return self._family != socket.AF_UNSPEC
 
+	@property
+	def isSingle(self):
+		"""Returns whether the object is a single IP address (not DNS and subnet)
+		"""
+		return self._plen == {socket.AF_INET: 32, socket.AF_INET6: 128}.get(self._family, -1000)
+
 	def __eq__(self, other):
 		if self._family == IPAddr.CIDR_RAW and not isinstance(other, IPAddr):
 			return self._raw == other
@@ -511,6 +538,11 @@ class IPAddr(object):
 		
 		return (self.addr & mask) == net.addr
 
+	def contains(self, ip):
+		"""Return whether the object (as network) contains given IP
+		"""
+		return isinstance(ip, IPAddr) and (ip == self or ip.isInNet(self))
+
 	# Pre-calculated map: addr to maskplen
 	def __getMaskMap():
 		m6 = (1 << 128)-1

fail2ban/server/jails.py

@@ -22,7 +22,10 @@ __copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2013- Yaroslav Halchenko"
 __license__ = "GPL"
 
 from threading import Lock
-from collections import Mapping
+try:
+	from collections.abc import Mapping
+except ImportError:
+	from collections import Mapping
 
 from ..exceptions import DuplicateJailException, UnknownJailException
 from .jail import Jail

fail2ban/server/mytime.py

@@ -121,8 +121,11 @@ class MyTime:
 
 		@return ISO-capable string representation of given unixTime
 		"""
-		return datetime.datetime.fromtimestamp(
+		# consider end of 9999th year (in GMT+23 to avoid year overflow in other TZ)
-			unixTime).replace(microsecond=0).strftime(format)
+		dt = datetime.datetime.fromtimestamp(
+			unixTime).replace(microsecond=0
+		) if unixTime < 253402214400 else datetime.datetime(9999, 12, 31, 23, 59, 59)
+		return dt.strftime(format)
 
 	## precreate/precompile primitives used in str2seconds:
 

fail2ban/server/observer.py

@@ -87,7 +87,7 @@ class ObserverThread(JailThread):
 		except KeyError:
 			raise KeyError("Invalid event index : %s" % i)
 
-	def __delitem__(self, name):
+	def __delitem__(self, i):
 		try:
 			del self._queue[i]
 		except KeyError:
@@ -232,7 +232,7 @@ class ObserverThread(JailThread):
 					if self._paused:
 						continue
 				else:
-					## notify event deleted (shutdown) - just sleep a litle bit (waiting for shutdown events, prevent high cpu usage)
+					## notify event deleted (shutdown) - just sleep a little bit (waiting for shutdown events, prevent high cpu usage)
 					time.sleep(ObserverThread.DEFAULT_SLEEP_INTERVAL)
 					## stop by shutdown and empty queue :
 					if not self.is_full:

fail2ban/server/server.py

@@ -34,7 +34,7 @@ import sys
 
 from .observer import Observers, ObserverThread
 from .jails import Jails
-from .filter import FileFilter, JournalFilter
+from .filter import DNSUtils, FileFilter, JournalFilter
 from .transmitter import Transmitter
 from .asyncserver import AsyncServer, AsyncServerException
 from .. import version
@@ -293,6 +293,11 @@ class Server:
 			for name in self.__jails.keys():
 				self.delJail(name, stop=False, join=True)
 
+	def clearCaches(self):
+		# we need to clear caches, to be able to recognize new IPs/families etc:
+		DNSUtils.CACHE_nameToIp.clear()
+		DNSUtils.CACHE_ipToName.clear()	
+
 	def reloadJails(self, name, opts, begin):
 		if begin:
 			# begin reload:
@@ -314,6 +319,8 @@ class Server:
 						if "--restart" in opts:
 							self.stopJail(name)
 				else:
+					# invalidate caches by reload
+					self.clearCaches()
 					# first unban all ips (will be not restored after (re)start):
 					if "--unban" in opts:
 						self.setUnbanIP()
@@ -540,6 +547,32 @@ class Server:
 			cnt += jail.actions.removeBannedIP(value, ifexists=ifexists)
 		return cnt
 		
+	def banned(self, name=None, ids=None):
+		if name is not None:
+			# single jail:
+			jails = [self.__jails[name]]
+		else:
+			# in all jails:
+			jails = self.__jails.values()
+		# check banned ids:
+		res = []
+		if name is None and ids:
+			for ip in ids:
+				ret = []
+				for jail in jails:
+					if jail.actions.getBanned([ip]):
+						ret.append(jail.name)
+				res.append(ret)
+		else:
+			for jail in jails:
+				ret = jail.actions.getBanned(ids)
+				if name is not None:
+					return ret
+					res.append(ret)
+				else:
+					res.append({jail.name: ret})
+		return res
+		
 	def getBanTime(self, name):
 		return self.__jails[name].actions.getBanTime()
 
@@ -777,6 +810,11 @@ class Server:
 				logSys.info("flush performed on %s" % self.__logTarget)
 			return "flushed"
 			
+	@staticmethod
+	def setIPv6IsAllowed(value):
+		value = _as_bool(value) if value != 'auto' else None
+		return DNSUtils.setIPv6IsAllowed(value)
+
 	def setThreadOptions(self, value):
 		for o, v in value.iteritems():
 			if o == 'stacksize':

fail2ban/server/strptime.py

@@ -30,17 +30,6 @@ locale_time = LocaleTime()
 TZ_ABBR_RE = r"[A-Z](?:[A-Z]{2,4})?"
 FIXED_OFFSET_TZ_RE = re.compile(r"(%s)?([+-][01]\d(?::?\d{2})?)?$" % (TZ_ABBR_RE,))
 
-def _getYearCentRE(cent=(0,3), distance=3, now=(MyTime.now(), MyTime.alternateNow)):
-	""" Build century regex for last year and the next years (distance).
-		
-	Thereby respect possible run in the test-cases (alternate date used there)
-	"""
-	cent = lambda year, f=cent[0], t=cent[1]: str(year)[f:t]
-	exprset = set( cent(now[0].year + i) for i in (-1, distance) )
-	if len(now) and now[1]:
-		exprset |= set( cent(now[1].year + i) for i in (-1, distance) )
-	return "(?:%s)" % "|".join(exprset) if len(exprset) > 1 else "".join(exprset)
-
 timeRE = TimeRE()
 
 # %k - one- or two-digit number giving the hour of the day (0-23) on a 24-hour clock,
@@ -63,20 +52,68 @@ timeRE['z'] = r"(?P<z>Z|UTC|GMT|[+-][01]\d(?::?\d{2})?)"
 timeRE['ExZ'] = r"(?P<Z>%s)" % (TZ_ABBR_RE,)
 timeRE['Exz'] = r"(?P<z>(?:%s)?[+-][01]\d(?::?\d{2})?|%s)" % (TZ_ABBR_RE, TZ_ABBR_RE)
 
+# overwrite default patterns, since they can be non-optimal:
+timeRE['d'] = r"(?P<d>[1-2]\d|[0 ]?[1-9]|3[0-1])"
+timeRE['m'] = r"(?P<m>0?[1-9]|1[0-2])"
+timeRE['Y'] = r"(?P<Y>\d{4})"
+timeRE['H'] = r"(?P<H>[0-1]?\d|2[0-3])"
+timeRE['M'] = r"(?P<M>[0-5]?\d)"
+timeRE['S'] = r"(?P<S>[0-5]?\d|6[0-1])"
+
 # Extend build-in TimeRE with some exact patterns
 # exact two-digit patterns:
-timeRE['Exd'] = r"(?P<d>3[0-1]|[1-2]\d|0[1-9])"
+timeRE['Exd'] = r"(?P<d>[1-2]\d|0[1-9]|3[0-1])"
-timeRE['Exm'] = r"(?P<m>1[0-2]|0[1-9])"
+timeRE['Exm'] = r"(?P<m>0[1-9]|1[0-2])"
-timeRE['ExH'] = r"(?P<H>2[0-3]|[0-1]\d)"
+timeRE['ExH'] = r"(?P<H>[0-1]\d|2[0-3])"
-timeRE['Exk'] = r" ?(?P<H>2[0-3]|[0-1]\d|\d)"
+timeRE['Exk'] = r" ?(?P<H>[0-1]?\d|2[0-3])"
 timeRE['Exl'] = r" ?(?P<I>1[0-2]|\d)"
 timeRE['ExM'] = r"(?P<M>[0-5]\d)"
-timeRE['ExS'] = r"(?P<S>6[0-1]|[0-5]\d)"
+timeRE['ExS'] = r"(?P<S>[0-5]\d|6[0-1])"
-# more precise year patterns, within same century of last year and
+
-# the next 3 years (for possible long uptime of fail2ban); thereby
+def _updateTimeRE():
-# respect possible run in the test-cases (alternate date used there):
+	def _getYearCentRE(cent=(0,3), distance=3, now=(MyTime.now(), MyTime.alternateNow)):
-timeRE['ExY'] = r"(?P<Y>%s\d)" % _getYearCentRE(cent=(0,3), distance=3)
+		""" Build century regex for last year and the next years (distance).
-timeRE['Exy'] = r"(?P<y>%s\d)" % _getYearCentRE(cent=(2,3), distance=3)
+			
+		Thereby respect possible run in the test-cases (alternate date used there)
+		"""
+		cent = lambda year, f=cent[0], t=cent[1]: str(year)[f:t]
+		def grp(exprset):
+			c = None
+			if len(exprset) > 1:
+				for i in exprset:
+					if c is None or i[0:-1] == c:
+						c = i[0:-1]
+					else:
+						c = None
+						break
+				if not c:
+					for i in exprset:
+						if c is None or i[0] == c:
+							c = i[0]
+						else:
+							c = None
+							break
+				if c:
+					return "%s%s" % (c, grp([i[len(c):] for i in exprset]))
+			return ("(?:%s)" % "|".join(exprset) if len(exprset[0]) > 1 else "[%s]" % "".join(exprset)) \
+				if len(exprset) > 1 else "".join(exprset)
+		exprset = set( cent(now[0].year + i) for i in (-1, distance) )
+		if len(now) > 1 and now[1]:
+			exprset |= set( cent(now[1].year + i) for i in xrange(-1, now[0].year-now[1].year+1, distance) )
+		return grp(sorted(list(exprset)))
+
+	# more precise year patterns, within same century of last year and
+	# the next 3 years (for possible long uptime of fail2ban); thereby
+	# respect possible run in the test-cases (alternate date used there):
+	if MyTime.alternateNowTime != 0:
+		timeRE['ExY'] = r"(?P<Y>%s\d)" % _getYearCentRE(cent=(0,3), distance=3)
+		timeRE['Exy'] = r"(?P<y>%s\d)" % _getYearCentRE(cent=(2,3), distance=3)
+	else: # accept years: 19xx|2xxx up to current century
+		timeRE['ExY'] = r"(?P<Y>(?:19\d{2}|%s\d))" % _getYearCentRE(cent=(0,3), distance=3, 
+			now=(MyTime.now(), datetime.datetime.fromtimestamp(978393600)))
+		timeRE['Exy'] = r"(?P<y>\d{2})"
+
+_updateTimeRE()
 
 def getTimePatternRE():
 	keys = timeRE.keys()
@@ -168,9 +205,9 @@ def reGroupDictStrptime(found_dict, msec=False, default_tz=None):
 	"""
 
 	now = \
-	year = month = day = hour = minute = tzoffset = \
+	year = month = day = tzoffset = \
 	weekday = julian = week_of_year = None
-	second = fraction = 0
+	hour = minute = second = fraction = 0
 	for key, val in found_dict.iteritems():
 		if val is None: continue
 		# Directives not explicitly handled below:
@@ -234,16 +271,12 @@ def reGroupDictStrptime(found_dict, msec=False, default_tz=None):
 			week_of_year = int(val)
 			# U starts week on Sunday, W - on Monday
 			week_of_year_start = 6 if key == 'U' else 0
-		elif key == 'z':
+		elif key in ('z', 'Z'):
 			z = val
 			if z in ("Z", "UTC", "GMT"):
 				tzoffset = 0
 			else:
 				tzoffset = zone2offset(z, 0); # currently offset-based only
-		elif key == 'Z':
-			z = val
-			if z in ("UTC", "GMT"):
-				tzoffset = 0
 
 	# Fail2Ban will assume it's this year
 	assume_year = False
@@ -291,9 +324,8 @@ def reGroupDictStrptime(found_dict, msec=False, default_tz=None):
 			date_result -= datetime.timedelta(days=1)
 	if assume_year:
 		if not now: now = MyTime.now()
-		if date_result > now:
+		if date_result > now + datetime.timedelta(days=1): # ignore by timezone issues (+24h)
-			# Could be last year?
+			# assume last year - also reset month and day as it's not yesterday...
-			# also reset month and day as it's not yesterday...
 			date_result = date_result.replace(
 				year=year-1, month=month, day=day)
 

fail2ban/server/transmitter.py

@@ -118,6 +118,9 @@ class Transmitter:
 			if len(value) == 1 and value[0] == "--all":
 				return self.__server.setUnbanIP()
 			return self.__server.setUnbanIP(None, value)
+		elif name == "banned":
+			# check IP is banned in all jails:
+			return self.__server.banned(None, command[1:])
 		elif name == "echo":
 			return command[1:]
 		elif name == "server-status":
@@ -170,6 +173,11 @@ class Transmitter:
 				return self.__server.getSyslogSocket()
 			else:
 				raise Exception("Failed to change syslog socket")
+		elif name == "allowipv6":
+			value = command[1]
+			self.__server.setIPv6IsAllowed(value)
+			if self.__quiet: return
+			return value
 		#Thread
 		elif name == "thread":
 			value = command[1]
@@ -274,7 +282,8 @@ class Transmitter:
 			value = command[2]
 			self.__server.setPrefRegex(name, value)
 			if self.__quiet: return
-			return self.__server.getPrefRegex(name)
+			v = self.__server.getPrefRegex(name)
+			return v.getRegex() if v else ""
 		elif command[1] == "addfailregex":
 			value = command[2]
 			self.__server.addFailRegex(name, value, multiple=multiple)
@@ -430,7 +439,10 @@ class Transmitter:
 				return None
 			else:
 				return db.purgeage
-		# Filter
+		# Jail, Filter
+		elif command[1] == "banned":
+			# check IP is banned in all jails:
+			return self.__server.banned(name, command[2:])
 		elif command[1] == "logpath":
 			return self.__server.getLogPath(name)
 		elif command[1] == "logencoding":
@@ -446,7 +458,8 @@ class Transmitter:
 		elif command[1] == "ignorecache":
 			return self.__server.getIgnoreCache(name)
 		elif command[1] == "prefregex":
-			return self.__server.getPrefRegex(name)
+			v = self.__server.getPrefRegex(name)
+			return v.getRegex() if v else ""
 		elif command[1] == "failregex":
 			return self.__server.getFailRegex(name)
 		elif command[1] == "ignoreregex":

fail2ban/server/utils.py

@@ -125,6 +125,10 @@ class Utils():
 			with self.__lock:
 				self._cache.pop(k, None)
 
+		def clear(self):
+			with self.__lock:
+				self._cache.clear()
+
 
 	@staticmethod
 	def setFBlockMode(fhandle, value):
@@ -328,11 +332,9 @@ class Utils():
 					timeout_expr = lambda: time.time() > time0
 				else:
 					timeout_expr = timeout
-				if not interval:
-					interval = Utils.DEFAULT_SLEEP_INTERVAL
 			if timeout_expr():
 				break
-			stm = min(stm + interval, Utils.DEFAULT_SLEEP_TIME)
+			stm = min(stm + (interval or Utils.DEFAULT_SLEEP_INTERVAL), Utils.DEFAULT_SLEEP_TIME)
 			time.sleep(stm)
 		return ret
 

fail2ban/tests/action_d/test_badips.py

@@ -1,157 +0,0 @@
-# emacs: -*- mode: python; py-indent-offset: 4; indent-tabs-mode: t -*-
-# vi: set ft=python sts=4 ts=4 sw=4 noet :
-
-# This file is part of Fail2Ban.
-#
-# Fail2Ban is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2 of the License, or
-# (at your option) any later version.
-#
-# Fail2Ban is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Fail2Ban; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
-
-import os
-import unittest
-import sys
-from functools import wraps
-from socket import timeout
-from ssl import SSLError
-
-from ..actiontestcase import CallingMap
-from ..dummyjail import DummyJail
-from ..servertestcase import IPAddr
-from ..utils import LogCaptureTestCase, CONFIG_DIR
-
-if sys.version_info >= (3, ): # pragma: 2.x no cover
-	from urllib.error import HTTPError, URLError
-else: # pragma: 3.x no cover
-	from urllib2 import HTTPError, URLError
-
-def skip_if_not_available(f):
-	"""Helper to decorate tests to skip in case of timeout/http-errors like "502 bad gateway".
-	"""
-	@wraps(f)
-	def wrapper(self, *args):
-		try:
-			return f(self, *args)
-		except (SSLError, HTTPError, URLError, timeout) as e: # pragma: no cover - timeout/availability issues
-			if not isinstance(e, timeout) and 'timed out' not in str(e):
-				if not hasattr(e, 'code') or e.code > 200 and e.code <= 404:
-					raise
-			raise unittest.SkipTest('Skip test because of %s' % e)
-	return wrapper
-
-if sys.version_info >= (2,7): # pragma: no cover - may be unavailable
-	class BadIPsActionTest(LogCaptureTestCase):
-
-		available = True, None
-		pythonModule = None
-		modAction = None
-
-		@skip_if_not_available
-		def setUp(self):
-			"""Call before every test case."""
-			super(BadIPsActionTest, self).setUp()
-			unittest.F2B.SkipIfNoNetwork()
-
-			self.jail = DummyJail()
-
-			self.jail.actions.add("test")
-
-			pythonModuleName = os.path.join(CONFIG_DIR, "action.d", "badips.py")
-
-			# check availability (once if not alive, used shorter timeout as in test cases):
-			if BadIPsActionTest.available[0]:
-				if not BadIPsActionTest.modAction:
-					if not BadIPsActionTest.pythonModule:
-						BadIPsActionTest.pythonModule = self.jail.actions._load_python_module(pythonModuleName)
-					BadIPsActionTest.modAction = BadIPsActionTest.pythonModule.Action
-					self.jail.actions._load_python_module(pythonModuleName)
-				BadIPsActionTest.available = BadIPsActionTest.modAction.isAvailable(timeout=2 if unittest.F2B.fast else 30)
-			if not BadIPsActionTest.available[0]:
-				raise unittest.SkipTest('Skip test because service is not available: %s' % BadIPsActionTest.available[1])
-
-			self.jail.actions.add("badips", pythonModuleName, initOpts={
-				'category': "ssh",
-				'banaction': "test",
-				'age': "2w",
-				'score': 5,
-				#'key': "fail2ban-test-suite",
-				#'bankey': "fail2ban-test-suite",
-				'timeout': (3 if unittest.F2B.fast else 60),
-				})
-			self.action = self.jail.actions["badips"]
-
-		def tearDown(self):
-			"""Call after every test case."""
-			# Must cancel timer!
-			if self.action._timer:
-				self.action._timer.cancel()
-			super(BadIPsActionTest, self).tearDown()
-
-		@skip_if_not_available
-		def testCategory(self):
-			categories = self.action.getCategories()
-			self.assertIn("ssh", categories)
-			self.assertTrue(len(categories) >= 10)
-
-			self.assertRaises(
-				ValueError, setattr, self.action, "category",
-				"invalid-category")
-
-			# Not valid for reporting category...
-			self.assertRaises(
-				ValueError, setattr, self.action, "category", "mail")
-			# but valid for blacklisting.
-			self.action.bancategory = "mail"
-
-		@skip_if_not_available
-		def testScore(self):
-			self.assertRaises(ValueError, setattr, self.action, "score", -5)
-			self.action.score = 3
-			self.action.score = "3"
-
-		@skip_if_not_available
-		def testBanaction(self):
-			self.assertRaises(
-				ValueError, setattr, self.action, "banaction",
-				"invalid-action")
-			self.action.banaction = "test"
-
-		@skip_if_not_available
-		def testUpdateperiod(self):
-			self.assertRaises(
-				ValueError, setattr, self.action, "updateperiod", -50)
-			self.assertRaises(
-				ValueError, setattr, self.action, "updateperiod", 0)
-			self.action.updateperiod = 900
-			self.action.updateperiod = "900"
-
-		@skip_if_not_available
-		def testStartStop(self):
-			self.action.start()
-			self.assertTrue(len(self.action._bannedips) > 10,
-				"%s is fewer as 10: %r" % (len(self.action._bannedips), self.action._bannedips))
-			self.action.stop()
-			self.assertTrue(len(self.action._bannedips) == 0)
-
-		@skip_if_not_available
-		def testBanIP(self):
-			aInfo = CallingMap({
-				'ip': IPAddr('192.0.2.1')
-			})
-			self.action.ban(aInfo)
-			self.assertLogged('badips.com: ban', wait=True)
-			self.pruneLog()
-			# produce an error using wrong category/IP:
-			self.action._category = 'f2b-this-category-dont-available-test-suite-only'
-			aInfo['ip'] = ''
-			self.assertRaises(BadIPsActionTest.pythonModule.HTTPError, self.action.ban, aInfo)
-			self.assertLogged('IP is invalid', 'invalid category', wait=True, all=False)

fail2ban/tests/actionstestcase.py

@@ -96,6 +96,8 @@ class ExecuteActions(LogCaptureTestCase):
 		self.assertLogged("stdout: %r" % 'ip flush', "stdout: %r" % 'ip stop')
 		self.assertEqual(self.__actions.status(),[("Currently banned", 0 ),
                ("Total banned", 0 ), ("Banned IP list", [] )])
+		self.assertEqual(self.__actions.status('short'),[("Currently banned", 0 ),
+               ("Total banned", 0 )])
 
 	def testAddActionPython(self):
 		self.__actions.add(

fail2ban/tests/actiontestcase.py

@@ -252,7 +252,7 @@ class CommandActionTest(LogCaptureTestCase):
 		delattr(self.__action, 'ac')
 		# produce self-referencing query except:
 		self.assertRaisesRegexp(ValueError, r"possible self referencing definitions in query",
-			lambda: self.__action.replaceTag("<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x>>>>>>>>>>>>>>>>>>>>>", 
+			lambda: self.__action.replaceTag("<x"*30+">"*30,
 				self.__action._properties, conditional="family=inet6")
 		)
 

fail2ban/tests/banmanagertestcase.py

@@ -29,6 +29,7 @@ import unittest
 from .utils import setUpMyTime, tearDownMyTime
 
 from ..server.banmanager import BanManager
+from ..server.ipdns import DNSUtils
 from ..server.ticket import BanTicket
 
 class AddFailure(unittest.TestCase):
@@ -154,6 +155,21 @@ class AddFailure(unittest.TestCase):
 		finally:
 			self.__banManager.setBanTime(btime)
 
+	def testBanList(self):
+		tickets = [
+			BanTicket('192.0.2.1', 1167605999.0),
+			BanTicket('192.0.2.2', 1167605999.0),
+		]
+		tickets[1].setBanTime(-1)
+		for t in tickets:
+			self.__banManager.addBanTicket(t)
+		self.assertSortedEqual(self.__banManager.getBanList(ordered=True, withTime=True),
+			[
+			  '192.0.2.1 \t2006-12-31 23:59:59 + 600 = 2007-01-01 00:09:59',
+			  '192.0.2.2 \t2006-12-31 23:59:59 + -1 = 9999-12-31 23:59:59'
+			]
+		)
+
 
 class StatusExtendedCymruInfo(unittest.TestCase):
 	def setUp(self):
@@ -161,10 +177,10 @@ class StatusExtendedCymruInfo(unittest.TestCase):
 		super(StatusExtendedCymruInfo, self).setUp()
 		unittest.F2B.SkipIfNoNetwork()
 		setUpMyTime()
-		self.__ban_ip = "93.184.216.34"
+		self.__ban_ip = iter(DNSUtils.dnsToIp("resolver1.opendns.com")).next()
-		self.__asn = "15133"
+		self.__asn = "36692"
-		self.__country = "EU"
+		self.__country = "US"
-		self.__rir = "ripencc"
+		self.__rir = "arin"
 		ticket = BanTicket(self.__ban_ip, 1167605999.0)
 		self.__banManager = BanManager()
 		self.assertTrue(self.__banManager.addBanTicket(ticket))

fail2ban/tests/clientreadertestcase.py

@@ -87,6 +87,21 @@ option = %s
 		self.assertTrue(self.c.read(f))	# we got some now
 		return self.c.getOptions('section', [("int", 'option')])['option']
 
+	def testConvert(self):
+		self.c.add_section("Definition")
+		self.c.set("Definition", "a", "1")
+		self.c.set("Definition", "b", "1")
+		self.c.set("Definition", "c", "test")
+		opts = self.c.getOptions("Definition", 
+			(('int', 'a', 0), ('bool', 'b', 0), ('int', 'c', 0)))
+		self.assertSortedEqual(opts, {'a': 1, 'b': True, 'c': 0})
+		opts = self.c.getOptions("Definition", 
+			(('int', 'a'), ('bool', 'b'), ('int', 'c')))
+		self.assertSortedEqual(opts, {'a': 1, 'b': True, 'c': None})
+		opts = self.c.getOptions("Definition", 
+			{'a': ('int', 0), 'b': ('bool', 0), 'c': ('int', 0)})
+		self.assertSortedEqual(opts, {'a': 1, 'b': True, 'c': 0})
+
 	def testInaccessibleFile(self):
 		f = os.path.join(self.d, "d.conf")  # inaccessible file
 		self._write('d.conf', 0)
@@ -249,6 +264,17 @@ class JailReaderTest(LogCaptureTestCase):
 	def __init__(self, *args, **kwargs):
 		super(JailReaderTest, self).__init__(*args, **kwargs)
 
+	def testSplitWithOptions(self):
+		# covering all separators - new-line and spaces:
+		for sep in ('\n', '\t', ' '):
+			self.assertEqual(splitWithOptions('a%sb' % (sep,)),           ['a',           'b'])
+			self.assertEqual(splitWithOptions('a[x=y]%sb' % (sep,)),      ['a[x=y]',      'b'])
+			self.assertEqual(splitWithOptions('a[x=y][z=z]%sb' % (sep,)), ['a[x=y][z=z]', 'b'])
+			self.assertEqual(splitWithOptions('a[x="y][z"]%sb' % (sep,)), ['a[x="y][z"]', 'b'])
+			self.assertEqual(splitWithOptions('a[x="y z"]%sb' % (sep,)),  ['a[x="y z"]',  'b'])
+			self.assertEqual(splitWithOptions('a[x="y\tz"]%sb' % (sep,)), ['a[x="y\tz"]', 'b'])
+			self.assertEqual(splitWithOptions('a[x="y\nz"]%sb' % (sep,)), ['a[x="y\nz"]', 'b'])
+
 	def testIncorrectJail(self):
 		jail = JailReader('XXXABSENTXXX', basedir=CONFIG_DIR, share_config=CONFIG_DIR_SHARE_CFG)
 		self.assertRaises(ValueError, jail.read)
@@ -355,13 +381,16 @@ class JailReaderTest(LogCaptureTestCase):
 		self.assertEqual(('mail.who_is', {'a':'cat', 'b':'dog'}), extractOptions("mail.who_is[a=cat,b=dog]"))
 		self.assertEqual(('mail--ho_is', {}), extractOptions("mail--ho_is"))
 
-		self.assertEqual(('mail--ho_is', {}), extractOptions("mail--ho_is['s']"))
-		#print(self.getLog())
-		#self.assertLogged("Invalid argument ['s'] in ''s''")
-
 		self.assertEqual(('mail', {'a': ','}), extractOptions("mail[a=',']"))
+		self.assertEqual(('mail', {'a': 'b'}), extractOptions("mail[a=b, ]"))
 
-		#self.assertRaises(ValueError, extractOptions ,'mail-how[')
+		self.assertRaises(ValueError, extractOptions ,'mail-how[')
+
+		self.assertRaises(ValueError, extractOptions, """mail[a="test with interim (wrong) "" quotes"]""")
+		self.assertRaises(ValueError, extractOptions, """mail[a='test with interim (wrong) '' quotes']""")
+		self.assertRaises(ValueError, extractOptions, """mail[a='x, y, z', b=x, y, z]""")
+
+		self.assertRaises(ValueError, extractOptions, """mail['s']""")
 
 		# Empty option
 		option = "abc[]"
@@ -429,8 +458,6 @@ class JailReaderTest(LogCaptureTestCase):
 			('sender', 'f2b-test@example.com'), ('blocklist_de_apikey', 'test-key'), 
 			('action', 
 				'%(action_blocklist_de)s\n'
-				'%(action_badips_report)s\n'
-				'%(action_badips)s\n'
 				'mynetwatchman[port=1234,protocol=udp,agent="%(fail2ban_agent)s"]'
 			),
 		))
@@ -444,16 +471,14 @@ class JailReaderTest(LogCaptureTestCase):
 			if len(cmd) <= 4:
 				continue
 			# differentiate between set and multi-set (wrop it here to single set):
-			if cmd[0] == 'set' and (cmd[4] == 'agent' or cmd[4].endswith('badips.py')):
+			if cmd[0] == 'set' and cmd[4] == 'agent':
 				act.append(cmd)
 			elif cmd[0] == 'multi-set':
 				act.extend([['set'] + cmd[1:4] + o for o in cmd[4] if o[0] == 'agent'])
 		useragent = 'Fail2Ban/%s' % version
-		self.assertEqual(len(act), 4)
+		self.assertEqual(len(act), 2)
 		self.assertEqual(act[0], ['set', 'blocklisttest', 'action', 'blocklist_de', 'agent', useragent])
-		self.assertEqual(act[1], ['set', 'blocklisttest', 'action', 'badips', 'agent', useragent])
+		self.assertEqual(act[1], ['set', 'blocklisttest', 'action', 'mynetwatchman', 'agent', useragent])
-		self.assertEqual(eval(act[2][5]).get('agent', '<wrong>'), useragent)
-		self.assertEqual(act[3], ['set', 'blocklisttest', 'action', 'mynetwatchman', 'agent', useragent])
 
 	@with_tmpdir
 	def testGlob(self, d):
@@ -483,14 +508,12 @@ class JailReaderTest(LogCaptureTestCase):
 		self.assertRaises(NoSectionError, c.getOptions, 'test', {})
 
 
-class FilterReaderTest(unittest.TestCase):
+class FilterReaderTest(LogCaptureTestCase):
-
-	def __init__(self, *args, **kwargs):
-		super(FilterReaderTest, self).__init__(*args, **kwargs)
-		self.__share_cfg = {}
 
 	def testConvert(self):
-		output = [['multi-set', 'testcase01', 'addfailregex', [
+		output = [
+			['set', 'testcase01', 'maxlines', 1],
+			['multi-set', 'testcase01', 'addfailregex', [
 			"^\\s*(?:\\S+ )?(?:kernel: \\[\\d+\\.\\d+\\] )?(?:@vserver_\\S+ )"
 			"?(?:(?:\\[\\d+\\])?:\\s+[\\[\\(]?sshd(?:\\(\\S+\\))?[\\]\\)]?:?|"
 			"[\\[\\(]?sshd(?:\\(\\S+\\))?[\\]\\)]?:?(?:\\[\\d+\\])?:)?\\s*(?:"
@@ -512,7 +535,6 @@ class FilterReaderTest(unittest.TestCase):
 			['set', 'testcase01', 'addjournalmatch',
 				"FIELD= with spaces ", "+", "AFIELD= with + char and spaces"],
 			['set', 'testcase01', 'datepattern', "%Y %m %d %H:%M:%S"],
-			['set', 'testcase01', 'maxlines', 1], # Last for overide test
 		]
 		filterReader = FilterReader("testcase01", "testcase01", {})
 		filterReader.setBaseDir(TEST_FILES_DIR)
@@ -529,9 +551,18 @@ class FilterReaderTest(unittest.TestCase):
 		filterReader.read()
 		#filterReader.getOptions(["failregex", "ignoreregex"])
 		filterReader.getOptions(None)
-		output[-1][-1] = "5"
+		output[0][-1] = 5; # maxlines = 5
 		self.assertSortedEqual(filterReader.convert(), output)
 
+	def testConvertOptions(self):
+		filterReader = FilterReader("testcase01", "testcase01", {'maxlines': '<test>', 'test': 'X'},
+		  share_config=TEST_FILES_DIR_SHARE_CFG, basedir=TEST_FILES_DIR)
+		filterReader.read()
+		filterReader.getOptions(None)
+		opts = filterReader.getCombined();
+		self.assertNotEqual(opts['maxlines'], 'X'); # wrong int value 'X' for 'maxlines'
+		self.assertLogged("Wrong int value 'X' for 'maxlines'. Using default one:")
+
 	def testFilterReaderSubstitionDefault(self):
 		output = [['set', 'jailname', 'addfailregex', 'to=sweet@example.com fromip=<IP>']]
 		filterReader = FilterReader('substition', "jailname", {},
@@ -541,6 +572,17 @@ class FilterReaderTest(unittest.TestCase):
 		c = filterReader.convert()
 		self.assertSortedEqual(c, output)
 
+	def testFilterReaderSubstKnown(self):
+		# testcase02.conf + testcase02.local, test covering that known/option is not overridden
+		# with unmodified (not available) value of option from .local config file, so wouldn't
+		# cause self-recursion if option already has a reference to known/option in .conf file.
+		filterReader = FilterReader('testcase02', "jailname", {},
+		  share_config=TEST_FILES_DIR_SHARE_CFG, basedir=TEST_FILES_DIR)
+		filterReader.read()
+		filterReader.getOptions(None)
+		opts = filterReader.getCombined()
+		self.assertTrue('sshd' in opts['failregex'])
+		
 	def testFilterReaderSubstitionSet(self):
 		output = [['set', 'jailname', 'addfailregex', 'to=sour@example.com fromip=<IP>']]
 		filterReader = FilterReader('substition', "jailname", {'honeypot': 'sour@example.com'},
@@ -709,9 +751,9 @@ class JailsReaderTest(LogCaptureTestCase):
 		         ['add', 'tz_correct', 'auto'],
 			 ['start', 'tz_correct'],
 			 ['config-error',
-				"Jail 'brokenactiondef' skipped, because of wrong configuration: Invalid action definition 'joho[foo'"],
+				"Jail 'brokenactiondef' skipped, because of wrong configuration: Invalid action definition 'joho[foo': unexpected option syntax"],
 			 ['config-error',
-				"Jail 'brokenfilterdef' skipped, because of wrong configuration: Invalid filter definition 'flt[test'"],
+				"Jail 'brokenfilterdef' skipped, because of wrong configuration: Invalid filter definition 'flt[test': unexpected option syntax"],
 			 ['config-error',
 				"Jail 'missingaction' skipped, because of wrong configuration: Unable to read action 'noactionfileforthisaction'"],
 			 ['config-error',
@@ -932,6 +974,7 @@ class JailsReaderTest(LogCaptureTestCase):
 		  ['set', 'syslogsocket', 'auto'],
 		  ['set', 'loglevel', "INFO"],
 		  ['set', 'logtarget', '/var/log/fail2ban.log'],
+		  ['set', 'allowipv6', 'auto'],
 		  ['set', 'dbfile', '/var/lib/fail2ban/fail2ban.sqlite3'],
 		  ['set', 'dbmaxmatches', 10],
 		  ['set', 'dbpurgeage', '1d'],

fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf

@@ -57,11 +57,10 @@ mdre-normal =
 
 mdre-ddos =  ^%(__prefix_line_sl)sDid not receive identification string from <HOST>
              ^%(__prefix_line_sl)sBad protocol version identification '.*' from <HOST>
-             ^%(__prefix_line_sl)sConnection closed by%(__authng_user)s <HOST>%(__on_port_opt)s\s+\[preauth\]\s*$
+             ^%(__prefix_line_sl)sConnection (?:closed|reset) by%(__authng_user)s <HOST>%(__on_port_opt)s\s+\[preauth\]\s*$
-             ^%(__prefix_line_sl)sConnection reset by <HOST>
              ^%(__prefix_line_ml1)sSSH: Server;Ltype: (?:Authname|Version|Kex);Remote: <HOST>-\d+;[A-Z]\w+:.*%(__prefix_line_ml2)sRead from socket failed: Connection reset by peer%(__suff)s$
 
-mdre-extra = ^%(__prefix_line_sl)sReceived disconnect from <HOST>%(__on_port_opt)s:\s*14: No supported authentication methods available
+mdre-extra = ^%(__prefix_line_sl)sReceived disconnect from <HOST>%(__on_port_opt)s:\s*14: No(?: supported)? authentication methods available
              ^%(__prefix_line_sl)sUnable to negotiate with <HOST>%(__on_port_opt)s: no matching <__alg_match> found.
              ^%(__prefix_line_ml1)sConnection from <HOST>%(__on_port_opt)s%(__prefix_line_ml2)sUnable to negotiate a <__alg_match>
              ^%(__prefix_line_ml1)sConnection from <HOST>%(__on_port_opt)s%(__prefix_line_ml2)sno matching <__alg_match> found:

fail2ban/tests/databasetestcase.py

@@ -29,7 +29,7 @@ import tempfile
 import sqlite3
 import shutil
 
-from ..server.filter import FileContainer
+from ..server.filter import FileContainer, Filter
 from ..server.mytime import MyTime
 from ..server.ticket import FailTicket
 from ..server.actions import Actions, Utils
@@ -212,19 +212,20 @@ class DatabaseTest(LogCaptureTestCase):
 			self.jail.name in self.db.getJailNames(True),
 			"Jail not added to database")
 
-	def testAddLog(self):
+	def _testAddLog(self):
 		self.testAddJail() # Jail required
 
 		_, filename = tempfile.mkstemp(".log", "Fail2BanDb_")
 		self.fileContainer = FileContainer(filename, "utf-8")
 
-		self.db.addLog(self.jail, self.fileContainer)
+		pos = self.db.addLog(self.jail, self.fileContainer)
+		self.assertTrue(pos is None); # unknown previously
 
 		self.assertIn(filename, self.db.getLogPaths(self.jail))
 		os.remove(filename)
 
 	def testUpdateLog(self):
-		self.testAddLog() # Add log file
+		self._testAddLog() # Add log file
 
 		# Write some text
 		filename = self.fileContainer.getFileName()
@@ -544,17 +545,21 @@ class DatabaseTest(LogCaptureTestCase):
 		self.testAddJail() # Jail required
 		self.jail.database = self.db
 		self.db.addJail(self.jail)
-		actions = Actions(self.jail)
+		actions = self.jail.actions
 		actions.add(
 			"action_checkainfo",
 			os.path.join(TEST_FILES_DIR, "action.d/action_checkainfo.py"),
 			{})
+		actions.banManager.setBanTotal(20)
+		self.jail._Jail__filter = flt = Filter(self.jail)
+		flt.failManager.setFailTotal(50)
 		ticket = FailTicket("1.2.3.4")
 		ticket.setAttempt(5)
 		ticket.setMatches(['test', 'test'])
 		self.jail.putFailTicket(ticket)
 		actions._Actions__checkBan()
 		self.assertLogged("ban ainfo %s, %s, %s, %s" % (True, True, True, True))
+		self.assertLogged("jail info %d, %d, %d, %d" % (1, 21, 0, 50))
 
 	def testDelAndAddJail(self):
 		self.testAddJail() # Add jail

fail2ban/tests/datedetectortestcase.py

@@ -516,6 +516,9 @@ class CustomDateFormatsTest(unittest.TestCase):
 			(1072746123.0 - 3600, "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %Z)?", "[2003-12-30 01:02:03] server ..."),
 			(1072746123.0,        "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %z)?", "[2003-12-30 01:02:03 UTC] server ..."),
 			(1072746123.0,        "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %Z)?", "[2003-12-30 01:02:03 UTC] server ..."),
+			(1072746123.0,        "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %z)?", "[2003-12-30 01:02:03 Z] server ..."),
+			(1072746123.0,        "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %z)?", "[2003-12-30 01:02:03 +0000] server ..."),
+			(1072746123.0,        "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %Z)?", "[2003-12-30 01:02:03 Z] server ..."),
 		):
 			logSys.debug('== test: %r', (matched, dp, line))
 			if dp is None:
@@ -551,6 +554,9 @@ class CustomDateFormatsTest(unittest.TestCase):
 			(1123970401.0,  "^%ExH:%ExM:%ExS**", '00:00:01'),
 			# cover date with current year, in test cases now == Aug 2005 -> back to last year (Sep 2004):
 			(1094068799.0,  "^%m/%d %ExH:%ExM:%ExS**", '09/01 21:59:59'),
+			# no time (only date) in pattern, assume local 00:00:00 for H:M:S :
+			(1093989600.0,  "^%Y-%m-%d**",   '2004-09-01'),
+			(1093996800.0,  "^%Y-%m-%d%z**", '2004-09-01Z'),
 		):
 			logSys.debug('== test: %r', (matched, dp, line))
 			dd = DateDetector()

fail2ban/tests/fail2banclienttestcase.py

@@ -37,7 +37,7 @@ from threading import Thread
 
 from ..client import fail2banclient, fail2banserver, fail2bancmdline
 from ..client.fail2bancmdline import Fail2banCmdLine
-from ..client.fail2banclient import exec_command_line as _exec_client, VisualWait
+from ..client.fail2banclient import exec_command_line as _exec_client, CSocket, VisualWait
 from ..client.fail2banserver import Fail2banServer, exec_command_line as _exec_server
 from .. import protocol
 from ..server import server
@@ -230,7 +230,7 @@ def _start_params(tmp, use_stock=False, use_stock_cfg=None,
 			os.symlink(os.path.abspath(pjoin(STOCK_CONF_DIR, n)), pjoin(cfg, n))
 	if create_before_start:
 		for n in create_before_start:
-			_write_file(n % {'tmp': tmp}, 'w', '')
+			_write_file(n % {'tmp': tmp}, 'w')
 	# parameters (sock/pid and config, increase verbosity, set log, etc.):
 	vvv, llev = (), "INFO"
 	if unittest.F2B.log_level < logging.INFO: # pragma: no cover
@@ -453,6 +453,14 @@ class Fail2banClientServerBase(LogCaptureTestCase):
 		self.assertRaises(exitType, self.exec_command_line[0],
 			(self.exec_command_line[1:] + startparams + args))
 
+	def execCmdDirect(self, startparams, *args):
+		sock = startparams[startparams.index('-s')+1]
+		s = CSocket(sock)
+		try:
+			return s.send(args)
+		finally:
+			s.close()
+
 	#
 	# Common tests
 	#
@@ -647,12 +655,6 @@ class Fail2banClientTest(Fail2banClientServerBase):
 		self.assertLogged("Base configuration directory " + pjoin(tmp, "miss") + " does not exist")
 		self.pruneLog()
 
-		## wrong socket
-		self.execCmd(FAILED, (),
-			"--async", "-c", pjoin(tmp, "config"), "-s", pjoin(tmp, "miss/f2b.sock"), "start")
-		self.assertLogged("There is no directory " + pjoin(tmp, "miss") + " to contain the socket file")
-		self.pruneLog()
-
 		## not running
 		self.execCmd(FAILED, (),
 			"-c", pjoin(tmp, "config"), "-s", pjoin(tmp, "f2b.sock"), "reload")
@@ -748,12 +750,6 @@ class Fail2banServerTest(Fail2banClientServerBase):
 		self.assertLogged("Base configuration directory " + pjoin(tmp, "miss") + " does not exist")
 		self.pruneLog()
 
-		## wrong socket
-		self.execCmd(FAILED, (),
-			"-c", pjoin(tmp, "config"), "-x", "-s", pjoin(tmp, "miss/f2b.sock"))
-		self.assertLogged("There is no directory " + pjoin(tmp, "miss") + " to contain the socket file")
-		self.pruneLog()
-
 		## already exists:
 		open(pjoin(tmp, "f2b.sock"), 'a').close()
 		self.execCmd(FAILED, (),
@@ -892,7 +888,7 @@ class Fail2banServerTest(Fail2banClientServerBase):
 				"action = ",
 				"         test-action2[name='%(__name__)s', restore='restored: <restored>', info=', err-code: <F-ERRCODE>']" \
 					if 2 in actions else "",
-				"         test-action2[name='%(__name__)s', actname=test-action3, _exec_once=1, restore='restored: <restored>']"
+				"         test-action2[name='%(__name__)s', actname=test-action3, _exec_once=1, restore='restored: <restored>',"
 										" actionflush=<_use_flush_>]" \
 					if 3 in actions else "",
 				"logpath = " + test2log,
@@ -941,10 +937,8 @@ class Fail2banServerTest(Fail2banClientServerBase):
 			"Jail 'broken-jail' skipped, because of wrong configuration", all=True)
 		
 		# enable both jails, 3 logs for jail1, etc...
-		# truncate test-log - we should not find unban/ban again by reload:
 		self.pruneLog("[test-phase 1b]")
 		_write_jail_cfg(actions=[1,2])
-		_write_file(test1log, "w+")
 		if unittest.F2B.log_level < logging.DEBUG: # pragma: no cover
 			_out_file(test1log)
 		self.execCmd(SUCCESS, startparams, "reload")
@@ -1007,7 +1001,7 @@ class Fail2banServerTest(Fail2banClientServerBase):
 		
 		self.pruneLog("[test-phase 2b]")
 		# write new failures:
-		_write_file(test2log, "w+", *(
+		_write_file(test2log, "a+", *(
 			(str(int(MyTime.time())) + "   error 403 from 192.0.2.2: test 2",) * 3 +
 		  (str(int(MyTime.time())) + "   error 403 from 192.0.2.3: test 2",) * 3 +
 		  (str(int(MyTime.time())) + " failure 401 from 192.0.2.4: test 2",) * 3 +
@@ -1041,10 +1035,30 @@ class Fail2banServerTest(Fail2banClientServerBase):
 			all=True)
 		# if observer available wait for it becomes idle (write all tickets to db):
 		_observer_wait_idle()
-
+		# test banned command:
-		# rotate logs:
+		self.assertSortedEqual(self.execCmdDirect(startparams,
-		_write_file(test1log, "w+")
+			'banned'), (0, [
-		_write_file(test2log, "w+")
+				{'test-jail1': ['192.0.2.4', '192.0.2.1', '192.0.2.8', '192.0.2.3', '192.0.2.2']},
+				{'test-jail2': ['192.0.2.4', '192.0.2.9', '192.0.2.8']}
+			]
+		))
+		self.assertSortedEqual(self.execCmdDirect(startparams,
+			'banned', '192.0.2.1', '192.0.2.4', '192.0.2.222'), (0, [
+			  ['test-jail1'], ['test-jail1', 'test-jail2'], []
+			]
+		))
+		self.assertSortedEqual(self.execCmdDirect(startparams,
+			'get', 'test-jail1', 'banned')[1], [
+				'192.0.2.4', '192.0.2.1', '192.0.2.8', '192.0.2.3', '192.0.2.2'])
+		self.assertSortedEqual(self.execCmdDirect(startparams,
+			'get', 'test-jail2', 'banned')[1], [
+				'192.0.2.4', '192.0.2.9', '192.0.2.8'])
+		self.assertEqual(self.execCmdDirect(startparams,
+			'get', 'test-jail1', 'banned', '192.0.2.3')[1],  1)
+		self.assertEqual(self.execCmdDirect(startparams,
+			'get', 'test-jail1', 'banned', '192.0.2.9')[1],  0)
+		self.assertEqual(self.execCmdDirect(startparams,
+			'get', 'test-jail1', 'banned', '192.0.2.3', '192.0.2.9')[1],  [1, 0])
 
 		# restart jail without unban all:
 		self.pruneLog("[test-phase 2c]")
@@ -1163,7 +1177,7 @@ class Fail2banServerTest(Fail2banClientServerBase):
 
 		# now write failures again and check already banned (jail1 was alive the whole time) and new bans occurred (jail1 was alive the whole time):
 		self.pruneLog("[test-phase 5]")
-		_write_file(test1log, "w+", *(
+		_write_file(test1log, "a+", *(
 			(str(int(MyTime.time())) + " failure 401 from 192.0.2.1: test 5",) * 3 + 
 			(str(int(MyTime.time())) + "   error 403 from 192.0.2.5: test 5",) * 3 +
 			(str(int(MyTime.time())) + " failure 401 from 192.0.2.6: test 5",) * 3
@@ -1183,13 +1197,41 @@ class Fail2banServerTest(Fail2banClientServerBase):
 		self.assertNotLogged("[test-jail1] Found 192.0.2.5")
 
 		# unban single ips:
-		self.pruneLog("[test-phase 6]")
+		self.pruneLog("[test-phase 6a]")
 		self.execCmd(SUCCESS, startparams,
 			"--async", "unban", "192.0.2.5", "192.0.2.6")
 		self.assertLogged(
 			"192.0.2.5 is not banned",
 			"[test-jail1] Unban 192.0.2.6", all=True, wait=MID_WAITTIME
 		)
+		# unban ips by subnet (cidr/mask):
+		self.pruneLog("[test-phase 6b]")
+		self.execCmd(SUCCESS, startparams,
+			"--async", "unban", "192.0.2.2/31")
+		self.assertLogged(
+			"[test-jail1] Unban 192.0.2.2",
+			"[test-jail1] Unban 192.0.2.3", all=True, wait=MID_WAITTIME
+		)		
+		self.execCmd(SUCCESS, startparams,
+			"--async", "unban", "192.0.2.8/31", "192.0.2.100/31")
+		self.assertLogged(
+			"[test-jail1] Unban 192.0.2.8",
+			"192.0.2.100/31 is not banned", all=True, wait=MID_WAITTIME)
+
+		# ban/unban subnet(s):
+		self.pruneLog("[test-phase 6c]")
+		self.execCmd(SUCCESS, startparams,
+			"--async", "set", "test-jail1", "banip", "192.0.2.96/28", "192.0.2.112/28")
+		self.assertLogged(
+			"[test-jail1] Ban 192.0.2.96/28",
+			"[test-jail1] Ban 192.0.2.112/28", all=True, wait=MID_WAITTIME
+		)
+		self.execCmd(SUCCESS, startparams,
+			"--async", "set", "test-jail1", "unbanip", "192.0.2.64/26"); # contains both subnets .96/28 and .112/28
+		self.assertLogged(
+			"[test-jail1] Unban 192.0.2.96/28",
+			"[test-jail1] Unban 192.0.2.112/28", all=True, wait=MID_WAITTIME
+		)
 
 		# reload all (one jail) with unban all:
 		self.pruneLog("[test-phase 7]")
@@ -1200,8 +1242,6 @@ class Fail2banServerTest(Fail2banClientServerBase):
 		self.assertLogged(
 			"Jail 'test-jail1' reloaded",
 			"[test-jail1] Unban 192.0.2.1",
-			"[test-jail1] Unban 192.0.2.2",
-			"[test-jail1] Unban 192.0.2.3",
 			"[test-jail1] Unban 192.0.2.4", all=True
 		)
 		# no restart occurred, no more ban (unbanned all using option "--unban"):
@@ -1209,8 +1249,6 @@ class Fail2banServerTest(Fail2banClientServerBase):
 			"Jail 'test-jail1' stopped",
 			"Jail 'test-jail1' started",
 			"[test-jail1] Ban 192.0.2.1",
-			"[test-jail1] Ban 192.0.2.2",
-			"[test-jail1] Ban 192.0.2.3",
 			"[test-jail1] Ban 192.0.2.4", all=True
 		)
 
@@ -1282,7 +1320,7 @@ class Fail2banServerTest(Fail2banClientServerBase):
 			'backend = polling',
 			'usedns = no',
 			'logpath = %(tmp)s/blck-failures.log',
-			'action = nginx-block-map[blck_lst_reload="", blck_lst_file="%(tmp)s/blck-lst.map"]',
+			'action = nginx-block-map[srv_cmd="echo nginx", srv_pid="%(tmp)s/f2b.pid", blck_lst_file="%(tmp)s/blck-lst.map"]',
 			'         blocklist_de[actionban=\'curl() { echo "*** curl" "$*";}; <Definition/actionban>\', email="Fail2Ban <fail2ban@localhost>", '
 													  'apikey="TEST-API-KEY", agent="fail2ban-test-agent", service=<name>]',
 			'filter =',
@@ -1322,6 +1360,8 @@ class Fail2banServerTest(Fail2banClientServerBase):
 		self.assertIn('\\125-000-004 1;\n', mp)
 		self.assertIn('\\125-000-005 1;\n', mp)
 
+		# check nginx reload is logged (pid of fail2ban is used to simulate success check nginx is running):
+		self.assertLogged("stdout: 'nginx -qt'", "stdout: 'nginx -s reload'", all=True)
 		# check blocklist_de substitution (e. g. new-line after <matches>):
 		self.assertLogged(
 			"stdout: '*** curl --fail --data-urlencode server=Fail2Ban <fail2ban@localhost>"
@@ -1364,8 +1404,9 @@ class Fail2banServerTest(Fail2banClientServerBase):
 		'jails': (
 			# default:
 			'''test_action = dummy[actionstart_on_demand=1, init="start: %(__name__)s", target="%(tmp)s/test.txt",
-      actionban='<known/actionban>;
+      actionban='<known/actionban>; echo "found: <jail.found> / <jail.found_total>, banned: <jail.banned> / <jail.banned_total>"
-        echo "<matches>"; printf "=====\\n%%b\\n=====\\n\\n" "<matches>" >> <target>']''',
+        echo "<matches>"; printf "=====\\n%%b\\n=====\\n\\n" "<matches>" >> <target>',
+      actionstop='<known/actionstop>; echo "stats <name> - found: <jail.found_total>, banned: <jail.banned_total>"']''',
 			# jail sendmail-auth:
 			'[sendmail-auth]',
 			'backend = polling',
@@ -1410,7 +1451,8 @@ class Fail2banServerTest(Fail2banClientServerBase):
 		_write_file(lgfn, "w+", *smaut_msg)
 		# wait and check it caused banned (and dump in the test-file):
 		self.assertLogged(
-			"[sendmail-auth] Ban 192.0.2.1", "1 ticket(s) in 'sendmail-auth'", all=True, wait=MID_WAITTIME)
+			"[sendmail-auth] Ban 192.0.2.1",  "stdout: 'found: 0 / 3, banned: 1 / 1'",
+			"1 ticket(s) in 'sendmail-auth'", all=True, wait=MID_WAITTIME)
 		_out_file(tofn)
 		td = _read_file(tofn)
 		# check matches (maxmatches = 2, so only 2 & 3 available):
@@ -1421,10 +1463,11 @@ class Fail2banServerTest(Fail2banClientServerBase):
 
 		self.pruneLog("[test-phase sendmail-reject]")
 		# write log:
-		_write_file(lgfn, "w+", *smrej_msg)
+		_write_file(lgfn, "a+", *smrej_msg)
 		# wait and check it caused banned (and dump in the test-file):
 		self.assertLogged(
-			"[sendmail-reject] Ban 192.0.2.2", "1 ticket(s) in 'sendmail-reject'", all=True, wait=MID_WAITTIME)
+			"[sendmail-reject] Ban 192.0.2.2", "stdout: 'found: 0 / 3, banned: 1 / 1'",
+			"1 ticket(s) in 'sendmail-reject'", all=True, wait=MID_WAITTIME)
 		_out_file(tofn)
 		td = _read_file(tofn)
 		# check matches (no maxmatches, so all matched messages are available):
@@ -1438,6 +1481,8 @@ class Fail2banServerTest(Fail2banClientServerBase):
 		# wait a bit:
 		self.assertLogged(
 			"Reload finished.",
+			"stdout: 'stats sendmail-auth - found: 3, banned: 1'",
+			"stdout: 'stats sendmail-reject - found: 3, banned: 1'",
 			"[sendmail-auth] Restore Ban 192.0.2.1", "1 ticket(s) in 'sendmail-auth'", all=True, wait=MID_WAITTIME)
 		# check matches again - (dbmaxmatches = 1), so it should be only last match after restart:
 		td = _read_file(tofn)
@@ -1546,7 +1591,7 @@ class Fail2banServerTest(Fail2banClientServerBase):
 		wakeObs = False
 		_observer_wait_before_incrban(lambda: wakeObs)
 		# write again (IP already bad):
-		_write_file(test1log, "w+", *(
+		_write_file(test1log, "a+", *(
 		  (str(int(MyTime.time())) + " failure 401 from 192.0.2.11: I'm very bad \"hacker\" `` $(echo test)",) * 2
 		))
 		# wait for ban:

fail2ban/tests/fail2banregextestcase.py

@@ -25,6 +25,7 @@ __license__ = "GPL"
 
 import os
 import sys
+import tempfile
 import unittest
 
 from ..client import fail2banregex
@@ -80,7 +81,13 @@ def _test_exec_command_line(*args):
 		sys.stderr = _org['stderr']
 	return _exit_code
 
+def _reset():
+	# reset global warn-counter:
+	from ..server.filter import _decode_line_warn
+	_decode_line_warn.clear()
+
 STR_00 = "Dec 31 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 192.0.2.0"
+STR_00_NODT = "[sshd] error: PAM: Authentication failure for kevin from 192.0.2.0"
 
 RE_00 = r"(?:(?:Authentication failure|Failed [-/\w+]+) for(?: [iI](?:llegal|nvalid) user)?|[Ii](?:llegal|nvalid) user|ROOT LOGIN REFUSED) .*(?: from|FROM) <HOST>"
 RE_00_ID = r"Authentication failure for <F-ID>.*?</F-ID> from <ADDR>$"
@@ -121,6 +128,7 @@ class Fail2banRegexTest(LogCaptureTestCase):
 		"""Call before every test case."""
 		LogCaptureTestCase.setUp(self)
 		setUpMyTime()
+		_reset()
 
 	def tearDown(self):
 		"""Call after every test case."""
@@ -140,6 +148,12 @@ class Fail2banRegexTest(LogCaptureTestCase):
 		))
 		self.assertLogged("Unable to compile regular expression")
 
+	def testWrongFilterOptions(self):
+		self.assertFalse(_test_exec(
+			"test", "flt[a='x,y,z',b=z,y,x]"
+		))
+		self.assertLogged("Wrong filter name or options", "wrong syntax at 14: y,x", all=True)
+
 	def testDirectFound(self):
 		self.assertTrue(_test_exec(
 			"--datepattern", r"^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?",
@@ -172,7 +186,7 @@ class Fail2banRegexTest(LogCaptureTestCase):
 			"--print-all-matched",
 			FILENAME_01, RE_00
 		))
-		self.assertLogged('Lines: 19 lines, 0 ignored, 13 matched, 6 missed')
+		self.assertLogged('Lines: 19 lines, 0 ignored, 16 matched, 3 missed')
 
 		self.assertLogged('Error decoding line');
 		self.assertLogged('Continuing to process line ignoring invalid characters')
@@ -186,7 +200,7 @@ class Fail2banRegexTest(LogCaptureTestCase):
 			"--print-all-matched", "--raw",
 			FILENAME_01, RE_00
 		))
-		self.assertLogged('Lines: 19 lines, 0 ignored, 16 matched, 3 missed')
+		self.assertLogged('Lines: 19 lines, 0 ignored, 19 matched, 0 missed')
 
 	def testDirectRE_1raw_noDns(self):
 		self.assertTrue(_test_exec(
@@ -194,7 +208,7 @@ class Fail2banRegexTest(LogCaptureTestCase):
 			"--print-all-matched", "--raw", "--usedns=no",
 			FILENAME_01, RE_00
 		))
-		self.assertLogged('Lines: 19 lines, 0 ignored, 13 matched, 6 missed')
+		self.assertLogged('Lines: 19 lines, 0 ignored, 16 matched, 3 missed')
 		# usage of <F-ID>\S+</F-ID> causes raw handling automatically:
 		self.pruneLog()
 		self.assertTrue(_test_exec(
@@ -340,6 +354,23 @@ class Fail2banRegexTest(LogCaptureTestCase):
 		self.assertTrue(_test_exec('-o', 'id', STR_00, RE_00_ID))
 		self.assertLogged('kevin')
 		self.pruneLog()
+		# multiple id combined to a tuple (id, tuple_id):
+		self.assertTrue(_test_exec('-o', 'id', 
+			'1591983743.667 192.0.2.1 192.0.2.2',
+			r'^\s*<F-ID/> <F-TUPLE_ID>\S+</F-TUPLE_ID>'))
+		self.assertLogged(str(('192.0.2.1', '192.0.2.2')))
+		self.pruneLog()
+		# multiple id combined to a tuple, id first - (id, tuple_id_1, tuple_id_2):
+		self.assertTrue(_test_exec('-o', 'id', 
+			'1591983743.667 left 192.0.2.3 right',
+			r'^\s*<F-TUPLE_ID_1>\S+</F-TUPLE_ID_1> <F-ID/> <F-TUPLE_ID_2>\S+</F-TUPLE_ID_2>'))
+		self.pruneLog()
+		# id had higher precedence as ip-address:
+		self.assertTrue(_test_exec('-o', 'id', 
+			'1591983743.667 left [192.0.2.4]:12345 right',
+			r'^\s*<F-TUPLE_ID_1>\S+</F-TUPLE_ID_1> <F-ID><ADDR>:<F-PORT/></F-ID> <F-TUPLE_ID_2>\S+</F-TUPLE_ID_2>'))
+		self.assertLogged(str(('[192.0.2.4]:12345', 'left', 'right')))
+		self.pruneLog()
 		# row with id :
 		self.assertTrue(_test_exec('-o', 'row', STR_00, RE_00_ID))
 		self.assertLogged("['kevin'", "'ip4': '192.0.2.0'", "'fid': 'kevin'", all=True)
@@ -361,6 +392,24 @@ class Fail2banRegexTest(LogCaptureTestCase):
 		self.assertLogged('192.0.2.0, kevin, inet4')
 		self.pruneLog()
 
+	def testNoDateTime(self):
+		# datepattern doesn't match:
+		self.assertTrue(_test_exec('-d', '{^LN-BEG}EPOCH', '-o', 'Found-ID:<F-ID>', STR_00_NODT, RE_00_ID))
+		self.assertLogged(
+			"Found a match but no valid date/time found",
+			"Match without a timestamp:",
+			"Found-ID:kevin", all=True)
+		self.pruneLog()
+		# explicitly no datepattern:
+		self.assertTrue(_test_exec('-d', '{NONE}', '-o', 'Found-ID:<F-ID>', STR_00_NODT, RE_00_ID))
+		self.assertLogged(
+			"Found-ID:kevin", all=True)
+		self.assertNotLogged(
+			"Found a match but no valid date/time found",
+			"Match without a timestamp:", all=True)
+
+		self.pruneLog()
+
 	def testFrmtOutputWrapML(self):
 		unittest.F2B.SkipIfCfgMissing(stock=True)
 		# complex substitution using tags and message (ip, user, msg):
@@ -412,14 +461,8 @@ class Fail2banRegexTest(LogCaptureTestCase):
 			FILENAME_ZZZ_GEN, FILENAME_ZZZ_GEN
 		))
 
-	def _reset(self):
-		# reset global warn-counter:
-		from ..server.filter import _decode_line_warn
-		_decode_line_warn.clear()
-
 	def testWronChar(self):
 		unittest.F2B.SkipIfCfgMissing(stock=True)
-		self._reset()
 		self.assertTrue(_test_exec(
 		"-l", "notice", # put down log-level, because of too many debug-messages
 			"--datepattern", r"^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?",
@@ -435,7 +478,6 @@ class Fail2banRegexTest(LogCaptureTestCase):
 
 	def testWronCharDebuggex(self):
 		unittest.F2B.SkipIfCfgMissing(stock=True)
-		self._reset()
 		self.assertTrue(_test_exec(
 		"-l", "notice", # put down log-level, because of too many debug-messages
 			"--datepattern", r"^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?",
@@ -448,6 +490,36 @@ class Fail2banRegexTest(LogCaptureTestCase):
 
 		self.assertLogged('https://')
 
+	def testNLCharAsPartOfUniChar(self):
+		fname = tempfile.mktemp(prefix='tmp_fail2ban', suffix='uni')
+		# test two multi-byte encodings (both contains `\x0A` in either \x02\x0A or \x0A\x02):
+		for enc in ('utf-16be', 'utf-16le'):
+			self.pruneLog("[test-phase encoding=%s]" % enc)
+			try:
+				fout = open(fname, 'wb')
+				# test on unicode string containing \x0A as part of uni-char,
+				# it must produce exactly 2 lines (both are failures):
+				for l in (
+					u'1490349000 \u20AC Failed auth: invalid user Test\u020A from 192.0.2.1\n',
+					u'1490349000 \u20AC Failed auth: invalid user TestI from 192.0.2.2\n'
+				):
+					fout.write(l.encode(enc))
+				fout.close()
+
+				self.assertTrue(_test_exec(
+				"-l", "notice", # put down log-level, because of too many debug-messages
+					"--encoding", enc,
+					"--datepattern", r"^EPOCH",
+					fname, r"Failed .* from <HOST>",
+				))
+
+				self.assertLogged(" encoding : %s" % enc,
+					"Lines: 2 lines, 0 ignored, 2 matched, 0 missed", all=True)
+				self.assertNotLogged("Missed line(s)")
+			finally:
+				fout.close()
+				os.unlink(fname)
+
 	def testExecCmdLine_Usage(self):
 		self.assertNotEqual(_test_exec_command_line(), 0)
 		self.pruneLog()
@@ -485,7 +557,7 @@ class Fail2banRegexTest(LogCaptureTestCase):
 
 	def testLogtypeSystemdJournal(self): # pragma: no cover
 		if not fail2banregex.FilterSystemd:
-			raise unittest.SkipTest('Skip test because no systemd backand available')
+			raise unittest.SkipTest('Skip test because no systemd backend available')
 		self.assertTrue(_test_exec(
 			"systemd-journal", FILTER_ZZZ_GEN
 			  +'[journalmatch="SYSLOG_IDENTIFIER=\x01\x02dummy\x02\x01",'

fail2ban/tests/files/action.d/action_checkainfo.py

@@ -8,6 +8,9 @@ class TestAction(ActionBase):
         self._logSys.info("ban ainfo %s, %s, %s, %s",
           aInfo["ipmatches"] != '', aInfo["ipjailmatches"] != '', aInfo["ipfailures"] > 0, aInfo["ipjailfailures"] > 0
         )
+        self._logSys.info("jail info %d, %d, %d, %d",
+          aInfo["jail.banned"], aInfo["jail.banned_total"], aInfo["jail.found"], aInfo["jail.found_total"]
+        )
 
     def unban(self, aInfo):
         pass

fail2ban/tests/files/filter.d/testcase02.conf

@@ -0,0 +1,12 @@
+[INCLUDES]
+
+# Read common prefixes. If any customizations available -- read them from
+# common.local
+before = testcase-common.conf
+
+[Definition]
+
+_daemon = sshd
+__prefix_line = %(known/__prefix_line)s(?:\w{14,20}: )?
+
+failregex = %(__prefix_line)s test

fail2ban/tests/files/filter.d/testcase02.local

@@ -0,0 +1,4 @@
+[Definition]
+
+# no options here, coverage for testFilterReaderSubstKnown:
+# avoid to overwrite known/option with unmodified (not available) value of option from .local config file

fail2ban/tests/files/logs/apache-modsecurity

@@ -6,3 +6,6 @@
 
 # failJSON: { "time": "2018-09-28T09:18:06", "match": true , "host": "192.0.2.1", "desc": "two client entries in message (gh-2247)" }
 [Sat Sep 28 09:18:06 2018] [error] [client 192.0.2.1:55555] [client 192.0.2.1] ModSecurity: [file "/etc/httpd/modsecurity.d/10_asl_rules.conf"] [line "635"] [id "340069"] [rev "4"] [msg "Atomicorp.com UNSUPPORTED DELAYED Rules: Web vulnerability scanner"] [severity "CRITICAL"] Access denied with code 403 (phase 2). Pattern match "(?:nessus(?:_is_probing_you_|test)|^/w00tw00t\\\\.at\\\\.)" at REQUEST_URI. [hostname "192.81.249.191"] [uri "/w00tw00t.at.blackhats.romanian.anti-sec:)"] [unique_id "4Q6RdsBR@b4AAA65LRUAAAAA"] 
+
+# failJSON: { "time": "2020-05-09T00:35:52", "match": true , "host": "192.0.2.2", "desc": "new format - apache 2.4 and php-fpm (gh-2717)" }
+[Sat May 09 00:35:52.389262 2020] [:error] [pid 22406:tid 139985298601728] [client 192.0.2.2:47762] [client 192.0.2.2] ModSecurity: Access denied with code 401 (phase 2). Operator EQ matched 1 at IP:blocked. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_wp_login.conf"] [line "14"] [id "500000"] [msg "Ip address blocked for 15 minutes, more than 5 login attempts in 3 minutes."] [hostname "example.com"] [uri "/wp-login.php"] [unique_id "XrYlGL5IY3I@EoLOgAAAA8"], referer: https://example.com/wp-login.php