From 884f708bd7f4b48c90503752d927975ba10dafbc Mon Sep 17 00:00:00 2001 From: Michael Orlitzky Date: Sun, 15 Jul 2018 09:07:28 -0400 Subject: [PATCH 001/240] fail2ban/files: rename "gentoo" files to "openrc". We ship a service script and configuration file for "gentoo" that are actually more generally applicable: they work on any system where OpenRC is used. This commit simply renames the files from "gentoo" to "openrc" to reflect the fact that they are in no way Gentoo-specific. --- MANIFEST | 4 ++-- files/{gentoo-confd => fail2ban-openrc.conf} | 0 files/{gentoo-initd => fail2ban-openrc.init} | 0 3 files changed, 2 insertions(+), 2 deletions(-) rename files/{gentoo-confd => fail2ban-openrc.conf} (100%) rename files/{gentoo-initd => fail2ban-openrc.init} (100%) diff --git a/MANIFEST b/MANIFEST index 3ea2816b..ca9be123 100644 --- a/MANIFEST +++ b/MANIFEST @@ -375,8 +375,8 @@ files/fail2ban.service.in files/fail2ban-tmpfiles.conf files/fail2ban.upstart files/gen_badbots -files/gentoo-confd -files/gentoo-initd +files/fail2ban-openrc.conf +files/fail2ban-openrc.init files/ipmasq-ZZZzzz_fail2ban.rul files/logwatch/fail2ban files/logwatch/fail2ban-0.8.log diff --git a/files/gentoo-confd b/files/fail2ban-openrc.conf similarity index 100% rename from files/gentoo-confd rename to files/fail2ban-openrc.conf diff --git a/files/gentoo-initd b/files/fail2ban-openrc.init similarity index 100% rename from files/gentoo-initd rename to files/fail2ban-openrc.init From 1cec3d05b8a052423a7ef4f8205c9d63b1ce9b07 Mon Sep 17 00:00:00 2001 From: Michael Orlitzky Date: Sun, 15 Jul 2018 09:11:53 -0400 Subject: [PATCH 002/240] files/fail2ban-openrc.conf: remove hard-coded paths. There were two paths mentioned in comments in the fail2ban OpenRC conf file, but those paths aren't guaranteed to be correct (until/unless we integrate the conf file with the build system). The first comment referenced the physical location of the associated init script, and in my opinion is not useful to an end user in the first place. It has been removed: OpenRC users know what this file is for, there's no reason to repeat it in a comment. The second comment contained an absolute path to fail2ban-client, and I've removed the leading path components because "fail2ban-client" is generally run from your $PATH. --- files/fail2ban-openrc.conf | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/files/fail2ban-openrc.conf b/files/fail2ban-openrc.conf index 00d19f8b..1c589763 100644 --- a/files/fail2ban-openrc.conf +++ b/files/fail2ban-openrc.conf @@ -1,6 +1,4 @@ -# Config file for /etc/init.d/fail2ban -# -# For information on options, see "/usr/bin/fail2ban-client -h". +# For available options, plase run "fail2ban-client -h". FAIL2BAN_OPTIONS="" From eb58e90ba9ccd049fae8bd92d8f828df6fdb287e Mon Sep 17 00:00:00 2001 From: Michael Orlitzky Date: Sun, 15 Jul 2018 09:17:29 -0400 Subject: [PATCH 003/240] files/fail2ban-openrc.conf: remove a commented example setting. Our OpenRC conf file already tells users how to find the available options that can be placed in the FAIL2BAN_OPTIONS variable, so having a specific example of, FAIL2BAN_OPTIONS="-x" doesn't provide much more information. In fact, it makes you wonder why it's there in the first place: does the init script have some kind of problem with stale sockets? It used to, but that problem has been fixed. This commit removes the redundant example. --- files/fail2ban-openrc.conf | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/files/fail2ban-openrc.conf b/files/fail2ban-openrc.conf index 1c589763..1a2450e2 100644 --- a/files/fail2ban-openrc.conf +++ b/files/fail2ban-openrc.conf @@ -1,6 +1,2 @@ # For available options, plase run "fail2ban-client -h". - -FAIL2BAN_OPTIONS="" - -# Force execution of the server even if the socket already exists: -#FAIL2BAN_OPTIONS="-x" +#FAIL2BAN_OPTIONS="" From 64ec399542f20582090a69b77f4c5b90f7b41019 Mon Sep 17 00:00:00 2001 From: Michael Orlitzky Date: Sun, 15 Jul 2018 12:59:18 -0400 Subject: [PATCH 004/240] files/fail2ban-openrc.init: drop "need net" dependency. The "need net" dependency in our OpenRC service script was incorrect: the fail2ban service does not need a working WAN to function. This issue is well-documented and is covered in the OpenRC Service Script Guide, currently located at https://github.com/OpenRC/openrc/blob/master/service-script-guide.md --- files/fail2ban-openrc.init | 1 - 1 file changed, 1 deletion(-) diff --git a/files/fail2ban-openrc.init b/files/fail2ban-openrc.init index 0fb157cd..69025499 100755 --- a/files/fail2ban-openrc.init +++ b/files/fail2ban-openrc.init @@ -26,7 +26,6 @@ extra_started_commands="reload showlog" FAIL2BAN="/usr/bin/fail2ban-client ${FAIL2BAN_OPTIONS}" depend() { - need net need logger after iptables } From af24c52558296e52e90cb55fad449b2bf9e996c6 Mon Sep 17 00:00:00 2001 From: Michael Orlitzky Date: Sun, 15 Jul 2018 13:05:11 -0400 Subject: [PATCH 005/240] files/fail2ban-openrc.init: change "need logger" dependency to "use logger". Our OpenRC service script contained a "need logger" dependency, which meant that the life cycle of the fail2ban service was tied to that of the system logger service. That isn't quite correct: fail2ban functions fine even if the system logger is stopped: 1. fail2ban is capable of analyzing non-syslog log files. 2. Even if fail2ban is solely analyzing syslog files, we don't want to stop the fail2ban service simply because syslog was stopped -- fail2ban just won't see any new log lines until syslog is started again. This commit changes the "need net" dependency to "use net", which will still attempt to start the system logger service, but which won't kill fail2ban if the system logger is ever stopped. --- files/fail2ban-openrc.init | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/files/fail2ban-openrc.init b/files/fail2ban-openrc.init index 69025499..138bff30 100755 --- a/files/fail2ban-openrc.init +++ b/files/fail2ban-openrc.init @@ -26,7 +26,7 @@ extra_started_commands="reload showlog" FAIL2BAN="/usr/bin/fail2ban-client ${FAIL2BAN_OPTIONS}" depend() { - need logger + use logger after iptables } From bc4a742e32f610879fcdb1af7677e46007ac0682 Mon Sep 17 00:00:00 2001 From: Michael Orlitzky Date: Sun, 15 Jul 2018 13:13:13 -0400 Subject: [PATCH 006/240] files/fail2ban-openrc.init: replace FAIL2BAN with standard OpenRC variables. The FAIL2BAN variable in our OpenRC service script was a combination of two standard OpenRC variables, "command" and "command_args". This commit simply replaces the custom variable with the two standard ones. This will aid future simplifications of the service script. --- files/fail2ban-openrc.init | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/files/fail2ban-openrc.init b/files/fail2ban-openrc.init index 138bff30..1374da3d 100755 --- a/files/fail2ban-openrc.init +++ b/files/fail2ban-openrc.init @@ -23,7 +23,8 @@ description_reload="reload configuration" description_showlog="show fail2ban logs" extra_started_commands="reload showlog" -FAIL2BAN="/usr/bin/fail2ban-client ${FAIL2BAN_OPTIONS}" +command="/usr/bin/fail2ban-client" +command_args="${FAIL2BAN_OPTIONS}" depend() { use logger @@ -37,20 +38,20 @@ start() { # bug 347477 rm -f /var/run/fail2ban/fail2ban.sock || return 1 start-stop-daemon --start --pidfile /var/run/fail2ban/fail2ban.pid \ - -- ${FAIL2BAN} start + -- ${command} ${command_args} start eend $? "Failed to start fail2ban" } stop() { ebegin "Stopping fail2ban" start-stop-daemon --stop --pidfile /var/run/fail2ban/fail2ban.pid --retry 30 \ - -- ${FAIL2BAN} stop + -- ${command} ${command_args} stop eend $? "Failed to stop fail2ban" } reload() { ebegin "Reloading fail2ban" - ${FAIL2BAN} reload + ${command} ${command_args} reload eend $? "Failed to reload fail2ban" } From 115024d14a0b1a5a4b96175712f257b408883f38 Mon Sep 17 00:00:00 2001 From: Michael Orlitzky Date: Sun, 15 Jul 2018 13:15:44 -0400 Subject: [PATCH 007/240] files/fail2ban-openrc.init: use a variable for the pid file location. OpenRC has a special variable "pidfile" that should be used to store the location of the daemon's PID file. This commit replaces two instances of said location with one variable. --- files/fail2ban-openrc.init | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/files/fail2ban-openrc.init b/files/fail2ban-openrc.init index 1374da3d..9969a3e0 100755 --- a/files/fail2ban-openrc.init +++ b/files/fail2ban-openrc.init @@ -25,6 +25,7 @@ extra_started_commands="reload showlog" command="/usr/bin/fail2ban-client" command_args="${FAIL2BAN_OPTIONS}" +pidfile="/run/${RC_SVCNAME}/${RC_SVCNAME}.pid" depend() { use logger @@ -37,14 +38,14 @@ start() { # remove stalled sock file after system crash # bug 347477 rm -f /var/run/fail2ban/fail2ban.sock || return 1 - start-stop-daemon --start --pidfile /var/run/fail2ban/fail2ban.pid \ + start-stop-daemon --start --pidfile "${pidfile}" \ -- ${command} ${command_args} start eend $? "Failed to start fail2ban" } stop() { ebegin "Stopping fail2ban" - start-stop-daemon --stop --pidfile /var/run/fail2ban/fail2ban.pid --retry 30 \ + start-stop-daemon --stop --pidfile "${pidfile}" --retry 30 \ -- ${command} ${command_args} stop eend $? "Failed to stop fail2ban" } From 0b146208eb3a6f3ceb04d59e9e89c1cc914854ea Mon Sep 17 00:00:00 2001 From: Michael Orlitzky Date: Sun, 15 Jul 2018 13:32:03 -0400 Subject: [PATCH 008/240] files/fail2ban-openrc.init: move pre-flight checks into start_pre(). Our OpenRC service script performs two tasks before starting the service: 1. It removes any stake sockets (from e.g. a system crash). 2. It ensures that the PID file directory exists. These have both been moved into the "start_pre" phase, which is designed to do such things (and will allow us to simplify the "start" phase in the future). The existing "mkdir -p" has also been converted into a "checkpath -d" command which is built-in to OpenRC. --- files/fail2ban-openrc.init | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/files/fail2ban-openrc.init b/files/fail2ban-openrc.init index 9969a3e0..e1cf5273 100755 --- a/files/fail2ban-openrc.init +++ b/files/fail2ban-openrc.init @@ -32,12 +32,16 @@ depend() { after iptables } +start_pre() { + checkpath -d "${pidfile%/*}" || return 1 + + # Remove stale socket after system crash, Gentoo bug 347477 + rm -f /var/run/fail2ban/fail2ban.sock || return 1 +} + start() { ebegin "Starting fail2ban" - mkdir -p /var/run/fail2ban || return 1 - # remove stalled sock file after system crash - # bug 347477 - rm -f /var/run/fail2ban/fail2ban.sock || return 1 + start-stop-daemon --start --pidfile "${pidfile}" \ -- ${command} ${command_args} start eend $? "Failed to start fail2ban" From e0097aefb95bac48c992f6b5ae58fc99140ecc03 Mon Sep 17 00:00:00 2001 From: Michael Orlitzky Date: Sun, 15 Jul 2018 13:37:00 -0400 Subject: [PATCH 009/240] files/fail2ban-openrc.init: use RC_SVCNAME instead of hard-coding the name. If our service is installed under some other name, then we don't want the service script to say things like "Starting fail2ban..." because the name "fail2ban" won't make any sense at that point. Instead, we use the $RC_SVCNAME variable to ensure that the service name matches what we tell the user. Typically, however, $RC_SVCNAME will still be "fail2ban". --- files/fail2ban-openrc.init | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/files/fail2ban-openrc.init b/files/fail2ban-openrc.init index e1cf5273..bcbdf8fd 100755 --- a/files/fail2ban-openrc.init +++ b/files/fail2ban-openrc.init @@ -40,24 +40,24 @@ start_pre() { } start() { - ebegin "Starting fail2ban" + ebegin "Starting ${RC_SVCNAME}" start-stop-daemon --start --pidfile "${pidfile}" \ -- ${command} ${command_args} start - eend $? "Failed to start fail2ban" + eend $? "Failed to start ${RC_SVCNAME}" } stop() { - ebegin "Stopping fail2ban" + ebegin "Stopping ${RC_SVCNAME}" start-stop-daemon --stop --pidfile "${pidfile}" --retry 30 \ -- ${command} ${command_args} stop - eend $? "Failed to stop fail2ban" + eend $? "Failed to stop ${RC_SVCNAME}" } reload() { - ebegin "Reloading fail2ban" + ebegin "Reloading ${RC_SVCNAME}" ${command} ${command_args} reload - eend $? "Failed to reload fail2ban" + eend $? "Failed to reload ${RC_SVCNAME}" } showlog(){ From c8ed0e0d913feeb361de800d1af6fd56e96bb840 Mon Sep 17 00:00:00 2001 From: Michael Orlitzky Date: Sun, 15 Jul 2018 13:44:53 -0400 Subject: [PATCH 010/240] files/fail2ban-openrc.init: use the standard OpenRC "retry" variable. If the "retry" variable is set in the service script, we don't have to pass it to start-stop-daemon explicitly. While we can't immediately eliminate any code with this change, it will be necessary later to adopt the default OpenRC stop() function. --- files/fail2ban-openrc.init | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/files/fail2ban-openrc.init b/files/fail2ban-openrc.init index bcbdf8fd..e3ddfe1a 100755 --- a/files/fail2ban-openrc.init +++ b/files/fail2ban-openrc.init @@ -26,6 +26,7 @@ extra_started_commands="reload showlog" command="/usr/bin/fail2ban-client" command_args="${FAIL2BAN_OPTIONS}" pidfile="/run/${RC_SVCNAME}/${RC_SVCNAME}.pid" +retry="30" depend() { use logger @@ -49,7 +50,7 @@ start() { stop() { ebegin "Stopping ${RC_SVCNAME}" - start-stop-daemon --stop --pidfile "${pidfile}" --retry 30 \ + start-stop-daemon --stop --pidfile "${pidfile}" --retry "${retry}" \ -- ${command} ${command_args} stop eend $? "Failed to stop ${RC_SVCNAME}" } From bb0f732ae69894b22306dd7efa213513e3acd8a2 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 14 Jan 2020 20:38:26 +0100 Subject: [PATCH 011/240] version bump (master is 1.0.x-dev now) --- .github/PULL_REQUEST_TEMPLATE.md | 3 ++- ChangeLog | 10 ++++++++++ fail2ban/version.py | 2 +- man/fail2ban-client.1 | 4 ++-- man/fail2ban-python.1 | 2 +- man/fail2ban-regex.1 | 2 +- man/fail2ban-server.1 | 4 ++-- man/fail2ban-testcases.1 | 2 +- 8 files changed, 20 insertions(+), 9 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 3a17ccc2..350d6ee2 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,7 +1,8 @@ Before submitting your PR, please review the following checklist: - [ ] **CHOOSE CORRECT BRANCH**: if filing a bugfix/enhancement - against 0.9.x series, choose `master` branch + against certain release version, choose `0.9`, `0.10` or `0.11` branch, + for dev-edition use `master` branch - [ ] **CONSIDER adding a unit test** if your PR resolves an issue - [ ] **LIST ISSUES** this PR resolves - [ ] **MAKE SURE** this PR doesn't break existing tests diff --git a/ChangeLog b/ChangeLog index cc0c6608..3f9c11b1 100644 --- a/ChangeLog +++ b/ChangeLog @@ -6,6 +6,16 @@ Fail2Ban: Changelog =================== +ver. 1.0.1-dev-1 (20??/??/??) - development nightly edition +----------- + +### Fixes + +### New Features + +### Enhancements + + ver. 0.11.1 (2020/01/11) - this-is-the-way ----------- diff --git a/fail2ban/version.py b/fail2ban/version.py index e5efbe77..078e47ef 100644 --- a/fail2ban/version.py +++ b/fail2ban/version.py @@ -24,7 +24,7 @@ __author__ = "Cyril Jaquier, Yaroslav Halchenko, Steven Hiscocks, Daniel Black" __copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2005-2016 Yaroslav Halchenko, 2013-2014 Steven Hiscocks, Daniel Black" __license__ = "GPL-v2+" -version = "0.11.1" +version = "1.0.1.dev1" def normVersion(): """ Returns fail2ban version in normalized machine-readable format""" diff --git a/man/fail2ban-client.1 b/man/fail2ban-client.1 index 745c080a..54b475e3 100644 --- a/man/fail2ban-client.1 +++ b/man/fail2ban-client.1 @@ -1,12 +1,12 @@ .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. -.TH FAIL2BAN-CLIENT "1" "January 2020" "fail2ban-client v0.11.1" "User Commands" +.TH FAIL2BAN-CLIENT "1" "January 2020" "fail2ban-client v1.0.1.dev1" "User Commands" .SH NAME fail2ban-client \- configure and control the server .SH SYNOPSIS .B fail2ban-client [\fI\,OPTIONS\/\fR] \fI\,\/\fR .SH DESCRIPTION -Fail2Ban v0.11.1 reads log file that contains password failure report +Fail2Ban v1.0.1.dev1 reads log file that contains password failure report and bans the corresponding IP addresses using firewall rules. .SH OPTIONS .TP diff --git a/man/fail2ban-python.1 b/man/fail2ban-python.1 index 2c9746ee..5237f0fc 100644 --- a/man/fail2ban-python.1 +++ b/man/fail2ban-python.1 @@ -1,5 +1,5 @@ .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. -.TH FAIL2BAN-PYTHON "1" "January 2020" "fail2ban-python 0.11.1" "User Commands" +.TH FAIL2BAN-PYTHON "1" "January 2020" "fail2ban-python 1.0.1.1" "User Commands" .SH NAME fail2ban-python \- a helper for Fail2Ban to assure that the same Python is used .SH DESCRIPTION diff --git a/man/fail2ban-regex.1 b/man/fail2ban-regex.1 index a703130c..d11fcf43 100644 --- a/man/fail2ban-regex.1 +++ b/man/fail2ban-regex.1 @@ -1,5 +1,5 @@ .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. -.TH FAIL2BAN-REGEX "1" "January 2020" "fail2ban-regex 0.11.1" "User Commands" +.TH FAIL2BAN-REGEX "1" "January 2020" "fail2ban-regex 1.0.1.dev1" "User Commands" .SH NAME fail2ban-regex \- test Fail2ban "failregex" option .SH SYNOPSIS diff --git a/man/fail2ban-server.1 b/man/fail2ban-server.1 index 418b46dd..ed8aa639 100644 --- a/man/fail2ban-server.1 +++ b/man/fail2ban-server.1 @@ -1,12 +1,12 @@ .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. -.TH FAIL2BAN-SERVER "1" "January 2020" "fail2ban-server v0.11.1" "User Commands" +.TH FAIL2BAN-SERVER "1" "January 2020" "fail2ban-server v1.0.1.dev1" "User Commands" .SH NAME fail2ban-server \- start the server .SH SYNOPSIS .B fail2ban-server [\fI\,OPTIONS\/\fR] .SH DESCRIPTION -Fail2Ban v0.11.1 reads log file that contains password failure report +Fail2Ban v1.0.1.dev1 reads log file that contains password failure report and bans the corresponding IP addresses using firewall rules. .SH OPTIONS .TP diff --git a/man/fail2ban-testcases.1 b/man/fail2ban-testcases.1 index 6c32cb9e..7a6fa4e4 100644 --- a/man/fail2ban-testcases.1 +++ b/man/fail2ban-testcases.1 @@ -1,5 +1,5 @@ .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. -.TH FAIL2BAN-TESTCASES "1" "January 2020" "fail2ban-testcases 0.11.1" "User Commands" +.TH FAIL2BAN-TESTCASES "1" "January 2020" "fail2ban-testcases 1.0.1.dev1" "User Commands" .SH NAME fail2ban-testcases \- run Fail2Ban unit-tests .SH SYNOPSIS From 3f5c382a988bb21f48a0fe3a26c2433d04f7076f Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Tue, 14 Jan 2020 20:55:24 +0100 Subject: [PATCH 012/240] readme: amend to bump version (v.1.0.1 is master now) --- README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 8e9f5c3a..fac3414d 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ / _|__ _(_) |_ ) |__ __ _ _ _ | _/ _` | | |/ /| '_ \/ _` | ' \ |_| \__,_|_|_/___|_.__/\__,_|_||_| - v0.11.0.dev1 20??/??/?? + v1.0.1.dev1 20??/??/?? ## Fail2Ban: ban hosts that cause multiple authentication errors @@ -46,11 +46,11 @@ Optional: To install: - tar xvfj fail2ban-0.11.0.tar.bz2 - cd fail2ban-0.11.0 + tar xvfj fail2ban-1.0.1.tar.bz2 + cd fail2ban-1.0.1 sudo python setup.py install -Alternatively, you can clone the source from GitHub to a directory of Your choice, and do the install from there. Pick the correct branch, for example, 0.11 +Alternatively, you can clone the source from GitHub to a directory of Your choice, and do the install from there. Pick the correct branch, for example, master or 0.11 git clone https://github.com/fail2ban/fail2ban.git cd fail2ban @@ -89,11 +89,11 @@ fail2ban(1) and jail.conf(5) manpages for further references. Code status: ------------ -* travis-ci.org: [![tests status](https://secure.travis-ci.org/fail2ban/fail2ban.svg?branch=0.11)](https://travis-ci.org/fail2ban/fail2ban?branch=0.11) (0.11 branch) / [![tests status](https://secure.travis-ci.org/fail2ban/fail2ban.svg?branch=0.10)](https://travis-ci.org/fail2ban/fail2ban?branch=0.10) (0.10 branch) +* travis-ci.org: [![tests status](https://secure.travis-ci.org/fail2ban/fail2ban.svg?branch=master)](https://travis-ci.org/fail2ban/fail2ban?branch=master) / [![tests status](https://secure.travis-ci.org/fail2ban/fail2ban.svg?branch=0.11)](https://travis-ci.org/fail2ban/fail2ban?branch=0.11) (0.11 branch) / [![tests status](https://secure.travis-ci.org/fail2ban/fail2ban.svg?branch=0.10)](https://travis-ci.org/fail2ban/fail2ban?branch=0.10) (0.10 branch) -* coveralls.io: [![Coverage Status](https://coveralls.io/repos/fail2ban/fail2ban/badge.svg?branch=0.11)](https://coveralls.io/github/fail2ban/fail2ban?branch=0.11) (0.11 branch) / [![Coverage Status](https://coveralls.io/repos/fail2ban/fail2ban/badge.svg?branch=0.10)](https://coveralls.io/github/fail2ban/fail2ban?branch=0.10) / (0.10 branch) +* coveralls.io: [![Coverage Status](https://coveralls.io/repos/fail2ban/fail2ban/badge.svg?branch=master)](https://coveralls.io/github/fail2ban/fail2ban?branch=master) / [![Coverage Status](https://coveralls.io/repos/fail2ban/fail2ban/badge.svg?branch=0.11)](https://coveralls.io/github/fail2ban/fail2ban?branch=0.11) (0.11 branch) / [![Coverage Status](https://coveralls.io/repos/fail2ban/fail2ban/badge.svg?branch=0.10)](https://coveralls.io/github/fail2ban/fail2ban?branch=0.10) / (0.10 branch) -* codecov.io: [![codecov.io](https://codecov.io/gh/fail2ban/fail2ban/coverage.svg?branch=0.11)](https://codecov.io/gh/fail2ban/fail2ban/branch/0.11) (0.11 branch) / [![codecov.io](https://codecov.io/gh/fail2ban/fail2ban/coverage.svg?branch=0.10)](https://codecov.io/gh/fail2ban/fail2ban/branch/0.10) (0.10 branch) +* codecov.io: [![codecov.io](https://codecov.io/gh/fail2ban/fail2ban/coverage.svg?branch=master)](https://codecov.io/gh/fail2ban/fail2ban/branch/master) / [![codecov.io](https://codecov.io/gh/fail2ban/fail2ban/coverage.svg?branch=0.11)](https://codecov.io/gh/fail2ban/fail2ban/branch/0.11) (0.11 branch) / [![codecov.io](https://codecov.io/gh/fail2ban/fail2ban/coverage.svg?branch=0.10)](https://codecov.io/gh/fail2ban/fail2ban/branch/0.10) (0.10 branch) Contact: -------- From ed4b5b8bdc8914e9d8dc68ca889f7c05ea194986 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 15 Jan 2020 13:06:21 +0100 Subject: [PATCH 013/240] change actioncheck behavior (on error only, gh-488) --- fail2ban/server/action.py | 51 ++++++++++++++++++------------- fail2ban/tests/actionstestcase.py | 6 ++++ fail2ban/tests/actiontestcase.py | 46 ++++++++++++++++++++++++++-- fail2ban/tests/servertestcase.py | 21 ++++++++++--- 4 files changed, 95 insertions(+), 29 deletions(-) diff --git a/fail2ban/server/action.py b/fail2ban/server/action.py index 5c817fc0..4df7217e 100644 --- a/fail2ban/server/action.py +++ b/fail2ban/server/action.py @@ -407,7 +407,7 @@ class CommandAction(ActionBase): cmd = self.replaceTag(tag, self._properties, conditional=('family='+family if family else ''), cache=self.__substCache) - if '<' not in cmd or not family: return cmd + if not family or '<' not in cmd: return cmd # replace family as dynamic tags, important - don't cache, no recursion and auto-escape here: cmd = self.replaceDynamicTags(cmd, {'family':family}) return cmd @@ -962,31 +962,38 @@ class CommandAction(ActionBase): except (KeyError, TypeError): family = '' - # invariant check: - if self.actioncheck: - # don't repair/restore if unban (no matter): - def _beforeRepair(): - if cmd == '' and not self._properties.get('actionrepair_on_unban'): - self._logSys.error("Invariant check failed. Unban is impossible.") + repcnt = 0 + while True: + + # got some error, do invariant check: + if repcnt and self.actioncheck: + # don't repair/restore if unban (no matter): + def _beforeRepair(): + if cmd == '' and not self._properties.get('actionrepair_on_unban'): + self._logSys.error("Invariant check failed. Unban is impossible.") + return False + return True + # check and repair if broken: + ret = self._invariantCheck(family, _beforeRepair, forceStart=(cmd != '')) + # if not sane (and not restored) return: + if ret != 1: return False - return True - # check and repair if broken: - ret = self._invariantCheck(family, _beforeRepair, forceStart=(cmd != '')) - # if not sane (and not restored) return: - if ret != 1: - return False - # Replace static fields - realCmd = self.replaceTag(cmd, self._properties, - conditional=('family='+family if family else ''), cache=self.__substCache) + # Replace static fields + realCmd = self.replaceTag(cmd, self._properties, + conditional=('family='+family if family else ''), cache=self.__substCache) - # Replace dynamical tags, important - don't cache, no recursion and auto-escape here - if aInfo is not None: - realCmd = self.replaceDynamicTags(realCmd, aInfo) - else: - realCmd = cmd + # Replace dynamical tags, important - don't cache, no recursion and auto-escape here + if aInfo is not None: + realCmd = self.replaceDynamicTags(realCmd, aInfo) + else: + realCmd = cmd - return self.executeCmd(realCmd, self.timeout) + # try execute command: + ret = self.executeCmd(realCmd, self.timeout) + repcnt += 1 + if ret or repcnt > 1: + return ret @staticmethod def executeCmd(realCmd, timeout=60, **kwargs): diff --git a/fail2ban/tests/actionstestcase.py b/fail2ban/tests/actionstestcase.py index a3c14a4b..b324b2c4 100644 --- a/fail2ban/tests/actionstestcase.py +++ b/fail2ban/tests/actionstestcase.py @@ -215,6 +215,9 @@ class ExecuteActions(LogCaptureTestCase): # flush for inet6 is intentionally "broken" here - test no unhandled except and invariant check: act['actionflush?family=inet6'] = act.actionflush + '; exit 1' act.actionstart_on_demand = True + # force errors via check in ban/unban: + act.actionban = " ; " + act.actionban + act.actionunban = " ; " + act.actionunban self.__actions.start() self.assertNotLogged("stdout: %r" % 'ip start') @@ -292,6 +295,9 @@ class ExecuteActions(LogCaptureTestCase): act['actionflush?family=inet6'] = act.actionflush + '; exit 1' act.actionstart_on_demand = True act.actionrepair_on_unban = True + # force errors via check in ban/unban: + act.actionban = " ; " + act.actionban + act.actionunban = " ; " + act.actionunban self.__actions.start() self.assertNotLogged("stdout: %r" % 'ip start') diff --git a/fail2ban/tests/actiontestcase.py b/fail2ban/tests/actiontestcase.py index 1a00c040..2dab3ab1 100644 --- a/fail2ban/tests/actiontestcase.py +++ b/fail2ban/tests/actiontestcase.py @@ -305,8 +305,8 @@ class CommandActionTest(LogCaptureTestCase): self.assertEqual(self.__action.actionstart, "touch '%s'" % tmp) self.__action.actionstop = "rm -f '%s'" % tmp self.assertEqual(self.__action.actionstop, "rm -f '%s'" % tmp) - self.__action.actionban = "echo -n" - self.assertEqual(self.__action.actionban, 'echo -n') + self.__action.actionban = " && echo -n" + self.assertEqual(self.__action.actionban, " && echo -n") self.__action.actioncheck = "[ -e '%s' ]" % tmp self.assertEqual(self.__action.actioncheck, "[ -e '%s' ]" % tmp) self.__action.actionunban = "true" @@ -316,6 +316,7 @@ class CommandActionTest(LogCaptureTestCase): self.assertNotLogged('returned') # no action was actually executed yet + # start on demand is false, so it should cause failure on first attempt of ban: self.__action.ban({'ip': None}) self.assertLogged('Invariant check failed') self.assertLogged('returned successfully') @@ -365,12 +366,51 @@ class CommandActionTest(LogCaptureTestCase): self.pruneLog('[phase 2]') self.__action.actionstart = "touch '%s'" % tmp self.__action.actionstop = "rm '%s'" % tmp - self.__action.actionban = """printf "%%%%b\n" >> '%s'""" % tmp + self.__action.actionban = """ && printf "%%%%b\n" >> '%s'""" % tmp self.__action.actioncheck = "[ -e '%s' ]" % tmp self.__action.ban({'ip': None}) self.assertLogged('Invariant check failed') self.assertNotLogged('Unable to restore environment') + @with_tmpdir + def testExecuteActionCheckOnBanFailure(self, tmp): + tmp += '/fail2ban.test' + self.__action.actionstart = "touch '%s'; echo 'started ...'" % tmp + self.__action.actionstop = "rm -f '%s'" % tmp + self.__action.actionban = "[ -e '%s' ] && echo 'banned '" % tmp + self.__action.actioncheck = "[ -e '%s' ] && echo 'check ok' || { echo 'check failed'; exit 1; }" % tmp + self.__action.actionrepair = "echo 'repair ...'; touch '%s'" % tmp + self.__action.actionstart_on_demand = False + self.__action.start() + # phase 1: with repair; + # phase 2: without repair (start/stop), not on demand; + # phase 3: without repair (start/stop), start on demand. + for i in (1, 2, 3): + self.pruneLog('[phase %s]' % i) + # 1st time with success ban: + self.__action.ban({'ip': '192.0.2.1'}) + self.assertLogged( + "stdout: %r" % 'banned 192.0.2.1', all=True) + self.assertNotLogged("Invariant check failed. Trying", + "stdout: %r" % 'check failed', + "stdout: %r" % ('repair ...' if self.__action.actionrepair else 'started ...'), + "stdout: %r" % 'check ok', all=True) + # force error in ban: + os.remove(tmp) + self.pruneLog() + # 2nd time with fail recognition, success repair, check and ban: + self.__action.ban({'ip': '192.0.2.2'}) + self.assertLogged("Invariant check failed. Trying", + "stdout: %r" % 'check failed', + "stdout: %r" % ('repair ...' if self.__action.actionrepair else 'started ...'), + "stdout: %r" % 'check ok', + "stdout: %r" % 'banned 192.0.2.2', all=True) + # repeat without repair (stop/start), herafter enable on demand: + if self.__action.actionrepair: + self.__action.actionrepair = "" + elif not self.__action.actionstart_on_demand: + self.__action.actionstart_on_demand = True + @with_tmpdir def testExecuteActionCheckRepairEnvironment(self, tmp): tmp += '/fail2ban.test' diff --git a/fail2ban/tests/servertestcase.py b/fail2ban/tests/servertestcase.py index 55e72455..71347c7e 100644 --- a/fail2ban/tests/servertestcase.py +++ b/fail2ban/tests/servertestcase.py @@ -2030,25 +2030,38 @@ class ServerConfigReaderTests(LogCaptureTestCase): action.ban(aInfos['ipv4']) if tests.get('ip4-start'): self.assertLogged(*tests.get('*-start', ())+tests['ip4-start'], all=True) if tests.get('ip6-start'): self.assertNotLogged(*tests['ip6-start'], all=True) - self.assertLogged(*tests.get('ip4-check',())+tests['ip4-ban'], all=True) + self.assertLogged(*tests['ip4-ban'], all=True) self.assertNotLogged(*tests['ip6'], all=True) # test unban ip4 : self.pruneLog('# === unban ipv4 ===') action.unban(aInfos['ipv4']) - self.assertLogged(*tests.get('ip4-check',())+tests['ip4-unban'], all=True) + self.assertLogged(*tests['ip4-unban'], all=True) self.assertNotLogged(*tests['ip6'], all=True) # test ban ip6 : self.pruneLog('# === ban ipv6 ===') action.ban(aInfos['ipv6']) if tests.get('ip6-start'): self.assertLogged(*tests.get('*-start', ())+tests['ip6-start'], all=True) if tests.get('ip4-start'): self.assertNotLogged(*tests['ip4-start'], all=True) - self.assertLogged(*tests.get('ip6-check',())+tests['ip6-ban'], all=True) + self.assertLogged(*tests['ip6-ban'], all=True) self.assertNotLogged(*tests['ip4'], all=True) # test unban ip6 : self.pruneLog('# === unban ipv6 ===') action.unban(aInfos['ipv6']) - self.assertLogged(*tests.get('ip6-check',())+tests['ip6-unban'], all=True) + self.assertLogged(*tests['ip6-unban'], all=True) self.assertNotLogged(*tests['ip4'], all=True) + # test invariant check (normally on demand in error case only): + if tests.get('ip4-check'): + self.pruneLog('# === check ipv4 ===') + action._invariantCheck(aInfos['ipv4']['family']) + self.assertLogged(*tests['ip4-check'], all=True) + if tests.get('ip6-check') and tests['ip6-check'] != tests['ip4-check']: + self.assertNotLogged(*tests['ip6-check'], all=True) + if tests.get('ip6-check'): + self.pruneLog('# === check ipv6 ===') + action._invariantCheck(aInfos['ipv6']['family']) + self.assertLogged(*tests['ip6-check'], all=True) + if tests.get('ip4-check') and tests['ip4-check'] != tests['ip6-check']: + self.assertNotLogged(*tests['ip4-check'], all=True) # test flush for actions should supported this: if tests.get('flush'): self.pruneLog('# === flush ===') From 8cbc1e0ebb91142a34756fd2f4416b08d773deaa Mon Sep 17 00:00:00 2001 From: sebres Date: Thu, 16 Jan 2020 16:51:27 +0100 Subject: [PATCH 014/240] ChangeLog (change actioncheck behavior) --- ChangeLog | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/ChangeLog b/ChangeLog index 3f9c11b1..73e0e047 100644 --- a/ChangeLog +++ b/ChangeLog @@ -9,11 +9,20 @@ Fail2Ban: Changelog ver. 1.0.1-dev-1 (20??/??/??) - development nightly edition ----------- +### Compatibility: +* to v.0.11: + - due to change of `actioncheck` behavior (gh-488), some actions can be incompatible as regards + the invariant check, if `actionban` or `actionunban` would not throw an error (exit code + different from 0) in case of unsane environment. + ### Fixes ### New Features ### Enhancements +* `actioncheck` behavior is changed now (gh-488), so invariant check as well as restore or repair + of sane environment (in case of recognized unsane state) would only occur on action errors (e. g. + if ban or unban operations are exiting with other code as 0) ver. 0.11.1 (2020/01/11) - this-is-the-way From 303861d7c752492a245bcf954fc4edf336b9e35d Mon Sep 17 00:00:00 2001 From: Mihail Politaev Date: Thu, 30 Jan 2020 21:17:32 +0200 Subject: [PATCH 015/240] Using native firewalld ipset implementation By creating additional action file firewallcmd-ipset-native.conf --- config/action.d/firewallcmd-ipset-native.conf | 77 +++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 config/action.d/firewallcmd-ipset-native.conf diff --git a/config/action.d/firewallcmd-ipset-native.conf b/config/action.d/firewallcmd-ipset-native.conf new file mode 100644 index 00000000..757d46ad --- /dev/null +++ b/config/action.d/firewallcmd-ipset-native.conf @@ -0,0 +1,77 @@ +# Fail2Ban action file for firewall-cmd using native ipset implementation +# +# This requires: +# ipset (package: ipset) +# firewall-cmd (package: firewalld) +# +# This is for ipset protocol 6 (and hopefully later) (ipset v6.14). +# Use ipset -V to see the protocol and version. +# +# IPset was a feature introduced in the linux kernel 2.6.39 and 3.0.0 kernels. +# +# If you are running on an older kernel you make need to patch in external +# modules. + +[INCLUDES] + +before = firewallcmd-common.conf + +[Definition] + +actionstart = firewall-cmd --permanent --new-ipset= --type=hash:ip --option=timeout= + firewall-cmd --reload + firewall-cmd --direct --add-rule filter 0 -m set --match-set src -j + +actionstop = firewall-cmd --direct --remove-rule filter 0 -m set --match-set src -j + firewall-cmd --permanent --delete-ipset= + firewall-cmd --reload + +actionban = firewall-cmd --ipset= --add-entry= + +actionunban = firewall-cmd --ipset= --remove-entry= + +[Init] + +# Option: chain +# Notes specifies the iptables chain to which the fail2ban rules should be +# added +# Values: [ STRING ] +# +chain = INPUT_direct + +# Option: bantime +# Notes: specifies the bantime in seconds (handled internally rather than by fail2ban) +# Values: [ NUM ] Default: 600 + +bantime = 86400 + +# Option: actiontype +# Notes.: defines additions to the blocking rule +# Values: leave empty to block all attempts from the host +# Default: Value of the multiport +actiontype = + +# Option: allports +# Notes.: default addition to block all ports +# Usage.: use in jail config: banaction = firewallcmd-ipset[actiontype=] +# for all protocols: banaction = firewallcmd-ipset[actiontype=""] +allports = -p + +# Option: multiport +# Notes.: addition to block access only to specific ports +# Usage.: use in jail config: banaction = firewallcmd-ipset[actiontype=] +multiport = -p -m multiport --dports + +ipmset = f2b- +familyopt = + +[Init?family=inet6] + +ipmset = f2b-6 +familyopt = family inet6 + + +# DEV NOTES: +# +# Author: Edgar Hoch and Daniel Black and Mihail Politaev +# firewallcmd-new / iptables-ipset-proto6 combined for maximium goodness From d26209e2c6683298ef8662025693428aec5211a2 Mon Sep 17 00:00:00 2001 From: sebres Date: Thu, 16 Jan 2020 20:08:35 +0100 Subject: [PATCH 016/240] first attempt to make certain standard actions breakdown safe starting with iptables: - better check mechanism (using `-C`, option `--check` is available long time); - additionally iptables is a replacement for iptables-common now, several actions using this as include now become obsolete; - many features of different iptables actions are combinable as single chain/rule (can be supplied to action as parameters); --- MANIFEST | 1 - config/action.d/iptables-allports.conf | 46 +------ config/action.d/iptables-common.conf | 92 -------------- config/action.d/iptables-ipset-proto4.conf | 2 +- .../iptables-ipset-proto6-allports.conf | 2 +- config/action.d/iptables-ipset-proto6.conf | 2 +- config/action.d/iptables-multiport-log.conf | 2 +- config/action.d/iptables-multiport.conf | 44 +------ config/action.d/iptables-new.conf | 45 +------ config/action.d/iptables-xt_recent-echo.conf | 4 +- config/action.d/iptables.conf | 117 ++++++++++++++++-- .../symbiosis-blacklist-allports.conf | 7 +- fail2ban/tests/clientreadertestcase.py | 2 +- fail2ban/tests/servertestcase.py | 22 ++-- 14 files changed, 141 insertions(+), 247 deletions(-) delete mode 100644 config/action.d/iptables-common.conf diff --git a/MANIFEST b/MANIFEST index 630df5ea..f6322573 100644 --- a/MANIFEST +++ b/MANIFEST @@ -25,7 +25,6 @@ config/action.d/hostsdeny.conf config/action.d/ipfilter.conf config/action.d/ipfw.conf config/action.d/iptables-allports.conf -config/action.d/iptables-common.conf config/action.d/iptables.conf config/action.d/iptables-ipset-proto4.conf config/action.d/iptables-ipset-proto6-allports.conf diff --git a/config/action.d/iptables-allports.conf b/config/action.d/iptables-allports.conf index caf9ab81..51c4694d 100644 --- a/config/action.d/iptables-allports.conf +++ b/config/action.d/iptables-allports.conf @@ -4,52 +4,12 @@ # Modified: Yaroslav O. Halchenko # made active on all ports from original iptables.conf # -# +# Obsolete: superseded by iptables[type=allports] [INCLUDES] -before = iptables-common.conf - +before = iptables.conf [Definition] -# Option: actionstart -# Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false). -# Values: CMD -# -actionstart = -N f2b- - -A f2b- -j - -I -p -j f2b- - -# Option: actionstop -# Notes.: command executed at the stop of jail (or at the end of Fail2Ban) -# Values: CMD -# -actionstop = -D -p -j f2b- - - -X f2b- - -# Option: actioncheck -# Notes.: command executed once before each actionban command -# Values: CMD -# -actioncheck = -n -L | grep -q 'f2b-[ \t]' - -# Option: actionban -# Notes.: command executed when banning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionban = -I f2b- 1 -s -j - -# Option: actionunban -# Notes.: command executed when unbanning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionunban = -D f2b- -s -j - -[Init] - +type = allports diff --git a/config/action.d/iptables-common.conf b/config/action.d/iptables-common.conf deleted file mode 100644 index e016ef2f..00000000 --- a/config/action.d/iptables-common.conf +++ /dev/null @@ -1,92 +0,0 @@ -# Fail2Ban configuration file -# -# Author: Daniel Black -# -# This is a included configuration file and includes the definitions for the iptables -# used in all iptables based actions by default. -# -# The user can override the defaults in iptables-common.local -# -# Modified: Alexander Koeppe , Serg G. Brester -# made config file IPv6 capable (see new section Init?family=inet6) - -[INCLUDES] - -after = iptables-blocktype.local - iptables-common.local -# iptables-blocktype.local is obsolete - -[Definition] - -# Option: actionflush -# Notes.: command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action) -# Values: CMD -# -actionflush = -F f2b- - - -[Init] - -# Option: chain -# Notes specifies the iptables chain to which the Fail2Ban rules should be -# added -# Values: STRING Default: INPUT -chain = INPUT - -# Default name of the chain -# -name = default - -# Option: port -# Notes.: specifies port to monitor -# Values: [ NUM | STRING ] Default: -# -port = ssh - -# Option: protocol -# Notes.: internally used by config reader for interpolations. -# Values: [ tcp | udp | icmp | all ] Default: tcp -# -protocol = tcp - -# Option: blocktype -# Note: This is what the action does with rules. This can be any jump target -# as per the iptables man page (section 8). Common values are DROP -# REJECT, REJECT --reject-with icmp-port-unreachable -# Values: STRING -blocktype = REJECT --reject-with icmp-port-unreachable - -# Option: returntype -# Note: This is the default rule on "actionstart". This should be RETURN -# in all (blocking) actions, except REJECT in allowing actions. -# Values: STRING -returntype = RETURN - -# Option: lockingopt -# Notes.: Option was introduced to iptables to prevent multiple instances from -# running concurrently and causing irratic behavior. -w was introduced -# in iptables 1.4.20, so might be absent on older systems -# See https://github.com/fail2ban/fail2ban/issues/1122 -# Values: STRING -lockingopt = -w - -# Option: iptables -# Notes.: Actual command to be executed, including common to all calls options -# Values: STRING -iptables = iptables - - -[Init?family=inet6] - -# Option: blocktype (ipv6) -# Note: This is what the action does with rules. This can be any jump target -# as per the iptables man page (section 8). Common values are DROP -# REJECT, REJECT --reject-with icmp6-port-unreachable -# Values: STRING -blocktype = REJECT --reject-with icmp6-port-unreachable - -# Option: iptables (ipv6) -# Notes.: Actual command to be executed, including common to all calls options -# Values: STRING -iptables = ip6tables - diff --git a/config/action.d/iptables-ipset-proto4.conf b/config/action.d/iptables-ipset-proto4.conf index 99ebbf8c..b46893cb 100644 --- a/config/action.d/iptables-ipset-proto4.conf +++ b/config/action.d/iptables-ipset-proto4.conf @@ -19,7 +19,7 @@ [INCLUDES] -before = iptables-common.conf +before = iptables.conf [Definition] diff --git a/config/action.d/iptables-ipset-proto6-allports.conf b/config/action.d/iptables-ipset-proto6-allports.conf index c851233c..bbef6fb7 100644 --- a/config/action.d/iptables-ipset-proto6-allports.conf +++ b/config/action.d/iptables-ipset-proto6-allports.conf @@ -18,7 +18,7 @@ [INCLUDES] -before = iptables-common.conf +before = iptables.conf [Definition] diff --git a/config/action.d/iptables-ipset-proto6.conf b/config/action.d/iptables-ipset-proto6.conf index 12c3ddd6..4fa0860d 100644 --- a/config/action.d/iptables-ipset-proto6.conf +++ b/config/action.d/iptables-ipset-proto6.conf @@ -18,7 +18,7 @@ [INCLUDES] -before = iptables-common.conf +before = iptables.conf [Definition] diff --git a/config/action.d/iptables-multiport-log.conf b/config/action.d/iptables-multiport-log.conf index df126dbf..322a7491 100644 --- a/config/action.d/iptables-multiport-log.conf +++ b/config/action.d/iptables-multiport-log.conf @@ -11,7 +11,7 @@ [INCLUDES] -before = iptables-common.conf +before = iptables.conf [Definition] diff --git a/config/action.d/iptables-multiport.conf b/config/action.d/iptables-multiport.conf index 41b00c54..008208e0 100644 --- a/config/action.d/iptables-multiport.conf +++ b/config/action.d/iptables-multiport.conf @@ -3,50 +3,12 @@ # Author: Cyril Jaquier # Modified by Yaroslav Halchenko for multiport banning # +# Obsolete: superseded by iptables[type=multiport] [INCLUDES] -before = iptables-common.conf +before = iptables.conf [Definition] -# Option: actionstart -# Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false). -# Values: CMD -# -actionstart = -N f2b- - -A f2b- -j - -I -p -m multiport --dports -j f2b- - -# Option: actionstop -# Notes.: command executed at the stop of jail (or at the end of Fail2Ban) -# Values: CMD -# -actionstop = -D -p -m multiport --dports -j f2b- - - -X f2b- - -# Option: actioncheck -# Notes.: command executed once before each actionban command -# Values: CMD -# -actioncheck = -n -L | grep -q 'f2b-[ \t]' - -# Option: actionban -# Notes.: command executed when banning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionban = -I f2b- 1 -s -j - -# Option: actionunban -# Notes.: command executed when unbanning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionunban = -D f2b- -s -j - -[Init] - +type = multiport diff --git a/config/action.d/iptables-new.conf b/config/action.d/iptables-new.conf index 39a17099..170cb934 100644 --- a/config/action.d/iptables-new.conf +++ b/config/action.d/iptables-new.conf @@ -4,51 +4,12 @@ # Copied from iptables.conf and modified by Yaroslav Halchenko # to fulfill the needs of bugreporter dbts#350746. # -# +# Obsolete: superseded by iptables[pre-rule='-m state --state NEW'] [INCLUDES] -before = iptables-common.conf +before = iptables.conf [Definition] -# Option: actionstart -# Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false). -# Values: CMD -# -actionstart = -N f2b- - -A f2b- -j - -I -m state --state NEW -p --dport -j f2b- - -# Option: actionstop -# Notes.: command executed at the stop of jail (or at the end of Fail2Ban) -# Values: CMD -# -actionstop = -D -m state --state NEW -p --dport -j f2b- - - -X f2b- - -# Option: actioncheck -# Notes.: command executed once before each actionban command -# Values: CMD -# -actioncheck = -n -L | grep -q 'f2b-[ \t]' - -# Option: actionban -# Notes.: command executed when banning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionban = -I f2b- 1 -s -j - -# Option: actionunban -# Notes.: command executed when unbanning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionunban = -D f2b- -s -j - -[Init] - +pre-rule = -m state --state NEW \ No newline at end of file diff --git a/config/action.d/iptables-xt_recent-echo.conf b/config/action.d/iptables-xt_recent-echo.conf index 97449222..7e24db7a 100644 --- a/config/action.d/iptables-xt_recent-echo.conf +++ b/config/action.d/iptables-xt_recent-echo.conf @@ -7,7 +7,7 @@ [INCLUDES] -before = iptables-common.conf +before = iptables.conf [Definition] @@ -52,7 +52,7 @@ actionstop = echo / > /proc/net/xt_recent/ # Notes.: command executed once before each actionban command # Values: CMD # -actioncheck = test -e /proc/net/xt_recent/ +actioncheck = { ; } && test -e /proc/net/xt_recent/ # Option: actionban # Notes.: command executed when banning an IP. Take care that the diff --git a/config/action.d/iptables.conf b/config/action.d/iptables.conf index 8ed5fdad..d31b49aa 100644 --- a/config/action.d/iptables.conf +++ b/config/action.d/iptables.conf @@ -1,28 +1,42 @@ # Fail2Ban configuration file # -# Author: Cyril Jaquier +# Authors: Sergey G. Brester (sebres), Cyril Jaquier, Daniel Black, +# Yaroslav O. Halchenko, Alexander Koeppe et al. # -# - -[INCLUDES] - -before = iptables-common.conf [Definition] +# Option: type +# Notes.: type of the action. +# Values: [ oneport | multiport | allports ] Default: oneport +# +type = oneport + +# Option: rule +# Notes.: type of the iptables rule. +# Values: [ iptables | ipset ] Default: iptables +# +rule = iptables + +# Option: actionflush +# Notes.: command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action) +# Values: CMD +# +actionflush = -F f2b- + # Option: actionstart # Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false). # Values: CMD # actionstart = -N f2b- -A f2b- -j - -I -p --dport -j f2b- + -I %(_ipt_chain_rule)s # Option: actionstop # Notes.: command executed at the stop of jail (or at the end of Fail2Ban) # Values: CMD # -actionstop = -D -p --dport -j f2b- +actionstop = -D %(_ipt_chain_rule)s -X f2b- @@ -30,7 +44,7 @@ actionstop = -D -p --dport -j f2b- # Notes.: command executed once before each actionban command # Values: CMD # -actioncheck = -n -L | grep -q 'f2b-[ \t]' +actioncheck = %(_ipt_check_rule)s # Option: actionban # Notes.: command executed when banning an IP. Take care that the @@ -48,5 +62,90 @@ actionban = -I f2b- 1 -s -j # actionunban = -D f2b- -s -j +# Option: pre-rule +# Notes.: prefix parameter(s) inserted to the begin of rule. No default (empty) +# +pre-rule = + +# Several capabilities used internaly: + +_ipt_chain_rule = /_chain_rule> +_ipt_check_rule = -C %(_ipt_chain_rule)s + +[ipt_oneport] + +_chain_rule = -p --dport -j f2b- + +[ipt_multiport] + +_chain_rule = -p -m multiport --dports -j f2b- + +[ipt_allports] + +_chain_rule = -p -j f2b- + + [Init] +# Option: chain +# Notes specifies the iptables chain to which the Fail2Ban rules should be +# added +# Values: STRING Default: INPUT +chain = INPUT + +# Default name of the chain +# +name = default + +# Option: port +# Notes.: specifies port to monitor +# Values: [ NUM | STRING ] Default: +# +port = ssh + +# Option: protocol +# Notes.: internally used by config reader for interpolations. +# Values: [ tcp | udp | icmp | all ] Default: tcp +# +protocol = tcp + +# Option: blocktype +# Note: This is what the action does with rules. This can be any jump target +# as per the iptables man page (section 8). Common values are DROP +# REJECT, REJECT --reject-with icmp-port-unreachable +# Values: STRING +blocktype = REJECT --reject-with icmp-port-unreachable + +# Option: returntype +# Note: This is the default rule on "actionstart". This should be RETURN +# in all (blocking) actions, except REJECT in allowing actions. +# Values: STRING +returntype = RETURN + +# Option: lockingopt +# Notes.: Option was introduced to iptables to prevent multiple instances from +# running concurrently and causing irratic behavior. -w was introduced +# in iptables 1.4.20, so might be absent on older systems +# See https://github.com/fail2ban/fail2ban/issues/1122 +# Values: STRING +lockingopt = -w + +# Option: iptables +# Notes.: Actual command to be executed, including common to all calls options +# Values: STRING +iptables = iptables + + +[Init?family=inet6] + +# Option: blocktype (ipv6) +# Note: This is what the action does with rules. This can be any jump target +# as per the iptables man page (section 8). Common values are DROP +# REJECT, REJECT --reject-with icmp6-port-unreachable +# Values: STRING +blocktype = REJECT --reject-with icmp6-port-unreachable + +# Option: iptables (ipv6) +# Notes.: Actual command to be executed, including common to all calls options +# Values: STRING +iptables = ip6tables diff --git a/config/action.d/symbiosis-blacklist-allports.conf b/config/action.d/symbiosis-blacklist-allports.conf index 6fb7d0af..7208b293 100644 --- a/config/action.d/symbiosis-blacklist-allports.conf +++ b/config/action.d/symbiosis-blacklist-allports.conf @@ -5,7 +5,7 @@ [INCLUDES] -before = iptables-common.conf +before = iptables.conf [Definition] @@ -41,6 +41,11 @@ actionban = echo 'all' >| /etc/symbiosis/firewall/blacklist.d/.auto actionunban = rm -f /etc/symbiosis/firewall/blacklist.d/.auto -D -s -j || : +# [TODO] Flushing is currently not implemented for symbiosis blacklist.d +# +actionflush = + + [Init] # Option: chain diff --git a/fail2ban/tests/clientreadertestcase.py b/fail2ban/tests/clientreadertestcase.py index 2c1d0a0e..c44e72ec 100644 --- a/fail2ban/tests/clientreadertestcase.py +++ b/fail2ban/tests/clientreadertestcase.py @@ -656,7 +656,7 @@ class JailsReaderTestCache(LogCaptureTestCase): cnt = self._getLoggedReadCount(r'filter\.d/common\.conf') self.assertTrue(cnt == 1, "Unexpected count by reading of filter files, cnt = %s" % cnt) # same with action: - cnt = self._getLoggedReadCount(r'action\.d/iptables-common\.conf') + cnt = self._getLoggedReadCount(r'action\.d/iptables\.conf') self.assertTrue(cnt == 1, "Unexpected count by reading of action files, cnt = %s" % cnt) finally: configparserinc.logLevel = saved_ll diff --git a/fail2ban/tests/servertestcase.py b/fail2ban/tests/servertestcase.py index 71347c7e..4ecb1ca5 100644 --- a/fail2ban/tests/servertestcase.py +++ b/fail2ban/tests/servertestcase.py @@ -1511,10 +1511,10 @@ class ServerConfigReaderTests(LogCaptureTestCase): "`ip6tables -w -X f2b-j-w-iptables-mp`", ), 'ip4-check': ( - r"""`iptables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-mp[ \t]'`""", + r"""`iptables -w -C INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`""", ), 'ip6-check': ( - r"""`ip6tables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-mp[ \t]'`""", + r"""`ip6tables -w -C INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`""", ), 'ip4-ban': ( r"`iptables -w -I f2b-j-w-iptables-mp 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", @@ -1555,10 +1555,10 @@ class ServerConfigReaderTests(LogCaptureTestCase): "`ip6tables -w -X f2b-j-w-iptables-ap`", ), 'ip4-check': ( - r"""`iptables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-ap[ \t]'`""", + r"""`iptables -w -C INPUT -p tcp -j f2b-j-w-iptables-ap`""", ), 'ip6-check': ( - r"""`ip6tables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-ap[ \t]'`""", + r"""`ip6tables -w -C INPUT -p tcp -j f2b-j-w-iptables-ap`""", ), 'ip4-ban': ( r"`iptables -w -I f2b-j-w-iptables-ap 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", @@ -1645,7 +1645,7 @@ class ServerConfigReaderTests(LogCaptureTestCase): r"`ipset del f2b-j-w-iptables-ipset-ap6 2001:db8:: -exist`", ), }), - # iptables -- + # iptables (oneport) -- ('j-w-iptables', 'iptables[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain=""]', { 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), 'ip4-start': ( @@ -1671,10 +1671,10 @@ class ServerConfigReaderTests(LogCaptureTestCase): "`ip6tables -w -X f2b-j-w-iptables`", ), 'ip4-check': ( - r"""`iptables -w -n -L INPUT | grep -q 'f2b-j-w-iptables[ \t]'`""", + r"""`iptables -w -C INPUT -p tcp --dport http -j f2b-j-w-iptables`""", ), 'ip6-check': ( - r"""`ip6tables -w -n -L INPUT | grep -q 'f2b-j-w-iptables[ \t]'`""", + r"""`ip6tables -w -C INPUT -p tcp --dport http -j f2b-j-w-iptables`""", ), 'ip4-ban': ( r"`iptables -w -I f2b-j-w-iptables 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", @@ -1715,10 +1715,10 @@ class ServerConfigReaderTests(LogCaptureTestCase): "`ip6tables -w -X f2b-j-w-iptables-new`", ), 'ip4-check': ( - r"""`iptables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-new[ \t]'`""", + r"""`iptables -w -C INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`""", ), 'ip6-check': ( - r"""`ip6tables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-new[ \t]'`""", + r"""`ip6tables -w -C INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`""", ), 'ip4-ban': ( r"`iptables -w -I f2b-j-w-iptables-new 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", @@ -1749,10 +1749,10 @@ class ServerConfigReaderTests(LogCaptureTestCase): "`if [ `id -u` -eq 0 ];then ip6tables -w -D INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre6 -j REJECT --reject-with icmp6-port-unreachable;fi`", ), 'ip4-check': ( - r"`test -e /proc/net/xt_recent/f2b-j-w-iptables-xtre`", + r"`{ iptables -w -C INPUT -p tcp --dport ssh -j f2b-j-w-iptables-xtre; } && test -e /proc/net/xt_recent/f2b-j-w-iptables-xtre`", ), 'ip6-check': ( - r"`test -e /proc/net/xt_recent/f2b-j-w-iptables-xtre6`", + r"`{ ip6tables -w -C INPUT -p tcp --dport ssh -j f2b-j-w-iptables-xtre; } && test -e /proc/net/xt_recent/f2b-j-w-iptables-xtre6`", ), 'ip4-ban': ( r"`echo +192.0.2.1 > /proc/net/xt_recent/f2b-j-w-iptables-xtre`", From ceeba99f25763378c19e5779ddbd0ac7d31caa88 Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 17 Jan 2020 08:56:15 +0100 Subject: [PATCH 017/240] replace internals of several iptables-ipset actions using internals of iptables include: - better check mechanism (using `-C`, option `--check` is available long time); - additionally iptables-ipset is a common action for iptables-ipset-proto6-* now (which become obsolete now); - many features of different iptables actions are combinable as single chain/rule (can be supplied to action as parameters); - tests adjusted. --- config/action.d/iptables-ipset-proto4.conf | 5 +- .../iptables-ipset-proto6-allports.conf | 60 +------------- config/action.d/iptables-ipset-proto6.conf | 60 +------------- config/action.d/iptables-ipset.conf | 82 +++++++++++++++++++ config/action.d/iptables.conf | 15 ++-- fail2ban/tests/servertestcase.py | 20 ++++- 6 files changed, 115 insertions(+), 127 deletions(-) create mode 100644 config/action.d/iptables-ipset.conf diff --git a/config/action.d/iptables-ipset-proto4.conf b/config/action.d/iptables-ipset-proto4.conf index b46893cb..2e2f5779 100644 --- a/config/action.d/iptables-ipset-proto4.conf +++ b/config/action.d/iptables-ipset-proto4.conf @@ -28,7 +28,7 @@ before = iptables.conf # Values: CMD # actionstart = ipset --create f2b- iphash - -I -p -m multiport --dports -m set --match-set f2b- src -j + -I %(_ipt_chain_rule)s # Option: actionflush @@ -61,5 +61,6 @@ actionban = ipset --test f2b- || ipset --add f2b- # actionunban = ipset --test f2b- && ipset --del f2b- -[Init] +# Several capabilities used internaly: +rule-jump = -m set --match-set f2b- src -j diff --git a/config/action.d/iptables-ipset-proto6-allports.conf b/config/action.d/iptables-ipset-proto6-allports.conf index bbef6fb7..1aa7fd6f 100644 --- a/config/action.d/iptables-ipset-proto6-allports.conf +++ b/config/action.d/iptables-ipset-proto6-allports.conf @@ -15,65 +15,13 @@ # # Modified: Alexander Koeppe , Serg G. Brester # made config file IPv6 capable (see new section Init?family=inet6) +# +# Obsolete: superseded by iptables-ipset[type=allports] [INCLUDES] -before = iptables.conf +before = iptables-ipset.conf [Definition] -# Option: actionstart -# Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false). -# Values: CMD -# -actionstart = ipset create hash:ip timeout - -I -m set --match-set src -j - -# Option: actionflush -# Notes.: command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action) -# Values: CMD -# -actionflush = ipset flush - -# Option: actionstop -# Notes.: command executed at the stop of jail (or at the end of Fail2Ban) -# Values: CMD -# -actionstop = -D -m set --match-set src -j - - ipset destroy - -# Option: actionban -# Notes.: command executed when banning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionban = ipset add timeout -exist - -actionprolong = %(actionban)s - -# Option: actionunban -# Notes.: command executed when unbanning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionunban = ipset del -exist - -[Init] - -# Option: default-timeout -# Notes: specifies default timeout in seconds (handled default ipset timeout only) -# Values: [ NUM ] Default: 600 - -default-timeout = 600 - -ipmset = f2b- -familyopt = - - -[Init?family=inet6] - -ipmset = f2b-6 -familyopt = family inet6 +type = allports diff --git a/config/action.d/iptables-ipset-proto6.conf b/config/action.d/iptables-ipset-proto6.conf index 4fa0860d..ef744984 100644 --- a/config/action.d/iptables-ipset-proto6.conf +++ b/config/action.d/iptables-ipset-proto6.conf @@ -15,65 +15,13 @@ # # Modified: Alexander Koeppe , Serg G. Brester # made config file IPv6 capable (see new section Init?family=inet6) +# +# Obsolete: superseded by iptables-ipset[type=multiport] [INCLUDES] -before = iptables.conf +before = iptables-ipset.conf [Definition] -# Option: actionstart -# Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false). -# Values: CMD -# -actionstart = ipset create hash:ip timeout - -I -p -m multiport --dports -m set --match-set src -j - -# Option: actionflush -# Notes.: command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action) -# Values: CMD -# -actionflush = ipset flush - -# Option: actionstop -# Notes.: command executed at the stop of jail (or at the end of Fail2Ban) -# Values: CMD -# -actionstop = -D -p -m multiport --dports -m set --match-set src -j - - ipset destroy - -# Option: actionban -# Notes.: command executed when banning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionban = ipset add timeout -exist - -actionprolong = %(actionban)s - -# Option: actionunban -# Notes.: command executed when unbanning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionunban = ipset del -exist - -[Init] - -# Option: default-timeout -# Notes: specifies default timeout in seconds (handled default ipset timeout only) -# Values: [ NUM ] Default: 600 - -default-timeout = 600 - -ipmset = f2b- -familyopt = - - -[Init?family=inet6] - -ipmset = f2b-6 -familyopt = family inet6 +type = multiport diff --git a/config/action.d/iptables-ipset.conf b/config/action.d/iptables-ipset.conf new file mode 100644 index 00000000..076aabe6 --- /dev/null +++ b/config/action.d/iptables-ipset.conf @@ -0,0 +1,82 @@ +# Fail2Ban configuration file +# +# Authors: Sergey G Brester (sebres), Daniel Black, Alexander Koeppe +# +# This is for ipset protocol 6 (and hopefully later) (ipset v6.14). +# Use ipset -V to see the protocol and version. Version 4 should use +# iptables-ipset-proto4.conf. +# +# This requires the program ipset which is normally in package called ipset. +# +# IPset was a feature introduced in the linux kernel 2.6.39 and 3.0.0 kernels. +# +# If you are running on an older kernel you make need to patch in external +# modules. +# + +[INCLUDES] + +before = iptables.conf + +[Definition] + +# Option: actionstart +# Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false). +# Values: CMD +# +actionstart = ipset create hash:ip timeout + -I %(_ipt_chain_rule)s + +# Option: actionflush +# Notes.: command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action) +# Values: CMD +# +actionflush = ipset flush + +# Option: actionstop +# Notes.: command executed at the stop of jail (or at the end of Fail2Ban) +# Values: CMD +# +actionstop = -D %(_ipt_chain_rule)s + + ipset destroy + +# Option: actionban +# Notes.: command executed when banning an IP. Take care that the +# command is executed with Fail2Ban user rights. +# Tags: See jail.conf(5) man page +# Values: CMD +# +actionban = ipset add timeout -exist + +actionprolong = %(actionban)s + +# Option: actionunban +# Notes.: command executed when unbanning an IP. Take care that the +# command is executed with Fail2Ban user rights. +# Tags: See jail.conf(5) man page +# Values: CMD +# +actionunban = ipset del -exist + +# Several capabilities used internaly: + +rule-jump = -m set --match-set src -j + + +[Init] + +# Option: default-timeout +# Notes: specifies default timeout in seconds (handled default ipset timeout only) +# Values: [ NUM ] Default: 600 + +default-timeout = 600 + +ipmset = f2b- +familyopt = + + +[Init?family=inet6] + +ipmset = f2b-6 +familyopt = family inet6 diff --git a/config/action.d/iptables.conf b/config/action.d/iptables.conf index d31b49aa..d8845173 100644 --- a/config/action.d/iptables.conf +++ b/config/action.d/iptables.conf @@ -12,12 +12,6 @@ # type = oneport -# Option: rule -# Notes.: type of the iptables rule. -# Values: [ iptables | ipset ] Default: iptables -# -rule = iptables - # Option: actionflush # Notes.: command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action) # Values: CMD @@ -67,22 +61,25 @@ actionunban = -D f2b- -s -j # pre-rule = +rule-jump = -j <_ipt_rule_target> + # Several capabilities used internaly: _ipt_chain_rule = /_chain_rule> _ipt_check_rule = -C %(_ipt_chain_rule)s +_ipt_rule_target = f2b- [ipt_oneport] -_chain_rule = -p --dport -j f2b- +_chain_rule = -p --dport [ipt_multiport] -_chain_rule = -p -m multiport --dports -j f2b- +_chain_rule = -p -m multiport --dports [ipt_allports] -_chain_rule = -p -j f2b- +_chain_rule = -p [Init] diff --git a/fail2ban/tests/servertestcase.py b/fail2ban/tests/servertestcase.py index 4ecb1ca5..301d52e5 100644 --- a/fail2ban/tests/servertestcase.py +++ b/fail2ban/tests/servertestcase.py @@ -1596,6 +1596,12 @@ class ServerConfigReaderTests(LogCaptureTestCase): "`ipset flush f2b-j-w-iptables-ipset6`", "`ipset destroy f2b-j-w-iptables-ipset6`", ), + 'ip4-check': ( + r"""`iptables -w -C INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable`""", + ), + 'ip6-check': ( + r"""`ip6tables -w -C INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`""", + ), 'ip4-ban': ( r"`ipset add f2b-j-w-iptables-ipset 192.0.2.1 timeout 600 -exist`", ), @@ -1614,24 +1620,30 @@ class ServerConfigReaderTests(LogCaptureTestCase): 'ip4': (' f2b-j-w-iptables-ipset-ap ',), 'ip6': (' f2b-j-w-iptables-ipset-ap6 ',), 'ip4-start': ( "`ipset create f2b-j-w-iptables-ipset-ap hash:ip timeout 0`", - "`iptables -w -I INPUT -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", + "`iptables -w -I INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", ), 'ip6-start': ( "`ipset create f2b-j-w-iptables-ipset-ap6 hash:ip timeout 0 family inet6`", - "`ip6tables -w -I INPUT -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", + "`ip6tables -w -I INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", ), 'flush': ( "`ipset flush f2b-j-w-iptables-ipset-ap`", "`ipset flush f2b-j-w-iptables-ipset-ap6`", ), 'stop': ( - "`iptables -w -D INPUT -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", + "`iptables -w -D INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", "`ipset flush f2b-j-w-iptables-ipset-ap`", "`ipset destroy f2b-j-w-iptables-ipset-ap`", - "`ip6tables -w -D INPUT -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", + "`ip6tables -w -D INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", "`ipset flush f2b-j-w-iptables-ipset-ap6`", "`ipset destroy f2b-j-w-iptables-ipset-ap6`", ), + 'ip4-check': ( + r"""`iptables -w -C INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`""", + ), + 'ip6-check': ( + r"""`ip6tables -w -C INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`""", + ), 'ip4-ban': ( r"`ipset add f2b-j-w-iptables-ipset-ap 192.0.2.1 timeout 600 -exist`", ), From 2fd6b478a969bbe602d470ec6526caeaf0c159d3 Mon Sep 17 00:00:00 2001 From: "Brian J. Murrell" Date: Sun, 16 Feb 2020 10:38:36 -0500 Subject: [PATCH 018/240] FreeIPA renames named to named-pkcs11 FreeIPA renames the BIND9 named daemon to named-pkcs11, so extend the REGEX match to look for either variant. Signed-off-by: Brian J. Murrell --- config/filter.d/named-refused.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/filter.d/named-refused.conf b/config/filter.d/named-refused.conf index 0d5a6275..51505288 100644 --- a/config/filter.d/named-refused.conf +++ b/config/filter.d/named-refused.conf @@ -22,7 +22,7 @@ [Definition] # Daemon name -_daemon=named +_daemon=named(-pkcs11)? # Shortcuts for easier comprehension of the failregex From ede200970859a6f1491aabdeb68c7d24a6052e36 Mon Sep 17 00:00:00 2001 From: Jordi Sanfeliu Date: Fri, 3 Apr 2020 12:52:19 +0200 Subject: [PATCH 019/240] added new jail (and filter) Monitorix --- config/filter.d/monitorix.conf | 27 +++++++++++++++++++++++++++ config/jail.conf | 5 +++++ 2 files changed, 32 insertions(+) create mode 100644 config/filter.d/monitorix.conf diff --git a/config/filter.d/monitorix.conf b/config/filter.d/monitorix.conf new file mode 100644 index 00000000..3979ed43 --- /dev/null +++ b/config/filter.d/monitorix.conf @@ -0,0 +1,27 @@ +# Fail2Ban filter for Monitorix (HTTP built-in server) +# + +[INCLUDES] + +before = common.conf + +[Definition] + +_daemon = monitorix-httpd + +# Option: failregex +# Notes.: regex to match the password failures messages in the logfile. The +# host must be matched by a group named "host". The tag "" can +# be used for standard IP/hostname matching and is only an alias for +# (?:::f{4,6}:)?(?P\S+) +# Values: TEXT +# +failregex = NOTEXIST - \[\] .* + AUTHERR - \[\] .* + NOTALLOWED - \[\] .* + +# Option: ignoreregex +# Notes.: regex to ignore. If this regex matches, the line is ignored. +# Values: TEXT +# +ignoreregex = diff --git a/config/jail.conf b/config/jail.conf index c7177f13..02ce55f1 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -951,3 +951,8 @@ logpath = %(apache_error_log)s # see `filter.d/traefik-auth.conf` for details and service example. port = http,https logpath = /var/log/traefik/access.log + + +[monitorix] +port = 8080 +logpath = /var/log/monitorix-httpd From 412120ac3c3f1c9a3fa8c7676059ab2be0f87c6a Mon Sep 17 00:00:00 2001 From: aresdr Date: Sat, 30 May 2020 15:25:31 -0700 Subject: [PATCH 020/240] Update drupal-auth.conf Small fix for Drupal 8. D8 uses "Login attempt failed from" while D7 uses "Login attempt failed for". The referer part is a must currently, but some requests did not have one and are not failing. --- config/filter.d/drupal-auth.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/filter.d/drupal-auth.conf b/config/filter.d/drupal-auth.conf index b60abe3e..2d4cbe9f 100644 --- a/config/filter.d/drupal-auth.conf +++ b/config/filter.d/drupal-auth.conf @@ -14,7 +14,7 @@ before = common.conf [Definition] -failregex = ^%(__prefix_line)s(https?:\/\/)([\da-z\.-]+)\.([a-z\.]{2,6})(\/[\w\.-]+)*\|\d{10}\|user\|\|.+\|.+\|\d\|.*\|Login attempt failed for .+\.$ +failregex = ^%(__prefix_line)s(https?:\/\/)([\da-z\.-]+)\.([a-z\.]{2,6})(\/[\w\.-]+)*\|\d{10}\|user\|\|.+\|.*\|\d\|.*\|Login attempt failed (?:for|from) .+\.$ ignoreregex = From 3c83c19070b8e79e2d4392071d02b4ea99643d87 Mon Sep 17 00:00:00 2001 From: Jan Przybylak Date: Sat, 6 Jun 2020 19:51:46 +0200 Subject: [PATCH 021/240] Added filter nginx-bad-request --- config/filter.d/nginx-bad-request.conf | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 config/filter.d/nginx-bad-request.conf diff --git a/config/filter.d/nginx-bad-request.conf b/config/filter.d/nginx-bad-request.conf new file mode 100644 index 00000000..ea26d56a --- /dev/null +++ b/config/filter.d/nginx-bad-request.conf @@ -0,0 +1,13 @@ +# Fail2Ban filter to match bad requests to nginx +# + +[Definition] + +# The request often doesn't contain a method, only some encoded garbage +failregex = ^ \- \S+ \[\] \".+\" 400 .+$ + +datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]%%f)?(?:\s*%%z)? + ^[^\[]*\[({DATE}) + {^LN-BEG} + +# Author: Jan Przybylak From d7ef5d166db58ec402408fa97c0b291906c3d8c9 Mon Sep 17 00:00:00 2001 From: Jan Przybylak Date: Thu, 11 Jun 2020 16:44:48 +0200 Subject: [PATCH 022/240] Removed vulnerable catchall & anchor --- config/filter.d/nginx-bad-request.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/filter.d/nginx-bad-request.conf b/config/filter.d/nginx-bad-request.conf index ea26d56a..03721f86 100644 --- a/config/filter.d/nginx-bad-request.conf +++ b/config/filter.d/nginx-bad-request.conf @@ -4,7 +4,7 @@ [Definition] # The request often doesn't contain a method, only some encoded garbage -failregex = ^ \- \S+ \[\] \".+\" 400 .+$ +failregex = ^ \- \S+ \[\] \"[^\"]+\" 400 datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]%%f)?(?:\s*%%z)? ^[^\[]*\[({DATE}) From a5ab4406d8b7d34da8e653d8d5debd2c0735b484 Mon Sep 17 00:00:00 2001 From: Jan Przybylak Date: Sun, 21 Jun 2020 18:24:09 +0200 Subject: [PATCH 023/240] Removed unnecessary escape sequence This commit also contains changes to match requests that are 100% empty (by using "*" instead of "+" in the regex) --- config/filter.d/nginx-bad-request.conf | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/config/filter.d/nginx-bad-request.conf b/config/filter.d/nginx-bad-request.conf index 03721f86..2b8f5ab6 100644 --- a/config/filter.d/nginx-bad-request.conf +++ b/config/filter.d/nginx-bad-request.conf @@ -4,7 +4,8 @@ [Definition] # The request often doesn't contain a method, only some encoded garbage -failregex = ^ \- \S+ \[\] \"[^\"]+\" 400 +# This will also match requests that are entirely empty +failregex = ^ - \S+ \[\] "[^"]*" 400 datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]%%f)?(?:\s*%%z)? ^[^\[]*\[({DATE}) From 56fefe9240a6cc06650b91743339451caf6aa2f1 Mon Sep 17 00:00:00 2001 From: Jan Przybylak Date: Sun, 21 Jun 2020 18:25:27 +0200 Subject: [PATCH 024/240] Added test file "nginx-bad-request" I tested with `./fail2ban-testcases testSampleRegex`, which did not return any errors. --- fail2ban/tests/files/logs/nginx-bad-request | 23 +++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 fail2ban/tests/files/logs/nginx-bad-request diff --git a/fail2ban/tests/files/logs/nginx-bad-request b/fail2ban/tests/files/logs/nginx-bad-request new file mode 100644 index 00000000..a9ff6497 --- /dev/null +++ b/fail2ban/tests/files/logs/nginx-bad-request @@ -0,0 +1,23 @@ +# failJSON: { "time": "2015-01-20T19:53:28", "match": true , "host": "12.34.56.78" } +12.34.56.78 - - [20/Jan/2015:19:53:28 +0100] "" 400 47 "-" "-" "-" + +# failJSON: { "time": "2015-01-20T19:53:28", "match": true , "host": "12.34.56.78" } +12.34.56.78 - root [20/Jan/2015:19:53:28 +0100] "" 400 47 "-" "-" "-" + +# failJSON: { "time": "2015-01-20T19:53:28", "match": true , "host": "12.34.56.78" } +12.34.56.78 - - [20/Jan/2015:19:53:28 +0100] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 47 "-" "-" "-" + +# failJSON: { "time": "2015-01-20T19:53:28", "match": true , "host": "12.34.56.78" } +12.34.56.78 - - [20/Jan/2015:19:53:28 +0100] "GET //admin/pma/scripts/setup.php HTTP/1.1" 400 47 "-" "-" "-" + +# failJSON: { "time": "2015-01-20T19:54:28", "match": true , "host": "12.34.56.78" } +12.34.56.78 - - [20/Jan/2015:19:54:28 +0100] "HELP" 400 47 "-" "-" "-" + +# failJSON: { "time": "2015-01-20T19:55:28", "match": true , "host": "12.34.56.78" } +12.34.56.78 - - [20/Jan/2015:19:55:28 +0100] "batman" 400 47 "-" "-" "-" + +# failJSON: { "time": "2015-01-20T01:17:07", "match": true , "host": "7.8.9.10" } +7.8.9.10 - root [20/Jan/2015:01:17:07 +0100] "CONNECT 123.123.123.123 HTTP/1.1" 400 162 "-" "-" "-" + +# failJSON: { "time": "2014-12-12T22:59:02", "match": true , "host": "2.5.2.5" } +2.5.2.5 - tomcat [12/Dec/2014:22:59:02 +0100] "GET /cgi-bin/tools/tools.pl HTTP/1.1" 400 162 "-" "-" "-" \ No newline at end of file From fd25c4cbb813db2562e5a2e54d4510eac291a897 Mon Sep 17 00:00:00 2001 From: TorontoMedia <10255876+TorontoMedia@users.noreply.github.com> Date: Sun, 28 Jun 2020 12:58:41 -0400 Subject: [PATCH 025/240] Remove duplicate method and rename invalid parameter --- fail2ban/server/failmanager.py | 4 ---- fail2ban/server/observer.py | 2 +- 2 files changed, 1 insertion(+), 5 deletions(-) diff --git a/fail2ban/server/failmanager.py b/fail2ban/server/failmanager.py index bc09c0e2..97b4c7a4 100644 --- a/fail2ban/server/failmanager.py +++ b/fail2ban/server/failmanager.py @@ -59,10 +59,6 @@ class FailManager: with self.__lock: return len(self.__failList), sum([f.getRetry() for f in self.__failList.values()]) - def getFailTotal(self): - with self.__lock: - return self.__failTotal - def setMaxRetry(self, value): self.__maxRetry = value diff --git a/fail2ban/server/observer.py b/fail2ban/server/observer.py index c19549ba..f5ba20d9 100644 --- a/fail2ban/server/observer.py +++ b/fail2ban/server/observer.py @@ -87,7 +87,7 @@ class ObserverThread(JailThread): except KeyError: raise KeyError("Invalid event index : %s" % i) - def __delitem__(self, name): + def __delitem__(self, i): try: del self._queue[i] except KeyError: From 2216fd8da4e95564bc4cd0047ffae08d24d3d17d Mon Sep 17 00:00:00 2001 From: Chris Caron Date: Tue, 4 Aug 2020 19:04:05 -0400 Subject: [PATCH 026/240] Add Apprise Support (50+ Notifications) --- MANIFEST | 1 + config/action.d/apprise.conf | 47 ++++++++++++++++++++++++++++++++++++ config/jail.conf | 9 +++++++ 3 files changed, 57 insertions(+) create mode 100644 config/action.d/apprise.conf diff --git a/MANIFEST b/MANIFEST index ed441bac..d02b8bd0 100644 --- a/MANIFEST +++ b/MANIFEST @@ -3,6 +3,7 @@ bin/fail2ban-regex bin/fail2ban-server bin/fail2ban-testcases ChangeLog +config/action.d/apprise.conf config/action.d/abuseipdb.conf config/action.d/apf.conf config/action.d/badips.conf diff --git a/config/action.d/apprise.conf b/config/action.d/apprise.conf new file mode 100644 index 00000000..ac54d8fd --- /dev/null +++ b/config/action.d/apprise.conf @@ -0,0 +1,47 @@ +# Fail2Ban configuration file +# +# Author: Chris Caron +# +# + +[Definition] + +# Option: actionstart +# Notes.: command executed once at the start of Fail2Ban. +# Values: CMD +# +actionstart = printf %%b "The jail as been started successfully." |apprise -t "[Fail2Ban] : started on `uname -n`" -c /etc/fail2ban/apprise.conf + +# Option: actionstop +# Notes.: command executed once at the end of Fail2Ban +# Values: CMD +# +actionstop = printf %%b "The jail has been stopped." |apprise -t "[Fail2Ban] : stopped on `uname -n`" -c /etc/fail2ban/apprise.conf + +# Option: actioncheck +# Notes.: command executed once before each actionban command +# Values: CMD +# +actioncheck = + +# Option: actionban +# Notes.: command executed when banning an IP. Take care that the +# command is executed with Fail2Ban user rights. +# Tags: See jail.conf(5) man page +# Values: CMD +# +actionban = printf %%b "The IP has just been banned by Fail2Ban after attempts against " | apprise -n "warning" -t "[Fail2Ban] : banned $ + +# Option: actionunban +# Notes.: command executed when unbanning an IP. Take care that the +# command is executed with Fail2Ban user rights. +# Tags: See jail.conf(5) man page +# Values: CMD +# +actionunban = + +[Init] + +# Define location of the default apprise configuration file to use +# +config = /etc/fail2ban/apprise.conf diff --git a/config/jail.conf b/config/jail.conf index 6e8a6a2f..dbca2e8f 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -227,6 +227,15 @@ action_mwl = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(proto action_xarf = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"] xarf-login-attack[service=%(__name__)s, sender="%(sender)s", logpath="%(logpath)s", port="%(port)s"] +# ban & send a notification to one or more of the 50+ services supported by Apprise. +# See https://github.com/caronc/apprise/wiki for details on what is supported. +# +# You may optionally over-ride the default configuration line (containing the Apprise URLs) +# by using 'apprise[name=%(__name__)s, config="/alternate/path/to/apprise.cfg"]' otherwise +# /etc/fail2ban/apprise.conf is sourced for your supported notification configuration. +action_apprise = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"] + apprise[name=%(__name__)s] + # ban IP on CloudFlare & send an e-mail with whois report and relevant log lines # to the destemail. action_cf_mwl = cloudflare[cfuser="%(cfemail)s", cftoken="%(cfapikey)s"] From 70c601e9e53a44a6a2e950ee8e64705d14f09bc7 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Wed, 2 Sep 2020 20:47:05 +0200 Subject: [PATCH 027/240] involve config parameter (replaces hard-coded path); fixed typo in actionban (looks like copy&paste from trimmed tty) --- config/action.d/apprise.conf | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/config/action.d/apprise.conf b/config/action.d/apprise.conf index ac54d8fd..37c42ea2 100644 --- a/config/action.d/apprise.conf +++ b/config/action.d/apprise.conf @@ -10,13 +10,13 @@ # Notes.: command executed once at the start of Fail2Ban. # Values: CMD # -actionstart = printf %%b "The jail as been started successfully." |apprise -t "[Fail2Ban] : started on `uname -n`" -c /etc/fail2ban/apprise.conf +actionstart = printf %%b "The jail as been started successfully." | -t "[Fail2Ban] : started on `uname -n`" # Option: actionstop # Notes.: command executed once at the end of Fail2Ban # Values: CMD # -actionstop = printf %%b "The jail has been stopped." |apprise -t "[Fail2Ban] : stopped on `uname -n`" -c /etc/fail2ban/apprise.conf +actionstop = printf %%b "The jail has been stopped." | -t "[Fail2Ban] : stopped on `uname -n`" # Option: actioncheck # Notes.: command executed once before each actionban command @@ -30,7 +30,7 @@ actioncheck = # Tags: See jail.conf(5) man page # Values: CMD # -actionban = printf %%b "The IP has just been banned by Fail2Ban after attempts against " | apprise -n "warning" -t "[Fail2Ban] : banned $ +actionban = printf %%b "The IP has just been banned by Fail2Ban after attempts against " | -n "warning" -t "[Fail2Ban] : banned from `uname -n`" # Option: actionunban # Notes.: command executed when unbanning an IP. Take care that the @@ -45,3 +45,5 @@ actionunban = # Define location of the default apprise configuration file to use # config = /etc/fail2ban/apprise.conf +# +apprise = apprise -c "" From 9d77fb2b4c5f4471e1abef0974c70c1ff622ff75 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Tue, 3 Nov 2020 13:15:42 +0100 Subject: [PATCH 028/240] 1st try of GH actions flow (CI only, no coverage atm) --- .github/workflows/main.yml | 64 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 .github/workflows/main.yml diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml new file mode 100644 index 00000000..154a75ae --- /dev/null +++ b/.github/workflows/main.yml @@ -0,0 +1,64 @@ +name: CI + +# Controls when the action will run. Triggers the workflow on push or pull request +# events but only for the master branch +on: + push: + paths-ignore: + - 'doc/**' + - 'files/**' + - 'man/**' + pull_request: + paths-ignore: + - 'doc/**' + - 'files/**' + - 'man/**' + +# A workflow run is made up of one or more jobs that can run sequentially or in parallel +jobs: + # This workflow contains a single job called "build" + build: + # The type of runner that the job will run on + runs-on: ubuntu-20.04 + strategy: + matrix: + python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, pypy2, pypy3] + fail-fast: false + # Steps represent a sequence of tasks that will be executed as part of the job + steps: + # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it + - uses: actions/checkout@v2 + + - name: Set up Python + uses: actions/setup-python@v2 + with: + python-version: ${{ matrix.python-version }} + + - name: Python version + run: | + F2B_PY=$(python -c "import sys; print(sys.version)") + echo "Python: ${{ matrix.python-version }} -- $F2B_PY" + F2B_PY=${F2B_PY:0:1} + echo "Set F2B_PY=$F2B_PY" + echo "F2B_PY=$F2B_PY" >> $GITHUB_ENV + + - name: Install dependencies + run: | + if [[ "$F2B_PY" = 3 ]] && ! command -v 2to3x -v 2to3 > /dev/null; then + python -m pip install --upgrade pip + pip install 2to3 + fi + + - name: Before scripts + run: | + cd "$GITHUB_WORKSPACE" + # Manually execute 2to3 for now + if [[ "$F2B_PY" = 3 ]]; then echo "2to3 ..." && ./fail2ban-2to3; fi + # (debug) output current preferred encoding: + python -c 'import locale, sys; from fail2ban.helpers import PREFER_ENC; print(PREFER_ENC, locale.getpreferredencoding(), (sys.stdout and sys.stdout.encoding))' + + - name: Test suite + run: if [[ "$F2B_PY" = 2 ]]; then python setup.py test; else python bin/fail2ban-testcases --verbosity=2; fi + + #- name: Test initd scripts + # run: shellcheck -s bash -e SC1090,SC1091 files/debian-initd From 753fff9c15387d62e32bc9794cb7a4714b9dd359 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Mon, 23 Nov 2020 18:38:41 +0100 Subject: [PATCH 029/240] amend to #2750, add jail for new filter nginx-bad-request --- config/jail.conf | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/config/jail.conf b/config/jail.conf index e6961a18..28f259a0 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -392,7 +392,10 @@ logpath = %(nginx_error_log)s port = http,https logpath = %(nginx_error_log)s -maxretry = 2 + +[nginx-bad-request] +port = http,https +logpath = %(nginx_access_log)s # Ban attackers that try to use PHP's URL-fopen() functionality From 0f27a8add82dd75c9cb984505732b56c1f625313 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 24 Nov 2020 19:28:46 +0100 Subject: [PATCH 030/240] datedetector, strptime: token `%Z` must recognize zone abbreviation `Z` (GMT/UTC) also, similar to `%z`; more test cases added. --- fail2ban/server/strptime.py | 8 ++------ fail2ban/tests/datedetectortestcase.py | 3 +++ 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/fail2ban/server/strptime.py b/fail2ban/server/strptime.py index 1464a96d..5fcfb959 100644 --- a/fail2ban/server/strptime.py +++ b/fail2ban/server/strptime.py @@ -234,16 +234,12 @@ def reGroupDictStrptime(found_dict, msec=False, default_tz=None): week_of_year = int(val) # U starts week on Sunday, W - on Monday week_of_year_start = 6 if key == 'U' else 0 - elif key == 'z': + elif key in ('z', 'Z'): z = val if z in ("Z", "UTC", "GMT"): tzoffset = 0 - else: + elif key == 'z': tzoffset = zone2offset(z, 0); # currently offset-based only - elif key == 'Z': - z = val - if z in ("UTC", "GMT"): - tzoffset = 0 # Fail2Ban will assume it's this year assume_year = False diff --git a/fail2ban/tests/datedetectortestcase.py b/fail2ban/tests/datedetectortestcase.py index d6370fc4..ee126f72 100644 --- a/fail2ban/tests/datedetectortestcase.py +++ b/fail2ban/tests/datedetectortestcase.py @@ -516,6 +516,9 @@ class CustomDateFormatsTest(unittest.TestCase): (1072746123.0 - 3600, "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %Z)?", "[2003-12-30 01:02:03] server ..."), (1072746123.0, "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %z)?", "[2003-12-30 01:02:03 UTC] server ..."), (1072746123.0, "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %Z)?", "[2003-12-30 01:02:03 UTC] server ..."), + (1072746123.0, "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %z)?", "[2003-12-30 01:02:03 Z] server ..."), + (1072746123.0, "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %z)?", "[2003-12-30 01:02:03 +0000] server ..."), + (1072746123.0, "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %Z)?", "[2003-12-30 01:02:03 Z] server ..."), ): logSys.debug('== test: %r', (matched, dp, line)) if dp is None: From b8e8a87ee93985d36c8f53bf5089650b34c874e1 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 24 Nov 2020 20:21:49 +0100 Subject: [PATCH 031/240] small amend to 0f27a8add82dd75c9cb984505732b56c1f625313 - datedetector, strptime: token `%Z` recognizes all known zone abbreviation besides Z, GMT, UTC correctly, if it is matching (`%z` remains unchanged for backwards-compatibility, see comment in code); test cases fixed (PDT zone will be found now). --- fail2ban/server/strptime.py | 2 +- fail2ban/tests/files/logs/monit | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/fail2ban/server/strptime.py b/fail2ban/server/strptime.py index 5fcfb959..7f11402a 100644 --- a/fail2ban/server/strptime.py +++ b/fail2ban/server/strptime.py @@ -238,7 +238,7 @@ def reGroupDictStrptime(found_dict, msec=False, default_tz=None): z = val if z in ("Z", "UTC", "GMT"): tzoffset = 0 - elif key == 'z': + else: tzoffset = zone2offset(z, 0); # currently offset-based only # Fail2Ban will assume it's this year diff --git a/fail2ban/tests/files/logs/monit b/fail2ban/tests/files/logs/monit index 8dbddaf6..36f1c1e4 100644 --- a/fail2ban/tests/files/logs/monit +++ b/fail2ban/tests/files/logs/monit @@ -1,7 +1,7 @@ # Previous version -- -# failJSON: { "time": "2005-04-16T21:05:29", "match": true , "host": "69.93.127.111" } +# failJSON: { "time": "2005-04-17T06:05:29", "match": true , "host": "69.93.127.111" } [PDT Apr 16 21:05:29] error : Warning: Client '69.93.127.111' supplied unknown user 'foo' accessing monit httpd -# failJSON: { "time": "2005-04-16T20:59:33", "match": true , "host": "97.113.189.111" } +# failJSON: { "time": "2005-04-17T05:59:33", "match": true , "host": "97.113.189.111" } [PDT Apr 16 20:59:33] error : Warning: Client '97.113.189.111' supplied wrong password for user 'admin' accessing monit httpd # Current version -- corresponding "https://bitbucket.org/tildeslash/monit/src/6905335aa903d425cae732cab766bd88ea5f2d1d/src/http/processor.c?at=master&fileviewer=file-view-default#processor.c-728" From 27c40a77a31c4c72b9076f43962debcf8d7d6359 Mon Sep 17 00:00:00 2001 From: stepodev Date: Wed, 25 Nov 2020 20:57:06 +0100 Subject: [PATCH 032/240] add nginx-tls-downgrade --- ChangeLog | 1 + config/filter.d/nginx-tls-fallback.conf | 16 ++++++++++++++++ config/jail.conf | 3 +++ fail2ban/tests/files/logs/nginx-tls-fallback | 9 +++++++++ 4 files changed, 29 insertions(+) create mode 100644 config/filter.d/nginx-tls-fallback.conf create mode 100644 fail2ban/tests/files/logs/nginx-tls-fallback diff --git a/ChangeLog b/ChangeLog index 5f1fc313..bd4a369d 100644 --- a/ChangeLog +++ b/ChangeLog @@ -21,6 +21,7 @@ ver. 1.0.1-dev-1 (20??/??/??) - development nightly edition * `actioncheck` behavior is changed now (gh-488), so invariant check as well as restore or repair of sane environment (in case of recognized unsane state) would only occur on action errors (e. g. if ban or unban operations are exiting with other code as 0) +* `filter.d/nginx-tls-fallback` -- filter added for tls downgrade probes ver. 0.11.2 (2020/11/23) - heal-the-world-with-security-tools diff --git a/config/filter.d/nginx-tls-fallback.conf b/config/filter.d/nginx-tls-fallback.conf new file mode 100644 index 00000000..aaa84fb9 --- /dev/null +++ b/config/filter.d/nginx-tls-fallback.conf @@ -0,0 +1,16 @@ +# fail2ban filter configuration for nginx +# Ban people checking for TLS_FALLBACK_SCSV repeatedly +# https://stackoverflow.com/questions/28010492/nginx-critical-error-with-ssl-handshaking/28010608#28010608 + +[Definition] + + +failregex = ^ \[crit\] \d+#\d+: \*\d+ SSL_do_handshake\(\) failed.*?ssl3_get_record.*?too.*?, client: , server: \S+$ + +ignoreregex = + +datepattern = {^LN-BEG} + +# Author: Stephan Orlowsky +# maybe not restrictive enough, will also match: +#"[crit] 76952#76952: *5062354 SSL_do_handshake() failed ssl3_get_record too, client: 0.0.0.0, server: thisshouldntmatch" diff --git a/config/jail.conf b/config/jail.conf index 28f259a0..a7818a06 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -397,6 +397,9 @@ logpath = %(nginx_error_log)s port = http,https logpath = %(nginx_access_log)s +[nginx-tls-fallback] +port = http,https +logpath = %(nginx_access_log)s # Ban attackers that try to use PHP's URL-fopen() functionality # through GET/POST variables. - Experimental, with more than a year diff --git a/fail2ban/tests/files/logs/nginx-tls-fallback b/fail2ban/tests/files/logs/nginx-tls-fallback new file mode 100644 index 00000000..bd51fbbf --- /dev/null +++ b/fail2ban/tests/files/logs/nginx-tls-fallback @@ -0,0 +1,9 @@ + +# failJSON: { "time": "2020-11-25T14:42:16", "match": true , "host": "142.93.180.14" } +2020/11/25 14:42:16 [crit] 76952#76952: *2454307 SSL_do_handshake() failed (SSL: error:1408F0C6:SSL routines:ssl3_get_record:packet length too long) while SSL handshaking, client: 142.93.180.14, server: 0.0.0.0:443 +# failJSON: { "time": "2020-11-25T15:47:47", "match": true , "host": "80.191.166.166" } +2020/11/25 15:47:47 [crit] 76952#76952: *5062354 SSL_do_handshake() failed (SSL: error:1408F0A0:SSL routines:ssl3_get_record:length too short) while SSL handshaking, client: 80.191.166.166, server: 0.0.0.0:443 +# failJSON: { "time": "2020-11-25T16:48:08", "match": true , "host": "5.126.32.148" } +2020/11/25 16:48:08 [crit] 76952#76952: *7976400 SSL_do_handshake() failed (SSL: error:1408F096:SSL routines:ssl3_get_record:encrypted length too long) while SSL handshaking, client: 5.126.32.148, server: 0.0.0.0:443 +# failJSON: { "time": "2020-11-25T16:02:45", "match": false } +2020/11/25 16:02:45 [error] 76952#76952: *5645766 connect() failed (111: Connection refused) while connecting to upstream, client: 5.126.32.148, server: www.google.de, request: "GET /admin/config HTTP/2.0", upstream: "http://127.0.0.1:3000/admin/config", host: "www.google.de" From c0256724a7b091cba7c774b1774d6e1dc6ee604a Mon Sep 17 00:00:00 2001 From: stepodev Date: Wed, 25 Nov 2020 21:30:21 +0100 Subject: [PATCH 033/240] fix monitoring wrong error log. was access log, should be error.log --- config/jail.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/jail.conf b/config/jail.conf index a7818a06..f7eeeeb8 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -399,7 +399,7 @@ logpath = %(nginx_access_log)s [nginx-tls-fallback] port = http,https -logpath = %(nginx_access_log)s +logpath = %(nginx_error_log)s # Ban attackers that try to use PHP's URL-fopen() functionality # through GET/POST variables. - Experimental, with more than a year From d959f6d19926ff2b75322acb008dc692b6a29aa5 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Thu, 26 Nov 2020 12:25:32 +0100 Subject: [PATCH 034/240] Update nginx-tls-fallback.conf more precise and conclusive regex without catch-all's --- config/filter.d/nginx-tls-fallback.conf | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/config/filter.d/nginx-tls-fallback.conf b/config/filter.d/nginx-tls-fallback.conf index aaa84fb9..b9eeac04 100644 --- a/config/filter.d/nginx-tls-fallback.conf +++ b/config/filter.d/nginx-tls-fallback.conf @@ -5,12 +5,10 @@ [Definition] -failregex = ^ \[crit\] \d+#\d+: \*\d+ SSL_do_handshake\(\) failed.*?ssl3_get_record.*?too.*?, client: , server: \S+$ +failregex = ^\s*\[crit\] \d+#\d+: \*\d+ SSL_do_handshake\(\) failed \(SSL: error:\S+(?: \S+){1,3} too (?:long|short)\)[^,]*, client: ignoreregex = datepattern = {^LN-BEG} # Author: Stephan Orlowsky -# maybe not restrictive enough, will also match: -#"[crit] 76952#76952: *5062354 SSL_do_handshake() failed ssl3_get_record too, client: 0.0.0.0, server: thisshouldntmatch" From d0ba27cf46411be9104ad5f90e8d39551419fa2a Mon Sep 17 00:00:00 2001 From: stepodev Date: Mon, 30 Nov 2020 12:14:49 +0100 Subject: [PATCH 035/240] move nginx-tls-fallback rules to nginx-http-auth --- ChangeLog | 2 -- config/filter.d/nginx-http-auth.conf | 18 +++++++++++++++++- config/filter.d/nginx-tls-fallback.conf | 14 -------------- config/jail.conf | 4 ---- fail2ban/tests/files/logs/nginx-http-auth | 18 ++++++++++++++++++ fail2ban/tests/files/logs/nginx-tls-fallback | 9 --------- 6 files changed, 35 insertions(+), 30 deletions(-) delete mode 100644 config/filter.d/nginx-tls-fallback.conf delete mode 100644 fail2ban/tests/files/logs/nginx-tls-fallback diff --git a/ChangeLog b/ChangeLog index bd4a369d..f9fb8a33 100644 --- a/ChangeLog +++ b/ChangeLog @@ -21,8 +21,6 @@ ver. 1.0.1-dev-1 (20??/??/??) - development nightly edition * `actioncheck` behavior is changed now (gh-488), so invariant check as well as restore or repair of sane environment (in case of recognized unsane state) would only occur on action errors (e. g. if ban or unban operations are exiting with other code as 0) -* `filter.d/nginx-tls-fallback` -- filter added for tls downgrade probes - ver. 0.11.2 (2020/11/23) - heal-the-world-with-security-tools ----------- diff --git a/config/filter.d/nginx-http-auth.conf b/config/filter.d/nginx-http-auth.conf index 93341cd2..d5655fa8 100644 --- a/config/filter.d/nginx-http-auth.conf +++ b/config/filter.d/nginx-http-auth.conf @@ -3,15 +3,31 @@ [Definition] +mdre-auth = ^ \[error\] \d+#\d+: \*\d+ user "(?:[^"]+|.*?)":? (?:password mismatch|was not found in "[^\"]*"), client: , server: \S*, request: "\S+ \S+ HTTP/\d+\.\d+", host: "\S+"(?:, referrer: "\S+")?\s*$ +mdre-fallback = ^\s*\[crit\] \d+#\d+: \*\d+ SSL_do_handshake\(\) failed \(SSL: error:\S+(?: \S+){1,3} too (?:long|short)\)[^,]*, client: -failregex = ^ \[error\] \d+#\d+: \*\d+ user "(?:[^"]+|.*?)":? (?:password mismatch|was not found in "[^\"]*"), client: , server: \S*, request: "\S+ \S+ HTTP/\d+\.\d+", host: "\S+"(?:, referrer: "\S+")?\s*$ + +mdre-normal = %(mdre-auth)s +mdre-aggressive = %(mdre-auth)s + %(mdre-fallback)s + +failregex = > ignoreregex = datepattern = {^LN-BEG} +mode = normal + # DEV NOTES: +# mdre-auth: # Based on samples in https://github.com/fail2ban/fail2ban/pull/43/files # Extensive search of all nginx auth failures not done yet. # # Author: Daniel Black + +# mdre-fallback: +# Ban people checking for TLS_FALLBACK_SCSV repeatedly +# https://stackoverflow.com/questions/28010492/nginx-critical-error-with-ssl-handshaking/28010608#28010608 +# Author: Stephan Orlowsky + diff --git a/config/filter.d/nginx-tls-fallback.conf b/config/filter.d/nginx-tls-fallback.conf deleted file mode 100644 index b9eeac04..00000000 --- a/config/filter.d/nginx-tls-fallback.conf +++ /dev/null @@ -1,14 +0,0 @@ -# fail2ban filter configuration for nginx -# Ban people checking for TLS_FALLBACK_SCSV repeatedly -# https://stackoverflow.com/questions/28010492/nginx-critical-error-with-ssl-handshaking/28010608#28010608 - -[Definition] - - -failregex = ^\s*\[crit\] \d+#\d+: \*\d+ SSL_do_handshake\(\) failed \(SSL: error:\S+(?: \S+){1,3} too (?:long|short)\)[^,]*, client: - -ignoreregex = - -datepattern = {^LN-BEG} - -# Author: Stephan Orlowsky diff --git a/config/jail.conf b/config/jail.conf index f7eeeeb8..3d971892 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -397,10 +397,6 @@ logpath = %(nginx_error_log)s port = http,https logpath = %(nginx_access_log)s -[nginx-tls-fallback] -port = http,https -logpath = %(nginx_error_log)s - # Ban attackers that try to use PHP's URL-fopen() functionality # through GET/POST variables. - Experimental, with more than a year # of usage in production environments. diff --git a/fail2ban/tests/files/logs/nginx-http-auth b/fail2ban/tests/files/logs/nginx-http-auth index c9c96807..fb24b242 100644 --- a/fail2ban/tests/files/logs/nginx-http-auth +++ b/fail2ban/tests/files/logs/nginx-http-auth @@ -1,3 +1,4 @@ +# filterOptions: [{"mode": "normal"}, {"mode": "auth"}] # failJSON: { "time": "2012-04-09T11:53:29", "match": true , "host": "192.0.43.10" } 2012/04/09 11:53:29 [error] 2865#0: *66647 user "xyz" was not found in "/var/www/.htpasswd", client: 192.0.43.10, server: www.myhost.com, request: "GET / HTTP/1.1", host: "www.myhost.com" @@ -11,3 +12,20 @@ 2014/04/03 22:20:38 [error] 30708#0: *3 user "scriben dio": password mismatch, client: 192.0.2.1, server: , request: "GET / HTTP/1.1", host: "localhost:8443" # failJSON: { "time": "2014-04-03T22:20:40", "match": true, "host": "192.0.2.2", "desc": "trying injection on user name"} 2014/04/03 22:20:40 [error] 30708#0: *3 user "test": password mismatch, client: 127.0.0.1, server: test, request: "GET / HTTP/1.1", host: "localhost:8443"": was not found in "/etc/nginx/.htpasswd", client: 192.0.2.2, server: , request: "GET / HTTP/1.1", host: "localhost:8443" + +# filterOptions: [{"mode": "fallback"}] + +# failJSON: { "time": "2020-11-25T14:42:16", "match": true , "host": "142.93.180.14" } +2020/11/25 14:42:16 [crit] 76952#76952: *2454307 SSL_do_handshake() failed (SSL: error:1408F0C6:SSL routines:ssl3_get_record:packet length too long) while SSL handshaking, client: 142.93.180.14, server: 0.0.0.0:443 +# failJSON: { "time": "2020-11-25T15:47:47", "match": true , "host": "80.191.166.166" } +2020/11/25 15:47:47 [crit] 76952#76952: *5062354 SSL_do_handshake() failed (SSL: error:1408F0A0:SSL routines:ssl3_get_record:length too short) while SSL handshaking, client: 80.191.166.166, server: 0.0.0.0:443 +# failJSON: { "time": "2020-11-25T16:48:08", "match": true , "host": "5.126.32.148" } +2020/11/25 16:48:08 [crit] 76952#76952: *7976400 SSL_do_handshake() failed (SSL: error:1408F096:SSL routines:ssl3_get_record:encrypted length too long) while SSL handshaking, client: 5.126.32.148, server: 0.0.0.0:443 +# failJSON: { "time": "2020-11-25T16:02:45", "match": false } +2020/11/25 16:02:45 [error] 76952#76952: *5645766 connect() failed (111: Connection refused) while connecting to upstream, client: 5.126.32.148, server: www.google.de, request: "GET /admin/config HTTP/2.0", upstream: "http://127.0.0.1:3000/admin/config", host: "www.google.de" + +# filterOptions: [{"mode": "aggressive"}] +# failJSON: { "time": "2020-11-25T14:42:16", "match": true , "host": "142.93.180.14" } +2020/11/25 14:42:16 [crit] 76952#76952: *2454307 SSL_do_handshake() failed (SSL: error:1408F0C6:SSL routines:ssl3_get_record:packet length too long) while SSL handshaking, client: 142.93.180.14, server: 0.0.0.0:443 +# failJSON: { "time": "2012-04-09T11:53:29", "match": true , "host": "192.0.43.10" } +2012/04/09 11:53:29 [error] 2865#0: *66647 user "xyz" was not found in "/var/www/.htpasswd", client: 192.0.43.10, server: www.myhost.com, request: "GET / HTTP/1.1", host: "www.myhost.com" diff --git a/fail2ban/tests/files/logs/nginx-tls-fallback b/fail2ban/tests/files/logs/nginx-tls-fallback deleted file mode 100644 index bd51fbbf..00000000 --- a/fail2ban/tests/files/logs/nginx-tls-fallback +++ /dev/null @@ -1,9 +0,0 @@ - -# failJSON: { "time": "2020-11-25T14:42:16", "match": true , "host": "142.93.180.14" } -2020/11/25 14:42:16 [crit] 76952#76952: *2454307 SSL_do_handshake() failed (SSL: error:1408F0C6:SSL routines:ssl3_get_record:packet length too long) while SSL handshaking, client: 142.93.180.14, server: 0.0.0.0:443 -# failJSON: { "time": "2020-11-25T15:47:47", "match": true , "host": "80.191.166.166" } -2020/11/25 15:47:47 [crit] 76952#76952: *5062354 SSL_do_handshake() failed (SSL: error:1408F0A0:SSL routines:ssl3_get_record:length too short) while SSL handshaking, client: 80.191.166.166, server: 0.0.0.0:443 -# failJSON: { "time": "2020-11-25T16:48:08", "match": true , "host": "5.126.32.148" } -2020/11/25 16:48:08 [crit] 76952#76952: *7976400 SSL_do_handshake() failed (SSL: error:1408F096:SSL routines:ssl3_get_record:encrypted length too long) while SSL handshaking, client: 5.126.32.148, server: 0.0.0.0:443 -# failJSON: { "time": "2020-11-25T16:02:45", "match": false } -2020/11/25 16:02:45 [error] 76952#76952: *5645766 connect() failed (111: Connection refused) while connecting to upstream, client: 5.126.32.148, server: www.google.de, request: "GET /admin/config HTTP/2.0", upstream: "http://127.0.0.1:3000/admin/config", host: "www.google.de" From cecc3d62ffac65b3545f665dee51eff58c3a7f6e Mon Sep 17 00:00:00 2001 From: stepodev Date: Mon, 30 Nov 2020 12:26:32 +0100 Subject: [PATCH 036/240] add mode explanation to nginx-http-auth in jail.conf --- config/jail.conf | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/config/jail.conf b/config/jail.conf index 3d971892..1b82b648 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -375,8 +375,11 @@ banaction = %(banaction_allports)s logpath = /opt/openhab/logs/request.log +# To use more aggressive http-auth modes set filter parameter "mode" in jail.local: +# normal (default), aggressive (combines all), auth or fallback +# See "tests/files/logs/nginx-http-auth" or "filter.d/nginx-http-auth.conf" for usage example and details. [nginx-http-auth] - +# mode = normal port = http,https logpath = %(nginx_error_log)s From 27e435a7f5bee112c0b2d79fb0d48898afd16ffd Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 22 Dec 2020 18:17:27 +0100 Subject: [PATCH 037/240] fix cymru test cases --- fail2ban/tests/banmanagertestcase.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/fail2ban/tests/banmanagertestcase.py b/fail2ban/tests/banmanagertestcase.py index c2c64a35..6f2e1732 100644 --- a/fail2ban/tests/banmanagertestcase.py +++ b/fail2ban/tests/banmanagertestcase.py @@ -27,6 +27,7 @@ __license__ = "GPL" import unittest from ..server.banmanager import BanManager +from ..server.ipdns import DNSUtils from ..server.ticket import BanTicket class AddFailure(unittest.TestCase): @@ -137,10 +138,10 @@ class StatusExtendedCymruInfo(unittest.TestCase): """Call before every test case.""" super(StatusExtendedCymruInfo, self).setUp() unittest.F2B.SkipIfNoNetwork() - self.__ban_ip = "93.184.216.34" - self.__asn = "15133" - self.__country = "EU" - self.__rir = "ripencc" + self.__ban_ip = iter(DNSUtils.dnsToIp("resolver1.opendns.com")).next() + self.__asn = "36692" + self.__country = "US" + self.__rir = "arin" ticket = BanTicket(self.__ban_ip, 1167605999.0) self.__banManager = BanManager() self.assertTrue(self.__banManager.addBanTicket(ticket)) From 3bb19ecc9d9fe18c3b5ac6a25844d17839543111 Mon Sep 17 00:00:00 2001 From: Tim Gates Date: Wed, 23 Dec 2020 12:20:42 +1100 Subject: [PATCH 038/240] docs: fix simple typo, litle -> little There is a small typo in fail2ban/client/fail2banclient.py, fail2ban/server/observer.py. Should read `little` rather than `litle`. --- fail2ban/client/fail2banclient.py | 2 +- fail2ban/server/observer.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/fail2ban/client/fail2banclient.py b/fail2ban/client/fail2banclient.py index 6ea18fda..a7053034 100755 --- a/fail2ban/client/fail2banclient.py +++ b/fail2ban/client/fail2banclient.py @@ -230,7 +230,7 @@ class Fail2banClient(Fail2banCmdLine, Thread): logSys.log(5, ' client phase %s', phase) if not stream: return False - # wait a litle bit for phase "start-ready" before enter active waiting: + # wait a little bit for phase "start-ready" before enter active waiting: if phase is not None: Utils.wait_for(lambda: phase.get('start-ready', None) is not None, 0.5, 0.001) phase['configure'] = (True if stream else False) diff --git a/fail2ban/server/observer.py b/fail2ban/server/observer.py index f5ba20d9..996eec4a 100644 --- a/fail2ban/server/observer.py +++ b/fail2ban/server/observer.py @@ -232,7 +232,7 @@ class ObserverThread(JailThread): if self._paused: continue else: - ## notify event deleted (shutdown) - just sleep a litle bit (waiting for shutdown events, prevent high cpu usage) + ## notify event deleted (shutdown) - just sleep a little bit (waiting for shutdown events, prevent high cpu usage) time.sleep(ObserverThread.DEFAULT_SLEEP_INTERVAL) ## stop by shutdown and empty queue : if not self.is_full: From ba7daef86c521ceda4ee48b5f221485dd98a13ad Mon Sep 17 00:00:00 2001 From: defanor Date: Thu, 24 Dec 2020 06:55:01 +0300 Subject: [PATCH 039/240] Handle postscreen's PREGREET and HANGUP messages Provoking those seems to be a popular activity among spammers. --- config/filter.d/postfix.conf | 4 +++- fail2ban/tests/files/logs/postfix | 5 +++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/config/filter.d/postfix.conf b/config/filter.d/postfix.conf index fb690fb0..69b4ab48 100644 --- a/config/filter.d/postfix.conf +++ b/config/filter.d/postfix.conf @@ -37,7 +37,9 @@ mdre-rbl = ^RCPT from [^[]*\[\]%(_port)s: [45]54 [45]\.7\.1 Service unava mdpr-more = %(mdpr-normal)s mdre-more = %(mdre-normal)s -mdpr-ddos = (?:lost connection after(?! DATA) [A-Z]+|disconnect(?= from \S+(?: \S+=\d+)* auth=0/(?:[1-9]|\d\d+))) +# Includes some of the log messages described in +# . +mdpr-ddos = (?:lost connection after(?! DATA) [A-Z]+|disconnect(?= from \S+(?: \S+=\d+)* auth=0/(?:[1-9]|\d\d+))|(?:PREGREET \d+|HANGUP) after \S+) mdre-ddos = ^from [^[]*\[\]%(_port)s:? mdpr-extra = (?:%(mdpr-auth)s|%(mdpr-normal)s) diff --git a/fail2ban/tests/files/logs/postfix b/fail2ban/tests/files/logs/postfix index 6e2dc460..9f74e155 100644 --- a/fail2ban/tests/files/logs/postfix +++ b/fail2ban/tests/files/logs/postfix @@ -151,6 +151,11 @@ Feb 18 09:48:04 xxx postfix/smtpd[23]: lost connection after AUTH from unknown[1 # failJSON: { "time": "2005-02-18T09:48:04", "match": true , "host": "192.0.2.23" } Feb 18 09:48:04 xxx postfix/smtpd[23]: lost connection after AUTH from unknown[192.0.2.23] +# failJSON: { "time": "2004-12-23T19:39:13", "match": true , "host": "192.0.2.2" } +Dec 23 19:39:13 xxx postfix/postscreen[21057]: PREGREET 14 after 0.08 from [192.0.2.2]:59415: EHLO ylmf-pc\r\n +# failJSON: { "time": "2004-12-24T00:54:36", "match": true , "host": "192.0.2.3" } +Dec 24 00:54:36 xxx postfix/postscreen[22515]: HANGUP after 16 from [192.0.2.3]:48119 in tests after SMTP handshake + # filterOptions: [{}, {"mode": "ddos"}, {"mode": "aggressive"}] # failJSON: { "match": false, "desc": "don't affect lawful data (sporadical connection aborts within DATA-phase, see gh-1813 for discussion)" } Feb 18 09:50:05 xxx postfix/smtpd[42]: lost connection after DATA from good-host.example.com[192.0.2.10] From 73b39e08947b1858d7574085c3f74318e101e0de Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 29 Dec 2020 21:22:47 +0100 Subject: [PATCH 040/240] filter.d/named-refused.conf: fixes prefix for messages from systemd journal (no mandatory space ahead, because don't have timestamp) closes gh-2899 --- config/filter.d/named-refused.conf | 2 +- fail2ban/tests/files/logs/named-refused | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/config/filter.d/named-refused.conf b/config/filter.d/named-refused.conf index 0d5a6275..8a0b1b8c 100644 --- a/config/filter.d/named-refused.conf +++ b/config/filter.d/named-refused.conf @@ -32,7 +32,7 @@ __daemon_combs_re=(?:%(__pid_re)s?:\s+%(__daemon_re)s|%(__daemon_re)s%(__pid_re) # hostname daemon_id spaces # this can be optional (for instance if we match named native log files) -__line_prefix=(?:\s\S+ %(__daemon_combs_re)s\s+)? +__line_prefix=(?:\s*\S+ %(__daemon_combs_re)s\s+)? prefregex = ^%(__line_prefix)s(?: error:)?\s*client(?: @\S*)? #\S+(?: \([\S.]+\))?: .+\s(?:denied|\(NOTAUTH\))\s*$ diff --git a/fail2ban/tests/files/logs/named-refused b/fail2ban/tests/files/logs/named-refused index 0250276c..c06a4146 100644 --- a/fail2ban/tests/files/logs/named-refused +++ b/fail2ban/tests/files/logs/named-refused @@ -26,3 +26,8 @@ Aug 27 16:58:31 vhost1-ua named[29206]: client 176.9.92.38#42592 (simmarket.com. # failJSON: { "time": "2004-08-27T16:59:00", "match": true , "host": "192.0.2.1", "desc": "new log format, 9.11.0 (#2406)" } Aug 27 16:59:00 host named[28098]: client @0x7f6450002ef0 192.0.2.1#23332 (example.com): bad zone transfer request: 'test.com/IN': non-authoritative zone (NOTAUTH) + +# filterOptions: {"logtype": "journal"} + +# failJSON: { "match": true , "host": "192.0.2.1", "desc": "systemd-journal entry" } +atom named[1806]: client @0x7fb13400eec0 192.0.2.1#61977 (.): query (cache) './ANY/IN' denied From 747d4683221b5584f9663695fb48145689b42ceb Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 4 Jan 2021 02:42:38 +0100 Subject: [PATCH 041/240] fixes century selector of %ExY and %Exy in datepattern for tests, considering interval from 2005 (alternate now) to now; + better grouping algorithm for resulting century RE --- fail2ban/server/strptime.py | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/fail2ban/server/strptime.py b/fail2ban/server/strptime.py index 1464a96d..39fc7958 100644 --- a/fail2ban/server/strptime.py +++ b/fail2ban/server/strptime.py @@ -36,10 +36,30 @@ def _getYearCentRE(cent=(0,3), distance=3, now=(MyTime.now(), MyTime.alternateNo Thereby respect possible run in the test-cases (alternate date used there) """ cent = lambda year, f=cent[0], t=cent[1]: str(year)[f:t] + def grp(exprset): + c = None + if len(exprset) > 1: + for i in exprset: + if c is None or i[0:-1] == c: + c = i[0:-1] + else: + c = None + break + if not c: + for i in exprset: + if c is None or i[0] == c: + c = i[0] + else: + c = None + break + if c: + return "%s%s" % (c, grp([i[len(c):] for i in exprset])) + return ("(?:%s)" % "|".join(exprset) if len(exprset[0]) > 1 else "[%s]" % "".join(exprset)) \ + if len(exprset) > 1 else "".join(exprset) exprset = set( cent(now[0].year + i) for i in (-1, distance) ) if len(now) and now[1]: - exprset |= set( cent(now[1].year + i) for i in (-1, distance) ) - return "(?:%s)" % "|".join(exprset) if len(exprset) > 1 else "".join(exprset) + exprset |= set( cent(now[1].year + i) for i in xrange(-1, now[0].year-now[1].year+1, distance) ) + return grp(sorted(list(exprset))) timeRE = TimeRE() From 0a7f5b7b5c7b352e14c13a8058d0a88753e5f48a Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Wed, 6 Jan 2021 14:29:18 +0100 Subject: [PATCH 042/240] CI: add python 3.10 to github actions --- .github/workflows/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 7a1d31df..c5bb5df2 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -22,7 +22,7 @@ jobs: runs-on: ubuntu-20.04 strategy: matrix: - python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, pypy2, pypy3] + python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10, pypy2, pypy3] fail-fast: false # Steps represent a sequence of tasks that will be executed as part of the job steps: From 3c8aa0e6ba4ca46aa730bbb957653e0468365b49 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Wed, 6 Jan 2021 14:34:08 +0100 Subject: [PATCH 043/240] amend with correct yaml notation (float vs string) --- .github/workflows/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index c5bb5df2..7d69d2e5 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -22,7 +22,7 @@ jobs: runs-on: ubuntu-20.04 strategy: matrix: - python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10, pypy2, pypy3] + python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, '3.10', pypy2, pypy3] fail-fast: false # Steps represent a sequence of tasks that will be executed as part of the job steps: From 097d9ea7fe4cb2e9285203ca56646d0ea003b064 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Wed, 6 Jan 2021 14:36:09 +0100 Subject: [PATCH 044/240] try wildcard (3.10 is still alpha) --- .github/workflows/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 7d69d2e5..78f40b8a 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -22,7 +22,7 @@ jobs: runs-on: ubuntu-20.04 strategy: matrix: - python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, '3.10', pypy2, pypy3] + python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, '3.10.*', pypy2, pypy3] fail-fast: false # Steps represent a sequence of tasks that will be executed as part of the job steps: From c9907bef0d6fd4d26c9f7a92611cfd03ea434dca Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Wed, 6 Jan 2021 14:37:34 +0100 Subject: [PATCH 045/240] CI: 3.10.0-alpha.4 --- .github/workflows/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 78f40b8a..07086636 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -22,7 +22,7 @@ jobs: runs-on: ubuntu-20.04 strategy: matrix: - python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, '3.10.*', pypy2, pypy3] + python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, '3.10.0-alpha.4', pypy2, pypy3] fail-fast: false # Steps represent a sequence of tasks that will be executed as part of the job steps: From 2781e7b9d1111005e2bc97c96ddb53f0fbf47f99 Mon Sep 17 00:00:00 2001 From: Michael Haro Date: Fri, 8 Jan 2021 15:01:25 -0800 Subject: [PATCH 046/240] Add blank line so markdown renders correctly --- FILTERS | 1 + 1 file changed, 1 insertion(+) diff --git a/FILTERS b/FILTERS index e114973a..2ed6281d 100644 --- a/FILTERS +++ b/FILTERS @@ -278,6 +278,7 @@ to tune it. fail2ban-regex -D ... will present Debuggex URLs for the regexs and sample log files that you pass into it. In general use when using regex debuggers for generating fail2ban filters: + * use regex from the ./fail2ban-regex output (to ensure all substitutions are done) * replace with (?&.ipv4) From 9df332fdef853c0ae27578d84beda3a1c41deee1 Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 11 Jan 2021 15:10:53 +0100 Subject: [PATCH 047/240] filter.d/apache-overflows.conf: extended to match AH00126 error (Invalid URI ...); closes gh-2908 --- config/filter.d/apache-overflows.conf | 2 +- fail2ban/tests/files/logs/apache-overflows | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/config/filter.d/apache-overflows.conf b/config/filter.d/apache-overflows.conf index 02a2ef20..0f54da11 100644 --- a/config/filter.d/apache-overflows.conf +++ b/config/filter.d/apache-overflows.conf @@ -8,7 +8,7 @@ before = apache-common.conf [Definition] -failregex = ^%(_apache_error_client)s (?:(?:AH0013[456]: )?Invalid (method|URI) in request\b|(?:AH00565: )?request failed: URI too long \(longer than \d+\)|request failed: erroneous characters after protocol string:|(?:AH00566: )?request failed: invalid characters in URI\b) +failregex = ^%(_apache_error_client)s (?:(?:AH001[23][456]: )?Invalid (method|URI) in request\b|(?:AH00565: )?request failed: URI too long \(longer than \d+\)|request failed: erroneous characters after protocol string:|(?:AH00566: )?request failed: invalid characters in URI\b) ignoreregex = diff --git a/fail2ban/tests/files/logs/apache-overflows b/fail2ban/tests/files/logs/apache-overflows index 376114c4..4be013eb 100644 --- a/fail2ban/tests/files/logs/apache-overflows +++ b/fail2ban/tests/files/logs/apache-overflows @@ -3,6 +3,8 @@ [Tue Mar 16 15:39:29 2010] [error] [client 58.179.109.179] Invalid URI in request \xf9h\xa9\xf3\x88\x8cXKj \xbf-l*4\x87n\xe4\xfe\xd4\x1d\x06\x8c\xf8m\\rS\xf6n\xeb\x8 # failJSON: { "time": "2010-03-15T15:44:47", "match": true , "host": "121.222.2.133" } [Mon Mar 15 15:44:47 2010] [error] [client 121.222.2.133] Invalid URI in request n\xed*\xbe*\xab\xefd\x80\xb5\xae\xf6\x01\x10M?\xf2\xce\x13\x9c\xd7\xa0N\xa7\xdb%0\xde\xe0\xfc\xd2\xa0\xfe\xe9w\xee\xc4`v\x9b[{\x0c:\xcb\x93\xc6\xa0\x93\x9c`l\\\x8d\xc9 +# failJSON: { "time": "2010-03-15T16:04:06", "match": true , "host": "192.0.2.1", "desc": "AH00126 failure, gh-2908" } +[Sat Mar 15 16:04:06.105212 2010] [core:error] [pid 17408] [client 192.0.2.1:55280] AH00126: Invalid URI in request GET /static/../../../a/../../../../etc/passwd HTTP/1.1 # http://forum.nconf.org/viewtopic.php?f=14&t=427&p=1488 # failJSON: { "time": "2010-07-30T11:23:54", "match": true , "host": "10.85.6.69" } From 5f3f4d1e2ffcf99384c96c9800d1aa913ccce81b Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Mon, 11 Jan 2021 15:23:40 +0100 Subject: [PATCH 048/240] action.d/cloudflare.conf: better IPv6 capability closes gh-2891 --- config/action.d/cloudflare.conf | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/config/action.d/cloudflare.conf b/config/action.d/cloudflare.conf index 361cb177..4af87080 100644 --- a/config/action.d/cloudflare.conf +++ b/config/action.d/cloudflare.conf @@ -44,7 +44,7 @@ actioncheck = #actionban = curl -s -o /dev/null https://www.cloudflare.com/api_json.html -d 'a=ban' -d 'tkn=' -d 'email=' -d 'key=' # API v4 actionban = curl -s -o /dev/null -X POST <_cf_api_prms> \ - -d '{"mode":"block","configuration":{"target":"ip","value":""},"notes":"Fail2Ban "}' \ + -d '{"mode":"block","configuration":{"target":"","value":""},"notes":"Fail2Ban "}' \ <_cf_api_url> # Option: actionunban @@ -59,7 +59,7 @@ actionban = curl -s -o /dev/null -X POST <_cf_api_prms> \ #actionunban = curl -s -o /dev/null https://www.cloudflare.com/api_json.html -d 'a=nul' -d 'tkn=' -d 'email=' -d 'key=' # API v4 actionunban = id=$(curl -s -X GET <_cf_api_prms> \ - "<_cf_api_url>?mode=block&configuration_target=ip&configuration_value=&page=1&per_page=1¬es=Fail2Ban%%20" \ + "<_cf_api_url>?mode=block&configuration_target=&configuration_value=&page=1&per_page=1¬es=Fail2Ban%%20" \ | { jq -r '.result[0].id' 2>/dev/null || tr -d '\n' | sed -nE 's/^.*"result"\s*:\s*\[\s*\{\s*"id"\s*:\s*"([^"]+)".*$/\1/p'; }) if [ -z "$id" ]; then echo ": id for cannot be found"; exit 0; fi; curl -s -o /dev/null -X DELETE <_cf_api_prms> "<_cf_api_url>/$id" @@ -81,3 +81,8 @@ _cf_api_prms = -H 'X-Auth-Email: ' -H 'X-Auth-Key: ' -H 'Conten cftoken = cfuser = + +cftarget = ip + +[Init?family=inet6] +cftarget = ip6 From 164105fab12a8d97050eb1930cbc338b66d6e8eb Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Sat, 16 Jan 2021 17:10:12 +0100 Subject: [PATCH 049/240] added new parameter `namespace` for systemd backend closes gh-2910 --- fail2ban/server/filtersystemd.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/fail2ban/server/filtersystemd.py b/fail2ban/server/filtersystemd.py index 47fc891e..1b33b115 100644 --- a/fail2ban/server/filtersystemd.py +++ b/fail2ban/server/filtersystemd.py @@ -94,6 +94,11 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover # be sure all journal types will be opened if files specified (don't set flags): if 'files' not in args or not len(args['files']): args['flags'] = 4 + + try: + args['namespace'] = kwargs.pop('namespace') + except KeyError: + pass return args From 0f44a3408aa88d32ab2c4583ac6f54ea82220069 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 19 Jan 2021 17:07:03 +0100 Subject: [PATCH 050/240] amend to 747d4683221b5584f9663695fb48145689b42ceb: fail2ban-regex: loosen up date patterns %ExY, %Exy - let accept every year from 19xx up to current century (+3 years) --- fail2ban/client/fail2banregex.py | 11 +++-- fail2ban/server/datedetector.py | 2 +- fail2ban/server/datetemplate.py | 3 +- fail2ban/server/strptime.py | 81 ++++++++++++++++++-------------- 4 files changed, 55 insertions(+), 42 deletions(-) diff --git a/fail2ban/client/fail2banregex.py b/fail2ban/client/fail2banregex.py index e7a4e214..5d5f4a1c 100644 --- a/fail2ban/client/fail2banregex.py +++ b/fail2ban/client/fail2banregex.py @@ -39,7 +39,6 @@ import os import shlex import sys import time -import time import urllib from optparse import OptionParser, Option @@ -52,7 +51,7 @@ except ImportError: from ..version import version, normVersion from .filterreader import FilterReader -from ..server.filter import Filter, FileContainer +from ..server.filter import Filter, FileContainer, MyTime from ..server.failregex import Regex, RegexException from ..helpers import str2LogLevel, getVerbosityFormat, FormatterWithTraceBack, getLogger, \ @@ -269,15 +268,19 @@ class Fail2banRegex(object): self.setJournalMatch(shlex.split(opts.journalmatch)) if opts.timezone: self._filter.setLogTimeZone(opts.timezone) + self._filter.checkFindTime = False + if True: # not opts.out: + MyTime.setAlternateNow(0); # accept every date (years from 19xx up to end of current century, '%ExY' and 'Exy' patterns) + from ..server.strptime import _updateTimeRE + _updateTimeRE() if opts.datepattern: self.setDatePattern(opts.datepattern) if opts.usedns: self._filter.setUseDns(opts.usedns) self._filter.returnRawHost = opts.raw - self._filter.checkFindTime = False self._filter.checkAllRegex = opts.checkAllRegex and not opts.out # ignore pending (without ID/IP), added to matches if it hits later (if ID/IP can be retreved) - self._filter.ignorePending = opts.out + self._filter.ignorePending = bool(opts.out) # callback to increment ignored RE's by index (during process): self._filter.onIgnoreRegex = self._onIgnoreRegex self._backend = 'auto' diff --git a/fail2ban/server/datedetector.py b/fail2ban/server/datedetector.py index 90a70b0d..ecc9d935 100644 --- a/fail2ban/server/datedetector.py +++ b/fail2ban/server/datedetector.py @@ -35,7 +35,7 @@ from ..helpers import getLogger # Gets the instance of the logger. logSys = getLogger(__name__) -logLevel = 6 +logLevel = 5 RE_DATE_PREMATCH = re.compile(r"(? 1: - for i in exprset: - if c is None or i[0:-1] == c: - c = i[0:-1] - else: - c = None - break - if not c: - for i in exprset: - if c is None or i[0] == c: - c = i[0] - else: - c = None - break - if c: - return "%s%s" % (c, grp([i[len(c):] for i in exprset])) - return ("(?:%s)" % "|".join(exprset) if len(exprset[0]) > 1 else "[%s]" % "".join(exprset)) \ - if len(exprset) > 1 else "".join(exprset) - exprset = set( cent(now[0].year + i) for i in (-1, distance) ) - if len(now) and now[1]: - exprset |= set( cent(now[1].year + i) for i in xrange(-1, now[0].year-now[1].year+1, distance) ) - return grp(sorted(list(exprset))) - timeRE = TimeRE() # %k - one- or two-digit number giving the hour of the day (0-23) on a 24-hour clock, @@ -92,11 +61,51 @@ timeRE['Exk'] = r" ?(?P2[0-3]|[0-1]\d|\d)" timeRE['Exl'] = r" ?(?P1[0-2]|\d)" timeRE['ExM'] = r"(?P[0-5]\d)" timeRE['ExS'] = r"(?P6[0-1]|[0-5]\d)" -# more precise year patterns, within same century of last year and -# the next 3 years (for possible long uptime of fail2ban); thereby -# respect possible run in the test-cases (alternate date used there): -timeRE['ExY'] = r"(?P%s\d)" % _getYearCentRE(cent=(0,3), distance=3) -timeRE['Exy'] = r"(?P%s\d)" % _getYearCentRE(cent=(2,3), distance=3) + +def _updateTimeRE(): + def _getYearCentRE(cent=(0,3), distance=3, now=(MyTime.now(), MyTime.alternateNow)): + """ Build century regex for last year and the next years (distance). + + Thereby respect possible run in the test-cases (alternate date used there) + """ + cent = lambda year, f=cent[0], t=cent[1]: str(year)[f:t] + def grp(exprset): + c = None + if len(exprset) > 1: + for i in exprset: + if c is None or i[0:-1] == c: + c = i[0:-1] + else: + c = None + break + if not c: + for i in exprset: + if c is None or i[0] == c: + c = i[0] + else: + c = None + break + if c: + return "%s%s" % (c, grp([i[len(c):] for i in exprset])) + return ("(?:%s)" % "|".join(exprset) if len(exprset[0]) > 1 else "[%s]" % "".join(exprset)) \ + if len(exprset) > 1 else "".join(exprset) + exprset = set( cent(now[0].year + i) for i in (-1, distance) ) + if len(now) > 1 and now[1]: + exprset |= set( cent(now[1].year + i) for i in xrange(-1, now[0].year-now[1].year+1, distance) ) + return grp(sorted(list(exprset))) + + # more precise year patterns, within same century of last year and + # the next 3 years (for possible long uptime of fail2ban); thereby + # respect possible run in the test-cases (alternate date used there): + if MyTime.alternateNowTime != 0: + timeRE['ExY'] = r"(?P%s\d)" % _getYearCentRE(cent=(0,3), distance=3) + timeRE['Exy'] = r"(?P%s\d)" % _getYearCentRE(cent=(2,3), distance=3) + else: # accept years: 19xx|2xxx up to current century + timeRE['ExY'] = r"(?P(?:19\d{2}|%s\d))" % _getYearCentRE(cent=(0,3), distance=3, + now=(MyTime.now(), datetime.datetime.fromtimestamp(978393600))) + timeRE['Exy'] = r"(?P\d{2})" + +_updateTimeRE() def getTimePatternRE(): keys = timeRE.keys() From 913c37db80d3f78c37e7df2f18a503baf3f6dabc Mon Sep 17 00:00:00 2001 From: sebres Date: Thu, 21 Jan 2021 18:56:25 +0100 Subject: [PATCH 051/240] more fixes and optimizations, better RE's for patterns, allow parse date without time with such a datepattern (assume 00:00:00 then), etc --- fail2ban/server/strptime.py | 22 +++++++++++++++------- fail2ban/tests/datedetectortestcase.py | 3 +++ 2 files changed, 18 insertions(+), 7 deletions(-) diff --git a/fail2ban/server/strptime.py b/fail2ban/server/strptime.py index 83cb3989..69514b20 100644 --- a/fail2ban/server/strptime.py +++ b/fail2ban/server/strptime.py @@ -52,15 +52,23 @@ timeRE['z'] = r"(?PZ|UTC|GMT|[+-][01]\d(?::?\d{2})?)" timeRE['ExZ'] = r"(?P%s)" % (TZ_ABBR_RE,) timeRE['Exz'] = r"(?P(?:%s)?[+-][01]\d(?::?\d{2})?|%s)" % (TZ_ABBR_RE, TZ_ABBR_RE) +# overwrite default patterns, since they can be non-optimal: +timeRE['d'] = r"(?P[1-2]\d|[0 ]?[1-9]|3[0-1])" +timeRE['m'] = r"(?P0?[1-9]|1[0-2])" +timeRE['Y'] = r"(?P\d{4})" +timeRE['H'] = r"(?P[0-1]?\d|2[0-3])" +timeRE['M'] = r"(?P[0-5]?\d)" +timeRE['S'] = r"(?P[0-5]?\d|6[0-1])" + # Extend build-in TimeRE with some exact patterns # exact two-digit patterns: -timeRE['Exd'] = r"(?P3[0-1]|[1-2]\d|0[1-9])" -timeRE['Exm'] = r"(?P1[0-2]|0[1-9])" -timeRE['ExH'] = r"(?P2[0-3]|[0-1]\d)" -timeRE['Exk'] = r" ?(?P2[0-3]|[0-1]\d|\d)" +timeRE['Exd'] = r"(?P[1-2]\d|0[1-9]|3[0-1])" +timeRE['Exm'] = r"(?P0[1-9]|1[0-2])" +timeRE['ExH'] = r"(?P[0-1]\d|2[0-3])" +timeRE['Exk'] = r" ?(?P[0-1]?\d|2[0-3])" timeRE['Exl'] = r" ?(?P1[0-2]|\d)" timeRE['ExM'] = r"(?P[0-5]\d)" -timeRE['ExS'] = r"(?P6[0-1]|[0-5]\d)" +timeRE['ExS'] = r"(?P[0-5]\d|6[0-1])" def _updateTimeRE(): def _getYearCentRE(cent=(0,3), distance=3, now=(MyTime.now(), MyTime.alternateNow)): @@ -197,9 +205,9 @@ def reGroupDictStrptime(found_dict, msec=False, default_tz=None): """ now = \ - year = month = day = hour = minute = tzoffset = \ + year = month = day = tzoffset = \ weekday = julian = week_of_year = None - second = fraction = 0 + hour = minute = second = fraction = 0 for key, val in found_dict.iteritems(): if val is None: continue # Directives not explicitly handled below: diff --git a/fail2ban/tests/datedetectortestcase.py b/fail2ban/tests/datedetectortestcase.py index d6370fc4..b8e8451e 100644 --- a/fail2ban/tests/datedetectortestcase.py +++ b/fail2ban/tests/datedetectortestcase.py @@ -551,6 +551,9 @@ class CustomDateFormatsTest(unittest.TestCase): (1123970401.0, "^%ExH:%ExM:%ExS**", '00:00:01'), # cover date with current year, in test cases now == Aug 2005 -> back to last year (Sep 2004): (1094068799.0, "^%m/%d %ExH:%ExM:%ExS**", '09/01 21:59:59'), + # no time (only date) in pattern, assume local 00:00:00 for H:M:S : + (1093989600.0, "^%Y-%m-%d**", '2004-09-01'), + (1093996800.0, "^%Y-%m-%d%z**", '2004-09-01Z'), ): logSys.debug('== test: %r', (matched, dp, line)) dd = DateDetector() From 3700a9e5230822d33b2377afed9b6de38676e564 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 26 Jan 2021 20:25:58 +0100 Subject: [PATCH 052/240] invalidate IP/DNS caches by reload, so inter alia would allow to recognize IPv6IsAllowed immediately, previously retarded up to cache max-time (5m); closes gh-2804 --- fail2ban/server/server.py | 9 ++++++++- fail2ban/tests/utils.py | 2 ++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/fail2ban/server/server.py b/fail2ban/server/server.py index b12c8f9f..475fd706 100644 --- a/fail2ban/server/server.py +++ b/fail2ban/server/server.py @@ -33,7 +33,7 @@ import stat import sys from .jails import Jails -from .filter import FileFilter, JournalFilter +from .filter import DNSUtils, FileFilter, JournalFilter from .transmitter import Transmitter from .asyncserver import AsyncServer, AsyncServerException from .. import version @@ -274,6 +274,11 @@ class Server: for name in self.__jails.keys(): self.delJail(name, stop=False, join=True) + def clearCaches(self): + # we need to clear caches, to be able to recognize new IPs/families etc: + DNSUtils.CACHE_nameToIp.clear() + DNSUtils.CACHE_ipToName.clear() + def reloadJails(self, name, opts, begin): if begin: # begin reload: @@ -295,6 +300,8 @@ class Server: if "--restart" in opts: self.stopJail(name) else: + # invalidate caches by reload + self.clearCaches() # first unban all ips (will be not restored after (re)start): if "--unban" in opts: self.setUnbanIP() diff --git a/fail2ban/tests/utils.py b/fail2ban/tests/utils.py index b54581f5..921427db 100644 --- a/fail2ban/tests/utils.py +++ b/fail2ban/tests/utils.py @@ -320,6 +320,7 @@ def initTests(opts): # precache all invalid ip's (TEST-NET-1, ..., TEST-NET-3 according to RFC 5737): c = DNSUtils.CACHE_ipToName + c.clear = lambda: logSys.warn('clear CACHE_ipToName is disabled in test suite') # increase max count and max time (too many entries, long time testing): c.setOptions(maxCount=10000, maxTime=5*60) for i in xrange(256): @@ -337,6 +338,7 @@ def initTests(opts): c.set('8.8.4.4', 'dns.google') # precache all dns to ip's used in test cases: c = DNSUtils.CACHE_nameToIp + c.clear = lambda: logSys.warn('clear CACHE_nameToIp is disabled in test suite') for i in ( ('999.999.999.999', set()), ('abcdef.abcdef', set()), From c75748c5d3b4de1e51318e1db4a5b72519c47c1f Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 27 Jan 2021 17:05:37 +0100 Subject: [PATCH 053/240] fail2ban.conf: added new fail2ban configuration option "allowipv6" (default auto), can be used to allow or disallow IPv6 interface in fail2ban immediately by start (e. g. if fail2ban starts before network interfaces). closes gh-2804 --- config/fail2ban.conf | 6 ++++ fail2ban/client/fail2banreader.py | 2 ++ fail2ban/server/ipdns.py | 41 +++++++++++++++++++------- fail2ban/server/server.py | 5 ++++ fail2ban/server/transmitter.py | 5 ++++ fail2ban/tests/clientreadertestcase.py | 1 + fail2ban/tests/servertestcase.py | 15 +++++++++- man/jail.conf.5 | 5 ++++ 8 files changed, 69 insertions(+), 11 deletions(-) diff --git a/config/fail2ban.conf b/config/fail2ban.conf index f3867839..601402d8 100644 --- a/config/fail2ban.conf +++ b/config/fail2ban.conf @@ -55,6 +55,12 @@ socket = /var/run/fail2ban/fail2ban.sock # pidfile = /var/run/fail2ban/fail2ban.pid +# Option: allowipv6 +# Notes.: Allows IPv6 interface: +# Default: auto +# Values: [ auto yes (on, true, 1) no (off, false, 0) ] Default: auto +#allowipv6 = auto + # Options: dbfile # Notes.: Set the file for the fail2ban persistent data to be stored. # A value of ":memory:" means database is only stored in memory diff --git a/fail2ban/client/fail2banreader.py b/fail2ban/client/fail2banreader.py index 3270b767..1f135cf8 100644 --- a/fail2ban/client/fail2banreader.py +++ b/fail2ban/client/fail2banreader.py @@ -53,6 +53,7 @@ class Fail2banReader(ConfigReader): opts = [["string", "loglevel", "INFO" ], ["string", "logtarget", "STDERR"], ["string", "syslogsocket", "auto"], + ["string", "allowipv6", "auto"], ["string", "dbfile", "/var/lib/fail2ban/fail2ban.sqlite3"], ["int", "dbmaxmatches", None], ["string", "dbpurgeage", "1d"]] @@ -74,6 +75,7 @@ class Fail2banReader(ConfigReader): # Also dbfile should be set before all other database options. # So adding order indices into items, to be stripped after sorting, upon return order = {"thread":0, "syslogsocket":11, "loglevel":12, "logtarget":13, + "allowipv6": 14, "dbfile":50, "dbmaxmatches":51, "dbpurgeage":51} stream = list() for opt in self.__opts: diff --git a/fail2ban/server/ipdns.py b/fail2ban/server/ipdns.py index 571ccc4f..5f3e4571 100644 --- a/fail2ban/server/ipdns.py +++ b/fail2ban/server/ipdns.py @@ -169,27 +169,31 @@ class DNSUtils: DNSUtils.CACHE_ipToName.set(key, name) return name + # key find cached own hostnames (this tuple-key cannot be used elsewhere): + _getSelfNames_key = ('self','dns') + @staticmethod def getSelfNames(): """Get own host names of self""" - # try find cached own hostnames (this tuple-key cannot be used elsewhere): - key = ('self','dns') - names = DNSUtils.CACHE_ipToName.get(key) + # try find cached own hostnames: + names = DNSUtils.CACHE_ipToName.get(DNSUtils._getSelfNames_key) # get it using different ways (a set with names of localhost, hostname, fully qualified): if names is None: names = set([ 'localhost', DNSUtils.getHostname(False), DNSUtils.getHostname(True) ]) - set(['']) # getHostname can return '' # cache and return : - DNSUtils.CACHE_ipToName.set(key, names) + DNSUtils.CACHE_ipToName.set(DNSUtils._getSelfNames_key, names) return names + # key to find cached own IPs (this tuple-key cannot be used elsewhere): + _getSelfIPs_key = ('self','ips') + @staticmethod def getSelfIPs(): """Get own IP addresses of self""" - # try find cached own IPs (this tuple-key cannot be used elsewhere): - key = ('self','ips') - ips = DNSUtils.CACHE_nameToIp.get(key) + # to find cached own IPs: + ips = DNSUtils.CACHE_nameToIp.get(DNSUtils._getSelfIPs_key) # get it using different ways (a set with IPs of localhost, hostname, fully qualified): if ips is None: ips = set() @@ -199,13 +203,30 @@ class DNSUtils: except Exception as e: # pragma: no cover logSys.warning("Retrieving own IPs of %s failed: %s", hostname, e) # cache and return : - DNSUtils.CACHE_nameToIp.set(key, ips) + DNSUtils.CACHE_nameToIp.set(DNSUtils._getSelfIPs_key, ips) return ips + _IPv6IsAllowed = None + + @staticmethod + def setIPv6IsAllowed(value): + DNSUtils._IPv6IsAllowed = value + logSys.debug("IPv6 is %s", ('on' if value else 'off') if value is not None else 'auto') + return value + + # key to find cached value of IPv6 allowance (this tuple-key cannot be used elsewhere): + _IPv6IsAllowed_key = ('self','ipv6-allowed') + @staticmethod def IPv6IsAllowed(): - # return os.path.exists("/proc/net/if_inet6") || any((':' in ip) for ip in DNSUtils.getSelfIPs()) - return any((':' in ip.ntoa) for ip in DNSUtils.getSelfIPs()) + if DNSUtils._IPv6IsAllowed is not None: + return DNSUtils._IPv6IsAllowed + v = DNSUtils.CACHE_nameToIp.get(DNSUtils._IPv6IsAllowed_key) + if v is not None: + return v + v = any((':' in ip.ntoa) for ip in DNSUtils.getSelfIPs()) + DNSUtils.CACHE_nameToIp.set(DNSUtils._IPv6IsAllowed_key, v) + return v ## diff --git a/fail2ban/server/server.py b/fail2ban/server/server.py index 475fd706..bd2c7ad3 100644 --- a/fail2ban/server/server.py +++ b/fail2ban/server/server.py @@ -770,6 +770,11 @@ class Server: logSys.info("flush performed on %s" % self.__logTarget) return "flushed" + @staticmethod + def setIPv6IsAllowed(value): + value = _as_bool(value) if value != 'auto' else None + return DNSUtils.setIPv6IsAllowed(value) + def setThreadOptions(self, value): for o, v in value.iteritems(): if o == 'stacksize': diff --git a/fail2ban/server/transmitter.py b/fail2ban/server/transmitter.py index 10cfd163..e40f2f51 100644 --- a/fail2ban/server/transmitter.py +++ b/fail2ban/server/transmitter.py @@ -173,6 +173,11 @@ class Transmitter: return self.__server.getSyslogSocket() else: raise Exception("Failed to change syslog socket") + elif name == "allowipv6": + value = command[1] + self.__server.setIPv6IsAllowed(value) + if self.__quiet: return + return value #Thread elif name == "thread": value = command[1] diff --git a/fail2ban/tests/clientreadertestcase.py b/fail2ban/tests/clientreadertestcase.py index 2cfaff77..54850bca 100644 --- a/fail2ban/tests/clientreadertestcase.py +++ b/fail2ban/tests/clientreadertestcase.py @@ -975,6 +975,7 @@ class JailsReaderTest(LogCaptureTestCase): ['set', 'syslogsocket', 'auto'], ['set', 'loglevel', "INFO"], ['set', 'logtarget', '/var/log/fail2ban.log'], + ['set', 'allowipv6', 'auto'], ['set', 'dbfile', '/var/lib/fail2ban/fail2ban.sqlite3'], ['set', 'dbmaxmatches', 10], ['set', 'dbpurgeage', '1d'], diff --git a/fail2ban/tests/servertestcase.py b/fail2ban/tests/servertestcase.py index eaf1b346..b7b9d802 100644 --- a/fail2ban/tests/servertestcase.py +++ b/fail2ban/tests/servertestcase.py @@ -35,7 +35,7 @@ import platform from ..server.failregex import Regex, FailRegex, RegexException from ..server import actions as _actions from ..server.server import Server -from ..server.ipdns import IPAddr +from ..server.ipdns import DNSUtils, IPAddr from ..server.jail import Jail from ..server.jailthread import JailThread from ..server.ticket import BanTicket @@ -175,6 +175,19 @@ class Transmitter(TransmitterBase): def testVersion(self): self.assertEqual(self.transm.proceed(["version"]), (0, version.version)) + def testSetIPv6(self): + try: + self.assertEqual(self.transm.proceed(["set", "allowipv6", 'yes']), (0, 'yes')) + self.assertTrue(DNSUtils.IPv6IsAllowed()) + self.assertLogged("IPv6 is on"); self.pruneLog() + self.assertEqual(self.transm.proceed(["set", "allowipv6", 'no']), (0, 'no')) + self.assertFalse(DNSUtils.IPv6IsAllowed()) + self.assertLogged("IPv6 is off"); self.pruneLog() + finally: + # restore back to auto: + self.assertEqual(self.transm.proceed(["set", "allowipv6", "auto"]), (0, "auto")) + self.assertLogged("IPv6 is auto"); self.pruneLog() + def testSleep(self): if not unittest.F2B.fast: t0 = time.time() diff --git a/man/jail.conf.5 b/man/jail.conf.5 index dc226ac2..788fad2b 100644 --- a/man/jail.conf.5 +++ b/man/jail.conf.5 @@ -151,6 +151,11 @@ PID filename. Default: /var/run/fail2ban/fail2ban.pid .br This is used to store the process ID of the fail2ban server. .TP +.B allowipv6 +option to allow IPv6 interface - auto, yes (on, true, 1) or no (off, false, 0). Default: auto +.br +This value can be used to declare fail2ban whether IPv6 is allowed or not. +.TP .B dbfile Database filename. Default: /var/lib/fail2ban/fail2ban.sqlite3 .br From dc4ee5aa47e211bb8ca7417c73b61d4c3dc44a14 Mon Sep 17 00:00:00 2001 From: "Brian J. Murrell" Date: Mon, 18 Jan 2021 17:43:06 -0500 Subject: [PATCH 054/240] Add transport to asterisk RE Call rejection messages from Asterisk can have the transport prefixed to the IP address. Signed-off-by: Brian J. Murrell --- config/filter.d/asterisk.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/filter.d/asterisk.conf b/config/filter.d/asterisk.conf index 68472495..e15d7bfe 100644 --- a/config/filter.d/asterisk.conf +++ b/config/filter.d/asterisk.conf @@ -21,7 +21,7 @@ log_prefix= (?:NOTICE|SECURITY|WARNING)%(__pid_re)s:?(?:\[C-[\da-f]*\])?:? [^:]+ prefregex = ^%(__prefix_line)s%(log_prefix)s .+$ failregex = ^Registration from '[^']*' failed for '(:\d+)?' - (?:Wrong password|Username/auth name mismatch|No matching peer found|Not a local domain|Device does not match ACL|Peer is not supposed to register|ACL error \(permit/deny\)|Not a local domain)$ - ^Call from '[^']*' \(:\d+\) to extension '[^']*' rejected because extension not found in context + ^Call from '[^']*' \((?:(?:TCP|UDP):)?:\d+\) to extension '[^']*' rejected because extension not found in context ^(?:Host )? (?:failed (?:to authenticate\b|MD5 authentication\b)|tried to authenticate with nonexistent user\b) ^No registration for peer '[^']*' \(from \)$ ^hacking attempt detected ''$ From 69c96c00c0ed6cdeb2e90a496e91b569c14cef14 Mon Sep 17 00:00:00 2001 From: "Brian J. Murrell" Date: Mon, 18 Jan 2021 17:51:38 -0500 Subject: [PATCH 055/240] Log entries for updated failregex Add a sample failregex. Signed-off-by: Brian J. Murrell --- fail2ban/tests/files/logs/asterisk | 2 ++ 1 file changed, 2 insertions(+) diff --git a/fail2ban/tests/files/logs/asterisk b/fail2ban/tests/files/logs/asterisk index 76ec40b2..0c4f81bf 100644 --- a/fail2ban/tests/files/logs/asterisk +++ b/fail2ban/tests/files/logs/asterisk @@ -19,6 +19,8 @@ [2012-02-13 17:44:26] NOTICE[1638] chan_iax2.c: Host 1.2.3.4 failed MD5 authentication for 'Fail2ban' (e7df7cd2ca07f4f1ab415d457a6e1c13 != 53ac4bc41ee4ec77888ed4aa50677247) # failJSON: { "time": "2013-02-05T23:44:42", "match": true , "host": "1.2.3.4" } [2013-02-05 23:44:42] NOTICE[436][C-00000fa9] chan_sip.c: Call from '' (1.2.3.4:10836) to extension '0972598285108' rejected because extension not found in context 'default'. +# failJSON: { "time": "Jan 18 17:39:50", "match": true , "host": "1.2.3.4" } +[Jan 18 17:39:50] NOTICE[12049]: res_pjsip_session.c:2337 new_invite: Call from 'anonymous' (TCP:[1.2.3.4]:61470) to extension '9011+442037690237' rejected because extension not found in context 'default'. # failJSON: { "time": "2013-03-26T15:47:54", "match": true , "host": "1.2.3.4" } [2013-03-26 15:47:54] NOTICE[1237] chan_sip.c: Registration from '"100"sip:100@1.2.3.4' failed for '1.2.3.4:23930' - No matching peer found # failJSON: { "time": "2013-05-13T07:10:53", "match": true , "host": "1.2.3.4" } From 7f185a828e6573669535b152fb945a366270d809 Mon Sep 17 00:00:00 2001 From: "Brian J. Murrell" Date: Tue, 19 Jan 2021 08:42:05 -0500 Subject: [PATCH 056/240] Update date in failJSON The date format in failJSON is specific, so convert the date to use that format. --- fail2ban/tests/files/logs/asterisk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fail2ban/tests/files/logs/asterisk b/fail2ban/tests/files/logs/asterisk index 0c4f81bf..ab31fa6f 100644 --- a/fail2ban/tests/files/logs/asterisk +++ b/fail2ban/tests/files/logs/asterisk @@ -19,7 +19,7 @@ [2012-02-13 17:44:26] NOTICE[1638] chan_iax2.c: Host 1.2.3.4 failed MD5 authentication for 'Fail2ban' (e7df7cd2ca07f4f1ab415d457a6e1c13 != 53ac4bc41ee4ec77888ed4aa50677247) # failJSON: { "time": "2013-02-05T23:44:42", "match": true , "host": "1.2.3.4" } [2013-02-05 23:44:42] NOTICE[436][C-00000fa9] chan_sip.c: Call from '' (1.2.3.4:10836) to extension '0972598285108' rejected because extension not found in context 'default'. -# failJSON: { "time": "Jan 18 17:39:50", "match": true , "host": "1.2.3.4" } +# failJSON: { "time": "2005-01-18T17:39:50", "match": true , "host": "1.2.3.4" } [Jan 18 17:39:50] NOTICE[12049]: res_pjsip_session.c:2337 new_invite: Call from 'anonymous' (TCP:[1.2.3.4]:61470) to extension '9011+442037690237' rejected because extension not found in context 'default'. # failJSON: { "time": "2013-03-26T15:47:54", "match": true , "host": "1.2.3.4" } [2013-03-26 15:47:54] NOTICE[1237] chan_sip.c: Registration from '"100"sip:100@1.2.3.4' failed for '1.2.3.4:23930' - No matching peer found From 366c64cb9da3c5d833eefc83f5bb200713db29b5 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 3 Feb 2021 14:45:30 +0100 Subject: [PATCH 057/240] extractOptions: ensure options are parsed completely - avoids unexpected skip or truncate of parameters, produces more verbose error message in case of incorrect syntax; added more tests covering several cases WARN: potential incompatibility (since it doesn't silently ignore wrong syntax anymore) --- fail2ban/client/fail2banregex.py | 40 +++++++++++++++---------- fail2ban/client/jailreader.py | 15 +++++----- fail2ban/helpers.py | 12 ++++++-- fail2ban/tests/clientreadertestcase.py | 17 ++++++----- fail2ban/tests/fail2banregextestcase.py | 6 ++++ 5 files changed, 57 insertions(+), 33 deletions(-) diff --git a/fail2ban/client/fail2banregex.py b/fail2ban/client/fail2banregex.py index 5d5f4a1c..90e178f9 100644 --- a/fail2ban/client/fail2banregex.py +++ b/fail2ban/client/fail2banregex.py @@ -35,6 +35,7 @@ __license__ = "GPL" import getopt import logging +import re import os import shlex import sys @@ -329,26 +330,33 @@ class Fail2banRegex(object): regex = regextype + 'regex' # try to check - we've case filter?[options...]?: basedir = self._opts.config + fltName = value fltFile = None fltOpt = {} if regextype == 'fail': - fltName, fltOpt = extractOptions(value) - if fltName is not None: - if "." in fltName[~5:]: - tryNames = (fltName,) - else: - tryNames = (fltName, fltName + '.conf', fltName + '.local') - for fltFile in tryNames: - if not "/" in fltFile: - if os.path.basename(basedir) == 'filter.d': - fltFile = os.path.join(basedir, fltFile) - else: - fltFile = os.path.join(basedir, 'filter.d', fltFile) + if re.search(r'^/{0,3}[\w/_\-.]+(?:\[.*\])?$', value): + try: + fltName, fltOpt = extractOptions(value) + if "." in fltName[~5:]: + tryNames = (fltName,) else: - basedir = os.path.dirname(fltFile) - if os.path.isfile(fltFile): - break - fltFile = None + tryNames = (fltName, fltName + '.conf', fltName + '.local') + for fltFile in tryNames: + if not "/" in fltFile: + if os.path.basename(basedir) == 'filter.d': + fltFile = os.path.join(basedir, fltFile) + else: + fltFile = os.path.join(basedir, 'filter.d', fltFile) + else: + basedir = os.path.dirname(fltFile) + if os.path.isfile(fltFile): + break + fltFile = None + except Exception as e: + output("ERROR: Wrong filter name or options: %s" % (str(e),)) + output(" while parsing: %s" % (value,)) + if self._verbose: raise(e) + return False # if it is filter file: if fltFile is not None: if (basedir == self._opts.config diff --git a/fail2ban/client/jailreader.py b/fail2ban/client/jailreader.py index 1d7db0dc..0a27c644 100644 --- a/fail2ban/client/jailreader.py +++ b/fail2ban/client/jailreader.py @@ -133,9 +133,10 @@ class JailReader(ConfigReader): # Read filter flt = self.__opts["filter"] if flt: - filterName, filterOpt = extractOptions(flt) - if not filterName: - raise JailDefError("Invalid filter definition %r" % flt) + try: + filterName, filterOpt = extractOptions(flt) + except ValueError as e: + raise JailDefError("Invalid filter definition %r: %s" % (flt, e)) self.__filter = FilterReader( filterName, self.__name, filterOpt, share_config=self.share_config, basedir=self.getBaseDir()) @@ -167,10 +168,10 @@ class JailReader(ConfigReader): if not act: # skip empty actions continue # join with previous line if needed (consider possible new-line): - actName, actOpt = extractOptions(act) - prevln = '' - if not actName: - raise JailDefError("Invalid action definition %r" % act) + try: + actName, actOpt = extractOptions(act) + except ValueError as e: + raise JailDefError("Invalid action definition %r: %s" % (act, e)) if actName.endswith(".py"): self.__actions.append([ "set", diff --git a/fail2ban/helpers.py b/fail2ban/helpers.py index c45be849..5c1750a6 100644 --- a/fail2ban/helpers.py +++ b/fail2ban/helpers.py @@ -371,7 +371,7 @@ OPTION_CRE = re.compile(r"^([^\[]+)(?:\[(.*)\])?\s*$", re.DOTALL) # since v0.10 separator extended with `]\s*[` for support of multiple option groups, syntax # `action = act[p1=...][p2=...]` OPTION_EXTRACT_CRE = re.compile( - r'([\w\-_\.]+)=(?:"([^"]*)"|\'([^\']*)\'|([^,\]]*))(?:,|\]\s*\[|$)', re.DOTALL) + r'\s*([\w\-_\.]+)=(?:"([^"]*)"|\'([^\']*)\'|([^,\]]*))(?:,|\]\s*\[|$|(?P.+))|,?\s*$|(?P.+)', re.DOTALL) # split by new-line considering possible new-lines within options [...]: OPTION_SPLIT_CRE = re.compile( r'(?:[^\[\s]+(?:\s*\[\s*(?:[\w\-_\.]+=(?:"[^"]*"|\'[^\']*\'|[^,\]]*)\s*(?:,|\]\s*\[)?\s*)*\])?\s*|\S+)(?=\n\s*|\s+|$)', re.DOTALL) @@ -379,13 +379,19 @@ OPTION_SPLIT_CRE = re.compile( def extractOptions(option): match = OPTION_CRE.match(option) if not match: - # TODO proper error handling - return None, None + raise ValueError("unexpected option syntax") option_name, optstr = match.groups() option_opts = dict() if optstr: for optmatch in OPTION_EXTRACT_CRE.finditer(optstr): + if optmatch.group("wrngA"): + raise ValueError("unexpected syntax at %d after option %r: %s" % ( + optmatch.start("wrngA"), optmatch.group(1), optmatch.group("wrngA")[0:25])) + if optmatch.group("wrngB"): + raise ValueError("expected option, wrong syntax at %d: %s" % ( + optmatch.start("wrngB"), optmatch.group("wrngB")[0:25])) opt = optmatch.group(1) + if not opt: continue value = [ val for val in optmatch.group(2,3,4) if val is not None][0] option_opts[opt.strip()] = value.strip() diff --git a/fail2ban/tests/clientreadertestcase.py b/fail2ban/tests/clientreadertestcase.py index 54850bca..e92edd48 100644 --- a/fail2ban/tests/clientreadertestcase.py +++ b/fail2ban/tests/clientreadertestcase.py @@ -381,13 +381,16 @@ class JailReaderTest(LogCaptureTestCase): self.assertEqual(('mail.who_is', {'a':'cat', 'b':'dog'}), extractOptions("mail.who_is[a=cat,b=dog]")) self.assertEqual(('mail--ho_is', {}), extractOptions("mail--ho_is")) - self.assertEqual(('mail--ho_is', {}), extractOptions("mail--ho_is['s']")) - #print(self.getLog()) - #self.assertLogged("Invalid argument ['s'] in ''s''") - self.assertEqual(('mail', {'a': ','}), extractOptions("mail[a=',']")) + self.assertEqual(('mail', {'a': 'b'}), extractOptions("mail[a=b, ]")) - #self.assertRaises(ValueError, extractOptions ,'mail-how[') + self.assertRaises(ValueError, extractOptions ,'mail-how[') + + self.assertRaises(ValueError, extractOptions, """mail[a="test with interim (wrong) "" quotes"]""") + self.assertRaises(ValueError, extractOptions, """mail[a='test with interim (wrong) '' quotes']""") + self.assertRaises(ValueError, extractOptions, """mail[a='x, y, z', b=x, y, z]""") + + self.assertRaises(ValueError, extractOptions, """mail['s']""") # Empty option option = "abc[]" @@ -752,9 +755,9 @@ class JailsReaderTest(LogCaptureTestCase): ['add', 'tz_correct', 'auto'], ['start', 'tz_correct'], ['config-error', - "Jail 'brokenactiondef' skipped, because of wrong configuration: Invalid action definition 'joho[foo'"], + "Jail 'brokenactiondef' skipped, because of wrong configuration: Invalid action definition 'joho[foo': unexpected option syntax"], ['config-error', - "Jail 'brokenfilterdef' skipped, because of wrong configuration: Invalid filter definition 'flt[test'"], + "Jail 'brokenfilterdef' skipped, because of wrong configuration: Invalid filter definition 'flt[test': unexpected option syntax"], ['config-error', "Jail 'missingaction' skipped, because of wrong configuration: Unable to read action 'noactionfileforthisaction'"], ['config-error', diff --git a/fail2ban/tests/fail2banregextestcase.py b/fail2ban/tests/fail2banregextestcase.py index 4d878a24..85fe4f15 100644 --- a/fail2ban/tests/fail2banregextestcase.py +++ b/fail2ban/tests/fail2banregextestcase.py @@ -141,6 +141,12 @@ class Fail2banRegexTest(LogCaptureTestCase): )) self.assertLogged("Unable to compile regular expression") + def testWrongFilterOptions(self): + self.assertFalse(_test_exec( + "test", "flt[a='x,y,z',b=z,y,x]" + )) + self.assertLogged("Wrong filter name or options", "wrong syntax at 14: y,x", all=True) + def testDirectFound(self): self.assertTrue(_test_exec( "--datepattern", r"^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?", From d678440658541a3c02c8d57695bd363f72147724 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Thu, 11 Feb 2021 18:32:32 +0100 Subject: [PATCH 058/240] more precise RE (avoids weakness with catch-all's and is injection safe) --- config/filter.d/drupal-auth.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/filter.d/drupal-auth.conf b/config/filter.d/drupal-auth.conf index 2d4cbe9f..2404cc6d 100644 --- a/config/filter.d/drupal-auth.conf +++ b/config/filter.d/drupal-auth.conf @@ -14,7 +14,7 @@ before = common.conf [Definition] -failregex = ^%(__prefix_line)s(https?:\/\/)([\da-z\.-]+)\.([a-z\.]{2,6})(\/[\w\.-]+)*\|\d{10}\|user\|\|.+\|.*\|\d\|.*\|Login attempt failed (?:for|from) .+\.$ +failregex = ^%(__prefix_line)s(?:https?:\/\/)[^|]+\|[^|]+\|[^|]+\|\|(?:[^|]*\|)*Login attempt failed (?:for|from) [^|]+\.$ ignoreregex = From f4f92aa72d4124fbdff615e14fc5bfb8d66d7b0a Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Thu, 11 Feb 2021 18:56:53 +0100 Subject: [PATCH 059/240] more tests covering different cases, injections attempt etc --- fail2ban/tests/files/logs/drupal-auth | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/fail2ban/tests/files/logs/drupal-auth b/fail2ban/tests/files/logs/drupal-auth index 5e7194d9..4d063e55 100644 --- a/fail2ban/tests/files/logs/drupal-auth +++ b/fail2ban/tests/files/logs/drupal-auth @@ -3,5 +3,15 @@ Apr 26 13:15:25 webserver example.com: https://example.com|1430068525|user|1.2.3 # failJSON: { "time": "2005-04-26T13:15:25", "match": true , "host": "1.2.3.4" } Apr 26 13:15:25 webserver example.com: https://example.com/subdir|1430068525|user|1.2.3.4|https://example.com/subdir/user|https://example.com/subdir/user|0||Login attempt failed for drupaladmin. -# failJSON: { "time": "2005-04-26T13:19:08", "match": false , "host": "1.2.3.4" } +# failJSON: { "time": "2005-04-26T13:19:08", "match": false , "host": "1.2.3.4", "user": "drupaladmin" } Apr 26 13:19:08 webserver example.com: https://example.com|1430068748|user|1.2.3.4|https://example.com/user|https://example.com/user|1||Session opened for drupaladmin. + +# failJSON: { "time": "2005-04-26T13:20:00", "match": false, "desc": "attempt to inject on URI (pipe, login failed for), not a failure, gh-2742" } +Apr 26 13:20:00 host drupal-site: https://example.com|1613063581|user|192.0.2.5|https://example.com/user/login?test=%7C&test2=%7C...|https://example.com/user/login?test=|&test2=|0||Login attempt failed for tester|2||Session revisited for drupaladmin. + +# failJSON: { "time": "2005-04-26T13:20:01", "match": true , "host": "192.0.2.7", "user": "Jack Sparrow", "desc": "log-format change - for -> from, user name with space, gh-2742" } +Apr 26 13:20:01 mweb drupal_site[24864]: https://www.example.com|1613058599|user|192.0.2.7|https://www.example.com/en/user/login|https://www.example.com/en/user/login|0||Login attempt failed from Jack Sparrow. +# failJSON: { "time": "2005-04-26T13:20:02", "match": true , "host": "192.0.2.4", "desc": "attempt to inject on URI (pipe), login failed, gh-2742" } +Apr 26 13:20:02 host drupal-site: https://example.com|1613063581|user|192.0.2.4|https://example.com/user/login?test=%7C&test2=%7C|https://example.com/user/login?test=|&test2=||0||Login attempt failed from 192.0.2.4. +# failJSON: { "time": "2005-04-26T13:20:03", "match": false, "desc": "attempt to inject on URI (pipe, login failed from), not a failure, gh-2742" } +Apr 26 13:20:03 host drupal-site: https://example.com|1613063581|user|192.0.2.5|https://example.com/user/login?test=%7C&test2=%7C...|https://example.com/user/login?test=|&test2=|0||Login attempt failed from 1.2.3.4|2||Session revisited for drupaladmin. From 8ae9208454e426aa87b96ba5df26036c4ae5cefd Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Mon, 8 Feb 2021 16:44:27 +0100 Subject: [PATCH 060/240] try to provide coverage for 3.10-alpha.5 (#2931) --- .github/workflows/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 7a1d31df..262448c2 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -22,7 +22,7 @@ jobs: runs-on: ubuntu-20.04 strategy: matrix: - python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, pypy2, pypy3] + python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, '3.10.0-alpha.5', pypy2, pypy3] fail-fast: false # Steps represent a sequence of tasks that will be executed as part of the job steps: From 2b6bb2c1bed8f7009631e8f8c306fa3160324a49 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Mon, 8 Feb 2021 17:19:24 +0100 Subject: [PATCH 061/240] follow bpo-37324: :ref:`collections-abstract-base-classes` moved to the :mod:`collections.abc` module (since 3.10-alpha.5 `MutableMapping` is missing in collections module) --- fail2ban/server/action.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/fail2ban/server/action.py b/fail2ban/server/action.py index 3bc48fe0..f0f1e6f5 100644 --- a/fail2ban/server/action.py +++ b/fail2ban/server/action.py @@ -30,7 +30,10 @@ import tempfile import threading import time from abc import ABCMeta -from collections import MutableMapping +try: + from collections.abc import MutableMapping +except ImportError: + from collections import MutableMapping from .failregex import mapTag2Opt from .ipdns import DNSUtils From 42dee38ad2ac5c3f23bdf297d824022923270dd9 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Mon, 8 Feb 2021 17:25:45 +0100 Subject: [PATCH 062/240] amend for `Mapping` --- fail2ban/server/actions.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/fail2ban/server/actions.py b/fail2ban/server/actions.py index b7b95b44..897d907c 100644 --- a/fail2ban/server/actions.py +++ b/fail2ban/server/actions.py @@ -28,7 +28,10 @@ import logging import os import sys import time -from collections import Mapping +try: + from collections.abc import Mapping +except ImportError: + from collections import Mapping try: from collections import OrderedDict except ImportError: From 9f1d1f4fbd0804695a976beb191f2c49a2739834 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Mon, 8 Feb 2021 17:35:59 +0100 Subject: [PATCH 063/240] amend for `Mapping` (jails) --- fail2ban/server/jails.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/fail2ban/server/jails.py b/fail2ban/server/jails.py index 972a8c4b..27e12ddf 100644 --- a/fail2ban/server/jails.py +++ b/fail2ban/server/jails.py @@ -22,7 +22,10 @@ __copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2013- Yaroslav Halchenko" __license__ = "GPL" from threading import Lock -from collections import Mapping +try: + from collections.abc import Mapping +except ImportError: + from collections import Mapping from ..exceptions import DuplicateJailException, UnknownJailException from .jail import Jail From abc5a4e062417139893ba1f402e09e96f8887479 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 17 Feb 2021 19:02:22 +0100 Subject: [PATCH 064/240] ChangeLog (#2742) --- ChangeLog | 1 + 1 file changed, 1 insertion(+) diff --git a/ChangeLog b/ChangeLog index 6c530871..e5e7b485 100644 --- a/ChangeLog +++ b/ChangeLog @@ -18,6 +18,7 @@ ver. 1.0.1-dev-1 (20??/??/??) - development nightly edition different from 0) in case of unsane environment. ### Fixes +* `filter.d/drupal-auth.conf` more strict regex, extended to match "Login attempt failed from" (gh-2742) ### New Features and Enhancements * `actioncheck` behavior is changed now (gh-488), so invariant check as well as restore or repair From 55d7d9e214f72bbe4f39a2d17aa004d80bfc7299 Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 22 Feb 2021 18:39:58 +0100 Subject: [PATCH 065/240] *WiP* try to solve RC on jails with too many failures without ban, gh-2945 ... --- fail2ban/server/failmanager.py | 5 +++-- fail2ban/server/filter.py | 34 +++++++++++++++++++++++------- fail2ban/server/filtergamin.py | 16 +++----------- fail2ban/server/filterpoll.py | 10 ++------- fail2ban/server/filterpyinotify.py | 15 ++++++------- fail2ban/server/filtersystemd.py | 13 ++++++------ fail2ban/tests/filtertestcase.py | 14 ++++++------ 7 files changed, 53 insertions(+), 54 deletions(-) diff --git a/fail2ban/server/failmanager.py b/fail2ban/server/failmanager.py index 4173a233..64576dbd 100644 --- a/fail2ban/server/failmanager.py +++ b/fail2ban/server/failmanager.py @@ -124,9 +124,10 @@ class FailManager: return len(self.__failList) def cleanup(self, time): + time -= self.__maxTime with self.__lock: todelete = [fid for fid,item in self.__failList.iteritems() \ - if item.getTime() + self.__maxTime <= time] + if item.getTime() <= time] if len(todelete) == len(self.__failList): # remove all: self.__failList = dict() @@ -140,7 +141,7 @@ class FailManager: else: # create new dictionary without items to be deleted: self.__failList = dict((fid,item) for fid,item in self.__failList.iteritems() \ - if item.getTime() + self.__maxTime > time) + if item.getTime() > time) self.__bgSvc.service() def delFailure(self, fid): diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index 4e947d27..0f4e9e5b 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -93,6 +93,8 @@ class Filter(JailThread): ## Store last time stamp, applicable for multi-line self.__lastTimeText = "" self.__lastDate = None + ## Next service (cleanup) time + self.__nextSvcTime = -(1<<63) ## if set, treat log lines without explicit time zone to be in this time zone self.__logtimezone = None ## Default or preferred encoding (to decode bytes from file or journal): @@ -114,10 +116,10 @@ class Filter(JailThread): self.checkFindTime = True ## shows that filter is in operation mode (processing new messages): self.inOperation = True - ## if true prevents against retarded banning in case of RC by too many failures (disabled only for test purposes): - self.banASAP = True ## Ticks counter self.ticks = 0 + ## Processed lines counter + self.procLines = 0 ## Thread name: self.name="f2b/f."+self.jailName @@ -441,12 +443,23 @@ class Filter(JailThread): def performBan(self, ip=None): """Performs a ban for IPs (or given ip) that are reached maxretry of the jail.""" - try: # pragma: no branch - exception is the only way out - while True: + while True: + try: ticket = self.failManager.toBan(ip) - self.jail.putFailTicket(ticket) - except FailManagerEmpty: - self.failManager.cleanup(MyTime.time()) + except FailManagerEmpty: + break + self.jail.putFailTicket(ticket) + if ip: break + self.performSvc() + + def performSvc(self, force=False): + """Performs a service tasks (clean failure list).""" + tm = MyTime.time() + # avoid too early clean up: + if force or tm >= self.__nextSvcTime: + self.__nextSvcTime = tm + 5 + # clean up failure list: + self.failManager.cleanup(tm) def addAttempt(self, ip, *matches): """Generate a failed attempt for ip""" @@ -694,8 +707,12 @@ class Filter(JailThread): attempts = self.failManager.addFailure(tick) # avoid RC on busy filter (too many failures) - if attempts for IP/ID reached maxretry, # we can speedup ban, so do it as soon as possible: - if self.banASAP and attempts >= self.failManager.getMaxRetry(): + if attempts >= self.failManager.getMaxRetry(): self.performBan(ip) + self.procLines += 1 + # every 100 lines check need to perform service tasks: + if self.procLines % 100 == 0: + self.performSvc() # reset (halve) error counter (successfully processed line): if self._errors: self._errors //= 2 @@ -1064,6 +1081,7 @@ class FileFilter(Filter): # is created and is added to the FailManager. def getFailures(self, filename, inOperation=None): + if self.idle: return False log = self.getLog(filename) if log is None: logSys.error("Unable to get failures in %s", filename) diff --git a/fail2ban/server/filtergamin.py b/fail2ban/server/filtergamin.py index 078246de..c5373445 100644 --- a/fail2ban/server/filtergamin.py +++ b/fail2ban/server/filtergamin.py @@ -55,7 +55,6 @@ class FilterGamin(FileFilter): def __init__(self, jail): FileFilter.__init__(self, jail) - self.__modified = False # Gamin monitor self.monitor = gamin.WatchMonitor() fd = self.monitor.get_fd() @@ -67,21 +66,9 @@ class FilterGamin(FileFilter): logSys.log(4, "Got event: " + repr(event) + " for " + path) if event in (gamin.GAMCreated, gamin.GAMChanged, gamin.GAMExists): logSys.debug("File changed: " + path) - self.__modified = True self.ticks += 1 - self._process_file(path) - - def _process_file(self, path): - """Process a given file - - TODO -- RF: - this is a common logic and must be shared/provided by FileFilter - """ self.getFailures(path) - if not self.banASAP: # pragma: no cover - self.performBan() - self.__modified = False ## # Add a log file path @@ -128,6 +115,9 @@ class FilterGamin(FileFilter): Utils.wait_for(lambda: not self.active or self._handleEvents(), self.sleeptime) self.ticks += 1 + if self.ticks % 10 == 0: + self.performSvc() + logSys.debug("[%s] filter terminated", self.jailName) return True diff --git a/fail2ban/server/filterpoll.py b/fail2ban/server/filterpoll.py index 7bbdfc5c..7ee00540 100644 --- a/fail2ban/server/filterpoll.py +++ b/fail2ban/server/filterpoll.py @@ -27,9 +27,7 @@ __license__ = "GPL" import os import time -from .failmanager import FailManagerEmpty from .filter import FileFilter -from .mytime import MyTime from .utils import Utils from ..helpers import getLogger, logging @@ -55,7 +53,6 @@ class FilterPoll(FileFilter): def __init__(self, jail): FileFilter.__init__(self, jail) - self.__modified = False ## The time of the last modification of the file. self.__prevStats = dict() self.__file404Cnt = dict() @@ -115,13 +112,10 @@ class FilterPoll(FileFilter): break for filename in modlst: self.getFailures(filename) - self.__modified = True self.ticks += 1 - if self.__modified: - if not self.banASAP: # pragma: no cover - self.performBan() - self.__modified = False + if self.ticks % 10 == 0: + self.performSvc() except Exception as e: # pragma: no cover if not self.active: # if not active - error by stop... break diff --git a/fail2ban/server/filterpyinotify.py b/fail2ban/server/filterpyinotify.py index 9796e26f..d62348a2 100644 --- a/fail2ban/server/filterpyinotify.py +++ b/fail2ban/server/filterpyinotify.py @@ -75,7 +75,6 @@ class FilterPyinotify(FileFilter): def __init__(self, jail): FileFilter.__init__(self, jail) - self.__modified = False # Pyinotify watch manager self.__monitor = pyinotify.WatchManager() self.__notifier = None @@ -140,9 +139,6 @@ class FilterPyinotify(FileFilter): """ if not self.idle: self.getFailures(path) - if not self.banASAP: # pragma: no cover - self.performBan() - self.__modified = False def _addPending(self, path, reason, isDir=False): if path not in self.__pending: @@ -352,9 +348,14 @@ class FilterPyinotify(FileFilter): if not self.active: break self.__notifier.read_events() + self.ticks += 1 + # check pending files/dirs (logrotate ready): - if not self.idle: - self._checkPending() + if self.idle: + continue + self._checkPending() + if self.ticks % 10 == 0: + self.performSvc() except Exception as e: # pragma: no cover if not self.active: # if not active - error by stop... @@ -364,8 +365,6 @@ class FilterPyinotify(FileFilter): # incr common error counter: self.commonError() - self.ticks += 1 - logSys.debug("[%s] filter exited (pyinotifier)", self.jailName) self.__notifier = None diff --git a/fail2ban/server/filtersystemd.py b/fail2ban/server/filtersystemd.py index 1b33b115..925109d1 100644 --- a/fail2ban/server/filtersystemd.py +++ b/fail2ban/server/filtersystemd.py @@ -322,13 +322,12 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover break else: break - if self.__modified: - if not self.banASAP: # pragma: no cover - self.performBan() - self.__modified = 0 - # update position in log (time and iso string): - if self.jail.database is not None: - self.jail.database.updateJournal(self.jail, 'systemd-journal', line[1], line[0][1]) + self.__modified = 0 + if self.ticks % 10 == 0: + self.performSvc() + # update position in log (time and iso string): + if self.jail.database is not None: + self.jail.database.updateJournal(self.jail, 'systemd-journal', line[1], line[0][1]) except Exception as e: # pragma: no cover if not self.active: # if not active - error by stop... break diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 2dac91d1..15882ea0 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -800,7 +800,6 @@ class LogFileMonitor(LogCaptureTestCase): _, self.name = tempfile.mkstemp('fail2ban', 'monitorfailures') self.file = open(self.name, 'a') self.filter = FilterPoll(DummyJail()) - self.filter.banASAP = False # avoid immediate ban in this tests self.filter.addLogPath(self.name, autoSeek=False) self.filter.active = True self.filter.addFailRegex(r"(?:(?:Authentication failure|Failed [-/\w+]+) for(?: [iI](?:llegal|nvalid) user)?|[Ii](?:llegal|nvalid) user|ROOT LOGIN REFUSED) .*(?: from|FROM) ") @@ -952,15 +951,18 @@ class LogFileMonitor(LogCaptureTestCase): self.file.close() self.file = _copy_lines_between_files(GetFailures.FILENAME_01, self.name, n=14, mode='w') + print('=========='*10) self.filter.getFailures(self.name) + print('=========='*10) self.assertRaises(FailManagerEmpty, self.filter.failManager.toBan) self.assertEqual(self.filter.failManager.getFailTotal(), 2) # move aside, but leaving the handle still open... + print('=========='*10) os.rename(self.name, self.name + '.bak') _copy_lines_between_files(GetFailures.FILENAME_01, self.name, skip=14, n=1).close() self.filter.getFailures(self.name) - _assert_correct_last_attempt(self, self.filter, GetFailures.FAILURES_01) + #_assert_correct_last_attempt(self, self.filter, GetFailures.FAILURES_01) self.assertEqual(self.filter.failManager.getFailTotal(), 3) @@ -1018,7 +1020,6 @@ def get_monitor_failures_testcase(Filter_): self.file = open(self.name, 'a') self.jail = DummyJail() self.filter = Filter_(self.jail) - self.filter.banASAP = False # avoid immediate ban in this tests self.filter.addLogPath(self.name, autoSeek=False) # speedup search using exact date pattern: self.filter.setDatePattern(r'^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?') @@ -1277,14 +1278,14 @@ def get_monitor_failures_testcase(Filter_): # tail written before, so let's not copy anything yet #_copy_lines_between_files(GetFailures.FILENAME_01, self.name, n=100) # we should detect the failures - self.assert_correct_last_attempt(GetFailures.FAILURES_01, count=6) # was needed if we write twice above + self.assert_correct_last_attempt(GetFailures.FAILURES_01, count=3) # was needed if we write twice above # now copy and get even more _copy_lines_between_files(GetFailures.FILENAME_01, self.file, skip=12, n=3) # check for 3 failures (not 9), because 6 already get above... self.assert_correct_last_attempt(GetFailures.FAILURES_01) # total count in this test: - self.assertEqual(self.filter.failManager.getFailTotal(), 12) + self.assertEqual(self.filter.failManager.getFailTotal(), 9) cls = MonitorFailures cls.__qualname__ = cls.__name__ = "MonitorFailures<%s>(%s)" \ @@ -1316,7 +1317,6 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover def _initFilter(self, **kwargs): self._getRuntimeJournal() # check journal available self.filter = Filter_(self.jail, **kwargs) - self.filter.banASAP = False # avoid immediate ban in this tests self.filter.addJournalMatch([ "SYSLOG_IDENTIFIER=fail2ban-testcases", "TEST_FIELD=1", @@ -1570,7 +1570,6 @@ class GetFailures(LogCaptureTestCase): setUpMyTime() self.jail = DummyJail() self.filter = FileFilter(self.jail) - self.filter.banASAP = False # avoid immediate ban in this tests self.filter.active = True # speedup search using exact date pattern: self.filter.setDatePattern(r'^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?') @@ -1771,7 +1770,6 @@ class GetFailures(LogCaptureTestCase): self.pruneLog("[test-phase useDns=%s]" % useDns) jail = DummyJail() filter_ = FileFilter(jail, useDns=useDns) - filter_.banASAP = False # avoid immediate ban in this tests filter_.active = True filter_.failManager.setMaxRetry(1) # we might have just few failures From e353fb802442309d0b6fbfe86cf6d0ab286c6626 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 23 Feb 2021 02:46:44 +0100 Subject: [PATCH 066/240] fixed test cases (ban ASAP also followed in test suite now, so failure reached maxretry causes immediate ban now) --- fail2ban/tests/filtertestcase.py | 120 +++++++++++++++++-------------- 1 file changed, 66 insertions(+), 54 deletions(-) diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 15882ea0..fe37ea29 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -164,18 +164,25 @@ def _assert_correct_last_attempt(utest, filter_, output, count=None): # get fail ticket from jail found.append(_ticket_tuple(filter_.getFailTicket())) else: - # when we are testing without jails - # wait for failures (up to max time) - Utils.wait_for( - lambda: filter_.failManager.getFailCount() >= (tickcount, failcount), - _maxWaitTime(10)) - # get fail ticket(s) from filter - while tickcount: - try: - found.append(_ticket_tuple(filter_.failManager.toBan())) - except FailManagerEmpty: - break - tickcount -= 1 + # when we are testing without jails wait for failures (up to max time) + if filter_.jail: + while True: + t = filter_.jail.getFailTicket() + if not t: break + found.append(_ticket_tuple(t)) + if found: + tickcount -= len(found) + if tickcount > 0: + Utils.wait_for( + lambda: filter_.failManager.getFailCount() >= (tickcount, failcount), + _maxWaitTime(10)) + # get fail ticket(s) from filter + while tickcount: + try: + found.append(_ticket_tuple(filter_.failManager.toBan())) + except FailManagerEmpty: + break + tickcount -= 1 if not isinstance(output[0], (tuple,list)): utest.assertEqual(len(found), 1) @@ -951,14 +958,11 @@ class LogFileMonitor(LogCaptureTestCase): self.file.close() self.file = _copy_lines_between_files(GetFailures.FILENAME_01, self.name, n=14, mode='w') - print('=========='*10) self.filter.getFailures(self.name) - print('=========='*10) self.assertRaises(FailManagerEmpty, self.filter.failManager.toBan) self.assertEqual(self.filter.failManager.getFailTotal(), 2) # move aside, but leaving the handle still open... - print('=========='*10) os.rename(self.name, self.name + '.bak') _copy_lines_between_files(GetFailures.FILENAME_01, self.name, skip=14, n=1).close() self.filter.getFailures(self.name) @@ -1112,12 +1116,13 @@ def get_monitor_failures_testcase(Filter_): skip=12, n=3, mode='w') self.assert_correct_last_attempt(GetFailures.FAILURES_01) - def _wait4failures(self, count=2): + def _wait4failures(self, count=2, waitEmpty=True): # Poll might need more time - self.assertTrue(self.isEmpty(_maxWaitTime(5)), - "Queue must be empty but it is not: %s." - % (', '.join([str(x) for x in self.jail.queue]))) - self.assertRaises(FailManagerEmpty, self.filter.failManager.toBan) + if waitEmpty: + self.assertTrue(self.isEmpty(_maxWaitTime(5)), + "Queue must be empty but it is not: %s." + % (', '.join([str(x) for x in self.jail.queue]))) + self.assertRaises(FailManagerEmpty, self.filter.failManager.toBan) Utils.wait_for(lambda: self.filter.failManager.getFailTotal() >= count, _maxWaitTime(10)) self.assertEqual(self.filter.failManager.getFailTotal(), count) @@ -1283,9 +1288,9 @@ def get_monitor_failures_testcase(Filter_): # now copy and get even more _copy_lines_between_files(GetFailures.FILENAME_01, self.file, skip=12, n=3) # check for 3 failures (not 9), because 6 already get above... - self.assert_correct_last_attempt(GetFailures.FAILURES_01) + self.assert_correct_last_attempt(GetFailures.FAILURES_01, count=3) # total count in this test: - self.assertEqual(self.filter.failManager.getFailTotal(), 9) + self._wait4failures(12, False) cls = MonitorFailures cls.__qualname__ = cls.__name__ = "MonitorFailures<%s>(%s)" \ @@ -1640,6 +1645,7 @@ class GetFailures(LogCaptureTestCase): [u'Aug 14 11:%d:59 i60p295 sshd[12365]: Failed publickey for roehl from ::ffff:141.3.81.106 port 51332 ssh2' % m for m in 53, 54, 57, 58]) + self.filter.setMaxRetry(4) self.filter.addLogPath(GetFailures.FILENAME_02, autoSeek=0) self.filter.addFailRegex(r"Failed .* from ") self.filter.getFailures(GetFailures.FILENAME_02) @@ -1648,6 +1654,7 @@ class GetFailures(LogCaptureTestCase): def testGetFailures03(self): output = ('203.162.223.135', 6, 1124013600.0) + self.filter.setMaxRetry(6) self.filter.addLogPath(GetFailures.FILENAME_03, autoSeek=0) self.filter.addFailRegex(r"error,relay=,.*550 User unknown") self.filter.getFailures(GetFailures.FILENAME_03) @@ -1656,6 +1663,7 @@ class GetFailures(LogCaptureTestCase): def testGetFailures03_InOperation(self): output = ('203.162.223.135', 9, 1124013600.0) + self.filter.setMaxRetry(9) self.filter.addLogPath(GetFailures.FILENAME_03, autoSeek=0) self.filter.addFailRegex(r"error,relay=,.*550 User unknown") self.filter.getFailures(GetFailures.FILENAME_03, inOperation=True) @@ -1673,7 +1681,7 @@ class GetFailures(LogCaptureTestCase): def testGetFailures03_Seek2(self): # same test as above but with seek to 'Aug 14 11:59:04' - so other output ... output = ('203.162.223.135', 2, 1124013600.0) - self.filter.setMaxRetry(1) + self.filter.setMaxRetry(2) self.filter.addLogPath(GetFailures.FILENAME_03, autoSeek=output[2]) self.filter.addFailRegex(r"error,relay=,.*550 User unknown") @@ -1683,10 +1691,12 @@ class GetFailures(LogCaptureTestCase): def testGetFailures04(self): # because of not exact time in testcase04.log (no year), we should always use our test time: self.assertEqual(MyTime.time(), 1124013600) - # should find exact 4 failures for *.186 and 2 failures for *.185 - output = (('212.41.96.186', 4, 1124013600.0), - ('212.41.96.185', 2, 1124013598.0)) - + # should find exact 4 failures for *.186 and 2 failures for *.185, but maxretry is 2, so 3 tickets: + output = ( + ('212.41.96.186', 2, 1124013480.0), + ('212.41.96.186', 2, 1124013600.0), + ('212.41.96.185', 2, 1124013598.0) + ) # speedup search using exact date pattern: self.filter.setDatePattern((r'^%ExY(?P<_sep>[-/.])%m(?P=_sep)%d[T ]%H:%M:%S(?:[.,]%f)?(?:\s*%z)?', r'^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?', @@ -1743,9 +1753,11 @@ class GetFailures(LogCaptureTestCase): unittest.F2B.SkipIfNoNetwork() # We should still catch failures with usedns = no ;-) output_yes = ( - ('93.184.216.34', 2, 1124013539.0, - [u'Aug 14 11:54:59 i60p295 sshd[12365]: Failed publickey for roehl from example.com port 51332 ssh2', - u'Aug 14 11:58:59 i60p295 sshd[12365]: Failed publickey for roehl from ::ffff:93.184.216.34 port 51332 ssh2'] + ('93.184.216.34', 1, 1124013299.0, + [u'Aug 14 11:54:59 i60p295 sshd[12365]: Failed publickey for roehl from example.com port 51332 ssh2'] + ), + ('93.184.216.34', 1, 1124013539.0, + [u'Aug 14 11:58:59 i60p295 sshd[12365]: Failed publickey for roehl from ::ffff:93.184.216.34 port 51332 ssh2'] ), ('2606:2800:220:1:248:1893:25c8:1946', 1, 1124013299.0, [u'Aug 14 11:54:59 i60p295 sshd[12365]: Failed publickey for roehl from example.com port 51332 ssh2'] @@ -1779,8 +1791,11 @@ class GetFailures(LogCaptureTestCase): _assert_correct_last_attempt(self, filter_, output) def testGetFailuresMultiRegex(self): - output = ('141.3.81.106', 8, 1124013541.0) + output = [ + ('141.3.81.106', 8, 1124013541.0) + ] + self.filter.setMaxRetry(8) self.filter.addLogPath(GetFailures.FILENAME_02, autoSeek=False) self.filter.addFailRegex(r"Failed .* from ") self.filter.addFailRegex(r"Accepted .* from ") @@ -1798,26 +1813,25 @@ class GetFailures(LogCaptureTestCase): self.assertRaises(FailManagerEmpty, self.filter.failManager.toBan) def testGetFailuresMultiLine(self): - output = [("192.0.43.10", 2, 1124013599.0), - ("192.0.43.11", 1, 1124013598.0)] + output = [ + ("192.0.43.10", 1, 1124013598.0), + ("192.0.43.10", 1, 1124013599.0), + ("192.0.43.11", 1, 1124013598.0) + ] self.filter.addLogPath(GetFailures.FILENAME_MULTILINE, autoSeek=False) self.filter.setMaxLines(100) self.filter.addFailRegex(r"^.*rsyncd\[(?P\d+)\]: connect from .+ \(\)$^.+ rsyncd\[(?P=pid)\]: rsync error: .*$") self.filter.setMaxRetry(1) self.filter.getFailures(GetFailures.FILENAME_MULTILINE) - - foundList = [] - while True: - try: - foundList.append( - _ticket_tuple(self.filter.failManager.toBan())[0:3]) - except FailManagerEmpty: - break - self.assertSortedEqual(foundList, output) + + _assert_correct_last_attempt(self, self.filter, output) def testGetFailuresMultiLineIgnoreRegex(self): - output = [("192.0.43.10", 2, 1124013599.0)] + output = [ + ("192.0.43.10", 1, 1124013598.0), + ("192.0.43.10", 1, 1124013599.0) + ] self.filter.addLogPath(GetFailures.FILENAME_MULTILINE, autoSeek=False) self.filter.setMaxLines(100) self.filter.addFailRegex(r"^.*rsyncd\[(?P\d+)\]: connect from .+ \(\)$^.+ rsyncd\[(?P=pid)\]: rsync error: .*$") @@ -1826,14 +1840,17 @@ class GetFailures(LogCaptureTestCase): self.filter.getFailures(GetFailures.FILENAME_MULTILINE) - _assert_correct_last_attempt(self, self.filter, output.pop()) + _assert_correct_last_attempt(self, self.filter, output) self.assertRaises(FailManagerEmpty, self.filter.failManager.toBan) def testGetFailuresMultiLineMultiRegex(self): - output = [("192.0.43.10", 2, 1124013599.0), + output = [ + ("192.0.43.10", 1, 1124013598.0), + ("192.0.43.10", 1, 1124013599.0), ("192.0.43.11", 1, 1124013598.0), - ("192.0.43.15", 1, 1124013598.0)] + ("192.0.43.15", 1, 1124013598.0) + ] self.filter.addLogPath(GetFailures.FILENAME_MULTILINE, autoSeek=False) self.filter.setMaxLines(100) self.filter.addFailRegex(r"^.*rsyncd\[(?P\d+)\]: connect from .+ \(\)$^.+ rsyncd\[(?P=pid)\]: rsync error: .*$") @@ -1842,14 +1859,9 @@ class GetFailures(LogCaptureTestCase): self.filter.getFailures(GetFailures.FILENAME_MULTILINE) - foundList = [] - while True: - try: - foundList.append( - _ticket_tuple(self.filter.failManager.toBan())[0:3]) - except FailManagerEmpty: - break - self.assertSortedEqual(foundList, output) + _assert_correct_last_attempt(self, self.filter, output) + + self.assertRaises(FailManagerEmpty, self.filter.failManager.toBan) class DNSUtilsTests(unittest.TestCase): From 92a224217496fe3114fc7ee9f80708c00804ec03 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 23 Feb 2021 15:54:48 +0100 Subject: [PATCH 067/240] amend fixing journal tests (systemd backend only) --- fail2ban/tests/filtertestcase.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index fe37ea29..b9b7e8aa 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -1517,7 +1517,7 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover "SYSLOG_IDENTIFIER=fail2ban-testcases", "TEST_FIELD=1", "TEST_UUID=%s" % self.test_uuid]) - self.assert_correct_ban("193.168.0.128", 4) + self.assert_correct_ban("193.168.0.128", 3) _copy_lines_to_journal( self.test_file, self.journal_fields, n=6, skip=10) # we should detect the failures @@ -1531,7 +1531,7 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover self.test_file, self.journal_fields, skip=15, n=4) self.waitForTicks(1) self.assertTrue(self.isFilled(10)) - self.assert_correct_ban("87.142.124.10", 4) + self.assert_correct_ban("87.142.124.10", 3) # Add direct utf, unicode, blob: for l in ( "error: PAM: Authentication failure for \xe4\xf6\xfc\xdf from 192.0.2.1", From 6f4b6ec8ccdb68c75aec8225d8fa2b03ed19f320 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 24 Feb 2021 13:05:04 +0100 Subject: [PATCH 068/240] action.d/badips.* removed (badips.com is no longer active, gh-2889) --- MANIFEST | 3 - config/action.d/badips.conf | 19 -- config/action.d/badips.py | 391 ------------------------- config/jail.conf | 14 - fail2ban/tests/action_d/test_badips.py | 157 ---------- fail2ban/tests/clientreadertestcase.py | 10 +- 6 files changed, 3 insertions(+), 591 deletions(-) delete mode 100644 config/action.d/badips.conf delete mode 100644 config/action.d/badips.py delete mode 100644 fail2ban/tests/action_d/test_badips.py diff --git a/MANIFEST b/MANIFEST index 50f308db..efe87085 100644 --- a/MANIFEST +++ b/MANIFEST @@ -5,8 +5,6 @@ bin/fail2ban-testcases ChangeLog config/action.d/abuseipdb.conf config/action.d/apf.conf -config/action.d/badips.conf -config/action.d/badips.py config/action.d/blocklist_de.conf config/action.d/bsd-ipfw.conf config/action.d/cloudflare.conf @@ -219,7 +217,6 @@ fail2ban/setup.py fail2ban-testcases-all fail2ban-testcases-all-python3 fail2ban/tests/action_d/__init__.py -fail2ban/tests/action_d/test_badips.py fail2ban/tests/action_d/test_smtp.py fail2ban/tests/actionstestcase.py fail2ban/tests/actiontestcase.py diff --git a/config/action.d/badips.conf b/config/action.d/badips.conf deleted file mode 100644 index 6f9513f6..00000000 --- a/config/action.d/badips.conf +++ /dev/null @@ -1,19 +0,0 @@ -# Fail2ban reporting to badips.com -# -# Note: This reports an IP only and does not actually ban traffic. Use -# another action in the same jail if you want bans to occur. -# -# Set the category to the appropriate value before use. -# -# To get see register and optional key to get personalised graphs see: -# http://www.badips.com/blog/personalized-statistics-track-the-attackers-of-all-your-servers-with-one-key - -[Definition] - -actionban = curl --fail --user-agent "" http://www.badips.com/add// - -[Init] - -# Option: category -# Notes.: Values are from the list here: http://www.badips.com/get/categories -category = diff --git a/config/action.d/badips.py b/config/action.d/badips.py deleted file mode 100644 index 805120e9..00000000 --- a/config/action.d/badips.py +++ /dev/null @@ -1,391 +0,0 @@ -# emacs: -*- mode: python; py-indent-offset: 4; indent-tabs-mode: t -*- -# vi: set ft=python sts=4 ts=4 sw=4 noet : - -# This file is part of Fail2Ban. -# -# Fail2Ban is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# Fail2Ban is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Fail2Ban; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - -import sys -if sys.version_info < (2, 7): # pragma: no cover - raise ImportError("badips.py action requires Python >= 2.7") -import json -import threading -import logging -if sys.version_info >= (3, ): # pragma: 2.x no cover - from urllib.request import Request, urlopen - from urllib.parse import urlencode - from urllib.error import HTTPError -else: # pragma: 3.x no cover - from urllib2 import Request, urlopen, HTTPError - from urllib import urlencode - -from fail2ban.server.actions import Actions, ActionBase, BanTicket -from fail2ban.helpers import splitwords, str2LogLevel - - - -class BadIPsAction(ActionBase): # pragma: no cover - may be unavailable - """Fail2Ban action which reports bans to badips.com, and also - blacklist bad IPs listed on badips.com by using another action's - ban method. - - Parameters - ---------- - jail : Jail - The jail which the action belongs to. - name : str - Name assigned to the action. - category : str - Valid badips.com category for reporting failures. - score : int, optional - Minimum score for bad IPs. Default 3. - age : str, optional - Age of last report for bad IPs, per badips.com syntax. - Default "24h" (24 hours) - banaction : str, optional - Name of banaction to use for blacklisting bad IPs. If `None`, - no blacklist of IPs will take place. - Default `None`. - bancategory : str, optional - Name of category to use for blacklisting, which can differ - from category used for reporting. e.g. may want to report - "postfix", but want to use whole "mail" category for blacklist. - Default `category`. - bankey : str, optional - Key issued by badips.com to retrieve personal list - of blacklist IPs. - updateperiod : int, optional - Time in seconds between updating bad IPs blacklist. - Default 900 (15 minutes) - loglevel : int/str, optional - Log level of the message when an IP is (un)banned. - Default `DEBUG`. - Can be also supplied as two-value list (comma- or space separated) to - provide level of the summary message when a group of IPs is (un)banned. - Example `DEBUG,INFO`. - agent : str, optional - User agent transmitted to server. - Default `Fail2Ban/ver.` - - Raises - ------ - ValueError - If invalid `category`, `score`, `banaction` or `updateperiod`. - """ - - TIMEOUT = 10 - _badips = "https://www.badips.com" - def _Request(self, url, **argv): - return Request(url, headers={'User-Agent': self.agent}, **argv) - - def __init__(self, jail, name, category, score=3, age="24h", - banaction=None, bancategory=None, bankey=None, updateperiod=900, - loglevel='DEBUG', agent="Fail2Ban", timeout=TIMEOUT): - super(BadIPsAction, self).__init__(jail, name) - - self.timeout = timeout - self.agent = agent - self.category = category - self.score = score - self.age = age - self.banaction = banaction - self.bancategory = bancategory or category - self.bankey = bankey - loglevel = splitwords(loglevel) - self.sumloglevel = str2LogLevel(loglevel[-1]) - self.loglevel = str2LogLevel(loglevel[0]) - self.updateperiod = updateperiod - - self._bannedips = set() - # Used later for threading.Timer for updating badips - self._timer = None - - @staticmethod - def isAvailable(timeout=1): - try: - response = urlopen(Request("/".join([BadIPsAction._badips]), - headers={'User-Agent': "Fail2Ban"}), timeout=timeout) - return True, '' - except Exception as e: # pragma: no cover - return False, e - - def logError(self, response, what=''): # pragma: no cover - sporadical (502: Bad Gateway, etc) - messages = {} - try: - messages = json.loads(response.read().decode('utf-8')) - except: - pass - self._logSys.error( - "%s. badips.com response: '%s'", what, - messages.get('err', 'Unknown')) - - def getCategories(self, incParents=False): - """Get badips.com categories. - - Returns - ------- - set - Set of categories. - - Raises - ------ - HTTPError - Any issues with badips.com request. - ValueError - If badips.com response didn't contain necessary information - """ - try: - response = urlopen( - self._Request("/".join([self._badips, "get", "categories"])), timeout=self.timeout) - except HTTPError as response: # pragma: no cover - self.logError(response, "Failed to fetch categories") - raise - else: - response_json = json.loads(response.read().decode('utf-8')) - if not 'categories' in response_json: - err = "badips.com response lacked categories specification. Response was: %s" \ - % (response_json,) - self._logSys.error(err) - raise ValueError(err) - categories = response_json['categories'] - categories_names = set( - value['Name'] for value in categories) - if incParents: - categories_names.update(set( - value['Parent'] for value in categories - if "Parent" in value)) - return categories_names - - def getList(self, category, score, age, key=None): - """Get badips.com list of bad IPs. - - Parameters - ---------- - category : str - Valid badips.com category. - score : int - Minimum score for bad IPs. - age : str - Age of last report for bad IPs, per badips.com syntax. - key : str, optional - Key issued by badips.com to fetch IPs reported with the - associated key. - - Returns - ------- - set - Set of bad IPs. - - Raises - ------ - HTTPError - Any issues with badips.com request. - """ - try: - url = "?".join([ - "/".join([self._badips, "get", "list", category, str(score)]), - urlencode({'age': age})]) - if key: - url = "&".join([url, urlencode({'key': key})]) - self._logSys.debug('badips.com: get list, url: %r', url) - response = urlopen(self._Request(url), timeout=self.timeout) - except HTTPError as response: # pragma: no cover - self.logError(response, "Failed to fetch bad IP list") - raise - else: - return set(response.read().decode('utf-8').split()) - - @property - def category(self): - """badips.com category for reporting IPs. - """ - return self._category - - @category.setter - def category(self, category): - if category not in self.getCategories(): - self._logSys.error("Category name '%s' not valid. " - "see badips.com for list of valid categories", - category) - raise ValueError("Invalid category: %s" % category) - self._category = category - - @property - def bancategory(self): - """badips.com bancategory for fetching IPs. - """ - return self._bancategory - - @bancategory.setter - def bancategory(self, bancategory): - if bancategory != "any" and bancategory not in self.getCategories(incParents=True): - self._logSys.error("Category name '%s' not valid. " - "see badips.com for list of valid categories", - bancategory) - raise ValueError("Invalid bancategory: %s" % bancategory) - self._bancategory = bancategory - - @property - def score(self): - """badips.com minimum score for fetching IPs. - """ - return self._score - - @score.setter - def score(self, score): - score = int(score) - if 0 <= score <= 5: - self._score = score - else: - raise ValueError("Score must be 0-5") - - @property - def banaction(self): - """Jail action to use for banning/unbanning. - """ - return self._banaction - - @banaction.setter - def banaction(self, banaction): - if banaction is not None and banaction not in self._jail.actions: - self._logSys.error("Action name '%s' not in jail '%s'", - banaction, self._jail.name) - raise ValueError("Invalid banaction") - self._banaction = banaction - - @property - def updateperiod(self): - """Period in seconds between banned bad IPs will be updated. - """ - return self._updateperiod - - @updateperiod.setter - def updateperiod(self, updateperiod): - updateperiod = int(updateperiod) - if updateperiod > 0: - self._updateperiod = updateperiod - else: - raise ValueError("Update period must be integer greater than 0") - - def _banIPs(self, ips): - for ip in ips: - try: - ai = Actions.ActionInfo(BanTicket(ip), self._jail) - self._jail.actions[self.banaction].ban(ai) - except Exception as e: - self._logSys.error( - "Error banning IP %s for jail '%s' with action '%s': %s", - ip, self._jail.name, self.banaction, e, - exc_info=self._logSys.getEffectiveLevel()<=logging.DEBUG) - else: - self._bannedips.add(ip) - self._logSys.log(self.loglevel, - "Banned IP %s for jail '%s' with action '%s'", - ip, self._jail.name, self.banaction) - - def _unbanIPs(self, ips): - for ip in ips: - try: - ai = Actions.ActionInfo(BanTicket(ip), self._jail) - self._jail.actions[self.banaction].unban(ai) - except Exception as e: - self._logSys.error( - "Error unbanning IP %s for jail '%s' with action '%s': %s", - ip, self._jail.name, self.banaction, e, - exc_info=self._logSys.getEffectiveLevel()<=logging.DEBUG) - else: - self._logSys.log(self.loglevel, - "Unbanned IP %s for jail '%s' with action '%s'", - ip, self._jail.name, self.banaction) - finally: - self._bannedips.remove(ip) - - def start(self): - """If `banaction` set, blacklists bad IPs. - """ - if self.banaction is not None: - self.update() - - def update(self): - """If `banaction` set, updates blacklisted IPs. - - Queries badips.com for list of bad IPs, removing IPs from the - blacklist if no longer present, and adds new bad IPs to the - blacklist. - """ - if self.banaction is not None: - if self._timer: - self._timer.cancel() - self._timer = None - - try: - ips = self.getList( - self.bancategory, self.score, self.age, self.bankey) - # Remove old IPs no longer listed - s = self._bannedips - ips - m = len(s) - self._unbanIPs(s) - # Add new IPs which are now listed - s = ips - self._bannedips - p = len(s) - self._banIPs(s) - if m != 0 or p != 0: - self._logSys.log(self.sumloglevel, - "Updated IPs for jail '%s' (-%d/+%d)", - self._jail.name, m, p) - self._logSys.debug( - "Next update for jail '%' in %i seconds", - self._jail.name, self.updateperiod) - finally: - self._timer = threading.Timer(self.updateperiod, self.update) - self._timer.start() - - def stop(self): - """If `banaction` set, clears blacklisted IPs. - """ - if self.banaction is not None: - if self._timer: - self._timer.cancel() - self._timer = None - self._unbanIPs(self._bannedips.copy()) - - def ban(self, aInfo): - """Reports banned IP to badips.com. - - Parameters - ---------- - aInfo : dict - Dictionary which includes information in relation to - the ban. - - Raises - ------ - HTTPError - Any issues with badips.com request. - """ - try: - url = "/".join([self._badips, "add", self.category, str(aInfo['ip'])]) - self._logSys.debug('badips.com: ban, url: %r', url) - response = urlopen(self._Request(url), timeout=self.timeout) - except HTTPError as response: # pragma: no cover - self.logError(response, "Failed to ban") - raise - else: - messages = json.loads(response.read().decode('utf-8')) - self._logSys.debug( - "Response from badips.com report: '%s'", - messages['suc']) - -Action = BadIPsAction diff --git a/config/jail.conf b/config/jail.conf index ddbcf61e..be035112 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -204,20 +204,6 @@ action_cf_mwl = cloudflare[cfuser="%(cfemail)s", cftoken="%(cfapikey)s"] # action_blocklist_de = blocklist_de[email="%(sender)s", service="%(__name__)s", apikey="%(blocklist_de_apikey)s", agent="%(fail2ban_agent)s"] -# Report ban via badips.com, and use as blacklist -# -# See BadIPsAction docstring in config/action.d/badips.py for -# documentation for this action. -# -# NOTE: This action relies on banaction being present on start and therefore -# should be last action defined for a jail. -# -action_badips = badips.py[category="%(__name__)s", banaction="%(banaction)s", agent="%(fail2ban_agent)s"] -# -# Report ban via badips.com (uses action.d/badips.conf for reporting only) -# -action_badips_report = badips[category="%(__name__)s", agent="%(fail2ban_agent)s"] - # Report ban via abuseipdb.com. # # See action.d/abuseipdb.conf for usage example and details. diff --git a/fail2ban/tests/action_d/test_badips.py b/fail2ban/tests/action_d/test_badips.py deleted file mode 100644 index 013c0fdb..00000000 --- a/fail2ban/tests/action_d/test_badips.py +++ /dev/null @@ -1,157 +0,0 @@ -# emacs: -*- mode: python; py-indent-offset: 4; indent-tabs-mode: t -*- -# vi: set ft=python sts=4 ts=4 sw=4 noet : - -# This file is part of Fail2Ban. -# -# Fail2Ban is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# Fail2Ban is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Fail2Ban; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - -import os -import unittest -import sys -from functools import wraps -from socket import timeout -from ssl import SSLError - -from ..actiontestcase import CallingMap -from ..dummyjail import DummyJail -from ..servertestcase import IPAddr -from ..utils import LogCaptureTestCase, CONFIG_DIR - -if sys.version_info >= (3, ): # pragma: 2.x no cover - from urllib.error import HTTPError, URLError -else: # pragma: 3.x no cover - from urllib2 import HTTPError, URLError - -def skip_if_not_available(f): - """Helper to decorate tests to skip in case of timeout/http-errors like "502 bad gateway". - """ - @wraps(f) - def wrapper(self, *args): - try: - return f(self, *args) - except (SSLError, HTTPError, URLError, timeout) as e: # pragma: no cover - timeout/availability issues - if not isinstance(e, timeout) and 'timed out' not in str(e): - if not hasattr(e, 'code') or e.code > 200 and e.code <= 404: - raise - raise unittest.SkipTest('Skip test because of %s' % e) - return wrapper - -if sys.version_info >= (2,7): # pragma: no cover - may be unavailable - class BadIPsActionTest(LogCaptureTestCase): - - available = True, None - pythonModule = None - modAction = None - - @skip_if_not_available - def setUp(self): - """Call before every test case.""" - super(BadIPsActionTest, self).setUp() - unittest.F2B.SkipIfNoNetwork() - - self.jail = DummyJail() - - self.jail.actions.add("test") - - pythonModuleName = os.path.join(CONFIG_DIR, "action.d", "badips.py") - - # check availability (once if not alive, used shorter timeout as in test cases): - if BadIPsActionTest.available[0]: - if not BadIPsActionTest.modAction: - if not BadIPsActionTest.pythonModule: - BadIPsActionTest.pythonModule = self.jail.actions._load_python_module(pythonModuleName) - BadIPsActionTest.modAction = BadIPsActionTest.pythonModule.Action - self.jail.actions._load_python_module(pythonModuleName) - BadIPsActionTest.available = BadIPsActionTest.modAction.isAvailable(timeout=2 if unittest.F2B.fast else 30) - if not BadIPsActionTest.available[0]: - raise unittest.SkipTest('Skip test because service is not available: %s' % BadIPsActionTest.available[1]) - - self.jail.actions.add("badips", pythonModuleName, initOpts={ - 'category': "ssh", - 'banaction': "test", - 'age': "2w", - 'score': 5, - #'key': "fail2ban-test-suite", - #'bankey': "fail2ban-test-suite", - 'timeout': (3 if unittest.F2B.fast else 60), - }) - self.action = self.jail.actions["badips"] - - def tearDown(self): - """Call after every test case.""" - # Must cancel timer! - if self.action._timer: - self.action._timer.cancel() - super(BadIPsActionTest, self).tearDown() - - @skip_if_not_available - def testCategory(self): - categories = self.action.getCategories() - self.assertIn("ssh", categories) - self.assertTrue(len(categories) >= 10) - - self.assertRaises( - ValueError, setattr, self.action, "category", - "invalid-category") - - # Not valid for reporting category... - self.assertRaises( - ValueError, setattr, self.action, "category", "mail") - # but valid for blacklisting. - self.action.bancategory = "mail" - - @skip_if_not_available - def testScore(self): - self.assertRaises(ValueError, setattr, self.action, "score", -5) - self.action.score = 3 - self.action.score = "3" - - @skip_if_not_available - def testBanaction(self): - self.assertRaises( - ValueError, setattr, self.action, "banaction", - "invalid-action") - self.action.banaction = "test" - - @skip_if_not_available - def testUpdateperiod(self): - self.assertRaises( - ValueError, setattr, self.action, "updateperiod", -50) - self.assertRaises( - ValueError, setattr, self.action, "updateperiod", 0) - self.action.updateperiod = 900 - self.action.updateperiod = "900" - - @skip_if_not_available - def testStartStop(self): - self.action.start() - self.assertTrue(len(self.action._bannedips) > 10, - "%s is fewer as 10: %r" % (len(self.action._bannedips), self.action._bannedips)) - self.action.stop() - self.assertTrue(len(self.action._bannedips) == 0) - - @skip_if_not_available - def testBanIP(self): - aInfo = CallingMap({ - 'ip': IPAddr('192.0.2.1') - }) - self.action.ban(aInfo) - self.assertLogged('badips.com: ban', wait=True) - self.pruneLog() - # produce an error using wrong category/IP: - self.action._category = 'f2b-this-category-dont-available-test-suite-only' - aInfo['ip'] = '' - self.assertRaises(BadIPsActionTest.pythonModule.HTTPError, self.action.ban, aInfo) - self.assertLogged('IP is invalid', 'invalid category', wait=True, all=False) diff --git a/fail2ban/tests/clientreadertestcase.py b/fail2ban/tests/clientreadertestcase.py index e92edd48..4029c753 100644 --- a/fail2ban/tests/clientreadertestcase.py +++ b/fail2ban/tests/clientreadertestcase.py @@ -458,8 +458,6 @@ class JailReaderTest(LogCaptureTestCase): ('sender', 'f2b-test@example.com'), ('blocklist_de_apikey', 'test-key'), ('action', '%(action_blocklist_de)s\n' - '%(action_badips_report)s\n' - '%(action_badips)s\n' 'mynetwatchman[port=1234,protocol=udp,agent="%(fail2ban_agent)s"]' ), )) @@ -473,16 +471,14 @@ class JailReaderTest(LogCaptureTestCase): if len(cmd) <= 4: continue # differentiate between set and multi-set (wrop it here to single set): - if cmd[0] == 'set' and (cmd[4] == 'agent' or cmd[4].endswith('badips.py')): + if cmd[0] == 'set' and cmd[4] == 'agent': act.append(cmd) elif cmd[0] == 'multi-set': act.extend([['set'] + cmd[1:4] + o for o in cmd[4] if o[0] == 'agent']) useragent = 'Fail2Ban/%s' % version - self.assertEqual(len(act), 4) + self.assertEqual(len(act), 2) self.assertEqual(act[0], ['set', 'blocklisttest', 'action', 'blocklist_de', 'agent', useragent]) - self.assertEqual(act[1], ['set', 'blocklisttest', 'action', 'badips', 'agent', useragent]) - self.assertEqual(eval(act[2][5]).get('agent', ''), useragent) - self.assertEqual(act[3], ['set', 'blocklisttest', 'action', 'mynetwatchman', 'agent', useragent]) + self.assertEqual(act[1], ['set', 'blocklisttest', 'action', 'mynetwatchman', 'agent', useragent]) @with_tmpdir def testGlob(self, d): From 63acc862b139bb1d8b45edb0b3716044b36c7113 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 24 Feb 2021 18:21:42 +0100 Subject: [PATCH 069/240] `action.d/nginx-block-map.conf`: reload nginx only if it is running (also avoid error in nginx-errorlog, gh-2949) and better test coverage for the action --- config/action.d/nginx-block-map.conf | 11 +++++++++-- fail2ban/tests/fail2banclienttestcase.py | 4 +++- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/config/action.d/nginx-block-map.conf b/config/action.d/nginx-block-map.conf index ee702907..0de382bd 100644 --- a/config/action.d/nginx-block-map.conf +++ b/config/action.d/nginx-block-map.conf @@ -84,8 +84,15 @@ srv_cfg_path = /etc/nginx/ #srv_cmd = nginx -c %(srv_cfg_path)s/nginx.conf srv_cmd = nginx -# first test configuration is correct, hereafter send reload signal: -blck_lst_reload = %(srv_cmd)s -qt; if [ $? -eq 0 ]; then +# pid file (used to check nginx is running): +srv_pid = /run/nginx.pid + +# command used to check whether nginx is running and configuration is valid: +srv_is_running = [ -f "%(srv_pid)s" ] +srv_check_cmd = %(srv_is_running)s && %(srv_cmd)s -qt + +# first test nginx is running and configuration is correct, hereafter send reload signal: +blck_lst_reload = %(srv_check_cmd)s; if [ $? -eq 0 ]; then %(srv_cmd)s -s reload; if [ $? -ne 0 ]; then echo 'reload failed.'; fi; fi; diff --git a/fail2ban/tests/fail2banclienttestcase.py b/fail2ban/tests/fail2banclienttestcase.py index 03b1d7ce..d1aec5ab 100644 --- a/fail2ban/tests/fail2banclienttestcase.py +++ b/fail2ban/tests/fail2banclienttestcase.py @@ -1281,7 +1281,7 @@ class Fail2banServerTest(Fail2banClientServerBase): 'backend = polling', 'usedns = no', 'logpath = %(tmp)s/blck-failures.log', - 'action = nginx-block-map[blck_lst_reload="", blck_lst_file="%(tmp)s/blck-lst.map"]', + 'action = nginx-block-map[srv_cmd="echo nginx", srv_pid="%(tmp)s/f2b.pid", blck_lst_file="%(tmp)s/blck-lst.map"]', ' blocklist_de[actionban=\'curl() { echo "*** curl" "$*";}; \', email="Fail2Ban ", ' 'apikey="TEST-API-KEY", agent="fail2ban-test-agent", service=]', 'filter =', @@ -1321,6 +1321,8 @@ class Fail2banServerTest(Fail2banClientServerBase): self.assertIn('\\125-000-004 1;\n', mp) self.assertIn('\\125-000-005 1;\n', mp) + # check nginx reload is logged (pid of fail2ban is used to simulate success check nginx is running): + self.assertLogged("stdout: 'nginx -qt'", "stdout: 'nginx -s reload'", all=True) # check blocklist_de substitution (e. g. new-line after ): self.assertLogged( "stdout: '*** curl --fail --data-urlencode server=Fail2Ban " From a45b1c974c969e9486326872447c33feca9ce0b2 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 2 Mar 2021 19:24:03 +0100 Subject: [PATCH 070/240] filter.d/ignorecommands/apache-fakegooglebot: added timeout parameter (default 55 seconds) - avoid fail with timeout (default 1 minute) by reverse lookup on some slow DNS services (googlebots must be resolved fast); closes gh-2951 --- .../ignorecommands/apache-fakegooglebot | 25 +++++++++++++------ fail2ban/tests/filtertestcase.py | 13 +++++----- 2 files changed, 25 insertions(+), 13 deletions(-) diff --git a/config/filter.d/ignorecommands/apache-fakegooglebot b/config/filter.d/ignorecommands/apache-fakegooglebot index b11f0d98..8351efa2 100755 --- a/config/filter.d/ignorecommands/apache-fakegooglebot +++ b/config/filter.d/ignorecommands/apache-fakegooglebot @@ -6,24 +6,35 @@ # import sys from fail2ban.server.ipdns import DNSUtils, IPAddr +from threading import Thread def process_args(argv): - if len(argv) != 2: - raise ValueError("Please provide a single IP as an argument. Got: %s\n" - % (argv[1:])) + if len(argv) - 1 not in (1, 2): + raise ValueError("Usage %s ip ?timeout?. Got: %s\n" + % (argv[0], argv[1:])) ip = argv[1] if not IPAddr(ip).isValid: raise ValueError("Argument must be a single valid IP. Got: %s\n" % ip) - return ip + return argv[1:] google_ips = None -def is_googlebot(ip): +def is_googlebot(ip, timeout=55): import re - host = DNSUtils.ipToName(ip) + timeout = float(timeout or 0) + if timeout: + def ipToNameTO(host, ip, timeout): + host[0] = DNSUtils.ipToName(ip) + host = [None] + th = Thread(target=ipToNameTO, args=(host, ip, timeout)); th.daemon=True; th.start() + th.join(timeout) + host = host[0] + else: + host = DNSUtils.ipToName(ip) + if not host or not re.match(r'.*\.google(bot)?\.com$', host): return False host_ips = DNSUtils.dnsToIp(host) @@ -31,7 +42,7 @@ def is_googlebot(ip): if __name__ == '__main__': # pragma: no cover try: - ret = is_googlebot(process_args(sys.argv)) + ret = is_googlebot(*process_args(sys.argv)) except ValueError as e: sys.stderr.write(str(e)) sys.exit(2) diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index b9b7e8aa..4f716663 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -606,13 +606,14 @@ class IgnoreIPDNS(LogCaptureTestCase): cmd = os.path.join(STOCK_CONF_DIR, "filter.d/ignorecommands/apache-fakegooglebot") ## below test direct as python module: mod = Utils.load_python_module(cmd) - self.assertFalse(mod.is_googlebot(mod.process_args([cmd, "128.178.222.69"]))) - self.assertFalse(mod.is_googlebot(mod.process_args([cmd, "192.0.2.1"]))) + self.assertFalse(mod.is_googlebot(*mod.process_args([cmd, "128.178.222.69"]))) + self.assertFalse(mod.is_googlebot(*mod.process_args([cmd, "192.0.2.1"]))) + self.assertFalse(mod.is_googlebot(*mod.process_args([cmd, "192.0.2.1", 0.1]))) bot_ips = ['66.249.66.1'] for ip in bot_ips: - self.assertTrue(mod.is_googlebot(mod.process_args([cmd, str(ip)])), "test of googlebot ip %s failed" % ip) - self.assertRaises(ValueError, lambda: mod.is_googlebot(mod.process_args([cmd]))) - self.assertRaises(ValueError, lambda: mod.is_googlebot(mod.process_args([cmd, "192.0"]))) + self.assertTrue(mod.is_googlebot(*mod.process_args([cmd, str(ip)])), "test of googlebot ip %s failed" % ip) + self.assertRaises(ValueError, lambda: mod.is_googlebot(*mod.process_args([cmd]))) + self.assertRaises(ValueError, lambda: mod.is_googlebot(*mod.process_args([cmd, "192.0"]))) ## via command: self.filter.ignoreCommand = cmd + " " for ip in bot_ips: @@ -624,7 +625,7 @@ class IgnoreIPDNS(LogCaptureTestCase): self.pruneLog() self.filter.ignoreCommand = cmd + " bad arguments " self.assertFalse(self.filter.inIgnoreIPList("192.0")) - self.assertLogged('Please provide a single IP as an argument.') + self.assertLogged('Usage') From 04aba6168c5f9b4b7b2bc4e2f4d128b62c3b7633 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 3 Mar 2021 13:02:00 +0100 Subject: [PATCH 071/240] fixed typo, `--` is not expected in options declaration, so `--dump-pretty` did never work (only `--dp` is working) --- fail2ban/client/fail2bancmdline.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fail2ban/client/fail2bancmdline.py b/fail2ban/client/fail2bancmdline.py index 03683cad..c2f6d0be 100644 --- a/fail2ban/client/fail2bancmdline.py +++ b/fail2ban/client/fail2bancmdline.py @@ -192,7 +192,7 @@ class Fail2banCmdLine(): cmdOpts = 'hc:s:p:xfbdtviqV' cmdLongOpts = ['loglevel=', 'logtarget=', 'syslogsocket=', 'test', 'async', 'conf=', 'pidfile=', 'pname=', 'socket=', - 'timeout=', 'str2sec=', 'help', 'version', 'dp', '--dump-pretty'] + 'timeout=', 'str2sec=', 'help', 'version', 'dp', 'dump-pretty'] optList, self._args = getopt.getopt(self._argv[1:], cmdOpts, cmdLongOpts) except getopt.GetoptError: self.dispUsage() From df5e024fb8819e90ee2b3435eb4f519e77c26495 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 3 Mar 2021 20:13:04 +0100 Subject: [PATCH 072/240] new issue templates --- .github/ISSUE_TEMPLATE.md | 49 ---------------- .github/ISSUE_TEMPLATE/bug_report.md | 70 +++++++++++++++++++++++ .github/ISSUE_TEMPLATE/feature_request.md | 35 ++++++++++++ .github/ISSUE_TEMPLATE/filter_request.md | 59 +++++++++++++++++++ 4 files changed, 164 insertions(+), 49 deletions(-) delete mode 100644 .github/ISSUE_TEMPLATE.md create mode 100644 .github/ISSUE_TEMPLATE/bug_report.md create mode 100644 .github/ISSUE_TEMPLATE/feature_request.md create mode 100644 .github/ISSUE_TEMPLATE/filter_request.md diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md deleted file mode 100644 index cb4b4bc6..00000000 --- a/.github/ISSUE_TEMPLATE.md +++ /dev/null @@ -1,49 +0,0 @@ -_We will be very grateful, if your problem was described as completely as possible, -enclosing excerpts from logs (if possible within DEBUG mode, if no errors evident -within INFO mode), and configuration in particular of effected relevant settings -(e.g., with ` fail2ban-client -d | grep 'affected-jail-name' ` for a particular -jail troubleshooting). -Thank you in advance for the details, because such issues like "It does not work" -alone could not help to resolve anything! -Thanks! (remove this paragraph and other comments upon reading)_ - -### Environment: - -_Fill out and check (`[x]`) the boxes which apply. If your Fail2Ban version is outdated, -and you can't verify that the issue persists in the recent release, better seek support -from the distribution you obtained Fail2Ban from_ - -- Fail2Ban version (including any possible distribution suffixes): -- OS, including release name/version: -- [ ] Fail2Ban installed via OS/distribution mechanisms -- [ ] You have not applied any additional foreign patches to the codebase -- [ ] Some customizations were done to the configuration (provide details below is so) - -### The issue: - -_Summary here_ - -#### Steps to reproduce - -#### Expected behavior - -#### Observed behavior - -#### Any additional information - -### Configuration, dump and another helpful excerpts - -#### Any customizations done to /etc/fail2ban/ configuration -``` -``` - -#### Relevant parts of /var/log/fail2ban.log file: -_preferably obtained while running fail2ban with `loglevel = 4`_ - -``` -``` - -#### Relevant lines from monitored log files in question: - -``` -``` \ No newline at end of file diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 00000000..33d94e10 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,70 @@ +--- +name: Bug report +about: Report a bug within the fail2ban engines (not filters or jails) +title: '[BR]: ' +labels: bug +assignees: '' + +--- + + + +### Environment: + + + +- Fail2Ban version : +- OS, including release name/version : +- [ ] Fail2Ban installed via OS/distribution mechanisms +- [ ] You have not applied any additional foreign patches to the codebase +- [ ] Some customizations were done to the configuration (provide details below is so) + +### The issue: + + + +#### Steps to reproduce + +#### Expected behavior + +#### Observed behavior + +#### Any additional information + + +### Configuration, dump and another helpful excerpts + +#### Any customizations done to /etc/fail2ban/ configuration + +``` +``` + +#### Relevant parts of /var/log/fail2ban.log file: + + +``` +``` + +#### Relevant lines from monitored log files: + +``` +``` diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 00000000..41812e82 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,35 @@ +--- +name: Feature request +about: Suggest an idea or an enhancement for this project +title: '[RFE]: ' +labels: enhancement +assignees: '' + +--- + + + +#### Feature request type + + +#### Description + + +#### Considered alternatives + + +#### Any additional information + diff --git a/.github/ISSUE_TEMPLATE/filter_request.md b/.github/ISSUE_TEMPLATE/filter_request.md new file mode 100644 index 00000000..eeaba5ae --- /dev/null +++ b/.github/ISSUE_TEMPLATE/filter_request.md @@ -0,0 +1,59 @@ +--- +name: Filter Request +about: Request a new jail or filter to be supported or existing filter extended with new failregex. +title: '[FR]: ' +labels: filter-request +assignees: '' + +--- + + + +### Environment: + + + +- Fail2Ban version : +- OS, including release name/version : + +#### Service, project or product which log or journal should be monitored + +- Name of filter or jail in Fail2Ban (if already exists) : +- Service, project or product name, including release name/version : +- Repository or URL (if known) : +- Service type : +- Ports and protocols the service is listening : + +#### Log or journal information + + + + +- Log file name(s) : + + + +- Journal identifier or unit name : + +#### Any additional information + + +### Relevant lines from monitored log files: + +#### failures in sense of fail2ban filter (fail2ban must match): + +``` +``` + +#### legitimate messages (fail2ban should not consider as failures): + +``` +``` From 08393f9d82d4929ce3ee09bd9f3c685d38b2428d Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Wed, 3 Mar 2021 20:28:27 +0100 Subject: [PATCH 073/240] Update filter_request.md --- .github/ISSUE_TEMPLATE/filter_request.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/filter_request.md b/.github/ISSUE_TEMPLATE/filter_request.md index eeaba5ae..caf02f90 100644 --- a/.github/ISSUE_TEMPLATE/filter_request.md +++ b/.github/ISSUE_TEMPLATE/filter_request.md @@ -1,6 +1,6 @@ --- -name: Filter Request -about: Request a new jail or filter to be supported or existing filter extended with new failregex. +name: Filter request +about: Request a new jail or filter to be supported or existing filter extended with new failregex title: '[FR]: ' labels: filter-request assignees: '' From f15ed356198728c18470794ce6d88fb786571dc4 Mon Sep 17 00:00:00 2001 From: Mike Gabriel Date: Thu, 25 Feb 2021 20:13:18 +0100 Subject: [PATCH 074/240] config/: Add support for filtering out detected port scans via scanlogd. --- config/filter.d/scanlogd.conf | 17 +++++++++++++++++ config/jail.conf | 3 +++ 2 files changed, 20 insertions(+) create mode 100644 config/filter.d/scanlogd.conf diff --git a/config/filter.d/scanlogd.conf b/config/filter.d/scanlogd.conf new file mode 100644 index 00000000..65ad63f6 --- /dev/null +++ b/config/filter.d/scanlogd.conf @@ -0,0 +1,17 @@ +# Fail2Ban filter for port scans detected by scanlogd + +[INCLUDES] + +# Read common prefixes. If any customizations available -- read them from +# common.local +before = common.conf + +[Definition] + +_daemon = scanlogd + +failregex = ^%(__prefix_line)s\ to\ [\.:0-9a-f]+\ ports\ [\ \.,0-9]+,\ f.......,\ TOS\ [0-9]+,\ TTL\ [0-9]+\ \@[0-9]{1,2}:[0-9]{2}:[0-9]{2}$ + +ignoreregex = + +# Author: Mike Gabriel diff --git a/config/jail.conf b/config/jail.conf index 28f259a0..d6d8af67 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -965,3 +965,6 @@ logpath = %(apache_error_log)s # see `filter.d/traefik-auth.conf` for details and service example. port = http,https logpath = /var/log/traefik/access.log + +[scanlogd] +logpath = %{syslog_local0} From 529866b2bb41586342652af19aaca7435ab28c28 Mon Sep 17 00:00:00 2001 From: oukb <5672797+oukb@users.noreply.github.com> Date: Mon, 8 Mar 2021 19:14:28 +0300 Subject: [PATCH 075/240] nsd.conf: fix for the current log format New nsd 4.3.5 log format: | [2021-03-05 05:25:14.562] nsd[160800]: info: axfr for example.com. from 192.35.168.32 refused, no acl matches | [2021-03-06 05:24:33.223] nsd[356033]: info: axfr for localhost. from 192.35.168.160 refused, no acl matches | [2021-03-07 05:23:26.641] nsd[547893]: info: axfr for example.com. from 192.35.168.64 refused, no acl matches | [2021-03-08 05:18:54.067] nsd[739606]: info: axfr for example.com. from 192.35.168.32 refused, no acl matches --- config/filter.d/nsd.conf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/config/filter.d/nsd.conf b/config/filter.d/nsd.conf index bfd99544..9399db23 100644 --- a/config/filter.d/nsd.conf +++ b/config/filter.d/nsd.conf @@ -23,9 +23,9 @@ _daemon = nsd # Values: TEXT failregex = ^%(__prefix_line)sinfo: ratelimit block .* query TYPE255$ - ^%(__prefix_line)sinfo: .* refused, no acl matches\.$ + ^%(__prefix_line)sinfo: .* refused, no acl matches ignoreregex = datepattern = {^LN-BEG}Epoch - {^LN-BEG} \ No newline at end of file + {^LN-BEG} From 725354c79315d8eac6cdc7ffa550a8acb9a01d73 Mon Sep 17 00:00:00 2001 From: sebres Date: Sat, 20 Mar 2021 22:33:31 +0100 Subject: [PATCH 076/240] action info extended with new members for jail info (usable as tags in command actions): `jail.found`, `jail.found_total` - current and total found failures `jail.banned`, `jail.banned_total` - current and total bans closes #10 --- fail2ban/server/actions.py | 55 ++++++++++--------- fail2ban/tests/databasetestcase.py | 8 ++- fail2ban/tests/fail2banclienttestcase.py | 13 +++-- .../tests/files/action.d/action_checkainfo.py | 3 + 4 files changed, 48 insertions(+), 31 deletions(-) diff --git a/fail2ban/server/actions.py b/fail2ban/server/actions.py index 91e1ebaf..e07ffb17 100644 --- a/fail2ban/server/actions.py +++ b/fail2ban/server/actions.py @@ -84,7 +84,7 @@ class Actions(JailThread, Mapping): self._jail = jail self._actions = OrderedDict() ## The ban manager. - self.__banManager = BanManager() + self.banManager = BanManager() self.banEpoch = 0 self.__lastConsistencyCheckTM = 0 ## Precedence of ban (over unban), so max number of tickets banned (to call an unban check): @@ -203,7 +203,7 @@ class Actions(JailThread, Mapping): def setBanTime(self, value): value = MyTime.str2seconds(value) - self.__banManager.setBanTime(value) + self.banManager.setBanTime(value) logSys.info(" banTime: %s" % value) ## @@ -212,10 +212,10 @@ class Actions(JailThread, Mapping): # @return the time def getBanTime(self): - return self.__banManager.getBanTime() + return self.banManager.getBanTime() def getBanned(self, ids): - lst = self.__banManager.getBanList() + lst = self.banManager.getBanList() if not ids: return lst if len(ids) == 1: @@ -230,7 +230,7 @@ class Actions(JailThread, Mapping): list The list of banned IP addresses. """ - return self.__banManager.getBanList(ordered=True, withTime=withTime) + return self.banManager.getBanList(ordered=True, withTime=withTime) def addBannedIP(self, ip): """Ban an IP or list of IPs.""" @@ -282,7 +282,7 @@ class Actions(JailThread, Mapping): if db and self._jail.database is not None: self._jail.database.delBan(self._jail, ip) # Find the ticket with the IP. - ticket = self.__banManager.getTicketByID(ip) + ticket = self.banManager.getTicketByID(ip) if ticket is not None: # Unban the IP. self.__unBan(ticket) @@ -291,7 +291,7 @@ class Actions(JailThread, Mapping): if not isinstance(ip, IPAddr): ipa = IPAddr(ip) if not ipa.isSingle: # subnet (mask/cidr) or raw (may be dns/hostname): - ips = filter(ipa.contains, self.__banManager.getBanList()) + ips = filter(ipa.contains, self.banManager.getBanList()) if ips: return self.removeBannedIP(ips, db, ifexists) # not found: @@ -350,7 +350,7 @@ class Actions(JailThread, Mapping): continue # wait for ban (stop if gets inactive, pending ban or unban): bancnt = 0 - wt = min(self.sleeptime, self.__banManager._nextUnbanTime - MyTime.time()) + wt = min(self.sleeptime, self.banManager._nextUnbanTime - MyTime.time()) logSys.log(5, "Actions: wait for pending tickets %s (default %s)", wt, self.sleeptime) if Utils.wait_for(lambda: not self.active or self._jail.hasFailTickets, wt): bancnt = self.__checkBan() @@ -397,7 +397,12 @@ class Actions(JailThread, Mapping): "ipfailures": lambda self: self._mi4ip(True).getAttempt(), "ipjailfailures": lambda self: self._mi4ip().getAttempt(), # raw ticket info: - "raw-ticket": lambda self: repr(self.__ticket) + "raw-ticket": lambda self: repr(self.__ticket), + # jail info: + "jail.banned": lambda self: self.__jail.actions.banManager.size(), + "jail.banned_total": lambda self: self.__jail.actions.banManager.getBanTotal(), + "jail.found": lambda self: self.__jail.filter.failManager.size(), + "jail.found_total": lambda self: self.__jail.filter.failManager.getFailTotal() } __slots__ = CallingMap.__slots__ + ('__ticket', '__jail', '__mi4ip') @@ -494,11 +499,11 @@ class Actions(JailThread, Mapping): for ticket in tickets: bTicket = BanTicket.wrap(ticket) - btime = ticket.getBanTime(self.__banManager.getBanTime()) + btime = ticket.getBanTime(self.banManager.getBanTime()) ip = bTicket.getIP() aInfo = self._getActionInfo(bTicket) reason = {} - if self.__banManager.addBanTicket(bTicket, reason=reason): + if self.banManager.addBanTicket(bTicket, reason=reason): cnt += 1 # report ticket to observer, to check time should be increased and hereafter observer writes ban to database (asynchronous) if Observers.Main is not None and not bTicket.restored: @@ -557,7 +562,7 @@ class Actions(JailThread, Mapping): # and increase ticket time if "bantime.increment" set) if cnt: logSys.debug("Banned %s / %s, %s ticket(s) in %r", cnt, - self.__banManager.getBanTotal(), self.__banManager.size(), self._jail.name) + self.banManager.getBanTotal(), self.banManager.size(), self._jail.name) return cnt def __reBan(self, ticket, actions=None, log=True): @@ -597,7 +602,7 @@ class Actions(JailThread, Mapping): def _prolongBan(self, ticket): # prevent to prolong ticket that was removed in-between, # if it in ban list - ban time already prolonged (and it stays there): - if not self.__banManager._inBanList(ticket): return + if not self.banManager._inBanList(ticket): return # do actions : aInfo = None for name, action in self._actions.iteritems(): @@ -622,13 +627,13 @@ class Actions(JailThread, Mapping): Unban IP addresses which are outdated. """ - lst = self.__banManager.unBanList(MyTime.time(), maxCount) + lst = self.banManager.unBanList(MyTime.time(), maxCount) for ticket in lst: self.__unBan(ticket) cnt = len(lst) if cnt: logSys.debug("Unbanned %s, %s ticket(s) in %r", - cnt, self.__banManager.size(), self._jail.name) + cnt, self.banManager.size(), self._jail.name) return cnt def __flushBan(self, db=False, actions=None, stop=False): @@ -642,10 +647,10 @@ class Actions(JailThread, Mapping): log = True if actions is None: logSys.debug(" Flush ban list") - lst = self.__banManager.flushBanList() + lst = self.banManager.flushBanList() else: log = False # don't log "[jail] Unban ..." if removing actions only. - lst = iter(self.__banManager) + lst = iter(self.banManager) cnt = 0 # first we'll execute flush for actions supporting this operation: unbactions = {} @@ -682,7 +687,7 @@ class Actions(JailThread, Mapping): self.__unBan(ticket, actions=actions, log=log) cnt += 1 logSys.debug(" Unbanned %s, %s ticket(s) in %r", - cnt, self.__banManager.size(), self._jail.name) + cnt, self.banManager.size(), self._jail.name) return cnt def __unBan(self, ticket, actions=None, log=True): @@ -725,18 +730,18 @@ class Actions(JailThread, Mapping): logSys.warning("Unsupported extended jail status flavor %r. Supported: %s" % (flavor, supported_flavors)) # Always print this information (basic) if flavor != "short": - banned = self.__banManager.getBanList() + banned = self.banManager.getBanList() cnt = len(banned) else: - cnt = self.__banManager.size() + cnt = self.banManager.size() ret = [("Currently banned", cnt), - ("Total banned", self.__banManager.getBanTotal())] + ("Total banned", self.banManager.getBanTotal())] if flavor != "short": ret += [("Banned IP list", banned)] if flavor == "cymru": - cymru_info = self.__banManager.getBanListExtendedCymruInfo() + cymru_info = self.banManager.getBanListExtendedCymruInfo() ret += \ - [("Banned ASN list", self.__banManager.geBanListExtendedASN(cymru_info)), - ("Banned Country list", self.__banManager.geBanListExtendedCountry(cymru_info)), - ("Banned RIR list", self.__banManager.geBanListExtendedRIR(cymru_info))] + [("Banned ASN list", self.banManager.geBanListExtendedASN(cymru_info)), + ("Banned Country list", self.banManager.geBanListExtendedCountry(cymru_info)), + ("Banned RIR list", self.banManager.geBanListExtendedRIR(cymru_info))] return ret diff --git a/fail2ban/tests/databasetestcase.py b/fail2ban/tests/databasetestcase.py index a8e2ceae..298730ae 100644 --- a/fail2ban/tests/databasetestcase.py +++ b/fail2ban/tests/databasetestcase.py @@ -29,7 +29,7 @@ import tempfile import sqlite3 import shutil -from ..server.filter import FileContainer +from ..server.filter import FileContainer, Filter from ..server.mytime import MyTime from ..server.ticket import FailTicket from ..server.actions import Actions, Utils @@ -544,17 +544,21 @@ class DatabaseTest(LogCaptureTestCase): self.testAddJail() # Jail required self.jail.database = self.db self.db.addJail(self.jail) - actions = Actions(self.jail) + actions = self.jail.actions actions.add( "action_checkainfo", os.path.join(TEST_FILES_DIR, "action.d/action_checkainfo.py"), {}) + actions.banManager.setBanTotal(20) + self.jail._Jail__filter = flt = Filter(self.jail) + flt.failManager.setFailTotal(50) ticket = FailTicket("1.2.3.4") ticket.setAttempt(5) ticket.setMatches(['test', 'test']) self.jail.putFailTicket(ticket) actions._Actions__checkBan() self.assertLogged("ban ainfo %s, %s, %s, %s" % (True, True, True, True)) + self.assertLogged("jail info %d, %d, %d, %d" % (1, 21, 0, 50)) def testDelAndAddJail(self): self.testAddJail() # Add jail diff --git a/fail2ban/tests/fail2banclienttestcase.py b/fail2ban/tests/fail2banclienttestcase.py index 244d23b0..5d322f70 100644 --- a/fail2ban/tests/fail2banclienttestcase.py +++ b/fail2ban/tests/fail2banclienttestcase.py @@ -1410,8 +1410,9 @@ class Fail2banServerTest(Fail2banClientServerBase): 'jails': ( # default: '''test_action = dummy[actionstart_on_demand=1, init="start: %(__name__)s", target="%(tmp)s/test.txt", - actionban='; - echo ""; printf "=====\\n%%b\\n=====\\n\\n" "" >> ']''', + actionban='; echo "found: / , banned: / " + echo ""; printf "=====\\n%%b\\n=====\\n\\n" "" >> ', + actionstop='; echo "stats - found: , banned: "']''', # jail sendmail-auth: '[sendmail-auth]', 'backend = polling', @@ -1456,7 +1457,8 @@ class Fail2banServerTest(Fail2banClientServerBase): _write_file(lgfn, "w+", *smaut_msg) # wait and check it caused banned (and dump in the test-file): self.assertLogged( - "[sendmail-auth] Ban 192.0.2.1", "1 ticket(s) in 'sendmail-auth'", all=True, wait=MID_WAITTIME) + "[sendmail-auth] Ban 192.0.2.1", "stdout: 'found: 0 / 3, banned: 1 / 1'", + "1 ticket(s) in 'sendmail-auth'", all=True, wait=MID_WAITTIME) _out_file(tofn) td = _read_file(tofn) # check matches (maxmatches = 2, so only 2 & 3 available): @@ -1470,7 +1472,8 @@ class Fail2banServerTest(Fail2banClientServerBase): _write_file(lgfn, "w+", *smrej_msg) # wait and check it caused banned (and dump in the test-file): self.assertLogged( - "[sendmail-reject] Ban 192.0.2.2", "1 ticket(s) in 'sendmail-reject'", all=True, wait=MID_WAITTIME) + "[sendmail-reject] Ban 192.0.2.2", "stdout: 'found: 0 / 3, banned: 1 / 1'", + "1 ticket(s) in 'sendmail-reject'", all=True, wait=MID_WAITTIME) _out_file(tofn) td = _read_file(tofn) # check matches (no maxmatches, so all matched messages are available): @@ -1484,6 +1487,8 @@ class Fail2banServerTest(Fail2banClientServerBase): # wait a bit: self.assertLogged( "Reload finished.", + "stdout: 'stats sendmail-auth - found: 3, banned: 1'", + "stdout: 'stats sendmail-reject - found: 3, banned: 1'", "[sendmail-auth] Restore Ban 192.0.2.1", "1 ticket(s) in 'sendmail-auth'", all=True, wait=MID_WAITTIME) # check matches again - (dbmaxmatches = 1), so it should be only last match after restart: td = _read_file(tofn) diff --git a/fail2ban/tests/files/action.d/action_checkainfo.py b/fail2ban/tests/files/action.d/action_checkainfo.py index 63dd4f5b..c5eaf0f8 100644 --- a/fail2ban/tests/files/action.d/action_checkainfo.py +++ b/fail2ban/tests/files/action.d/action_checkainfo.py @@ -8,6 +8,9 @@ class TestAction(ActionBase): self._logSys.info("ban ainfo %s, %s, %s, %s", aInfo["ipmatches"] != '', aInfo["ipjailmatches"] != '', aInfo["ipfailures"] > 0, aInfo["ipjailfailures"] > 0 ) + self._logSys.info("jail info %d, %d, %d, %d", + aInfo["jail.banned"], aInfo["jail.banned_total"], aInfo["jail.found"], aInfo["jail.found_total"] + ) def unban(self, aInfo): pass From 9bdc4be6cce7398cf51d811b7d71e1999c262fb3 Mon Sep 17 00:00:00 2001 From: sebres Date: Sun, 21 Mar 2021 23:35:09 +0100 Subject: [PATCH 077/240] stability: better recognition of rotation (e. g. on hash collision, consider current size and last known position now), no hash of empty file (or not fulfilled line), etc; performance: avoid unnecessary seek to start of file and hash calculation - now it occurs only if file really rotated (ino changing or size shrinking), otherwise not earlier than in 30 seconds; avoid unneeded log-rotation in tests --- fail2ban/server/database.py | 2 +- fail2ban/server/filter.py | 109 ++++++++++++++--------- fail2ban/tests/databasetestcase.py | 7 +- fail2ban/tests/fail2banclienttestcase.py | 16 ++-- fail2ban/tests/filtertestcase.py | 19 ++-- 5 files changed, 88 insertions(+), 65 deletions(-) diff --git a/fail2ban/server/database.py b/fail2ban/server/database.py index ed736a7a..86b0ea68 100644 --- a/fail2ban/server/database.py +++ b/fail2ban/server/database.py @@ -502,7 +502,7 @@ class Fail2BanDb(object): except TypeError: firstLineMD5 = None - if not firstLineMD5 and (pos or md5): + if firstLineMD5 is None and (pos or md5 is not None): cur.execute( "INSERT OR REPLACE INTO logs(jail, path, firstlinemd5, lastfilepos) " "VALUES(?, ?, ?, ?)", (jail.name, name, md5, pos)) diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index 7ad8a462..79cde6b5 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -1155,6 +1155,8 @@ class FileFilter(Filter): if logSys.getEffectiveLevel() <= logging.DEBUG: logSys.debug("Seek to find time %s (%s), file size %s", date, MyTime.time2str(date), fs) + if not fs: + return minp = container.getPos() maxp = fs tryPos = minp @@ -1281,20 +1283,25 @@ class FileContainer: self.setEncoding(encoding) self.__tail = tail self.__handler = None + self.__pos = 0 + self.__pos4hash = 0 + self.__hash = '' + self.__hashNextTime = time.time() + 30 # Try to open the file. Raises an exception if an error occurred. handler = open(filename, 'rb') - stats = os.fstat(handler.fileno()) - self.__ino = stats.st_ino try: - firstLine = handler.readline() - # Computes the MD5 of the first line. - self.__hash = md5sum(firstLine).hexdigest() - # Start at the beginning of file if tail mode is off. - if tail: - handler.seek(0, 2) - self.__pos = handler.tell() - else: - self.__pos = 0 + stats = os.fstat(handler.fileno()) + self.__ino = stats.st_ino + if stats.st_size: + firstLine = handler.readline() + # first line available and contains new-line: + if firstLine != firstLine.rstrip('\r\n'): + # Computes the MD5 of the first line. + self.__hash = md5sum(firstLine).hexdigest() + # if tail mode scroll to the end of file + if tail: + handler.seek(0, 2) + self.__pos = handler.tell() finally: handler.close() ## shows that log is in operation mode (expecting new messages only from here): @@ -1304,6 +1311,10 @@ class FileContainer: return self.__filename def getFileSize(self): + h = self.__handler + if h is not None: + stats = os.fstat(h.fileno()) + return stats.st_size return os.path.getsize(self.__filename); def setEncoding(self, encoding): @@ -1322,38 +1333,54 @@ class FileContainer: def setPos(self, value): self.__pos = value - def open(self): - self.__handler = open(self.__filename, 'rb') - # Set the file descriptor to be FD_CLOEXEC - fd = self.__handler.fileno() - flags = fcntl.fcntl(fd, fcntl.F_GETFD) - fcntl.fcntl(fd, fcntl.F_SETFD, flags | fcntl.FD_CLOEXEC) - # Stat the file before even attempting to read it - stats = os.fstat(self.__handler.fileno()) - if not stats.st_size: - # yoh: so it is still an empty file -- nothing should be - # read from it yet - # print "D: no content -- return" - return False - firstLine = self.__handler.readline() - # Computes the MD5 of the first line. - myHash = md5sum(firstLine).hexdigest() - ## print "D: fn=%s hashes=%s/%s inos=%s/%s pos=%s rotate=%s" % ( - ## self.__filename, self.__hash, myHash, stats.st_ino, self.__ino, self.__pos, - ## self.__hash != myHash or self.__ino != stats.st_ino) - ## sys.stdout.flush() - # Compare hash and inode - if self.__hash != myHash or self.__ino != stats.st_ino: - logSys.log(logging.MSG, "Log rotation detected for %s", self.__filename) - self.__hash = myHash - self.__ino = stats.st_ino - self.__pos = 0 - # Sets the file pointer to the last position. - self.__handler.seek(self.__pos) + def open(self, forcePos=None): + h = open(self.__filename, 'rb') + try: + # Set the file descriptor to be FD_CLOEXEC + fd = h.fileno() + flags = fcntl.fcntl(fd, fcntl.F_GETFD) + fcntl.fcntl(fd, fcntl.F_SETFD, flags | fcntl.FD_CLOEXEC) + myHash = self.__hash + # Stat the file before even attempting to read it + stats = os.fstat(h.fileno()) + rotflg = stats.st_size < self.__pos or stats.st_ino != self.__ino + if rotflg or not len(myHash) or time.time() > self.__hashNextTime: + myHash = '' + firstLine = h.readline() + # Computes the MD5 of the first line (if it is complete) + if firstLine != firstLine.rstrip('\r\n'): + myHash = md5sum(firstLine).hexdigest() + self.__hashNextTime = time.time() + 30 + elif stats.st_size == self.__pos: + myHash = self.__hash + # Compare size, hash and inode + if rotflg or myHash != self.__hash: + if self.__hash != '': + logSys.log(logging.MSG, "Log rotation detected for %s, reason: %r", self.__filename, + (stats.st_size, self.__pos, stats.st_ino, self.__ino, myHash, self.__hash)) + self.__ino = stats.st_ino + self.__pos = 0 + self.__hash = myHash + # if nothing to read from file yet (empty or no new data): + if forcePos is not None: + self.__pos = forcePos + elif stats.st_size <= self.__pos: + return False + # Sets the file pointer to the last position. + h.seek(self.__pos) + # leave file open (to read content): + self.__handler = h; h = None + finally: + # close (no content or error only) + if h: + h.close(); h = None return True def seek(self, offs, endLine=True): h = self.__handler + if h is None: + self.open(offs) + h = self.__handler # seek to given position h.seek(offs, 0) # goto end of next line @@ -1394,14 +1421,12 @@ class FileContainer: self.getFileName(), self.getEncoding(), self.__handler.readline()) def close(self): - if not self.__handler is None: + if self.__handler is not None: # Saves the last position. self.__pos = self.__handler.tell() # Closes the file. self.__handler.close() self.__handler = None - ## print "D: Closed %s with pos %d" % (handler, self.__pos) - ## sys.stdout.flush() _decode_line_warn = Utils.Cache(maxCount=1000, maxTime=24*60*60); diff --git a/fail2ban/tests/databasetestcase.py b/fail2ban/tests/databasetestcase.py index 298730ae..6692b238 100644 --- a/fail2ban/tests/databasetestcase.py +++ b/fail2ban/tests/databasetestcase.py @@ -212,19 +212,20 @@ class DatabaseTest(LogCaptureTestCase): self.jail.name in self.db.getJailNames(True), "Jail not added to database") - def testAddLog(self): + def _testAddLog(self): self.testAddJail() # Jail required _, filename = tempfile.mkstemp(".log", "Fail2BanDb_") self.fileContainer = FileContainer(filename, "utf-8") - self.db.addLog(self.jail, self.fileContainer) + pos = self.db.addLog(self.jail, self.fileContainer) + self.assertTrue(pos is None); # unknown previously self.assertIn(filename, self.db.getLogPaths(self.jail)) os.remove(filename) def testUpdateLog(self): - self.testAddLog() # Add log file + self._testAddLog() # Add log file # Write some text filename = self.fileContainer.getFileName() diff --git a/fail2ban/tests/fail2banclienttestcase.py b/fail2ban/tests/fail2banclienttestcase.py index 5d322f70..0cbda94f 100644 --- a/fail2ban/tests/fail2banclienttestcase.py +++ b/fail2ban/tests/fail2banclienttestcase.py @@ -230,7 +230,7 @@ def _start_params(tmp, use_stock=False, use_stock_cfg=None, os.symlink(os.path.abspath(pjoin(STOCK_CONF_DIR, n)), pjoin(cfg, n)) if create_before_start: for n in create_before_start: - _write_file(n % {'tmp': tmp}, 'w', '') + _write_file(n % {'tmp': tmp}, 'w') # parameters (sock/pid and config, increase verbosity, set log, etc.): vvv, llev = (), "INFO" if unittest.F2B.log_level < logging.INFO: # pragma: no cover @@ -937,10 +937,8 @@ class Fail2banServerTest(Fail2banClientServerBase): "Jail 'broken-jail' skipped, because of wrong configuration", all=True) # enable both jails, 3 logs for jail1, etc... - # truncate test-log - we should not find unban/ban again by reload: self.pruneLog("[test-phase 1b]") _write_jail_cfg(actions=[1,2]) - _write_file(test1log, "w+") if unittest.F2B.log_level < logging.DEBUG: # pragma: no cover _out_file(test1log) self.execCmd(SUCCESS, startparams, "reload") @@ -1003,7 +1001,7 @@ class Fail2banServerTest(Fail2banClientServerBase): self.pruneLog("[test-phase 2b]") # write new failures: - _write_file(test2log, "w+", *( + _write_file(test2log, "a+", *( (str(int(MyTime.time())) + " error 403 from 192.0.2.2: test 2",) * 3 + (str(int(MyTime.time())) + " error 403 from 192.0.2.3: test 2",) * 3 + (str(int(MyTime.time())) + " failure 401 from 192.0.2.4: test 2",) * 3 + @@ -1062,10 +1060,6 @@ class Fail2banServerTest(Fail2banClientServerBase): self.assertEqual(self.execCmdDirect(startparams, 'get', 'test-jail1', 'banned', '192.0.2.3', '192.0.2.9')[1], [1, 0]) - # rotate logs: - _write_file(test1log, "w+") - _write_file(test2log, "w+") - # restart jail without unban all: self.pruneLog("[test-phase 2c]") self.execCmd(SUCCESS, startparams, @@ -1183,7 +1177,7 @@ class Fail2banServerTest(Fail2banClientServerBase): # now write failures again and check already banned (jail1 was alive the whole time) and new bans occurred (jail1 was alive the whole time): self.pruneLog("[test-phase 5]") - _write_file(test1log, "w+", *( + _write_file(test1log, "a+", *( (str(int(MyTime.time())) + " failure 401 from 192.0.2.1: test 5",) * 3 + (str(int(MyTime.time())) + " error 403 from 192.0.2.5: test 5",) * 3 + (str(int(MyTime.time())) + " failure 401 from 192.0.2.6: test 5",) * 3 @@ -1469,7 +1463,7 @@ class Fail2banServerTest(Fail2banClientServerBase): self.pruneLog("[test-phase sendmail-reject]") # write log: - _write_file(lgfn, "w+", *smrej_msg) + _write_file(lgfn, "a+", *smrej_msg) # wait and check it caused banned (and dump in the test-file): self.assertLogged( "[sendmail-reject] Ban 192.0.2.2", "stdout: 'found: 0 / 3, banned: 1 / 1'", @@ -1597,7 +1591,7 @@ class Fail2banServerTest(Fail2banClientServerBase): wakeObs = False _observer_wait_before_incrban(lambda: wakeObs) # write again (IP already bad): - _write_file(test1log, "w+", *( + _write_file(test1log, "a+", *( (str(int(MyTime.time())) + " failure 401 from 192.0.2.11: I'm very bad \"hacker\" `` $(echo test)",) * 2 )) # wait for ban: diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 28b2d357..799adfd3 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -195,7 +195,7 @@ def _assert_correct_last_attempt(utest, filter_, output, count=None): _assert_equal_entries(utest, f, o) -def _copy_lines_between_files(in_, fout, n=None, skip=0, mode='a', terminal_line=""): +def _copy_lines_between_files(in_, fout, n=None, skip=0, mode='a', terminal_line="", lines=None): """Copy lines from one file to another (which might be already open) Returns open fout @@ -212,9 +212,9 @@ def _copy_lines_between_files(in_, fout, n=None, skip=0, mode='a', terminal_line fin.readline() # Read i = 0 - lines = [] + if not lines: lines = [] while n is None or i < n: - l = FileContainer.decode_line(in_, 'UTF-8', fin.readline()).rstrip('\r\n') + l = fin.readline().decode('UTF-8', 'replace').rstrip('\r\n') if terminal_line is not None and l == terminal_line: break lines.append(l) @@ -222,6 +222,7 @@ def _copy_lines_between_files(in_, fout, n=None, skip=0, mode='a', terminal_line # Write: all at once and flush if isinstance(fout, str): fout = open(fout, mode) + DefLogSys.debug(' ++ write %d test lines', len(lines)) fout.write('\n'.join(lines)+'\n') fout.flush() if isinstance(in_, str): # pragma: no branch - only used with str in test cases @@ -253,7 +254,7 @@ def _copy_lines_to_journal(in_, fields={},n=None, skip=0, terminal_line=""): # p # Read/Write i = 0 while n is None or i < n: - l = FileContainer.decode_line(in_, 'UTF-8', fin.readline()).rstrip('\r\n') + l = fin.readline().decode('UTF-8', 'replace').rstrip('\r\n') if terminal_line is not None and l == terminal_line: break journal.send(MESSAGE=l.strip(), **fields) @@ -1136,13 +1137,15 @@ def get_monitor_failures_testcase(Filter_): # move aside, but leaving the handle still open... os.rename(self.name, self.name + '.bak') - _copy_lines_between_files(GetFailures.FILENAME_01, self.name, skip=14, n=1).close() + _copy_lines_between_files(GetFailures.FILENAME_01, self.name, skip=14, n=1, + lines=["Aug 14 11:59:59 [logrotate] rotation 1"]).close() self.assert_correct_last_attempt(GetFailures.FAILURES_01) self.assertEqual(self.filter.failManager.getFailTotal(), 3) # now remove the moved file _killfile(None, self.name + '.bak') - _copy_lines_between_files(GetFailures.FILENAME_01, self.name, skip=12, n=3).close() + _copy_lines_between_files(GetFailures.FILENAME_01, self.name, skip=12, n=3, + lines=["Aug 14 11:59:59 [logrotate] rotation 2"]).close() self.assert_correct_last_attempt(GetFailures.FAILURES_01) self.assertEqual(self.filter.failManager.getFailTotal(), 6) @@ -1196,7 +1199,7 @@ def get_monitor_failures_testcase(Filter_): os.rename(tmpsub1, tmpsub2 + 'a') os.mkdir(tmpsub1) self.file = _copy_lines_between_files(GetFailures.FILENAME_01, self.name, - skip=12, n=1, mode='w') + skip=12, n=1, mode='w', lines=["Aug 14 11:59:59 [logrotate] rotation 1"]) self.file.close() self._wait4failures(2) @@ -1207,7 +1210,7 @@ def get_monitor_failures_testcase(Filter_): os.mkdir(tmpsub1) self.waitForTicks(2) self.file = _copy_lines_between_files(GetFailures.FILENAME_01, self.name, - skip=12, n=1, mode='w') + skip=12, n=1, mode='w', lines=["Aug 14 11:59:59 [logrotate] rotation 2"]) self.file.close() self._wait4failures(3) From 343ccd7e8a139a4c6526682290dcbf14b4384973 Mon Sep 17 00:00:00 2001 From: sebres Date: Sun, 21 Mar 2021 23:35:38 +0100 Subject: [PATCH 078/240] small optimization --- fail2ban/server/utils.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/fail2ban/server/utils.py b/fail2ban/server/utils.py index 294d147f..8483b013 100644 --- a/fail2ban/server/utils.py +++ b/fail2ban/server/utils.py @@ -332,11 +332,9 @@ class Utils(): timeout_expr = lambda: time.time() > time0 else: timeout_expr = timeout - if not interval: - interval = Utils.DEFAULT_SLEEP_INTERVAL if timeout_expr(): break - stm = min(stm + interval, Utils.DEFAULT_SLEEP_TIME) + stm = min(stm + (interval or Utils.DEFAULT_SLEEP_INTERVAL), Utils.DEFAULT_SLEEP_TIME) time.sleep(stm) return ret From e587526ede759977b8b483deeaa644f45f97cea3 Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 22 Mar 2021 00:55:28 +0100 Subject: [PATCH 079/240] tests: add missing constraint (causing incomplete comparison in below cycle if fewer lines as expected was found) --- fail2ban/tests/filtertestcase.py | 1 + 1 file changed, 1 insertion(+) diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 4f716663..3cc17fb1 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -188,6 +188,7 @@ def _assert_correct_last_attempt(utest, filter_, output, count=None): utest.assertEqual(len(found), 1) _assert_equal_entries(utest, found[0], output, count) else: + utest.assertEqual(len(found), len(output)) # sort by string representation of ip (multiple failures with different ips): found = sorted(found, key=lambda x: str(x)) output = sorted(output, key=lambda x: str(x)) From 996920cdaa7b678afc64d1bef6a07ed857c81468 Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 22 Mar 2021 00:49:55 +0100 Subject: [PATCH 080/240] in operation mode the filter reads only complete lines (ended with new-line) now, otherwise it would wait for end of line (for its completion) --- fail2ban/server/filter.py | 38 +++++++++++++++++++++++++++++--------- 1 file changed, 29 insertions(+), 9 deletions(-) diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index 79cde6b5..845b069d 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -1131,14 +1131,14 @@ class FileFilter(Filter): while not self.idle: line = log.readline() if not self.active: break; # jail has been stopped - if not line: + if line is None: # The jail reached the bottom, simply set in operation for this log # (since we are first time at end of file, growing is only possible after modifications): log.inOperation = True break # acquire in operation from log and process: self.inOperation = inOperation if inOperation is not None else log.inOperation - self.processLineAndAdd(line.rstrip('\r\n')) + self.processLineAndAdd(line) finally: log.close() db = self.jail.database @@ -1180,8 +1180,8 @@ class FileFilter(Filter): dateTimeMatch = None nextp = None while True: - line = container.readline() - if not line: + line = container.readline(False) + if line is None: break (timeMatch, template) = self.dateDetector.matchTime(line) if timeMatch: @@ -1295,7 +1295,7 @@ class FileContainer: if stats.st_size: firstLine = handler.readline() # first line available and contains new-line: - if firstLine != firstLine.rstrip('\r\n'): + if firstLine != firstLine.rstrip(b'\r\n'): # Computes the MD5 of the first line. self.__hash = md5sum(firstLine).hexdigest() # if tail mode scroll to the end of file @@ -1348,7 +1348,7 @@ class FileContainer: myHash = '' firstLine = h.readline() # Computes the MD5 of the first line (if it is complete) - if firstLine != firstLine.rstrip('\r\n'): + if firstLine != firstLine.rstrip(b'\r\n'): myHash = md5sum(firstLine).hexdigest() self.__hashNextTime = time.time() + 30 elif stats.st_size == self.__pos: @@ -1414,15 +1414,35 @@ class FileContainer: line = line.decode(enc, 'replace') return line - def readline(self): + def readline(self, complete=True): + """Read line from file + + In opposite to pythons readline it doesn't return new-line, + so returns either the line if line is complete (and complete=True) or None + if line is not complete (and complete=True) or there is no content to read. + If line is complete (and complete is True), it also shift current known + position to begin of next line. + """ if self.__handler is None: return "" + rl = self.__handler.readline() + if rl == b'': + return None + # trim new-line here and check the line was written complete (contains a new-line): + l = rl.rstrip(b'\r\n') + if self.inOperation and complete: + if l == rl: + # not fulfilled - seek back and return: + self.__handler.seek(self.__pos, 0) + return None + # shift position (to be able to seek back above): + self.__pos += len(rl) return FileContainer.decode_line( - self.getFileName(), self.getEncoding(), self.__handler.readline()) + self.getFileName(), self.getEncoding(), l) def close(self): if self.__handler is not None: - # Saves the last position. + # Saves the last real position. self.__pos = self.__handler.tell() # Closes the file. self.__handler.close() From d135aeea16343152e47191559b17912df9373705 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 24 Mar 2021 14:12:11 +0100 Subject: [PATCH 081/240] fixes restore of original logging withing tests (`LogCaptureTestCase.tearDown`) - python 3 seemed still to log wordy after tear down (setting of log.level does not restore the level for related log objects - e. g. for logger of `fail2ban.jail` etc, so `fail2ban-testcases '(testVersion|testLongName).*servertest'` generating messages in stdout handler in testLongName) --- fail2ban/tests/servertestcase.py | 10 +++++----- fail2ban/tests/utils.py | 8 +++----- 2 files changed, 8 insertions(+), 10 deletions(-) diff --git a/fail2ban/tests/servertestcase.py b/fail2ban/tests/servertestcase.py index b7b9d802..fc505552 100644 --- a/fail2ban/tests/servertestcase.py +++ b/fail2ban/tests/servertestcase.py @@ -66,9 +66,12 @@ class TestServer(Server): class TransmitterBase(LogCaptureTestCase): + TEST_SRV_CLASS = TestServer + def setUp(self): """Call before every test case.""" super(TransmitterBase, self).setUp() + self.server = self.TEST_SRV_CLASS() self.transm = self.server._Server__transm # To test thransmitter we don't need to start server... #self.server.start('/dev/null', '/dev/null', force=False) @@ -157,10 +160,6 @@ class TransmitterBase(LogCaptureTestCase): class Transmitter(TransmitterBase): - def setUp(self): - self.server = TestServer() - super(Transmitter, self).setUp() - def testServerIsNotStarted(self): # so far isStarted only tested but not used otherwise # and here we don't really .start server @@ -893,8 +892,9 @@ class Transmitter(TransmitterBase): class TransmitterLogging(TransmitterBase): + TEST_SRV_CLASS = Server + def setUp(self): - self.server = Server() super(TransmitterLogging, self).setUp() self.server.setLogTarget("/dev/null") self.server.setLogLevel("CRITICAL") diff --git a/fail2ban/tests/utils.py b/fail2ban/tests/utils.py index 921427db..e2c4bccc 100644 --- a/fail2ban/tests/utils.py +++ b/fail2ban/tests/utils.py @@ -47,7 +47,7 @@ from ..server import asyncserver from ..version import version -logSys = getLogger(__name__) +logSys = getLogger("fail2ban") TEST_NOW = 1124013600 @@ -126,9 +126,6 @@ def getOptParser(doc=""): def initProcess(opts): # Logger: - global logSys - logSys = getLogger("fail2ban") - llev = None if opts.log_level is not None: # pragma: no cover # so we had explicit settings @@ -777,8 +774,9 @@ class LogCaptureTestCase(unittest.TestCase): """Call after every test case.""" # print "O: >>%s<<" % self._log.getvalue() self.pruneLog() + self._log.close() logSys.handlers = self._old_handlers - logSys.level = self._old_level + logSys.setLevel(self._old_level) super(LogCaptureTestCase, self).tearDown() def _is_logged(self, *s, **kwargs): From cbac7c176a93207027a53aa4591342f7268a1547 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 23 Mar 2021 12:57:42 +0100 Subject: [PATCH 082/240] readline fixed to consider interim new-line character as part of code point in multi-byte logs (e. g. unicode: utf-16be, utf-16le); suppress warning "Error decoding line" for incomplete line (produced by not fully read multi-byte new-line character at end of data); added test coverage for such logs --- fail2ban/server/filter.py | 59 ++++++++++++++++++++++------- fail2ban/tests/filtertestcase.py | 64 +++++++++++++++++++++++++++----- 2 files changed, 100 insertions(+), 23 deletions(-) diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index 845b069d..16279627 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -1398,6 +1398,9 @@ class FileContainer: try: return line.decode(enc, 'strict') except (UnicodeDecodeError, UnicodeEncodeError) as e: + # avoid warning if got incomplete end of line (e. g. '\n' in "...[0A" followed by "00]..." for utf-16le: + if (e.end == len(line) and line[e.start] in b'\r\n'): + return line[0:e.start].decode(enc, 'replace') global _decode_line_warn lev = 7 if not _decode_line_warn.get(filename, 0): @@ -1406,9 +1409,9 @@ class FileContainer: logSys.log(lev, "Error decoding line from '%s' with '%s'.", filename, enc) if logSys.getEffectiveLevel() <= lev: - logSys.log(lev, "Consider setting logencoding=utf-8 (or another appropriate" - " encoding) for this jail. Continuing" - " to process line ignoring invalid characters: %r", + logSys.log(lev, + "Consider setting logencoding to appropriate encoding for this jail. " + "Continuing to process line ignoring invalid characters: %r", line) # decode with replacing error chars: line = line.decode(enc, 'replace') @@ -1422,23 +1425,51 @@ class FileContainer: if line is not complete (and complete=True) or there is no content to read. If line is complete (and complete is True), it also shift current known position to begin of next line. + + Also it is safe against interim new-line bytes (e. g. part of multi-byte char) + in given encoding. """ if self.__handler is None: return "" - rl = self.__handler.readline() - if rl == b'': + # read raw bytes up to \n char: + b = self.__handler.readline() + if not b: return None - # trim new-line here and check the line was written complete (contains a new-line): - l = rl.rstrip(b'\r\n') - if self.inOperation and complete: - if l == rl: + bl = len(b) + # convert to log-encoding (new-line char could disappear if it is part of multi-byte sequence): + r = FileContainer.decode_line( + self.getFileName(), self.getEncoding(), b) + # trim new-line at end and check the line was written complete (contains a new-line): + l = r.rstrip('\r\n') + if complete: + if l == r: + # try to fill buffer in order to find line-end in log encoding: + fnd = 0 + while 1: + r = self.__handler.readline() + if not r: + break + b += r + bl += len(r) + # convert to log-encoding: + r = FileContainer.decode_line( + self.getFileName(), self.getEncoding(), b) + # ensure new-line is not in the middle (buffered 2 strings, e. g. in utf-16le it is "...[0A"+"00]..."): + e = r.find('\n') + if e >= 0 and e != len(r)-1: + l, r = r[0:e], r[0:e+1] + # back to bytes and get offset to seek after NL: + r = r.encode(self.getEncoding(), 'replace') + self.__handler.seek(-bl+len(r), 1) + return l + # trim new-line at end and check the line was written complete (contains a new-line): + l = r.rstrip('\r\n') + if l != r: + return l # not fulfilled - seek back and return: - self.__handler.seek(self.__pos, 0) + self.__handler.seek(-bl, 1) return None - # shift position (to be able to seek back above): - self.__pos += len(rl) - return FileContainer.decode_line( - self.getFileName(), self.getEncoding(), l) + return l def close(self): if self.__handler is not None: diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 799adfd3..f8621f29 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -644,6 +644,19 @@ class LogFile(LogCaptureTestCase): self.filter = FilterPoll(None) self.assertRaises(IOError, self.filter.addLogPath, LogFile.MISSING) + def testDecodeLineWarn(self): + # incomplete line (missing byte at end), warning is suppressed: + l = u"correct line\n" + r = l.encode('utf-16le') + self.assertEqual(FileContainer.decode_line('TESTFILE', 'utf-16le', r), l) + self.assertEqual(FileContainer.decode_line('TESTFILE', 'utf-16le', r[0:-1]), l[0:-1]) + self.assertNotLogged('Error decoding line') + # complete line (incorrect surrogate in the middle), warning is there: + r = b"incorrect \xc8\x0a line\n" + l = r.decode('utf-8', 'replace') + self.assertEqual(FileContainer.decode_line('TESTFILE', 'utf-8', r), l) + self.assertLogged('Error decoding line') + class LogFileFilterPoll(unittest.TestCase): @@ -1633,16 +1646,49 @@ class GetFailures(LogCaptureTestCase): def testCRLFFailures01(self): # We first adjust logfile/failures to end with CR+LF fname = tempfile.mktemp(prefix='tmp_fail2ban', suffix='crlf') - # poor man unix2dos: - fin, fout = open(GetFailures.FILENAME_01, 'rb'), open(fname, 'wb') - for l in fin.read().splitlines(): - fout.write(l + b'\r\n') - fin.close() - fout.close() + try: + # poor man unix2dos: + fin, fout = open(GetFailures.FILENAME_01, 'rb'), open(fname, 'wb') + for l in fin.read().splitlines(): + fout.write(l + b'\r\n') + fin.close() + fout.close() - # now see if we should be getting the "same" failures - self.testGetFailures01(filename=fname) - _killfile(fout, fname) + # now see if we should be getting the "same" failures + self.testGetFailures01(filename=fname) + finally: + _killfile(fout, fname) + + def testNLCharAsPartOfUniChar(self): + fname = tempfile.mktemp(prefix='tmp_fail2ban', suffix='crlf') + # test two multi-byte encodings (both contains `\x0A` in either \x02\x0A or \x0A\x02): + for enc in ('utf-16be', 'utf-16le'): + self.pruneLog("[test-phase encoding=%s]" % enc) + try: + fout = open(fname, 'wb') + tm = int(time.time()) + # test on unicode string containing \x0A as part of uni-char, + # it must produce exactly 2 lines (both are failures): + for l in ( + u'%s \u20AC Failed auth: invalid user Test\u020A from 192.0.2.1\n' % tm, + u'%s \u20AC Failed auth: invalid user TestI from 192.0.2.2\n' % tm + ): + fout.write(l.encode(enc)) + fout.close() + + self.filter.setLogEncoding(enc) + self.filter.addLogPath(fname, autoSeek=0) + self.filter.setDatePattern((r'^EPOCH',)) + self.filter.addFailRegex(r"Failed .* from ") + self.filter.getFailures(fname) + self.assertLogged( + "[DummyJail] Found 192.0.2.1", + "[DummyJail] Found 192.0.2.2", all=True, wait=True) + finally: + _killfile(fout, fname) + self.filter.delLogPath(fname) + # must find 4 failures and generate 2 tickets (2 IPs with each 2 failures): + self.assertEqual(self.filter.failManager.getFailCount(), (2, 4)) def testGetFailures02(self): output = ('141.3.81.106', 4, 1124013539.0, From 9659033523e27f62b2c6ad7db5e1671e58d4f045 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 24 Mar 2021 16:19:06 +0100 Subject: [PATCH 083/240] fail2ban-regex: reimplemented log-file iterator - uses FileContainer facilities now instead of direct read from file and decode; fail2banregextestcase.py extended to cover proper line-ending handling by interim NL char as part of multi-byte encodings (utf-16be, utf-16le) --- fail2ban/client/fail2banregex.py | 13 ++++---- fail2ban/server/filter.py | 5 ++- fail2ban/tests/fail2banregextestcase.py | 44 +++++++++++++++++++++---- fail2ban/tests/filtertestcase.py | 2 +- 4 files changed, 49 insertions(+), 15 deletions(-) diff --git a/fail2ban/client/fail2banregex.py b/fail2ban/client/fail2banregex.py index 90e178f9..26e7394a 100644 --- a/fail2ban/client/fail2banregex.py +++ b/fail2ban/client/fail2banregex.py @@ -289,9 +289,6 @@ class Fail2banRegex(object): def output(self, line): if not self._opts.out: output(line) - def decode_line(self, line): - return FileContainer.decode_line('', self._encoding, line) - def encode_line(self, line): return line.encode(self._encoding, 'ignore') @@ -724,8 +721,12 @@ class Fail2banRegex(object): return True def file_lines_gen(self, hdlr): - for line in hdlr: - yield self.decode_line(line) + while 1: + line = hdlr.readline() + if line is None: + break + yield line + hdlr.close() def start(self, args): @@ -745,7 +746,7 @@ class Fail2banRegex(object): if os.path.isfile(cmd_log): try: - hdlr = open(cmd_log, 'rb') + hdlr = FileContainer(cmd_log, self._encoding, doOpen=True) self.output( "Use log file : %s" % cmd_log ) self.output( "Use encoding : %s" % self._encoding ) test_lines = self.file_lines_gen(hdlr) diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index 16279627..9a2df255 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -1278,7 +1278,7 @@ except ImportError: # pragma: no cover class FileContainer: - def __init__(self, filename, encoding, tail=False): + def __init__(self, filename, encoding, tail=False, doOpen=False): self.__filename = filename self.setEncoding(encoding) self.__tail = tail @@ -1289,6 +1289,9 @@ class FileContainer: self.__hashNextTime = time.time() + 30 # Try to open the file. Raises an exception if an error occurred. handler = open(filename, 'rb') + if doOpen: # fail2ban-regex only (don't need to reopen it and check for rotation) + self.__handler = handler + return try: stats = os.fstat(handler.fileno()) self.__ino = stats.st_ino diff --git a/fail2ban/tests/fail2banregextestcase.py b/fail2ban/tests/fail2banregextestcase.py index 0a33fd9d..32acc0ae 100644 --- a/fail2ban/tests/fail2banregextestcase.py +++ b/fail2ban/tests/fail2banregextestcase.py @@ -25,6 +25,7 @@ __license__ = "GPL" import os import sys +import tempfile import unittest from ..client import fail2banregex @@ -80,6 +81,11 @@ def _test_exec_command_line(*args): sys.stderr = _org['stderr'] return _exit_code +def _reset(): + # reset global warn-counter: + from ..server.filter import _decode_line_warn + _decode_line_warn.clear() + STR_00 = "Dec 31 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 192.0.2.0" STR_00_NODT = "[sshd] error: PAM: Authentication failure for kevin from 192.0.2.0" @@ -122,6 +128,7 @@ class Fail2banRegexTest(LogCaptureTestCase): """Call before every test case.""" LogCaptureTestCase.setUp(self) setUpMyTime() + _reset() def tearDown(self): """Call after every test case.""" @@ -454,14 +461,8 @@ class Fail2banRegexTest(LogCaptureTestCase): FILENAME_ZZZ_GEN, FILENAME_ZZZ_GEN )) - def _reset(self): - # reset global warn-counter: - from ..server.filter import _decode_line_warn - _decode_line_warn.clear() - def testWronChar(self): unittest.F2B.SkipIfCfgMissing(stock=True) - self._reset() self.assertTrue(_test_exec( "-l", "notice", # put down log-level, because of too many debug-messages "--datepattern", r"^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?", @@ -477,7 +478,6 @@ class Fail2banRegexTest(LogCaptureTestCase): def testWronCharDebuggex(self): unittest.F2B.SkipIfCfgMissing(stock=True) - self._reset() self.assertTrue(_test_exec( "-l", "notice", # put down log-level, because of too many debug-messages "--datepattern", r"^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?", @@ -490,6 +490,36 @@ class Fail2banRegexTest(LogCaptureTestCase): self.assertLogged('https://') + def testNLCharAsPartOfUniChar(self): + fname = tempfile.mktemp(prefix='tmp_fail2ban', suffix='uni') + # test two multi-byte encodings (both contains `\x0A` in either \x02\x0A or \x0A\x02): + for enc in ('utf-16be', 'utf-16le'): + self.pruneLog("[test-phase encoding=%s]" % enc) + try: + fout = open(fname, 'wb') + # test on unicode string containing \x0A as part of uni-char, + # it must produce exactly 2 lines (both are failures): + for l in ( + u'1490349000 \u20AC Failed auth: invalid user Test\u020A from 192.0.2.1\n', + u'1490349000 \u20AC Failed auth: invalid user TestI from 192.0.2.2\n' + ): + fout.write(l.encode(enc)) + fout.close() + + self.assertTrue(_test_exec( + "-l", "notice", # put down log-level, because of too many debug-messages + "--encoding", enc, + "--datepattern", r"^EPOCH", + fname, r"Failed .* from ", + )) + + self.assertLogged(" encoding : %s" % enc, + "Lines: 2 lines, 0 ignored, 2 matched, 0 missed", all=True) + self.assertNotLogged("Missed line(s)") + finally: + fout.close() + os.unlink(fname) + def testExecCmdLine_Usage(self): self.assertNotEqual(_test_exec_command_line(), 0) self.pruneLog() diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index f8621f29..319f16de 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -1660,7 +1660,7 @@ class GetFailures(LogCaptureTestCase): _killfile(fout, fname) def testNLCharAsPartOfUniChar(self): - fname = tempfile.mktemp(prefix='tmp_fail2ban', suffix='crlf') + fname = tempfile.mktemp(prefix='tmp_fail2ban', suffix='uni') # test two multi-byte encodings (both contains `\x0A` in either \x02\x0A or \x0A\x02): for enc in ('utf-16be', 'utf-16le'): self.pruneLog("[test-phase encoding=%s]" % enc) From ccf4f3a07dbe5e21dbca1e0f3e702cbdaaeabaf3 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 24 Mar 2021 17:22:05 +0100 Subject: [PATCH 084/240] amend with common log-file iterator in fail2ban-regex and test-suite (in sample regex factory also) --- fail2ban/client/fail2banregex.py | 12 ++---------- fail2ban/server/filter.py | 17 ++++++++++++++--- fail2ban/tests/samplestestcase.py | 15 ++++++++------- 3 files changed, 24 insertions(+), 20 deletions(-) diff --git a/fail2ban/client/fail2banregex.py b/fail2ban/client/fail2banregex.py index 26e7394a..5921dfdd 100644 --- a/fail2ban/client/fail2banregex.py +++ b/fail2ban/client/fail2banregex.py @@ -720,14 +720,6 @@ class Fail2banRegex(object): return True - def file_lines_gen(self, hdlr): - while 1: - line = hdlr.readline() - if line is None: - break - yield line - hdlr.close() - def start(self, args): cmd_log, cmd_regex = args[:2] @@ -746,10 +738,10 @@ class Fail2banRegex(object): if os.path.isfile(cmd_log): try: - hdlr = FileContainer(cmd_log, self._encoding, doOpen=True) + test_lines = FileContainer(cmd_log, self._encoding, doOpen=True) + self.output( "Use log file : %s" % cmd_log ) self.output( "Use encoding : %s" % self._encoding ) - test_lines = self.file_lines_gen(hdlr) except IOError as e: # pragma: no cover output( e ) return False diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index 9a2df255..e16d86c9 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -1280,6 +1280,7 @@ class FileContainer: def __init__(self, filename, encoding, tail=False, doOpen=False): self.__filename = filename + self.waitForLineEnd = True self.setEncoding(encoding) self.__tail = tail self.__handler = None @@ -1469,9 +1470,10 @@ class FileContainer: l = r.rstrip('\r\n') if l != r: return l - # not fulfilled - seek back and return: - self.__handler.seek(-bl, 1) - return None + if self.waitForLineEnd: + # not fulfilled - seek back and return: + self.__handler.seek(-bl, 1) + return None return l def close(self): @@ -1482,6 +1484,15 @@ class FileContainer: self.__handler.close() self.__handler = None + def __iter__(self): + return self + def next(self): + line = self.readline() + if line is None: + self.close() + raise StopIteration + return line + _decode_line_warn = Utils.Cache(maxCount=1000, maxTime=24*60*60); diff --git a/fail2ban/tests/samplestestcase.py b/fail2ban/tests/samplestestcase.py index 5a72ffa9..b33b46c1 100644 --- a/fail2ban/tests/samplestestcase.py +++ b/fail2ban/tests/samplestestcase.py @@ -23,7 +23,6 @@ __copyright__ = "Copyright (c) 2013 Steven Hiscocks" __license__ = "GPL" import datetime -import fileinput import inspect import json import os @@ -156,12 +155,15 @@ def testSampleRegexsFactory(name, basedir): i = 0 while i < len(filenames): filename = filenames[i]; i += 1; - logFile = fileinput.FileInput(os.path.join(TEST_FILES_DIR, "logs", - filename), mode='rb') + logFile = FileContainer(os.path.join(TEST_FILES_DIR, "logs", + filename), 'UTF-8', doOpen=True) + # avoid errors if no NL char at end of test log-file: + logFile.waitForLineEnd = False ignoreBlock = False + lnnum = 0 for line in logFile: - line = FileContainer.decode_line(logFile.filename(), 'UTF-8', line) + lnnum += 1 jsonREMatch = re.match("^#+ ?(failJSON|(?:file|filter)Options|addFILE):(.+)$", line) if jsonREMatch: try: @@ -201,9 +203,8 @@ def testSampleRegexsFactory(name, basedir): # failJSON - faildata contains info of the failure to check it. except ValueError as e: # pragma: no cover - we've valid json's raise ValueError("%s: %s:%i" % - (e, logFile.filename(), logFile.filelineno())) + (e, logFile.getFileName(), lnnum)) line = next(logFile) - line = FileContainer.decode_line(logFile.filename(), 'UTF-8', line) elif ignoreBlock or line.startswith("#") or not line.strip(): continue else: # pragma: no cover - normally unreachable @@ -298,7 +299,7 @@ def testSampleRegexsFactory(name, basedir): import pprint raise AssertionError("%s: %s on: %s:%i, line:\n %s\nregex (%s):\n %s\n" "faildata: %s\nfail: %s" % ( - fltName, e, logFile.filename(), logFile.filelineno(), + fltName, e, logFile.getFileName(), lnnum, line, failregex, regexList[failregex] if failregex != -1 else None, '\n'.join(pprint.pformat(faildata).splitlines()), '\n'.join(pprint.pformat(fail).splitlines()))) From 4b17dddc2389a71bba15be9a3129b03665f853b0 Mon Sep 17 00:00:00 2001 From: sebres Date: Thu, 25 Mar 2021 12:05:09 +0100 Subject: [PATCH 085/240] update ChangeLog --- ChangeLog | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/ChangeLog b/ChangeLog index e5e7b485..4fe8acdc 100644 --- a/ChangeLog +++ b/ChangeLog @@ -18,12 +18,17 @@ ver. 1.0.1-dev-1 (20??/??/??) - development nightly edition different from 0) in case of unsane environment. ### Fixes +* readline fixed to consider interim new-line character as part of code point in multi-byte logs + (e. g. unicode encoding like utf-16be, utf-16le); * `filter.d/drupal-auth.conf` more strict regex, extended to match "Login attempt failed from" (gh-2742) ### New Features and Enhancements * `actioncheck` behavior is changed now (gh-488), so invariant check as well as restore or repair of sane environment (in case of recognized unsane state) would only occur on action errors (e. g. if ban or unban operations are exiting with other code as 0) +* better recognition of log rotation, better performance by reopen: avoid unnecessary seek to begin of file + (and hash calculation) +* file filter reads only complete lines (ended with new-line) now, so waits for end of line (for its completion) ver. 0.11.2 (2020/11/23) - heal-the-world-with-security-tools From b259e819114e587e0e9b17253860dd48f9579997 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 24 Mar 2021 17:23:14 +0100 Subject: [PATCH 086/240] test-suite: skip testFQDN if no network --- fail2ban/tests/filtertestcase.py | 1 + 1 file changed, 1 insertion(+) diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 319f16de..27f5695e 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -2252,6 +2252,7 @@ class DNSUtilsNetworkTests(unittest.TestCase): ip1 = IPAddr('2606:2800:220:1:248:1893:25c8:1946'); ip2 = IPAddr('2606:2800:220:1:248:1893:25c8:1946'); self.assertEqual(id(ip1), id(ip2)) def testFQDN(self): + unittest.F2B.SkipIfNoNetwork() sname = DNSUtils.getHostname(fqdn=False) lname = DNSUtils.getHostname(fqdn=True) # FQDN is not localhost if short hostname is not localhost too (or vice versa): From 2686811593a57bab72f691a5c1c93ffb877d4395 Mon Sep 17 00:00:00 2001 From: j-marz Date: Sun, 28 Mar 2021 21:19:10 +1100 Subject: [PATCH 087/240] Updated zoneminder filter Support new log format, ERR instead of WAR. Add detection of non-existent user login attempts --- config/filter.d/zoneminder.conf | 16 +++++++++++----- fail2ban/tests/files/logs/zoneminder | 6 ++++++ 2 files changed, 17 insertions(+), 5 deletions(-) diff --git a/config/filter.d/zoneminder.conf b/config/filter.d/zoneminder.conf index cc82755a..1af97c7d 100644 --- a/config/filter.d/zoneminder.conf +++ b/config/filter.d/zoneminder.conf @@ -5,17 +5,23 @@ before = apache-common.conf [Definition] -# pattern: [Wed Apr 27 23:12:07.736196 2016] [:error] [pid 2460] [client 10.1.1.1:47296] WAR [Login denied for user "test"], referer: https://zoneminderurl/index.php -# -# +# patterns: + # [Mon Mar 28 16:50:49.522240 2016] [:error] [pid 1795] [client 10.1.1.1:50700] WAR [Login denied for user "username1"], referer: https://zoneminder/ + # [Sun Mar 28 16:53:00.472693 2021] [php7:notice] [pid 11328] [client 10.1.1.1:39568] ERR [Could not retrieve user test details], referer: https://zm/ + # [Sun Mar 28 16:59:14.150625 2021] [php7:notice] [pid 11336] [client 10.1.1.1:39654] ERR [Login denied for user "john"], referer: https://zm/ + # Option: failregex -# Notes.: regex to match the password failure messages in the logfile. +# Notes.: regex to match the login failure and non-existent user error messages in the logfile. failregex = ^%(_apache_error_client)s WAR \[Login denied for user "[^"]*"\] + ^%(_apache_error_client)s ERR \[Login denied for user "[^"]*"\] + ^%(_apache_error_client)s ERR \[Could not retrieve user \w* details\] ignoreregex = # Notes: -# Tested on Zoneminder 1.29.0 +# Tested on Zoneminder 1.29 and 1.35.21 +# +# Zoneminer versions > 1.3x use "ERR" and < 1.3x use "WAR" level logs, so i've kept both for compatibility reasons # # Author: John Marzella diff --git a/fail2ban/tests/files/logs/zoneminder b/fail2ban/tests/files/logs/zoneminder index abd49869..f4b6bd3e 100644 --- a/fail2ban/tests/files/logs/zoneminder +++ b/fail2ban/tests/files/logs/zoneminder @@ -1,2 +1,8 @@ # failJSON: { "time": "2016-03-28T16:50:49", "match": true , "host": "10.1.1.1" } [Mon Mar 28 16:50:49.522240 2016] [:error] [pid 1795] [client 10.1.1.1:50700] WAR [Login denied for user "username1"], referer: https://zoneminder/ + +# failJSON: { "time": "2021-03-28T16:53:00", "match": true , "host": "10.1.1.1" } +[Sun Mar 28 16:53:00.472693 2021] [php7:notice] [pid 11328] [client 10.1.1.1:39568] ERR [Could not retrieve user username1 details], referer: https://zm/zm/?view=logout + +# failJSON: { "time": "2021-03-28T16:59:14", "match": true , "host": "10.1.1.1" } +[Sun Mar 28 16:59:14.150625 2021] [php7:notice] [pid 11336] [client 10.1.1.1:39654] ERR [Login denied for user "username1"], referer: https://zm/zm/? From 5d8f5004718d38146c6821f37eaeb3c7b0415d9d Mon Sep 17 00:00:00 2001 From: j-marz Date: Mon, 29 Mar 2021 08:36:53 +1100 Subject: [PATCH 088/240] updated formatting to pass tests --- config/filter.d/zoneminder.conf | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/config/filter.d/zoneminder.conf b/config/filter.d/zoneminder.conf index 1af97c7d..7da40968 100644 --- a/config/filter.d/zoneminder.conf +++ b/config/filter.d/zoneminder.conf @@ -5,11 +5,10 @@ before = apache-common.conf [Definition] -# patterns: - # [Mon Mar 28 16:50:49.522240 2016] [:error] [pid 1795] [client 10.1.1.1:50700] WAR [Login denied for user "username1"], referer: https://zoneminder/ - # [Sun Mar 28 16:53:00.472693 2021] [php7:notice] [pid 11328] [client 10.1.1.1:39568] ERR [Could not retrieve user test details], referer: https://zm/ - # [Sun Mar 28 16:59:14.150625 2021] [php7:notice] [pid 11336] [client 10.1.1.1:39654] ERR [Login denied for user "john"], referer: https://zm/ - +# patterns: [Mon Mar 28 16:50:49.522240 2016] [:error] [pid 1795] [client 10.1.1.1:50700] WAR [Login denied for user "username1"], referer: https://zoneminder/ +# [Sun Mar 28 16:53:00.472693 2021] [php7:notice] [pid 11328] [client 10.1.1.1:39568] ERR [Could not retrieve user test details], referer: https://zm/ +# [Sun Mar 28 16:59:14.150625 2021] [php7:notice] [pid 11336] [client 10.1.1.1:39654] ERR [Login denied for user "john"], referer: https://zm/ +# # Option: failregex # Notes.: regex to match the login failure and non-existent user error messages in the logfile. From 5aa20c30d8b283970e4b3ee0938f729850ebb618 Mon Sep 17 00:00:00 2001 From: Markus Felten Date: Fri, 12 Feb 2021 09:29:03 +0100 Subject: [PATCH 089/240] fix: add journalmatch to nginx filters --- config/filter.d/nginx-bad-request.conf | 2 ++ config/filter.d/nginx-botsearch.conf | 4 +++- config/filter.d/nginx-http-auth.conf | 2 ++ config/filter.d/nginx-limit-req.conf | 3 +++ 4 files changed, 10 insertions(+), 1 deletion(-) diff --git a/config/filter.d/nginx-bad-request.conf b/config/filter.d/nginx-bad-request.conf index 2b8f5ab6..12c14ab7 100644 --- a/config/filter.d/nginx-bad-request.conf +++ b/config/filter.d/nginx-bad-request.conf @@ -11,4 +11,6 @@ datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,] ^[^\[]*\[({DATE}) {^LN-BEG} +journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx + # Author: Jan Przybylak diff --git a/config/filter.d/nginx-botsearch.conf b/config/filter.d/nginx-botsearch.conf index 0be895b2..2bd23072 100644 --- a/config/filter.d/nginx-botsearch.conf +++ b/config/filter.d/nginx-botsearch.conf @@ -17,7 +17,9 @@ datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,] ^[^\[]*\[({DATE}) {^LN-BEG} +journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx + # DEV Notes: # Based on apache-botsearch filter # -# Author: Frantisek Sumsal \ No newline at end of file +# Author: Frantisek Sumsal diff --git a/config/filter.d/nginx-http-auth.conf b/config/filter.d/nginx-http-auth.conf index 93341cd2..3a2e314b 100644 --- a/config/filter.d/nginx-http-auth.conf +++ b/config/filter.d/nginx-http-auth.conf @@ -10,6 +10,8 @@ ignoreregex = datepattern = {^LN-BEG} +journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx + # DEV NOTES: # Based on samples in https://github.com/fail2ban/fail2ban/pull/43/files # Extensive search of all nginx auth failures not done yet. diff --git a/config/filter.d/nginx-limit-req.conf b/config/filter.d/nginx-limit-req.conf index e23548ab..2f45e831 100644 --- a/config/filter.d/nginx-limit-req.conf +++ b/config/filter.d/nginx-limit-req.conf @@ -44,3 +44,6 @@ failregex = ^\s*\[[a-z]+\] \d+#\d+: \*\d+ limiting requests, excess: [\d\.]+ by ignoreregex = datepattern = {^LN-BEG} + +journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx + From 9eaa2322b0913c805cfa7d45be96d0067ffe61b1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=BCdiger=20Olschewsky?= Date: Mon, 24 Feb 2020 16:35:42 +0100 Subject: [PATCH 090/240] Filter and Defaults for Microsoft SQL Server --- config/filter.d/mssql-auth.conf | 15 +++++++++++++++ config/jail.conf | 8 ++++++++ fail2ban/tests/files/logs/mssql-auth | 3 +++ 3 files changed, 26 insertions(+) create mode 100644 config/filter.d/mssql-auth.conf create mode 100644 fail2ban/tests/files/logs/mssql-auth diff --git a/config/filter.d/mssql-auth.conf b/config/filter.d/mssql-auth.conf new file mode 100644 index 00000000..a1813c83 --- /dev/null +++ b/config/filter.d/mssql-auth.conf @@ -0,0 +1,15 @@ +# Fail2Ban filter for failed MSSQL Server authentication attempts + +[Definition] + +failregex = Logon\s+Login failed for user ('.*')(.*)\[CLIENT: \]$ + + +# DEV Notes: +# Tested with SQL Server 2019 on Ubuntu 18.04 +# +# Example: +# 2020-02-24 14:48:55.12 Logon Login failed for user 'root'. Reason: Could not find a login matching the name provided. [CLIENT: 127.0.0.1] +# +# Author: Rüdiger Olschewsky +# \ No newline at end of file diff --git a/config/jail.conf b/config/jail.conf index 20958d11..aeff3616 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -786,6 +786,14 @@ logpath = %(mysql_log)s backend = %(mysql_backend)s +[mssql-auth] +# Default configuration for Microsoft SQL Server for Linux +# See the 'mssql-conf' manpage how to change logpath or port +logpath = /var/opt/mssql/log/errorlog +port = 1433 +filter = mssql-auth + + # Log wrong MongoDB auth (for details see filter 'filter.d/mongodb-auth.conf') [mongodb-auth] # change port when running with "--shardsvr" or "--configsvr" runtime operation diff --git a/fail2ban/tests/files/logs/mssql-auth b/fail2ban/tests/files/logs/mssql-auth new file mode 100644 index 00000000..5a97f53e --- /dev/null +++ b/fail2ban/tests/files/logs/mssql-auth @@ -0,0 +1,3 @@ +2020-02-24 16:05:21.00 Logon Login failed for user 'Backend'. Reason: Could not find a login matching the name provided. [CLIENT: 212.96.131.253] +2020-02-24 16:30:25.88 Logon Login failed for user '===)jf02hüas9ä##22f'. Reason: Could not find a login matching the name provided. [CLIENT: 148.86.203.199] +2020-02-24 16:31:12.20 Logon Login failed for user ''. Reason: An attempt to login using SQL authentication failed. Server is configured for Integrated authentication only. [CLIENT: 105.254.136.171] \ No newline at end of file From 7f38b80d351cc51cd3273ad93cd1939a4b1abe1e Mon Sep 17 00:00:00 2001 From: sebres Date: Sat, 3 Apr 2021 20:16:47 +0200 Subject: [PATCH 091/240] precise regex (left anchor and fewer catch-all's); fixed tests (added failJSON and more tests for some corner-cases around new RE) --- config/filter.d/mssql-auth.conf | 2 +- fail2ban/tests/files/logs/mssql-auth | 14 +++++++++++--- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/config/filter.d/mssql-auth.conf b/config/filter.d/mssql-auth.conf index a1813c83..65bbd917 100644 --- a/config/filter.d/mssql-auth.conf +++ b/config/filter.d/mssql-auth.conf @@ -2,7 +2,7 @@ [Definition] -failregex = Logon\s+Login failed for user ('.*')(.*)\[CLIENT: \]$ +failregex = ^\s*Logon\s+Login failed for user '(?:[^']*|.*)'\. [^'\[]+\[CLIENT: \]$ # DEV Notes: diff --git a/fail2ban/tests/files/logs/mssql-auth b/fail2ban/tests/files/logs/mssql-auth index 5a97f53e..1c9b65ec 100644 --- a/fail2ban/tests/files/logs/mssql-auth +++ b/fail2ban/tests/files/logs/mssql-auth @@ -1,3 +1,11 @@ -2020-02-24 16:05:21.00 Logon Login failed for user 'Backend'. Reason: Could not find a login matching the name provided. [CLIENT: 212.96.131.253] -2020-02-24 16:30:25.88 Logon Login failed for user '===)jf02hüas9ä##22f'. Reason: Could not find a login matching the name provided. [CLIENT: 148.86.203.199] -2020-02-24 16:31:12.20 Logon Login failed for user ''. Reason: An attempt to login using SQL authentication failed. Server is configured for Integrated authentication only. [CLIENT: 105.254.136.171] \ No newline at end of file +# failJSON: { "time": "2020-02-24T16:05:21", "match": true , "host": "192.0.2.1" } +2020-02-24 16:05:21.00 Logon Login failed for user 'Backend'. Reason: Could not find a login matching the name provided. [CLIENT: 192.0.2.1] +# failJSON: { "time": "2020-02-24T16:30:25", "match": true , "host": "192.0.2.2" } +2020-02-24 16:30:25.88 Logon Login failed for user '===)jf02hüas9ä##22f'. Reason: Could not find a login matching the name provided. [CLIENT: 192.0.2.2] +# failJSON: { "time": "2020-02-24T16:31:12", "match": true , "host": "192.0.2.3" } +2020-02-24 16:31:12.20 Logon Login failed for user ''. Reason: An attempt to login using SQL authentication failed. Server is configured for Integrated authentication only. [CLIENT: 192.0.2.3] + +# failJSON: { "time": "2020-02-24T16:31:26", "match": true , "host": "192.0.2.4", "user":"O'Leary" } +2020-02-24 16:31:26.01 Logon Login failed for user 'O'Leary'. Reason: Could not find a login matching the name provided. [CLIENT: 192.0.2.4] +# failJSON: { "time": "2020-02-24T16:31:26", "match": false, "desc": "test injection in possibly unescaped foreign input" } +2020-02-24 16:31:26.02 Wrong data received: Logon Login failed for user 'test'. Reason: Could not find a login matching the name provided. [CLIENT: 192.0.2.5] From 1215cb28ac6a79acda2699694556bc1a6cef1e45 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Sat, 3 Apr 2021 20:58:26 +0200 Subject: [PATCH 092/240] Update nsd --- fail2ban/tests/files/logs/nsd | 2 ++ 1 file changed, 2 insertions(+) diff --git a/fail2ban/tests/files/logs/nsd b/fail2ban/tests/files/logs/nsd index a33a52a9..63c162e9 100644 --- a/fail2ban/tests/files/logs/nsd +++ b/fail2ban/tests/files/logs/nsd @@ -2,3 +2,5 @@ [1387288694] nsd[7745]: info: ratelimit block example.com. type any target 192.0.2.0/24 query 192.0.2.105 TYPE255 # failJSON: { "time": "2013-12-18T07:42:15", "match": true , "host": "192.0.2.115" } [1387348935] nsd[23600]: info: axfr for zone domain.nl. from client 192.0.2.115 refused, no acl matches. +# failJSON: { "time": "2021-03-05T05:25:14", "match": true , "host": "192.0.2.32", "desc": "new format, no client after from, no dot at end, gh-2965" } +[2021-03-05 05:25:14.562] nsd[160800]: info: axfr for example.com. from 192.0.2.32 refused, no acl matches From a838deba7f8066d50984ed24983fb90f15edfaaf Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Sat, 3 Apr 2021 21:00:14 +0200 Subject: [PATCH 093/240] restore anchor (e. g. catch all in the middle), dot is optional now, RE rewritten a bit more precise --- config/filter.d/nsd.conf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/config/filter.d/nsd.conf b/config/filter.d/nsd.conf index 9399db23..0589c16c 100644 --- a/config/filter.d/nsd.conf +++ b/config/filter.d/nsd.conf @@ -22,8 +22,8 @@ _daemon = nsd # (?:::f{4,6}:)?(?P[\w\-.^_]+) # Values: TEXT -failregex = ^%(__prefix_line)sinfo: ratelimit block .* query TYPE255$ - ^%(__prefix_line)sinfo: .* refused, no acl matches +failregex = ^%(__prefix_line)sinfo: ratelimit block .* query TYPE255$ + ^%(__prefix_line)sinfo: .* from(?: client)? refused, no acl matches\.?$ ignoreregex = From 0c4d356d118d1282d8bdbed21cb514f02953e65f Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Sat, 3 Apr 2021 23:10:51 +0200 Subject: [PATCH 094/240] added test log-file --- fail2ban/tests/files/logs/scanlogd | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 fail2ban/tests/files/logs/scanlogd diff --git a/fail2ban/tests/files/logs/scanlogd b/fail2ban/tests/files/logs/scanlogd new file mode 100644 index 00000000..5a97c578 --- /dev/null +++ b/fail2ban/tests/files/logs/scanlogd @@ -0,0 +1,8 @@ +# failJSON: { "time": "2005-03-05T21:44:43", "match": true , "host": "192.0.2.123" } +Mar 5 21:44:43 srv scanlogd: 192.0.2.123 to 192.0.2.1 ports 80, 81, 83, 88, 99, 443, 1080, 3128, ..., f????uxy, TOS 00, TTL 49 @20:44:43 +# failJSON: { "time": "2005-03-05T21:44:44", "match": true , "host": "192.0.2.123" } +Mar 5 21:44:44 srv scanlogd: 192.0.2.123 to 192.0.2.1 ports 497, 515, 544, 543, 464, 513, ..., fSrpauxy, TOS 00 @09:04:25 +# failJSON: { "time": "2005-03-05T21:44:45", "match": true , "host": "192.0.2.123" } +Mar 5 21:44:45 srv scanlogd: 192.0.2.123 to 192.0.2.1 ports 593, 548, 636, 646, 625, 631, ..., fSrpauxy, TOS 00, TTL 239 @17:34:00 +# failJSON: { "time": "2005-03-05T21:44:46", "match": true , "host": "192.0.2.123" } +Mar 5 21:44:46 srv scanlogd: 192.0.2.123 to 192.0.2.1 ports 22, 26, 37, 80, 25, 79, ..., fSrpauxy, TOS 00 @22:38:37 From 14edeed310ea5d74a13ec72d415b934cb4f7beb1 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Sat, 3 Apr 2021 23:24:55 +0200 Subject: [PATCH 095/240] fixed regex (don't need to match whole line, e. g. every port etc) --- config/filter.d/scanlogd.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/filter.d/scanlogd.conf b/config/filter.d/scanlogd.conf index 65ad63f6..15fea329 100644 --- a/config/filter.d/scanlogd.conf +++ b/config/filter.d/scanlogd.conf @@ -10,7 +10,7 @@ before = common.conf _daemon = scanlogd -failregex = ^%(__prefix_line)s\ to\ [\.:0-9a-f]+\ ports\ [\ \.,0-9]+,\ f.......,\ TOS\ [0-9]+,\ TTL\ [0-9]+\ \@[0-9]{1,2}:[0-9]{2}:[0-9]{2}$ +failregex = ^%(__prefix_line)s(?::) to \S+ ports\b ignoreregex = From 977dfe4bd762474e4e2d077e258c49772a48f6f3 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Sat, 3 Apr 2021 23:29:16 +0200 Subject: [PATCH 096/240] small amend: sport after saddr is optional format of message: saddr[:sport] to daddr [and others,] ports port[, port...], ..., flags[, TOS TOS][, TTL TTL] @HH:MM:SS --- config/filter.d/scanlogd.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/filter.d/scanlogd.conf b/config/filter.d/scanlogd.conf index 15fea329..d3fe78b0 100644 --- a/config/filter.d/scanlogd.conf +++ b/config/filter.d/scanlogd.conf @@ -10,7 +10,7 @@ before = common.conf _daemon = scanlogd -failregex = ^%(__prefix_line)s(?::) to \S+ ports\b +failregex = ^%(__prefix_line)s(?::)? to \S+ ports\b ignoreregex = From 2d51240b3e9e19a4259c03058ddde6ce8698397b Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Sat, 3 Apr 2021 23:33:49 +0200 Subject: [PATCH 097/240] correction for default log interpolation and added allports banaction --- config/jail.conf | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/config/jail.conf b/config/jail.conf index d6d8af67..52cc161d 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -967,4 +967,5 @@ port = http,https logpath = /var/log/traefik/access.log [scanlogd] -logpath = %{syslog_local0} +logpath = %(syslog_local0)s +banaction = %(banaction_allports)s From 7579072e3b1d0663ac50ff067a953d4fe8c28ac7 Mon Sep 17 00:00:00 2001 From: Michele Mondelli Date: Thu, 31 Dec 2020 17:03:19 +0100 Subject: [PATCH 098/240] docs: fix typos --- config/jail.conf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/config/jail.conf b/config/jail.conf index b4b9f249..cd802a57 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -67,7 +67,7 @@ before = paths-debian.conf # more aggressive example of formula has the same values only for factor "2.0 / 2.885385" : #bantime.formula = ban.Time * math.exp(float(ban.Count+1)*banFactor)/math.exp(1*banFactor) -# "bantime.multipliers" used to calculate next value of ban time instead of formula, coresponding +# "bantime.multipliers" used to calculate next value of ban time instead of formula, corresponding # previously ban count and given "bantime.factor" (for multipliers default is 1); # following example grows ban time by 1, 2, 4, 8, 16 ... and if last ban count greater as multipliers count, # always used last multiplier (64 in example), for factor '1' and original ban time 600 - 10.6 hours @@ -77,7 +77,7 @@ before = paths-debian.conf #bantime.multipliers = 1 5 30 60 300 720 1440 2880 # "bantime.overalljails" (if true) specifies the search of IP in the database will be executed -# cross over all jails, if false (dafault), only current jail of the ban IP will be searched +# cross over all jails, if false (default), only current jail of the ban IP will be searched #bantime.overalljails = false # -------------------- From c5d43d7573437e401f991a782e106a851ef9975d Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Sun, 4 Apr 2021 00:00:59 +0200 Subject: [PATCH 099/240] Update ChangeLog --- ChangeLog | 3 +++ 1 file changed, 3 insertions(+) diff --git a/ChangeLog b/ChangeLog index f9fb8a33..56b8c757 100644 --- a/ChangeLog +++ b/ChangeLog @@ -21,6 +21,9 @@ ver. 1.0.1-dev-1 (20??/??/??) - development nightly edition * `actioncheck` behavior is changed now (gh-488), so invariant check as well as restore or repair of sane environment (in case of recognized unsane state) would only occur on action errors (e. g. if ban or unban operations are exiting with other code as 0) +* `filter.d/nginx-http-auth.conf` - extended with parameter mode, so additionally to `auth` (or `normal`) + mode `fallback` (or combined as `aggressive`) can find SSL errors while SSL handshaking, gh-2881 + ver. 0.11.2 (2020/11/23) - heal-the-world-with-security-tools ----------- From b2f6a3a658f83fa6e2ec80deffa44c837775e80a Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Sun, 4 Apr 2021 00:21:59 +0200 Subject: [PATCH 100/240] remove unneeded substitution it is enough to add `apprise` to action --- config/jail.conf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/config/jail.conf b/config/jail.conf index dbca2e8f..e17a1892 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -231,10 +231,10 @@ action_xarf = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(prot # See https://github.com/caronc/apprise/wiki for details on what is supported. # # You may optionally over-ride the default configuration line (containing the Apprise URLs) -# by using 'apprise[name=%(__name__)s, config="/alternate/path/to/apprise.cfg"]' otherwise +# by using 'apprise[config="/alternate/path/to/apprise.cfg"]' otherwise # /etc/fail2ban/apprise.conf is sourced for your supported notification configuration. -action_apprise = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"] - apprise[name=%(__name__)s] +# action = %(action_)s +# apprise # ban IP on CloudFlare & send an e-mail with whois report and relevant log lines # to the destemail. From 8b741129a51c98b6178f08bcca9af88a4486201f Mon Sep 17 00:00:00 2001 From: Jordi Sanfeliu Date: Wed, 14 Apr 2021 11:26:52 +0200 Subject: [PATCH 101/240] Create monitorix-httpd --- fail2ban/tests/files/logs/monitorix-httpd | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 fail2ban/tests/files/logs/monitorix-httpd diff --git a/fail2ban/tests/files/logs/monitorix-httpd b/fail2ban/tests/files/logs/monitorix-httpd new file mode 100644 index 00000000..3166dbd3 --- /dev/null +++ b/fail2ban/tests/files/logs/monitorix-httpd @@ -0,0 +1,10 @@ +Wed Apr 14 08:11:01 2021 - OK - [127.0.0.1] "GET /monitorix-cgi/monitorix.cgi - Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0" +Wed Apr 14 08:11:01 2021 - OK - [127.0.0.1] "GET /monitorix/css/black.css - Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0" +Wed Apr 14 08:11:01 2021 - OK - [127.0.0.1] "GET /monitorix/imgs/fs01.1day.png - Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0" +Wed Apr 14 08:11:02 2021 - OK - [127.0.0.1] "GET /monitorix/imgs/fs03.1day.png - Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0" +Wed Apr 14 08:11:02 2021 - OK - [127.0.0.1] "GET /monitorix/imgs/fs04.1day.png - Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0" +Wed Apr 14 08:11:02 2021 - OK - [127.0.0.1] "GET /monitorix/imgs/fs02.1day.png - Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0" +Wed Apr 14 08:54:22 2021 - NOTEXIST - [127.0.0.1] File does not exist: /manager/html +Wed Apr 14 11:24:31 2021 - NOTALLOWED - [127.0.0.1] Access not allowed: /monitorix/ +Wed Apr 14 11:26:08 2021 - AUTHERR - [127.0.0.1] Authentication error: /monitorix/ +Wed Apr 14 11:26:09 2021 - AUTHERR - [127.0.0.1] Authentication error: /monitorix/ From 63b3f39adc7323873d7f558a6060f79ce3e9a273 Mon Sep 17 00:00:00 2001 From: Jordi Sanfeliu Date: Wed, 14 Apr 2021 11:30:48 +0200 Subject: [PATCH 102/240] Rename monitorix-httpd to monitorix --- fail2ban/tests/files/logs/{monitorix-httpd => monitorix} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename fail2ban/tests/files/logs/{monitorix-httpd => monitorix} (100%) diff --git a/fail2ban/tests/files/logs/monitorix-httpd b/fail2ban/tests/files/logs/monitorix similarity index 100% rename from fail2ban/tests/files/logs/monitorix-httpd rename to fail2ban/tests/files/logs/monitorix From b6fac90b5a57934749f30a0f9c626b075fcf51d8 Mon Sep 17 00:00:00 2001 From: Jordi Sanfeliu Date: Wed, 14 Apr 2021 11:46:23 +0200 Subject: [PATCH 103/240] Update monitorix --- fail2ban/tests/files/logs/monitorix | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/fail2ban/tests/files/logs/monitorix b/fail2ban/tests/files/logs/monitorix index 3166dbd3..e6ad6dc6 100644 --- a/fail2ban/tests/files/logs/monitorix +++ b/fail2ban/tests/files/logs/monitorix @@ -1,10 +1,8 @@ +# failJSON: { "time": "2021-04-14T08:11:01", "match": false, "desc": "should be ignored: successful request" } Wed Apr 14 08:11:01 2021 - OK - [127.0.0.1] "GET /monitorix-cgi/monitorix.cgi - Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0" -Wed Apr 14 08:11:01 2021 - OK - [127.0.0.1] "GET /monitorix/css/black.css - Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0" -Wed Apr 14 08:11:01 2021 - OK - [127.0.0.1] "GET /monitorix/imgs/fs01.1day.png - Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0" -Wed Apr 14 08:11:02 2021 - OK - [127.0.0.1] "GET /monitorix/imgs/fs03.1day.png - Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0" -Wed Apr 14 08:11:02 2021 - OK - [127.0.0.1] "GET /monitorix/imgs/fs04.1day.png - Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0" -Wed Apr 14 08:11:02 2021 - OK - [127.0.0.1] "GET /monitorix/imgs/fs02.1day.png - Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0" +# failJSON: { "time": "2021-04-14T08:54:22", "match": true, "host": "127.0.0.1", "desc": "file does not exist" } Wed Apr 14 08:54:22 2021 - NOTEXIST - [127.0.0.1] File does not exist: /manager/html +# failJSON: { "time": "2021-04-14T11:24:31", "match": true, "host": "127.0.0.1", "desc": "access not allowed" } Wed Apr 14 11:24:31 2021 - NOTALLOWED - [127.0.0.1] Access not allowed: /monitorix/ +# failJSON: { "time": "2021-04-14T11:26:08", "match": true, "host": "127.0.0.1", "desc": "authentication error" } Wed Apr 14 11:26:08 2021 - AUTHERR - [127.0.0.1] Authentication error: /monitorix/ -Wed Apr 14 11:26:09 2021 - AUTHERR - [127.0.0.1] Authentication error: /monitorix/ From ab0847e2d59611dd81e2f6a1dd30c1f136dce958 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Wed, 14 Apr 2021 13:06:58 +0200 Subject: [PATCH 104/240] more precise anchored RE (also combining all 3 REs in a single regex) --- config/filter.d/monitorix.conf | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/config/filter.d/monitorix.conf b/config/filter.d/monitorix.conf index 3979ed43..ff69f1bc 100644 --- a/config/filter.d/monitorix.conf +++ b/config/filter.d/monitorix.conf @@ -16,9 +16,7 @@ _daemon = monitorix-httpd # (?:::f{4,6}:)?(?P\S+) # Values: TEXT # -failregex = NOTEXIST - \[\] .* - AUTHERR - \[\] .* - NOTALLOWED - \[\] .* +failregex = ^(?:\s+-)?\s*(?:NOTEXIST|AUTHERR|NOTALLOWED) - \b # Option: ignoreregex # Notes.: regex to ignore. If this regex matches, the line is ignored. From f0214b3d36ea08b318c52a2d51fce50bf9ec0cce Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 20 Apr 2021 18:13:40 +0200 Subject: [PATCH 105/240] filter.d/sendmail-reject.conf: fixed regex to consider "Connection rate limit exceeded" with different combination of arguments --- config/filter.d/sendmail-reject.conf | 4 ++-- fail2ban/tests/files/logs/sendmail-reject | 3 +++ 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/config/filter.d/sendmail-reject.conf b/config/filter.d/sendmail-reject.conf index e8b766c5..966d880c 100644 --- a/config/filter.d/sendmail-reject.conf +++ b/config/filter.d/sendmail-reject.conf @@ -21,12 +21,12 @@ before = common.conf _daemon = (?:(sm-(mta|acceptingconnections)|sendmail)) __prefix_line = %(known/__prefix_line)s(?:\w{14,20}: )? -addr = (?:IPv6:|) +addr = (?:(?:IPv6:)?|) prefregex = ^%(__prefix_line)s.+$ cmnfailre = ^ruleset=check_rcpt, arg1=(?P<\S+@\S+>), relay=(\S+ )?\[%(addr)s\](?: \(may be forged\))?, reject=(550 5\.7\.1 (?P=email)\.\.\. Relaying denied\. (IP name possibly forged \[(\d+\.){3}\d+\]|Proper authentication required\.|IP name lookup failed \[(\d+\.){3}\d+\])|553 5\.1\.8 (?P=email)\.\.\. Domain of sender address \S+ does not exist|550 5\.[71]\.1 (?P=email)\.\.\. (Rejected: .*|User unknown))$ - ^ruleset=check_relay, arg1=(?P\S+), arg2=%(addr)s, relay=((?P=dom) )?\[(\d+\.){3}\d+\](?: \(may be forged\))?, reject=421 4\.3\.2 (Connection rate limit exceeded\.|Too many open connections\.)$ + ^ruleset=check_relay(?:, arg\d+=\S*)*, relay=(\S+ )?\[%(addr)s\](?: \(may be forged\))?, reject=421 4\.3\.2 (Connection rate limit exceeded\.|Too many open connections\.)$ ^rejecting commands from (\S* )?\[%(addr)s\] due to pre-greeting traffic after \d+ seconds$ ^(?:\S+ )?\[%(addr)s\]: (?:(?i)expn|vrfy) \S+ \[rejected\]$ ^<[^@]+@[^>]+>\.\.\. No such user here$ diff --git a/fail2ban/tests/files/logs/sendmail-reject b/fail2ban/tests/files/logs/sendmail-reject index 99c1877c..ed3143a8 100644 --- a/fail2ban/tests/files/logs/sendmail-reject +++ b/fail2ban/tests/files/logs/sendmail-reject @@ -40,6 +40,9 @@ Feb 27 15:49:07 batman sm-mta[88390]: ruleset=check_relay, arg1=189-30-205-74.pa # failJSON: { "time": "2005-02-19T18:01:50", "match": true , "host": "196.213.73.146" } Feb 19 18:01:50 batman sm-mta[78152]: ruleset=check_relay, arg1=[196.213.73.146], arg2=196.213.73.146, relay=[196.213.73.146], reject=421 4.3.2 Connection rate limit exceeded. +# failJSON: { "time": "2005-02-19T20:17:12", "match": true , "host": "192.0.2.123" } +Feb 19 20:17:12 server sm-mta[201892]: ruleset=check_relay, arg1=[192.0.2.123], arg2=192.0.2.123, relay=host.example.com [192.0.2.123] (may be forged), reject=421 4.3.2 Connection rate limit exceeded. + # failJSON: { "time": "2005-02-27T10:53:06", "match": true , "host": "209.15.212.253" } Feb 27 10:53:06 batman sm-mta[44307]: s1R9r60D044307: rejecting commands from [209.15.212.253] due to pre-greeting traffic after 0 seconds # failJSON: { "time": "2005-02-27T10:53:07", "match": true , "host": "1.2.3.4" } From d3f5d2d52b1e1b1577dbf29e931fbdbe59f56109 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 21 Apr 2021 11:50:07 +0200 Subject: [PATCH 106/240] documentation (interpolation tags) --- man/jail.conf.5 | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/man/jail.conf.5 b/man/jail.conf.5 index 788fad2b..5f29161d 100644 --- a/man/jail.conf.5 +++ b/man/jail.conf.5 @@ -481,13 +481,29 @@ is the regex (\fBreg\fRular \fBex\fRpression) that will match failed attempts. T .IP \fI\fR - regex for IPv4 addresses. .IP -\fI\fR - regex for IPv6 addresses (also IP enclosed in brackets). +\fI\fR - regex for IPv6 addresses. .IP \fI\fR - regex to match hostnames. .IP \fI\fR - helper regex to match CIDR (simple integer form of net-mask). .IP \fI\fR - regex to match sub-net adresses (in form of IP/CIDR, also single IP is matched, so part /CIDR is optional). +.IP +\fI...\fR - free regex capturing group targeting identifier used for ban (instead of IP address or hostname). +.IP +\fI...\fR - free regex capturing named group stored in ticket, which can be used in action. +.nf +For example \fI[^@]+\fR matches and stores a user name, that can be used in action with interpolation tag \fI\fR. +.IP +\fI...\fR - free regex capturing alternative named group stored in ticket. +.nf +For example first found matched value defined in regex as \fI\fR, \fI\fR or \fI\fR would be stored as (if direct match is not found or empty). +.PP +Every of abovementioned tags can be specified in \fBprefregex\fR and in \fBfailregex\fR, thereby if specified in both, the value matched in \fBfailregex\fR overwrites a value matched in \fBprefregex\fR. +.TQ +All standard tags like IP4 or IP6 can be also specified with custom regex using \fI...\fR syntax, for example \fI(?:ip4:\\S+|ip6:\\S+)\fR. +.TQ +Tags \fI\fR, \fI\fR and \fI\fR would also match the IP address enclosed in square brackets. .PP \fBNOTE:\fR the \fBfailregex\fR will be applied to the remaining part of message after \fBprefregex\fR processing (if specified), which in turn takes place after \fBdatepattern\fR processing (whereby the string of timestamp matching the best pattern, cut out from the message). .PP From 319cfefac2b57c9ca7c9e4ce91b252dd7e20cfee Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 27 Apr 2021 13:34:22 +0200 Subject: [PATCH 107/240] fix travis build (unsupported pythons and pypy versions), update 3.10 in GH actions --- .github/workflows/main.yml | 2 +- .travis.yml | 10 +--------- 2 files changed, 2 insertions(+), 10 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 262448c2..4ea0e7eb 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -22,7 +22,7 @@ jobs: runs-on: ubuntu-20.04 strategy: matrix: - python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, '3.10.0-alpha.5', pypy2, pypy3] + python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, '3.10.0-beta.1', pypy2, pypy3] fail-fast: false # Steps represent a sequence of tasks that will be executed as part of the job steps: diff --git a/.travis.yml b/.travis.yml index 064b678b..398c120a 100644 --- a/.travis.yml +++ b/.travis.yml @@ -10,16 +10,8 @@ dist: xenial matrix: fast_finish: true include: - - python: 2.6 - dist: trusty # required for Python 2.6 - python: 2.7 - dist: trusty # required for packages like gamin - name: 2.7 (trusty) - - python: 2.7 - name: 2.7 (xenial) - - python: pypy - - python: 3.3 - dist: trusty + #- python: pypy - python: 3.4 - python: 3.5 - python: 3.6 From e4e7a83cffb4fb5cbb35c204795fba45c8f40c61 Mon Sep 17 00:00:00 2001 From: usernamepi <53445688+usernamepi@users.noreply.github.com> Date: Thu, 6 May 2021 13:44:36 +0200 Subject: [PATCH 108/240] Update ufw.conf Prerequisites: * The ss command is available, kernel is compiled with option CONFIG_INET_DIAG_DESTROY. * Ufw version is => 0.36 (released in 2018) * Now using "prepend" instead of "insert" to be able to handle IPv6 addresses correctly. The current action will fail for IPv6 addresses. * Now application names containing a space should handled correctly, solves https://github.com/fail2ban/fail2ban/pull/1532 * Now closing IPv4 and IPv6 connections (if any) from the ip that is being banned. The current action will leave them open. Using ss to accomplish this. For this to work the kernel needs to be compiled with the CONFIG_INET_DIAG_DESTROY option. My system apparently is compiled that way. --- config/action.d/ufw.conf | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/config/action.d/ufw.conf b/config/action.d/ufw.conf index d2f731f2..b47fa7e7 100644 --- a/config/action.d/ufw.conf +++ b/config/action.d/ufw.conf @@ -13,17 +13,26 @@ actionstop = actioncheck = -actionban = [ -n "" ] && app="app " - ufw insert from to $app +# ufw does "quickly process packets for which we already have a connection" in before.rules, +# therefore all related sockets should be closed +# actionban is using `ss` to do so, this only handles IPv4 and IPv6. -actionunban = [ -n "" ] && app="app " - ufw delete from to $app +actionban = if [ -n "" ] && ufw app info "" + then + ufw prepend from to app "" comment "" + else + ufw prepend from to comment "" + fi + ss -K dst [] + +actionunban = if [ -n "" ] && ufw app info "" + then + ufw delete from to app "" + else + ufw delete from to + fi [Init] -# Option: insertpos -# Notes.: The position number in the firewall list to insert the block rule -insertpos = 1 - # Option: blocktype # Notes.: reject or deny blocktype = reject From 5debaa4cac2723fa863ede9ed32c19cc82c71786 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Thu, 6 May 2021 20:23:58 +0200 Subject: [PATCH 109/240] option "add", can be set to "insert " instead of prepend (customization or backwards compat) --- config/action.d/ufw.conf | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/config/action.d/ufw.conf b/config/action.d/ufw.conf index b47fa7e7..bf06fe67 100644 --- a/config/action.d/ufw.conf +++ b/config/action.d/ufw.conf @@ -19,9 +19,9 @@ actioncheck = actionban = if [ -n "" ] && ufw app info "" then - ufw prepend from to app "" comment "" + ufw from to app "" comment "" else - ufw prepend from to comment "" + ufw from to comment "" fi ss -K dst [] @@ -33,6 +33,10 @@ actionunban = if [ -n "" ] && ufw app info "" fi [Init] +# Option: add +# Notes.: can be set to "insert 1" to insert a rule at certain position (here 1): +add = prepend + # Option: blocktype # Notes.: reject or deny blocktype = reject From 8f6a8df3a45395620e434fd15b4ede694a1d00aa Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Thu, 6 May 2021 21:47:06 +0200 Subject: [PATCH 110/240] added new options `kill-mode` and `kill`, which makes the drop of all connections optional --- config/action.d/ufw.conf | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/config/action.d/ufw.conf b/config/action.d/ufw.conf index bf06fe67..cf8c22be 100644 --- a/config/action.d/ufw.conf +++ b/config/action.d/ufw.conf @@ -23,7 +23,7 @@ actionban = if [ -n "" ] && ufw app info "" else ufw from to comment "" fi - ss -K dst [] + actionunban = if [ -n "" ] && ufw app info "" then @@ -32,6 +32,21 @@ actionunban = if [ -n "" ] && ufw app info "" ufw delete from to fi +# Option: kill-mode +# Notes.: can be set to ss (may be extended later with other modes) to immediately drop all connections from banned IP, default empty (no kill) +# Example: banaction = ufw[kill-mode=ss] +kill-mode = + +# intern conditional parameter used to provide killing mode after ban: +_kill_ = +_kill_ss = ss -K dst "[]" + +# Option: kill +# Notes.: can be used to specify custom killing feature, by default depending on option kill-mode +# Examples: banaction = ufw[kill='ss -K "( sport = :http || sport = :https )" dst "[]"'] + banaction = ufw[kill='cutter ""'] +kill = <_kill_> + [Init] # Option: add # Notes.: can be set to "insert 1" to insert a rule at certain position (here 1): From 2958ad8636e9b3082c49537d7a56ae3b8cf12012 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Thu, 6 May 2021 22:19:38 +0200 Subject: [PATCH 111/240] Update ChangeLog --- ChangeLog | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/ChangeLog b/ChangeLog index 4b0733d7..175d087a 100644 --- a/ChangeLog +++ b/ChangeLog @@ -20,6 +20,9 @@ ver. 1.0.1-dev-1 (20??/??/??) - development nightly edition ### Fixes * readline fixed to consider interim new-line character as part of code point in multi-byte logs (e. g. unicode encoding like utf-16be, utf-16le); +* `action.d/ufw.conf`: + - fixed handling on IPv6 (using prepend, gh-2331, gh-3018) + - application names containing spaces can be used now (gh-656, gh-1532, gh-3018) * `filter.d/drupal-auth.conf` more strict regex, extended to match "Login attempt failed from" (gh-2742) ### New Features and Enhancements @@ -29,6 +32,9 @@ ver. 1.0.1-dev-1 (20??/??/??) - development nightly edition * better recognition of log rotation, better performance by reopen: avoid unnecessary seek to begin of file (and hash calculation) * file filter reads only complete lines (ended with new-line) now, so waits for end of line (for its completion) +* `action.d/ufw.conf` (gh-3018): + - new option `add` (default `prepend`), can be supplied as `insert 1` for ufw versions before v.0.36 (gh-2331, gh-3018) + - new options `kill-mode` and `kill` to drop established connections of intruder (see action for details, gh-3018) * `filter.d/nginx-http-auth.conf` - extended with parameter mode, so additionally to `auth` (or `normal`) mode `fallback` (or combined as `aggressive`) can find SSL errors while SSL handshaking, gh-2881 From 88f779ed241e2fccbaeb65281d336ff0f1706b07 Mon Sep 17 00:00:00 2001 From: usernamepi <53445688+usernamepi@users.noreply.github.com> Date: Thu, 6 May 2021 23:23:39 +0200 Subject: [PATCH 112/240] ufw.conf, amend to #3018 - add missing option for comment (#3019) --- config/action.d/ufw.conf | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/config/action.d/ufw.conf b/config/action.d/ufw.conf index cf8c22be..b5d1213c 100644 --- a/config/action.d/ufw.conf +++ b/config/action.d/ufw.conf @@ -64,6 +64,10 @@ destination = any # Notes.: application from sudo ufw app list application = +# Option: comment +# Notes.: comment for rule added by fail2ban +comment = by Fail2Ban after attempts against + # DEV NOTES: # # Author: Guilhem Lettron From 2918849f9e781fc27a0fd0ba052c1a97d96d02bc Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 7 May 2021 01:10:26 +0200 Subject: [PATCH 113/240] fixes precise year pattern %ExY - accept years 20xx up to current century (using almost the same pattern in tests and production now) --- fail2ban/server/strptime.py | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/fail2ban/server/strptime.py b/fail2ban/server/strptime.py index 69514b20..6f88add1 100644 --- a/fail2ban/server/strptime.py +++ b/fail2ban/server/strptime.py @@ -104,14 +104,14 @@ def _updateTimeRE(): # more precise year patterns, within same century of last year and # the next 3 years (for possible long uptime of fail2ban); thereby - # respect possible run in the test-cases (alternate date used there): - if MyTime.alternateNowTime != 0: - timeRE['ExY'] = r"(?P%s\d)" % _getYearCentRE(cent=(0,3), distance=3) - timeRE['Exy'] = r"(?P%s\d)" % _getYearCentRE(cent=(2,3), distance=3) - else: # accept years: 19xx|2xxx up to current century - timeRE['ExY'] = r"(?P(?:19\d{2}|%s\d))" % _getYearCentRE(cent=(0,3), distance=3, - now=(MyTime.now(), datetime.datetime.fromtimestamp(978393600))) - timeRE['Exy'] = r"(?P\d{2})" + # consider possible run in the test-cases (alternate date used there), + # so accept years: 20xx (from test-date or 2001 up to current century) + timeRE['ExY'] = r"(?P%s\d)" % _getYearCentRE(cent=(0,3), distance=3, + now=(datetime.datetime.now(), datetime.datetime.fromtimestamp( + min(MyTime.alternateNowTime or 978393600, 978393600)) + ) + ) + timeRE['Exy'] = r"(?P\d{2})" _updateTimeRE() From ef5c826c74c160d33d4b79b23d564f944d16a9dd Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 7 May 2021 01:16:48 +0200 Subject: [PATCH 114/240] fixes search for the best datepattern (gh-3020) - e. g. if line is too short, boundaries check for previously known unprecise pattern may fail on incomplete lines (logging break-off, no flush, etc) --- fail2ban/server/datedetector.py | 4 ++-- fail2ban/tests/fail2banregextestcase.py | 12 +++++++++++- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/fail2ban/server/datedetector.py b/fail2ban/server/datedetector.py index ecc9d935..b90e1b26 100644 --- a/fail2ban/server/datedetector.py +++ b/fail2ban/server/datedetector.py @@ -365,10 +365,10 @@ class DateDetector(object): # with space or some special char), otherwise possible collision/pattern switch: if (( line[distance-1:distance] == self.__lastPos[1] or - (line[distance] == self.__lastPos[2] and not self.__lastPos[2].isalnum()) + (line[distance:distance+1] == self.__lastPos[2] and not self.__lastPos[2].isalnum()) ) and ( line[endpos:endpos+1] == self.__lastEndPos[2] or - (line[endpos-1] == self.__lastEndPos[1] and not self.__lastEndPos[1].isalnum()) + (line[endpos-1:endpos] == self.__lastEndPos[1] and not self.__lastEndPos[1].isalnum()) )): # search in line part only: log(logLevel-1, " boundaries are correct, search in part %r", line[distance:endpos]) diff --git a/fail2ban/tests/fail2banregextestcase.py b/fail2ban/tests/fail2banregextestcase.py index 85fe4f15..1c55e227 100644 --- a/fail2ban/tests/fail2banregextestcase.py +++ b/fail2ban/tests/fail2banregextestcase.py @@ -384,7 +384,17 @@ class Fail2banRegexTest(LogCaptureTestCase): "Found a match but no valid date/time found", "Match without a timestamp:", all=True) - self.pruneLog() + def testIncompleteDateTime(self): + # datepattern in followed lines doesn't match previously known pattern + line is too short + # (logging break-off, no flush, etc): + self.assertTrue(_test_exec( + '-o', 'Found-ADDR:', + '192.0.2.1 - - [02/May/2021:18:40:55 +0100] "GET / HTTP/1.1" 302 328 "-" "Mozilla/5.0" "-"\n' + '192.0.2.2 - - [02/May/2021:18:40:55 +0100\n' + '192.0.2.3 - - [02/May/2021:18:40:55', + '^')) + self.assertLogged( + "Found-ADDR:192.0.2.1", "Found-ADDR:192.0.2.2", "Found-ADDR:192.0.2.3", all=True) def testFrmtOutputWrapML(self): unittest.F2B.SkipIfCfgMissing(stock=True) From 4f8427178a805fe7388c5fc8cfb16e2e601af920 Mon Sep 17 00:00:00 2001 From: usernamepi <53445688+usernamepi@users.noreply.github.com> Date: Fri, 7 May 2021 18:23:40 +0200 Subject: [PATCH 115/240] Missing comment "#" (#3022) Missed this ... but the logs showed it. --- config/action.d/ufw.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/action.d/ufw.conf b/config/action.d/ufw.conf index b5d1213c..088a5a9d 100644 --- a/config/action.d/ufw.conf +++ b/config/action.d/ufw.conf @@ -44,7 +44,7 @@ _kill_ss = ss -K dst "[]" # Option: kill # Notes.: can be used to specify custom killing feature, by default depending on option kill-mode # Examples: banaction = ufw[kill='ss -K "( sport = :http || sport = :https )" dst "[]"'] - banaction = ufw[kill='cutter ""'] +# banaction = ufw[kill='cutter ""'] kill = <_kill_> [Init] From 3f9cf278534b7e8051590767c682a0114c5f6907 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Tue, 11 May 2021 13:47:48 +0200 Subject: [PATCH 116/240] filter.d/apache-fakegooglebot.conf: better, more precise regex and datepattern (closes possible weakness like #3013) --- config/filter.d/apache-fakegooglebot.conf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/config/filter.d/apache-fakegooglebot.conf b/config/filter.d/apache-fakegooglebot.conf index 729410ad..ee23656a 100644 --- a/config/filter.d/apache-fakegooglebot.conf +++ b/config/filter.d/apache-fakegooglebot.conf @@ -2,11 +2,11 @@ [Definition] -failregex = ^ .*Googlebot.*$ +failregex = ^\s* \S+ \S+(?: \S+)?\s+\S+ "[A-Z]+ /\S* [^"]*" \d+ \d+ \"[^"]*\" "[^"]*\bGooglebot/[^"]*" ignoreregex = -datepattern = ^[^\[]*\[({DATE}) +datepattern = ^[^\[]*(\[{DATE}\s*\]) {^LN-BEG} # DEV Notes: From 5e3de882af2b8f48e6de728c86b5483aad05a93c Mon Sep 17 00:00:00 2001 From: Ioannis Cherouvim <743305+cherouvim@users.noreply.github.com> Date: Wed, 12 May 2021 14:18:46 +0300 Subject: [PATCH 117/240] docs: Typo. (#3025) --- man/jail.conf.5 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/man/jail.conf.5 b/man/jail.conf.5 index 5f29161d..8ff407f0 100644 --- a/man/jail.conf.5 +++ b/man/jail.conf.5 @@ -271,7 +271,7 @@ effective ban duration (in seconds or time abbreviation format). time interval (in seconds or time abbreviation format) before the current time where failures will count towards a ban. .TP .B maxretry -number of failures that have to occur in the last \fBfindtime\fR seconds to ban then IP. +number of failures that have to occur in the last \fBfindtime\fR seconds to ban the IP. .TP .B backend backend to be used to detect changes in the logpath. From 80b1007a8fece5817f36ba19061d8359201669c5 Mon Sep 17 00:00:00 2001 From: Michael Orlitzky Date: Sun, 15 Jul 2018 13:54:22 -0400 Subject: [PATCH 118/240] files/fail2ban-openrc.init: remove the "showlog" command. The extra "showlog" command in our OpenRC service script was more trouble than it was worth: the only thing it did was call "less" on a log file, and the service script is only guessing at the location of the log file (only the fail2ban server knows its true location). It's not like "/etc/init.d/fail2ban showlog" is that much easier to type than "less /var/log/fail2ban.log" in the first place, so I think the extra complexity (5 more lines in the service script) is not worth it. --- files/fail2ban-openrc.init | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/files/fail2ban-openrc.init b/files/fail2ban-openrc.init index e3ddfe1a..2de5ae33 100755 --- a/files/fail2ban-openrc.init +++ b/files/fail2ban-openrc.init @@ -20,8 +20,7 @@ description="Daemon to ban hosts that cause multiple authentication errors" description_reload="reload configuration" -description_showlog="show fail2ban logs" -extra_started_commands="reload showlog" +extra_started_commands="reload" command="/usr/bin/fail2ban-client" command_args="${FAIL2BAN_OPTIONS}" @@ -60,7 +59,3 @@ reload() { ${command} ${command_args} reload eend $? "Failed to reload ${RC_SVCNAME}" } - -showlog(){ - less /var/log/fail2ban.log -} From 654fda8a50f65c6b329d75cbac91a50aa5a8a8f5 Mon Sep 17 00:00:00 2001 From: Michael Orlitzky Date: Sun, 15 Jul 2018 14:08:33 -0400 Subject: [PATCH 119/240] files/fail2ban-openrc*: let start-stop-daemon manage the server. There are two ways that it would make sense to write the OpenRC service script for fail2ban: 1. Use the fail2ban-client program to stop, start, reload, etc. the server; and try to figure out whether or not it worked afterwards. 2. Use the start-stop-daemon program built into OpenRC to manage the fail2ban-server process. This works only for starting and stopping, because the "reload" command is sent over an undocumented protocol, but has the benefit that you get immediate feedback about the result of calling fail2ban-server. The existing service script combined the two in a way that appeared to work, but didn't make too much sense. It used start-stop-daemon to initiate the fail2ban-client program with either a "start" or "stop" argument. So long as everything goes fine, that appears to work. But the start-stop-daemon is not actually monitoring the fail2ban-client program; it's supposed to be monitoring the fail2ban-server process that gets started as side-effect. The existing stop() function does not do quite what you'd expect; for example the "stop" command is never sent. Again, the daemon does ultimately get stopped so long as the hard-coded PID file contains what you think it does -- so it "works" -- but is misleading. This commit changes everything to use the second approach above, where start-stop-daemon manages everything. This was done mainly to simplify the service script, because now the default start() and stop() phases can be used, allowing us to delete them from our copy. One might worry that there is some special magic behind "fail2ban-client start" and "fail2ban-client stop", however that does not appear to be the case. Admittedly, if in the future those two commands begin to do something nonstandard, the service script would need to be changed again to take the first approach above and use fail2ban-client for everything. --- files/fail2ban-openrc.conf | 2 +- files/fail2ban-openrc.init | 31 +++++++++---------------------- 2 files changed, 10 insertions(+), 23 deletions(-) diff --git a/files/fail2ban-openrc.conf b/files/fail2ban-openrc.conf index 1a2450e2..8493b03c 100644 --- a/files/fail2ban-openrc.conf +++ b/files/fail2ban-openrc.conf @@ -1,2 +1,2 @@ -# For available options, plase run "fail2ban-client -h". +# For available options, plase run "fail2ban-server --help". #FAIL2BAN_OPTIONS="" diff --git a/files/fail2ban-openrc.init b/files/fail2ban-openrc.init index 2de5ae33..21e251db 100755 --- a/files/fail2ban-openrc.init +++ b/files/fail2ban-openrc.init @@ -18,13 +18,15 @@ # Author: Sireyessire, Cyril Jaquier # -description="Daemon to ban hosts that cause multiple authentication errors" +description="Ban hosts that cause multiple authentication errors" description_reload="reload configuration" extra_started_commands="reload" -command="/usr/bin/fail2ban-client" -command_args="${FAIL2BAN_OPTIONS}" +# The fail2ban-client program is also capable of starting and stopping +# the server, but things are simpler if we let start-stop-daemon do it. +command="/usr/bin/fail2ban-server" pidfile="/run/${RC_SVCNAME}/${RC_SVCNAME}.pid" +command_args="${FAIL2BAN_OPTIONS} -p ${pidfile}" retry="30" depend() { @@ -34,28 +36,13 @@ depend() { start_pre() { checkpath -d "${pidfile%/*}" || return 1 - - # Remove stale socket after system crash, Gentoo bug 347477 - rm -f /var/run/fail2ban/fail2ban.sock || return 1 -} - -start() { - ebegin "Starting ${RC_SVCNAME}" - - start-stop-daemon --start --pidfile "${pidfile}" \ - -- ${command} ${command_args} start - eend $? "Failed to start ${RC_SVCNAME}" -} - -stop() { - ebegin "Stopping ${RC_SVCNAME}" - start-stop-daemon --stop --pidfile "${pidfile}" --retry "${retry}" \ - -- ${command} ${command_args} stop - eend $? "Failed to stop ${RC_SVCNAME}" } reload() { + # The fail2ban-client uses an undocumented protocol to tell + # the server to reload(), so we have to use it here rather + # than e.g. sending a signal to the server daemon. ebegin "Reloading ${RC_SVCNAME}" - ${command} ${command_args} reload + "${command%/*}/fail2ban-client" ${command_args} reload eend $? "Failed to reload ${RC_SVCNAME}" } From 4e7419e71f82481d321ac8c1b05ad0175d27e32d Mon Sep 17 00:00:00 2001 From: Michael Orlitzky Date: Sun, 15 Jul 2018 14:12:20 -0400 Subject: [PATCH 120/240] files/fail2ban-openrc.conf: add back the "-x" example. I've removed the stale socket cleanup from our OpenRC service script: * Cleaning up stale sockets isn't really the job of the service script. * The ability to ignore a stale socket is already built into the server. With it gone, maybe the "-x" is a useful example to have in the conf file (although it's commented-out by default, anyway). --- files/fail2ban-openrc.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/files/fail2ban-openrc.conf b/files/fail2ban-openrc.conf index 8493b03c..9454ef68 100644 --- a/files/fail2ban-openrc.conf +++ b/files/fail2ban-openrc.conf @@ -1,2 +1,2 @@ # For available options, plase run "fail2ban-server --help". -#FAIL2BAN_OPTIONS="" +#FAIL2BAN_OPTIONS="-x" From e6a9f109c5349041987e64909917256e9c6e4229 Mon Sep 17 00:00:00 2001 From: Michael Orlitzky Date: Sun, 15 Jul 2018 14:22:35 -0400 Subject: [PATCH 121/240] files/fail2ban-openrc.init: force the socket location in the service script. The socket location needs to be set in the service script for the same reason that the PID file location does: because the service script is taking responsibility for ensuring that its parent directory exists and has the correct permissions. We can't do that if the end user is allowed to move the PID file or socket somewhere else (without parsing the config file, which has other security implications). --- files/fail2ban-openrc.init | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/files/fail2ban-openrc.init b/files/fail2ban-openrc.init index 21e251db..8473da26 100755 --- a/files/fail2ban-openrc.init +++ b/files/fail2ban-openrc.init @@ -22,11 +22,23 @@ description="Ban hosts that cause multiple authentication errors" description_reload="reload configuration" extra_started_commands="reload" +# Can't (and shouldn't) be changed by the end-user. +FAIL2BAN_RUNDIR="/run/${RC_SVCNAME}" +FAIL2BAN_SOCKET="${FAIL2BAN_RUNDIR}/${RC_SVCNAME}.sock" + +# This should be replaced by the build system, eventually. +FAIL2BAN_BINDIR="/usr/bin" + # The fail2ban-client program is also capable of starting and stopping # the server, but things are simpler if we let start-stop-daemon do it. -command="/usr/bin/fail2ban-server" -pidfile="/run/${RC_SVCNAME}/${RC_SVCNAME}.pid" -command_args="${FAIL2BAN_OPTIONS} -p ${pidfile}" +command="${FAIL2BAN_BINDIR}/fail2ban-server" +pidfile="${FAIL2BAN_RUNDIR}/${RC_SVCNAME}.pid" + +# We force the pidfile/socket location in this service script because +# we're taking responsibility for ensuring that their parent directory +# exists and has the correct permissions (which we can't do if the +# user is allowed to change them). +command_args="${FAIL2BAN_OPTIONS} -p ${pidfile} -s ${FAIL2BAN_SOCKET}" retry="30" depend() { @@ -35,7 +47,7 @@ depend() { } start_pre() { - checkpath -d "${pidfile%/*}" || return 1 + checkpath -d "${FAIL2BAN_RUNDIR}" || return 1 } reload() { @@ -43,6 +55,6 @@ reload() { # the server to reload(), so we have to use it here rather # than e.g. sending a signal to the server daemon. ebegin "Reloading ${RC_SVCNAME}" - "${command%/*}/fail2ban-client" ${command_args} reload + "${FAIL2BAN_BINDIR}/fail2ban-client" ${command_args} reload eend $? "Failed to reload ${RC_SVCNAME}" } From dd0f3487578e6feb54a54e774fcc40e390d769f8 Mon Sep 17 00:00:00 2001 From: Michael Orlitzky Date: Sun, 15 Jul 2018 14:32:07 -0400 Subject: [PATCH 122/240] files/fail2ban-openrc.init: replace @BINDIR@ at build-time. This commit renames fail2ban-openrc.init to fail2ban-openrc.init.in, and replaces the hard-coded value "/usr/bin" with "@BINDIR@" therein. At build-time, setup.py will replace that string with the correct value, and rename the file (without the ".in" suffix). This mimics the procedure done for "fail2ban-service.in" entirely. --- MANIFEST | 2 +- ...an-openrc.init => fail2ban-openrc.init.in} | 7 +--- setup.py | 39 ++++++++++--------- 3 files changed, 24 insertions(+), 24 deletions(-) rename files/{fail2ban-openrc.init => fail2ban-openrc.init.in} (91%) diff --git a/MANIFEST b/MANIFEST index ca9be123..5194e75b 100644 --- a/MANIFEST +++ b/MANIFEST @@ -376,7 +376,7 @@ files/fail2ban-tmpfiles.conf files/fail2ban.upstart files/gen_badbots files/fail2ban-openrc.conf -files/fail2ban-openrc.init +files/fail2ban-openrc.init.in files/ipmasq-ZZZzzz_fail2ban.rul files/logwatch/fail2ban files/logwatch/fail2ban-0.8.log diff --git a/files/fail2ban-openrc.init b/files/fail2ban-openrc.init.in similarity index 91% rename from files/fail2ban-openrc.init rename to files/fail2ban-openrc.init.in index 8473da26..f59a0a20 100755 --- a/files/fail2ban-openrc.init +++ b/files/fail2ban-openrc.init.in @@ -26,12 +26,9 @@ extra_started_commands="reload" FAIL2BAN_RUNDIR="/run/${RC_SVCNAME}" FAIL2BAN_SOCKET="${FAIL2BAN_RUNDIR}/${RC_SVCNAME}.sock" -# This should be replaced by the build system, eventually. -FAIL2BAN_BINDIR="/usr/bin" - # The fail2ban-client program is also capable of starting and stopping # the server, but things are simpler if we let start-stop-daemon do it. -command="${FAIL2BAN_BINDIR}/fail2ban-server" +command="@BINDIR@/fail2ban-server" pidfile="${FAIL2BAN_RUNDIR}/${RC_SVCNAME}.pid" # We force the pidfile/socket location in this service script because @@ -55,6 +52,6 @@ reload() { # the server to reload(), so we have to use it here rather # than e.g. sending a signal to the server daemon. ebegin "Reloading ${RC_SVCNAME}" - "${FAIL2BAN_BINDIR}/fail2ban-client" ${command_args} reload + "@BINDIR@/fail2ban-client" ${command_args} reload eend $? "Failed to reload ${RC_SVCNAME}" } diff --git a/setup.py b/setup.py index 8da29268..e18c99f3 100755 --- a/setup.py +++ b/setup.py @@ -95,24 +95,27 @@ class install_scripts_f2b(install_scripts): if install_dir.startswith(root): install_dir = install_dir[len(root):] except: # pragma: no cover - print('WARNING: Cannot find root-base option, check the bin-path to fail2ban-scripts in "fail2ban.service".') - print('Creating %s/fail2ban.service (from fail2ban.service.in): @BINDIR@ -> %s' % (buildroot, install_dir)) - with open(os.path.join(source_dir, 'files/fail2ban.service.in'), 'r') as fn: - lines = fn.readlines() - fn = None - if not dry_run: - fn = open(os.path.join(buildroot, 'fail2ban.service'), 'w') - try: - for ln in lines: - ln = re.sub(r'@BINDIR@', lambda v: install_dir, ln) - if dry_run: - sys.stdout.write(' | ' + ln) - continue - fn.write(ln) - finally: - if fn: fn.close() - if dry_run: - print(' `') + print('WARNING: Cannot find root-base option, check the bin-path to fail2ban-scripts in "fail2ban.service" and "fail2ban-openrc.init".') + + scripts = ['fail2ban.service', 'fail2ban-openrc.init'] + for script in scripts: + print('Creating %s/%s (from %s.in): @BINDIR@ -> %s' % (buildroot, script, script, install_dir)) + with open(os.path.join(source_dir, 'files/%s.in' % script), 'r') as fn: + lines = fn.readlines() + fn = None + if not dry_run: + fn = open(os.path.join(buildroot, script), 'w') + try: + for ln in lines: + ln = re.sub(r'@BINDIR@', lambda v: install_dir, ln) + if dry_run: + sys.stdout.write(' | ' + ln) + continue + fn.write(ln) + finally: + if fn: fn.close() + if dry_run: + print(' `') # Wrapper to specify fail2ban own options: From 36a7abe82fa17ed4cb697f745a48f77afd0618c8 Mon Sep 17 00:00:00 2001 From: Michael Orlitzky Date: Sun, 15 Jul 2018 16:28:20 -0400 Subject: [PATCH 123/240] files/fail2ban-openrc.init.in: mention that "reload" doesn't drop bans. The description of the "reload" OpenRC command just said that it would reload the configuration, which is true but not totally helpful. This commit updates it to mention that your existing bans won't be dropped, in contrast with the "restart" command that does drop your bans. --- files/fail2ban-openrc.init.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/files/fail2ban-openrc.init.in b/files/fail2ban-openrc.init.in index f59a0a20..a2f4d34f 100755 --- a/files/fail2ban-openrc.init.in +++ b/files/fail2ban-openrc.init.in @@ -19,7 +19,7 @@ # description="Ban hosts that cause multiple authentication errors" -description_reload="reload configuration" +description_reload="reload configuration without dropping bans" extra_started_commands="reload" # Can't (and shouldn't) be changed by the end-user. From 87e9cff065c9add436e4a9488239a89835b1bc48 Mon Sep 17 00:00:00 2001 From: Michael Orlitzky Date: Sun, 15 Jul 2018 16:49:49 -0400 Subject: [PATCH 124/240] files/fail2ban-openrc.init.in: remove redundant "return" from start_pre. OpenRC functions will exit with the return code from the last command by default, so there's no need for the "|| return 1" in our single-line start_pre() phase. --- files/fail2ban-openrc.init.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/files/fail2ban-openrc.init.in b/files/fail2ban-openrc.init.in index a2f4d34f..ad977274 100755 --- a/files/fail2ban-openrc.init.in +++ b/files/fail2ban-openrc.init.in @@ -44,7 +44,7 @@ depend() { } start_pre() { - checkpath -d "${FAIL2BAN_RUNDIR}" || return 1 + checkpath -d "${FAIL2BAN_RUNDIR}" } reload() { From 4d2841832cc3b38175bd0f0c3e6b8e143b5a0426 Mon Sep 17 00:00:00 2001 From: Michael Orlitzky Date: Sun, 15 Jul 2018 17:02:25 -0400 Subject: [PATCH 125/240] files/fail2ban-openrc.init.in: don't restart() with a broken config. This commit adds a new function checkconfig() to the OpenRC service script. All it does is run the server with the "--test" flag in addition to the usual command-line arguments. The new command is not user-facing, but lets us avoid restarting the daemon with a broken config. That helps when the user changes his configuration while the daemon is running, and then tries to restart() not knowing that the new config is broken. A priori, we would stop the daemon and then the error would only become visible when the subsequent start() command failed. Refusing to stop() with a broken configuration is a nicer thing to do. --- files/fail2ban-openrc.init.in | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/files/fail2ban-openrc.init.in b/files/fail2ban-openrc.init.in index ad977274..20465254 100755 --- a/files/fail2ban-openrc.init.in +++ b/files/fail2ban-openrc.init.in @@ -43,14 +43,39 @@ depend() { after iptables } +checkconfig() { + "${command}" ${command_args} --test +} + start_pre() { + # If this isn't a restart, make sure that the user's config isn't + # busted before we try to start the daemon (this will produce + # better error messages than if we just try to start it blindly). + # + # If, on the other hand, this *is* a restart, then the stop_pre + # action will have ensured that the config is usable and we don't + # need to do that again. + if [ "${RC_CMD}" != "restart" ] ; then + checkconfig || return $? + fi checkpath -d "${FAIL2BAN_RUNDIR}" } +stop_pre() { + # If this is a restart, check to make sure the user's config + # isn't busted before we stop the running daemon. + if [ "${RC_CMD}" = "restart" ] ; then + checkconfig || return $? + fi +} + reload() { # The fail2ban-client uses an undocumented protocol to tell # the server to reload(), so we have to use it here rather - # than e.g. sending a signal to the server daemon. + # than e.g. sending a signal to the server daemon. Note that + # the reload will fail (on the server side) if the new config + # is invalid; we therefore don't need to test it ourselves + # with checkconfig() before initiating the reload. ebegin "Reloading ${RC_SVCNAME}" "@BINDIR@/fail2ban-client" ${command_args} reload eend $? "Failed to reload ${RC_SVCNAME}" From 78dddb75e6f3f8f433a6f08add13bffbad90fc7c Mon Sep 17 00:00:00 2001 From: Michael Orlitzky Date: Sun, 15 Jul 2018 18:28:37 -0400 Subject: [PATCH 126/240] files/fail2ban-openrc.init.in: add a comment about @RUNDIR@ in the future. --- files/fail2ban-openrc.init.in | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/files/fail2ban-openrc.init.in b/files/fail2ban-openrc.init.in index 20465254..2c56ee3a 100755 --- a/files/fail2ban-openrc.init.in +++ b/files/fail2ban-openrc.init.in @@ -23,6 +23,10 @@ description_reload="reload configuration without dropping bans" extra_started_commands="reload" # Can't (and shouldn't) be changed by the end-user. +# +# Note that @BINDIR@ is already supplied by the build system. Some +# day, it might be nice to have @RUNDIR@ supplied by the build system +# as well, so that we don't have to hard-code /run here. FAIL2BAN_RUNDIR="/run/${RC_SVCNAME}" FAIL2BAN_SOCKET="${FAIL2BAN_RUNDIR}/${RC_SVCNAME}.sock" From 2367ad115c1d20daaca886ebe4db81edf06df577 Mon Sep 17 00:00:00 2001 From: j-marz Date: Thu, 20 May 2021 09:15:45 +1000 Subject: [PATCH 127/240] fixed typo in comment --- config/filter.d/zoneminder.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/filter.d/zoneminder.conf b/config/filter.d/zoneminder.conf index 7da40968..b3c0be72 100644 --- a/config/filter.d/zoneminder.conf +++ b/config/filter.d/zoneminder.conf @@ -21,6 +21,6 @@ ignoreregex = # Notes: # Tested on Zoneminder 1.29 and 1.35.21 # -# Zoneminer versions > 1.3x use "ERR" and < 1.3x use "WAR" level logs, so i've kept both for compatibility reasons +# Zoneminder versions > 1.3x use "ERR" and < 1.3x use "WAR" level logs, so i've kept both for compatibility reasons # # Author: John Marzella From ec4e0dd65b1f2483a3e2413a61442eb544ae8c16 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Fri, 21 May 2021 13:00:24 +0200 Subject: [PATCH 128/240] padding with space, prefregex, regex review (simplifying, capture user name, consider possible space char in user name) --- config/filter.d/zoneminder.conf | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/config/filter.d/zoneminder.conf b/config/filter.d/zoneminder.conf index b3c0be72..8e8ed432 100644 --- a/config/filter.d/zoneminder.conf +++ b/config/filter.d/zoneminder.conf @@ -6,15 +6,16 @@ before = apache-common.conf [Definition] # patterns: [Mon Mar 28 16:50:49.522240 2016] [:error] [pid 1795] [client 10.1.1.1:50700] WAR [Login denied for user "username1"], referer: https://zoneminder/ -# [Sun Mar 28 16:53:00.472693 2021] [php7:notice] [pid 11328] [client 10.1.1.1:39568] ERR [Could not retrieve user test details], referer: https://zm/ -# [Sun Mar 28 16:59:14.150625 2021] [php7:notice] [pid 11336] [client 10.1.1.1:39654] ERR [Login denied for user "john"], referer: https://zm/ +# [Sun Mar 28 16:53:00.472693 2021] [php7:notice] [pid 11328] [client 10.1.1.1:39568] ERR [Could not retrieve user test details], referer: https://zm/ +# [Sun Mar 28 16:59:14.150625 2021] [php7:notice] [pid 11336] [client 10.1.1.1:39654] ERR [Login denied for user "john"], referer: https://zm/ # # Option: failregex # Notes.: regex to match the login failure and non-existent user error messages in the logfile. -failregex = ^%(_apache_error_client)s WAR \[Login denied for user "[^"]*"\] - ^%(_apache_error_client)s ERR \[Login denied for user "[^"]*"\] - ^%(_apache_error_client)s ERR \[Could not retrieve user \w* details\] +prefregex = ^%(_apache_error_client)s (?:ERR|WAR) \[(?:Login denied|Could not retrieve).*$ + +failregex = ^\[Login denied for user "[^"]*"\] + ^\[Could not retrieve user \S* ignoreregex = From 1627d4f573a1c763d00bee881a245e4c29c3f0b1 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 25 May 2021 23:15:31 +0200 Subject: [PATCH 129/240] filter.d/sendmail-auth.conf: user not found, closes gh-3030 --- config/filter.d/sendmail-auth.conf | 1 + fail2ban/tests/files/logs/sendmail-auth | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/config/filter.d/sendmail-auth.conf b/config/filter.d/sendmail-auth.conf index 84fcbdda..4601a1fc 100644 --- a/config/filter.d/sendmail-auth.conf +++ b/config/filter.d/sendmail-auth.conf @@ -16,6 +16,7 @@ prefregex = ^%(__prefix_line)s.+$ failregex = ^(\S+ )?\[%(addr)s\]( \(may be forged\))?: possible SMTP attack: command=AUTH, count=\d+$ ^AUTH failure \(LOGIN\):(?: [^:]+:)? authentication failure: checkpass failed, user=(?:\S+|.*?), relay=(?:\S+ )?\[%(addr)s\](?: \(may be forged\))?$ + ^AUTH failure \([^\)]+\):(?: [^:]+:)? user not found: [^,]*, user=(?:\S+|.*?), relay=(?:\S+ )?\[%(addr)s\](?: \(may be forged\))?$ ignoreregex = journalmatch = _SYSTEMD_UNIT=sendmail.service diff --git a/fail2ban/tests/files/logs/sendmail-auth b/fail2ban/tests/files/logs/sendmail-auth index 93bf0b14..baa0a33f 100644 --- a/fail2ban/tests/files/logs/sendmail-auth +++ b/fail2ban/tests/files/logs/sendmail-auth @@ -22,3 +22,8 @@ Feb 24 14:00:01 server sendmail[3529566]: xA32R2PQ3529566: [192.0.2.2]: possible Feb 25 04:02:27 relay1 sendmail[16664]: 06I02CNi016764: AUTH failure (LOGIN): authentication failure (-13) SASL(-13): authentication failure: checkpass failed, user=user@example.com, relay=example.com [192.0.2.3] (may be forged) # failJSON: { "time": "2005-02-25T04:02:28", "match": true , "host": "192.0.2.4", "desc": "injection attempt on user name" } Feb 25 04:02:28 relay1 sendmail[16665]: 06I02CNi016765: AUTH failure (LOGIN): authentication failure (-13) SASL(-13): authentication failure: checkpass failed, user=criminal, relay=[192.0.2.100], relay=[192.0.2.4] (may be forged) + +# failJSON: { "time": "2005-05-24T01:58:40", "match": true , "host": "192.0.2.5", "desc": "user not found (gh-3030)" } +May 24 01:58:40 server sm-mta[65696]: 14NNwaRl065696: AUTH failure (DIGEST-MD5): user not found (-20) SASL(-13): user not found: unable to canonify user and get auxprops, user=scanner, relay=[192.0.2.5] +# failJSON: { "time": "2005-05-24T01:59:07", "match": true , "host": "192.0.2.6", "desc": "user not found (gh-3030)" } +May 24 01:59:07 server sm-mta[65815]: 14NNx65Q065815: AUTH failure (CRAM-MD5): user not found (-20) SASL(-13): user not found: user: scan@server.example.com property: userPassword not found in sasldb /usr/local/etc/sasldb2, user=scan, relay=[192.0.2.6] From 87f717e0e038afeb9b2ac98617fc3d907ddc49bc Mon Sep 17 00:00:00 2001 From: sebres Date: Sat, 29 May 2021 17:45:02 +0200 Subject: [PATCH 130/240] filter.d/sendmail-reject.conf: fix reverse DNS for ... (gh-3012) --- config/filter.d/sendmail-reject.conf | 2 +- fail2ban/tests/files/logs/sendmail-reject | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/config/filter.d/sendmail-reject.conf b/config/filter.d/sendmail-reject.conf index 966d880c..41035e5f 100644 --- a/config/filter.d/sendmail-reject.conf +++ b/config/filter.d/sendmail-reject.conf @@ -25,7 +25,7 @@ addr = (?:(?:IPv6:)?|) prefregex = ^%(__prefix_line)s.+$ -cmnfailre = ^ruleset=check_rcpt, arg1=(?P<\S+@\S+>), relay=(\S+ )?\[%(addr)s\](?: \(may be forged\))?, reject=(550 5\.7\.1 (?P=email)\.\.\. Relaying denied\. (IP name possibly forged \[(\d+\.){3}\d+\]|Proper authentication required\.|IP name lookup failed \[(\d+\.){3}\d+\])|553 5\.1\.8 (?P=email)\.\.\. Domain of sender address \S+ does not exist|550 5\.[71]\.1 (?P=email)\.\.\. (Rejected: .*|User unknown))$ +cmnfailre = ^ruleset=check_rcpt, arg1=(?P<\S+@\S+>), relay=(\S+ )?\[%(addr)s\](?: \(may be forged\))?, reject=(?:550 5\.7\.1(?: (?P=email)\.\.\.)?(?: Relaying denied\.)? (?:IP name possibly forged \[(\d+\.){3}\d+\]|Proper authentication required\.|IP name lookup failed \[(\d+\.){3}\d+\]|Fix reverse DNS for \S+)|553 5\.1\.8(?: (?P=email)\.\.\.)? Domain of sender address \S+ does not exist|550 5\.[71]\.1 (?P=email)\.\.\. (Rejected: .*|User unknown))$ ^ruleset=check_relay(?:, arg\d+=\S*)*, relay=(\S+ )?\[%(addr)s\](?: \(may be forged\))?, reject=421 4\.3\.2 (Connection rate limit exceeded\.|Too many open connections\.)$ ^rejecting commands from (\S* )?\[%(addr)s\] due to pre-greeting traffic after \d+ seconds$ ^(?:\S+ )?\[%(addr)s\]: (?:(?i)expn|vrfy) \S+ \[rejected\]$ diff --git a/fail2ban/tests/files/logs/sendmail-reject b/fail2ban/tests/files/logs/sendmail-reject index ed3143a8..8debe7ca 100644 --- a/fail2ban/tests/files/logs/sendmail-reject +++ b/fail2ban/tests/files/logs/sendmail-reject @@ -72,6 +72,8 @@ Feb 13 01:16:50 batman sm-mta[25815]: s1D0GoSs025815: [217.193.142.180]: vrfy in # failJSON: { "time": "2005-02-22T14:02:44", "match": true , "host": "24.73.201.194" } Feb 22 14:02:44 batman sm-mta[4030]: s1MD2hsd004030: rrcs-24-73-201-194.se.biz.rr.com [24.73.201.194]: VRFY root [rejected] +# failJSON: { "time": "2005-02-22T15:20:27", "match": true , "host": "192.0.2.5", "desc": "Fix reverse DNS for ... (gh-3012)" } +Feb 22 15:20:27 localhost sm-mta[275631]: 13O9Ixhq275631: ruleset=check_rcpt, arg1=, relay=[192.0.2.5], reject=550 5.7.1 ... Fix reverse DNS for 192.0.2.5 # failJSON: { "match": false } Nov 3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026250: ... No such user here @@ -109,4 +111,4 @@ Mar 29 22:51:43 server sendmail[3529565]: xA32R2PQ3529565: [192.0.2.2] did not i # failJSON: { "time": "2005-03-29T22:51:45", "match": true , "host": "192.0.2.3", "desc": "sendmail 8.15.2 default names IPv4/6 (gh-2787)" } Mar 29 22:51:45 server sm-mta[50437]: 06QDQnNf050437: example.com [192.0.2.3] did not issue MAIL/EXPN/VRFY/ETRN during connection to IPv4 # failJSON: { "time": "2005-03-29T22:51:46", "match": true , "host": "2001:DB8::1", "desc": "IPv6" } -Mar 29 22:51:46 server sm-mta[50438]: 06QDQnNf050438: example.com [IPv6:2001:DB8::1] did not issue MAIL/EXPN/VRFY/ETRN during connection to IPv6 \ No newline at end of file +Mar 29 22:51:46 server sm-mta[50438]: 06QDQnNf050438: example.com [IPv6:2001:DB8::1] did not issue MAIL/EXPN/VRFY/ETRN during connection to IPv6 From ae3e9b9149c36e3334447e869667bcf171ff0a3a Mon Sep 17 00:00:00 2001 From: sebres Date: Sat, 29 May 2021 19:21:27 +0200 Subject: [PATCH 131/240] filter.d/postfix.conf: extended to cover 2 new vectors: - RCPT from unknown, 504 5.5.2, need fully-qualified hostname, gh-2995 - 550 5.7.25 Client host rejected, gh-2996 review combining several regex to single one --- config/filter.d/postfix.conf | 10 +++------- fail2ban/tests/files/logs/postfix | 5 +++++ 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/config/filter.d/postfix.conf b/config/filter.d/postfix.conf index fb690fb0..17982f3e 100644 --- a/config/filter.d/postfix.conf +++ b/config/filter.d/postfix.conf @@ -12,16 +12,12 @@ before = common.conf _daemon = postfix(-\w+)?/\w+(?:/smtp[ds])? _port = (?::\d+)? +_pref = (?:RCPT|EHLO|VRFY) prefregex = ^%(__prefix_line)s> .+$ mdpr-normal = (?:\w+: reject:|(?:improper command pipelining|too many errors) after \S+) -mdre-normal=^RCPT from [^[]*\[\]%(_port)s: 55[04] 5\.7\.1\s - ^RCPT from [^[]*\[\]%(_port)s: 45[04] 4\.7\.\d+ (?:Service unavailable\b|Client host rejected: cannot find your (reverse )?hostname\b) - ^RCPT from [^[]*\[\]%(_port)s: 450 4\.7\.\d+ (<[^>]*>)?: Helo command rejected: Host not found\b - ^EHLO from [^[]*\[\]%(_port)s: 504 5\.5\.\d+ (<[^>]*>)?: Helo command rejected: need fully-qualified hostname\b - ^(RCPT|VRFY) from [^[]*\[\]%(_port)s: 550 5\.1\.1\s - ^RCPT from [^[]*\[\]%(_port)s: 450 4\.1\.\d+ (<[^>]*>)?: Sender address rejected: Domain not found\b +mdre-normal=^%(_pref)s from [^[]*\[\]%(_port)s: [45][50][04] [45]\.\d\.\d+ (?:(?:<[^>]*>)?: )?(?:(?:Helo command|(?:Sender|Recipient) address) rejected: )?(?:Service unavailable|User unknown|Client host rejected|Relay access denied|(?:Host|Domain) not found|need fully-qualified hostname|match)\b ^from [^[]*\[\]%(_port)s:? mdpr-auth = warning: @@ -31,7 +27,7 @@ mdre-auth2= ^[^[]*\[\]%(_port)s: SASL ((?i)LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5 # Mode "rbl" currently included in mode "normal", but if needed for jail "postfix-rbl" only: mdpr-rbl = %(mdpr-normal)s -mdre-rbl = ^RCPT from [^[]*\[\]%(_port)s: [45]54 [45]\.7\.1 Service unavailable; Client host \[\S+\] blocked\b +mdre-rbl = ^%(_pref)s from [^[]*\[\]%(_port)s: [45]54 [45]\.7\.1 Service unavailable; Client host \[\S+\] blocked\b # Mode "rbl" currently included in mode "normal" (within 1st rule) mdpr-more = %(mdpr-normal)s diff --git a/fail2ban/tests/files/logs/postfix b/fail2ban/tests/files/logs/postfix index 6e2dc460..d46e6538 100644 --- a/fail2ban/tests/files/logs/postfix +++ b/fail2ban/tests/files/logs/postfix @@ -35,6 +35,11 @@ Nov 22 22:33:44 xxx postfix/smtpd[11111]: NOQUEUE: reject: RCPT from 1-2-3-4.exa # failJSON: { "time": "2005-01-31T13:55:24", "match": true , "host": "78.107.251.238" } Jan 31 13:55:24 xxx postfix/smtpd[3462]: NOQUEUE: reject: EHLO from s271272.static.corbina.ru[78.107.251.238]: 504 5.5.2 : Helo command rejected: need fully-qualified hostname; proto=SMTP helo= +# failJSON: { "time": "2005-04-06T13:05:01", "match": true , "host": "192.0.2.116", "desc": "RCPT from unknown, gh-2995" } +Apr 6 13:05:01 server postfix/smtpd[20589]: NOQUEUE: reject: RCPT from unknown[192.0.2.116]: 504 5.5.2 : Helo command rejected: need fully-qualified hostname; from= to= proto=ESMTP helo= +# failJSON: { "time": "2005-04-07T03:10:56", "match": true , "host": "192.0.2.246", "desc": "550 5.7.25 Client host rejected, gh-2996" } +Apr 7 03:10:56 server postfix/smtpd[7754]: NOQUEUE: reject: RCPT from unknown[192.0.2.246]: 550 5.7.25 Client host rejected: cannot find your hostname, [192.0.2.246]; from= to= proto=ESMTP helo=<[192.0.2.246]> + # failJSON: { "time": "2005-01-31T13:55:24", "match": true , "host": "78.107.251.238" } Jan 31 13:55:24 xxx postfix-incoming/smtpd[3462]: NOQUEUE: reject: EHLO from s271272.static.corbina.ru[78.107.251.238]: 504 5.5.2 : Helo command rejected: need fully-qualified hostname; proto=SMTP helo= From c5f1598a215b337ce5a93243fe17896edff780e6 Mon Sep 17 00:00:00 2001 From: sebres Date: Sat, 29 May 2021 19:48:24 +0200 Subject: [PATCH 132/240] filter.d/postfix.conf: extended to cover new vectors: - reject: BDAT/DATA from (gh-2927) - (since regex is more precise now) token selector changed to `[A-Z]{4}`, e. g. no matter what a command is supplied now (RCPT, EHLO, VRFY, DATA, BDAT or something else) - matches "Command rejected" and "Data command rejected" now --- config/filter.d/postfix.conf | 6 +++--- fail2ban/tests/files/logs/postfix | 5 +++++ 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/config/filter.d/postfix.conf b/config/filter.d/postfix.conf index 17982f3e..01d8cb0b 100644 --- a/config/filter.d/postfix.conf +++ b/config/filter.d/postfix.conf @@ -12,12 +12,12 @@ before = common.conf _daemon = postfix(-\w+)?/\w+(?:/smtp[ds])? _port = (?::\d+)? -_pref = (?:RCPT|EHLO|VRFY) +_pref = [A-Z]{4} prefregex = ^%(__prefix_line)s> .+$ -mdpr-normal = (?:\w+: reject:|(?:improper command pipelining|too many errors) after \S+) -mdre-normal=^%(_pref)s from [^[]*\[\]%(_port)s: [45][50][04] [45]\.\d\.\d+ (?:(?:<[^>]*>)?: )?(?:(?:Helo command|(?:Sender|Recipient) address) rejected: )?(?:Service unavailable|User unknown|Client host rejected|Relay access denied|(?:Host|Domain) not found|need fully-qualified hostname|match)\b +mdpr-normal = (?:\w+: (?:milter-)?reject:|(?:improper command pipelining|too many errors) after \S+) +mdre-normal=^%(_pref)s from [^[]*\[\]%(_port)s: [45][50][04] [45]\.\d\.\d+ (?:(?:<[^>]*>)?: )?(?:(?:Helo command|(?:Sender|Recipient) address) rejected: )?(?:Service unavailable|User unknown|(?:Client host|Command|Data command) rejected|Relay access denied|(?:Host|Domain) not found|need fully-qualified hostname|match)\b ^from [^[]*\[\]%(_port)s:? mdpr-auth = warning: diff --git a/fail2ban/tests/files/logs/postfix b/fail2ban/tests/files/logs/postfix index d46e6538..85b61ea6 100644 --- a/fail2ban/tests/files/logs/postfix +++ b/fail2ban/tests/files/logs/postfix @@ -35,6 +35,11 @@ Nov 22 22:33:44 xxx postfix/smtpd[11111]: NOQUEUE: reject: RCPT from 1-2-3-4.exa # failJSON: { "time": "2005-01-31T13:55:24", "match": true , "host": "78.107.251.238" } Jan 31 13:55:24 xxx postfix/smtpd[3462]: NOQUEUE: reject: EHLO from s271272.static.corbina.ru[78.107.251.238]: 504 5.5.2 : Helo command rejected: need fully-qualified hostname; proto=SMTP helo= +# failJSON: { "time": "2005-03-7T02:09:33", "match": true , "host": "192.0.2.151", "desc": "reject: DATA from, gh-2927" } +Mar 7 02:09:33 server postfix/smtpd[27246]: 1D8CC1CA0A7F: milter-reject: DATA from 66-220-155-151.mail-mail.facebook.com[192.0.2.151]: 550 5.7.1 Command rejected; from= to= proto=ESMTP helo=<192-0-2-151.mail-mail.example.com> +# failJSON: { "time": "2005-03-11T23:27:54", "match": true , "host": "192.0.2.109", "desc": "reject: BDAT from, gh-2927" } +Mar 11 23:27:54 server postfix-smo/submission/smtpd[22427]: 44JCRG5tYPzCqt2: reject: BDAT from signing-milter.example.com[192.0.2.109]: 550 5.5.3 : Data command rejected: Multi-recipient bounce; from=<> to= proto=ESMTP helo= + # failJSON: { "time": "2005-04-06T13:05:01", "match": true , "host": "192.0.2.116", "desc": "RCPT from unknown, gh-2995" } Apr 6 13:05:01 server postfix/smtpd[20589]: NOQUEUE: reject: RCPT from unknown[192.0.2.116]: 504 5.5.2 : Helo command rejected: need fully-qualified hostname; from= to= proto=ESMTP helo= # failJSON: { "time": "2005-04-07T03:10:56", "match": true , "host": "192.0.2.246", "desc": "550 5.7.25 Client host rejected, gh-2996" } From 8afea374943b6a2589bf3964f15af0aa17c6c369 Mon Sep 17 00:00:00 2001 From: sebres Date: Sat, 29 May 2021 20:09:57 +0200 Subject: [PATCH 133/240] filter.d/sendmail-auth.conf: covering several "authentication failure" messages, sendmail 8.16.1 (gh-2757) --- config/filter.d/sendmail-auth.conf | 3 +-- fail2ban/tests/files/logs/sendmail-auth | 5 +++++ 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/config/filter.d/sendmail-auth.conf b/config/filter.d/sendmail-auth.conf index 4601a1fc..de1f8e36 100644 --- a/config/filter.d/sendmail-auth.conf +++ b/config/filter.d/sendmail-auth.conf @@ -15,8 +15,7 @@ addr = (?:IPv6:|) prefregex = ^%(__prefix_line)s.+$ failregex = ^(\S+ )?\[%(addr)s\]( \(may be forged\))?: possible SMTP attack: command=AUTH, count=\d+$ - ^AUTH failure \(LOGIN\):(?: [^:]+:)? authentication failure: checkpass failed, user=(?:\S+|.*?), relay=(?:\S+ )?\[%(addr)s\](?: \(may be forged\))?$ - ^AUTH failure \([^\)]+\):(?: [^:]+:)? user not found: [^,]*, user=(?:\S+|.*?), relay=(?:\S+ )?\[%(addr)s\](?: \(may be forged\))?$ + ^AUTH failure \([^\)]+\):(?: [^:]+:)? (?:authentication failure|user not found): [^,]*, user=(?:\S+|.*?), relay=(?:\S+ )?\[%(addr)s\](?: \(may be forged\))?$ ignoreregex = journalmatch = _SYSTEMD_UNIT=sendmail.service diff --git a/fail2ban/tests/files/logs/sendmail-auth b/fail2ban/tests/files/logs/sendmail-auth index baa0a33f..f88cde86 100644 --- a/fail2ban/tests/files/logs/sendmail-auth +++ b/fail2ban/tests/files/logs/sendmail-auth @@ -27,3 +27,8 @@ Feb 25 04:02:28 relay1 sendmail[16665]: 06I02CNi016765: AUTH failure (LOGIN): au May 24 01:58:40 server sm-mta[65696]: 14NNwaRl065696: AUTH failure (DIGEST-MD5): user not found (-20) SASL(-13): user not found: unable to canonify user and get auxprops, user=scanner, relay=[192.0.2.5] # failJSON: { "time": "2005-05-24T01:59:07", "match": true , "host": "192.0.2.6", "desc": "user not found (gh-3030)" } May 24 01:59:07 server sm-mta[65815]: 14NNx65Q065815: AUTH failure (CRAM-MD5): user not found (-20) SASL(-13): user not found: user: scan@server.example.com property: userPassword not found in sasldb /usr/local/etc/sasldb2, user=scan, relay=[192.0.2.6] + +# failJSON: { "time": "2005-05-29T23:14:04", "match": true , "host": "192.0.2.7", "desc": "authentication failure, sendmail 8.16.1 (gh-2757)" } +May 29 23:14:04 mail sendmail[5976]: 09DJDgOM005976: AUTH failure (login): authentication failure (-13) SASL(-13): authentication failure: checkpass failed, user=test, relay=host.example.com [192.0.2.7] (may be forged) +# failJSON: { "time": "2005-05-29T23:14:04", "match": true , "host": "192.0.2.8", "desc": "authentication failure, sendmail 8.16.1 (gh-2757)" } +May 29 23:14:04 mail sendmail[5976]: 09DJDgOM005976: AUTH failure (PLAIN): authentication failure (-13) SASL(-13): authentication failure: Password verification failed, user=test, relay=host.example.com [192.0.2.8] From 6be1a5a0b1b74ff8c1f3cb07e540f24260a125bf Mon Sep 17 00:00:00 2001 From: sebres Date: Sat, 29 May 2021 20:25:28 +0200 Subject: [PATCH 134/240] filter.d/dovecot.conf: fixed "Authentication failure" regex, matches "Password mismatch" in title case (gh-2880) --- config/filter.d/dovecot.conf | 4 ++-- fail2ban/tests/files/logs/dovecot | 3 +++ 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/config/filter.d/dovecot.conf b/config/filter.d/dovecot.conf index 98d9f43b..7ec9a30c 100644 --- a/config/filter.d/dovecot.conf +++ b/config/filter.d/dovecot.conf @@ -14,8 +14,8 @@ prefregex = ^%(__prefix_line)s(?:%(_auth_worker)s(?:\([^\)]+\))?: )?(?:%(__pam_a failregex = ^authentication failure; logname=\S* uid=\S* euid=\S* tty=dovecot ruser=\S* rhost=(?:\s+user=\S*)?\s*$ ^(?:Aborted login|Disconnected|Remote closed connection|Client has quit the connection)(?::(?: [^ \(]+)+)? \((?:auth failed, \d+ attempts(?: in \d+ secs)?|tried to use (?:disabled|disallowed) \S+ auth|proxy dest auth failed)\):(?: user=<[^>]*>,)?(?: method=\S+,)? rip=(?:[^>]*(?:, session=<\S+>)?)\s*$ - ^pam\(\S+,(?:,\S*)?\): pam_authenticate\(\) failed: (?:User not known to the underlying authentication module: \d+ Time\(s\)|Authentication failure \(password mismatch\?\)|Permission denied)\s*$ - ^[a-z\-]{3,15}\(\S*,(?:,\S*)?\): (?:unknown user|invalid credentials|Password mismatch) + ^pam\(\S+,(?:,\S*)?\): pam_authenticate\(\) failed: (?:User not known to the underlying authentication module: \d+ Time\(s\)|Authentication failure \([Pp]assword mismatch\?\)|Permission denied)\s*$ + ^[a-z\-]{3,15}\(\S*,(?:,\S*)?\): (?:[Uu]nknown user|[Ii]nvalid credentials|[Pp]assword mismatch) > mdre-aggressive = ^(?:Aborted login|Disconnected|Remote closed connection|Client has quit the connection)(?::(?: [^ \(]+)+)? \((?:no auth attempts|disconnected before auth was ready,|client didn't finish \S+ auth,)(?: (?:in|waited) \d+ secs)?\):(?: user=<[^>]*>,)?(?: method=\S+,)? rip=(?:[^>]*(?:, session=<\S+>)?)\s*$ diff --git a/fail2ban/tests/files/logs/dovecot b/fail2ban/tests/files/logs/dovecot index c1ed28f9..17a16b9f 100644 --- a/fail2ban/tests/files/logs/dovecot +++ b/fail2ban/tests/files/logs/dovecot @@ -34,6 +34,9 @@ Jul 02 13:49:32 hostname dovecot[442]: dovecot: auth(default): pam(account@MYSER # failJSON: { "time": "2005-01-29T05:32:50", "match": true , "host": "1.2.3.4" } Jan 29 05:32:50 mail dovecot: auth-worker(304): pam(username,1.2.3.4): pam_authenticate() failed: Authentication failure (password mismatch?) +# failJSON: { "time": "2005-01-29T18:55:55", "match": true , "host": "192.0.2.4", "desc": "Password mismatch (title case, gh-2880)" } +Jan 29 18:55:55 mail dovecot: auth-worker(12182): pam(user,192.0.2.4): pam_authenticate() failed: Authentication failure (Password mismatch?) + # failJSON: { "time": "2005-01-29T05:13:40", "match": true , "host": "1.2.3.4" } Jan 29 05:13:40 mail dovecot: auth-worker(31326): pam(username,1.2.3.4): unknown user From 8b984a0135be5ea284a2345cce0e3458dfb95d7b Mon Sep 17 00:00:00 2001 From: sebres Date: Sat, 29 May 2021 20:47:56 +0200 Subject: [PATCH 135/240] filter.d\exim-common.conf: pid-prefix extended to match `mx1 exim[...]:` (gh-2553) --- config/filter.d/exim-common.conf | 2 +- fail2ban/tests/files/logs/exim | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/config/filter.d/exim-common.conf b/config/filter.d/exim-common.conf index b3b25750..36644e94 100644 --- a/config/filter.d/exim-common.conf +++ b/config/filter.d/exim-common.conf @@ -12,7 +12,7 @@ after = exim-common.local host_info_pre = (?:H=([\w.-]+ )?(?:\(\S+\) )?)? host_info_suf = (?::\d+)?(?: I=\[\S+\](:\d+)?)?(?: U=\S+)?(?: P=e?smtp)?(?: F=(?:<>|[^@]+@\S+))?\s host_info = %(host_info_pre)s\[\]%(host_info_suf)s -pid = (?: \[\d+\])? +pid = (?: \[\d+\]| \w+ exim\[\d+\]:)? # DEV Notes: # From exim source code: ./src/receive.c:add_host_info_for_log diff --git a/fail2ban/tests/files/logs/exim b/fail2ban/tests/files/logs/exim index 79437a90..e88f06ef 100644 --- a/fail2ban/tests/files/logs/exim +++ b/fail2ban/tests/files/logs/exim @@ -43,6 +43,9 @@ # failJSON: { "time": "2014-01-12T02:07:48", "match": true , "host": "85.214.85.40" } 2014-01-12 02:07:48 dovecot_login authenticator failed for h1832461.stratoserver.net (User) [85.214.85.40]: 535 Incorrect authentication data (set_id=scanner) +# failJSON: { "time": "2019-10-22T03:39:17", "match": true , "host": "192.0.2.37", "desc": "pid-prefix in form of 'mx1 exim[...]:', gh-2553" } +2019-10-22 03:39:17 mx1 exim[29786]: dovecot_login authenticator failed for (User) [192.0.2.37]: 535 Incorrect authentication data (set_id=test@domain.com) + # failJSON: { "time": "2014-12-02T03:00:23", "match": true , "host": "193.254.202.35" } 2014-12-02 03:00:23 auth_plain authenticator failed for (rom182) [193.254.202.35]:41556 I=[10.0.0.1]:25: 535 Incorrect authentication data (set_id=webmaster) From 92f90038fa67d719c55c75f5eff8059e849fe747 Mon Sep 17 00:00:00 2001 From: sebres Date: Sat, 29 May 2021 21:12:34 +0200 Subject: [PATCH 136/240] filter.d/dovecot.conf: extended to match prefix like `conn unix:auth-worker (uid=143): auth-worker<13247>:` (authenticate from external service like exim), gh-2553 --- config/filter.d/dovecot.conf | 3 ++- fail2ban/tests/files/logs/dovecot | 5 +++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/config/filter.d/dovecot.conf b/config/filter.d/dovecot.conf index 7ec9a30c..9c817720 100644 --- a/config/filter.d/dovecot.conf +++ b/config/filter.d/dovecot.conf @@ -8,9 +8,10 @@ before = common.conf [Definition] _auth_worker = (?:dovecot: )?auth(?:-worker)? +_auth_worker_info = (?:conn \w+:auth(?:-worker)? \(uid=\w+\): auth(?:-worker)?<\d+>: )? _daemon = (?:dovecot(?:-auth)?|auth) -prefregex = ^%(__prefix_line)s(?:%(_auth_worker)s(?:\([^\)]+\))?: )?(?:%(__pam_auth)s(?:\(dovecot:auth\))?: |(?:pop3|imap|managesieve|submission)-login: )?(?:Info: )?.+$ +prefregex = ^%(__prefix_line)s(?:%(_auth_worker)s(?:\([^\)]+\))?: )?(?:%(__pam_auth)s(?:\(dovecot:auth\))?: |(?:pop3|imap|managesieve|submission)-login: )?(?:Info: )?%(_auth_worker_info)s.+$ failregex = ^authentication failure; logname=\S* uid=\S* euid=\S* tty=dovecot ruser=\S* rhost=(?:\s+user=\S*)?\s*$ ^(?:Aborted login|Disconnected|Remote closed connection|Client has quit the connection)(?::(?: [^ \(]+)+)? \((?:auth failed, \d+ attempts(?: in \d+ secs)?|tried to use (?:disabled|disallowed) \S+ auth|proxy dest auth failed)\):(?: user=<[^>]*>,)?(?: method=\S+,)? rip=(?:[^>]*(?:, session=<\S+>)?)\s*$ diff --git a/fail2ban/tests/files/logs/dovecot b/fail2ban/tests/files/logs/dovecot index 17a16b9f..6bcf8c5b 100644 --- a/fail2ban/tests/files/logs/dovecot +++ b/fail2ban/tests/files/logs/dovecot @@ -55,6 +55,11 @@ Jun 11 13:57:17 main dovecot: auth: sql(admin@example.ru,192.168.144.226,<6rXunF #failJSON: { "time": "2005-06-11T13:57:17", "match": true, "host": "192.168.178.25", "desc": "allow more verbose logging, gh-2573" } Jun 11 13:57:17 main dovecot: auth: ldap(user@server.org,192.168.178.25,): Password mismatch (for LDAP bind) (SHA1 of given password: f638ff) +# failJSON: { "time": "2005-06-12T11:48:12", "match": true , "host": "192.0.2.6" } +Jun 12 11:48:12 auth-worker(80180): Info: conn unix:auth-worker (uid=143): auth-worker<13247>: sql(support,192.0.2.6): unknown user +# failJSON: { "time": "2005-06-12T23:06:05", "match": true , "host": "192.0.2.7" } +Jun 12 23:06:05 auth-worker(57065): Info: conn unix:auth-worker (uid=143): auth-worker<225622>: sql(user@domain.com,192.0.2.7,): Password mismatch + # failJSON: { "time": "2005-01-29T14:38:51", "match": true , "host": "192.0.2.6", "desc": "PAM Permission denied (gh-1897)" } Jan 29 14:38:51 example.com dovecot[24941]: auth-worker(30165): pam(user@example.com,192.0.2.6,): pam_authenticate() failed: Permission denied From c7a86b4616ef1bd43cf96874710b53b42d065684 Mon Sep 17 00:00:00 2001 From: sebres Date: Sat, 29 May 2021 22:59:55 +0200 Subject: [PATCH 137/240] action.d/firewallcmd-ipset.conf: amend to #2620: - combines actions `firewallcmd-ipset` and `firewallcmd-ipset-native` (parameter `ipsettype=firewalld`); - IPv6-capability for firewalld ipset; - no internal timeout handling by default; - no permanent rules yet --- config/action.d/firewallcmd-ipset-native.conf | 77 ------------------- config/action.d/firewallcmd-ipset.conf | 43 +++++++++-- 2 files changed, 38 insertions(+), 82 deletions(-) delete mode 100644 config/action.d/firewallcmd-ipset-native.conf diff --git a/config/action.d/firewallcmd-ipset-native.conf b/config/action.d/firewallcmd-ipset-native.conf deleted file mode 100644 index 757d46ad..00000000 --- a/config/action.d/firewallcmd-ipset-native.conf +++ /dev/null @@ -1,77 +0,0 @@ -# Fail2Ban action file for firewall-cmd using native ipset implementation -# -# This requires: -# ipset (package: ipset) -# firewall-cmd (package: firewalld) -# -# This is for ipset protocol 6 (and hopefully later) (ipset v6.14). -# Use ipset -V to see the protocol and version. -# -# IPset was a feature introduced in the linux kernel 2.6.39 and 3.0.0 kernels. -# -# If you are running on an older kernel you make need to patch in external -# modules. - -[INCLUDES] - -before = firewallcmd-common.conf - -[Definition] - -actionstart = firewall-cmd --permanent --new-ipset= --type=hash:ip --option=timeout= - firewall-cmd --reload - firewall-cmd --direct --add-rule filter 0 -m set --match-set src -j - -actionstop = firewall-cmd --direct --remove-rule filter 0 -m set --match-set src -j - firewall-cmd --permanent --delete-ipset= - firewall-cmd --reload - -actionban = firewall-cmd --ipset= --add-entry= - -actionunban = firewall-cmd --ipset= --remove-entry= - -[Init] - -# Option: chain -# Notes specifies the iptables chain to which the fail2ban rules should be -# added -# Values: [ STRING ] -# -chain = INPUT_direct - -# Option: bantime -# Notes: specifies the bantime in seconds (handled internally rather than by fail2ban) -# Values: [ NUM ] Default: 600 - -bantime = 86400 - -# Option: actiontype -# Notes.: defines additions to the blocking rule -# Values: leave empty to block all attempts from the host -# Default: Value of the multiport -actiontype = - -# Option: allports -# Notes.: default addition to block all ports -# Usage.: use in jail config: banaction = firewallcmd-ipset[actiontype=] -# for all protocols: banaction = firewallcmd-ipset[actiontype=""] -allports = -p - -# Option: multiport -# Notes.: addition to block access only to specific ports -# Usage.: use in jail config: banaction = firewallcmd-ipset[actiontype=] -multiport = -p -m multiport --dports - -ipmset = f2b- -familyopt = - -[Init?family=inet6] - -ipmset = f2b-6 -familyopt = family inet6 - - -# DEV NOTES: -# -# Author: Edgar Hoch and Daniel Black and Mihail Politaev -# firewallcmd-new / iptables-ipset-proto6 combined for maximium goodness diff --git a/config/action.d/firewallcmd-ipset.conf b/config/action.d/firewallcmd-ipset.conf index c89a0243..e1cb67e7 100644 --- a/config/action.d/firewallcmd-ipset.conf +++ b/config/action.d/firewallcmd-ipset.conf @@ -18,21 +18,46 @@ before = firewallcmd-common.conf [Definition] -actionstart = ipset create hash:ip timeout +actionstart = /actionstart> firewall-cmd --direct --add-rule filter 0 -m set --match-set src -j -actionflush = ipset flush +actionflush = /actionflush> actionstop = firewall-cmd --direct --remove-rule filter 0 -m set --match-set src -j - ipset destroy + /actionstop> -actionban = ipset add timeout -exist +actionban = /actionban> # actionprolong = %(actionban)s +actionunban = /actionunban> + +[ipstype_ipset] + +actionstart = ipset create hash:ip timeout + +actionflush = ipset flush + +actionstop = ipset destroy + +actionban = ipset add timeout -exist + actionunban = ipset del -exist +[ipstype_firewalld] + +actionstart = firewall-cmd --direct --new-ipset= --type=hash:ip --option=timeout= + +# TODO: there doesn't seem to be an explicit way to invoke the ipset flush function using firewall-cmd +actionflush = + +actionstop = firewall-cmd --direct --delete-ipset= + +actionban = firewall-cmd --ipset= --add-entry= + +actionunban = firewall-cmd --ipset= --remove-entry= + [Init] # Option: chain @@ -56,6 +81,12 @@ ipsettime = 0 # banaction = %(known/banaction)s[ipsettime=''] timeout-bantime = $([ "" -le 2147483 ] && echo "" || echo 0) +# Option: ipsettype +# Notes.: defines type of ipset used for match-set (firewalld or ipset) +# Values: firewalld or ipset +# Default: ipset +ipsettype = ipset + # Option: actiontype # Notes.: defines additions to the blocking rule # Values: leave empty to block all attempts from the host @@ -75,14 +106,16 @@ multiport = -p -m multiport --dports "$(echo '' | sed s/:/-/g)" ipmset = f2b- familyopt = +firewalld_familyopt = [Init?family=inet6] ipmset = f2b-6 familyopt = family inet6 +firewalld_familyopt = --option=family=inet6 # DEV NOTES: # -# Author: Edgar Hoch and Daniel Black +# Author: Edgar Hoch, Daniel Black, Sergey Brester and Mihail Politaev # firewallcmd-new / iptables-ipset-proto6 combined for maximium goodness From bbfff1828061514e48395a5dbc5c1f9f81625e82 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Thu, 3 Jun 2021 12:02:08 +0200 Subject: [PATCH 138/240] action.d/ufw.conf: amend to #3018: parameter `kill-mode` extended with conntrack --- config/action.d/ufw.conf | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/config/action.d/ufw.conf b/config/action.d/ufw.conf index 088a5a9d..c9ff7f37 100644 --- a/config/action.d/ufw.conf +++ b/config/action.d/ufw.conf @@ -33,13 +33,14 @@ actionunban = if [ -n "" ] && ufw app info "" fi # Option: kill-mode -# Notes.: can be set to ss (may be extended later with other modes) to immediately drop all connections from banned IP, default empty (no kill) +# Notes.: can be set to ss or conntrack (may be extended later with other modes) to immediately drop all connections from banned IP, default empty (no kill) # Example: banaction = ufw[kill-mode=ss] kill-mode = # intern conditional parameter used to provide killing mode after ban: _kill_ = _kill_ss = ss -K dst "[]" +_kill_conntrack = conntrack -D -s "" # Option: kill # Notes.: can be used to specify custom killing feature, by default depending on option kill-mode From 43f2923fbd2507b6aa9c357ead261f1800555ec6 Mon Sep 17 00:00:00 2001 From: sebres Date: Thu, 10 Jun 2021 15:06:54 +0200 Subject: [PATCH 139/240] filter.d/postfix.conf: matches rejects with "undeliverable address" (sender/recipient verification, gh-3039) additionally to "Unknown user"; both are configurable now via extended parameter and can be disabled using `exre-user=` supplied in filter parameters --- config/filter.d/postfix.conf | 5 ++++- fail2ban/tests/files/logs/postfix | 3 +++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/config/filter.d/postfix.conf b/config/filter.d/postfix.conf index 72fa3c06..31c06605 100644 --- a/config/filter.d/postfix.conf +++ b/config/filter.d/postfix.conf @@ -16,8 +16,11 @@ _pref = [A-Z]{4} prefregex = ^%(__prefix_line)s> .+$ +# Extended RE for normal mode to match reject by unknown users or undeliverable address, can be set to empty to avoid this: +exre-user = |[Uu](?:ser unknown|ndeliverable address) + mdpr-normal = (?:\w+: (?:milter-)?reject:|(?:improper command pipelining|too many errors) after \S+) -mdre-normal=^%(_pref)s from [^[]*\[\]%(_port)s: [45][50][04] [45]\.\d\.\d+ (?:(?:<[^>]*>)?: )?(?:(?:Helo command|(?:Sender|Recipient) address) rejected: )?(?:Service unavailable|User unknown|(?:Client host|Command|Data command) rejected|Relay access denied|(?:Host|Domain) not found|need fully-qualified hostname|match)\b +mdre-normal=^%(_pref)s from [^[]*\[\]%(_port)s: [45][50][04] [45]\.\d\.\d+ (?:(?:<[^>]*>)?: )?(?:(?:Helo command|(?:Sender|Recipient) address) rejected: )?(?:Service unavailable|(?:Client host|Command|Data command) rejected|Relay access denied|(?:Host|Domain) not found|need fully-qualified hostname|match%(exre-user)s)\b ^from [^[]*\[\]%(_port)s:? mdpr-auth = warning: diff --git a/fail2ban/tests/files/logs/postfix b/fail2ban/tests/files/logs/postfix index 580e0d03..c973377e 100644 --- a/fail2ban/tests/files/logs/postfix +++ b/fail2ban/tests/files/logs/postfix @@ -15,6 +15,9 @@ Aug 10 10:55:38 f-vanier-bourgeois postfix/smtpd[2162]: NOQUEUE: reject: VRFY fr # failJSON: { "time": "2005-08-13T15:45:46", "match": true , "host": "192.0.2.1" } Aug 13 15:45:46 server postfix/smtpd[13844]: 00ADB3C0899: reject: RCPT from example.com[192.0.2.1]: 550 5.1.1 : Recipient address rejected: User unknown in local recipient table; from= to= proto=ESMTP helo= +# failJSON: { "time": "2005-05-19T00:00:30", "match": true , "host": "192.0.2.2", "desc": "undeliverable address (sender/recipient verification, gh-3039)" } +May 19 00:00:30 proxy2 postfix/smtpd[16123]: NOQUEUE: reject: RCPT from example.net[192.0.2.2]: 550 5.1.1 : Recipient address rejected: undeliverable address: verification failed; from= to= proto=ESMTP helo= + # failJSON: { "time": "2005-01-12T11:07:49", "match": true , "host": "181.21.131.88" } Jan 12 11:07:49 emf1pt2-2-35-70 postfix/smtpd[13767]: improper command pipelining after DATA from unknown[181.21.131.88]: From 579c6a94aff0bb680090b3be1aef91491cf0a061 Mon Sep 17 00:00:00 2001 From: sebres Date: Thu, 10 Jun 2021 15:23:24 +0200 Subject: [PATCH 140/240] filter.d/postfix.conf: mode `ddos` (and `aggressive`) extended to consider abusive handling of clients hitting command limit (gh-3040) --- config/filter.d/postfix.conf | 2 +- fail2ban/tests/files/logs/postfix | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/config/filter.d/postfix.conf b/config/filter.d/postfix.conf index 31c06605..b374f472 100644 --- a/config/filter.d/postfix.conf +++ b/config/filter.d/postfix.conf @@ -38,7 +38,7 @@ mdre-more = %(mdre-normal)s # Includes some of the log messages described in # . -mdpr-ddos = (?:lost connection after(?! DATA) [A-Z]+|disconnect(?= from \S+(?: \S+=\d+)* auth=0/(?:[1-9]|\d\d+))|(?:PREGREET \d+|HANGUP) after \S+) +mdpr-ddos = (?:lost connection after(?! DATA) [A-Z]+|disconnect(?= from \S+(?: \S+=\d+)* auth=0/(?:[1-9]|\d\d+))|(?:PREGREET \d+|HANGUP) after \S+|COMMAND (?:TIME|COUNT|LENGTH) LIMIT) mdre-ddos = ^from [^[]*\[\]%(_port)s:? mdpr-extra = (?:%(mdpr-auth)s|%(mdpr-normal)s) diff --git a/fail2ban/tests/files/logs/postfix b/fail2ban/tests/files/logs/postfix index c973377e..d1e534e3 100644 --- a/fail2ban/tests/files/logs/postfix +++ b/fail2ban/tests/files/logs/postfix @@ -169,6 +169,12 @@ Dec 23 19:39:13 xxx postfix/postscreen[21057]: PREGREET 14 after 0.08 from [192. # failJSON: { "time": "2004-12-24T00:54:36", "match": true , "host": "192.0.2.3" } Dec 24 00:54:36 xxx postfix/postscreen[22515]: HANGUP after 16 from [192.0.2.3]:48119 in tests after SMTP handshake +# failJSON: { "time": "2005-06-08T23:14:28", "match": true , "host": "192.0.2.77", "desc": "abusive clients hitting command limit, see see http://www.postfix.org/POSTSCREEN_README.html (gh-3040)" } +Jun 8 23:14:28 proxy2 postfix/postscreen[473]: COMMAND TIME LIMIT from [192.0.2.77]:3608 after CONNECT +# failJSON: { "time": "2005-06-08T23:14:54", "match": true , "host": "192.0.2.26", "desc": "abusive clients hitting command limit (gh-3040)" } +Jun 8 23:14:54 proxy2 postfix/postscreen[473]: COMMAND COUNT LIMIT from [192.0.2.26]:15592 after RCPT + + # filterOptions: [{}, {"mode": "ddos"}, {"mode": "aggressive"}] # failJSON: { "match": false, "desc": "don't affect lawful data (sporadical connection aborts within DATA-phase, see gh-1813 for discussion)" } Feb 18 09:50:05 xxx postfix/smtpd[42]: lost connection after DATA from good-host.example.com[192.0.2.10] From 410a6ce5c80dd981c22752da034f2529b5eee844 Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 21 Jun 2021 17:12:53 +0200 Subject: [PATCH 141/240] fixed possible RCE vulnerability, unset escape variable (default tilde) stops consider "~" char after new-line as composing escape sequence --- config/action.d/complain.conf | 2 +- config/action.d/dshield.conf | 2 +- config/action.d/mail-buffered.conf | 8 ++++---- config/action.d/mail-whois-lines.conf | 2 +- config/action.d/mail-whois.conf | 6 +++--- config/action.d/mail.conf | 6 +++--- 6 files changed, 13 insertions(+), 13 deletions(-) diff --git a/config/action.d/complain.conf b/config/action.d/complain.conf index 3a5f882c..4d73b058 100644 --- a/config/action.d/complain.conf +++ b/config/action.d/complain.conf @@ -102,7 +102,7 @@ logpath = /dev/null # Notes.: Your system mail command. Is passed 2 args: subject and recipient # Values: CMD # -mailcmd = mail -s +mailcmd = mail -E 'set escape' -s # Option: mailargs # Notes.: Additional arguments to mail command. e.g. for standard Unix mail: diff --git a/config/action.d/dshield.conf b/config/action.d/dshield.conf index c128bef3..3d5a7a53 100644 --- a/config/action.d/dshield.conf +++ b/config/action.d/dshield.conf @@ -179,7 +179,7 @@ tcpflags = # Notes.: Your system mail command. Is passed 2 args: subject and recipient # Values: CMD # -mailcmd = mail -s +mailcmd = mail -E 'set escape' -s # Option: mailargs # Notes.: Additional arguments to mail command. e.g. for standard Unix mail: diff --git a/config/action.d/mail-buffered.conf b/config/action.d/mail-buffered.conf index 325f185b..79b84104 100644 --- a/config/action.d/mail-buffered.conf +++ b/config/action.d/mail-buffered.conf @@ -17,7 +17,7 @@ actionstart = printf %%b "Hi,\n The jail has been started successfully.\n Output will be buffered until lines are available.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : started on " + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : started on " # Option: actionstop # Notes.: command executed at the stop of jail (or at the end of Fail2Ban) @@ -28,13 +28,13 @@ actionstop = if [ -f ]; then These hosts have been banned by Fail2Ban.\n `cat ` Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : Summary from " + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : Summary from " rm fi printf %%b "Hi,\n The jail has been stopped.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : stopped on " + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : stopped on " # Option: actioncheck # Notes.: command executed once before each actionban command @@ -55,7 +55,7 @@ actionban = printf %%b "`date`: ( failures)\n" >> These hosts have been banned by Fail2Ban.\n `cat ` \nRegards,\n - Fail2Ban"|mail -s "[Fail2Ban] : Summary" + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : Summary" rm fi diff --git a/config/action.d/mail-whois-lines.conf b/config/action.d/mail-whois-lines.conf index 3a3e56b2..d2818cb9 100644 --- a/config/action.d/mail-whois-lines.conf +++ b/config/action.d/mail-whois-lines.conf @@ -72,7 +72,7 @@ actionunban = # Notes.: Your system mail command. Is passed 2 args: subject and recipient # Values: CMD # -mailcmd = mail -s +mailcmd = mail -E 'set escape' -s # Default name of the chain # diff --git a/config/action.d/mail-whois.conf b/config/action.d/mail-whois.conf index 7fea34c4..ab33b616 100644 --- a/config/action.d/mail-whois.conf +++ b/config/action.d/mail-whois.conf @@ -20,7 +20,7 @@ norestored = 1 actionstart = printf %%b "Hi,\n The jail has been started successfully.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : started on " + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : started on " # Option: actionstop # Notes.: command executed at the stop of jail (or at the end of Fail2Ban) @@ -29,7 +29,7 @@ actionstart = printf %%b "Hi,\n actionstop = printf %%b "Hi,\n The jail has been stopped.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : stopped on " + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : stopped on " # Option: actioncheck # Notes.: command executed once before each actionban command @@ -49,7 +49,7 @@ actionban = printf %%b "Hi,\n Here is more information about :\n `%(_whois_command)s`\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : banned from " + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : banned from " # Option: actionunban # Notes.: command executed when unbanning an IP. Take care that the diff --git a/config/action.d/mail.conf b/config/action.d/mail.conf index 5d8c0e15..f4838ddc 100644 --- a/config/action.d/mail.conf +++ b/config/action.d/mail.conf @@ -16,7 +16,7 @@ norestored = 1 actionstart = printf %%b "Hi,\n The jail has been started successfully.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : started on " + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : started on " # Option: actionstop # Notes.: command executed at the stop of jail (or at the end of Fail2Ban) @@ -25,7 +25,7 @@ actionstart = printf %%b "Hi,\n actionstop = printf %%b "Hi,\n The jail has been stopped.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : stopped on " + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : stopped on " # Option: actioncheck # Notes.: command executed once before each actionban command @@ -43,7 +43,7 @@ actionban = printf %%b "Hi,\n The IP has just been banned by Fail2Ban after attempts against .\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : banned from " + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : banned from " # Option: actionunban # Notes.: command executed when unbanning an IP. Take care that the From 2ed414ed09b3bb4c478abc9366a1ff22024a33c9 Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 21 Jun 2021 17:12:53 +0200 Subject: [PATCH 142/240] fixed possible RCE vulnerability, unset escape variable (default tilde) stops consider "~" char after new-line as composing escape sequence closes GHSA-m985-3f3v-cwmm for 0.9 --- config/action.d/complain.conf | 2 +- config/action.d/dshield.conf | 2 +- config/action.d/mail-buffered.conf | 8 ++++---- config/action.d/mail-whois-lines.conf | 12 +++++++++--- config/action.d/mail-whois.conf | 6 +++--- config/action.d/mail.conf | 6 +++--- 6 files changed, 21 insertions(+), 15 deletions(-) diff --git a/config/action.d/complain.conf b/config/action.d/complain.conf index 9247803e..993a31e4 100644 --- a/config/action.d/complain.conf +++ b/config/action.d/complain.conf @@ -79,7 +79,7 @@ logpath = /dev/null # Notes.: Your system mail command. Is passed 2 args: subject and recipient # Values: CMD # -mailcmd = mail -s +mailcmd = mail -E 'set escape' -s # Option: mailargs # Notes.: Additional arguments to mail command. e.g. for standard Unix mail: diff --git a/config/action.d/dshield.conf b/config/action.d/dshield.conf index a0041986..ae7014fa 100644 --- a/config/action.d/dshield.conf +++ b/config/action.d/dshield.conf @@ -176,7 +176,7 @@ tcpflags = # Notes.: Your system mail command. Is passed 2 args: subject and recipient # Values: CMD # -mailcmd = mail -s +mailcmd = mail -E 'set escape' -s # Option: mailargs # Notes.: Additional arguments to mail command. e.g. for standard Unix mail: diff --git a/config/action.d/mail-buffered.conf b/config/action.d/mail-buffered.conf index 914d4a5a..f530e958 100644 --- a/config/action.d/mail-buffered.conf +++ b/config/action.d/mail-buffered.conf @@ -14,7 +14,7 @@ actionstart = printf %%b "Hi,\n The jail has been started successfully.\n Output will be buffered until lines are available.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : started on `uname -n`" + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : started on `uname -n`" # Option: actionstop # Notes.: command executed once at the end of Fail2Ban @@ -25,13 +25,13 @@ actionstop = if [ -f ]; then These hosts have been banned by Fail2Ban.\n `cat ` Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : Summary from `uname -n`" + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : Summary from `uname -n`" rm fi printf %%b "Hi,\n The jail has been stopped.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : stopped on `uname -n`" + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : stopped on `uname -n`" # Option: actioncheck # Notes.: command executed once before each actionban command @@ -52,7 +52,7 @@ actionban = printf %%b "`date`: ( failures)\n" >> These hosts have been banned by Fail2Ban.\n `cat ` \nRegards,\n - Fail2Ban"|mail -s "[Fail2Ban] : Summary" + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : Summary" rm fi diff --git a/config/action.d/mail-whois-lines.conf b/config/action.d/mail-whois-lines.conf index 6e39c605..39f78957 100644 --- a/config/action.d/mail-whois-lines.conf +++ b/config/action.d/mail-whois-lines.conf @@ -17,7 +17,7 @@ before = mail-whois-common.conf actionstart = printf %%b "Hi,\n The jail has been started successfully.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : started on `uname -n`" + Fail2Ban" | "[Fail2Ban] : started on `uname -n`" # Option: actionstop # Notes.: command executed once at the end of Fail2Ban @@ -26,7 +26,7 @@ actionstart = printf %%b "Hi,\n actionstop = printf %%b "Hi,\n The jail has been stopped.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : stopped on `uname -n`" + Fail2Ban" | "[Fail2Ban] : stopped on `uname -n`" # Option: actioncheck # Notes.: command executed once before each actionban command @@ -48,7 +48,7 @@ actionban = printf %%b "Hi,\n Lines containing IP: in \n `grep -E '(^|[^0-9])([^0-9]|$)' `\n\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : banned from `uname -n`" + Fail2Ban" | "[Fail2Ban] : banned from `uname -n`" # Option: actionunban # Notes.: command executed when unbanning an IP. Take care that the @@ -60,6 +60,12 @@ actionunban = [Init] +# Option: mailcmd +# Notes.: Your system mail command. Is passed 2 args: subject and recipient +# Values: CMD +# +mailcmd = mail -E 'set escape' -s + # Default name of the chain # name = default diff --git a/config/action.d/mail-whois.conf b/config/action.d/mail-whois.conf index 018c327d..2c346bfe 100644 --- a/config/action.d/mail-whois.conf +++ b/config/action.d/mail-whois.conf @@ -17,7 +17,7 @@ before = mail-whois-common.conf actionstart = printf %%b "Hi,\n The jail has been started successfully.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : started on `uname -n`" + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : started on `uname -n`" # Option: actionstop # Notes.: command executed once at the end of Fail2Ban @@ -26,7 +26,7 @@ actionstart = printf %%b "Hi,\n actionstop = printf %%b "Hi,\n The jail has been stopped.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : stopped on `uname -n`" + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : stopped on `uname -n`" # Option: actioncheck # Notes.: command executed once before each actionban command @@ -46,7 +46,7 @@ actionban = printf %%b "Hi,\n Here is more information about :\n `%(_whois_command)s`\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : banned from `uname -n`" + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : banned from `uname -n`" # Option: actionunban # Notes.: command executed when unbanning an IP. Take care that the diff --git a/config/action.d/mail.conf b/config/action.d/mail.conf index 7bf51a1d..9755f097 100644 --- a/config/action.d/mail.conf +++ b/config/action.d/mail.conf @@ -13,7 +13,7 @@ actionstart = printf %%b "Hi,\n The jail has been started successfully.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : started on `uname -n`" + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : started on `uname -n`" # Option: actionstop # Notes.: command executed once at the end of Fail2Ban @@ -22,7 +22,7 @@ actionstart = printf %%b "Hi,\n actionstop = printf %%b "Hi,\n The jail has been stopped.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : stopped on `uname -n`" + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : stopped on `uname -n`" # Option: actioncheck # Notes.: command executed once before each actionban command @@ -40,7 +40,7 @@ actionban = printf %%b "Hi,\n The IP has just been banned by Fail2Ban after attempts against .\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : banned from `uname -n`" + Fail2Ban"|mail -E 'set escape' -s "[Fail2Ban] : banned from `uname -n`" # Option: actionunban # Notes.: command executed when unbanning an IP. Take care that the From 39fe0bdce612b2a3cb649866da8247d51f5731ab Mon Sep 17 00:00:00 2001 From: Alex Porto dos Santos <31667409+alexporto2200@users.noreply.github.com> Date: Tue, 17 Aug 2021 10:39:04 -0300 Subject: [PATCH 143/240] Update Readme Add python3-setuptools or python-setuptools for dependencies. On some distributions this doesn't come by default, it would be nice to let users know about this in the documentation. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index fac3414d..060b250f 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,7 @@ this case, you should use that instead.** Required: - [Python2 >= 2.6 or Python >= 3.2](https://www.python.org) or [PyPy](https://pypy.org) +- python3-setuptools or python-setuptools for run installer. Optional: - [pyinotify >= 0.8.3](https://github.com/seb-m/pyinotify), may require: From ade79635b27c408b811ffb3dd9a806a0963771c5 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Tue, 17 Aug 2021 16:41:20 +0200 Subject: [PATCH 144/240] distutils/setuptools only required if installing from source --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 060b250f..ba5c149e 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ this case, you should use that instead.** Required: - [Python2 >= 2.6 or Python >= 3.2](https://www.python.org) or [PyPy](https://pypy.org) -- python3-setuptools or python-setuptools for run installer. +- python-setuptools, python-distutils or python3-setuptools for installation from source Optional: - [pyinotify >= 0.8.3](https://github.com/seb-m/pyinotify), may require: From 2f99d5accb5b349817c4f8f3dc73a0162b05cab3 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 8 Sep 2021 18:22:31 +0200 Subject: [PATCH 145/240] test coverage for unhandled exception in run of several filter (gh-3097) --- fail2ban/server/filter.py | 2 +- fail2ban/server/filterpoll.py | 2 +- fail2ban/server/filterpyinotify.py | 2 +- fail2ban/server/filtersystemd.py | 2 +- fail2ban/tests/filtertestcase.py | 18 ++++++++++++++++++ 5 files changed, 22 insertions(+), 4 deletions(-) diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index f514b337..e9000447 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -722,7 +722,7 @@ class Filter(JailThread): # incr common error counter: self.commonError() - def commonError(self): + def commonError(self, reason="common", exc=None): # incr error counter, stop processing (going idle) after 100th error : self._errors += 1 # sleep a little bit (to get around time-related errors): diff --git a/fail2ban/server/filterpoll.py b/fail2ban/server/filterpoll.py index 7ee00540..196955e5 100644 --- a/fail2ban/server/filterpoll.py +++ b/fail2ban/server/filterpoll.py @@ -122,7 +122,7 @@ class FilterPoll(FileFilter): logSys.error("Caught unhandled exception in main cycle: %r", e, exc_info=logSys.getEffectiveLevel()<=logging.DEBUG) # incr common error counter: - self.commonError() + self.commonError("unhandled", e) logSys.debug("[%s] filter terminated", self.jailName) return True diff --git a/fail2ban/server/filterpyinotify.py b/fail2ban/server/filterpyinotify.py index d62348a2..b9936df5 100644 --- a/fail2ban/server/filterpyinotify.py +++ b/fail2ban/server/filterpyinotify.py @@ -363,7 +363,7 @@ class FilterPyinotify(FileFilter): logSys.error("Caught unhandled exception in main cycle: %r", e, exc_info=logSys.getEffectiveLevel()<=logging.DEBUG) # incr common error counter: - self.commonError() + self.commonError("unhandled", e) logSys.debug("[%s] filter exited (pyinotifier)", self.jailName) self.__notifier = None diff --git a/fail2ban/server/filtersystemd.py b/fail2ban/server/filtersystemd.py index 925109d1..d70f9259 100644 --- a/fail2ban/server/filtersystemd.py +++ b/fail2ban/server/filtersystemd.py @@ -334,7 +334,7 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover logSys.error("Caught unhandled exception in main cycle: %r", e, exc_info=logSys.getEffectiveLevel()<=logging.DEBUG) # incr common error counter: - self.commonError() + self.commonError("unhandled", e) logSys.debug("[%s] filter terminated", self.jailName) diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 3cc17fb1..5166bc43 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -979,6 +979,10 @@ class CommonMonitorTestCase(unittest.TestCase): super(CommonMonitorTestCase, self).setUp() self._failTotal = 0 + def tearDown(self): + super(CommonMonitorTestCase, self).tearDown() + self.assertFalse(hasattr(self, "_unexpectedError")) + def waitFailTotal(self, count, delay=1): """Wait up to `delay` sec to assure that expected failure `count` reached """ @@ -1004,6 +1008,16 @@ class CommonMonitorTestCase(unittest.TestCase): last_ticks = self.filter.ticks return Utils.wait_for(lambda: self.filter.ticks >= last_ticks + ticks, _maxWaitTime(delay)) + def commonFltError(self, reason="common", exc=None): + """ Mock-up for default common error handler to find catched unhandled exceptions + could occur in filters + """ + self._commonFltError(reason, exc) + if reason == "unhandled": + DefLogSys.critical("Caught unhandled exception in main cycle of %r : %r", self.filter, exc, exc_info=True) + self._unexpectedError = True + # self.assertNotEqual(reason, "unhandled") + def get_monitor_failures_testcase(Filter_): """Generator of TestCase's for different filters/backends @@ -1026,6 +1040,8 @@ def get_monitor_failures_testcase(Filter_): self.file = open(self.name, 'a') self.jail = DummyJail() self.filter = Filter_(self.jail) + # mock-up common error to find catched unhandled exceptions: + self._commonFltError, self.filter.commonError = self.filter.commonError, self.commonFltError self.filter.addLogPath(self.name, autoSeek=False) # speedup search using exact date pattern: self.filter.setDatePattern(r'^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?') @@ -1324,6 +1340,8 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover def _initFilter(self, **kwargs): self._getRuntimeJournal() # check journal available self.filter = Filter_(self.jail, **kwargs) + # mock-up common error to find catched unhandled exceptions: + self._commonFltError, self.filter.commonError = self.filter.commonError, self.commonFltError self.filter.addJournalMatch([ "SYSLOG_IDENTIFIER=fail2ban-testcases", "TEST_FIELD=1", From 1e4a14fb25d88e32f3ca9c06fb1d6b8d3b4813ab Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 8 Sep 2021 19:16:49 +0200 Subject: [PATCH 146/240] pyinotify: fixes sporadic runtime error "dictionary changed size during iteration" (if something outside changes the pending dict during _checkPending evaluation) - simply deserialize to a list for iteration, without any lock, because unneeded here due to small and mostly empty dictionary (logrotate, etc), not to mention that pending check is normally called once per minute; don't call process file inside of server thread calling of addLogPath (always retard it as pending event); ensure to wake-up as soon as possible to process pending events (e. g. if file gets added). --- fail2ban/server/filterpyinotify.py | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/fail2ban/server/filterpyinotify.py b/fail2ban/server/filterpyinotify.py index b9936df5..5f449cad 100644 --- a/fail2ban/server/filterpyinotify.py +++ b/fail2ban/server/filterpyinotify.py @@ -165,7 +165,7 @@ class FilterPyinotify(FileFilter): return found = {} minTime = 60 - for path, (retardTM, isDir) in self.__pending.iteritems(): + for path, (retardTM, isDir) in list(self.__pending.items()): if ntm - self.__pendingChkTime < retardTM: if minTime > retardTM: minTime = retardTM continue @@ -268,15 +268,13 @@ class FilterPyinotify(FileFilter): def _addLogPath(self, path): self._addFileWatcher(path) - # initial scan: + # notify (wake up if in waiting): if self.active: - # we can execute it right now: - self._process_file(path) - else: - # retard until filter gets started, isDir=None signals special case: process file only (don't need to refresh monitor): - self._addPending(path, ('INITIAL', path), isDir=None) + self.__pendingMinTime = 0 + # retard until filter gets started, isDir=None signals special case: process file only (don't need to refresh monitor): + self._addPending(path, ('INITIAL', path), isDir=None) - ## + ## # Delete a log path # # @param path the log file to delete @@ -341,12 +339,17 @@ class FilterPyinotify(FileFilter): self.__notifier.process_events() # wait for events / timeout: - notify_maxtout = self.__notify_maxtout def __check_events(): - return not self.active or self.__notifier.check_events(timeout=notify_maxtout) - if Utils.wait_for(__check_events, min(self.sleeptime, self.__pendingMinTime)): + return ( + not self.active + or bool(self.__notifier.check_events(timeout=self.__notify_maxtout)) + or (self.__pendingMinTime and self.__pending) + ) + wres = Utils.wait_for(__check_events, min(self.sleeptime, self.__pendingMinTime)) + if wres: if not self.active: break - self.__notifier.read_events() + if not isinstance(wres, dict): + self.__notifier.read_events() self.ticks += 1 From e323c148e13a141b22ea047b04a83a4cf248a7ac Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 8 Sep 2021 19:42:08 +0200 Subject: [PATCH 147/240] backend systemd: fixes error "local variable 'line' referenced before assignment", introduced in 55d7d9e214f72bbe4f39a2d17aa004d80bfc7299; don't update database too often (every 10 ticks or ~ 10 seconds in production); closes gh-3097 --- fail2ban/server/filtersystemd.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/fail2ban/server/filtersystemd.py b/fail2ban/server/filtersystemd.py index d70f9259..88f8c292 100644 --- a/fail2ban/server/filtersystemd.py +++ b/fail2ban/server/filtersystemd.py @@ -61,6 +61,7 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover # Initialise systemd-journal connection self.__journal = journal.Reader(**jrnlargs) self.__matches = [] + self.__nextUpdateTM = 0 self.setDatePattern(None) logSys.debug("Created FilterSystemd") @@ -285,6 +286,7 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover except OSError: pass # Reading failure, so safe to ignore + line = None while self.active: # wait for records (or for timeout in sleeptime seconds): try: @@ -326,8 +328,15 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover if self.ticks % 10 == 0: self.performSvc() # update position in log (time and iso string): - if self.jail.database is not None: + if (line and self.jail.database and ( + self.ticks % 10 == 0 + or MyTime.time() >= self.__nextUpdateTM + or not self.active + ) + ): self.jail.database.updateJournal(self.jail, 'systemd-journal', line[1], line[0][1]) + self.__nextUpdateTM = MyTime.time() + Utils.DEFAULT_SLEEP_TIME * 5 + line = None except Exception as e: # pragma: no cover if not self.active: # if not active - error by stop... break From ba282b794c97279906c26fdcae93180621bc3067 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 8 Sep 2021 19:56:02 +0200 Subject: [PATCH 148/240] pyinotify: amend to 1e4a14fb25d88e32f3ca9c06fb1d6b8d3b4813ab: one fix more for sporadic runtime error "dictionary changed size during iteration" (watched files) --- fail2ban/server/filterpyinotify.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fail2ban/server/filterpyinotify.py b/fail2ban/server/filterpyinotify.py index 5f449cad..16b6cfd5 100644 --- a/fail2ban/server/filterpyinotify.py +++ b/fail2ban/server/filterpyinotify.py @@ -188,7 +188,7 @@ class FilterPyinotify(FileFilter): self._refreshWatcher(path, isDir=isDir) if isDir: # check all files belong to this dir: - for logpath in self.__watchFiles: + for logpath in list(self.__watchFiles): if logpath.startswith(path + pathsep): # if still no file - add to pending, otherwise refresh and process: if not os.path.isfile(logpath): @@ -285,7 +285,7 @@ class FilterPyinotify(FileFilter): logSys.error("Failed to remove watch on path: %s", path) path_dir = dirname(path) - for k in self.__watchFiles: + for k in list(self.__watchFiles): if k.startswith(path_dir + pathsep): path_dir = None break From d709ec8179e9a589bc2d979b5f33192b1ce023f5 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 8 Sep 2021 20:00:41 +0200 Subject: [PATCH 149/240] GH actions: use newest python version for 3.10 (3.10.0-rc.2) --- .github/workflows/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 4ea0e7eb..231cca1e 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -22,7 +22,7 @@ jobs: runs-on: ubuntu-20.04 strategy: matrix: - python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, '3.10.0-beta.1', pypy2, pypy3] + python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, '3.10.0-rc.2', pypy2, pypy3] fail-fast: false # Steps represent a sequence of tasks that will be executed as part of the job steps: From 1414a44b8ee166e63b28c5f4d800b821fa9ea62c Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Sun, 19 Sep 2021 18:24:36 +0200 Subject: [PATCH 150/240] Update main.yml CI: try to install dependencies via apt, add build test --- .github/workflows/main.yml | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 231cca1e..c60ab05c 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -44,12 +44,15 @@ jobs: - name: Install dependencies run: | - python -m pip install --upgrade pip + #python -m pip install --upgrade pip if [[ "$F2B_PY" = 3 ]] && ! command -v 2to3x -v 2to3 > /dev/null; then - pip install 2to3 + #pip install 2to3 + sudo apt-get -y install 2to3 fi - pip install systemd-python || echo 'systemd not available' - pip install pyinotify || echo 'inotify not available' + #pip install systemd-python || echo 'systemd not available' + #pip install pyinotify || echo 'inotify not available' + sudo apt-get -y install python${F2B_PY/2/}-systemd || echo 'systemd not available' + sudo apt-get -y install python${F2B_PY/2/}-pyinotify || echo 'inotify not available' - name: Before scripts run: | @@ -61,6 +64,9 @@ jobs: - name: Test suite run: if [[ "$F2B_PY" = 2 ]]; then python setup.py test; else python bin/fail2ban-testcases --verbosity=2; fi - + + - name: Build + run: python setup.py build + #- name: Test initd scripts # run: shellcheck -s bash -e SC1090,SC1091 files/debian-initd From 7f22c4873aed3b5ffce0953f079f3c1977297c9a Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Sun, 19 Sep 2021 18:36:02 +0200 Subject: [PATCH 151/240] remove 2to3 in setup (should be called outside before setup) --- setup.py | 8 -------- 1 file changed, 8 deletions(-) diff --git a/setup.py b/setup.py index 2e2a77fa..f4c2550f 100755 --- a/setup.py +++ b/setup.py @@ -39,14 +39,6 @@ from distutils.command.build_scripts import build_scripts if setuptools is None: from distutils.command.install import install from distutils.command.install_scripts import install_scripts -try: - # python 3.x - from distutils.command.build_py import build_py_2to3 - from distutils.command.build_scripts import build_scripts_2to3 - _2to3 = True -except ImportError: - # python 2.x - _2to3 = False import os from os.path import isfile, join, isdir, realpath From 5ac303df8a171f748330d4c645ccbf1c2c7f3497 Mon Sep 17 00:00:00 2001 From: sebres Date: Sun, 19 Sep 2021 18:49:18 +0200 Subject: [PATCH 152/240] fix gh-3098: build fails with error in fail2ban setup command: use_2to3 is invalid (setuptools 58+) --- setup.py | 16 +--------------- 1 file changed, 1 insertion(+), 15 deletions(-) diff --git a/setup.py b/setup.py index f4c2550f..98413273 100755 --- a/setup.py +++ b/setup.py @@ -48,7 +48,7 @@ import warnings from glob import glob from fail2ban.setup import updatePyExec - +from fail2ban.version import version source_dir = os.path.realpath(os.path.dirname( # __file__ seems to be overwritten sometimes on some python versions (e.g. bug of 2.6 by running under cProfile, etc.): @@ -112,22 +112,12 @@ class install_scripts_f2b(install_scripts): # Wrapper to specify fail2ban own options: class install_command_f2b(install): user_options = install.user_options + [ - ('disable-2to3', None, 'Specify to deactivate 2to3, e.g. if the install runs from fail2ban test-cases.'), ('without-tests', None, 'without tests files installation'), ] def initialize_options(self): - self.disable_2to3 = None self.without_tests = not with_tests install.initialize_options(self) def finalize_options(self): - global _2to3 - ## in the test cases 2to3 should be already done (fail2ban-2to3): - if self.disable_2to3: - _2to3 = False - if _2to3: - cmdclass = self.distribution.cmdclass - cmdclass['build_py'] = build_py_2to3 - cmdclass['build_scripts'] = build_scripts_2to3 if self.without_tests: self.distribution.scripts.remove('bin/fail2ban-testcases') @@ -178,7 +168,6 @@ commands.''' if setuptools: setup_extra = { 'test_suite': "fail2ban.tests.utils.gatherTests", - 'use_2to3': True, } else: setup_extra = {} @@ -202,9 +191,6 @@ if platform_system in ('linux', 'solaris', 'sunos') or platform_system.startswit ('/usr/share/doc/fail2ban', doc_files) ) -# Get version number, avoiding importing fail2ban. -# This is due to tests not functioning for python3 as 2to3 takes place later -exec(open(join("fail2ban", "version.py")).read()) setup( name = "fail2ban", From d6b884f3b72b8a42b21da863836569ef6836c2ea Mon Sep 17 00:00:00 2001 From: sebres Date: Sun, 19 Sep 2021 18:52:34 +0200 Subject: [PATCH 153/240] amend to fix gh-3098: no option `--disable-2to3` anymore --- fail2ban/tests/misctestcase.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fail2ban/tests/misctestcase.py b/fail2ban/tests/misctestcase.py index 458e9a23..559a89f3 100644 --- a/fail2ban/tests/misctestcase.py +++ b/fail2ban/tests/misctestcase.py @@ -111,7 +111,7 @@ class SetupTest(unittest.TestCase): supdbgout = ' >/dev/null 2>&1' if unittest.F2B.log_level >= logging.DEBUG else '' # HEAVYDEBUG try: # try dry-run: - os.system("%s %s --dry-run install --disable-2to3 --root=%s%s" + os.system("%s %s --dry-run install --root=%s%s" % (sys.executable, self.setup , tmp, supdbgout)) # check nothing was created: self.assertTrue(not os.listdir(tmp)) @@ -127,7 +127,7 @@ class SetupTest(unittest.TestCase): # suppress stdout (and stderr) if not heavydebug supdbgout = ' >/dev/null' if unittest.F2B.log_level >= logging.DEBUG else '' # HEAVYDEBUG try: - self.assertEqual(os.system("%s %s install --disable-2to3 --root=%s%s" + self.assertEqual(os.system("%s %s install --root=%s%s" % (sys.executable, self.setup, tmp, supdbgout)), 0) def strippath(l): From a98cc08b31b41e9d344e26c6eb0d63d54b5f1242 Mon Sep 17 00:00:00 2001 From: Daniel Brooks Date: Sat, 10 Jul 2021 21:34:46 -0700 Subject: [PATCH 154/240] Updated the warning messages created when fail2ban sees unexpected timestamps to improve their grammar and to remove jargon. Partially fixes #2822 --- fail2ban/server/filter.py | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index e9000447..f9ea9025 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -662,9 +662,10 @@ class Filter(JailThread): if (date is None or date < MyTime.time() - 60 or date > MyTime.time() + 60): # log time zone issue as warning once per day: self._logWarnOnce("_next_simByTimeWarn", - ("Simulate NOW in operation since found time has too large deviation %s ~ %s +/- %s", - date, MyTime.time(), 60), - ("Please check jail has possibly a timezone issue. Line with odd timestamp: %s", + ("Found at least one log entry with a timestamp more" + + " than a minute from the current time. This is rarely a problem."), + ("Please check this jail and associated logs as" + + " there is potentially a timezone or latency problem: %s", line)) # simulate now as date: date = MyTime.time() @@ -674,9 +675,11 @@ class Filter(JailThread): if date is not None and date < MyTime.time() - self.getFindTime(): # log time zone issue as warning once per day: self._logWarnOnce("_next_ignByTimeWarn", - ("Ignore line since time %s < %s - %s", - date, MyTime.time(), self.getFindTime()), - ("Please check jail has possibly a timezone issue. Line with odd timestamp: %s", + ("Ignoring all log entries older than %s; these are probably" + + " messages generated while fail2ban was not running.", + self.getFindTime()), + ("Please check this jail and associated logs as" + + " there is potentially a timezone or latency problem: %s", line)) # ignore - too old (obsolete) entry: return [] From 320a3dcdd5a8b3019df3dfb21022dd4d80851f38 Mon Sep 17 00:00:00 2001 From: Daniel Brooks Date: Sat, 10 Jul 2021 21:48:32 -0700 Subject: [PATCH 155/240] remove old warnings from filtertestcase.py MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit assertLogged only checks that at least one listed message is found, so it isn’t necessary to repeat them in the test. --- fail2ban/tests/filtertestcase.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 5166bc43..9e81006b 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -457,8 +457,6 @@ class IgnoreIP(LogCaptureTestCase): for i in (1,2,3): self.filter.processLineAndAdd('2019-10-27 02:00:00 fail from 192.0.2.15'); # +3 = 3 self.assertLogged( - "Simulate NOW in operation since found time has too large deviation", - "Please check jail has possibly a timezone issue.", "192.0.2.15:1", "192.0.2.15:2", "192.0.2.15:3", "Total # of detected failures: 3.", wait=True) # From 1929e7a76b03becbece228edd8c65204a7e83371 Mon Sep 17 00:00:00 2001 From: Daniel Brooks Date: Mon, 12 Jul 2021 09:21:00 -0700 Subject: [PATCH 156/240] include more specific information in the warning --- fail2ban/server/filter.py | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index f9ea9025..e327228e 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -659,14 +659,19 @@ class Filter(JailThread): # if in operation (modifications have been really found): if self.inOperation: # if weird date - we'd simulate now for timeing issue (too large deviation from now): - if (date is None or date < MyTime.time() - 60 or date > MyTime.time() + 60): + delta = date - MyTime.time() + if (date is None or abs(delta) > 60): + latency = "This could indicate a latency problem." + timezone = "This looks like a timezone problem." # log time zone issue as warning once per day: self._logWarnOnce("_next_simByTimeWarn", - ("Found at least one log entry with a timestamp more" + - " than a minute from the current time. This is rarely a problem."), - ("Please check this jail and associated logs as" + - " there is potentially a timezone or latency problem: %s", - line)) + ("Found a log entry with a timestamp %ss %s the current time. %s", + abs(delta), + "before" if delta <= 0 else "after", + latency if -15*60 <= delta <= 0 else timezone), + ("Please check this jail and associated logs as there" + + " is potentially a timezone or latency problem: %s", + line)) # simulate now as date: date = MyTime.time() self.__lastDate = date From d7afcde2e10fa61bb4adb6604862074461e19f16 Mon Sep 17 00:00:00 2001 From: Daniel Brooks Date: Mon, 12 Jul 2021 17:09:58 -0700 Subject: [PATCH 157/240] add a warning message for dates in the future and a test that checks which message was output for which time deltas. --- fail2ban/server/filter.py | 11 ++++++++++- fail2ban/tests/filtertestcase.py | 30 ++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+), 1 deletion(-) diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index e327228e..16e7d2a6 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -662,13 +662,22 @@ class Filter(JailThread): delta = date - MyTime.time() if (date is None or abs(delta) > 60): latency = "This could indicate a latency problem." + synchronization = "This could be a clock synchronization" + \ + " problem; are you sure that NTP is set up?" timezone = "This looks like a timezone problem." + msg = "" + if -15*60 <= delta <= 0: + msg = latency + elif 0 < delta <= 15*60: + msg = synchronization + else: + msg = timezone # log time zone issue as warning once per day: self._logWarnOnce("_next_simByTimeWarn", ("Found a log entry with a timestamp %ss %s the current time. %s", abs(delta), "before" if delta <= 0 else "after", - latency if -15*60 <= delta <= 0 else timezone), + msg), ("Please check this jail and associated logs as there" + " is potentially a timezone or latency problem: %s", line)) diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 9e81006b..6b0218f0 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -479,6 +479,36 @@ class IgnoreIP(LogCaptureTestCase): finally: tearDownMyTime() + def testWrongTimeDelta(self): + try: + self.filter.addFailRegex('fail from $') + self.filter.setDatePattern(r'{^LN-BEG}%Y-%m-%d %H:%M:%S(?:\s*%Z)?\s') + self.filter.setMaxRetry(5); # don't ban here + self.filter.inOperation = True; # real processing (all messages are new) + expectable_messages = ["timezone problem.", "clock synchronization problem;", "latency problem."] + cases = [(-90*60, 0), + (-10*60, 1), + (-30, None), + (30, None), + (10*60, 2), + (90*60, 0)] + for idx, (delta, expect) in enumerate(cases): + MyTime.setTime(1572138000+delta) + setattr(self.filter, "_next_simByTimeWarn", -1) + self.pruneLog('[phase {phase}] log entries offset by {delta}s'.format(phase=idx+1, delta=delta)) + for i in (1,2,3): + self.filter.processLineAndAdd('2019-10-27 02:00:00 fail from 192.0.2.15'); + for (idx, msg) in enumerate(expectable_messages): + if idx == expect: + self.assertLogged(expectable_messages[idx]) + else: + self.assertNotLogged(expectable_messages[idx]) + self.assertLogged( + "192.0.2.15:1", "192.0.2.15:2", "192.0.2.15:3", + "Total # of detected failures: 3.", wait=True) + finally: + tearDownMyTime() + def testAddAttempt(self): self.filter.setMaxRetry(3) for i in xrange(1, 1+3): From ec043cd2029fe52d56cf526fac31e79b79d73467 Mon Sep 17 00:00:00 2001 From: sebres Date: Sun, 19 Sep 2021 21:58:42 +0200 Subject: [PATCH 158/240] simplifying logic and shortening messages (delta in minutes; removed clock synchronization, because it is rarely an issue on fail2ban side, e. g. for remote logs only, etc) --- fail2ban/server/filter.py | 41 +++++++++--------------- fail2ban/tests/filtertestcase.py | 53 ++++++++++++++------------------ 2 files changed, 38 insertions(+), 56 deletions(-) diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index 16e7d2a6..08bc2e59 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -655,46 +655,35 @@ class Filter(JailThread): tupleLine = ("", self.__lastTimeText, line) date = self.__lastDate - if self.checkFindTime: + if self.checkFindTime and date is not None: # if in operation (modifications have been really found): if self.inOperation: # if weird date - we'd simulate now for timeing issue (too large deviation from now): - delta = date - MyTime.time() - if (date is None or abs(delta) > 60): - latency = "This could indicate a latency problem." - synchronization = "This could be a clock synchronization" + \ - " problem; are you sure that NTP is set up?" - timezone = "This looks like a timezone problem." - msg = "" - if -15*60 <= delta <= 0: - msg = latency - elif 0 < delta <= 15*60: - msg = synchronization - else: - msg = timezone - # log time zone issue as warning once per day: + delta = int(date - MyTime.time()) + if abs(delta) > 60: + delta /= 60 + # log timing issue as warning once per day: self._logWarnOnce("_next_simByTimeWarn", - ("Found a log entry with a timestamp %ss %s the current time. %s", - abs(delta), - "before" if delta <= 0 else "after", - msg), - ("Please check this jail and associated logs as there" + - " is potentially a timezone or latency problem: %s", + ("Detected a log entry %sm %s the current time in operation mode. " + "This looks like a %s problem. Treating such entries as if they just happened.", + abs(delta), "before" if delta < 0 else "after", + "latency" if -55 <= delta < 0 else "timezone" + ), + ("Please check a jail for a timing issue. Line with odd timestamp: %s", line)) # simulate now as date: date = MyTime.time() self.__lastDate = date else: # in initialization (restore) phase, if too old - ignore: - if date is not None and date < MyTime.time() - self.getFindTime(): + if date < MyTime.time() - self.getFindTime(): # log time zone issue as warning once per day: self._logWarnOnce("_next_ignByTimeWarn", - ("Ignoring all log entries older than %s; these are probably" + + ("Ignoring all log entries older than %ss; these are probably" + " messages generated while fail2ban was not running.", self.getFindTime()), - ("Please check this jail and associated logs as" + - " there is potentially a timezone or latency problem: %s", - line)) + ("Please check a jail for a timing issue. Line with odd timestamp: %s", + line)) # ignore - too old (obsolete) entry: return [] diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 6b0218f0..8e151f77 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -444,11 +444,11 @@ class IgnoreIP(LogCaptureTestCase): def testTimeJump_InOperation(self): self._testTimeJump(inOperation=True) - def testWrongTimeZone(self): + def testWrongTimeOrTZ(self): try: self.filter.addFailRegex('fail from $') self.filter.setDatePattern(r'{^LN-BEG}%Y-%m-%d %H:%M:%S(?:\s*%Z)?\s') - self.filter.setMaxRetry(5); # don't ban here + self.filter.setMaxRetry(50); # don't ban here self.filter.inOperation = True; # real processing (all messages are new) # current time is 1h later than log-entries: MyTime.setTime(1572138000+3600) @@ -476,36 +476,29 @@ class IgnoreIP(LogCaptureTestCase): "Match without a timestamp:", "192.0.2.17:1", "192.0.2.17:2", "192.0.2.17:3", "Total # of detected failures: 9.", all=True, wait=True) - finally: - tearDownMyTime() - - def testWrongTimeDelta(self): - try: - self.filter.addFailRegex('fail from $') - self.filter.setDatePattern(r'{^LN-BEG}%Y-%m-%d %H:%M:%S(?:\s*%Z)?\s') - self.filter.setMaxRetry(5); # don't ban here - self.filter.inOperation = True; # real processing (all messages are new) - expectable_messages = ["timezone problem.", "clock synchronization problem;", "latency problem."] - cases = [(-90*60, 0), - (-10*60, 1), - (-30, None), - (30, None), - (10*60, 2), - (90*60, 0)] - for idx, (delta, expect) in enumerate(cases): + # + phase = 3 + for delta, expect in ( + (-90*60, "timezone"), #90 minutes after + (-60*60, "timezone"), #60 minutes after + (-10*60, "timezone"), #10 minutes after + (-59, None), #59 seconds after + (59, None), #59 seconds before + (61, "latency"), #>1 minute before + (55*60, "latency"), #55 minutes before + (90*60, "timezone") #90 minutes before + ): + phase += 1 MyTime.setTime(1572138000+delta) setattr(self.filter, "_next_simByTimeWarn", -1) - self.pruneLog('[phase {phase}] log entries offset by {delta}s'.format(phase=idx+1, delta=delta)) - for i in (1,2,3): - self.filter.processLineAndAdd('2019-10-27 02:00:00 fail from 192.0.2.15'); - for (idx, msg) in enumerate(expectable_messages): - if idx == expect: - self.assertLogged(expectable_messages[idx]) - else: - self.assertNotLogged(expectable_messages[idx]) - self.assertLogged( - "192.0.2.15:1", "192.0.2.15:2", "192.0.2.15:3", - "Total # of detected failures: 3.", wait=True) + self.pruneLog('[phase {phase}] log entries offset by {delta}s'.format(phase=phase, delta=delta)) + self.filter.processLineAndAdd('2019-10-27 02:00:00 fail from 192.0.2.15'); + self.assertLogged("Found 192.0.2.15", wait=True) + if expect: + self.assertLogged(("timezone problem", "latency problem")[int(expect == "latency")], all=True) + self.assertNotLogged(("timezone problem", "latency problem")[int(expect != "latency")]) + else: + self.assertNotLogged("timezone problem", "latency problem", all=True) finally: tearDownMyTime() From 621d8cae17f16cfea43a394149217963034d344b Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 20 Sep 2021 02:20:22 +0200 Subject: [PATCH 159/240] restore backwards compatibility for date None --- fail2ban/server/filter.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index 08bc2e59..f49fe198 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -618,6 +618,7 @@ class Filter(JailThread): noDate = False if date: tupleLine = line + line = "".join(line) self.__lastTimeText = tupleLine[1] self.__lastDate = date else: @@ -654,6 +655,8 @@ class Filter(JailThread): if self.__lastDate and self.__lastDate > MyTime.time() - 60: tupleLine = ("", self.__lastTimeText, line) date = self.__lastDate + elif self.checkFindTime and self.inOperation: + date = MyTime.time() if self.checkFindTime and date is not None: # if in operation (modifications have been really found): From 17eed32e03e86f6b42a0f0ad227345713b7c7cd7 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Tue, 21 Sep 2021 16:00:37 +0200 Subject: [PATCH 160/240] Update filtertestcase.py --- fail2ban/tests/filtertestcase.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 8e151f77..be874697 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -457,13 +457,18 @@ class IgnoreIP(LogCaptureTestCase): for i in (1,2,3): self.filter.processLineAndAdd('2019-10-27 02:00:00 fail from 192.0.2.15'); # +3 = 3 self.assertLogged( + "Detected a log entry 60m before the current time in operation mode. This looks like a timezone problem.", + "Please check a jail for a timing issue.", "192.0.2.15:1", "192.0.2.15:2", "192.0.2.15:3", - "Total # of detected failures: 3.", wait=True) + "Total # of detected failures: 3.", all=True, wait=True) # + setattr(self.filter, "_next_simByTimeWarn", -1) self.pruneLog("[phase 2] wrong TZ given in log") for i in (1,2,3): self.filter.processLineAndAdd('2019-10-27 04:00:00 GMT fail from 192.0.2.16'); # +3 = 6 self.assertLogged( + "Detected a log entry 120m after the current time in operation mode. This looks like a timezone problem.", + "Please check a jail for a timing issue.", "192.0.2.16:1", "192.0.2.16:2", "192.0.2.16:3", "Total # of detected failures: 6.", all=True, wait=True) self.assertNotLogged("Found a match but no valid date/time found") @@ -496,7 +501,7 @@ class IgnoreIP(LogCaptureTestCase): self.assertLogged("Found 192.0.2.15", wait=True) if expect: self.assertLogged(("timezone problem", "latency problem")[int(expect == "latency")], all=True) - self.assertNotLogged(("timezone problem", "latency problem")[int(expect != "latency")]) + self.assertNotLogged(("timezone problem", "latency problem")[int(expect != "latency")], all=True) else: self.assertNotLogged("timezone problem", "latency problem", all=True) finally: From d086317cc83ec7aa6e6772dc91525bce33ed7200 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Tue, 21 Sep 2021 16:05:53 +0200 Subject: [PATCH 161/240] Update filter.py --- fail2ban/server/filter.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index f49fe198..a13b1fb5 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -664,7 +664,7 @@ class Filter(JailThread): # if weird date - we'd simulate now for timeing issue (too large deviation from now): delta = int(date - MyTime.time()) if abs(delta) > 60: - delta /= 60 + delta //= 60 # log timing issue as warning once per day: self._logWarnOnce("_next_simByTimeWarn", ("Detected a log entry %sm %s the current time in operation mode. " From f8f59dd31a46f6ad35b38df1f29a8f81c7e88960 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Fri, 1 Oct 2021 14:58:25 +0200 Subject: [PATCH 162/240] added test cases covering different messages adjusted to new log-format (gh-3116) --- fail2ban/tests/files/logs/lighttpd-auth | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/fail2ban/tests/files/logs/lighttpd-auth b/fail2ban/tests/files/logs/lighttpd-auth index 184dba33..c8a922b5 100644 --- a/fail2ban/tests/files/logs/lighttpd-auth +++ b/fail2ban/tests/files/logs/lighttpd-auth @@ -1,4 +1,3 @@ -#authentification failure (mod_auth) # failJSON: { "time": "2011-12-25T17:09:20", "match": true , "host": "4.4.4.4" } 2011-12-25 17:09:20: (http_auth.c.875) password doesn't match for /gitweb/ username: francois, IP: 4.4.4.4 # failJSON: { "time": "2012-09-26T10:24:35", "match": true , "host": "4.4.4.4" } @@ -7,3 +6,9 @@ 2013-08-25 00:24:55: (http_auth.c.877) get_password failed, IP: 4.4.4.4 # failJSON: { "time": "2018-01-16T14:10:32", "match": true , "host": "192.0.2.1", "desc": "http_auth -> mod_auth, gh-2018" } 2018-01-16 14:10:32: (mod_auth.c.525) password doesn't match for /test-url username: test, IP: 192.0.2.1 +# failJSON: { "time": "2021-09-30T16:05:33", "match": true , "host": "192.0.2.2", "user":"test", "desc": "gh-3116" } +2021-09-30 16:05:33: mod_auth.c.828) password doesn't match for /secure/ username: test IP: 192.0.2.2 +# failJSON: { "time": "2021-09-30T17:44:37", "match": true , "host": "192.0.2.3", "user":"tester", "desc": "gh-3116" } +2021-09-30 17:44:37: (mod_auth.c.791) digest: auth failed for tester : wrong password, IP: 192.0.2.3 +# failJSON: { "time": "2021-09-30T17:44:37", "match": true , "host": "192.0.2.4", "desc": "gh-3116" } +2021-09-30 17:44:37: (mod_auth.c.791) digest: auth failed: uri mismatch (/uri1 != /uri2), IP: 192.0.2.4 From ba839af8ad594439002f468b71b4c6caafb5de03 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Fri, 1 Oct 2021 15:03:24 +0200 Subject: [PATCH 163/240] filter.d/lighttpd-auth.conf: adjusted to the current source code + avoiding catch-all's, etc (gh-3116) --- config/filter.d/lighttpd-auth.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/filter.d/lighttpd-auth.conf b/config/filter.d/lighttpd-auth.conf index a68f4f4d..dcf19d3e 100644 --- a/config/filter.d/lighttpd-auth.conf +++ b/config/filter.d/lighttpd-auth.conf @@ -3,7 +3,7 @@ [Definition] -failregex = ^: \((?:http|mod)_auth\.c\.\d+\) (?:password doesn\'t match .* username: .*|digest: auth failed for .*: wrong password|get_password failed), IP: \s*$ +failregex = ^\s*(?:: )?\(?(?:http|mod)_auth\.c\.\d+\) (?:password doesn\'t match for (?:\S+|.*?) username:\s+(?:\S+|.*?)\s*|digest: auth failed(?: for\s+(?:\S+|.*?)\s*)?: (?:wrong password|uri mismatch \([^\)]*\))|get_password failed),? IP: \s*$ ignoreregex = From 1e8ce20b325e1a60c92025e0512fe0a547552293 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Sz=C3=A9pe?= Date: Sun, 17 Oct 2021 19:53:10 +0200 Subject: [PATCH 164/240] Display ChangeLog as a Markdown document --- .gitattributes | 1 + 1 file changed, 1 insertion(+) create mode 100644 .gitattributes diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 00000000..0cbdbf83 --- /dev/null +++ b/.gitattributes @@ -0,0 +1 @@ +ChangeLog linguist-language=Markdown From d91d949e95d5fbf70ca8679319dac7c8c2ba7a61 Mon Sep 17 00:00:00 2001 From: Mike Gilbert Date: Tue, 8 Jun 2021 17:48:12 -0400 Subject: [PATCH 165/240] tests: improve detection of readable systemd journal Look for system.journal in journal sub-directory. Add -readable to the find command. Bug: https://bugs.gentoo.org/794931 --- fail2ban/tests/filtertestcase.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index be874697..2b3dd995 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -1396,7 +1396,7 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover # check one at at time until the first hit for systemd_var in 'system-runtime-logs', 'system-state-logs': tmp = Utils.executeCmd( - 'find "$(systemd-path %s)" -name system.journal' % systemd_var, + 'find "$(systemd-path %s)/journal" -name system.journal -readable' % systemd_var, timeout=10, shell=True, output=True ) self.assertTrue(tmp) From 1cd3118da37b2281f841adf298fa20eff81b5017 Mon Sep 17 00:00:00 2001 From: Sylvestre Ledru Date: Sat, 23 Oct 2021 17:05:30 +0200 Subject: [PATCH 166/240] Fix typo found by lintian (Debian static analyzer) --- ChangeLog | 2 +- fail2ban/protocol.py | 2 +- man/fail2ban-client.1 | 2 +- man/jail.conf.5 | 4 ++-- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/ChangeLog b/ChangeLog index 175d087a..a022f17c 100644 --- a/ChangeLog +++ b/ChangeLog @@ -222,7 +222,7 @@ Yes, Hrrrm... ### New Features * new replacement tags for failregex to match subnets in form of IP-addresses with CIDR mask (gh-2559): - `` - helper regex to match CIDR (simple integer form of net-mask); - - `` - regex to match sub-net adresses (in form of IP/CIDR, also single IP is matched, so part /CIDR is optional); + - `` - regex to match sub-net addresses (in form of IP/CIDR, also single IP is matched, so part /CIDR is optional); * grouped tags (``, ``, ``) recognize IP addresses enclosed in square brackets * new failregex-flag tag `` for failregex, signaled that the access to service was gained (ATM used similar to tag ``, but it does not add the log-line to matches, gh-2279) diff --git a/fail2ban/protocol.py b/fail2ban/protocol.py index 0a4c84ed..58102b55 100644 --- a/fail2ban/protocol.py +++ b/fail2ban/protocol.py @@ -134,7 +134,7 @@ protocol = [ ["get ignoreregex", "gets the list of regular expressions which matches patterns to ignore for "], ["get findtime", "gets the time for which the filter will look back for failures for "], ["get bantime", "gets the time a host is banned for "], -["get datepattern", "gets the patern used to match date/times for "], +["get datepattern", "gets the pattern used to match date/times for "], ["get usedns", "gets the usedns setting for "], ["get banip [|--with-time]", "gets the list of of banned IP addresses for . Optionally the separator character ('', default is space) or the option '--with-time' (printing the times of ban) may be specified. The IPs are ordered by end of ban."], ["get maxretry", "gets the number of failures allowed for "], diff --git a/man/fail2ban-client.1 b/man/fail2ban-client.1 index 81473daa..2da467ec 100644 --- a/man/fail2ban-client.1 +++ b/man/fail2ban-client.1 @@ -415,7 +415,7 @@ gets the time a host is banned for .TP \fBget datepattern\fR -gets the patern used to match +gets the pattern used to match date/times for .TP \fBget usedns\fR diff --git a/man/jail.conf.5 b/man/jail.conf.5 index 8ff407f0..052fce80 100644 --- a/man/jail.conf.5 +++ b/man/jail.conf.5 @@ -487,7 +487,7 @@ is the regex (\fBreg\fRular \fBex\fRpression) that will match failed attempts. T .IP \fI\fR - helper regex to match CIDR (simple integer form of net-mask). .IP -\fI\fR - regex to match sub-net adresses (in form of IP/CIDR, also single IP is matched, so part /CIDR is optional). +\fI\fR - regex to match sub-net addresses (in form of IP/CIDR, also single IP is matched, so part /CIDR is optional). .IP \fI...\fR - free regex capturing group targeting identifier used for ban (instead of IP address or hostname). .IP @@ -540,7 +540,7 @@ There are several prefixes and words with special meaning that could be specifie .IP \fI{UNB}\fR - prefix to disable automatic word boundaries in regex. .IP -\fI{NONE}\fR - value would allow to find failures totally without date-time in log message. Filter will use now as a timestamp (or last known timestamp from previous line with timestamp). +\fI{NONE}\fR - value would allow one to find failures totally without date-time in log message. Filter will use now as a timestamp (or last known timestamp from previous line with timestamp). .RE .TP \fBjournalmatch\fR From 3245b8018b2606a5278df42790fac06d5a455f00 Mon Sep 17 00:00:00 2001 From: Sylvestre Ledru Date: Sat, 23 Oct 2021 17:38:20 +0200 Subject: [PATCH 167/240] Add the Debian path to roundcube error logs --- config/paths-debian.conf | 2 ++ 1 file changed, 2 insertions(+) diff --git a/config/paths-debian.conf b/config/paths-debian.conf index e096f972..1f5ea37d 100644 --- a/config/paths-debian.conf +++ b/config/paths-debian.conf @@ -26,3 +26,5 @@ exim_main_log = /var/log/exim4/mainlog # was in debian squeezy but not in wheezy # /etc/proftpd/proftpd.conf (SystemLog) proftpd_log = /var/log/proftpd/proftpd.log + +roundcube_errors_log = /var/log/roundcube/errors.log From 4b54a07d71a6ce1c85a3eae92bace6c0dadcdcfb Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 1 Nov 2021 11:43:54 +0100 Subject: [PATCH 168/240] Revert "`action.d/firewallcmd-*.conf` (multiport only): fixed port range selector, replacing `:` with `-`;" This reverts the incompatibility #3047 introduced by commit a038fd5dfe8cb0714472833604735b83462a217d (#2821). --- config/action.d/firewallcmd-ipset.conf | 2 +- config/action.d/firewallcmd-multiport.conf | 4 +-- config/action.d/firewallcmd-new.conf | 4 +-- config/action.d/firewallcmd-rich-rules.conf | 4 +-- fail2ban/tests/servertestcase.py | 32 ++++++++++----------- 5 files changed, 23 insertions(+), 23 deletions(-) diff --git a/config/action.d/firewallcmd-ipset.conf b/config/action.d/firewallcmd-ipset.conf index 66358f23..42513933 100644 --- a/config/action.d/firewallcmd-ipset.conf +++ b/config/action.d/firewallcmd-ipset.conf @@ -69,7 +69,7 @@ allports = -p # Option: multiport # Notes.: addition to block access only to specific ports # Usage.: use in jail config: banaction = firewallcmd-ipset[actiontype=] -multiport = -p -m multiport --dports "$(echo '' | sed s/:/-/g)" +multiport = -p -m multiport --dports ipmset = f2b- familyopt = diff --git a/config/action.d/firewallcmd-multiport.conf b/config/action.d/firewallcmd-multiport.conf index 0c401f1b..81540e5b 100644 --- a/config/action.d/firewallcmd-multiport.conf +++ b/config/action.d/firewallcmd-multiport.conf @@ -11,9 +11,9 @@ before = firewallcmd-common.conf actionstart = firewall-cmd --direct --add-chain filter f2b- firewall-cmd --direct --add-rule filter f2b- 1000 -j RETURN - firewall-cmd --direct --add-rule filter 0 -m conntrack --ctstate NEW -p -m multiport --dports "$(echo '' | sed s/:/-/g)" -j f2b- + firewall-cmd --direct --add-rule filter 0 -m conntrack --ctstate NEW -p -m multiport --dports -j f2b- -actionstop = firewall-cmd --direct --remove-rule filter 0 -m conntrack --ctstate NEW -p -m multiport --dports "$(echo '' | sed s/:/-/g)" -j f2b- +actionstop = firewall-cmd --direct --remove-rule filter 0 -m conntrack --ctstate NEW -p -m multiport --dports -j f2b- firewall-cmd --direct --remove-rules filter f2b- firewall-cmd --direct --remove-chain filter f2b- diff --git a/config/action.d/firewallcmd-new.conf b/config/action.d/firewallcmd-new.conf index 7b08603c..b06f5ccd 100644 --- a/config/action.d/firewallcmd-new.conf +++ b/config/action.d/firewallcmd-new.conf @@ -10,9 +10,9 @@ before = firewallcmd-common.conf actionstart = firewall-cmd --direct --add-chain filter f2b- firewall-cmd --direct --add-rule filter f2b- 1000 -j RETURN - firewall-cmd --direct --add-rule filter 0 -m state --state NEW -p -m multiport --dports "$(echo '' | sed s/:/-/g)" -j f2b- + firewall-cmd --direct --add-rule filter 0 -m state --state NEW -p -m multiport --dports -j f2b- -actionstop = firewall-cmd --direct --remove-rule filter 0 -m state --state NEW -p -m multiport --dports "$(echo '' | sed s/:/-/g)" -j f2b- +actionstop = firewall-cmd --direct --remove-rule filter 0 -m state --state NEW -p -m multiport --dports -j f2b- firewall-cmd --direct --remove-rules filter f2b- firewall-cmd --direct --remove-chain filter f2b- diff --git a/config/action.d/firewallcmd-rich-rules.conf b/config/action.d/firewallcmd-rich-rules.conf index 803d7d12..75a27d88 100644 --- a/config/action.d/firewallcmd-rich-rules.conf +++ b/config/action.d/firewallcmd-rich-rules.conf @@ -37,8 +37,8 @@ actioncheck = fwcmd_rich_rule = rule family='' source address='' port port='$p' protocol='' %(rich-suffix)s -actionban = ports="$(echo '' | sed s/:/-/g)"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="%(fwcmd_rich_rule)s"; done +actionban = ports=""; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="%(fwcmd_rich_rule)s"; done -actionunban = ports="$(echo '' | sed s/:/-/g)"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --remove-rich-rule="%(fwcmd_rich_rule)s"; done +actionunban = ports=""; for p in $(echo $ports | tr ", " " "); do firewall-cmd --remove-rich-rule="%(fwcmd_rich_rule)s"; done rich-suffix = \ No newline at end of file diff --git a/fail2ban/tests/servertestcase.py b/fail2ban/tests/servertestcase.py index fc505552..6cf45f53 100644 --- a/fail2ban/tests/servertestcase.py +++ b/fail2ban/tests/servertestcase.py @@ -1793,18 +1793,18 @@ class ServerConfigReaderTests(LogCaptureTestCase): 'ip4-start': ( "`firewall-cmd --direct --add-chain ipv4 filter f2b-j-w-fwcmd-mp`", "`firewall-cmd --direct --add-rule ipv4 filter f2b-j-w-fwcmd-mp 1000 -j RETURN`", - """`firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports "$(echo 'http,https' | sed s/:/-/g)" -j f2b-j-w-fwcmd-mp`""", + "`firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports http,https -j f2b-j-w-fwcmd-mp`", ), 'ip6-start': ( "`firewall-cmd --direct --add-chain ipv6 filter f2b-j-w-fwcmd-mp`", "`firewall-cmd --direct --add-rule ipv6 filter f2b-j-w-fwcmd-mp 1000 -j RETURN`", - """`firewall-cmd --direct --add-rule ipv6 filter INPUT_direct 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports "$(echo 'http,https' | sed s/:/-/g)" -j f2b-j-w-fwcmd-mp`""", + "`firewall-cmd --direct --add-rule ipv6 filter INPUT_direct 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports http,https -j f2b-j-w-fwcmd-mp`", ), 'stop': ( - """`firewall-cmd --direct --remove-rule ipv4 filter INPUT_direct 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports "$(echo 'http,https' | sed s/:/-/g)" -j f2b-j-w-fwcmd-mp`""", + "`firewall-cmd --direct --remove-rule ipv4 filter INPUT_direct 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports http,https -j f2b-j-w-fwcmd-mp`", "`firewall-cmd --direct --remove-rules ipv4 filter f2b-j-w-fwcmd-mp`", "`firewall-cmd --direct --remove-chain ipv4 filter f2b-j-w-fwcmd-mp`", - """`firewall-cmd --direct --remove-rule ipv6 filter INPUT_direct 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports "$(echo 'http,https' | sed s/:/-/g)" -j f2b-j-w-fwcmd-mp`""", + "`firewall-cmd --direct --remove-rule ipv6 filter INPUT_direct 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports http,https -j f2b-j-w-fwcmd-mp`", "`firewall-cmd --direct --remove-rules ipv6 filter f2b-j-w-fwcmd-mp`", "`firewall-cmd --direct --remove-chain ipv6 filter f2b-j-w-fwcmd-mp`", ), @@ -1872,21 +1872,21 @@ class ServerConfigReaderTests(LogCaptureTestCase): 'ip4': (' f2b-j-w-fwcmd-ipset ',), 'ip6': (' f2b-j-w-fwcmd-ipset6 ',), 'ip4-start': ( "`ipset create f2b-j-w-fwcmd-ipset hash:ip timeout 0 `", - """`firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 0 -p tcp -m multiport --dports "$(echo 'http' | sed s/:/-/g)" -m set --match-set f2b-j-w-fwcmd-ipset src -j REJECT --reject-with icmp-port-unreachable`""", + "`firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset src -j REJECT --reject-with icmp-port-unreachable`", ), 'ip6-start': ( "`ipset create f2b-j-w-fwcmd-ipset6 hash:ip timeout 0 family inet6`", - """`firewall-cmd --direct --add-rule ipv6 filter INPUT_direct 0 -p tcp -m multiport --dports "$(echo 'http' | sed s/:/-/g)" -m set --match-set f2b-j-w-fwcmd-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`""", + "`firewall-cmd --direct --add-rule ipv6 filter INPUT_direct 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", ), 'flush': ( "`ipset flush f2b-j-w-fwcmd-ipset`", "`ipset flush f2b-j-w-fwcmd-ipset6`", ), 'stop': ( - """`firewall-cmd --direct --remove-rule ipv4 filter INPUT_direct 0 -p tcp -m multiport --dports "$(echo 'http' | sed s/:/-/g)" -m set --match-set f2b-j-w-fwcmd-ipset src -j REJECT --reject-with icmp-port-unreachable`""", + "`firewall-cmd --direct --remove-rule ipv4 filter INPUT_direct 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset src -j REJECT --reject-with icmp-port-unreachable`", "`ipset flush f2b-j-w-fwcmd-ipset`", "`ipset destroy f2b-j-w-fwcmd-ipset`", - """`firewall-cmd --direct --remove-rule ipv6 filter INPUT_direct 0 -p tcp -m multiport --dports "$(echo 'http' | sed s/:/-/g)" -m set --match-set f2b-j-w-fwcmd-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`""", + "`firewall-cmd --direct --remove-rule ipv6 filter INPUT_direct 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", "`ipset flush f2b-j-w-fwcmd-ipset6`", "`ipset destroy f2b-j-w-fwcmd-ipset6`", ), @@ -1943,32 +1943,32 @@ class ServerConfigReaderTests(LogCaptureTestCase): ('j-fwcmd-rr', 'firewallcmd-rich-rules[port="22:24", protocol="tcp"]', { 'ip4': ("family='ipv4'", "icmp-port-unreachable",), 'ip6': ("family='ipv6'", 'icmp6-port-unreachable',), 'ip4-ban': ( - """`ports="$(echo '22:24' | sed s/:/-/g)"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="rule family='ipv4' source address='192.0.2.1' port port='$p' protocol='tcp' reject type='icmp-port-unreachable'"; done`""", + """`ports="22:24"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="rule family='ipv4' source address='192.0.2.1' port port='$p' protocol='tcp' reject type='icmp-port-unreachable'"; done`""", ), 'ip4-unban': ( - """`ports="$(echo '22:24' | sed s/:/-/g)"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --remove-rich-rule="rule family='ipv4' source address='192.0.2.1' port port='$p' protocol='tcp' reject type='icmp-port-unreachable'"; done`""", + """`ports="22:24"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --remove-rich-rule="rule family='ipv4' source address='192.0.2.1' port port='$p' protocol='tcp' reject type='icmp-port-unreachable'"; done`""", ), 'ip6-ban': ( - """ `ports="$(echo '22:24' | sed s/:/-/g)"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="rule family='ipv6' source address='2001:db8::' port port='$p' protocol='tcp' reject type='icmp6-port-unreachable'"; done`""", + """ `ports="22:24"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="rule family='ipv6' source address='2001:db8::' port port='$p' protocol='tcp' reject type='icmp6-port-unreachable'"; done`""", ), 'ip6-unban': ( - """`ports="$(echo '22:24' | sed s/:/-/g)"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --remove-rich-rule="rule family='ipv6' source address='2001:db8::' port port='$p' protocol='tcp' reject type='icmp6-port-unreachable'"; done`""", + """`ports="22:24"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --remove-rich-rule="rule family='ipv6' source address='2001:db8::' port port='$p' protocol='tcp' reject type='icmp6-port-unreachable'"; done`""", ), }), # firewallcmd-rich-logging -- ('j-fwcmd-rl', 'firewallcmd-rich-logging[port="22:24", protocol="tcp"]', { 'ip4': ("family='ipv4'", "icmp-port-unreachable",), 'ip6': ("family='ipv6'", 'icmp6-port-unreachable',), 'ip4-ban': ( - """`ports="$(echo '22:24' | sed s/:/-/g)"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="rule family='ipv4' source address='192.0.2.1' port port='$p' protocol='tcp' log prefix='f2b-j-fwcmd-rl' level='info' limit value='1/m' reject type='icmp-port-unreachable'"; done`""", + """`ports="22:24"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="rule family='ipv4' source address='192.0.2.1' port port='$p' protocol='tcp' log prefix='f2b-j-fwcmd-rl' level='info' limit value='1/m' reject type='icmp-port-unreachable'"; done`""", ), 'ip4-unban': ( - """`ports="$(echo '22:24' | sed s/:/-/g)"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --remove-rich-rule="rule family='ipv4' source address='192.0.2.1' port port='$p' protocol='tcp' log prefix='f2b-j-fwcmd-rl' level='info' limit value='1/m' reject type='icmp-port-unreachable'"; done`""", + """`ports="22:24"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --remove-rich-rule="rule family='ipv4' source address='192.0.2.1' port port='$p' protocol='tcp' log prefix='f2b-j-fwcmd-rl' level='info' limit value='1/m' reject type='icmp-port-unreachable'"; done`""", ), 'ip6-ban': ( - """ `ports="$(echo '22:24' | sed s/:/-/g)"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="rule family='ipv6' source address='2001:db8::' port port='$p' protocol='tcp' log prefix='f2b-j-fwcmd-rl' level='info' limit value='1/m' reject type='icmp6-port-unreachable'"; done`""", + """ `ports="22:24"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="rule family='ipv6' source address='2001:db8::' port port='$p' protocol='tcp' log prefix='f2b-j-fwcmd-rl' level='info' limit value='1/m' reject type='icmp6-port-unreachable'"; done`""", ), 'ip6-unban': ( - """`ports="$(echo '22:24' | sed s/:/-/g)"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --remove-rich-rule="rule family='ipv6' source address='2001:db8::' port port='$p' protocol='tcp' log prefix='f2b-j-fwcmd-rl' level='info' limit value='1/m' reject type='icmp6-port-unreachable'"; done`""", + """`ports="22:24"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --remove-rich-rule="rule family='ipv6' source address='2001:db8::' port port='$p' protocol='tcp' log prefix='f2b-j-fwcmd-rl' level='info' limit value='1/m' reject type='icmp6-port-unreachable'"; done`""", ), }), ) From 7678f5982733f1d20bbec8d04006ac78cded10db Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 21 Sep 2021 20:42:51 +0200 Subject: [PATCH 169/240] better format of time delta (using seconds2str); increase stability for systemd test-cases --- fail2ban/server/filter.py | 7 ++--- fail2ban/server/mytime.py | 48 ++++++++++++++++++++++++++++++++ fail2ban/tests/filtertestcase.py | 8 ++++-- fail2ban/tests/misctestcase.py | 12 ++++++++ 4 files changed, 68 insertions(+), 7 deletions(-) diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index a13b1fb5..d2cd0ce9 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -664,13 +664,12 @@ class Filter(JailThread): # if weird date - we'd simulate now for timeing issue (too large deviation from now): delta = int(date - MyTime.time()) if abs(delta) > 60: - delta //= 60 # log timing issue as warning once per day: self._logWarnOnce("_next_simByTimeWarn", - ("Detected a log entry %sm %s the current time in operation mode. " + ("Detected a log entry %s %s the current time in operation mode. " "This looks like a %s problem. Treating such entries as if they just happened.", - abs(delta), "before" if delta < 0 else "after", - "latency" if -55 <= delta < 0 else "timezone" + MyTime.seconds2str(abs(delta)), "before" if delta < 0 else "after", + "latency" if -3300 <= delta < 0 else "timezone" ), ("Please check a jail for a timing issue. Line with odd timestamp: %s", line)) diff --git a/fail2ban/server/mytime.py b/fail2ban/server/mytime.py index 49199887..d09c9d41 100644 --- a/fail2ban/server/mytime.py +++ b/fail2ban/server/mytime.py @@ -161,3 +161,51 @@ class MyTime: val = rexp.sub(rpl, val) val = MyTime._str2sec_fini.sub(r"\1+\2", val) return eval(val) + + class seconds2str(): + """Converts seconds to string on demand (if string representation needed). + Ex: seconds2str(86400*390) = 1y 3w 4d + seconds2str(86400*368) = 1y 3d + seconds2str(86400*365.5) = 1y + seconds2str(86400*2+3600*7+60*15) = 2d 7h 15m + seconds2str(86400*2+3599) = 2d 1h + seconds2str(3600-5) = 1h + seconds2str(3600-10) = 59m 50s + seconds2str(59) = 59s + """ + def __init__(self, sec): + self.sec = sec + def __str__(self): + # s = str(datetime.timedelta(seconds=int(self.sec))) + # return s if s[-3:] != ":00" else s[:-3] + s = self.sec; r = ""; c = 3 + # automatic accuracy: round by large values (upto 1 minute, or 1 day by year): + if s >= 3570: + if s >= 31536000: + s = int(round(float(s)/86400)*86400) + elif s >= 86400: + s = int(round(float(s)/60)*60) + else: + s = int(round(float(s)/10)*10) + for n, m in ( + ('y', 31536000), # a year as 365*24*60*60 (don't need to consider leap year by this accuracy) + ('w', 604800), # a week as 24*60*60*7 + ('d', 86400), # a day as 24*60*60 + ('h', 3600), # a hour as 60*60 + ('m', 60), # a minute + ('s', 1) # a second + ): + if s >= m: + c -= 1 + r += ' ' + str(s//m) + n + s %= m + # stop if no remaining part or no repeat needed (too small granularity): + if not s or not c: break + elif c < 3: + # avoid too small granularity: + c -= 1 + if not c: break + # + return r[1:] + def __repr__(self): + return self.__str__() diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 2b3dd995..343cfe04 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -457,7 +457,7 @@ class IgnoreIP(LogCaptureTestCase): for i in (1,2,3): self.filter.processLineAndAdd('2019-10-27 02:00:00 fail from 192.0.2.15'); # +3 = 3 self.assertLogged( - "Detected a log entry 60m before the current time in operation mode. This looks like a timezone problem.", + "Detected a log entry 1h before the current time in operation mode. This looks like a timezone problem.", "Please check a jail for a timing issue.", "192.0.2.15:1", "192.0.2.15:2", "192.0.2.15:3", "Total # of detected failures: 3.", all=True, wait=True) @@ -467,7 +467,7 @@ class IgnoreIP(LogCaptureTestCase): for i in (1,2,3): self.filter.processLineAndAdd('2019-10-27 04:00:00 GMT fail from 192.0.2.16'); # +3 = 6 self.assertLogged( - "Detected a log entry 120m after the current time in operation mode. This looks like a timezone problem.", + "Detected a log entry 2h after the current time in operation mode. This looks like a timezone problem.", "Please check a jail for a timing issue.", "192.0.2.16:1", "192.0.2.16:2", "192.0.2.16:3", "Total # of detected failures: 6.", all=True, wait=True) @@ -1458,7 +1458,7 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover if idle: self.filter.sleeptime /= 100.0 self.filter.idle = True - self.waitForTicks(1) + self.waitForTicks(1) self.assertRaises(FailManagerEmpty, self.filter.failManager.toBan) # Now let's feed it with entries from the file @@ -1540,6 +1540,7 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover def test_delJournalMatch(self): self._initFilter() self.filter.start() + self.waitForTicks(1); # wait for start # Smoke test for removing of match # basic full test @@ -1572,6 +1573,7 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover def test_WrongChar(self): self._initFilter() self.filter.start() + self.waitForTicks(1); # wait for start # Now let's feed it with entries from the file _copy_lines_to_journal( self.test_file, self.journal_fields, skip=15, n=4) diff --git a/fail2ban/tests/misctestcase.py b/fail2ban/tests/misctestcase.py index 458e9a23..915ce7df 100644 --- a/fail2ban/tests/misctestcase.py +++ b/fail2ban/tests/misctestcase.py @@ -457,3 +457,15 @@ class MyTimeTest(unittest.TestCase): self.assertEqual(float(str2sec("1 month")) / 60 / 60 / 24, 30.4375) self.assertEqual(float(str2sec("1 year")) / 60 / 60 / 24, 365.25) + def testSec2Str(self): + sec2str = lambda s: str(MyTime.seconds2str(s)) + self.assertEqual(sec2str(86400*390), '1y 3w 4d') + self.assertEqual(sec2str(86400*368), '1y 3d') + self.assertEqual(sec2str(86400*365.49), '1y') + self.assertEqual(sec2str(86400*15), '2w 1d') + self.assertEqual(sec2str(86400*14-10), '2w') + self.assertEqual(sec2str(86400*2+3600*7+60*15), '2d 7h 15m') + self.assertEqual(sec2str(86400*2+3599), '2d 1h') + self.assertEqual(sec2str(3600-5), '1h') + self.assertEqual(sec2str(3600-10), '59m 50s') + self.assertEqual(sec2str(59), '59s') From 96661f25ab558b92b42d01c87d383664e69dd18b Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 20 Sep 2021 04:07:55 +0200 Subject: [PATCH 170/240] filtersystemd.py: fixes wrong time point of "in operation" mode todo: need more tests to cover any step of switch to inOperationMode (all branches) --- .github/workflows/main.yml | 5 ++- fail2ban/server/filtersystemd.py | 76 ++++++++++++++++++++++++-------- fail2ban/tests/filtertestcase.py | 23 ++++++++++ 3 files changed, 85 insertions(+), 19 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index c60ab05c..e346635d 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -33,7 +33,10 @@ jobs: uses: actions/setup-python@v2 with: python-version: ${{ matrix.python-version }} - + + - name: Grant systemd-journal access + run: sudo usermod -a -G systemd-journal "$USER" || echo 'no systemd-journal access' + - name: Python version run: | F2B_PY=$(python -c "import sys; print(sys.version)") diff --git a/fail2ban/server/filtersystemd.py b/fail2ban/server/filtersystemd.py index 88f8c292..a305b6cf 100644 --- a/fail2ban/server/filtersystemd.py +++ b/fail2ban/server/filtersystemd.py @@ -92,8 +92,8 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover try: args['flags'] = int(kwargs.pop('journalflags')) except KeyError: - # be sure all journal types will be opened if files specified (don't set flags): - if 'files' not in args or not len(args['files']): + # be sure all journal types will be opened if files/path specified (don't set flags): + if ('files' not in args or not len(args['files'])) and ('path' not in args or not args['path']): args['flags'] = 4 try: @@ -258,6 +258,10 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover date = datetime.datetime.fromtimestamp(date) self.__journal.seek_realtime(date) + def inOperationMode(self): + self.inOperation = True + logSys.info("[%s] Jail is in operation now (process new journal entries)", self.jailName) + ## # Main loop. # @@ -268,23 +272,44 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover if not self.getJournalMatch(): logSys.notice( - "Jail started without 'journalmatch' set. " + "[%s] Jail started without 'journalmatch' set. " "Jail regexs will be checked against all journal entries, " - "which is not advised for performance reasons.") + "which is not advised for performance reasons.", self.jailName) - # Try to obtain the last known time (position of journal) - start_time = 0 - if self.jail.database is not None: - start_time = self.jail.database.getJournalPos(self.jail, 'systemd-journal') or 0 - # Seek to max(last_known_time, now - findtime) in journal - start_time = max( start_time, MyTime.time() - int(self.getFindTime()) ) - self.seekToTime(start_time) - # Move back one entry to ensure do not end up in dead space - # if start time beyond end of journal + # Save current cursor position (to recognize in operation mode): + logentry = None try: - self.__journal.get_previous() + self.__journal.seek_tail() + logentry = self.__journal.get_previous() + self.__journal.get_next() except OSError: - pass # Reading failure, so safe to ignore + logentry = None # Reading failure, so safe to ignore + if logentry: + # Try to obtain the last known time (position of journal) + startTime = 0 + if self.jail.database is not None: + startTime = self.jail.database.getJournalPos(self.jail, 'systemd-journal') or 0 + # Seek to max(last_known_time, now - findtime) in journal + startTime = max( startTime, MyTime.time() - int(self.getFindTime()) ) + self.seekToTime(startTime) + # Not in operation while we'll read old messages ... + self.inOperation = False + # Save current time in order to check time to switch "in operation" mode + startTime = (1, MyTime.time(), logentry.get('__CURSOR')) + # Move back one entry to ensure do not end up in dead space + # if start time beyond end of journal + try: + self.__journal.get_previous() + except OSError: + pass # Reading failure, so safe to ignore + else: + # empty journal or no entries for current filter: + self.inOperationMode() + # seek_tail() seems to have a bug by no entries (could bypass some entries hereafter), so seek to now instead: + startTime = MyTime.time() + self.seekToTime(startTime) + # for possible future switches of in-operation mode: + startTime = (0, startTime) line = None while self.active: @@ -317,12 +342,27 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover e, exc_info=logSys.getEffectiveLevel() <= logging.DEBUG) self.ticks += 1 if logentry: - line = self.formatJournalEntry(logentry) - self.processLineAndAdd(*line) + line, tm = self.formatJournalEntry(logentry) + # switch "in operation" mode if we'll find start entry (+ some delta): + if not self.inOperation: + if tm >= MyTime.time() - 1: # reached now (approximated): + self.inOperationMode() + elif startTime[0] == 1: + # if it reached start entry (or get read time larger than start time) + if logentry.get('__CURSOR') == startTime[2] or tm > startTime[1]: + # give the filter same time it needed to reach the start entry: + startTime = (0, MyTime.time()*2 - startTime[1]) + elif tm > startTime[1]: # reached start time (approximated): + self.inOperationMode() + # process line + self.processLineAndAdd(line, tm) self.__modified += 1 if self.__modified >= 100: # todo: should be configurable break else: + # "in operation" mode since we don't have messages anymore (reached end of journal): + if not self.inOperation: + self.inOperationMode() break self.__modified = 0 if self.ticks % 10 == 0: @@ -334,7 +374,7 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover or not self.active ) ): - self.jail.database.updateJournal(self.jail, 'systemd-journal', line[1], line[0][1]) + self.jail.database.updateJournal(self.jail, 'systemd-journal', tm, line[1]) self.__nextUpdateTM = MyTime.time() + Utils.DEFAULT_SLEEP_TIME * 5 line = None except Exception as e: # pragma: no cover diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 343cfe04..32dcca5c 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -1537,6 +1537,29 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover _gen_falure("192.0.2.6") self.assertFalse(self.jail.getFailTicket()) + # now reset DB, so we'd find all messages before filter entering in operation mode: + self.filter.stop() + self.filter.join() + self.jail.database.updateJournal(self.jail, 'systemd-journal', MyTime.time()-10000, 'TEST') + self._initFilter() + self.filter.setMaxRetry(1) + states = [] + def _state(*args): + self.assertNotIn("** in operation", states) + self.assertFalse(self.filter.inOperation) + states.append("** process line: %r" % (args,)) + self.filter.processLineAndAdd = _state + def _inoper(): + self.assertNotIn("** in operation", states) + self.assertEqual(len(states), 11) + states.append("** in operation") + self.filter.__class__.inOperationMode(self.filter) + self.filter.inOperationMode = _inoper + self.filter.start() + self.waitForTicks(12) + self.assertTrue(Utils.wait_for(lambda: len(states) == 12, _maxWaitTime(10))) + self.assertEqual(states[-1], "** in operation") + def test_delJournalMatch(self): self._initFilter() self.filter.start() From 3b020988172c4c38d9baae36344b0d3cfa50332e Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 3 Nov 2021 15:35:59 +0100 Subject: [PATCH 171/240] several backends optimizations (in file and journal filters): - don't need to wait if we still had log-entries from last iteration (which got interrupted for servicing) - rewritten update log/journal position, it is more stable and faster now (fewer DB access and surely up-to-date at end) --- fail2ban/server/filter.py | 53 +++++++++++++++++++++++++++----- fail2ban/server/filtersystemd.py | 45 +++++++++++++++++++-------- fail2ban/server/jailthread.py | 8 +++++ 3 files changed, 86 insertions(+), 20 deletions(-) diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index d2cd0ce9..f8417d2d 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -104,6 +104,10 @@ class Filter(JailThread): ## Error counter (protected, so can be used in filter implementations) ## if it reached 100 (at once), run-cycle will go idle self._errors = 0 + ## Next time to update log or journal position in database: + self._nextUpdateTM = 0 + ## Pending updates (must be executed at next update time or during stop): + self._pendDBUpdates = {} ## return raw host (host is not dns): self.returnRawHost = False ## check each regex (used for test purposes): @@ -1023,9 +1027,6 @@ class FileFilter(Filter): log = self.__logs.pop(path) except KeyError: return - db = self.jail.database - if db is not None: - db.updateLog(self.jail, log) logSys.info("Removed logfile: %r", path) self._delLogPath(path) return @@ -1145,9 +1146,15 @@ class FileFilter(Filter): self.processLineAndAdd(line.rstrip('\r\n')) finally: log.close() - db = self.jail.database - if db is not None: - db.updateLog(self.jail, log) + if self.jail.database is not None: + self._pendDBUpdates[log] = 1 + if ( + self.ticks % 100 == 0 + or MyTime.time() >= self._nextUpdateTM + or not self.active + ): + self._updateDBPending() + self._nextUpdateTM = MyTime.time() + Utils.DEFAULT_SLEEP_TIME * 5 return True ## @@ -1247,12 +1254,33 @@ class FileFilter(Filter): ret.append(("File list", path)) return ret - def stop(self): - """Stop monitoring of log-file(s) + def _updateDBPending(self): + """Apply pending updates (log position) to database. """ + db = self.jail.database + while True: + try: + log, args = self._pendDBUpdates.popitem() + except KeyError: + break + db.updateLog(self.jail, log) + + def onStop(self): + """Stop monitoring of log-file(s). Invoked after run method. + """ + # ensure positions of pending logs are up-to-date: + if self._pendDBUpdates and self.jail.database: + self._updateDBPending() # stop files monitoring: for path in self.__logs.keys(): self.delLogPath(path) + + def stop(self): + """Stop filter + """ + # normally onStop will be called automatically in thread after its run ends, + # but for backwards compatibilities we'll invoke it in caller of stop method. + self.onStop() # stop thread: super(Filter, self).stop() @@ -1304,6 +1332,15 @@ class FileContainer: ## shows that log is in operation mode (expecting new messages only from here): self.inOperation = tail + def __hash__(self): + return hash(self.__filename) + def __eq__(self, other): + return (id(self) == id(other) or + self.__filename == (other.__filename if isinstance(other, FileContainer) else other) + ) + def __repr__(self): + return 'file-log:'+self.__filename + def getFileName(self): return self.__filename diff --git a/fail2ban/server/filtersystemd.py b/fail2ban/server/filtersystemd.py index a305b6cf..e2220105 100644 --- a/fail2ban/server/filtersystemd.py +++ b/fail2ban/server/filtersystemd.py @@ -61,7 +61,6 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover # Initialise systemd-journal connection self.__journal = journal.Reader(**jrnlargs) self.__matches = [] - self.__nextUpdateTM = 0 self.setDatePattern(None) logSys.debug("Created FilterSystemd") @@ -321,9 +320,10 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover #self.__journal.wait(self.sleeptime) != journal.NOP ## ## wait for entries without sleep in intervals, because "sleeping" in journal.wait: - Utils.wait_for(lambda: not self.active or \ - self.__journal.wait(Utils.DEFAULT_SLEEP_INTERVAL) != journal.NOP, - self.sleeptime, 0.00001) + if not logentry: + Utils.wait_for(lambda: not self.active or \ + self.__journal.wait(Utils.DEFAULT_SLEEP_INTERVAL) != journal.NOP, + self.sleeptime, 0.00001) if self.idle: # because journal.wait will returns immediatelly if we have records in journal, # just wait a little bit here for not idle, to prevent hi-load: @@ -368,15 +368,17 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover if self.ticks % 10 == 0: self.performSvc() # update position in log (time and iso string): - if (line and self.jail.database and ( - self.ticks % 10 == 0 - or MyTime.time() >= self.__nextUpdateTM + if self.jail.database: + if line: + self._pendDBUpdates['systemd-journal'] = (tm, line[1]) + line = None + if self._pendDBUpdates and ( + self.ticks % 100 == 0 + or MyTime.time() >= self._nextUpdateTM or not self.active - ) - ): - self.jail.database.updateJournal(self.jail, 'systemd-journal', tm, line[1]) - self.__nextUpdateTM = MyTime.time() + Utils.DEFAULT_SLEEP_TIME * 5 - line = None + ): + self._updateDBPending() + self._nextUpdateTM = MyTime.time() + Utils.DEFAULT_SLEEP_TIME * 5 except Exception as e: # pragma: no cover if not self.active: # if not active - error by stop... break @@ -403,3 +405,22 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover ret.append(("Journal matches", [" + ".join(" ".join(match) for match in self.__matches)])) return ret + + def _updateDBPending(self): + """Apply pending updates (jornal position) to database. + """ + db = self.jail.database + while True: + try: + log, args = self._pendDBUpdates.popitem() + except KeyError: + break + db.updateJournal(self.jail, log, *args) + + def onStop(self): + """Stop monitoring of journal. Invoked after run method. + """ + # ensure positions of pending logs are up-to-date: + if self._pendDBUpdates and self.jail.database: + self._updateDBPending() + diff --git a/fail2ban/server/jailthread.py b/fail2ban/server/jailthread.py index 94f34542..67955a06 100644 --- a/fail2ban/server/jailthread.py +++ b/fail2ban/server/jailthread.py @@ -67,6 +67,8 @@ class JailThread(Thread): def run_with_except_hook(*args, **kwargs): try: run(*args, **kwargs) + # call on stop callback to do some finalizations: + self.onStop() except Exception as e: # avoid very sporadic error "'NoneType' object has no attribute 'exc_info'" (https://bugs.python.org/issue7336) # only extremely fast systems are affected ATM (2.7 / 3.x), if thread ends nothing is available here. @@ -97,6 +99,12 @@ class JailThread(Thread): self.active = True super(JailThread, self).start() + @abstractmethod + def onStop(self): # pragma: no cover - absract + """Abstract - Called when thread ends (after run). + """ + pass + def stop(self): """Sets `active` property to False, to flag run method to return. """ From 4fe4ac8dde6ba14841da598ec37f8c6911fe0f64 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 3 Nov 2021 15:58:57 +0100 Subject: [PATCH 172/240] amend to merge: replace timedelta string representation with new function seconds2str --- fail2ban/server/observer.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/fail2ban/server/observer.py b/fail2ban/server/observer.py index f5ba20d9..b585706f 100644 --- a/fail2ban/server/observer.py +++ b/fail2ban/server/observer.py @@ -462,7 +462,7 @@ class ObserverThread(JailThread): if ticket.getTime() > timeOfBan: logSys.info('[%s] IP %s is bad: %s # last %s - incr %s to %s' % (jail.name, ip, banCount, MyTime.time2str(timeOfBan), - datetime.timedelta(seconds=int(orgBanTime)), datetime.timedelta(seconds=int(banTime)))); + MyTime.seconds2str(orgBanTime), MyTime.seconds2str(banTime))) else: ticket.restored = True break @@ -491,8 +491,7 @@ class ObserverThread(JailThread): # if not permanent if btime != -1: bendtime = ticket.getTime() + btime - logtime = (datetime.timedelta(seconds=int(btime)), - MyTime.time2str(bendtime)) + logtime = (MyTime.seconds2str(btime), MyTime.time2str(bendtime)) # check ban is not too old : if bendtime < MyTime.time(): logSys.debug('Ignore old bantime %s', logtime[1]) From a57643404c98fee74398cb175e7de1d3883a6f99 Mon Sep 17 00:00:00 2001 From: sebres Date: Thu, 4 Nov 2021 14:34:04 +0100 Subject: [PATCH 173/240] mytime.seconds2str: small amend with speed-up, code simplification and few tests --- fail2ban/server/mytime.py | 69 ++++++++++++++++++++-------------- fail2ban/tests/misctestcase.py | 3 ++ 2 files changed, 43 insertions(+), 29 deletions(-) diff --git a/fail2ban/server/mytime.py b/fail2ban/server/mytime.py index d09c9d41..b6770721 100644 --- a/fail2ban/server/mytime.py +++ b/fail2ban/server/mytime.py @@ -178,34 +178,45 @@ class MyTime: def __str__(self): # s = str(datetime.timedelta(seconds=int(self.sec))) # return s if s[-3:] != ":00" else s[:-3] - s = self.sec; r = ""; c = 3 - # automatic accuracy: round by large values (upto 1 minute, or 1 day by year): - if s >= 3570: - if s >= 31536000: - s = int(round(float(s)/86400)*86400) - elif s >= 86400: - s = int(round(float(s)/60)*60) - else: - s = int(round(float(s)/10)*10) - for n, m in ( - ('y', 31536000), # a year as 365*24*60*60 (don't need to consider leap year by this accuracy) - ('w', 604800), # a week as 24*60*60*7 - ('d', 86400), # a day as 24*60*60 - ('h', 3600), # a hour as 60*60 - ('m', 60), # a minute - ('s', 1) # a second - ): - if s >= m: - c -= 1 - r += ' ' + str(s//m) + n - s %= m - # stop if no remaining part or no repeat needed (too small granularity): - if not s or not c: break - elif c < 3: - # avoid too small granularity: - c -= 1 - if not c: break - # - return r[1:] + s = self.sec; c = 3 + # automatic accuracy: round by large values (and maximally 3 groups) + if s >= 31536000: # a year as 365*24*60*60 (don't need to consider leap year by this accuracy) + s = int(round(float(s)/86400)) # round by a day + r = str(s//365) + 'y '; s %= 365 + if s >= 7: + r += str(s//7) + 'w '; s %= 7 + if s: + r += str(s) + 'd ' + return r[:-1] + if s >= 604800: # a week as 24*60*60*7 + s = int(round(float(s)/3600)) # round by a hour + r = str(s//168) + 'w '; s %= 168 + if s >= 24: + r += str(s//24) + 'd '; s %= 24 + if s: + r += str(s) + 'h ' + return r[:-1] + if s >= 86400: # a day as 24*60*60 + s = int(round(float(s)/60)) # round by a minute + r = str(s//1440) + 'd '; s %= 1440 + if s >= 60: + r += str(s//60) + 'h '; s %= 60 + if s: + r += str(s) + 'm ' + return r[:-1] + if s >= 3595: # a hour as 60*60 (- 5 seconds) + s = int(round(float(s)/10)) # round by 10 seconds + r = str(s//360) + 'h '; s %= 360 + if s >= 6: # a minute + r += str(s//6) + 'm '; s %= 6 + return r[:-1] + r = '' + if s >= 60: # a minute + r += str(s//60) + 'm '; s %= 60 + if s: # remaining seconds + r += str(s) + 's ' + elif not self.sec: # 0s + r = '0 ' + return r[:-1] def __repr__(self): return self.__str__() diff --git a/fail2ban/tests/misctestcase.py b/fail2ban/tests/misctestcase.py index 915ce7df..97669be0 100644 --- a/fail2ban/tests/misctestcase.py +++ b/fail2ban/tests/misctestcase.py @@ -466,6 +466,9 @@ class MyTimeTest(unittest.TestCase): self.assertEqual(sec2str(86400*14-10), '2w') self.assertEqual(sec2str(86400*2+3600*7+60*15), '2d 7h 15m') self.assertEqual(sec2str(86400*2+3599), '2d 1h') + self.assertEqual(sec2str(3600*3.52), '3h 31m') + self.assertEqual(sec2str(3600*2-5), '2h') self.assertEqual(sec2str(3600-5), '1h') self.assertEqual(sec2str(3600-10), '59m 50s') self.assertEqual(sec2str(59), '59s') + self.assertEqual(sec2str(0), '0') From 1bcb62e31ca0372abc4fefb81724f30fb6ff8837 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 3 Nov 2021 19:15:59 +0100 Subject: [PATCH 174/240] gh-actions: python releases upgrade + debug/test systemd backend availability in GHA --- .github/workflows/main.yml | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index e346635d..c0dc726b 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -22,7 +22,7 @@ jobs: runs-on: ubuntu-20.04 strategy: matrix: - python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, '3.10.0-rc.2', pypy2, pypy3] + python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, '3.10', '3.11.0-alpha.1', pypy2, pypy3] fail-fast: false # Steps represent a sequence of tasks that will be executed as part of the job steps: @@ -40,33 +40,44 @@ jobs: - name: Python version run: | F2B_PY=$(python -c "import sys; print(sys.version)") - echo "Python: ${{ matrix.python-version }} -- $F2B_PY" + echo "Python: ${{ matrix.python-version }} -- ${F2B_PY/$'\n'/ }" F2B_PY=${F2B_PY:0:1} echo "Set F2B_PY=$F2B_PY" echo "F2B_PY=$F2B_PY" >> $GITHUB_ENV - name: Install dependencies run: | - #python -m pip install --upgrade pip + if [[ "$F2B_PY" = 3 ]]; then python -m pip install --upgrade pip || echo "can't upgrade pip"; fi if [[ "$F2B_PY" = 3 ]] && ! command -v 2to3x -v 2to3 > /dev/null; then #pip install 2to3 sudo apt-get -y install 2to3 fi - #pip install systemd-python || echo 'systemd not available' - #pip install pyinotify || echo 'inotify not available' - sudo apt-get -y install python${F2B_PY/2/}-systemd || echo 'systemd not available' - sudo apt-get -y install python${F2B_PY/2/}-pyinotify || echo 'inotify not available' + #sudo apt-get -y install python${F2B_PY/2/}-pyinotify || echo 'inotify not available' + python -m pip install pyinotify || echo 'inotify not available' + #sudo apt-get -y install python${F2B_PY/2/}-systemd || echo 'systemd not available' + sudo apt-get -y install libsystemd-dev || echo 'systemd dependencies seems to be unavailable' + python -m pip install systemd-python || echo 'systemd not available' - name: Before scripts run: | cd "$GITHUB_WORKSPACE" # Manually execute 2to3 for now if [[ "$F2B_PY" = 3 ]]; then echo "2to3 ..." && ./fail2ban-2to3; fi + _debug() { echo -n "$1 "; err=$("${@:2}" 2>&1) && echo 'OK' || echo -e "FAIL\n$err"; } # (debug) output current preferred encoding: - python -c 'import locale, sys; from fail2ban.helpers import PREFER_ENC; print(PREFER_ENC, locale.getpreferredencoding(), (sys.stdout and sys.stdout.encoding))' + _debug 'Encodings:' python -c 'import locale, sys; from fail2ban.helpers import PREFER_ENC; print(PREFER_ENC, locale.getpreferredencoding(), (sys.stdout and sys.stdout.encoding))' + # (debug) backend availabilities: + echo 'Backends:' + _debug '- systemd:' python -c 'from fail2ban.server.filtersystemd import FilterSystemd' + #_debug '- systemd (root): ' sudo python -c 'from fail2ban.server.filtersystemd import FilterSystemd' + _debug '- pyinotify:' python -c 'from fail2ban.server.filterpyinotify import FilterPyinotify' - - name: Test suite - run: if [[ "$F2B_PY" = 2 ]]; then python setup.py test; else python bin/fail2ban-testcases --verbosity=2; fi + # - name: Test suite + # run: if [[ "$F2B_PY" = 2 ]]; then python setup.py test; else python bin/fail2ban-testcases --verbosity=2; fi + + - name: Test suite (debug some systemd tests only) + # run: python bin/fail2ban-testcases --verbosity=2 "[sS]ystemd|[jJ]ournal" + run: python bin/fail2ban-testcases --verbosity=2 -l 5 "test_WrongChar" - name: Build run: python setup.py build From a147a8b0e1b2f32b6f191932afd3c2db9765e2e3 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 1 Dec 2021 20:04:46 +0100 Subject: [PATCH 175/240] gh-actions: coverage for systemd backend (to monitor journals in test-suite in GHA-env we need to use 0 as default flags, because otherwise it cannot be found using SYSTEM_ONLY(4)) --- .github/workflows/main.yml | 14 ++++++++------ fail2ban/server/filtersystemd.py | 3 ++- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index c0dc726b..788b17e3 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -44,6 +44,8 @@ jobs: F2B_PY=${F2B_PY:0:1} echo "Set F2B_PY=$F2B_PY" echo "F2B_PY=$F2B_PY" >> $GITHUB_ENV + # for GHA we need to monitor all journals, since it cannot be found using SYSTEM_ONLY(4): + echo "F2B_SYSTEMD_DEFAULT_FLAGS=0" >> $GITHUB_ENV - name: Install dependencies run: | @@ -71,13 +73,13 @@ jobs: _debug '- systemd:' python -c 'from fail2ban.server.filtersystemd import FilterSystemd' #_debug '- systemd (root): ' sudo python -c 'from fail2ban.server.filtersystemd import FilterSystemd' _debug '- pyinotify:' python -c 'from fail2ban.server.filterpyinotify import FilterPyinotify' - - # - name: Test suite - # run: if [[ "$F2B_PY" = 2 ]]; then python setup.py test; else python bin/fail2ban-testcases --verbosity=2; fi - - name: Test suite (debug some systemd tests only) - # run: python bin/fail2ban-testcases --verbosity=2 "[sS]ystemd|[jJ]ournal" - run: python bin/fail2ban-testcases --verbosity=2 -l 5 "test_WrongChar" + - name: Test suite + run: if [[ "$F2B_PY" = 2 ]]; then python setup.py test; else python bin/fail2ban-testcases --verbosity=2; fi + + #- name: Test suite (debug some systemd tests only) + #run: python bin/fail2ban-testcases --verbosity=2 "[sS]ystemd|[jJ]ournal" + #run: python bin/fail2ban-testcases --verbosity=2 -l 5 "test_WrongChar" - name: Build run: python setup.py build diff --git a/fail2ban/server/filtersystemd.py b/fail2ban/server/filtersystemd.py index e2220105..6301b93a 100644 --- a/fail2ban/server/filtersystemd.py +++ b/fail2ban/server/filtersystemd.py @@ -23,6 +23,7 @@ __copyright__ = "Copyright (c) 2013 Steven Hiscocks" __license__ = "GPL" import datetime +import os import time from distutils.version import LooseVersion @@ -93,7 +94,7 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover except KeyError: # be sure all journal types will be opened if files/path specified (don't set flags): if ('files' not in args or not len(args['files'])) and ('path' not in args or not args['path']): - args['flags'] = 4 + args['flags'] = int(os.getenv("F2B_SYSTEMD_DEFAULT_FLAGS", 4)) try: args['namespace'] = kwargs.pop('namespace') From 0fa76ef75a7c6102f383c703fdcf6d53e45149b0 Mon Sep 17 00:00:00 2001 From: sebres Date: Thu, 2 Dec 2021 18:27:56 +0100 Subject: [PATCH 176/240] gh-actions: temporary ignore tests of systemd backend for python >= v.3.10 (otherwise it fails with "PY_SSIZE_T_CLEAN macro must be defined for '#' formats") --- .github/workflows/main.yml | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 788b17e3..ff31db19 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -41,9 +41,11 @@ jobs: run: | F2B_PY=$(python -c "import sys; print(sys.version)") echo "Python: ${{ matrix.python-version }} -- ${F2B_PY/$'\n'/ }" + F2B_PYV=$(echo "${F2B_PY}" | grep -oP '^\d+(?:\.\d+)') F2B_PY=${F2B_PY:0:1} - echo "Set F2B_PY=$F2B_PY" + echo "Set F2B_PY=$F2B_PY, F2B_PYV=$F2B_PYV" echo "F2B_PY=$F2B_PY" >> $GITHUB_ENV + echo "F2B_PYV=$F2B_PYV" >> $GITHUB_ENV # for GHA we need to monitor all journals, since it cannot be found using SYSTEM_ONLY(4): echo "F2B_SYSTEMD_DEFAULT_FLAGS=0" >> $GITHUB_ENV @@ -75,7 +77,15 @@ jobs: _debug '- pyinotify:' python -c 'from fail2ban.server.filterpyinotify import FilterPyinotify' - name: Test suite - run: if [[ "$F2B_PY" = 2 ]]; then python setup.py test; else python bin/fail2ban-testcases --verbosity=2; fi + run: | + if [[ "$F2B_PY" = 2 ]]; then + python setup.py test + elif dpkg --compare-versions "$F2B_PYV" lt 3.10; then + python bin/fail2ban-testcases --verbosity=2 + else + echo "Skip systemd backend since systemd-python module must be fixed for python >= v.3.10 in GHA ..." + python bin/fail2ban-testcases --verbosity=2 -i "[sS]ystemd|[jJ]ournal" + fi #- name: Test suite (debug some systemd tests only) #run: python bin/fail2ban-testcases --verbosity=2 "[sS]ystemd|[jJ]ournal" From 196c55e93103d1e56d1336f27a5f2591b4e54fef Mon Sep 17 00:00:00 2001 From: sebres Date: Sun, 19 Sep 2021 18:49:18 +0200 Subject: [PATCH 177/240] fix gh-3098: build fails with error in fail2ban setup command: use_2to3 is invalid (setuptools 58+) --- setup.py | 16 +--------------- 1 file changed, 1 insertion(+), 15 deletions(-) diff --git a/setup.py b/setup.py index f4c2550f..98413273 100755 --- a/setup.py +++ b/setup.py @@ -48,7 +48,7 @@ import warnings from glob import glob from fail2ban.setup import updatePyExec - +from fail2ban.version import version source_dir = os.path.realpath(os.path.dirname( # __file__ seems to be overwritten sometimes on some python versions (e.g. bug of 2.6 by running under cProfile, etc.): @@ -112,22 +112,12 @@ class install_scripts_f2b(install_scripts): # Wrapper to specify fail2ban own options: class install_command_f2b(install): user_options = install.user_options + [ - ('disable-2to3', None, 'Specify to deactivate 2to3, e.g. if the install runs from fail2ban test-cases.'), ('without-tests', None, 'without tests files installation'), ] def initialize_options(self): - self.disable_2to3 = None self.without_tests = not with_tests install.initialize_options(self) def finalize_options(self): - global _2to3 - ## in the test cases 2to3 should be already done (fail2ban-2to3): - if self.disable_2to3: - _2to3 = False - if _2to3: - cmdclass = self.distribution.cmdclass - cmdclass['build_py'] = build_py_2to3 - cmdclass['build_scripts'] = build_scripts_2to3 if self.without_tests: self.distribution.scripts.remove('bin/fail2ban-testcases') @@ -178,7 +168,6 @@ commands.''' if setuptools: setup_extra = { 'test_suite': "fail2ban.tests.utils.gatherTests", - 'use_2to3': True, } else: setup_extra = {} @@ -202,9 +191,6 @@ if platform_system in ('linux', 'solaris', 'sunos') or platform_system.startswit ('/usr/share/doc/fail2ban', doc_files) ) -# Get version number, avoiding importing fail2ban. -# This is due to tests not functioning for python3 as 2to3 takes place later -exec(open(join("fail2ban", "version.py")).read()) setup( name = "fail2ban", From 21d94ff178192116eb5288190d78dd4630534a7b Mon Sep 17 00:00:00 2001 From: sebres Date: Sun, 19 Sep 2021 18:52:34 +0200 Subject: [PATCH 178/240] amend to fix gh-3098: no option `--disable-2to3` anymore --- fail2ban/tests/misctestcase.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fail2ban/tests/misctestcase.py b/fail2ban/tests/misctestcase.py index 97669be0..e2faa6fd 100644 --- a/fail2ban/tests/misctestcase.py +++ b/fail2ban/tests/misctestcase.py @@ -111,7 +111,7 @@ class SetupTest(unittest.TestCase): supdbgout = ' >/dev/null 2>&1' if unittest.F2B.log_level >= logging.DEBUG else '' # HEAVYDEBUG try: # try dry-run: - os.system("%s %s --dry-run install --disable-2to3 --root=%s%s" + os.system("%s %s --dry-run install --root=%s%s" % (sys.executable, self.setup , tmp, supdbgout)) # check nothing was created: self.assertTrue(not os.listdir(tmp)) @@ -127,7 +127,7 @@ class SetupTest(unittest.TestCase): # suppress stdout (and stderr) if not heavydebug supdbgout = ' >/dev/null' if unittest.F2B.log_level >= logging.DEBUG else '' # HEAVYDEBUG try: - self.assertEqual(os.system("%s %s install --disable-2to3 --root=%s%s" + self.assertEqual(os.system("%s %s install --root=%s%s" % (sys.executable, self.setup, tmp, supdbgout)), 0) def strippath(l): From 8f83242c25a957d34478e9604600066695be7998 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Mon, 20 Dec 2021 15:39:57 +0100 Subject: [PATCH 179/240] suppress unneeded info (moved to debug level) see #3186 --- fail2ban/server/server.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fail2ban/server/server.py b/fail2ban/server/server.py index bd2c7ad3..627ffe8d 100644 --- a/fail2ban/server/server.py +++ b/fail2ban/server/server.py @@ -372,7 +372,7 @@ class Server: if isinstance(filter_, FileFilter): return filter_.getLogPaths() else: # pragma: systemd no cover - logSys.info("Jail %s is not a FileFilter instance" % name) + logSys.debug("Jail %s is not a FileFilter instance" % name) return [] def addJournalMatch(self, name, match): # pragma: systemd no cover @@ -390,7 +390,7 @@ class Server: if isinstance(filter_, JournalFilter): return filter_.getJournalMatch() else: - logSys.info("Jail %s is not a JournalFilter instance" % name) + logSys.debug("Jail %s is not a JournalFilter instance" % name) return [] def setLogEncoding(self, name, encoding): From 8bf15db688ac0f95f6f1489947ee00825340b23e Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 18 Jan 2022 15:06:16 +0100 Subject: [PATCH 180/240] filter.d/sshd.conf: `ddos` mode extended - recognizes new message "banner exchange: invalid format" generated by port scanner, https payload on ssh port; closes gh-3169 --- config/filter.d/sshd.conf | 1 + fail2ban/tests/files/logs/sshd | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/config/filter.d/sshd.conf b/config/filter.d/sshd.conf index e7942262..c265b9eb 100644 --- a/config/filter.d/sshd.conf +++ b/config/filter.d/sshd.conf @@ -75,6 +75,7 @@ mdre-ddos = ^Did not receive identification string from ^Bad protocol version identification '.*' from ^SSH: Server;Ltype: (?:Authname|Version|Kex);Remote: -\d+;[A-Z]\w+: ^Read from socket failed: Connection reset by peer + ^banner exchange: Connection from <__on_port_opt>: invalid format # same as mdre-normal-other, but as failure (without ) and [preauth] only: mdre-ddos-other = ^(Connection (?:closed|reset)|Disconnected) (?:by|from)%(__authng_user)s %(__on_port_opt)s\s+\[preauth\]\s*$ diff --git a/fail2ban/tests/files/logs/sshd b/fail2ban/tests/files/logs/sshd index 5d23f96f..ac8524a8 100644 --- a/fail2ban/tests/files/logs/sshd +++ b/fail2ban/tests/files/logs/sshd @@ -315,6 +315,11 @@ Feb 17 17:40:17 sshd[19725]: error: kex_exchange_identification: client sent inv # failJSON: { "time": "2005-02-17T17:40:18", "match": true , "host": "192.0.2.10", "desc": "ddos: flood attack vector, gh-2850" } Feb 17 17:40:18 sshd[19725]: error: kex_exchange_identification: Connection closed by remote host +# failJSON: { "match": false } +Mar 1 18:59:33 hostname sshd[1189575]: error: kex_exchange_identification: banner line too long +# failJSON: { "time": "2005-03-01T18:59:33", "match": true , "host": "192.0.2.12", "desc": "ddos: port scanner, https payload on ssh port (banner exchange: invalid format, gh-3169)" } +Mar 1 18:59:33 hostname sshd[1189575]: banner exchange: Connection from 192.0.2.12 port 44105: invalid format + # failJSON: { "time": "2005-03-15T09:21:01", "match": true , "host": "192.0.2.212", "desc": "DDOS mode causes failure on close within preauth stage" } Mar 15 09:21:01 host sshd[2717]: Connection closed by 192.0.2.212 [preauth] # failJSON: { "time": "2005-03-15T09:21:02", "match": true , "host": "192.0.2.212", "desc": "DDOS mode causes failure on close within preauth stage" } From bf689c27b833d4cafc6ce34ada214cd4df2d7d86 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 18 Jan 2022 15:30:27 +0100 Subject: [PATCH 181/240] filter.d/sshd.conf: `ddos` mode extended - recognizes messages "kex_exchange_identification: Connection closed / reset by pear" (fixed possible regression of f77398c49d4eeb529a1684a27dcfbf5b6aaafa66); closes gh-3086 --- config/filter.d/sshd.conf | 7 ++++--- fail2ban/tests/files/logs/sshd | 12 ++++++++++++ 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/config/filter.d/sshd.conf b/config/filter.d/sshd.conf index c265b9eb..d5d189b0 100644 --- a/config/filter.d/sshd.conf +++ b/config/filter.d/sshd.conf @@ -68,16 +68,17 @@ cmnfailed = > mdre-normal = # used to differentiate "connection closed" with and without `[preauth]` (fail/nofail cases in ddos mode) -mdre-normal-other = ^(Connection closed|Disconnected) (?:by|from)%(__authng_user)s (?:%(__suff)s|\s*)$ +mdre-normal-other = ^(Connection (?:closed|reset)|Disconnected) (?:by|from)%(__authng_user)s (?:%(__suff)s|\s*)$ mdre-ddos = ^Did not receive identification string from - ^kex_exchange_identification: (?:[Cc]lient sent invalid protocol identifier|[Cc]onnection closed by remote host) + ^kex_exchange_identification: (?:read: )?(?:[Cc]lient sent invalid protocol identifier|[Cc]onnection (?:closed by remote host|reset by peer)) ^Bad protocol version identification '.*' from ^SSH: Server;Ltype: (?:Authname|Version|Kex);Remote: -\d+;[A-Z]\w+: ^Read from socket failed: Connection reset by peer ^banner exchange: Connection from <__on_port_opt>: invalid format -# same as mdre-normal-other, but as failure (without ) and [preauth] only: +# same as mdre-normal-other, but as failure (without with [preauth] and with on no preauth phase as helper to identify address): mdre-ddos-other = ^(Connection (?:closed|reset)|Disconnected) (?:by|from)%(__authng_user)s %(__on_port_opt)s\s+\[preauth\]\s*$ + ^(Connection (?:closed|reset)|Disconnected) (?:by|from)%(__authng_user)s (?:%(__on_port_opt)s|\s*)$ mdre-extra = ^Received disconnect from %(__on_port_opt)s:\s*14: No(?: supported)? authentication methods available ^Unable to negotiate with %(__on_port_opt)s: no matching <__alg_match> found. diff --git a/fail2ban/tests/files/logs/sshd b/fail2ban/tests/files/logs/sshd index ac8524a8..99c3756b 100644 --- a/fail2ban/tests/files/logs/sshd +++ b/fail2ban/tests/files/logs/sshd @@ -333,6 +333,18 @@ Jun 6 04:17:04 host sshd[1189074]: Invalid user from 192.0.2.68 port 34916 # failJSON: { "time": "2005-06-06T04:17:09", "match": true , "host": "192.0.2.68", "dns": null, "user": "", "desc": "empty user, gh-2749" } Jun 6 04:17:09 host sshd[1189074]: Connection closed by invalid user 192.0.2.68 port 34916 [preauth] +# failJSON: { "match": false, "desc": "ddos-failure without IP, retarded, must be triggered with next (closed) message, gh-3086"} +Jun 7 04:10:35 host sshd[424228]: error: kex_exchange_identification: Connection closed by remote host +# failJSON: { "time": "2005-06-07T04:10:35", "match": true , "host": "192.0.2.15", "desc": "kex_exchange_identification: Connection closed, gh-3086" } +Jun 7 04:10:35 host sshd[424228]: Connection closed by 192.0.2.15 port 35352 + +# failJSON: { "match": false } +Jun 7 04:29:10 host sshd[649921]: Connection from 192.0.2.16 port 51280 on 192.0.2.16 port 22 rdomain "" +# failJSON: { "time": "2005-06-07T04:29:10", "match": true, "host": "192.0.2.16", "desc": "ddos-failure without IP, must be triggered here because it became known above, gh-3086"} +Jun 7 04:29:10 host sshd[649921]: error: kex_exchange_identification: read: Connection reset by peer +# failJSON: { "match": false, "desc": "Connection reset already triggered above (known IP, no-fail helper unused here)" } +Jun 7 04:29:10 host sshd[649921]: Connection reset by 192.0.2.16 port 51280 + # filterOptions: [{"mode": "extra"}, {"mode": "aggressive"}] # several other cases from gh-864: From 3d7e3bc2fb08a62a5e5f1b9826452d1748b47dba Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 24 Jan 2022 22:56:16 +0100 Subject: [PATCH 182/240] make ipset actions more breakdown-safe: start wouldn't fail if set with this name already exists (e. g. created by previous instance and don't deleted properly) --- config/action.d/firewallcmd-ipset.conf | 6 ++-- config/action.d/iptables-ipset.conf | 6 ++-- fail2ban/tests/servertestcase.py | 48 +++++++++++++------------- 3 files changed, 30 insertions(+), 30 deletions(-) diff --git a/config/action.d/firewallcmd-ipset.conf b/config/action.d/firewallcmd-ipset.conf index eaebdb3a..c36ba694 100644 --- a/config/action.d/firewallcmd-ipset.conf +++ b/config/action.d/firewallcmd-ipset.conf @@ -35,15 +35,15 @@ actionunban = /actionunban> [ipstype_ipset] -actionstart = ipset create hash:ip timeout +actionstart = ipset -exist create hash:ip timeout actionflush = ipset flush actionstop = ipset destroy -actionban = ipset add timeout -exist +actionban = ipset -exist add timeout -actionunban = ipset del -exist +actionunban = ipset -exist del [ipstype_firewalld] diff --git a/config/action.d/iptables-ipset.conf b/config/action.d/iptables-ipset.conf index 82d2b6b7..481fe753 100644 --- a/config/action.d/iptables-ipset.conf +++ b/config/action.d/iptables-ipset.conf @@ -24,7 +24,7 @@ before = iptables.conf # Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false). # Values: CMD # -actionstart = ipset create hash:ip timeout +actionstart = ipset -exist create hash:ip timeout -I %(_ipt_chain_rule)s # Option: actionflush @@ -47,7 +47,7 @@ actionstop = -D %(_ipt_chain_rule)s # Tags: See jail.conf(5) man page # Values: CMD # -actionban = ipset add timeout -exist +actionban = ipset -exist add timeout # actionprolong = %(actionban)s @@ -57,7 +57,7 @@ actionban = ipset add timeout -exist # Tags: See jail.conf(5) man page # Values: CMD # -actionunban = ipset del -exist +actionunban = ipset -exist del # Several capabilities used internaly: diff --git a/fail2ban/tests/servertestcase.py b/fail2ban/tests/servertestcase.py index 6aadec14..54088e8f 100644 --- a/fail2ban/tests/servertestcase.py +++ b/fail2ban/tests/servertestcase.py @@ -1594,11 +1594,11 @@ class ServerConfigReaderTests(LogCaptureTestCase): ('j-w-iptables-ipset', 'iptables-ipset-proto6[name=%(__name__)s, port="http", protocol="tcp", chain=""]', { 'ip4': (' f2b-j-w-iptables-ipset ',), 'ip6': (' f2b-j-w-iptables-ipset6 ',), 'ip4-start': ( - "`ipset create f2b-j-w-iptables-ipset hash:ip timeout 0 `", + "`ipset -exist create f2b-j-w-iptables-ipset hash:ip timeout 0 `", "`iptables -w -I INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable`", ), 'ip6-start': ( - "`ipset create f2b-j-w-iptables-ipset6 hash:ip timeout 0 family inet6`", + "`ipset -exist create f2b-j-w-iptables-ipset6 hash:ip timeout 0 family inet6`", "`ip6tables -w -I INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", ), 'flush': ( @@ -1620,27 +1620,27 @@ class ServerConfigReaderTests(LogCaptureTestCase): r"""`ip6tables -w -C INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`""", ), 'ip4-ban': ( - r"`ipset add f2b-j-w-iptables-ipset 192.0.2.1 timeout 0 -exist`", + r"`ipset -exist add f2b-j-w-iptables-ipset 192.0.2.1 timeout 0`", ), 'ip4-unban': ( - r"`ipset del f2b-j-w-iptables-ipset 192.0.2.1 -exist`", + r"`ipset -exist del f2b-j-w-iptables-ipset 192.0.2.1`", ), 'ip6-ban': ( - r"`ipset add f2b-j-w-iptables-ipset6 2001:db8:: timeout 0 -exist`", + r"`ipset -exist add f2b-j-w-iptables-ipset6 2001:db8:: timeout 0`", ), 'ip6-unban': ( - r"`ipset del f2b-j-w-iptables-ipset6 2001:db8:: -exist`", + r"`ipset -exist del f2b-j-w-iptables-ipset6 2001:db8::`", ), }), # iptables-ipset-proto6-allports -- ('j-w-iptables-ipset-ap', 'iptables-ipset-proto6-allports[name=%(__name__)s, chain=""]', { 'ip4': (' f2b-j-w-iptables-ipset-ap ',), 'ip6': (' f2b-j-w-iptables-ipset-ap6 ',), 'ip4-start': ( - "`ipset create f2b-j-w-iptables-ipset-ap hash:ip timeout 0 `", + "`ipset -exist create f2b-j-w-iptables-ipset-ap hash:ip timeout 0 `", "`iptables -w -I INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", ), 'ip6-start': ( - "`ipset create f2b-j-w-iptables-ipset-ap6 hash:ip timeout 0 family inet6`", + "`ipset -exist create f2b-j-w-iptables-ipset-ap6 hash:ip timeout 0 family inet6`", "`ip6tables -w -I INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", ), 'flush': ( @@ -1662,16 +1662,16 @@ class ServerConfigReaderTests(LogCaptureTestCase): r"""`ip6tables -w -C INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`""", ), 'ip4-ban': ( - r"`ipset add f2b-j-w-iptables-ipset-ap 192.0.2.1 timeout 0 -exist`", + r"`ipset -exist add f2b-j-w-iptables-ipset-ap 192.0.2.1 timeout 0`", ), 'ip4-unban': ( - r"`ipset del f2b-j-w-iptables-ipset-ap 192.0.2.1 -exist`", + r"`ipset -exist del f2b-j-w-iptables-ipset-ap 192.0.2.1`", ), 'ip6-ban': ( - r"`ipset add f2b-j-w-iptables-ipset-ap6 2001:db8:: timeout 0 -exist`", + r"`ipset -exist add f2b-j-w-iptables-ipset-ap6 2001:db8:: timeout 0`", ), 'ip6-unban': ( - r"`ipset del f2b-j-w-iptables-ipset-ap6 2001:db8:: -exist`", + r"`ipset -exist del f2b-j-w-iptables-ipset-ap6 2001:db8::`", ), }), # iptables (oneport) -- @@ -1949,11 +1949,11 @@ class ServerConfigReaderTests(LogCaptureTestCase): ('j-w-fwcmd-ipset', 'firewallcmd-ipset[name=%(__name__)s, port="http", protocol="tcp", chain=""]', { 'ip4': (' f2b-j-w-fwcmd-ipset ',), 'ip6': (' f2b-j-w-fwcmd-ipset6 ',), 'ip4-start': ( - "`ipset create f2b-j-w-fwcmd-ipset hash:ip timeout 0 `", + "`ipset -exist create f2b-j-w-fwcmd-ipset hash:ip timeout 0 `", "`firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset src -j REJECT --reject-with icmp-port-unreachable`", ), 'ip6-start': ( - "`ipset create f2b-j-w-fwcmd-ipset6 hash:ip timeout 0 family inet6`", + "`ipset -exist create f2b-j-w-fwcmd-ipset6 hash:ip timeout 0 family inet6`", "`firewall-cmd --direct --add-rule ipv6 filter INPUT_direct 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", ), 'flush': ( @@ -1969,27 +1969,27 @@ class ServerConfigReaderTests(LogCaptureTestCase): "`ipset destroy f2b-j-w-fwcmd-ipset6`", ), 'ip4-ban': ( - r"`ipset add f2b-j-w-fwcmd-ipset 192.0.2.1 timeout 0 -exist`", + r"`ipset -exist add f2b-j-w-fwcmd-ipset 192.0.2.1 timeout 0`", ), 'ip4-unban': ( - r"`ipset del f2b-j-w-fwcmd-ipset 192.0.2.1 -exist`", + r"`ipset -exist del f2b-j-w-fwcmd-ipset 192.0.2.1`", ), 'ip6-ban': ( - r"`ipset add f2b-j-w-fwcmd-ipset6 2001:db8:: timeout 0 -exist`", + r"`ipset -exist add f2b-j-w-fwcmd-ipset6 2001:db8:: timeout 0`", ), 'ip6-unban': ( - r"`ipset del f2b-j-w-fwcmd-ipset6 2001:db8:: -exist`", + r"`ipset -exist del f2b-j-w-fwcmd-ipset6 2001:db8::`", ), }), # firewallcmd-ipset (allports) -- ('j-w-fwcmd-ipset-ap', 'firewallcmd-ipset[name=%(__name__)s, actiontype=, protocol="tcp", chain=""]', { 'ip4': (' f2b-j-w-fwcmd-ipset-ap ',), 'ip6': (' f2b-j-w-fwcmd-ipset-ap6 ',), 'ip4-start': ( - "`ipset create f2b-j-w-fwcmd-ipset-ap hash:ip timeout 0 `", + "`ipset -exist create f2b-j-w-fwcmd-ipset-ap hash:ip timeout 0 `", "`firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 0 -p tcp -m set --match-set f2b-j-w-fwcmd-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", ), 'ip6-start': ( - "`ipset create f2b-j-w-fwcmd-ipset-ap6 hash:ip timeout 0 family inet6`", + "`ipset -exist create f2b-j-w-fwcmd-ipset-ap6 hash:ip timeout 0 family inet6`", "`firewall-cmd --direct --add-rule ipv6 filter INPUT_direct 0 -p tcp -m set --match-set f2b-j-w-fwcmd-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", ), 'flush': ( @@ -2005,16 +2005,16 @@ class ServerConfigReaderTests(LogCaptureTestCase): "`ipset destroy f2b-j-w-fwcmd-ipset-ap6`", ), 'ip4-ban': ( - r"`ipset add f2b-j-w-fwcmd-ipset-ap 192.0.2.1 timeout 0 -exist`", + r"`ipset -exist add f2b-j-w-fwcmd-ipset-ap 192.0.2.1 timeout 0`", ), 'ip4-unban': ( - r"`ipset del f2b-j-w-fwcmd-ipset-ap 192.0.2.1 -exist`", + r"`ipset -exist del f2b-j-w-fwcmd-ipset-ap 192.0.2.1`", ), 'ip6-ban': ( - r"`ipset add f2b-j-w-fwcmd-ipset-ap6 2001:db8:: timeout 0 -exist`", + r"`ipset -exist add f2b-j-w-fwcmd-ipset-ap6 2001:db8:: timeout 0`", ), 'ip6-unban': ( - r"`ipset del f2b-j-w-fwcmd-ipset-ap6 2001:db8:: -exist`", + r"`ipset -exist del f2b-j-w-fwcmd-ipset-ap6 2001:db8::`", ), }), # firewallcmd-rich-rules -- From b639c8869cedd44e11196242bbdc3ff33dfb3ad3 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 25 Jan 2022 00:00:40 +0100 Subject: [PATCH 183/240] make several iptables actions more breakdown-safe: start wouldn't fail if chain or rule already exists (e. g. created by previous instance and doesn't get purged properly); ultimately closes gh-980 --- config/action.d/iptables-ipset.conf | 2 +- config/action.d/iptables.conf | 5 +-- fail2ban/tests/servertestcase.py | 68 +++++++++++++++++------------ 3 files changed, 43 insertions(+), 32 deletions(-) diff --git a/config/action.d/iptables-ipset.conf b/config/action.d/iptables-ipset.conf index 481fe753..74dfe6c3 100644 --- a/config/action.d/iptables-ipset.conf +++ b/config/action.d/iptables-ipset.conf @@ -25,7 +25,7 @@ before = iptables.conf # Values: CMD # actionstart = ipset -exist create hash:ip timeout - -I %(_ipt_chain_rule)s + { %(_ipt_check_rule)s >/dev/null 2>&1; } || { -I %(_ipt_chain_rule)s; } # Option: actionflush # Notes.: command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action) diff --git a/config/action.d/iptables.conf b/config/action.d/iptables.conf index 7ca9c962..821a9ef1 100644 --- a/config/action.d/iptables.conf +++ b/config/action.d/iptables.conf @@ -22,9 +22,8 @@ actionflush = -F f2b- # Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false). # Values: CMD # -actionstart = -N f2b- - -A f2b- -j - -I %(_ipt_chain_rule)s +actionstart = { -C f2b- -j >/dev/null 2>&1; } || { -N f2b- || true; -A f2b- -j ; } + { %(_ipt_check_rule)s >/dev/null 2>&1; } || { -I %(_ipt_chain_rule)s; } # Option: actionstop # Notes.: command executed at the stop of jail (or at the end of Fail2Ban) diff --git a/fail2ban/tests/servertestcase.py b/fail2ban/tests/servertestcase.py index 54088e8f..8741893d 100644 --- a/fail2ban/tests/servertestcase.py +++ b/fail2ban/tests/servertestcase.py @@ -1506,14 +1506,16 @@ class ServerConfigReaderTests(LogCaptureTestCase): ('j-w-iptables-mp', 'iptables-multiport[name=%(__name__)s, bantime="10m", port="http,https", protocol="tcp", chain=""]', { 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), 'ip4-start': ( - "`iptables -w -N f2b-j-w-iptables-mp`", - "`iptables -w -A f2b-j-w-iptables-mp -j RETURN`", - "`iptables -w -I INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`", + "`{ iptables -w -C f2b-j-w-iptables-mp -j RETURN >/dev/null 2>&1; } || " + "{ iptables -w -N f2b-j-w-iptables-mp || true; iptables -w -A f2b-j-w-iptables-mp -j RETURN; }`", + "`{ iptables -w -C INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp >/dev/null 2>&1; } || " + "{ iptables -w -I INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp; }`", ), 'ip6-start': ( - "`ip6tables -w -N f2b-j-w-iptables-mp`", - "`ip6tables -w -A f2b-j-w-iptables-mp -j RETURN`", - "`ip6tables -w -I INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`", + "`{ ip6tables -w -C f2b-j-w-iptables-mp -j RETURN >/dev/null 2>&1; } || " + "{ ip6tables -w -N f2b-j-w-iptables-mp || true; ip6tables -w -A f2b-j-w-iptables-mp -j RETURN; }`", + "`{ ip6tables -w -C INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp >/dev/null 2>&1; } || ", + "{ ip6tables -w -I INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp; }`", ), 'flush': ( "`iptables -w -F f2b-j-w-iptables-mp`", @@ -1550,14 +1552,16 @@ class ServerConfigReaderTests(LogCaptureTestCase): ('j-w-iptables-ap', 'iptables-allports[name=%(__name__)s, bantime="10m", protocol="tcp", chain=""]', { 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), 'ip4-start': ( - "`iptables -w -N f2b-j-w-iptables-ap`", - "`iptables -w -A f2b-j-w-iptables-ap -j RETURN`", - "`iptables -w -I INPUT -p tcp -j f2b-j-w-iptables-ap`", + "`{ iptables -w -C f2b-j-w-iptables-ap -j RETURN >/dev/null 2>&1; } || " + "{ iptables -w -N f2b-j-w-iptables-ap || true; iptables -w -A f2b-j-w-iptables-ap -j RETURN; }`", + "`{ iptables -w -C INPUT -p tcp -j f2b-j-w-iptables-ap >/dev/null 2>&1; } || ", + "{ iptables -w -I INPUT -p tcp -j f2b-j-w-iptables-ap; }`", ), 'ip6-start': ( - "`ip6tables -w -N f2b-j-w-iptables-ap`", - "`ip6tables -w -A f2b-j-w-iptables-ap -j RETURN`", - "`ip6tables -w -I INPUT -p tcp -j f2b-j-w-iptables-ap`", + "`{ ip6tables -w -C f2b-j-w-iptables-ap -j RETURN >/dev/null 2>&1; } || " + "{ ip6tables -w -N f2b-j-w-iptables-ap || true; ip6tables -w -A f2b-j-w-iptables-ap -j RETURN; }`", + "`{ ip6tables -w -C INPUT -p tcp -j f2b-j-w-iptables-ap >/dev/null 2>&1; } || ", + "{ ip6tables -w -I INPUT -p tcp -j f2b-j-w-iptables-ap; }`", ), 'flush': ( "`iptables -w -F f2b-j-w-iptables-ap`", @@ -1595,11 +1599,13 @@ class ServerConfigReaderTests(LogCaptureTestCase): 'ip4': (' f2b-j-w-iptables-ipset ',), 'ip6': (' f2b-j-w-iptables-ipset6 ',), 'ip4-start': ( "`ipset -exist create f2b-j-w-iptables-ipset hash:ip timeout 0 `", - "`iptables -w -I INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable`", + "`{ iptables -w -C INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable >/dev/null 2>&1; } || " + "{ iptables -w -I INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable; }`", ), 'ip6-start': ( "`ipset -exist create f2b-j-w-iptables-ipset6 hash:ip timeout 0 family inet6`", - "`ip6tables -w -I INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", + "`{ ip6tables -w -C INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable >/dev/null 2>&1; } || " + "{ ip6tables -w -I INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable; }`", ), 'flush': ( "`ipset flush f2b-j-w-iptables-ipset`", @@ -1637,11 +1643,13 @@ class ServerConfigReaderTests(LogCaptureTestCase): 'ip4': (' f2b-j-w-iptables-ipset-ap ',), 'ip6': (' f2b-j-w-iptables-ipset-ap6 ',), 'ip4-start': ( "`ipset -exist create f2b-j-w-iptables-ipset-ap hash:ip timeout 0 `", - "`iptables -w -I INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", + "`{ iptables -w -C INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable >/dev/null 2>&1; } || " + "{ iptables -w -I INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable; }", ), 'ip6-start': ( "`ipset -exist create f2b-j-w-iptables-ipset-ap6 hash:ip timeout 0 family inet6`", - "`ip6tables -w -I INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", + "`{ ip6tables -w -C INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable >/dev/null 2>&1; } || " + "{ ip6tables -w -I INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable; }", ), 'flush': ( "`ipset flush f2b-j-w-iptables-ipset-ap`", @@ -1678,14 +1686,16 @@ class ServerConfigReaderTests(LogCaptureTestCase): ('j-w-iptables', 'iptables[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain=""]', { 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), 'ip4-start': ( - "`iptables -w -N f2b-j-w-iptables`", - "`iptables -w -A f2b-j-w-iptables -j RETURN`", - "`iptables -w -I INPUT -p tcp --dport http -j f2b-j-w-iptables`", + "`{ iptables -w -C f2b-j-w-iptables -j RETURN >/dev/null 2>&1; } || " + "{ iptables -w -N f2b-j-w-iptables || true; iptables -w -A f2b-j-w-iptables -j RETURN; }", + "`{ iptables -w -C INPUT -p tcp --dport http -j f2b-j-w-iptables >/dev/null 2>&1; } || " + "{ iptables -w -I INPUT -p tcp --dport http -j f2b-j-w-iptables; }`", ), 'ip6-start': ( - "`ip6tables -w -N f2b-j-w-iptables`", - "`ip6tables -w -A f2b-j-w-iptables -j RETURN`", - "`ip6tables -w -I INPUT -p tcp --dport http -j f2b-j-w-iptables`", + "`{ ip6tables -w -C f2b-j-w-iptables -j RETURN >/dev/null 2>&1; } || " + "{ ip6tables -w -N f2b-j-w-iptables || true; ip6tables -w -A f2b-j-w-iptables -j RETURN; }", + "`{ ip6tables -w -C INPUT -p tcp --dport http -j f2b-j-w-iptables >/dev/null 2>&1; } || " + "{ ip6tables -w -I INPUT -p tcp --dport http -j f2b-j-w-iptables; }`", ), 'flush': ( "`iptables -w -F f2b-j-w-iptables`", @@ -1722,14 +1732,16 @@ class ServerConfigReaderTests(LogCaptureTestCase): ('j-w-iptables-new', 'iptables-new[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain=""]', { 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), 'ip4-start': ( - "`iptables -w -N f2b-j-w-iptables-new`", - "`iptables -w -A f2b-j-w-iptables-new -j RETURN`", - "`iptables -w -I INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`", + "`{ iptables -w -C f2b-j-w-iptables-new -j RETURN >/dev/null 2>&1; } || " + "{ iptables -w -N f2b-j-w-iptables-new || true; iptables -w -A f2b-j-w-iptables-new -j RETURN; }`", + "`{ iptables -w -C INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new >/dev/null 2>&1; } || " + "{ iptables -w -I INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new; }`", ), 'ip6-start': ( - "`ip6tables -w -N f2b-j-w-iptables-new`", - "`ip6tables -w -A f2b-j-w-iptables-new -j RETURN`", - "`ip6tables -w -I INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`", + "`{ ip6tables -w -C f2b-j-w-iptables-new -j RETURN >/dev/null 2>&1; } || " + "{ ip6tables -w -N f2b-j-w-iptables-new || true; ip6tables -w -A f2b-j-w-iptables-new -j RETURN; }`", + "`{ ip6tables -w -C INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new >/dev/null 2>&1; } || " + "{ ip6tables -w -I INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new; }`", ), 'flush': ( "`iptables -w -F f2b-j-w-iptables-new`", From 06d2623c5e243e11fd45bcc98cbd099c3973e597 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 26 Jan 2022 21:51:11 +0100 Subject: [PATCH 184/240] iptables and iptables-ipset actions extended to support multiple protocols with single action for multiport or oneport type (back-ported from nftables action); amend to gh-980 fixing several actions (correctly supporting new enhancements now) --- config/action.d/iptables-ipset-proto4.conf | 4 +- config/action.d/iptables-ipset.conf | 4 +- config/action.d/iptables-xt_recent-echo.conf | 18 ++- config/action.d/iptables.conf | 27 +++- fail2ban/tests/servertestcase.py | 156 +++++++++++-------- 5 files changed, 133 insertions(+), 76 deletions(-) diff --git a/config/action.d/iptables-ipset-proto4.conf b/config/action.d/iptables-ipset-proto4.conf index 2e2f5779..37624284 100644 --- a/config/action.d/iptables-ipset-proto4.conf +++ b/config/action.d/iptables-ipset-proto4.conf @@ -28,7 +28,7 @@ before = iptables.conf # Values: CMD # actionstart = ipset --create f2b- iphash - -I %(_ipt_chain_rule)s + <_ipt_add_rules> # Option: actionflush @@ -41,7 +41,7 @@ actionflush = ipset --flush f2b- # Notes.: command executed at the stop of jail (or at the end of Fail2Ban) # Values: CMD # -actionstop = -D -p -m multiport --dports -m set --match-set f2b- src -j +actionstop = <_ipt_del_rules> ipset --destroy f2b- diff --git a/config/action.d/iptables-ipset.conf b/config/action.d/iptables-ipset.conf index 74dfe6c3..b44e6ec4 100644 --- a/config/action.d/iptables-ipset.conf +++ b/config/action.d/iptables-ipset.conf @@ -25,7 +25,7 @@ before = iptables.conf # Values: CMD # actionstart = ipset -exist create hash:ip timeout - { %(_ipt_check_rule)s >/dev/null 2>&1; } || { -I %(_ipt_chain_rule)s; } + <_ipt_add_rules> # Option: actionflush # Notes.: command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action) @@ -37,7 +37,7 @@ actionflush = ipset flush # Notes.: command executed at the stop of jail (or at the end of Fail2Ban) # Values: CMD # -actionstop = -D %(_ipt_chain_rule)s +actionstop = <_ipt_del_rules> ipset destroy diff --git a/config/action.d/iptables-xt_recent-echo.conf b/config/action.d/iptables-xt_recent-echo.conf index 7e24db7a..c3c175b3 100644 --- a/config/action.d/iptables-xt_recent-echo.conf +++ b/config/action.d/iptables-xt_recent-echo.conf @@ -11,6 +11,10 @@ before = iptables.conf [Definition] +_ipt_chain_rule = -m recent --update --seconds 3600 --name -j +_ipt_for_proto-iter = +_ipt_for_proto-done = + # Option: actionstart # Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false). # Values: CMD @@ -33,7 +37,9 @@ before = iptables.conf # own rules. The 3600 second timeout is independent and acts as a # safeguard in case the fail2ban process dies unexpectedly. The # shorter of the two timeouts actually matters. -actionstart = if [ `id -u` -eq 0 ];then -I -m recent --update --seconds 3600 --name -j ;fi +actionstart = if [ `id -u` -eq 0 ];then + { %(_ipt_check_rule)s >/dev/null 2>&1; } || { -I %(_ipt_chain_rule)s; } + fi # Option: actionflush # @@ -46,13 +52,15 @@ actionflush = # Values: CMD # actionstop = echo / > /proc/net/xt_recent/ - if [ `id -u` -eq 0 ];then -D -m recent --update --seconds 3600 --name -j ;fi + if [ `id -u` -eq 0 ];then + -D %(_ipt_chain_rule)s; + fi # Option: actioncheck -# Notes.: command executed once before each actionban command +# Notes.: command executed as invariant check (error by ban) # Values: CMD # -actioncheck = { ; } && test -e /proc/net/xt_recent/ +actioncheck = { -C %(_ipt_chain_rule)s; } && test -e /proc/net/xt_recent/ # Option: actionban # Notes.: command executed when banning an IP. Take care that the @@ -72,7 +80,7 @@ actionunban = echo - > /proc/net/xt_recent/ [Init] -iptname = f2b- +iptname = f2b- [Init?family=inet6] diff --git a/config/action.d/iptables.conf b/config/action.d/iptables.conf index 821a9ef1..67d496f5 100644 --- a/config/action.d/iptables.conf +++ b/config/action.d/iptables.conf @@ -23,13 +23,13 @@ actionflush = -F f2b- # Values: CMD # actionstart = { -C f2b- -j >/dev/null 2>&1; } || { -N f2b- || true; -A f2b- -j ; } - { %(_ipt_check_rule)s >/dev/null 2>&1; } || { -I %(_ipt_chain_rule)s; } + <_ipt_add_rules> # Option: actionstop # Notes.: command executed at the stop of jail (or at the end of Fail2Ban) # Values: CMD # -actionstop = -D %(_ipt_chain_rule)s +actionstop = <_ipt_del_rules> -X f2b- @@ -37,7 +37,7 @@ actionstop = -D %(_ipt_chain_rule)s # Notes.: command executed once before each actionban command # Values: CMD # -actioncheck = <_ipt_check_rule> +actioncheck = <_ipt_check_rules> # Option: actionban # Notes.: command executed when banning an IP. Take care that the @@ -64,21 +64,36 @@ rule-jump = -j <_ipt_rule_target> # Several capabilities used internaly: +_ipt_for_proto-iter = for proto in $(echo '' | sed 's/,/ /g'); do +_ipt_for_proto-done = done + +_ipt_add_rules = <_ipt_for_proto-iter> + { %(_ipt_check_rule)s >/dev/null 2>&1; } || { -I %(_ipt_chain_rule)s; } + <_ipt_for_proto-done> + +_ipt_del_rules = <_ipt_for_proto-iter> + -D %(_ipt_chain_rule)s + <_ipt_for_proto-done> + +_ipt_check_rules = <_ipt_for_proto-iter> + %(_ipt_check_rule)s + <_ipt_for_proto-done> + _ipt_chain_rule = /_chain_rule> _ipt_check_rule = -C %(_ipt_chain_rule)s _ipt_rule_target = f2b- [ipt_oneport] -_chain_rule = -p --dport +_chain_rule = -p $proto --dport [ipt_multiport] -_chain_rule = -p -m multiport --dports +_chain_rule = -p $proto -m multiport --dports [ipt_allports] -_chain_rule = -p +_chain_rule = -p $proto [Init] diff --git a/fail2ban/tests/servertestcase.py b/fail2ban/tests/servertestcase.py index 8741893d..b7ad4d32 100644 --- a/fail2ban/tests/servertestcase.py +++ b/fail2ban/tests/servertestcase.py @@ -1503,37 +1503,42 @@ class ServerConfigReaderTests(LogCaptureTestCase): ), }), # iptables-multiport -- - ('j-w-iptables-mp', 'iptables-multiport[name=%(__name__)s, bantime="10m", port="http,https", protocol="tcp", chain=""]', { + ('j-w-iptables-mp', 'iptables-multiport[name=%(__name__)s, bantime="10m", port="http,https", protocol="tcp,udp,sctp", chain=""]', { 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), + '*-start-stop-check': ( + # iterator over protocol is same for both families: + r"`for proto in $(echo 'tcp,udp,sctp' | sed 's/,/ /g'); do`", + r"`done`", + ), 'ip4-start': ( "`{ iptables -w -C f2b-j-w-iptables-mp -j RETURN >/dev/null 2>&1; } || " "{ iptables -w -N f2b-j-w-iptables-mp || true; iptables -w -A f2b-j-w-iptables-mp -j RETURN; }`", - "`{ iptables -w -C INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp >/dev/null 2>&1; } || " - "{ iptables -w -I INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp; }`", + "`{ iptables -w -C INPUT -p $proto -m multiport --dports http,https -j f2b-j-w-iptables-mp >/dev/null 2>&1; } || " + "{ iptables -w -I INPUT -p $proto -m multiport --dports http,https -j f2b-j-w-iptables-mp; }`", ), 'ip6-start': ( "`{ ip6tables -w -C f2b-j-w-iptables-mp -j RETURN >/dev/null 2>&1; } || " "{ ip6tables -w -N f2b-j-w-iptables-mp || true; ip6tables -w -A f2b-j-w-iptables-mp -j RETURN; }`", - "`{ ip6tables -w -C INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp >/dev/null 2>&1; } || ", - "{ ip6tables -w -I INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp; }`", + "`{ ip6tables -w -C INPUT -p $proto -m multiport --dports http,https -j f2b-j-w-iptables-mp >/dev/null 2>&1; } || ", + "{ ip6tables -w -I INPUT -p $proto -m multiport --dports http,https -j f2b-j-w-iptables-mp; }`", ), 'flush': ( "`iptables -w -F f2b-j-w-iptables-mp`", "`ip6tables -w -F f2b-j-w-iptables-mp`", ), 'stop': ( - "`iptables -w -D INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`", + "`iptables -w -D INPUT -p $proto -m multiport --dports http,https -j f2b-j-w-iptables-mp`", "`iptables -w -F f2b-j-w-iptables-mp`", "`iptables -w -X f2b-j-w-iptables-mp`", - "`ip6tables -w -D INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`", + "`ip6tables -w -D INPUT -p $proto -m multiport --dports http,https -j f2b-j-w-iptables-mp`", "`ip6tables -w -F f2b-j-w-iptables-mp`", "`ip6tables -w -X f2b-j-w-iptables-mp`", ), 'ip4-check': ( - r"""`iptables -w -C INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`""", + r"""`iptables -w -C INPUT -p $proto -m multiport --dports http,https -j f2b-j-w-iptables-mp`""", ), 'ip6-check': ( - r"""`ip6tables -w -C INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`""", + r"""`ip6tables -w -C INPUT -p $proto -m multiport --dports http,https -j f2b-j-w-iptables-mp`""", ), 'ip4-ban': ( r"`iptables -w -I f2b-j-w-iptables-mp 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", @@ -1549,37 +1554,42 @@ class ServerConfigReaderTests(LogCaptureTestCase): ), }), # iptables-allports -- - ('j-w-iptables-ap', 'iptables-allports[name=%(__name__)s, bantime="10m", protocol="tcp", chain=""]', { + ('j-w-iptables-ap', 'iptables-allports[name=%(__name__)s, bantime="10m", protocol="tcp,udp,sctp", chain=""]', { 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), + '*-start-stop-check': ( + # iterator over protocol is same for both families: + r"`for proto in $(echo 'tcp,udp,sctp' | sed 's/,/ /g'); do`", + r"`done`", + ), 'ip4-start': ( "`{ iptables -w -C f2b-j-w-iptables-ap -j RETURN >/dev/null 2>&1; } || " "{ iptables -w -N f2b-j-w-iptables-ap || true; iptables -w -A f2b-j-w-iptables-ap -j RETURN; }`", - "`{ iptables -w -C INPUT -p tcp -j f2b-j-w-iptables-ap >/dev/null 2>&1; } || ", - "{ iptables -w -I INPUT -p tcp -j f2b-j-w-iptables-ap; }`", + "`{ iptables -w -C INPUT -p $proto -j f2b-j-w-iptables-ap >/dev/null 2>&1; } || ", + "{ iptables -w -I INPUT -p $proto -j f2b-j-w-iptables-ap; }`", ), 'ip6-start': ( "`{ ip6tables -w -C f2b-j-w-iptables-ap -j RETURN >/dev/null 2>&1; } || " "{ ip6tables -w -N f2b-j-w-iptables-ap || true; ip6tables -w -A f2b-j-w-iptables-ap -j RETURN; }`", - "`{ ip6tables -w -C INPUT -p tcp -j f2b-j-w-iptables-ap >/dev/null 2>&1; } || ", - "{ ip6tables -w -I INPUT -p tcp -j f2b-j-w-iptables-ap; }`", + "`{ ip6tables -w -C INPUT -p $proto -j f2b-j-w-iptables-ap >/dev/null 2>&1; } || ", + "{ ip6tables -w -I INPUT -p $proto -j f2b-j-w-iptables-ap; }`", ), 'flush': ( "`iptables -w -F f2b-j-w-iptables-ap`", "`ip6tables -w -F f2b-j-w-iptables-ap`", ), 'stop': ( - "`iptables -w -D INPUT -p tcp -j f2b-j-w-iptables-ap`", + "`iptables -w -D INPUT -p $proto -j f2b-j-w-iptables-ap`", "`iptables -w -F f2b-j-w-iptables-ap`", "`iptables -w -X f2b-j-w-iptables-ap`", - "`ip6tables -w -D INPUT -p tcp -j f2b-j-w-iptables-ap`", + "`ip6tables -w -D INPUT -p $proto -j f2b-j-w-iptables-ap`", "`ip6tables -w -F f2b-j-w-iptables-ap`", "`ip6tables -w -X f2b-j-w-iptables-ap`", ), 'ip4-check': ( - r"""`iptables -w -C INPUT -p tcp -j f2b-j-w-iptables-ap`""", + r"""`iptables -w -C INPUT -p $proto -j f2b-j-w-iptables-ap`""", ), 'ip6-check': ( - r"""`ip6tables -w -C INPUT -p tcp -j f2b-j-w-iptables-ap`""", + r"""`ip6tables -w -C INPUT -p $proto -j f2b-j-w-iptables-ap`""", ), 'ip4-ban': ( r"`iptables -w -I f2b-j-w-iptables-ap 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", @@ -1597,33 +1607,38 @@ class ServerConfigReaderTests(LogCaptureTestCase): # iptables-ipset-proto6 -- ('j-w-iptables-ipset', 'iptables-ipset-proto6[name=%(__name__)s, port="http", protocol="tcp", chain=""]', { 'ip4': (' f2b-j-w-iptables-ipset ',), 'ip6': (' f2b-j-w-iptables-ipset6 ',), + '*-start-stop-check': ( + # iterator over protocol is same for both families: + "`for proto in $(echo 'tcp' | sed 's/,/ /g'); do`", + "`done`", + ), 'ip4-start': ( "`ipset -exist create f2b-j-w-iptables-ipset hash:ip timeout 0 `", - "`{ iptables -w -C INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable >/dev/null 2>&1; } || " - "{ iptables -w -I INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable; }`", + "`{ iptables -w -C INPUT -p $proto -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable >/dev/null 2>&1; } || " + "{ iptables -w -I INPUT -p $proto -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable; }`", ), 'ip6-start': ( "`ipset -exist create f2b-j-w-iptables-ipset6 hash:ip timeout 0 family inet6`", - "`{ ip6tables -w -C INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable >/dev/null 2>&1; } || " - "{ ip6tables -w -I INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable; }`", + "`{ ip6tables -w -C INPUT -p $proto -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable >/dev/null 2>&1; } || " + "{ ip6tables -w -I INPUT -p $proto -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable; }`", ), 'flush': ( "`ipset flush f2b-j-w-iptables-ipset`", "`ipset flush f2b-j-w-iptables-ipset6`", ), 'stop': ( - "`iptables -w -D INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable`", + "`iptables -w -D INPUT -p $proto -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable`", "`ipset flush f2b-j-w-iptables-ipset`", "`ipset destroy f2b-j-w-iptables-ipset`", - "`ip6tables -w -D INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", + "`ip6tables -w -D INPUT -p $proto -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", "`ipset flush f2b-j-w-iptables-ipset6`", "`ipset destroy f2b-j-w-iptables-ipset6`", ), 'ip4-check': ( - r"""`iptables -w -C INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable`""", + r"""`iptables -w -C INPUT -p $proto -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable`""", ), 'ip6-check': ( - r"""`ip6tables -w -C INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`""", + r"""`ip6tables -w -C INPUT -p $proto -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`""", ), 'ip4-ban': ( r"`ipset -exist add f2b-j-w-iptables-ipset 192.0.2.1 timeout 0`", @@ -1641,33 +1656,38 @@ class ServerConfigReaderTests(LogCaptureTestCase): # iptables-ipset-proto6-allports -- ('j-w-iptables-ipset-ap', 'iptables-ipset-proto6-allports[name=%(__name__)s, chain=""]', { 'ip4': (' f2b-j-w-iptables-ipset-ap ',), 'ip6': (' f2b-j-w-iptables-ipset-ap6 ',), + '*-start-stop-check': ( + # iterator over protocol is same for both families: + "`for proto in $(echo 'tcp' | sed 's/,/ /g'); do`", + "`done`", + ), 'ip4-start': ( "`ipset -exist create f2b-j-w-iptables-ipset-ap hash:ip timeout 0 `", - "`{ iptables -w -C INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable >/dev/null 2>&1; } || " - "{ iptables -w -I INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable; }", + "`{ iptables -w -C INPUT -p $proto -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable >/dev/null 2>&1; } || " + "{ iptables -w -I INPUT -p $proto -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable; }", ), 'ip6-start': ( "`ipset -exist create f2b-j-w-iptables-ipset-ap6 hash:ip timeout 0 family inet6`", - "`{ ip6tables -w -C INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable >/dev/null 2>&1; } || " - "{ ip6tables -w -I INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable; }", + "`{ ip6tables -w -C INPUT -p $proto -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable >/dev/null 2>&1; } || " + "{ ip6tables -w -I INPUT -p $proto -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable; }", ), 'flush': ( "`ipset flush f2b-j-w-iptables-ipset-ap`", "`ipset flush f2b-j-w-iptables-ipset-ap6`", ), 'stop': ( - "`iptables -w -D INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", + "`iptables -w -D INPUT -p $proto -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", "`ipset flush f2b-j-w-iptables-ipset-ap`", "`ipset destroy f2b-j-w-iptables-ipset-ap`", - "`ip6tables -w -D INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", + "`ip6tables -w -D INPUT -p $proto -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", "`ipset flush f2b-j-w-iptables-ipset-ap6`", "`ipset destroy f2b-j-w-iptables-ipset-ap6`", ), 'ip4-check': ( - r"""`iptables -w -C INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`""", + r"""`iptables -w -C INPUT -p $proto -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`""", ), 'ip6-check': ( - r"""`ip6tables -w -C INPUT -p tcp -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`""", + r"""`ip6tables -w -C INPUT -p $proto -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`""", ), 'ip4-ban': ( r"`ipset -exist add f2b-j-w-iptables-ipset-ap 192.0.2.1 timeout 0`", @@ -1685,35 +1705,40 @@ class ServerConfigReaderTests(LogCaptureTestCase): # iptables (oneport) -- ('j-w-iptables', 'iptables[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain=""]', { 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), + '*-start-stop-check': ( + # iterator over protocol is same for both families: + "`for proto in $(echo 'tcp' | sed 's/,/ /g'); do`", + "`done`", + ), 'ip4-start': ( "`{ iptables -w -C f2b-j-w-iptables -j RETURN >/dev/null 2>&1; } || " "{ iptables -w -N f2b-j-w-iptables || true; iptables -w -A f2b-j-w-iptables -j RETURN; }", - "`{ iptables -w -C INPUT -p tcp --dport http -j f2b-j-w-iptables >/dev/null 2>&1; } || " - "{ iptables -w -I INPUT -p tcp --dport http -j f2b-j-w-iptables; }`", + "`{ iptables -w -C INPUT -p $proto --dport http -j f2b-j-w-iptables >/dev/null 2>&1; } || " + "{ iptables -w -I INPUT -p $proto --dport http -j f2b-j-w-iptables; }`", ), 'ip6-start': ( "`{ ip6tables -w -C f2b-j-w-iptables -j RETURN >/dev/null 2>&1; } || " "{ ip6tables -w -N f2b-j-w-iptables || true; ip6tables -w -A f2b-j-w-iptables -j RETURN; }", - "`{ ip6tables -w -C INPUT -p tcp --dport http -j f2b-j-w-iptables >/dev/null 2>&1; } || " - "{ ip6tables -w -I INPUT -p tcp --dport http -j f2b-j-w-iptables; }`", + "`{ ip6tables -w -C INPUT -p $proto --dport http -j f2b-j-w-iptables >/dev/null 2>&1; } || " + "{ ip6tables -w -I INPUT -p $proto --dport http -j f2b-j-w-iptables; }`", ), 'flush': ( "`iptables -w -F f2b-j-w-iptables`", "`ip6tables -w -F f2b-j-w-iptables`", ), 'stop': ( - "`iptables -w -D INPUT -p tcp --dport http -j f2b-j-w-iptables`", + "`iptables -w -D INPUT -p $proto --dport http -j f2b-j-w-iptables`", "`iptables -w -F f2b-j-w-iptables`", "`iptables -w -X f2b-j-w-iptables`", - "`ip6tables -w -D INPUT -p tcp --dport http -j f2b-j-w-iptables`", + "`ip6tables -w -D INPUT -p $proto --dport http -j f2b-j-w-iptables`", "`ip6tables -w -F f2b-j-w-iptables`", "`ip6tables -w -X f2b-j-w-iptables`", ), 'ip4-check': ( - r"""`iptables -w -C INPUT -p tcp --dport http -j f2b-j-w-iptables`""", + r"""`iptables -w -C INPUT -p $proto --dport http -j f2b-j-w-iptables`""", ), 'ip6-check': ( - r"""`ip6tables -w -C INPUT -p tcp --dport http -j f2b-j-w-iptables`""", + r"""`ip6tables -w -C INPUT -p $proto --dport http -j f2b-j-w-iptables`""", ), 'ip4-ban': ( r"`iptables -w -I f2b-j-w-iptables 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", @@ -1731,35 +1756,40 @@ class ServerConfigReaderTests(LogCaptureTestCase): # iptables-new -- ('j-w-iptables-new', 'iptables-new[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain=""]', { 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), + '*-start-stop-check': ( + # iterator over protocol is same for both families: + "`for proto in $(echo 'tcp' | sed 's/,/ /g'); do`", + "`done`", + ), 'ip4-start': ( "`{ iptables -w -C f2b-j-w-iptables-new -j RETURN >/dev/null 2>&1; } || " "{ iptables -w -N f2b-j-w-iptables-new || true; iptables -w -A f2b-j-w-iptables-new -j RETURN; }`", - "`{ iptables -w -C INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new >/dev/null 2>&1; } || " - "{ iptables -w -I INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new; }`", + "`{ iptables -w -C INPUT -m state --state NEW -p $proto --dport http -j f2b-j-w-iptables-new >/dev/null 2>&1; } || " + "{ iptables -w -I INPUT -m state --state NEW -p $proto --dport http -j f2b-j-w-iptables-new; }`", ), 'ip6-start': ( "`{ ip6tables -w -C f2b-j-w-iptables-new -j RETURN >/dev/null 2>&1; } || " "{ ip6tables -w -N f2b-j-w-iptables-new || true; ip6tables -w -A f2b-j-w-iptables-new -j RETURN; }`", - "`{ ip6tables -w -C INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new >/dev/null 2>&1; } || " - "{ ip6tables -w -I INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new; }`", + "`{ ip6tables -w -C INPUT -m state --state NEW -p $proto --dport http -j f2b-j-w-iptables-new >/dev/null 2>&1; } || " + "{ ip6tables -w -I INPUT -m state --state NEW -p $proto --dport http -j f2b-j-w-iptables-new; }`", ), 'flush': ( "`iptables -w -F f2b-j-w-iptables-new`", "`ip6tables -w -F f2b-j-w-iptables-new`", ), 'stop': ( - "`iptables -w -D INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`", + "`iptables -w -D INPUT -m state --state NEW -p $proto --dport http -j f2b-j-w-iptables-new`", "`iptables -w -F f2b-j-w-iptables-new`", "`iptables -w -X f2b-j-w-iptables-new`", - "`ip6tables -w -D INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`", + "`ip6tables -w -D INPUT -m state --state NEW -p $proto --dport http -j f2b-j-w-iptables-new`", "`ip6tables -w -F f2b-j-w-iptables-new`", "`ip6tables -w -X f2b-j-w-iptables-new`", ), 'ip4-check': ( - r"""`iptables -w -C INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`""", + r"""`iptables -w -C INPUT -m state --state NEW -p $proto --dport http -j f2b-j-w-iptables-new`""", ), 'ip6-check': ( - r"""`ip6tables -w -C INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`""", + r"""`ip6tables -w -C INPUT -m state --state NEW -p $proto --dport http -j f2b-j-w-iptables-new`""", ), 'ip4-ban': ( r"`iptables -w -I f2b-j-w-iptables-new 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", @@ -1778,22 +1808,26 @@ class ServerConfigReaderTests(LogCaptureTestCase): ('j-w-iptables-xtre', 'iptables-xt_recent-echo[name=%(__name__)s, bantime="10m", chain=""]', { 'ip4': ('`iptables ', '/f2b-j-w-iptables-xtre`'), 'ip6': ('`ip6tables ', '/f2b-j-w-iptables-xtre6`'), 'ip4-start': ( - "`if [ `id -u` -eq 0 ];then iptables -w -I INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre -j REJECT --reject-with icmp-port-unreachable;fi`", + "`{ iptables -w -C INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre -j REJECT --reject-with icmp-port-unreachable >/dev/null 2>&1; } || { iptables -w -I INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre -j REJECT --reject-with icmp-port-unreachable; }`", ), 'ip6-start': ( - "`if [ `id -u` -eq 0 ];then ip6tables -w -I INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre6 -j REJECT --reject-with icmp6-port-unreachable;fi`", + "`{ ip6tables -w -C INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre6 -j REJECT --reject-with icmp6-port-unreachable >/dev/null 2>&1; } || { ip6tables -w -I INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre6 -j REJECT --reject-with icmp6-port-unreachable; }`", ), 'stop': ( "`echo / > /proc/net/xt_recent/f2b-j-w-iptables-xtre`", - "`if [ `id -u` -eq 0 ];then iptables -w -D INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre -j REJECT --reject-with icmp-port-unreachable;fi`", + "`if [ `id -u` -eq 0 ];then`", + "`iptables -w -D INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre -j REJECT --reject-with icmp-port-unreachable;`", + "`fi`", "`echo / > /proc/net/xt_recent/f2b-j-w-iptables-xtre6`", - "`if [ `id -u` -eq 0 ];then ip6tables -w -D INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre6 -j REJECT --reject-with icmp6-port-unreachable;fi`", + "`if [ `id -u` -eq 0 ];then`", + "`ip6tables -w -D INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre6 -j REJECT --reject-with icmp6-port-unreachable;`", + "`fi`", ), 'ip4-check': ( - r"`{ iptables -w -C INPUT -p tcp --dport ssh -j f2b-j-w-iptables-xtre; } && test -e /proc/net/xt_recent/f2b-j-w-iptables-xtre`", + r"`{ iptables -w -C INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre -j REJECT --reject-with icmp-port-unreachable; } && test -e /proc/net/xt_recent/f2b-j-w-iptables-xtre`", ), 'ip6-check': ( - r"`{ ip6tables -w -C INPUT -p tcp --dport ssh -j f2b-j-w-iptables-xtre; } && test -e /proc/net/xt_recent/f2b-j-w-iptables-xtre6`", + r"`{ ip6tables -w -C INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre6 -j REJECT --reject-with icmp6-port-unreachable; } && test -e /proc/net/xt_recent/f2b-j-w-iptables-xtre6`", ), 'ip4-ban': ( r"`echo +192.0.2.1 > /proc/net/xt_recent/f2b-j-w-iptables-xtre`", @@ -2101,7 +2135,7 @@ class ServerConfigReaderTests(LogCaptureTestCase): # test ban ip4 : self.pruneLog('# === ban-ipv4 ===') action.ban(aInfos['ipv4']) - if tests.get('ip4-start'): self.assertLogged(*tests.get('*-start', ())+tests['ip4-start'], all=True) + if tests.get('ip4-start'): self.assertLogged(*tests.get('*-start', tests.get('*-start-stop-check', ()))+tests['ip4-start'], all=True) if tests.get('ip6-start'): self.assertNotLogged(*tests['ip6-start'], all=True) self.assertLogged(*tests['ip4-ban'], all=True) self.assertNotLogged(*tests['ip6'], all=True) @@ -2113,7 +2147,7 @@ class ServerConfigReaderTests(LogCaptureTestCase): # test ban ip6 : self.pruneLog('# === ban ipv6 ===') action.ban(aInfos['ipv6']) - if tests.get('ip6-start'): self.assertLogged(*tests.get('*-start', ())+tests['ip6-start'], all=True) + if tests.get('ip6-start'): self.assertLogged(*tests.get('*-start', tests.get('*-start-stop-check', ()))+tests['ip6-start'], all=True) if tests.get('ip4-start'): self.assertNotLogged(*tests['ip4-start'], all=True) self.assertLogged(*tests['ip6-ban'], all=True) self.assertNotLogged(*tests['ip4'], all=True) @@ -2126,13 +2160,13 @@ class ServerConfigReaderTests(LogCaptureTestCase): if tests.get('ip4-check'): self.pruneLog('# === check ipv4 ===') action._invariantCheck(aInfos['ipv4']['family']) - self.assertLogged(*tests['ip4-check'], all=True) + self.assertLogged(*tests.get('*-check', tests.get('*-start-stop-check', ()))+tests['ip4-check'], all=True) if tests.get('ip6-check') and tests['ip6-check'] != tests['ip4-check']: self.assertNotLogged(*tests['ip6-check'], all=True) if tests.get('ip6-check'): self.pruneLog('# === check ipv6 ===') action._invariantCheck(aInfos['ipv6']['family']) - self.assertLogged(*tests['ip6-check'], all=True) + self.assertLogged(*tests.get('*-check', tests.get('*-start-stop-check', ()))+tests['ip6-check'], all=True) if tests.get('ip4-check') and tests['ip4-check'] != tests['ip6-check']: self.assertNotLogged(*tests['ip4-check'], all=True) # test flush for actions should supported this: @@ -2143,7 +2177,7 @@ class ServerConfigReaderTests(LogCaptureTestCase): # test stop : self.pruneLog('# === stop ===') action.stop() - if tests.get('stop'): self.assertLogged(*tests['stop'], all=True) + if tests.get('stop'): self.assertLogged(*tests.get('*-start-stop-check', ())+tests['stop'], all=True) def _executeMailCmd(self, realCmd, timeout=60): # replace pipe to mail with pipe to cat: From f4641dfc00570e7bf73c210c7917be8b551f9536 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 26 Jan 2022 17:56:00 +0100 Subject: [PATCH 185/240] observer API simplification (no failmanager in call of failureFound, jail.filter.failManager is enough) --- fail2ban/server/filter.py | 2 +- fail2ban/server/observer.py | 14 ++++---------- fail2ban/tests/observertestcase.py | 13 +++++++------ 3 files changed, 12 insertions(+), 17 deletions(-) diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index 6f1572ef..5af3626a 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -724,7 +724,7 @@ class Filter(JailThread): self.performBan(ip) # report to observer - failure was found, for possibly increasing of it retry counter (asynchronous) if Observers.Main is not None: - Observers.Main.add('failureFound', self.failManager, self.jail, tick) + Observers.Main.add('failureFound', self.jail, tick) self.procLines += 1 # every 100 lines check need to perform service tasks: if self.procLines % 100 == 0: diff --git a/fail2ban/server/observer.py b/fail2ban/server/observer.py index 241c677e..ecbcd5b7 100644 --- a/fail2ban/server/observer.py +++ b/fail2ban/server/observer.py @@ -364,7 +364,7 @@ class ObserverThread(JailThread): ## [Async] ban time increment functionality ... ## ----------------------------------------- - def failureFound(self, failManager, jail, ticket): + def failureFound(self, jail, ticket): """ Notify observer a failure for ip was found Observer will check ip was known (bad) and possibly increase an retry count @@ -380,7 +380,7 @@ class ObserverThread(JailThread): retryCount = 1 timeOfBan = None try: - maxRetry = failManager.getMaxRetry() + maxRetry = jail.filter.failManager.getMaxRetry() db = jail.database if db is not None: for banCount, timeOfBan, lastBanTime in db.getBan(ip, jail): @@ -403,18 +403,12 @@ class ObserverThread(JailThread): MyTime.time2str(unixTime), banCount, retryCount, (', Ban' if retryCount >= maxRetry else '')) # retryCount-1, because a ticket was already once incremented by filter self - retryCount = failManager.addFailure(ticket, retryCount - 1, True) + retryCount = jail.filter.failManager.addFailure(ticket, retryCount - 1, True) ticket.setBanCount(banCount) # after observe we have increased attempt count, compare it >= maxretry ... if retryCount >= maxRetry: # perform the banning of the IP now (again) - # [todo]: this code part will be used multiple times - optimize it later. - try: # pragma: no branch - exception is the only way out - while True: - ticket = failManager.toBan(ip) - jail.putFailTicket(ticket) - except FailManagerEmpty: - failManager.cleanup(MyTime.time()) + jail.filter.performBan(ip) except Exception as e: logSys.error('%s', e, exc_info=logSys.getEffectiveLevel()<=logging.DEBUG) diff --git a/fail2ban/tests/observertestcase.py b/fail2ban/tests/observertestcase.py index e379ccd1..38cfc881 100644 --- a/fail2ban/tests/observertestcase.py +++ b/fail2ban/tests/observertestcase.py @@ -450,7 +450,8 @@ class BanTimeIncrDB(LogCaptureTestCase): def testObserver(self): if Fail2BanDb is None: # pragma: no cover return - jail = self.jail + jail = self.jail = DummyJail(backend='polling') + jail.database = self.db self.db.addJail(jail) # we tests with initial ban time = 10 seconds: jail.actions.setBanTime(10) @@ -480,27 +481,27 @@ class BanTimeIncrDB(LogCaptureTestCase): # add failure: ip = "192.0.2.1" ticket = FailTicket(ip, stime-120, []) - failManager = FailManager() + failManager = jail.filter.failManager = FailManager() failManager.setMaxRetry(3) for i in xrange(3): failManager.addFailure(ticket) - obs.add('failureFound', failManager, jail, ticket) + obs.add('failureFound', jail, ticket) obs.wait_empty(5) self.assertEqual(ticket.getBanCount(), 0) # check still not ban : self.assertTrue(not jail.getFailTicket()) # add manually 4th times banned (added to bips - make ip bad): ticket.setBanCount(4) - self.db.addBan(self.jail, ticket) + self.db.addBan(jail, ticket) restored_tickets = self.db.getCurrentBans(jail=jail, fromtime=stime-120, correctBanTime=False) self.assertEqual(len(restored_tickets), 1) # check again, new ticket, new failmanager: ticket = FailTicket(ip, stime, []) - failManager = FailManager() + failManager = jail.filter.failManager = FailManager() failManager.setMaxRetry(3) # add once only - but bad - should be banned: failManager.addFailure(ticket) - obs.add('failureFound', failManager, self.jail, ticket) + obs.add('failureFound', jail, ticket) obs.wait_empty(5) # wait until ticket transfered from failmanager into jail: ticket2 = Utils.wait_for(jail.getFailTicket, 10) From 0f1706d4a10e2d4f08f1ee0cc92daea634d7b00b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A1szl=C3=B3=20K=C3=A1rolyi?= Date: Thu, 27 Jan 2022 11:28:20 +0000 Subject: [PATCH 186/240] Adjusting for updated dovecot log format This should now match: `Disconnected: Connection closed: read(size=1003) failed: Connection reset by peer (auth failed, 1 attempts in 0 secs): user=, rip=183.111.188.94, lip=127.0.0.19, session=` the issue is the `read(size=1003)` that probably has been added lately and which causes the rule not to discover the log message. --- config/filter.d/dovecot.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/filter.d/dovecot.conf b/config/filter.d/dovecot.conf index 9c817720..fcd28b58 100644 --- a/config/filter.d/dovecot.conf +++ b/config/filter.d/dovecot.conf @@ -14,7 +14,7 @@ _daemon = (?:dovecot(?:-auth)?|auth) prefregex = ^%(__prefix_line)s(?:%(_auth_worker)s(?:\([^\)]+\))?: )?(?:%(__pam_auth)s(?:\(dovecot:auth\))?: |(?:pop3|imap|managesieve|submission)-login: )?(?:Info: )?%(_auth_worker_info)s.+$ failregex = ^authentication failure; logname=\S* uid=\S* euid=\S* tty=dovecot ruser=\S* rhost=(?:\s+user=\S*)?\s*$ - ^(?:Aborted login|Disconnected|Remote closed connection|Client has quit the connection)(?::(?: [^ \(]+)+)? \((?:auth failed, \d+ attempts(?: in \d+ secs)?|tried to use (?:disabled|disallowed) \S+ auth|proxy dest auth failed)\):(?: user=<[^>]*>,)?(?: method=\S+,)? rip=(?:[^>]*(?:, session=<\S+>)?)\s*$ + ^(?:Aborted login|Disconnected|Remote closed connection|Client has quit the connection)(?::(?: [^ ]+)+)? \((?:auth failed, \d+ attempts(?: in \d+ secs)?|tried to use (?:disabled|disallowed) \S+ auth|proxy dest auth failed)\):(?: user=<[^>]*>,)?(?: method=\S+,)? rip=(?:[^>]*(?:, session=<\S+>)?)\s*$ ^pam\(\S+,(?:,\S*)?\): pam_authenticate\(\) failed: (?:User not known to the underlying authentication module: \d+ Time\(s\)|Authentication failure \([Pp]assword mismatch\?\)|Permission denied)\s*$ ^[a-z\-]{3,15}\(\S*,(?:,\S*)?\): (?:[Uu]nknown user|[Ii]nvalid credentials|[Pp]assword mismatch) > From af8a9f7ff9e93d59eaf48828c9af85049c3e0141 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Thu, 27 Jan 2022 17:44:58 +0100 Subject: [PATCH 187/240] added test to cover the new log-format --- fail2ban/tests/files/logs/dovecot | 2 ++ 1 file changed, 2 insertions(+) diff --git a/fail2ban/tests/files/logs/dovecot b/fail2ban/tests/files/logs/dovecot index 6bcf8c5b..e780411b 100644 --- a/fail2ban/tests/files/logs/dovecot +++ b/fail2ban/tests/files/logs/dovecot @@ -107,6 +107,8 @@ Jul 26 11:12:19 hostname dovecot: imap-login: Disconnected: Too many invalid com # failJSON: { "time": "2004-08-28T06:38:51", "match": true , "host": "192.0.2.3" } Aug 28 06:38:51 s166-62-100-187 dovecot: imap-login: Disconnected (auth failed, 1 attempts in 9 secs): user=, method=PLAIN, rip=192.0.2.3, lip=192.168.1.2, TLS: Disconnected, TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits) +# failJSON: { "time": "2004-08-28T06:38:52", "match": true , "host": "192.0.2.4", "desc": "open parenthesis in optional part between Disconnected and (auth failed ...), gh-3210" } +Aug 28 06:38:52 s166-62-100-187 dovecot: imap-login: Disconnected: Connection closed: read(size=1003) failed: Connection reset by peer (auth failed, 1 attempts in 0 secs): user=, rip=192.0.2.4, lip=127.0.0.19, session= # failJSON: { "time": "2004-08-29T03:17:18", "match": true , "host": "192.0.2.133" } Aug 29 03:17:18 server dovecot: submission-login: Client has quit the connection (auth failed, 1 attempts in 2 secs): user=, method=LOGIN, rip=192.0.2.133, lip=0.0.0.0 From dfc866ea410840a8c9bfba04d4fa92213221164d Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Thu, 27 Jan 2022 17:50:28 +0100 Subject: [PATCH 188/240] improve RE to solve conflict with expected another open parenthesis --- config/filter.d/dovecot.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/filter.d/dovecot.conf b/config/filter.d/dovecot.conf index fcd28b58..c55061c3 100644 --- a/config/filter.d/dovecot.conf +++ b/config/filter.d/dovecot.conf @@ -14,7 +14,7 @@ _daemon = (?:dovecot(?:-auth)?|auth) prefregex = ^%(__prefix_line)s(?:%(_auth_worker)s(?:\([^\)]+\))?: )?(?:%(__pam_auth)s(?:\(dovecot:auth\))?: |(?:pop3|imap|managesieve|submission)-login: )?(?:Info: )?%(_auth_worker_info)s.+$ failregex = ^authentication failure; logname=\S* uid=\S* euid=\S* tty=dovecot ruser=\S* rhost=(?:\s+user=\S*)?\s*$ - ^(?:Aborted login|Disconnected|Remote closed connection|Client has quit the connection)(?::(?: [^ ]+)+)? \((?:auth failed, \d+ attempts(?: in \d+ secs)?|tried to use (?:disabled|disallowed) \S+ auth|proxy dest auth failed)\):(?: user=<[^>]*>,)?(?: method=\S+,)? rip=(?:[^>]*(?:, session=<\S+>)?)\s*$ + ^(?:Aborted login|Disconnected|Remote closed connection|Client has quit the connection)(?:: (?:[^\(]+|\w+\([^\)]*\))+)? \((?:auth failed, \d+ attempts(?: in \d+ secs)?|tried to use (?:disabled|disallowed) \S+ auth|proxy dest auth failed)\):(?: user=<[^>]*>,)?(?: method=\S+,)? rip=(?:[^>]*(?:, session=<\S+>)?)\s*$ ^pam\(\S+,(?:,\S*)?\): pam_authenticate\(\) failed: (?:User not known to the underlying authentication module: \d+ Time\(s\)|Authentication failure \([Pp]assword mismatch\?\)|Permission denied)\s*$ ^[a-z\-]{3,15}\(\S*,(?:,\S*)?\): (?:[Uu]nknown user|[Ii]nvalid credentials|[Pp]assword mismatch) > From c7ae74ce178adb819cdb3342a248c4e30346bd33 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 8 Feb 2022 19:10:22 +0100 Subject: [PATCH 189/240] amend to a147a8b0e1b2f32b6f191932afd3c2db9765e2e3: systemd journal test-cases - additional check appropriate default settings (if testing as not root/sudoer) --- fail2ban/tests/filtertestcase.py | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 32dcca5c..f9fa5e97 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -1352,7 +1352,6 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover def setUp(self): """Call before every test case.""" super(MonitorJournalFailures, self).setUp() - self._runtimeJournal = None self.test_file = os.path.join(TEST_FILES_DIR, "testcase-journal.log") self.jail = DummyJail() self.filter = None @@ -1390,7 +1389,7 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover If not found, SkipTest exception will be raised. """ # we can cache it: - if self._runtimeJournal is None: + if not hasattr(MonitorJournalFailures, "_runtimeJournal"): # Depending on the system, it could be found under /run or /var/log (e.g. Debian) # which are pointed by different systemd-path variables. We will # check one at at time until the first hit @@ -1402,9 +1401,14 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover self.assertTrue(tmp) out = str(tmp[1].decode('utf-8')).split('\n')[0] if out: break - self._runtimeJournal = out - if self._runtimeJournal: - return self._runtimeJournal + # additional check appropriate default settings (if not root/sudoer and not already set): + if os.geteuid() != 0 and os.getenv("F2B_SYSTEMD_DEFAULT_FLAGS", None) is None: + # filter default SYSTEM_ONLY(4) is hardly usable for not root/sudoer tester, + # so back to default LOCAL_ONLY(1): + os.environ["F2B_SYSTEMD_DEFAULT_FLAGS"] = "0"; # or "1", what will be similar to journalflags=0 or ...=1 + MonitorJournalFailures._runtimeJournal = out + if MonitorJournalFailures._runtimeJournal: + return MonitorJournalFailures._runtimeJournal raise unittest.SkipTest('systemd journal seems to be not available (e. g. no rights to read)') def testJournalFilesArg(self): From 810386a265d621a0af9f4917f68fae9bf95d1ba4 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 8 Feb 2022 19:21:37 +0100 Subject: [PATCH 190/240] filter.d/dovecot.conf: parse everything in parenthesis by auth-worker info, e. g. can match (pid=...,uid=...) too (amend to 92f90038fa67d719c55c75f5eff8059e849fe747) --- config/filter.d/dovecot.conf | 2 +- fail2ban/tests/files/logs/dovecot | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/config/filter.d/dovecot.conf b/config/filter.d/dovecot.conf index 9c817720..b1df82f2 100644 --- a/config/filter.d/dovecot.conf +++ b/config/filter.d/dovecot.conf @@ -8,7 +8,7 @@ before = common.conf [Definition] _auth_worker = (?:dovecot: )?auth(?:-worker)? -_auth_worker_info = (?:conn \w+:auth(?:-worker)? \(uid=\w+\): auth(?:-worker)?<\d+>: )? +_auth_worker_info = (?:conn \w+:auth(?:-worker)? \([^\)]+\): auth(?:-worker)?<\d+>: )? _daemon = (?:dovecot(?:-auth)?|auth) prefregex = ^%(__prefix_line)s(?:%(_auth_worker)s(?:\([^\)]+\))?: )?(?:%(__pam_auth)s(?:\(dovecot:auth\))?: |(?:pop3|imap|managesieve|submission)-login: )?(?:Info: )?%(_auth_worker_info)s.+$ diff --git a/fail2ban/tests/files/logs/dovecot b/fail2ban/tests/files/logs/dovecot index 6bcf8c5b..7649cfeb 100644 --- a/fail2ban/tests/files/logs/dovecot +++ b/fail2ban/tests/files/logs/dovecot @@ -60,6 +60,11 @@ Jun 12 11:48:12 auth-worker(80180): Info: conn unix:auth-worker (uid=143): auth- # failJSON: { "time": "2005-06-12T23:06:05", "match": true , "host": "192.0.2.7" } Jun 12 23:06:05 auth-worker(57065): Info: conn unix:auth-worker (uid=143): auth-worker<225622>: sql(user@domain.com,192.0.2.7,): Password mismatch +# failJSON: { "time": "2005-06-15T11:28:21", "match": true , "host": "192.0.2.7" } +Jun 15 11:28:21 hostname dovecot: auth-worker(5787): conn unix:auth-worker (pid=27359,uid=97): auth-worker<55>: pam(webapps,192.0.2.7): unknown user +# failJSON: { "time": "2005-06-15T13:57:41", "match": true , "host": "192.0.2.7" } +Jun 15 13:57:41 hostname dovecot: auth-worker(3270): conn unix:auth-worker (pid=27359,uid=97): auth-worker<128>: pam(webapps,192.0.2.7): pam_authenticate() failed: Authentication failure (Password mismatch?) + # failJSON: { "time": "2005-01-29T14:38:51", "match": true , "host": "192.0.2.6", "desc": "PAM Permission denied (gh-1897)" } Jan 29 14:38:51 example.com dovecot[24941]: auth-worker(30165): pam(user@example.com,192.0.2.6,): pam_authenticate() failed: Permission denied From 8013cf0b900f2cacfdc1c9152c9b9847bfc41877 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 8 Feb 2022 19:57:40 +0100 Subject: [PATCH 191/240] python actions have no attribute 'consistencyCheck' by default; closes gh-3214 --- fail2ban/server/actions.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/fail2ban/server/actions.py b/fail2ban/server/actions.py index 897d907c..eb14c14f 100644 --- a/fail2ban/server/actions.py +++ b/fail2ban/server/actions.py @@ -513,9 +513,10 @@ class Actions(JailThread, Mapping): if bTicket.banEpoch == self.banEpoch and diftm > 3: # avoid too often checks: if not rebanacts and MyTime.time() > self.__lastConsistencyCheckTM + 3: - for action in self._actions.itervalues(): - action.consistencyCheck() self.__lastConsistencyCheckTM = MyTime.time() + for action in self._actions.itervalues(): + if hasattr(action, 'consistencyCheck'): + action.consistencyCheck() # check epoch in order to reban it: if bTicket.banEpoch < self.banEpoch: if not rebanacts: rebanacts = dict( From 498e473a10ee56aa6345b03cd3bf83e017df966c Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 9 Feb 2022 12:18:23 +0100 Subject: [PATCH 192/240] filter.d/courier-auth.conf: consider optional port after IP, regex is rewritten without catch-all's and right anchor, so it is more stable against further modifications now; closes #3211 --- config/filter.d/courier-auth.conf | 2 +- fail2ban/tests/files/logs/courier-auth | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/config/filter.d/courier-auth.conf b/config/filter.d/courier-auth.conf index 1ac33736..d5ba9c50 100644 --- a/config/filter.d/courier-auth.conf +++ b/config/filter.d/courier-auth.conf @@ -11,7 +11,7 @@ before = common.conf _daemon = (?:courier)?(?:imapd?|pop3d?)(?:login)?(?:-ssl)? -failregex = ^%(__prefix_line)sLOGIN FAILED, (?:user|method)=.*, ip=\[\]$ +failregex = ^%(__prefix_line)sLOGIN FAILED, (?:(?!ip=)(?:user=[^,]*|\w+=[^,]*), )*ip=\[\] ignoreregex = diff --git a/fail2ban/tests/files/logs/courier-auth b/fail2ban/tests/files/logs/courier-auth index 3505e109..8a20a27f 100644 --- a/fail2ban/tests/files/logs/courier-auth +++ b/fail2ban/tests/files/logs/courier-auth @@ -8,3 +8,5 @@ Nov 13 08:11:53 server imapd-ssl: LOGIN FAILED, user=user@domain.tld, ip=[::ffff Apr 17 19:17:11 SERVER courierpop3login: LOGIN FAILED, user=USER@EXAMPLE.org, ip=[::ffff:1.2.3.4] # failJSON: { "time": "2005-04-17T19:17:12", "match": true , "host": "192.0.2.4" } Apr 17 19:17:12 server imapd-ssl: LOGIN FAILED, method=PLAIN, ip=[::ffff:192.0.2.4] +# failJSON: { "time": "2005-04-27T09:00:00", "match": true , "user": "tester", "host": "192.0.2.5" } +Apr 27 09:00:00 servername imapd: LOGIN FAILED, user=tester, ip=[::ffff:192.0.2.5], port=[255] From cdb6a46945455ac49bce7efda25046140babcf9c Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 9 Feb 2022 14:47:40 +0100 Subject: [PATCH 193/240] systemd backend: better avoidance of landing in dead space by seeks over journals; increase verbosity and stability of few systemd tests (fixes sporadic timing issues); seekToTime doesn't need to convert float to datetime, because seek_realtime accepts it as unix time (we need to convert integers only, since it means microseconds and deprecated); --- fail2ban/server/filtersystemd.py | 21 +++++++++++---------- fail2ban/tests/filtertestcase.py | 30 +++++++++++++++++++----------- 2 files changed, 30 insertions(+), 21 deletions(-) diff --git a/fail2ban/server/filtersystemd.py b/fail2ban/server/filtersystemd.py index 6301b93a..8c67eedc 100644 --- a/fail2ban/server/filtersystemd.py +++ b/fail2ban/server/filtersystemd.py @@ -22,7 +22,6 @@ __author__ = "Steven Hiscocks" __copyright__ = "Copyright (c) 2013 Steven Hiscocks" __license__ = "GPL" -import datetime import os import time from distutils.version import LooseVersion @@ -254,8 +253,8 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover return ((logline[:0], date[0], logline.replace('\n', '\\n')), date[1]) def seekToTime(self, date): - if not isinstance(date, datetime.datetime): - date = datetime.datetime.fromtimestamp(date) + if isinstance(date, (int, long)): + date = float(date) self.__journal.seek_realtime(date) def inOperationMode(self): @@ -281,7 +280,8 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover try: self.__journal.seek_tail() logentry = self.__journal.get_previous() - self.__journal.get_next() + if logentry: + self.__journal.get_next() except OSError: logentry = None # Reading failure, so safe to ignore if logentry: @@ -296,12 +296,6 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover self.inOperation = False # Save current time in order to check time to switch "in operation" mode startTime = (1, MyTime.time(), logentry.get('__CURSOR')) - # Move back one entry to ensure do not end up in dead space - # if start time beyond end of journal - try: - self.__journal.get_previous() - except OSError: - pass # Reading failure, so safe to ignore else: # empty journal or no entries for current filter: self.inOperationMode() @@ -311,6 +305,13 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover # for possible future switches of in-operation mode: startTime = (0, startTime) + # Move back one entry to ensure do not end up in dead space + # if start time beyond end of journal + try: + self.__journal.get_previous() + except OSError: + pass # Reading failure, so safe to ignore + line = None while self.active: # wait for records (or for timeout in sleeptime seconds): diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index f9fa5e97..a9641eee 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -1514,7 +1514,7 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover # stop: self.filter.stop() self.filter.join() - MyTime.setTime(time.time() + 2) + MyTime.setTime(time.time() + 10) # update log manually (should cause a seek to end of log without wait for next second): self.jail.database.updateJournal(self.jail, 'systemd-journal', MyTime.time(), 'TEST') # check seek to last (simulated) position succeeds (without bans of previous copied tickets): @@ -1522,7 +1522,7 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover self._initFilter() self.filter.setMaxRetry(1) self.filter.start() - self.waitForTicks(1) + self.waitForTicks(2) # check new IP but no old IPs found: _gen_falure("192.0.2.5") self.assertFalse(self.jail.getFailTicket()) @@ -1535,8 +1535,8 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover self._initFilter() self.filter.setMaxRetry(1) self.filter.start() - self.waitForTicks(1) - MyTime.setTime(time.time() + 3) + self.waitForTicks(2) + MyTime.setTime(time.time() + 20) # check new IP but no old IPs found: _gen_falure("192.0.2.6") self.assertFalse(self.jail.getFailTicket()) @@ -1549,15 +1549,23 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover self.filter.setMaxRetry(1) states = [] def _state(*args): - self.assertNotIn("** in operation", states) - self.assertFalse(self.filter.inOperation) - states.append("** process line: %r" % (args,)) + try: + self.assertNotIn("** in operation", states) + self.assertFalse(self.filter.inOperation) + states.append("** process line: %r" % (args,)) + except Exception as e: + states.append("** failed: %r" % (e,)) + raise self.filter.processLineAndAdd = _state def _inoper(): - self.assertNotIn("** in operation", states) - self.assertEqual(len(states), 11) - states.append("** in operation") - self.filter.__class__.inOperationMode(self.filter) + try: + self.assertNotIn("** in operation", states) + self.assertEqual(len(states), 11) + states.append("** in operation") + self.filter.__class__.inOperationMode(self.filter) + except Exception as e: + states.append("** failed: %r" % (e,)) + raise self.filter.inOperationMode = _inoper self.filter.start() self.waitForTicks(12) From f380d6202d3760b3fbb718b5296061beec1787d0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A1szl=C3=B3=20K=C3=A1rolyi?= Date: Thu, 27 Jan 2022 12:28:20 +0100 Subject: [PATCH 194/240] cherry pick #3210 from master --- config/filter.d/dovecot.conf | 2 +- fail2ban/tests/files/logs/dovecot | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/config/filter.d/dovecot.conf b/config/filter.d/dovecot.conf index b1df82f2..0415ecb4 100644 --- a/config/filter.d/dovecot.conf +++ b/config/filter.d/dovecot.conf @@ -14,7 +14,7 @@ _daemon = (?:dovecot(?:-auth)?|auth) prefregex = ^%(__prefix_line)s(?:%(_auth_worker)s(?:\([^\)]+\))?: )?(?:%(__pam_auth)s(?:\(dovecot:auth\))?: |(?:pop3|imap|managesieve|submission)-login: )?(?:Info: )?%(_auth_worker_info)s.+$ failregex = ^authentication failure; logname=\S* uid=\S* euid=\S* tty=dovecot ruser=\S* rhost=(?:\s+user=\S*)?\s*$ - ^(?:Aborted login|Disconnected|Remote closed connection|Client has quit the connection)(?::(?: [^ \(]+)+)? \((?:auth failed, \d+ attempts(?: in \d+ secs)?|tried to use (?:disabled|disallowed) \S+ auth|proxy dest auth failed)\):(?: user=<[^>]*>,)?(?: method=\S+,)? rip=(?:[^>]*(?:, session=<\S+>)?)\s*$ + ^(?:Aborted login|Disconnected|Remote closed connection|Client has quit the connection)(?:: (?:[^\(]+|\w+\([^\)]*\))+)? \((?:auth failed, \d+ attempts(?: in \d+ secs)?|tried to use (?:disabled|disallowed) \S+ auth|proxy dest auth failed)\):(?: user=<[^>]*>,)?(?: method=\S+,)? rip=(?:[^>]*(?:, session=<\S+>)?)\s*$ ^pam\(\S+,(?:,\S*)?\): pam_authenticate\(\) failed: (?:User not known to the underlying authentication module: \d+ Time\(s\)|Authentication failure \([Pp]assword mismatch\?\)|Permission denied)\s*$ ^[a-z\-]{3,15}\(\S*,(?:,\S*)?\): (?:[Uu]nknown user|[Ii]nvalid credentials|[Pp]assword mismatch) > diff --git a/fail2ban/tests/files/logs/dovecot b/fail2ban/tests/files/logs/dovecot index 7649cfeb..75934c37 100644 --- a/fail2ban/tests/files/logs/dovecot +++ b/fail2ban/tests/files/logs/dovecot @@ -112,6 +112,8 @@ Jul 26 11:12:19 hostname dovecot: imap-login: Disconnected: Too many invalid com # failJSON: { "time": "2004-08-28T06:38:51", "match": true , "host": "192.0.2.3" } Aug 28 06:38:51 s166-62-100-187 dovecot: imap-login: Disconnected (auth failed, 1 attempts in 9 secs): user=, method=PLAIN, rip=192.0.2.3, lip=192.168.1.2, TLS: Disconnected, TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits) +# failJSON: { "time": "2004-08-28T06:38:52", "match": true , "host": "192.0.2.4", "desc": "open parenthesis in optional part between Disconnected and (auth failed ...), gh-3210" } +Aug 28 06:38:52 s166-62-100-187 dovecot: imap-login: Disconnected: Connection closed: read(size=1003) failed: Connection reset by peer (auth failed, 1 attempts in 0 secs): user=, rip=192.0.2.4, lip=127.0.0.19, session= # failJSON: { "time": "2004-08-29T03:17:18", "match": true , "host": "192.0.2.133" } Aug 29 03:17:18 server dovecot: submission-login: Client has quit the connection (auth failed, 1 attempts in 2 secs): user=, method=LOGIN, rip=192.0.2.133, lip=0.0.0.0 From a2431158f670867e65534674b640cd0672a85e49 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 9 Feb 2022 17:10:19 +0100 Subject: [PATCH 195/240] implements new interpolation variable `%(fail2ban_confpath)s` (automatically substituted from config-reader path, default `/etc/fail2ban` or `/usr/local/etc/fail2ban` depending on distribution); `ignorecommands_dir` is unneeded anymore, thus removed from `paths-common.conf`; fixes gh-3005 --- config/jail.conf | 2 +- config/paths-common.conf | 3 --- fail2ban/client/jailreader.py | 11 +++++++---- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/config/jail.conf b/config/jail.conf index e827167b..fe8db527 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -346,7 +346,7 @@ maxretry = 2 port = http,https logpath = %(apache_access_log)s maxretry = 1 -ignorecommand = %(ignorecommands_dir)s/apache-fakegooglebot +ignorecommand = %(fail2ban_confpath)s/filter.d/ignorecommands/apache-fakegooglebot [apache-modsecurity] diff --git a/config/paths-common.conf b/config/paths-common.conf index 7383cafe..4f6a5f71 100644 --- a/config/paths-common.conf +++ b/config/paths-common.conf @@ -91,6 +91,3 @@ mysql_log = %(syslog_daemon)s mysql_backend = %(default_backend)s roundcube_errors_log = /var/log/roundcube/errors - -# Directory with ignorecommand scripts -ignorecommands_dir = /etc/fail2ban/filter.d/ignorecommands diff --git a/fail2ban/client/jailreader.py b/fail2ban/client/jailreader.py index f3ccf7db..37746d4c 100644 --- a/fail2ban/client/jailreader.py +++ b/fail2ban/client/jailreader.py @@ -121,9 +121,12 @@ class JailReader(ConfigReader): def getOptions(self): + basedir = self.getBaseDir() + # Before interpolation (substitution) add static options always available as default: self.merge_defaults({ - "fail2ban_version": version + "fail2ban_version": version, + "fail2ban_confpath": basedir }) try: @@ -146,7 +149,7 @@ class JailReader(ConfigReader): raise JailDefError("Invalid filter definition %r: %s" % (flt, e)) self.__filter = FilterReader( filterName, self.__name, filterOpt, - share_config=self.share_config, basedir=self.getBaseDir()) + share_config=self.share_config, basedir=basedir) ret = self.__filter.read() if not ret: raise JailDefError("Unable to read the filter %r" % filterName) @@ -186,13 +189,13 @@ class JailReader(ConfigReader): "addaction", actOpt.pop("actname", os.path.splitext(actName)[0]), os.path.join( - self.getBaseDir(), "action.d", actName), + basedir, "action.d", actName), json.dumps(actOpt), ]) else: action = ActionReader( actName, self.__name, actOpt, - share_config=self.share_config, basedir=self.getBaseDir()) + share_config=self.share_config, basedir=basedir) ret = action.read() if ret: action.getOptions(self.__opts) From a98c4218c11938c18960e4487bee8ed43bdd7a44 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Wed, 9 Feb 2022 17:34:51 +0100 Subject: [PATCH 196/240] Create FUNDING.yml --- .github/FUNDING.yml | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 .github/FUNDING.yml diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml new file mode 100644 index 00000000..a8824045 --- /dev/null +++ b/.github/FUNDING.yml @@ -0,0 +1,15 @@ +# These are supported funding model platforms + +# Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2] +github: [sebres] + +#patreon: # Replace with a single Patreon username +#open_collective: # Replace with a single Open Collective username +#ko_fi: # Replace with a single Ko-fi username +#tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel +#community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry +#liberapay: # Replace with a single Liberapay username +#issuehunt: # Replace with a single IssueHunt username +#otechie: # Replace with a single Otechie username +#lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry +#custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2'] From 5bfd9992b470b724463ace3b807bf9cc43bea42b Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Wed, 9 Feb 2022 17:50:35 +0100 Subject: [PATCH 197/240] Update FUNDING.yml --- .github/FUNDING.yml | 13 +------------ 1 file changed, 1 insertion(+), 12 deletions(-) diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml index a8824045..543f316a 100644 --- a/.github/FUNDING.yml +++ b/.github/FUNDING.yml @@ -1,15 +1,4 @@ # These are supported funding model platforms -# Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2] github: [sebres] - -#patreon: # Replace with a single Patreon username -#open_collective: # Replace with a single Open Collective username -#ko_fi: # Replace with a single Ko-fi username -#tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel -#community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry -#liberapay: # Replace with a single Liberapay username -#issuehunt: # Replace with a single IssueHunt username -#otechie: # Replace with a single Otechie username -#lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry -#custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2'] +custom: [paypal.me/sebres] From ff7fe572bf90010ee40c9d9966e7f5ef3ee4a145 Mon Sep 17 00:00:00 2001 From: sebres Date: Thu, 10 Feb 2022 15:40:01 +0100 Subject: [PATCH 198/240] drop support for python 2.6 (hardly possible in modern CIs, new features would expect OrderedDicts, etc) --- .travis.yml | 3 +- ChangeLog | 2 + README.md | 2 +- fail2ban/server/actions.py | 5 +- fail2ban/server/server.py | 4 +- fail2ban/server/utils.py | 30 ++------ fail2ban/tests/actiontestcase.py | 108 ++++++++++++++--------------- fail2ban/tests/misctestcase.py | 16 ++--- fail2ban/tests/observertestcase.py | 2 +- fail2ban/tests/servertestcase.py | 26 ++----- 10 files changed, 78 insertions(+), 120 deletions(-) diff --git a/.travis.yml b/.travis.yml index 398c120a..502af5be 100644 --- a/.travis.yml +++ b/.travis.yml @@ -33,7 +33,8 @@ install: # coverage - travis_retry pip install coverage # coveralls (note coveralls doesn't support 2.6 now): - - if [[ $TRAVIS_PYTHON_VERSION != 2.6* ]]; then F2B_COV=1; else F2B_COV=0; fi + #- if [[ $TRAVIS_PYTHON_VERSION != 2.6* ]]; then F2B_COV=1; else F2B_COV=0; fi + - F2B_COV=1 - if [[ "$F2B_COV" = 1 ]]; then travis_retry pip install coveralls; fi # codecov: - travis_retry pip install codecov diff --git a/ChangeLog b/ChangeLog index a022f17c..d294ad54 100644 --- a/ChangeLog +++ b/ChangeLog @@ -10,6 +10,8 @@ ver. 1.0.1-dev-1 (20??/??/??) - development nightly edition ----------- ### Compatibility: +* the minimum supported python version is now 2.7, if you have previous python version + you can use the 0.11 version of fail2ban or upgrade python (or even build it from source). * potential incompatibility by parsing of options of `backend`, `filter` and `action` parameters (if they are partially incorrect), because fail2ban could throw an error now (doesn't silently bypass it anymore). * to v.0.11: diff --git a/README.md b/README.md index ba5c149e..6bf94c25 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ Installation: this case, you should use that instead.** Required: -- [Python2 >= 2.6 or Python >= 3.2](https://www.python.org) or [PyPy](https://pypy.org) +- [Python2 >= 2.7 or Python >= 3.2](https://www.python.org) or [PyPy](https://pypy.org) - python-setuptools, python-distutils or python3-setuptools for installation from source Optional: diff --git a/fail2ban/server/actions.py b/fail2ban/server/actions.py index 54e0c02a..0cd2ec33 100644 --- a/fail2ban/server/actions.py +++ b/fail2ban/server/actions.py @@ -32,10 +32,7 @@ try: from collections.abc import Mapping except ImportError: from collections import Mapping -try: - from collections import OrderedDict -except ImportError: - OrderedDict = dict +from collections import OrderedDict from .banmanager import BanManager, BanTicket from .ipdns import IPAddr diff --git a/fail2ban/server/server.py b/fail2ban/server/server.py index 3853bbc4..36ed1b0d 100644 --- a/fail2ban/server/server.py +++ b/fail2ban/server/server.py @@ -728,9 +728,7 @@ class Server: except (ValueError, KeyError): # pragma: no cover # Is known to be thrown after logging was shutdown once # with older Pythons -- seems to be safe to ignore there - # At least it was still failing on 2.6.2-0ubuntu1 (jaunty) - if (2, 6, 3) <= sys.version_info < (3,) or \ - (3, 2) <= sys.version_info: + if sys.version_info < (3,) or sys.version_info >= (3, 2): raise # detailed format by deep log levels (as DEBUG=10): if logger.getEffectiveLevel() <= logging.DEBUG: # pragma: no cover diff --git a/fail2ban/server/utils.py b/fail2ban/server/utils.py index 8483b013..18073ea7 100644 --- a/fail2ban/server/utils.py +++ b/fail2ban/server/utils.py @@ -30,11 +30,7 @@ import sys from threading import Lock import time from ..helpers import getLogger, _merge_dicts, uni_decode - -try: - from collections import OrderedDict -except ImportError: # pragma: 3.x no cover - OrderedDict = dict +from collections import OrderedDict if sys.version_info >= (3, 3): import importlib.machinery @@ -100,24 +96,12 @@ class Utils(): with self.__lock: # clean cache if max count reached: if len(cache) >= self.maxCount: - if OrderedDict is not dict: - # ordered (so remove some from ahead, FIFO) - while cache: - (ck, cv) = cache.popitem(last=False) - # if not yet expired (but has free slot for new entry): - if cv[1] > t and len(cache) < self.maxCount: - break - else: # pragma: 3.x no cover (dict is in 2.6 only) - remlst = [] - for (ck, cv) in cache.iteritems(): - # if expired: - if cv[1] <= t: - remlst.append(ck) - for ck in remlst: - self._cache.pop(ck, None) - # if still max count - remove any one: - while cache and len(cache) >= self.maxCount: - cache.popitem() + # ordered (so remove some from ahead, FIFO) + while cache: + (ck, cv) = cache.popitem(last=False) + # if not yet expired (but has free slot for new entry): + if cv[1] > t and len(cache) < self.maxCount: + break # set now: cache[k] = (v, t + self.maxTime) diff --git a/fail2ban/tests/actiontestcase.py b/fail2ban/tests/actiontestcase.py index 64476f56..ce5de483 100644 --- a/fail2ban/tests/actiontestcase.py +++ b/fail2ban/tests/actiontestcase.py @@ -75,61 +75,59 @@ class CommandActionTest(LogCaptureTestCase): lambda: substituteRecursiveTags({'A': 'to= fromip=', 'C': '', 'B': '', 'D': ''})) self.assertRaises(ValueError, lambda: substituteRecursiveTags({'failregex': 'to= fromip=', 'sweet': '', 'honeypot': '', 'ignoreregex': ''})) - # We need here an ordered, because the sequence of iteration is very important for this test - if OrderedDict: - # No cyclic recursion, just multiple replacement of tag , should be successful: - self.assertEqual(substituteRecursiveTags( OrderedDict( - (('X', 'x=x'), ('T', '1'), ('Z', ' '), ('Y', 'y=y'))) - ), {'X': 'x=x1', 'T': '1', 'Y': 'y=y1', 'Z': 'x=x1 1 y=y1'} - ) - # No cyclic recursion, just multiple replacement of tag in composite tags, should be successful: - self.assertEqual(substituteRecursiveTags( OrderedDict( - (('X', 'x=x <> <>'), ('R1', 'Z'), ('R2', 'Y'), ('T', '1'), ('Z', ' '), ('Y', 'y=y'))) - ), {'X': 'x=x1 1 y=y1 1 y=y1 y=y1', 'R1': 'Z', 'R2': 'Y', 'T': '1', 'Z': '1 y=y1', 'Y': 'y=y1'} - ) - # No cyclic recursion, just multiple replacement of same tags, should be successful: - self.assertEqual(substituteRecursiveTags( OrderedDict(( - ('actionstart', 'ipset create hash:ip timeout family \n -I '), - ('ipmset', 'f2b-'), - ('name', 'any'), - ('bantime', '600'), - ('ipsetfamily', 'inet'), - ('iptables', 'iptables '), - ('lockingopt', '-w'), - ('chain', 'INPUT'), - ('actiontype', ''), - ('multiport', '-p -m multiport --dports -m set --match-set src -j '), - ('protocol', 'tcp'), - ('port', 'ssh'), - ('blocktype', 'REJECT',), - )) - ), OrderedDict(( - ('actionstart', 'ipset create f2b-any hash:ip timeout 600 family inet\niptables -w -I INPUT -p tcp -m multiport --dports ssh -m set --match-set f2b-any src -j REJECT'), - ('ipmset', 'f2b-any'), - ('name', 'any'), - ('bantime', '600'), - ('ipsetfamily', 'inet'), - ('iptables', 'iptables -w'), - ('lockingopt', '-w'), - ('chain', 'INPUT'), - ('actiontype', '-p tcp -m multiport --dports ssh -m set --match-set f2b-any src -j REJECT'), - ('multiport', '-p tcp -m multiport --dports ssh -m set --match-set f2b-any src -j REJECT'), - ('protocol', 'tcp'), - ('port', 'ssh'), - ('blocktype', 'REJECT') - )) - ) - # Cyclic recursion by composite tag creation, tags "create" another tag, that closes cycle: - self.assertRaises(ValueError, lambda: substituteRecursiveTags( OrderedDict(( - ('A', '<>'), - ('B', 'D'), ('C', 'E'), - ('DE', 'cycle '), - )) )) - self.assertRaises(ValueError, lambda: substituteRecursiveTags( OrderedDict(( - ('DE', 'cycle '), - ('A', '<>'), - ('B', 'D'), ('C', 'E'), - )) )) + # No cyclic recursion, just multiple replacement of tag , should be successful: + self.assertEqual(substituteRecursiveTags( OrderedDict( + (('X', 'x=x'), ('T', '1'), ('Z', ' '), ('Y', 'y=y'))) + ), {'X': 'x=x1', 'T': '1', 'Y': 'y=y1', 'Z': 'x=x1 1 y=y1'} + ) + # No cyclic recursion, just multiple replacement of tag in composite tags, should be successful: + self.assertEqual(substituteRecursiveTags( OrderedDict( + (('X', 'x=x <> <>'), ('R1', 'Z'), ('R2', 'Y'), ('T', '1'), ('Z', ' '), ('Y', 'y=y'))) + ), {'X': 'x=x1 1 y=y1 1 y=y1 y=y1', 'R1': 'Z', 'R2': 'Y', 'T': '1', 'Z': '1 y=y1', 'Y': 'y=y1'} + ) + # No cyclic recursion, just multiple replacement of same tags, should be successful: + self.assertEqual(substituteRecursiveTags( OrderedDict(( + ('actionstart', 'ipset create hash:ip timeout family \n -I '), + ('ipmset', 'f2b-'), + ('name', 'any'), + ('bantime', '600'), + ('ipsetfamily', 'inet'), + ('iptables', 'iptables '), + ('lockingopt', '-w'), + ('chain', 'INPUT'), + ('actiontype', ''), + ('multiport', '-p -m multiport --dports -m set --match-set src -j '), + ('protocol', 'tcp'), + ('port', 'ssh'), + ('blocktype', 'REJECT',), + )) + ), OrderedDict(( + ('actionstart', 'ipset create f2b-any hash:ip timeout 600 family inet\niptables -w -I INPUT -p tcp -m multiport --dports ssh -m set --match-set f2b-any src -j REJECT'), + ('ipmset', 'f2b-any'), + ('name', 'any'), + ('bantime', '600'), + ('ipsetfamily', 'inet'), + ('iptables', 'iptables -w'), + ('lockingopt', '-w'), + ('chain', 'INPUT'), + ('actiontype', '-p tcp -m multiport --dports ssh -m set --match-set f2b-any src -j REJECT'), + ('multiport', '-p tcp -m multiport --dports ssh -m set --match-set f2b-any src -j REJECT'), + ('protocol', 'tcp'), + ('port', 'ssh'), + ('blocktype', 'REJECT') + )) + ) + # Cyclic recursion by composite tag creation, tags "create" another tag, that closes cycle: + self.assertRaises(ValueError, lambda: substituteRecursiveTags( OrderedDict(( + ('A', '<>'), + ('B', 'D'), ('C', 'E'), + ('DE', 'cycle '), + )) )) + self.assertRaises(ValueError, lambda: substituteRecursiveTags( OrderedDict(( + ('DE', 'cycle '), + ('A', '<>'), + ('B', 'D'), ('C', 'E'), + )) )) # missing tags are ok self.assertEqual(substituteRecursiveTags({'A': ''}), {'A': ''}) diff --git a/fail2ban/tests/misctestcase.py b/fail2ban/tests/misctestcase.py index e2faa6fd..4b026377 100644 --- a/fail2ban/tests/misctestcase.py +++ b/fail2ban/tests/misctestcase.py @@ -70,16 +70,10 @@ class HelpersTest(unittest.TestCase): self.assertEqual(splitwords(u' 1\n 2, 3'), ['1', '2', '3']) -if sys.version_info >= (2,7): - def _sh_call(cmd): - import subprocess - ret = subprocess.check_output(cmd, shell=True) - return uni_decode(ret).rstrip() -else: - def _sh_call(cmd): - import subprocess - ret = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout.read() - return uni_decode(ret).rstrip() +def _sh_call(cmd): + import subprocess + ret = subprocess.check_output(cmd, shell=True) + return uni_decode(ret).rstrip() def _getSysPythonVersion(): return _sh_call("fail2ban-python -c 'import sys; print(tuple(sys.version_info))'") @@ -92,7 +86,7 @@ class SetupTest(unittest.TestCase): unittest.F2B.SkipIfFast() setup = os.path.join(os.path.dirname(__file__), '..', '..', 'setup.py') self.setup = os.path.exists(setup) and setup or None - if not self.setup and sys.version_info >= (2,7): # pragma: no cover - running not out of the source + if not self.setup: # pragma: no cover - running not out of the source raise unittest.SkipTest( "Seems to be running not out of source distribution" " -- cannot locate setup.py") diff --git a/fail2ban/tests/observertestcase.py b/fail2ban/tests/observertestcase.py index 38cfc881..315c955a 100644 --- a/fail2ban/tests/observertestcase.py +++ b/fail2ban/tests/observertestcase.py @@ -181,7 +181,7 @@ class BanTimeIncrDB(LogCaptureTestCase): def setUp(self): """Call before every test case.""" super(BanTimeIncrDB, self).setUp() - if Fail2BanDb is None and sys.version_info >= (2,7): # pragma: no cover + if Fail2BanDb is None: # pragma: no cover raise unittest.SkipTest( "Unable to import fail2ban database module as sqlite is not " "available.") diff --git a/fail2ban/tests/servertestcase.py b/fail2ban/tests/servertestcase.py index b7ad4d32..62ae81fd 100644 --- a/fail2ban/tests/servertestcase.py +++ b/fail2ban/tests/servertestcase.py @@ -775,27 +775,11 @@ class Transmitter(TransmitterBase): def testPythonActionMethodsAndProperties(self): action = "TestCaseAction" - try: - out = self.transm.proceed( - ["set", self.jailName, "addaction", action, - os.path.join(TEST_FILES_DIR, "action.d", "action.py"), - '{"opt1": "value"}']) - self.assertEqual(out, (0, action)) - except AssertionError: - if ((2, 6) <= sys.version_info < (2, 6, 5)) \ - and '__init__() keywords must be strings' in out[1]: - # known issue http://bugs.python.org/issue2646 in 2.6 series - # since general Fail2Ban warnings are suppressed in normal - # operation -- let's issue Python's native warning here - import warnings - warnings.warn( - "Your version of Python %s seems to experience a known " - "issue forbidding correct operation of Fail2Ban: " - "http://bugs.python.org/issue2646 Upgrade your Python and " - "meanwhile other intestPythonActionMethodsAndProperties will " - "be skipped" % (sys.version)) - return - raise + out = self.transm.proceed( + ["set", self.jailName, "addaction", action, + os.path.join(TEST_FILES_DIR, "action.d", "action.py"), + '{"opt1": "value"}']) + self.assertEqual(out, (0, action)) self.assertSortedEqual( self.transm.proceed(["get", self.jailName, "actionproperties", action])[1], From 8b11c89ed4494215f8e117b946fdd2d4eb4541d7 Mon Sep 17 00:00:00 2001 From: sebres Date: Thu, 10 Feb 2022 17:04:47 +0100 Subject: [PATCH 199/240] amend to drop support of python 2.6 --- fail2ban/tests/utils.py | 17 ----------------- 1 file changed, 17 deletions(-) diff --git a/fail2ban/tests/utils.py b/fail2ban/tests/utils.py index e674ee9b..8bcc1431 100644 --- a/fail2ban/tests/utils.py +++ b/fail2ban/tests/utils.py @@ -255,23 +255,6 @@ def with_alt_time(f): return wrapper -# backwards compatibility to python 2.6: -if not hasattr(unittest, 'SkipTest'): # pragma: no cover - class SkipTest(Exception): - pass - unittest.SkipTest = SkipTest - _org_AddError = unittest._TextTestResult.addError - def addError(self, test, err): - if err[0] is SkipTest: - if self.showAll: - self.stream.writeln(str(err[1])) - elif self.dots: - self.stream.write('s') - self.stream.flush() - return - _org_AddError(self, test, err) - unittest._TextTestResult.addError = addError - def initTests(opts): ## if running from installer (setup.py): if not opts: From d17e61ed5bc95735b06f30855873fc15b7a98ac9 Mon Sep 17 00:00:00 2001 From: John Kristensen Date: Fri, 11 Feb 2022 17:33:07 +1100 Subject: [PATCH 200/240] Add missing assert in Fail2banRegexTest.testFrmtOutput There was no associated `assertLogged()` for the "multiple id combined to a tuple" test so nothing was actually being tested. --- fail2ban/tests/fail2banregextestcase.py | 1 + 1 file changed, 1 insertion(+) diff --git a/fail2ban/tests/fail2banregextestcase.py b/fail2ban/tests/fail2banregextestcase.py index 26b40abb..b526dbf5 100644 --- a/fail2ban/tests/fail2banregextestcase.py +++ b/fail2ban/tests/fail2banregextestcase.py @@ -364,6 +364,7 @@ class Fail2banRegexTest(LogCaptureTestCase): self.assertTrue(_test_exec('-o', 'id', '1591983743.667 left 192.0.2.3 right', r'^\s*\S+ \S+')) + self.assertLogged(str(('192.0.2.3', 'left', 'right'))) self.pruneLog() # id had higher precedence as ip-address: self.assertTrue(_test_exec('-o', 'id', From 96121830da3ed271b121c3edddca0153ac0cfe10 Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 11 Feb 2022 19:10:26 +0100 Subject: [PATCH 201/240] differentiate and (): if IP-address deviates from ID then `` is not `` anymore; introduces certain backwards incompatibility against actions that have used tag `` to get failure-ID, if IP-related tags (like `` or ``) used additionally to `` and they are different, see gh-3217 --- fail2ban/client/fail2banregex.py | 6 ++- fail2ban/server/actions.py | 10 ++-- fail2ban/server/database.py | 2 +- fail2ban/server/filter.py | 67 ++++++++++++------------- fail2ban/server/ipdns.py | 2 + fail2ban/server/observer.py | 8 +-- fail2ban/server/ticket.py | 17 +++---- fail2ban/tests/banmanagertestcase.py | 10 ++-- fail2ban/tests/databasetestcase.py | 20 ++++---- fail2ban/tests/fail2banregextestcase.py | 12 +++++ fail2ban/tests/failmanagertestcase.py | 8 +-- fail2ban/tests/filtertestcase.py | 6 +-- fail2ban/tests/observertestcase.py | 4 +- fail2ban/tests/tickettestcase.py | 2 + 14 files changed, 95 insertions(+), 79 deletions(-) diff --git a/fail2ban/client/fail2banregex.py b/fail2ban/client/fail2banregex.py index 5921dfdd..4ab8a8a3 100644 --- a/fail2ban/client/fail2banregex.py +++ b/fail2ban/client/fail2banregex.py @@ -520,10 +520,14 @@ class Fail2banRegex(object): def _prepaireOutput(self): """Prepares output- and fetch-function corresponding given '--out' option (format)""" ofmt = self._opts.out - if ofmt in ('id', 'ip'): + if ofmt in ('id', 'fid'): def _out(ret): for r in ret: output(r[1]) + elif ofmt == 'ip': + def _out(ret): + for r in ret: + output(r[3].get('ip', r[1])) elif ofmt == 'msg': def _out(ret): for r in ret: diff --git a/fail2ban/server/actions.py b/fail2ban/server/actions.py index 0cd2ec33..9b2aa049 100644 --- a/fail2ban/server/actions.py +++ b/fail2ban/server/actions.py @@ -497,7 +497,7 @@ class Actions(JailThread, Mapping): bTicket = BanTicket.wrap(ticket) btime = ticket.getBanTime(self.banManager.getBanTime()) - ip = bTicket.getIP() + ip = bTicket.getID() aInfo = self._getActionInfo(bTicket) reason = {} if self.banManager.addBanTicket(bTicket, reason=reason): @@ -575,10 +575,10 @@ class Actions(JailThread, Mapping): Ticket to reban """ actions = actions or self._actions - ip = ticket.getIP() + ip = ticket.getID() aInfo = self._getActionInfo(ticket) if log: - logSys.notice("[%s] Reban %s%s", self._jail.name, aInfo["ip"], (', action %r' % actions.keys()[0] if len(actions) == 1 else '')) + logSys.notice("[%s] Reban %s%s", self._jail.name, ip, (', action %r' % actions.keys()[0] if len(actions) == 1 else '')) for name, action in actions.iteritems(): try: logSys.debug("[%s] action %r: reban %s", self._jail.name, name, ip) @@ -703,10 +703,10 @@ class Actions(JailThread, Mapping): unbactions = self._actions else: unbactions = actions - ip = ticket.getIP() + ip = ticket.getID() aInfo = self._getActionInfo(ticket) if log: - logSys.notice("[%s] Unban %s", self._jail.name, aInfo["ip"]) + logSys.notice("[%s] Unban %s", self._jail.name, ip) for name, action in unbactions.iteritems(): try: logSys.debug("[%s] action %r: unban %s", self._jail.name, name, ip) diff --git a/fail2ban/server/database.py b/fail2ban/server/database.py index 86b0ea68..8a7abb3c 100644 --- a/fail2ban/server/database.py +++ b/fail2ban/server/database.py @@ -599,7 +599,7 @@ class Fail2BanDb(object): ticket : BanTicket Ticket of the ban to be added. """ - ip = str(ticket.getIP()) + ip = str(ticket.getID()) try: del self._bansMergedCache[(ip, jail)] except KeyError: diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index 5af3626a..86c6648e 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -553,7 +553,7 @@ class Filter(JailThread): ticket = None if isinstance(ip, FailTicket): ticket = ip - ip = ticket.getIP() + ip = ticket.getID() elif not isinstance(ip, IPAddr): ip = IPAddr(ip) return self._inIgnoreIPList(ip, ticket, log_ignore) @@ -702,10 +702,7 @@ class Filter(JailThread): """Processes the line for failures and populates failManager """ try: - for element in self.processLine(line, date): - ip = element[1] - unixTime = element[2] - fail = element[3] + for (_, ip, unixTime, fail) in self.processLine(line, date): logSys.debug("Processing line with time:%s and ip:%s", unixTime, ip) # ensure the time is not in the future, e. g. by some estimated (assumed) time: @@ -841,11 +838,9 @@ class Filter(JailThread): failList = list() ll = logSys.getEffectiveLevel() - returnRawHost = self.returnRawHost - cidr = IPAddr.CIDR_UNSPEC - if self.__useDns == "raw": - returnRawHost = True - cidr = IPAddr.CIDR_RAW + defcidr = IPAddr.CIDR_UNSPEC + if self.__useDns == "raw" or self.returnRawHost: + defcidr = IPAddr.CIDR_RAW if self.__lineBufferSize > 1: self.__lineBuffer.append(tupleLine) @@ -908,7 +903,8 @@ class Filter(JailThread): if not self.checkAllRegex or self.__lineBufferSize > 1: self.__lineBuffer, buf = failRegex.getUnmatchedTupleLines(), None # merge data if multi-line failure: - raw = returnRawHost + cidr = defcidr + raw = (defcidr == IPAddr.CIDR_RAW) if preGroups: currFail, fail = fail, preGroups.copy() fail.update(currFail) @@ -927,49 +923,50 @@ class Filter(JailThread): # failure-id: fid = fail.get('fid') # ip-address or host: - host = fail.get('ip4') - if host is not None: + ip = fail.get('ip4') + if ip is not None: cidr = int(fail.get('cidr') or IPAddr.FAM_IPv4) raw = True else: - host = fail.get('ip6') - if host is not None: + ip = fail.get('ip6') + if ip is not None: cidr = int(fail.get('cidr') or IPAddr.FAM_IPv6) raw = True - if host is None: - host = fail.get('dns') - if host is None: - # first try to check we have mlfid case (cache connection id): - if fid is None and mlfid is None: - # if no failure-id also (obscure case, wrong regex), throw error inside getFailID: - fid = failRegex.getFailID() - host = fid - cidr = IPAddr.CIDR_RAW - raw = True + else: + ip = fail.get('dns') + if ip is None: + # first try to check we have mlfid case (cache connection id): + if fid is None and mlfid is None: + # if no failure-id also (obscure case, wrong regex), throw error inside getFailID: + fid = failRegex.getFailID() + ip = fid + raw = True # if mlfid case (not failure): - if host is None: + if ip is None: if ll <= 7: logSys.log(7, "No failure-id by mlfid %r in regex %s: %s", mlfid, failRegexIndex, fail.get('mlfforget', "waiting for identifier")) fail['mlfpending'] = 1; # mark failure is pending if not self.checkAllRegex and self.ignorePending: return failList - ips = [None] + fids = [None] # if raw - add single ip or failure-id, # otherwise expand host to multiple ips using dns (or ignore it if not valid): elif raw: - ip = IPAddr(host, cidr) - # check host equal failure-id, if not - failure with complex id: - if fid is not None and fid != host: - ip = IPAddr(fid, IPAddr.CIDR_RAW) - ips = [ip] + # check ip/host equal failure-id, if not - failure with complex id: + if fid is None or fid == ip: + fid = IPAddr(ip, cidr) + else: + fail['ip'] = IPAddr(ip, cidr) + fid = IPAddr(fid, defcidr) + fids = [fid] # otherwise, try to use dns conversion: else: - ips = DNSUtils.textToIp(host, self.__useDns) + fids = DNSUtils.textToIp(ip, self.__useDns) # if checkAllRegex we must make a copy (to be sure next RE doesn't change merged/cached failure): if self.checkAllRegex and mlfid is not None: fail = fail.copy() # append failure with match to the list: - for ip in ips: - failList.append([failRegexIndex, ip, date, fail]) + for fid in fids: + failList.append([failRegexIndex, fid, date, fail]) if not self.checkAllRegex: break except RegexException as e: # pragma: no cover - unsure if reachable diff --git a/fail2ban/server/ipdns.py b/fail2ban/server/ipdns.py index d6dfbb9d..d917d031 100644 --- a/fail2ban/server/ipdns.py +++ b/fail2ban/server/ipdns.py @@ -257,6 +257,8 @@ class IPAddr(object): FAM_IPv6 = CIDR_RAW - socket.AF_INET6 def __new__(cls, ipstr, cidr=CIDR_UNSPEC): + if cidr == IPAddr.CIDR_UNSPEC and isinstance(ipstr, (tuple, list)): + cidr = IPAddr.CIDR_RAW if cidr == IPAddr.CIDR_RAW: # don't cache raw ip = super(IPAddr, cls).__new__(cls) ip.__init(ipstr, cidr) diff --git a/fail2ban/server/observer.py b/fail2ban/server/observer.py index ecbcd5b7..b1c9b37d 100644 --- a/fail2ban/server/observer.py +++ b/fail2ban/server/observer.py @@ -372,7 +372,7 @@ class ObserverThread(JailThread): # check jail active : if not jail.isAlive() or not jail.getBanTimeExtra("increment"): return - ip = ticket.getIP() + ip = ticket.getID() unixTime = ticket.getTime() logSys.debug("[%s] Observer: failure found %s", jail.name, ip) # increase retry count for known (bad) ip, corresponding banCount of it (one try will count than 2, 3, 5, 9 ...) : @@ -435,7 +435,7 @@ class ObserverThread(JailThread): if not jail.isAlive() or not jail.database: return banTime be = jail.getBanTimeExtra() - ip = ticket.getIP() + ip = ticket.getID() orgBanTime = banTime # check ip was already banned (increment time of ban): try: @@ -474,7 +474,7 @@ class ObserverThread(JailThread): return try: oldbtime = btime - ip = ticket.getIP() + ip = ticket.getID() logSys.debug("[%s] Observer: ban found %s, %s", jail.name, ip, btime) # if not permanent and ban time was not set - check time should be increased: if btime != -1 and ticket.getBanTime() is None: @@ -514,7 +514,7 @@ class ObserverThread(JailThread): """ try: btime = ticket.getBanTime() - ip = ticket.getIP() + ip = ticket.getID() logSys.debug("[%s] Observer: prolong %s, %s", jail.name, ip, btime) # prolong ticket via actions that expected this: jail.actions._prolongBan(ticket) diff --git a/fail2ban/server/ticket.py b/fail2ban/server/ticket.py index f99b6462..96e67773 100644 --- a/fail2ban/server/ticket.py +++ b/fail2ban/server/ticket.py @@ -33,7 +33,7 @@ logSys = getLogger(__name__) class Ticket(object): - __slots__ = ('_ip', '_flags', '_banCount', '_banTime', '_time', '_data', '_retry', '_lastReset') + __slots__ = ('_id', '_flags', '_banCount', '_banTime', '_time', '_data', '_retry', '_lastReset') MAX_TIME = 0X7FFFFFFFFFFF ;# 4461763-th year @@ -48,7 +48,7 @@ class Ticket(object): @param matches (log) lines caused the ticket """ - self.setIP(ip) + self.setID(ip) self._flags = 0; self._banCount = 0; self._banTime = None; @@ -65,7 +65,7 @@ class Ticket(object): def __str__(self): return "%s: ip=%s time=%s bantime=%s bancount=%s #attempts=%d matches=%r" % \ - (self.__class__.__name__.split('.')[-1], self._ip, self._time, + (self.__class__.__name__.split('.')[-1], self._id, self._time, self._banTime, self._banCount, self._data['failures'], self._data.get('matches', [])) @@ -74,7 +74,7 @@ class Ticket(object): def __eq__(self, other): try: - return self._ip == other._ip and \ + return self._id == other._id and \ round(self._time, 2) == round(other._time, 2) and \ self._data == other._data except AttributeError: @@ -86,18 +86,17 @@ class Ticket(object): if v is not None: setattr(self, n, v) - - def setIP(self, value): + def setID(self, value): # guarantee using IPAddr instead of unicode, str for the IP if isinstance(value, basestring): value = IPAddr(value) - self._ip = value + self._id = value def getID(self): - return self._data.get('fid', self._ip) + return self._id def getIP(self): - return self._ip + return self._data.get('ip', self._id) def setTime(self, value): self._time = value diff --git a/fail2ban/tests/banmanagertestcase.py b/fail2ban/tests/banmanagertestcase.py index ec8e6f9f..cf25ac0f 100644 --- a/fail2ban/tests/banmanagertestcase.py +++ b/fail2ban/tests/banmanagertestcase.py @@ -100,23 +100,23 @@ class AddFailure(unittest.TestCase): self.assertFalse(self.__banManager._inBanList(ticket)) def testBanTimeIncr(self): - ticket = BanTicket(self.__ticket.getIP(), self.__ticket.getTime()) + ticket = BanTicket(self.__ticket.getID(), self.__ticket.getTime()) ## increase twice and at end permanent, check time/count increase: c = 0 for i in (1000, 2000, -1): self.__banManager.addBanTicket(self.__ticket); c += 1 ticket.setBanTime(i) self.assertFalse(self.__banManager.addBanTicket(ticket)); # no incr of c (already banned) - self.assertEqual(str(self.__banManager.getTicketByID(ticket.getIP())), - "BanTicket: ip=%s time=%s bantime=%s bancount=%s #attempts=0 matches=[]" % (ticket.getIP(), ticket.getTime(), i, c)) + self.assertEqual(str(self.__banManager.getTicketByID(ticket.getID())), + "BanTicket: ip=%s time=%s bantime=%s bancount=%s #attempts=0 matches=[]" % (ticket.getID(), ticket.getTime(), i, c)) ## after permanent, it should remain permanent ban time (-1): self.__banManager.addBanTicket(self.__ticket); c += 1 ticket.setBanTime(-1) self.assertFalse(self.__banManager.addBanTicket(ticket)); # no incr of c (already banned) ticket.setBanTime(1000) self.assertFalse(self.__banManager.addBanTicket(ticket)); # no incr of c (already banned) - self.assertEqual(str(self.__banManager.getTicketByID(ticket.getIP())), - "BanTicket: ip=%s time=%s bantime=%s bancount=%s #attempts=0 matches=[]" % (ticket.getIP(), ticket.getTime(), -1, c)) + self.assertEqual(str(self.__banManager.getTicketByID(ticket.getID())), + "BanTicket: ip=%s time=%s bantime=%s bancount=%s #attempts=0 matches=[]" % (ticket.getID(), ticket.getTime(), -1, c)) def testUnban(self): btime = self.__banManager.getBanTime() diff --git a/fail2ban/tests/databasetestcase.py b/fail2ban/tests/databasetestcase.py index 6692b238..8cc394be 100644 --- a/fail2ban/tests/databasetestcase.py +++ b/fail2ban/tests/databasetestcase.py @@ -192,7 +192,7 @@ class DatabaseTest(LogCaptureTestCase): ticket.setAttempt(3) self.assertEqual(bans[0], ticket) # second ban found also: - self.assertEqual(bans[1].getIP(), "1.2.3.8") + self.assertEqual(bans[1].getID(), "1.2.3.8") # updated ? self.assertEqual(self.db.updateDb(Fail2BanDb.__version__), Fail2BanDb.__version__) # check current bans (should find 2 tickets after upgrade): @@ -312,7 +312,7 @@ class DatabaseTest(LogCaptureTestCase): for i, ticket in enumerate(tickets): DefLogSys.debug('readtickets[%d]: %r', i, readtickets[i].getData()) DefLogSys.debug(' == tickets[%d]: %r', i, ticket.getData()) - self.assertEqual(readtickets[i].getIP(), ticket.getIP()) + self.assertEqual(readtickets[i].getID(), ticket.getID()) self.assertEqual(len(readtickets[i].getMatches()), len(ticket.getMatches())) self.pruneLog('[test-phase 2] simulate errors') @@ -354,10 +354,10 @@ class DatabaseTest(LogCaptureTestCase): def testDelBan(self): tickets = self._testAdd3Bans() # delete single IP: - self.db.delBan(self.jail, tickets[0].getIP()) + self.db.delBan(self.jail, tickets[0].getID()) self.assertEqual(len(self.db.getBans(jail=self.jail)), 2) # delete two IPs: - self.db.delBan(self.jail, tickets[1].getIP(), tickets[2].getIP()) + self.db.delBan(self.jail, tickets[1].getID(), tickets[2].getID()) self.assertEqual(len(self.db.getBans(jail=self.jail)), 0) def testFlushBans(self): @@ -398,7 +398,7 @@ class DatabaseTest(LogCaptureTestCase): # should retrieve 2 matches only, but count of all attempts: self.db.maxMatches = maxMatches; ticket = self.db.getBansMerged("127.0.0.1") - self.assertEqual(ticket.getIP(), "127.0.0.1") + self.assertEqual(ticket.getID(), "127.0.0.1") self.assertEqual(ticket.getAttempt(), len(failures)) self.assertEqual(len(ticket.getMatches()), maxMatches) self.assertEqual(ticket.getMatches(), matches2find[-maxMatches:]) @@ -456,13 +456,13 @@ class DatabaseTest(LogCaptureTestCase): # All for IP 127.0.0.1 ticket = self.db.getBansMerged("127.0.0.1") - self.assertEqual(ticket.getIP(), "127.0.0.1") + self.assertEqual(ticket.getID(), "127.0.0.1") self.assertEqual(ticket.getAttempt(), 70) self.assertEqual(ticket.getMatches(), ["abc\n", "123\n", "ABC\n"]) # All for IP 127.0.0.1 for single jail ticket = self.db.getBansMerged("127.0.0.1", jail=self.jail) - self.assertEqual(ticket.getIP(), "127.0.0.1") + self.assertEqual(ticket.getID(), "127.0.0.1") self.assertEqual(ticket.getAttempt(), 30) self.assertEqual(ticket.getMatches(), ["abc\n", "123\n"]) @@ -490,8 +490,8 @@ class DatabaseTest(LogCaptureTestCase): tickets = self.db.getBansMerged() self.assertEqual(len(tickets), 2) self.assertSortedEqual( - list(set(ticket.getIP() for ticket in tickets)), - [ticket.getIP() for ticket in tickets]) + list(set(ticket.getID() for ticket in tickets)), + [ticket.getID() for ticket in tickets]) tickets = self.db.getBansMerged(jail=jail2) self.assertEqual(len(tickets), 1) @@ -510,7 +510,7 @@ class DatabaseTest(LogCaptureTestCase): tickets = self.db.getCurrentBans(jail=self.jail) self.assertEqual(len(tickets), 2) ticket = self.db.getCurrentBans(jail=None, ip="127.0.0.1"); - self.assertEqual(ticket.getIP(), "127.0.0.1") + self.assertEqual(ticket.getID(), "127.0.0.1") # positive case (1 ticket not yet expired): tickets = self.db.getCurrentBans(jail=self.jail, forbantime=15, diff --git a/fail2ban/tests/fail2banregextestcase.py b/fail2ban/tests/fail2banregextestcase.py index b526dbf5..97670f50 100644 --- a/fail2ban/tests/fail2banregextestcase.py +++ b/fail2ban/tests/fail2banregextestcase.py @@ -372,6 +372,18 @@ class Fail2banRegexTest(LogCaptureTestCase): r'^\s*\S+ : \S+')) self.assertLogged(str(('[192.0.2.4]:12345', 'left', 'right'))) self.pruneLog() + # ip is not id anymore (if IP-address deviates from ID): + self.assertTrue(_test_exec('-o', 'ip', + '1591983743.667 left [192.0.2.4]:12345 right', + r'^\s*\S+ : \S+')) + self.assertNotLogged(str(('[192.0.2.4]:12345', 'left', 'right'))) + self.assertLogged('192.0.2.4') + self.pruneLog() + self.assertTrue(_test_exec('-o', 'ID: | IP:', + '1591983743.667 left [192.0.2.4]:12345 right', + r'^\s*\S+ : \S+')) + self.assertLogged('ID:'+str(('[192.0.2.4]:12345', 'left', 'right'))+' | IP:192.0.2.4') + self.pruneLog() # row with id : self.assertTrue(_test_exec('-o', 'row', STR_00, RE_00_ID)) self.assertLogged("['kevin'", "'ip4': '192.0.2.0'", "'fid': 'kevin'", all=True) diff --git a/fail2ban/tests/failmanagertestcase.py b/fail2ban/tests/failmanagertestcase.py index a5425286..42b0fbd2 100644 --- a/fail2ban/tests/failmanagertestcase.py +++ b/fail2ban/tests/failmanagertestcase.py @@ -150,8 +150,8 @@ class AddFailure(unittest.TestCase): self.__failManager.setMaxRetry(5) #ticket = FailTicket('193.168.0.128', None) ticket = self.__failManager.toBan() - self.assertEqual(ticket.getIP(), "193.168.0.128") - self.assertTrue(isinstance(ticket.getIP(), (str, IPAddr))) + self.assertEqual(ticket.getID(), "193.168.0.128") + self.assertTrue(isinstance(ticket.getID(), (str, IPAddr))) # finish with rudimentary tests of the ticket # verify consistent str @@ -180,9 +180,9 @@ class AddFailure(unittest.TestCase): def testWindow(self): self._addDefItems() ticket = self.__failManager.toBan() - self.assertNotEqual(ticket.getIP(), "100.100.10.10") + self.assertNotEqual(ticket.getID(), "100.100.10.10") ticket = self.__failManager.toBan() - self.assertNotEqual(ticket.getIP(), "100.100.10.10") + self.assertNotEqual(ticket.getID(), "100.100.10.10") self.assertRaises(FailManagerEmpty, self.__failManager.toBan) def testBgService(self): diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 0c704767..017e54ec 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -141,7 +141,7 @@ def _ticket_tuple(ticket): """ attempts = ticket.getAttempt() date = ticket.getTime() - ip = ticket.getIP() + ip = ticket.getID() matches = ticket.getMatches() return (ip, attempts, date, matches) @@ -1460,7 +1460,7 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover self.assertTrue(ticket) attempts = ticket.getAttempt() - ip = ticket.getIP() + ip = ticket.getID() ticket.getMatches() self.assertEqual(ip, test_ip) @@ -1646,7 +1646,7 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover self.waitForTicks(1) self.waitFailTotal(6, 10) self.assertTrue(Utils.wait_for(lambda: len(self.jail) == 2, 10)) - self.assertSortedEqual([self.jail.getFailTicket().getIP(), self.jail.getFailTicket().getIP()], + self.assertSortedEqual([self.jail.getFailTicket().getID(), self.jail.getFailTicket().getID()], ["192.0.2.1", "192.0.2.2"]) cls = MonitorJournalFailures diff --git a/fail2ban/tests/observertestcase.py b/fail2ban/tests/observertestcase.py index 315c955a..9b44c6dd 100644 --- a/fail2ban/tests/observertestcase.py +++ b/fail2ban/tests/observertestcase.py @@ -367,14 +367,14 @@ class BanTimeIncrDB(LogCaptureTestCase): # this old ticket should be removed now: restored_tickets = self.db.getCurrentBans(fromtime=stime, correctBanTime=False) self.assertEqual(len(restored_tickets), 2) - self.assertEqual(restored_tickets[0].getIP(), ip) + self.assertEqual(restored_tickets[0].getID(), ip) # purge remove 1st ip self.db._purgeAge = -48*60*60 self.db.purge() restored_tickets = self.db.getCurrentBans(fromtime=stime, correctBanTime=False) self.assertEqual(len(restored_tickets), 1) - self.assertEqual(restored_tickets[0].getIP(), ip+'1') + self.assertEqual(restored_tickets[0].getID(), ip+'1') # this should purge all bans, bips and logs - nothing should be found now self.db._purgeAge = -240*60*60 diff --git a/fail2ban/tests/tickettestcase.py b/fail2ban/tests/tickettestcase.py index d7d5f19a..945da7bd 100644 --- a/fail2ban/tests/tickettestcase.py +++ b/fail2ban/tests/tickettestcase.py @@ -39,6 +39,7 @@ class TicketTests(unittest.TestCase): # Ticket t = Ticket('193.168.0.128', tm, matches) + self.assertEqual(t.getID(), '193.168.0.128') self.assertEqual(t.getIP(), '193.168.0.128') self.assertEqual(t.getTime(), tm) self.assertEqual(t.getMatches(), matches2) @@ -65,6 +66,7 @@ class TicketTests(unittest.TestCase): matches = ['first', 'second'] ft = FailTicket('193.168.0.128', tm, matches) ft.setBanTime(60*60) + self.assertEqual(ft.getID(), '193.168.0.128') self.assertEqual(ft.getIP(), '193.168.0.128') self.assertEqual(ft.getTime(), tm) self.assertEqual(ft.getMatches(), matches2) From b83712e3ec13383a2da961f2a13ed7cd9d09724a Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 11 Feb 2022 21:11:29 +0100 Subject: [PATCH 202/240] fail2ban-regex: accepts filter parameters with new-line --- fail2ban/client/fail2banregex.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fail2ban/client/fail2banregex.py b/fail2ban/client/fail2banregex.py index 4ab8a8a3..e0caae70 100644 --- a/fail2ban/client/fail2banregex.py +++ b/fail2ban/client/fail2banregex.py @@ -331,7 +331,7 @@ class Fail2banRegex(object): fltFile = None fltOpt = {} if regextype == 'fail': - if re.search(r'^/{0,3}[\w/_\-.]+(?:\[.*\])?$', value): + if re.search(r'^(?ms)/{0,3}[\w/_\-.]+(?:\[.*\])?$', value): try: fltName, fltOpt = extractOptions(value) if "." in fltName[~5:]: From c6e93db278005949a69173d268c0a461f2933d6d Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 11 Feb 2022 21:12:59 +0100 Subject: [PATCH 203/240] filter reader stream: don't need to generate None values from filter config --- fail2ban/client/filterreader.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/fail2ban/client/filterreader.py b/fail2ban/client/filterreader.py index 413f125e..24341014 100644 --- a/fail2ban/client/filterreader.py +++ b/fail2ban/client/filterreader.py @@ -72,8 +72,9 @@ class FilterReader(DefinitionInitConfigReader): def _fillStream(stream, opts, jailName): prio0idx = 0 for opt, value in opts.iteritems(): + # Do not send a command if the value is not set (empty). + if value is None: continue if opt in ("failregex", "ignoreregex"): - if value is None: continue multi = [] for regex in value.split('\n'): # Do not send a command if the rule is empty. @@ -91,8 +92,6 @@ class FilterReader(DefinitionInitConfigReader): elif opt in ('datepattern'): stream.append(["set", jailName, opt, value]) elif opt == 'journalmatch': - # Do not send a command if the match is empty. - if value is None: continue for match in value.split("\n"): if match == '': continue stream.append( From cf2695a253856aaedb5fe2db565f7835c6419135 Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 11 Feb 2022 21:13:30 +0100 Subject: [PATCH 204/240] more test cases (coverage for fail2ban-regex on constellations with different IP/ID) --- fail2ban/tests/fail2banregextestcase.py | 47 ++++++++++++++++++++++--- 1 file changed, 42 insertions(+), 5 deletions(-) diff --git a/fail2ban/tests/fail2banregextestcase.py b/fail2ban/tests/fail2banregextestcase.py index 97670f50..bc799b84 100644 --- a/fail2ban/tests/fail2banregextestcase.py +++ b/fail2ban/tests/fail2banregextestcase.py @@ -355,31 +355,31 @@ class Fail2banRegexTest(LogCaptureTestCase): self.assertLogged('kevin') self.pruneLog() # multiple id combined to a tuple (id, tuple_id): - self.assertTrue(_test_exec('-o', 'id', + self.assertTrue(_test_exec('-o', 'id', '-d', '{^LN-BEG}EPOCH', '1591983743.667 192.0.2.1 192.0.2.2', r'^\s* \S+')) self.assertLogged(str(('192.0.2.1', '192.0.2.2'))) self.pruneLog() # multiple id combined to a tuple, id first - (id, tuple_id_1, tuple_id_2): - self.assertTrue(_test_exec('-o', 'id', + self.assertTrue(_test_exec('-o', 'id', '-d', '{^LN-BEG}EPOCH', '1591983743.667 left 192.0.2.3 right', r'^\s*\S+ \S+')) self.assertLogged(str(('192.0.2.3', 'left', 'right'))) self.pruneLog() # id had higher precedence as ip-address: - self.assertTrue(_test_exec('-o', 'id', + self.assertTrue(_test_exec('-o', 'id', '-d', '{^LN-BEG}EPOCH', '1591983743.667 left [192.0.2.4]:12345 right', r'^\s*\S+ : \S+')) self.assertLogged(str(('[192.0.2.4]:12345', 'left', 'right'))) self.pruneLog() # ip is not id anymore (if IP-address deviates from ID): - self.assertTrue(_test_exec('-o', 'ip', + self.assertTrue(_test_exec('-o', 'ip', '-d', '{^LN-BEG}EPOCH', '1591983743.667 left [192.0.2.4]:12345 right', r'^\s*\S+ : \S+')) self.assertNotLogged(str(('[192.0.2.4]:12345', 'left', 'right'))) self.assertLogged('192.0.2.4') self.pruneLog() - self.assertTrue(_test_exec('-o', 'ID: | IP:', + self.assertTrue(_test_exec('-o', 'ID: | IP:', '-d', '{^LN-BEG}EPOCH', '1591983743.667 left [192.0.2.4]:12345 right', r'^\s*\S+ : \S+')) self.assertLogged('ID:'+str(('[192.0.2.4]:12345', 'left', 'right'))+' | IP:192.0.2.4') @@ -405,6 +405,43 @@ class Fail2banRegexTest(LogCaptureTestCase): self.assertLogged('192.0.2.0, kevin, inet4') self.pruneLog() + def testStalledIPByNoFailFrmtOutput(self): + opts = ( + '-c', CONFIG_DIR, + "-d", r"^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?", + ) + log = ( + 'May 27 00:16:33 host sshd[2364]: User root not allowed because account is locked\n' + 'May 27 00:16:33 host sshd[2364]: Received disconnect from 192.0.2.76 port 58846:11: Bye Bye [preauth]' + ) + _test = lambda *args: _test_exec(*(opts + args)) + # with MLFID from prefregex and IP after failure obtained from F-NOFAIL RE: + self.assertTrue(_test('-o', 'IP:', log, 'sshd')) + self.assertLogged('IP:192.0.2.76') + self.pruneLog() + # test diverse ID/IP constellations: + def _test_variants(flt="sshd", prefix=""): + # with different ID/IP from failregex (ID/User from first, IP from second message): + self.assertTrue(_test('-o', 'ID:"" | IP: | U:', log, + flt+'[failregex="' + '^'+prefix+'User \S+ not allowed\n' + '^'+prefix+'Received disconnect from ' + '"]')) + self.assertLogged('ID:"User root" | IP:192.0.2.76 | U:root') + self.pruneLog() + # with different ID/IP from failregex (User from first, ID and IP from second message): + self.assertTrue(_test('-o', 'ID:"" | IP: | U:', log, + flt+'[failregex="' + '^'+prefix+'User \S+ not allowed\n' + '^'+prefix+'Received disconnect from port \d+' + '"]')) + self.assertLogged('ID:"192.0.2.76 port 58846" | IP:192.0.2.76 | U:root') + self.pruneLog() + # first with sshd and prefregex: + _test_variants() + # the same without prefregex and MLFID directly in failregex (no merge with prefregex groups): + _test_variants('common', prefix="\s*\S+ sshd\[\d+\]:\s+") + def testNoDateTime(self): # datepattern doesn't match: self.assertTrue(_test_exec('-d', '{^LN-BEG}EPOCH', '-o', 'Found-ID:', STR_00_NODT, RE_00_ID)) From 8eb521694e2f7a5739442ba0e32b2413e9350d00 Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 11 Feb 2022 21:25:31 +0100 Subject: [PATCH 205/240] fulfill getIP with getID replacement; added simple tests for ticket --- fail2ban/server/jail.py | 2 +- fail2ban/tests/tickettestcase.py | 11 +++++++++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/fail2ban/server/jail.py b/fail2ban/server/jail.py index 673b6454..2c84e475 100644 --- a/fail2ban/server/jail.py +++ b/fail2ban/server/jail.py @@ -295,7 +295,7 @@ class Jail(object): ): try: #logSys.debug('restored ticket: %s', ticket) - if self.filter.inIgnoreIPList(ticket.getIP(), log_ignore=True): continue + if self.filter.inIgnoreIPList(ticket.getID(), log_ignore=True): continue # mark ticked was restored from database - does not put it again into db: ticket.restored = True # correct start time / ban time (by the same end of ban): diff --git a/fail2ban/tests/tickettestcase.py b/fail2ban/tests/tickettestcase.py index 945da7bd..771d2b50 100644 --- a/fail2ban/tests/tickettestcase.py +++ b/fail2ban/tests/tickettestcase.py @@ -118,6 +118,17 @@ class TicketTests(unittest.TestCase): self.assertEqual(ft2.getTime(), ft.getTime()) self.assertEqual(ft2.getBanTime(), ft.getBanTime()) + def testDiffIDAndIPTicket(self): + tm = MyTime.time() + # different ID (string) and IP: + t = Ticket('123-456-678', tm, data={'ip':'192.0.2.1'}) + self.assertEqual(t.getID(), '123-456-678') + self.assertEqual(t.getIP(), '192.0.2.1') + # different ID (tuple) and IP: + t = Ticket(('192.0.2.1', '5000'), tm, data={'ip':'192.0.2.1'}) + self.assertEqual(t.getID(), ('192.0.2.1', '5000')) + self.assertEqual(t.getIP(), '192.0.2.1') + def testTicketFlags(self): flags = ('restored', 'banned') ticket = Ticket('test', 0) From b903059419b45113f054a149a21abd5cf289d8c3 Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 18 Feb 2022 20:19:43 +0100 Subject: [PATCH 206/240] ChangeLog for RFE gh-3217 (with compat warning) --- ChangeLog | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/ChangeLog b/ChangeLog index d294ad54..0b5633c7 100644 --- a/ChangeLog +++ b/ChangeLog @@ -18,6 +18,9 @@ ver. 1.0.1-dev-1 (20??/??/??) - development nightly edition - due to change of `actioncheck` behavior (gh-488), some actions can be incompatible as regards the invariant check, if `actionban` or `actionunban` would not throw an error (exit code different from 0) in case of unsane environment. + - actions that have used tag `` (instead of `` or ``) to get failure-ID may become + incompatible, if filter uses IP-related tags (like `` or ``) additionally to `` + and the values are different (gh-3217) ### Fixes * readline fixed to consider interim new-line character as part of code point in multi-byte logs @@ -34,6 +37,8 @@ ver. 1.0.1-dev-1 (20??/??/??) - development nightly edition * better recognition of log rotation, better performance by reopen: avoid unnecessary seek to begin of file (and hash calculation) * file filter reads only complete lines (ended with new-line) now, so waits for end of line (for its completion) +* actions differentiate tags `` and `` (``), if IP-address deviates from ID then the value + of `` is not equal `` anymore (gh-3217) * `action.d/ufw.conf` (gh-3018): - new option `add` (default `prepend`), can be supplied as `insert 1` for ufw versions before v.0.36 (gh-2331, gh-3018) - new options `kill-mode` and `kill` to drop established connections of intruder (see action for details, gh-3018) From 1e5d5a446acd5171daf22975cb972695953a80f0 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Mon, 21 Feb 2022 16:59:38 +0100 Subject: [PATCH 207/240] highlighting got broken, so comment out unless GH/linguist gets fixed revert #3126 --- .gitattributes | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.gitattributes b/.gitattributes index 0cbdbf83..ef69dd82 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1 +1,2 @@ -ChangeLog linguist-language=Markdown +# unless GH/linguist gets fixed +#ChangeLog linguist-language=Markdown From 8e62c8a569949e6ba54a7e4ea026aae3c36b9ce3 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Mon, 21 Feb 2022 17:02:24 +0100 Subject: [PATCH 208/240] syntax --- ChangeLog | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ChangeLog b/ChangeLog index 0b5633c7..5b2acdc9 100644 --- a/ChangeLog +++ b/ChangeLog @@ -9,7 +9,7 @@ Fail2Ban: Changelog ver. 1.0.1-dev-1 (20??/??/??) - development nightly edition ----------- -### Compatibility: +### Compatibility * the minimum supported python version is now 2.7, if you have previous python version you can use the 0.11 version of fail2ban or upgrade python (or even build it from source). * potential incompatibility by parsing of options of `backend`, `filter` and `action` parameters (if they @@ -49,7 +49,7 @@ ver. 1.0.1-dev-1 (20??/??/??) - development nightly edition ver. 0.11.2 (2020/11/23) - heal-the-world-with-security-tools ----------- -### Compatibility: +### Compatibility * to v.0.10: - 0.11 is totally compatible to 0.10 (configuration- and API-related stuff), but the database got some new tables and fields (auto-converted during the first start), so once updated to 0.11, you From 45e08cc07f64f1dbae738a4da36878200ab42f4c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Sz=C3=A9pe?= Date: Mon, 21 Feb 2022 18:29:04 +0000 Subject: [PATCH 209/240] Render Changelog as Markdown --- ChangeLog | 1 + 1 file changed, 1 insertion(+) diff --git a/ChangeLog b/ChangeLog index 5b2acdc9..d386ceef 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,4 @@ + __ _ _ ___ _ / _|__ _(_) |_ ) |__ __ _ _ _ | _/ _` | | |/ /| '_ \/ _` | ' \ From 3a9f5c0b5dc0a164e0ec245887b893c8150f713a Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 21 Feb 2022 17:04:57 +0100 Subject: [PATCH 210/240] Revert "highlighting got broken, so comment out unless GH/linguist gets fixed" This reverts commit 1e5d5a446acd5171daf22975cb972695953a80f0. --- .gitattributes | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.gitattributes b/.gitattributes index ef69dd82..0cbdbf83 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,2 +1 @@ -# unless GH/linguist gets fixed -#ChangeLog linguist-language=Markdown +ChangeLog linguist-language=Markdown From 7eac4ac06fb03b8fce9b5d8bd368493482a1efe7 Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 11 Feb 2022 21:11:29 +0100 Subject: [PATCH 211/240] fail2ban-regex: accepts filter parameters with new-line --- fail2ban/client/fail2banregex.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fail2ban/client/fail2banregex.py b/fail2ban/client/fail2banregex.py index 90e178f9..8c03b2dd 100644 --- a/fail2ban/client/fail2banregex.py +++ b/fail2ban/client/fail2banregex.py @@ -334,7 +334,7 @@ class Fail2banRegex(object): fltFile = None fltOpt = {} if regextype == 'fail': - if re.search(r'^/{0,3}[\w/_\-.]+(?:\[.*\])?$', value): + if re.search(r'^(?ms)/{0,3}[\w/_\-.]+(?:\[.*\])?$', value): try: fltName, fltOpt = extractOptions(value) if "." in fltName[~5:]: From e2d50f38a6ef2511fee6b49f42b98f6d867625b2 Mon Sep 17 00:00:00 2001 From: sebres Date: Thu, 3 Mar 2022 15:04:34 +0100 Subject: [PATCH 212/240] amend to #2279: ensure that `` match would reset all pending multi-line failures --- fail2ban/server/filter.py | 2 ++ fail2ban/tests/fail2banregextestcase.py | 21 +++++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index f8417d2d..041773ab 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -793,6 +793,8 @@ class Filter(JailThread): # be sure we've correct current state ('nofail' and 'mlfgained' only from last failure) if mlfidGroups.pop('nofail', None): nfflgs |= 4 if mlfidGroups.pop('mlfgained', None): nfflgs |= 4 + # gained resets all pending failures (retaining users to check it later) + if nfflgs & 8: mlfidGroups.pop('mlfpending', None) # if we had no pending failures then clear the matches (they are already provided): if (nfflgs & 4) == 0 and not mlfidGroups.get('mlfpending', 0): mlfidGroups.pop("matches", None) diff --git a/fail2ban/tests/fail2banregextestcase.py b/fail2ban/tests/fail2banregextestcase.py index 1c55e227..00808ddd 100644 --- a/fail2ban/tests/fail2banregextestcase.py +++ b/fail2ban/tests/fail2banregextestcase.py @@ -440,6 +440,27 @@ class Fail2banRegexTest(LogCaptureTestCase): '192.0.2.1, git, '+lines[-1], all=True) + def testOutputNoPendingFailuresAfterGained(self): + unittest.F2B.SkipIfCfgMissing(stock=True) + # connect finished without authorization must generate a failure, because + # connect started will produce pending failure which gets reset by gained + # connect authorized. + self.assertTrue(_test_exec('-o', 'failure from == ==', + '-c', CONFIG_DIR, '-d', '{NONE}', + 'svc[1] connect started 192.0.2.3\n' + 'svc[1] connect finished 192.0.2.3\n' + 'svc[2] connect started 192.0.2.4\n' + 'svc[2] connect authorized 192.0.2.4\n' + 'svc[2] connect finished 192.0.2.4\n', + 'common[prefregex="^svc\[\d+\] connect .+$"' + ', failregex="' + '^started\n' + '^finished \n' + '^authorized ' + '", maxlines=1]' + )) + self.assertLogged('failure from == 192.0.2.3 ==') + self.assertNotLogged('failure from == 192.0.2.4 ==') def testWrongFilterFile(self): # use test log as filter file to cover eror cases... From 7e7b9f4a35ad39c7f6f3a3a0b64ec4781868b788 Mon Sep 17 00:00:00 2001 From: Logic-32 <25107222+Logic-32@users.noreply.github.com> Date: Wed, 27 Apr 2022 14:12:03 -0600 Subject: [PATCH 213/240] Adding support for Cloudflare Token API. Closes #3080 --- ChangeLog | 1 + config/action.d/cloudflare-token.conf | 92 +++++++++++++++++++++++++++ 2 files changed, 93 insertions(+) create mode 100644 config/action.d/cloudflare-token.conf diff --git a/ChangeLog b/ChangeLog index d386ceef..d5bbbd45 100644 --- a/ChangeLog +++ b/ChangeLog @@ -45,6 +45,7 @@ ver. 1.0.1-dev-1 (20??/??/??) - development nightly edition - new options `kill-mode` and `kill` to drop established connections of intruder (see action for details, gh-3018) * `filter.d/nginx-http-auth.conf` - extended with parameter mode, so additionally to `auth` (or `normal`) mode `fallback` (or combined as `aggressive`) can find SSL errors while SSL handshaking, gh-2881 +* `action.d/cloudflare-token.conf` - added support for Cloudflare Token APIs. This method is more restrictive and therefore safter than using API Keys. ver. 0.11.2 (2020/11/23) - heal-the-world-with-security-tools diff --git a/config/action.d/cloudflare-token.conf b/config/action.d/cloudflare-token.conf new file mode 100644 index 00000000..a660c281 --- /dev/null +++ b/config/action.d/cloudflare-token.conf @@ -0,0 +1,92 @@ +# +# Author: Logic-32 +# +# IMPORTANT +# +# Please set jail.local's permission to 640 because it contains your CF API token. +# +# This action depends on curl. +# +# To get your Cloudflare API token: https://developers.cloudflare.com/api/tokens/create/ +# +# Cloudflare Firewall API: https://developers.cloudflare.com/firewall/api/cf-firewall-rules/endpoints/ + +[Definition] + +# Option: actionstart +# Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false). +# Values: CMD +# +actionstart = + +# Option: actionstop +# Notes.: command executed at the stop of jail (or at the end of Fail2Ban) +# Values: CMD +# +actionstop = + +# Option: actioncheck +# Notes.: command executed once before each actionban command +# Values: CMD +# +actioncheck = + +# Option: actionban +# Notes.: command executed when banning an IP. Take care that the +# command is executed with Fail2Ban user rights. +# Tags: IP address +# number of failures +#