From fe9e85c71d73d7ec57048113768f2a02fa110ba6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Allan=20Nordh=C3=B8y?= Date: Fri, 10 Nov 2017 23:56:10 +0100 Subject: [PATCH 001/189] "Fail2Ban", other language improvements --- README.md | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 1138a3ac..71997c98 100644 --- a/README.md +++ b/README.md @@ -6,45 +6,45 @@ ## Fail2Ban: ban hosts that cause multiple authentication errors -Fail2Ban scans log files like `/var/log/auth.log` and bans IP addresses having +Fail2Ban scans log files like `/var/log/auth.log` and bans IP addresses conducting too many failed login attempts. It does this by updating system firewall rules to reject new connections from those IP addresses, for a configurable amount of time. Fail2Ban comes out-of-the-box ready to read many standard log files, -such as those for sshd and Apache, and is easy to configure to read any log -file you choose, for any error you choose. +such as those for sshd and Apache, and is easily configured to read any log +file of your choosing, for any error you wish. -Though Fail2Ban is able to reduce the rate of incorrect authentications -attempts, it cannot eliminate the risk that weak authentication presents. -Configure services to use only two factor or public/private authentication +Though Fail2Ban is able to reduce the rate of incorrect authentication +attempts, it cannot eliminate the risk presented by weak authentication. +Set up services to use only two factor, or public/private authentication mechanisms if you really want to protect services. -This README is a quick introduction to Fail2ban. More documentation, FAQ, HOWTOs -are available in fail2ban(1) manpage and on the website http://www.fail2ban.org +This README is a quick introduction to Fail2Ban. More documentation, FAQ, and HOWTOs +available in the fail2ban(1) manpage and on the website https://www.fail2ban.org Installation: ------------- -**It is possible that Fail2ban is already packaged for your distribution. In -this case, you should use it instead.** +**It is possible that Fail2Ban is already packaged for your distribution. In +this case, you should use that instead.** Required: -- [Python2 >= 2.6 or Python >= 3.2](http://www.python.org) or [PyPy](http://pypy.org) +- [Python2 >= 2.6 or Python >= 3.2](https://www.python.org) or [PyPy](https://pypy.org) Optional: - [pyinotify >= 0.8.3](https://github.com/seb-m/pyinotify) - - Linux >= 2.6.13 -- [gamin >= 0.0.21](http://www.gnome.org/~veillard/gamin) -- [systemd >= 204](http://www.freedesktop.org/wiki/Software/systemd) +- Linux >= 2.6.13 +- [gamin >= 0.0.21](https://www.gnome.org/~veillard/gamin) +- [systemd >= 204](https://www.freedesktop.org/wiki/Software/systemd) - [dnspython](http://www.dnspython.org/) -To install, just do: +To install: tar xvfj fail2ban-0.9.7.tar.bz2 cd fail2ban-0.9.7 python setup.py install This will install Fail2Ban into the python library directory. The executable -scripts are placed into `/usr/bin`, and configuration under `/etc/fail2ban`. +scripts are placed into `/usr/bin`, and configuration in `/etc/fail2ban`. Fail2Ban should be correctly installed now. Just type: @@ -86,7 +86,7 @@ Contact: See [CONTRIBUTING.md](https://github.com/fail2ban/fail2ban/blob/master/CONTRIBUTING.md) ### You just appreciate this program: -send kudos to the original author ([Cyril Jaquier](mailto: Cyril Jaquier )) +Send kudos to the original author ([Cyril Jaquier](mailto: Cyril Jaquier )) or *better* to the [mailing list](https://lists.sourceforge.net/lists/listinfo/fail2ban-users) since Fail2Ban is "community-driven" for years now. From 855f5d0ced70247ec80823ae4c8507adbd14554b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Allan=20Nordh=C3=B8y?= Date: Sat, 11 Nov 2017 14:03:15 +0100 Subject: [PATCH 002/189] to be found --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 71997c98..d97cdf38 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ Set up services to use only two factor, or public/private authentication mechanisms if you really want to protect services. This README is a quick introduction to Fail2Ban. More documentation, FAQ, and HOWTOs -available in the fail2ban(1) manpage and on the website https://www.fail2ban.org +to be found on fail2ban(1) manpage and the website: https://www.fail2ban.org Installation: ------------- From 9a38d5697ff1a876ee94f3ad273fd919b54535f1 Mon Sep 17 00:00:00 2001 From: sebres Date: Thu, 18 Jan 2018 16:40:48 +0100 Subject: [PATCH 003/189] bump version (0.10.2 -> 0.10.3.dev1) --- ChangeLog | 10 ++++++++++ README.md | 6 +++--- fail2ban/version.py | 2 +- man/fail2ban-client.1 | 4 ++-- man/fail2ban-regex.1 | 2 +- man/fail2ban-server.1 | 4 ++-- man/fail2ban-testcases.1 | 2 +- 7 files changed, 20 insertions(+), 10 deletions(-) diff --git a/ChangeLog b/ChangeLog index 29b07973..3813a156 100644 --- a/ChangeLog +++ b/ChangeLog @@ -31,6 +31,16 @@ Incompatibility list (compared to v.0.9): IPv6-capable now. +ver. 0.10.3-dev-1 (20??/??/??) - development edition +----------- + +### Fixes + +### New Features + +### Enhancements + + ver. 0.10.2 (2018/01/18) - nothing-burns-like-the-cold ----------- diff --git a/README.md b/README.md index 94485adb..84eb3302 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ / _|__ _(_) |_ ) |__ __ _ _ _ | _/ _` | | |/ /| '_ \/ _` | ' \ |_| \__,_|_|_/___|_.__/\__,_|_||_| - v0.10.2 2018/01/18 + v0.10.3.dev1 20??/??/?? ## Fail2Ban: ban hosts that cause multiple authentication errors @@ -43,8 +43,8 @@ Optional: To install, just do: - tar xvfj fail2ban-0.10.2.tar.bz2 - cd fail2ban-0.10.2 + tar xvfj fail2ban-0.10.3.tar.bz2 + cd fail2ban-0.10.3 python setup.py install This will install Fail2Ban into the python library directory. The executable diff --git a/fail2ban/version.py b/fail2ban/version.py index 67f9a6e6..fd16e76c 100644 --- a/fail2ban/version.py +++ b/fail2ban/version.py @@ -24,4 +24,4 @@ __author__ = "Cyril Jaquier, Yaroslav Halchenko, Steven Hiscocks, Daniel Black" __copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2005-2016 Yaroslav Halchenko, 2013-2014 Steven Hiscocks, Daniel Black" __license__ = "GPL-v2+" -version = "0.10.2" +version = "0.10.3.dev1" diff --git a/man/fail2ban-client.1 b/man/fail2ban-client.1 index ab9559fb..58abc9ff 100644 --- a/man/fail2ban-client.1 +++ b/man/fail2ban-client.1 @@ -1,12 +1,12 @@ .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. -.TH FAIL2BAN-CLIENT "1" "January 2018" "fail2ban-client v0.10.2" "User Commands" +.TH FAIL2BAN-CLIENT "1" "January 2018" "fail2ban-client v0.10.3.dev1" "User Commands" .SH NAME fail2ban-client \- configure and control the server .SH SYNOPSIS .B fail2ban-client [\fI\,OPTIONS\/\fR] \fI\,\/\fR .SH DESCRIPTION -Fail2Ban v0.10.2 reads log file that contains password failure report +Fail2Ban v0.10.3.dev1 reads log file that contains password failure report and bans the corresponding IP addresses using firewall rules. .SH OPTIONS .TP diff --git a/man/fail2ban-regex.1 b/man/fail2ban-regex.1 index c4fca08c..74e9c6bc 100644 --- a/man/fail2ban-regex.1 +++ b/man/fail2ban-regex.1 @@ -1,5 +1,5 @@ .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. -.TH FAIL2BAN-REGEX "1" "January 2018" "fail2ban-regex 0.10.2" "User Commands" +.TH FAIL2BAN-REGEX "1" "January 2018" "fail2ban-regex 0.10.3.dev1" "User Commands" .SH NAME fail2ban-regex \- test Fail2ban "failregex" option .SH SYNOPSIS diff --git a/man/fail2ban-server.1 b/man/fail2ban-server.1 index debfb232..63481c36 100644 --- a/man/fail2ban-server.1 +++ b/man/fail2ban-server.1 @@ -1,12 +1,12 @@ .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. -.TH FAIL2BAN-SERVER "1" "January 2018" "fail2ban-server v0.10.2" "User Commands" +.TH FAIL2BAN-SERVER "1" "January 2018" "fail2ban-server v0.10.3.dev1" "User Commands" .SH NAME fail2ban-server \- start the server .SH SYNOPSIS .B fail2ban-server [\fI\,OPTIONS\/\fR] .SH DESCRIPTION -Fail2Ban v0.10.2 reads log file that contains password failure report +Fail2Ban v0.10.3.dev1 reads log file that contains password failure report and bans the corresponding IP addresses using firewall rules. .SH OPTIONS .TP diff --git a/man/fail2ban-testcases.1 b/man/fail2ban-testcases.1 index 5762f827..19cd55a9 100644 --- a/man/fail2ban-testcases.1 +++ b/man/fail2ban-testcases.1 @@ -1,5 +1,5 @@ .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. -.TH FAIL2BAN-TESTCASES "1" "January 2018" "fail2ban-testcases 0.10.2" "User Commands" +.TH FAIL2BAN-TESTCASES "1" "January 2018" "fail2ban-testcases 0.10.3.dev1" "User Commands" .SH NAME fail2ban-testcases \- run Fail2Ban unit-tests .SH SYNOPSIS From 9d5f20aab265f7133bd87c5d859bdfc8531d558d Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 19 Jan 2018 12:32:24 +0100 Subject: [PATCH 004/189] FilterPyinotify: fixed sporadic test-case error (multi-threaded) - 'NoneType' object has no attribute 'stop'. --- fail2ban/server/filterpyinotify.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/fail2ban/server/filterpyinotify.py b/fail2ban/server/filterpyinotify.py index 17d387a1..4f3262b6 100644 --- a/fail2ban/server/filterpyinotify.py +++ b/fail2ban/server/filterpyinotify.py @@ -374,8 +374,11 @@ class FilterPyinotify(FileFilter): def stop(self): # stop filter thread: super(FilterPyinotify, self).stop() - if self.__notifier: # stop the notifier - self.__notifier.stop() + try: + if self.__notifier: # stop the notifier + self.__notifier.stop() + except AttributeError: # pragma: no cover + if self.__notifier: raise ## # Wait for exit with cleanup. From 8cfd97a68fccf942908bd024b5a359bbfebe1936 Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 22 Jan 2018 10:41:36 +0100 Subject: [PATCH 005/189] skip a testRepairDb if no sqlite3 command-helper available; code review (removed unnecessary code-pieces resp. code-duplication) closes #2026 --- fail2ban/tests/databasetestcase.py | 38 ++++++------------------------ 1 file changed, 7 insertions(+), 31 deletions(-) diff --git a/fail2ban/tests/databasetestcase.py b/fail2ban/tests/databasetestcase.py index 54d05677..1ee523d9 100644 --- a/fail2ban/tests/databasetestcase.py +++ b/fail2ban/tests/databasetestcase.py @@ -32,7 +32,7 @@ import shutil from ..server.filter import FileContainer from ..server.mytime import MyTime from ..server.ticket import FailTicket -from ..server.actions import Actions +from ..server.actions import Actions, Utils from .dummyjail import DummyJail try: from ..server.database import Fail2BanDb as Fail2BanDb @@ -85,13 +85,11 @@ class DatabaseTest(LogCaptureTestCase): os.remove(self.dbFilename) def testGetFilename(self): - if Fail2BanDb is None or self.db.filename == ':memory:': # pragma: no cover - return + if self.db.filename == ':memory:': # pragma: no cover + raise unittest.SkipTest("in :memory: database") self.assertEqual(self.dbFilename, self.db.filename) def testPurgeAge(self): - if Fail2BanDb is None: # pragma: no cover - return self.assertEqual(self.db.purgeage, 86400) self.db.purgeage = '1y6mon15d5h30m' self.assertEqual(self.db.purgeage, 48652200) @@ -99,16 +97,14 @@ class DatabaseTest(LogCaptureTestCase): self.assertEqual(self.db.purgeage, 48652200*2) def testCreateInvalidPath(self): - if Fail2BanDb is None: # pragma: no cover - return self.assertRaises( sqlite3.OperationalError, Fail2BanDb, "/this/path/should/not/exist") def testCreateAndReconnect(self): - if Fail2BanDb is None or self.db.filename == ':memory:': # pragma: no cover - return + if self.db.filename == ':memory:': # pragma: no cover + raise unittest.SkipTest("in :memory: database") self.testAddJail() # Reconnect... self.db = Fail2BanDb(self.dbFilename) @@ -118,8 +114,8 @@ class DatabaseTest(LogCaptureTestCase): "Jail not retained in Db after disconnect reconnect.") def testRepairDb(self): - if Fail2BanDb is None: # pragma: no cover - return + if not Utils.executeCmd("sqlite3 --version"): # pragma: no cover + raise unittest.SkipTest("no sqlite3 command") self.db = None if self.dbFilename is None: # pragma: no cover _, self.dbFilename = tempfile.mkstemp(".db", "fail2ban_") @@ -153,8 +149,6 @@ class DatabaseTest(LogCaptureTestCase): self.db = None def testUpdateDb(self): - if Fail2BanDb is None: # pragma: no cover - return self.db = None try: if self.dbFilename is None: # pragma: no cover @@ -174,8 +168,6 @@ class DatabaseTest(LogCaptureTestCase): os.remove(self.db._dbBackupFilename) def testAddJail(self): - if Fail2BanDb is None: # pragma: no cover - return self.jail = DummyJail() self.db.addJail(self.jail) self.assertTrue( @@ -183,8 +175,6 @@ class DatabaseTest(LogCaptureTestCase): "Jail not added to database") def testAddLog(self): - if Fail2BanDb is None: # pragma: no cover - return self.testAddJail() # Jail required _, filename = tempfile.mkstemp(".log", "Fail2BanDb_") @@ -196,8 +186,6 @@ class DatabaseTest(LogCaptureTestCase): os.remove(filename) def testUpdateLog(self): - if Fail2BanDb is None: # pragma: no cover - return self.testAddLog() # Add log file # Write some text @@ -237,8 +225,6 @@ class DatabaseTest(LogCaptureTestCase): os.remove(filename) def testAddBan(self): - if Fail2BanDb is None: # pragma: no cover - return self.testAddJail() ticket = FailTicket("127.0.0.1", 0, ["abc\n"]) self.db.addBan(self.jail, ticket) @@ -249,8 +235,6 @@ class DatabaseTest(LogCaptureTestCase): isinstance(tickets[0], FailTicket)) def testAddBanInvalidEncoded(self): - if Fail2BanDb is None: # pragma: no cover - return self.testAddJail() # invalid + valid, invalid + valid unicode, invalid + valid dual converted (like in filter:readline by fallback) ... tickets = [ @@ -304,8 +288,6 @@ class DatabaseTest(LogCaptureTestCase): self.assertEqual(len(self.db.getBans(jail=self.jail)), 0) def testGetBansWithTime(self): - if Fail2BanDb is None: # pragma: no cover - return self.testAddJail() self.db.addBan( self.jail, FailTicket("127.0.0.1", MyTime.time() - 60, ["abc\n"])) @@ -318,8 +300,6 @@ class DatabaseTest(LogCaptureTestCase): self.assertEqual(len(self.db.getBans(jail=self.jail,bantime=-1)), 2) def testGetBansMerged_MaxEntries(self): - if Fail2BanDb is None: # pragma: no cover - return self.testAddJail() maxEntries = 2 failures = ["abc\n", "123\n", "ABC\n", "1234\n"] @@ -349,8 +329,6 @@ class DatabaseTest(LogCaptureTestCase): self.assertEqual(ticket.getMatches(), failures[len(failures) - maxEntries:]) def testGetBansMerged(self): - if Fail2BanDb is None: # pragma: no cover - return self.testAddJail() jail2 = DummyJail() @@ -475,8 +453,6 @@ class DatabaseTest(LogCaptureTestCase): self.assertTrue(len(jails) == 0) def testPurge(self): - if Fail2BanDb is None: # pragma: no cover - return self.testAddJail() # Add jail self.db.purge() # Jail enabled by default so shouldn't be purged From af2de7ff2fe7ad8a306a8963467a54bdbae7f3a4 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Mon, 22 Jan 2018 21:08:39 -0500 Subject: [PATCH 006/189] RF: COND_FAMILIES - use tuple no need for a dict where tuple would be preferable (deterministic order) --- fail2ban/server/action.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fail2ban/server/action.py b/fail2ban/server/action.py index 31475d72..f06e28c0 100644 --- a/fail2ban/server/action.py +++ b/fail2ban/server/action.py @@ -383,7 +383,7 @@ class CommandAction(ActionBase): except ValueError as e: raise RuntimeError("Error %s action %s/%s: %r" % (operation, self._jail, self._name, e)) - COND_FAMILIES = {'inet4':1, 'inet6':1} + COND_FAMILIES = ('inet4', 'inet6') @property def _startOnDemand(self): From ba2538ba04b43eb0765b393ad78b5a1dc41111cd Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Mon, 22 Jan 2018 21:09:21 -0500 Subject: [PATCH 007/189] DOC: minor typos spotted around comments etc --- bin/fail2ban-server | 2 +- bin/fail2ban-testcases | 2 +- fail2ban/client/configparserinc.py | 2 +- fail2ban/client/fail2banregex.py | 2 +- fail2ban/client/fail2banserver.py | 4 ++-- fail2ban/helpers.py | 2 +- fail2ban/server/action.py | 10 +++++----- fail2ban/server/actions.py | 2 +- fail2ban/server/filter.py | 6 +++--- fail2ban/server/filterpoll.py | 2 +- fail2ban/tests/config/filter.d/test.conf | 2 +- 11 files changed, 18 insertions(+), 18 deletions(-) diff --git a/bin/fail2ban-server b/bin/fail2ban-server index 860a7607..03dc0fd3 100755 --- a/bin/fail2ban-server +++ b/bin/fail2ban-server @@ -22,7 +22,7 @@ Fail2Ban reads log file that contains password failure report and bans the corresponding IP addresses using firewall rules. -This tools starts/stops fail2ban server or does client/server communication, +This tool starts/stops fail2ban server or does client/server communication to change/read parameters of the server or jails. """ diff --git a/bin/fail2ban-testcases b/bin/fail2ban-testcases index 5539d4c6..ba3d90b9 100755 --- a/bin/fail2ban-testcases +++ b/bin/fail2ban-testcases @@ -31,7 +31,7 @@ import time import unittest # Check if local fail2ban module exists, and use if it exists by -# modifying the path. This is such that tests can be used in dev +# modifying the path. This is done so that tests can be used in dev # environment. if os.path.exists("fail2ban/__init__.py"): sys.path.insert(0, ".") diff --git a/fail2ban/client/configparserinc.py b/fail2ban/client/configparserinc.py index d220ed44..722f4618 100644 --- a/fail2ban/client/configparserinc.py +++ b/fail2ban/client/configparserinc.py @@ -20,7 +20,7 @@ # Author: Yaroslav Halchenko # Modified: Cyril Jaquier -__author__ = 'Yaroslav Halhenko, Serg G. Brester (aka sebres)' +__author__ = 'Yaroslav Halchenko, Serg G. Brester (aka sebres)' __copyright__ = 'Copyright (c) 2007 Yaroslav Halchenko, 2015 Serg G. Brester (aka sebres)' __license__ = 'GPL' diff --git a/fail2ban/client/fail2banregex.py b/fail2ban/client/fail2banregex.py index 84218149..c6338da5 100644 --- a/fail2ban/client/fail2banregex.py +++ b/fail2ban/client/fail2banregex.py @@ -226,7 +226,7 @@ class LineStats(object): class Fail2banRegex(object): def __init__(self, opts): - # set local protected memebers from given options: + # set local protected members from given options: self.__dict__.update(dict(('_'+o,v) for o,v in opts.__dict__.iteritems())) self._maxlines_set = False # so we allow to override maxlines in cmdline self._datepattern_set = False diff --git a/fail2ban/client/fail2banserver.py b/fail2ban/client/fail2banserver.py index 2dcaddf7..6c57fbf8 100644 --- a/fail2ban/client/fail2banserver.py +++ b/fail2ban/client/fail2banserver.py @@ -128,10 +128,10 @@ class Fail2banServer(Fail2banCmdLine): def getServerPath(): startdir = sys.path[0] exe = os.path.abspath(os.path.join(startdir, SERVER)) - if not os.path.isfile(exe): # may be uresolved in test-cases, so get relative starter (client): + if not os.path.isfile(exe): # may be unresolved in test-cases, so get relative starter (client): startdir = os.path.dirname(sys.argv[0]) exe = os.path.abspath(os.path.join(startdir, SERVER)) - if not os.path.isfile(exe): # may be uresolved in test-cases, so try to get relative bin-directory: + if not os.path.isfile(exe): # may be unresolved in test-cases, so try to get relative bin-directory: startdir = os.path.dirname(os.path.abspath(__file__)) startdir = os.path.join(os.path.dirname(os.path.dirname(startdir)), "bin") exe = os.path.abspath(os.path.join(startdir, SERVER)) diff --git a/fail2ban/helpers.py b/fail2ban/helpers.py index 378924df..98d59fa1 100644 --- a/fail2ban/helpers.py +++ b/fail2ban/helpers.py @@ -34,7 +34,7 @@ from .server.mytime import MyTime PREFER_ENC = locale.getpreferredencoding() -# correct prefered encoding if lang not set in environment: +# correct preferred encoding if lang not set in environment: if PREFER_ENC.startswith('ANSI_'): # pragma: no cover if all((os.getenv(v) in (None, "") for v in ('LANGUAGE', 'LC_ALL', 'LC_CTYPE', 'LANG'))): PREFER_ENC = 'UTF-8'; diff --git a/fail2ban/server/action.py b/fail2ban/server/action.py index f06e28c0..699d95eb 100644 --- a/fail2ban/server/action.py +++ b/fail2ban/server/action.py @@ -312,7 +312,7 @@ class CommandAction(ActionBase): def __setattr__(self, name, value): if not name.startswith('_') and not self.__init and not callable(value): - # special case for some pasrameters: + # special case for some parameters: if name in ('timeout', 'bantime'): value = str(MyTime.str2seconds(value)) # parameters changed - clear properties and substitution cache: @@ -337,7 +337,7 @@ class CommandAction(ActionBase): def _properties(self): """A dictionary of the actions properties. - This is used to subsitute "tags" in the commands. + This is used to substitute "tags" in the commands. """ # if we have a properties - return it: if self.__properties is not None: @@ -460,13 +460,13 @@ class CommandAction(ActionBase): """Executes the "actionflush" command. Command executed in order to flush all bans at once (e. g. by stop/shutdown - the system), instead of unbunning of each single ticket. + the system), instead of unbanning of each single ticket. Replaces the tags in the action command with actions properties and executes the resulting command. """ family = [] - # cumulate started families, if started on demand (conditional): + # collect started families, if started on demand (conditional): if self._startOnDemand: for f in CommandAction.COND_FAMILIES: if self.__started.get(f) == 1: # only real started: @@ -482,7 +482,7 @@ class CommandAction(ActionBase): and executes the resulting command. """ family = [] - # cumulate started families, if started on demand (conditional): + # collect started families, if started on demand (conditional): if self._startOnDemand: for f in CommandAction.COND_FAMILIES: if self.__started.get(f) == 1: # only real started: diff --git a/fail2ban/server/actions.py b/fail2ban/server/actions.py index fd4263c3..d90fca03 100644 --- a/fail2ban/server/actions.py +++ b/fail2ban/server/actions.py @@ -415,7 +415,7 @@ class Actions(JailThread, Mapping): diftm = ticket.getTime() - bTicket.getTime() # log already banned with following level: # DEBUG - before 3 seconds - certain interval for it, because of possible latency by recognizing in backends, etc. - # NOTICE - before 60 seconds - may still occurre if action are slow, or very high load in backend, + # NOTICE - before 60 seconds - may still occur if action is slow, or very high load in backend, # WARNING - after 60 seconds - very long time, something may be wrong ll = logging.DEBUG if diftm < 3 \ else logging.NOTICE if diftm < 60 \ diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index 25a0e687..9494b869 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -927,11 +927,11 @@ class FileFilter(Filter): if e.errno != 2: # errno.ENOENT logSys.exception(e) return False - except OSError as e: # pragma: no cover - requires race condition to tigger this + except OSError as e: # pragma: no cover - requires race condition to trigger this logSys.error("Error opening %s", filename) logSys.exception(e) return False - except Exception as e: # pragma: no cover - Requires implemention error in FileContainer to generate + except Exception as e: # pragma: no cover - Requires implementation error in FileContainer to generate logSys.error("Internal error in FileContainer open method - please report as a bug to https://github.com/fail2ban/fail2ban/issues") logSys.exception(e) return False @@ -1039,7 +1039,7 @@ class FileFilter(Filter): movecntr -= 1 if movecntr <= 0: break - # we have found large area without any date mached + # we have found large area without any date matched # or end of search - try min position (because can be end of previous line): if minp != lastPos: lastPos = tryPos = minp diff --git a/fail2ban/server/filterpoll.py b/fail2ban/server/filterpoll.py index 62cf76f0..5905c5b5 100644 --- a/fail2ban/server/filterpoll.py +++ b/fail2ban/server/filterpoll.py @@ -158,7 +158,7 @@ class FilterPoll(FileFilter): self.__prevStats[filename] = stats return True except Exception as e: - # stil alive (may be deleted because multi-threaded): + # still alive (may be deleted because multi-threaded): if not self.getLog(filename) or self.__prevStats.get(filename) is None: logSys.warning("Log %r seems to be down: %s", filename, e) return False diff --git a/fail2ban/tests/config/filter.d/test.conf b/fail2ban/tests/config/filter.d/test.conf index 9d08ef09..736c6297 100644 --- a/fail2ban/tests/config/filter.d/test.conf +++ b/fail2ban/tests/config/filter.d/test.conf @@ -9,5 +9,5 @@ where = conf failregex = failure <_daemon> (filter.d/test.%(where)s) [Init] -# test parameter, should be overriden in jail by "filter=test[one=1,...]" +# test parameter, should be overridden in jail by "filter=test[one=1,...]" one = *1* From 527bb9a7c35d1d59f43d6734a5112e4fe548b295 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 23 Jan 2018 08:48:36 -0500 Subject: [PATCH 008/189] dos2unix for helpers-common.conf Original report: http://bugs.debian.org/888110 --- config/action.d/helpers-common.conf | 32 ++++++++++++++--------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/config/action.d/helpers-common.conf b/config/action.d/helpers-common.conf index 5799d9d3..b036f68f 100644 --- a/config/action.d/helpers-common.conf +++ b/config/action.d/helpers-common.conf @@ -1,16 +1,16 @@ -[DEFAULT] - -# Usage: -# _grep_logs_args = 'test' -# (printf %%b "Log-excerpt contains 'test':\n"; %(_grep_logs)s; printf %%b "Log-excerpt contains 'test':\n") | mail ... -# -_grep_logs = logpath=""; grep -E %(_grep_logs_args)s $logpath | -_grep_logs_args = "(^|[^0-9a-fA-F:])$(echo '' | sed 's/\./\\./g')([^0-9a-fA-F:]|$)" - -# Used for actions, that should not by executed if ticket was restored: -_bypass_if_restored = if [ '' = '1' ]; then exit 0; fi; - -[Init] -greplimit = tail -n -grepmax = 1000 -grepopts = -m +[DEFAULT] + +# Usage: +# _grep_logs_args = 'test' +# (printf %%b "Log-excerpt contains 'test':\n"; %(_grep_logs)s; printf %%b "Log-excerpt contains 'test':\n") | mail ... +# +_grep_logs = logpath=""; grep -E %(_grep_logs_args)s $logpath | +_grep_logs_args = "(^|[^0-9a-fA-F:])$(echo '' | sed 's/\./\\./g')([^0-9a-fA-F:]|$)" + +# Used for actions, that should not by executed if ticket was restored: +_bypass_if_restored = if [ '' = '1' ]; then exit 0; fi; + +[Init] +greplimit = tail -n +grepmax = 1000 +grepopts = -m From d7e320b96d1339d19331d4cad05e77493e1ce669 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Allan=20Nordh=C3=B8y?= Date: Tue, 23 Jan 2018 21:09:53 +0100 Subject: [PATCH 009/189] reverting linux indentation --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d97cdf38..0f08b3e6 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@ Required: Optional: - [pyinotify >= 0.8.3](https://github.com/seb-m/pyinotify) -- Linux >= 2.6.13 +- ↓ Linux >= 2.6.13 - [gamin >= 0.0.21](https://www.gnome.org/~veillard/gamin) - [systemd >= 204](https://www.freedesktop.org/wiki/Software/systemd) - [dnspython](http://www.dnspython.org/) From 2f0bc491e23e08e61905fdea71b94d864bf40de6 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Sun, 21 Jan 2018 20:15:56 -0500 Subject: [PATCH 010/189] BF: use tests.utils.CONFIG_DIR instead of hardcoded "config" in fail2banclienttestcase Since otherwise cannot provide custom path to the config via env var and thus cannot test in a build directory which is out of source --- fail2ban/tests/fail2banclienttestcase.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fail2ban/tests/fail2banclienttestcase.py b/fail2ban/tests/fail2banclienttestcase.py index 1e2d1b33..65653a8e 100644 --- a/fail2ban/tests/fail2banclienttestcase.py +++ b/fail2ban/tests/fail2banclienttestcase.py @@ -44,13 +44,13 @@ from ..server import server from ..server.mytime import MyTime from ..server.utils import Utils from .utils import LogCaptureTestCase, logSys as DefLogSys, with_tmpdir, shutil, logging +from .utils import CONFIG_DIR as STOCK_CONF_DIR from ..helpers import getLogger # Gets the instance of the logger. logSys = getLogger(__name__) -STOCK_CONF_DIR = "config" STOCK = exists(pjoin(STOCK_CONF_DIR, 'fail2ban.conf')) CLIENT = "fail2ban-client" From 49be8de902eaff8351cba6626083513843fae1c9 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Sun, 21 Jan 2018 21:05:32 -0500 Subject: [PATCH 011/189] BF: look for system.journal also under system-state-logs (i.e. /var/log) as it happens on Debian systems --- fail2ban/tests/filtertestcase.py | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 1803974b..b707beb5 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -1173,11 +1173,22 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover super(MonitorJournalFailures, self).tearDown() def _getRuntimeJournal(self): - # retrieve current system journal path - tmp = Utils.executeCmd('find "$(systemd-path system-runtime-logs)" -name system.journal', - timeout=10, shell=True, output=True); - self.assertTrue(tmp) - return str(tmp[1].decode('utf-8')).split('\n')[0] + """Retrieve current system journal path + + If none found, None will be returned + """ + # Depending on the system, it could be found under /run or /var/log (e.g. Debian) + # which are pointed by different systemd-path variables. We will + # check one at at time until the first hit + for systemd_var in 'system-runtime-logs', 'system-state-logs': + tmp = Utils.executeCmd( + 'find "$(systemd-path %s)" -name system.journal' % systemd_var, + timeout=10, shell=True, output=True + ) + self.assertTrue(tmp) + out = str(tmp[1].decode('utf-8')).split('\n')[0] + if out: + return out def testJournalFilesArg(self): # retrieve current system journal path From a5b9128fccb6d35fbc22bb79ea81a9b529486b64 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Sun, 21 Jan 2018 22:01:24 -0500 Subject: [PATCH 012/189] BF: RF test for "being a root" to check if actually can read the file --- fail2ban/tests/filtertestcase.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index b707beb5..40cf51f9 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -675,7 +675,15 @@ class LogFileMonitor(LogCaptureTestCase): os.chmod(self.name, 0) self.filter.getFailures(self.name) failure_was_logged = self._is_logged('Unable to open %s' % self.name) - is_root = getpass.getuser() == 'root' + # verify that we cannot access the file. Checking by name of user is not + # sufficient since could be a fakeroot or some other super-user + try: + with open(self.name) as f: + f.read() + is_root = True + except IOError: + is_root = False + # If ran as root, those restrictive permissions would not # forbid log to be read. self.assertTrue(failure_was_logged != is_root) From 3f51c158cd7ead5bec90e5d18744d45a2c64712c Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Sun, 21 Jan 2018 23:25:26 -0500 Subject: [PATCH 013/189] Added manpage (still would need tuning) for fail2ban-python --- man/fail2ban-python.1 | 72 +++++++++++++++++++++++++++++++++++++++++ man/fail2ban-python.h2m | 9 ++++++ man/generate-man | 7 ++++ 3 files changed, 88 insertions(+) create mode 100644 man/fail2ban-python.1 create mode 100644 man/fail2ban-python.h2m diff --git a/man/fail2ban-python.1 b/man/fail2ban-python.1 new file mode 100644 index 00000000..e67e468a --- /dev/null +++ b/man/fail2ban-python.1 @@ -0,0 +1,72 @@ +.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.5. +.TH FAIL2BAN-PYTHON "1" "January 2018" "fail2ban-python f2bversion" "User Commands" +.SH NAME +fail2ban-python \- a helper for Fail2Ban to assure that the same Python is used +.SH DESCRIPTION +usage: ../bin/fail2ban\-python [option] ... [\-c cmd | \fB\-m\fR mod | file | \fB\-]\fR [arg] ... +Options and arguments (and corresponding environment variables): +\fB\-b\fR : issue warnings about comparing bytearray with unicode +.IP +(\fB\-bb\fR: issue errors) +.PP +\fB\-B\fR : don't write .py[co] files on import; also PYTHONDONTWRITEBYTECODE=x +\fB\-c\fR cmd : program passed in as string (terminates option list) +\fB\-d\fR : debug output from parser; also PYTHONDEBUG=x +\fB\-E\fR : ignore PYTHON* environment variables (such as PYTHONPATH) +\fB\-h\fR : print this help message and exit (also \fB\-\-help\fR) +\fB\-i\fR : inspect interactively after running script; forces a prompt even +.IP +if stdin does not appear to be a terminal; also PYTHONINSPECT=x +.PP +\fB\-m\fR mod : run library module as a script (terminates option list) +\fB\-O\fR : optimize generated bytecode slightly; also PYTHONOPTIMIZE=x +\fB\-OO\fR : remove doc\-strings in addition to the \fB\-O\fR optimizations +\fB\-R\fR : use a pseudo\-random salt to make hash() values of various types be +.IP +unpredictable between separate invocations of the interpreter, as +a defense against denial\-of\-service attacks +.PP +\fB\-Q\fR arg : division options: \fB\-Qold\fR (default), \fB\-Qwarn\fR, \fB\-Qwarnall\fR, \fB\-Qnew\fR +\fB\-s\fR : don't add user site directory to sys.path; also PYTHONNOUSERSITE +\fB\-S\fR : don't imply 'import site' on initialization +\fB\-t\fR : issue warnings about inconsistent tab usage (\fB\-tt\fR: issue errors) +\fB\-u\fR : unbuffered binary stdout and stderr; also PYTHONUNBUFFERED=x +.IP +see man page for details on internal buffering relating to '\-u' +.PP +\fB\-v\fR : verbose (trace import statements); also PYTHONVERBOSE=x +.IP +can be supplied multiple times to increase verbosity +.PP +\fB\-V\fR : print the Python version number and exit (also \fB\-\-version\fR) +\fB\-W\fR arg : warning control; arg is action:message:category:module:lineno +.IP +also PYTHONWARNINGS=arg +.PP +\fB\-x\fR : skip first line of source, allowing use of non\-Unix forms of #!cmd +\fB\-3\fR : warn about Python 3.x incompatibilities that 2to3 cannot trivially fix +file : program read from script file +\- : program read from stdin (default; interactive mode if a tty) +arg ...: arguments passed to program in sys.argv[1:] +.PP +Other environment variables: +PYTHONSTARTUP: file executed on interactive startup (no default) +PYTHONPATH : ':'\-separated list of directories prefixed to the +.TP +default module search path. +The result is sys.path. +.PP +PYTHONHOME : alternate directory (or :). +.IP +The default module search path uses /pythonX.X. +.PP +PYTHONCASEOK : ignore case in 'import' statements (Windows). +PYTHONIOENCODING: Encoding[:errors] used for stdin/stdout/stderr. +PYTHONHASHSEED: if this variable is set to 'random', the effect is the same +.IP +as specifying the \fB\-R\fR option: a random value is used to seed the hashes of +str, bytes and datetime objects. It can also be set to an integer +in the range [0,4294967295] to get hash values with a predictable seed. +.SH "SEE ALSO" +.br +fail2ban-client(1) diff --git a/man/fail2ban-python.h2m b/man/fail2ban-python.h2m new file mode 100644 index 00000000..ab4c3e65 --- /dev/null +++ b/man/fail2ban-python.h2m @@ -0,0 +1,9 @@ +Include file for help2man man page +$Id: $ + +[name] +fail2ban-python \- a helper for Fail2Ban to assure that the same Python is used + +[see also] +.br +fail2ban-client(1) diff --git a/man/generate-man b/man/generate-man index 1ce73f76..f7a7836c 100755 --- a/man/generate-man +++ b/man/generate-man @@ -4,6 +4,8 @@ set -eu export PYTHONPATH=.. +f2bversion=$(../bin/fail2ban-client --version | head -n1 | sed -e 's,.* v,,g') + # fail2ban-client echo -n "Generating fail2ban-client " help2man --section=1 --no-info --include=fail2ban-client.h2m --output fail2ban-client.1 ../bin/fail2ban-client @@ -35,6 +37,11 @@ for LINE in $LINES; do done echo "[done]" +# fail2ban-python +echo -n "Generating fail2ban-python " +help2man --version-string=f2bversion --section=1 --no-info --include=fail2ban-python.h2m --output fail2ban-python.1 ../bin/fail2ban-python +echo "[done]" + # fail2ban-server echo -n "Generating fail2ban-server " help2man --section=1 --no-info --include=fail2ban-server.h2m --output fail2ban-server.1 ../bin/fail2ban-server From 9af9ec25f5feb82f3aa0e7a1a8ae3e67a0215ed8 Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 22 Jan 2018 13:18:32 +0100 Subject: [PATCH 014/189] allow to override use_stock_cfg values (used as default value now), e. g. actions rest filters only, reject d1afbb566f0304487b5d578b4aacef8e647ee74b --- fail2ban/tests/fail2banclienttestcase.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fail2ban/tests/fail2banclienttestcase.py b/fail2ban/tests/fail2banclienttestcase.py index 65653a8e..e2cb8ab4 100644 --- a/fail2ban/tests/fail2banclienttestcase.py +++ b/fail2ban/tests/fail2banclienttestcase.py @@ -153,7 +153,7 @@ def _start_params(tmp, use_stock=False, use_stock_cfg=None, """Filters list of 'files' to contain only directories (under dir)""" return [f for f in files if isdir(pjoin(dir, f))] shutil.copytree(STOCK_CONF_DIR, cfg, ignore=ig_dirs) - use_stock_cfg = ('action.d', 'filter.d') + if use_stock_cfg is None: use_stock_cfg = ('action.d', 'filter.d') # replace fail2ban params (database with memory): r = re.compile(r'^dbfile\s*=') for line in fileinput.input(pjoin(cfg, "fail2ban.conf"), inplace=True): From 7a757645bb428510f76fc12eda87f96e6773b081 Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 22 Jan 2018 14:40:04 +0100 Subject: [PATCH 015/189] introduces new decorator/conditional helper in order to skip some STOCK-related test-cases (if running outside of stock-config environment). --- fail2ban/tests/action_d/test_smtp.py | 1 + fail2ban/tests/clientreadertestcase.py | 415 +++--- fail2ban/tests/fail2banclienttestcase.py | 7 +- fail2ban/tests/filtertestcase.py | 10 +- fail2ban/tests/servertestcase.py | 1594 +++++++++++----------- fail2ban/tests/utils.py | 28 + 6 files changed, 1042 insertions(+), 1013 deletions(-) diff --git a/fail2ban/tests/action_d/test_smtp.py b/fail2ban/tests/action_d/test_smtp.py index d0858b85..8d20055a 100644 --- a/fail2ban/tests/action_d/test_smtp.py +++ b/fail2ban/tests/action_d/test_smtp.py @@ -52,6 +52,7 @@ class SMTPActionTest(unittest.TestCase): def setUp(self): """Call before every test case.""" + unittest.F2B.SkipIfCfgMissing(action='smtp.py') super(SMTPActionTest, self).setUp() self.jail = DummyJail() pythonModule = os.path.join(CONFIG_DIR, "action.d", "smtp.py") diff --git a/fail2ban/tests/clientreadertestcase.py b/fail2ban/tests/clientreadertestcase.py index e8cd2912..0472b770 100644 --- a/fail2ban/tests/clientreadertestcase.py +++ b/fail2ban/tests/clientreadertestcase.py @@ -45,8 +45,6 @@ TEST_FILES_DIR_SHARE_CFG = {} from .utils import CONFIG_DIR CONFIG_DIR_SHARE_CFG = unittest.F2B.share_config -STOCK = os.path.exists(os.path.join('config', 'fail2ban.conf')) - IMPERFECT_CONFIG = os.path.join(os.path.dirname(__file__), 'config') IMPERFECT_CONFIG_SHARE_CFG = {} @@ -246,15 +244,15 @@ class JailReaderTest(LogCaptureTestCase): self.assertTrue(jail.isEnabled()) self.assertLogged("Invalid filter definition 'flt[test'") - if STOCK: - def testStockSSHJail(self): - jail = JailReader('sshd', basedir=CONFIG_DIR, share_config=CONFIG_DIR_SHARE_CFG) # we are running tests from root project dir atm - self.assertTrue(jail.read()) - self.assertTrue(jail.getOptions()) - self.assertFalse(jail.isEnabled()) - self.assertEqual(jail.getName(), 'sshd') - jail.setName('ssh-funky-blocker') - self.assertEqual(jail.getName(), 'ssh-funky-blocker') + def testStockSSHJail(self): + unittest.F2B.SkipIfCfgMissing(stock=True) + jail = JailReader('sshd', basedir=CONFIG_DIR, share_config=CONFIG_DIR_SHARE_CFG) # we are running tests from root project dir atm + self.assertTrue(jail.read()) + self.assertTrue(jail.getOptions()) + self.assertFalse(jail.isEnabled()) + self.assertEqual(jail.getName(), 'sshd') + jail.setName('ssh-funky-blocker') + self.assertEqual(jail.getName(), 'ssh-funky-blocker') def testSplitOption(self): # Simple example @@ -307,6 +305,7 @@ class JailReaderTest(LogCaptureTestCase): self.assertEqual(expected2, result) def testVersionAgent(self): + unittest.F2B.SkipIfCfgMissing(stock=True) jail = JailReader('blocklisttest', force_enable=True, basedir=CONFIG_DIR) # emulate jail.read(), because such jail not exists: ConfigReader.read(jail, "jail"); @@ -597,222 +596,226 @@ class JailsReaderTest(LogCaptureTestCase): self.assertNotLogged("Skipping...") self.assertLogged("No file(s) found for glob /weapons/of/mass/destruction") - if STOCK: - def testReadStockActionConf(self): - for actionConfig in glob.glob(os.path.join(CONFIG_DIR, 'action.d', '*.conf')): - actionName = os.path.basename(actionConfig).replace('.conf', '') - actionReader = ActionReader(actionName, "TEST", {}, basedir=CONFIG_DIR) - self.assertTrue(actionReader.read()) - try: - actionReader.getOptions({}) # populate _opts - except Exception as e: # pragma: no cover - self.fail("action %r\n%s: %s" % (actionName, type(e).__name__, e)) - if not actionName.endswith('-common'): - self.assertIn('Definition', actionReader.sections(), - msg="Action file %r is lacking [Definition] section" % actionConfig) - # all must have some actionban defined - self.assertTrue(actionReader._opts.get('actionban', '').strip(), - msg="Action file %r is lacking actionban" % actionConfig) - # test name of jail is set in options (also if not supplied within parameters): - opts = actionReader.getCombined( - ignore=CommandAction._escapedTags | set(('timeout', 'bantime'))) - self.assertEqual(opts.get('name'), 'TEST', - msg="Action file %r does not contains jail-name 'f2b-TEST'" % actionConfig) - # and the name is substituted (test several actions surely contains name-interpolation): - if actionName in ('pf', 'iptables-allports', 'iptables-multiport'): - #print('****', actionName, opts.get('actionstart', '')) - self.assertIn('f2b-TEST', opts.get('actionstart', ''), - msg="Action file %r: interpolation of actionstart does not contains jail-name 'f2b-TEST'" % actionConfig) + def testReadStockActionConf(self): + unittest.F2B.SkipIfCfgMissing(stock=True) + for actionConfig in glob.glob(os.path.join(CONFIG_DIR, 'action.d', '*.conf')): + actionName = os.path.basename(actionConfig).replace('.conf', '') + actionReader = ActionReader(actionName, "TEST", {}, basedir=CONFIG_DIR) + self.assertTrue(actionReader.read()) + try: + actionReader.getOptions({}) # populate _opts + except Exception as e: # pragma: no cover + self.fail("action %r\n%s: %s" % (actionName, type(e).__name__, e)) + if not actionName.endswith('-common'): + self.assertIn('Definition', actionReader.sections(), + msg="Action file %r is lacking [Definition] section" % actionConfig) + # all must have some actionban defined + self.assertTrue(actionReader._opts.get('actionban', '').strip(), + msg="Action file %r is lacking actionban" % actionConfig) + # test name of jail is set in options (also if not supplied within parameters): + opts = actionReader.getCombined( + ignore=CommandAction._escapedTags | set(('timeout', 'bantime'))) + self.assertEqual(opts.get('name'), 'TEST', + msg="Action file %r does not contains jail-name 'f2b-TEST'" % actionConfig) + # and the name is substituted (test several actions surely contains name-interpolation): + if actionName in ('pf', 'iptables-allports', 'iptables-multiport'): + #print('****', actionName, opts.get('actionstart', '')) + self.assertIn('f2b-TEST', opts.get('actionstart', ''), + msg="Action file %r: interpolation of actionstart does not contains jail-name 'f2b-TEST'" % actionConfig) - def testReadStockJailConf(self): - jails = JailsReader(basedir=CONFIG_DIR, share_config=CONFIG_DIR_SHARE_CFG) # we are running tests from root project dir atm - self.assertTrue(jails.read()) # opens fine - self.assertTrue(jails.getOptions()) # reads fine - comm_commands = jails.convert() - # by default None of the jails is enabled and we get no - # commands to communicate to the server - self.assertEqual(comm_commands, []) + def testReadStockJailConf(self): + unittest.F2B.SkipIfCfgMissing(stock=True) + jails = JailsReader(basedir=CONFIG_DIR, share_config=CONFIG_DIR_SHARE_CFG) # we are running tests from root project dir atm + self.assertTrue(jails.read()) # opens fine + self.assertTrue(jails.getOptions()) # reads fine + comm_commands = jails.convert() + # by default None of the jails is enabled and we get no + # commands to communicate to the server + self.assertEqual(comm_commands, []) - # TODO: make sure this is handled well - ## We should not "read" some bogus jail - #old_comm_commands = comm_commands[:] # make a copy - #self.assertRaises(ValueError, jails.getOptions, "BOGUS") - #self.printLog() - #self.assertLogged("No section: 'BOGUS'") - ## and there should be no side-effects - #self.assertEqual(jails.convert(), old_comm_commands) + # TODO: make sure this is handled well + ## We should not "read" some bogus jail + #old_comm_commands = comm_commands[:] # make a copy + #self.assertRaises(ValueError, jails.getOptions, "BOGUS") + #self.printLog() + #self.assertLogged("No section: 'BOGUS'") + ## and there should be no side-effects + #self.assertEqual(jails.convert(), old_comm_commands) - allFilters = set() + allFilters = set() - # All jails must have filter and action set - # TODO: evolve into a parametric test - for jail in jails.sections(): - if jail == 'INCLUDES': - continue - filterName = jails.get(jail, 'filter') - filterName, filterOpt = extractOptions(filterName) - allFilters.add(filterName) - self.assertTrue(len(filterName)) - # moreover we must have a file for it - # and it must be readable as a Filter - filterReader = FilterReader(filterName, jail, filterOpt, + # All jails must have filter and action set + # TODO: evolve into a parametric test + for jail in jails.sections(): + if jail == 'INCLUDES': + continue + filterName = jails.get(jail, 'filter') + filterName, filterOpt = extractOptions(filterName) + allFilters.add(filterName) + self.assertTrue(len(filterName)) + # moreover we must have a file for it + # and it must be readable as a Filter + filterReader = FilterReader(filterName, jail, filterOpt, + share_config=CONFIG_DIR_SHARE_CFG, basedir=CONFIG_DIR) + self.assertTrue(filterReader.read(),"Failed to read filter:" + filterName) # opens fine + filterReader.getOptions({}) # reads fine + + # test if filter has failregex set + self.assertTrue(filterReader._opts.get('failregex', '').strip()) + + actions = jails.get(jail, 'action') + self.assertTrue(len(actions.strip())) + + # somewhat duplicating here what is done in JailsReader if + # the jail is enabled + for act in actions.split('\n'): + actName, actOpt = extractOptions(act) + self.assertTrue(len(actName)) + self.assertTrue(isinstance(actOpt, dict)) + if actName == 'iptables-multiport': + self.assertIn('port', actOpt) + + actionReader = ActionReader(actName, jail, {}, share_config=CONFIG_DIR_SHARE_CFG, basedir=CONFIG_DIR) - self.assertTrue(filterReader.read(),"Failed to read filter:" + filterName) # opens fine - filterReader.getOptions({}) # reads fine + self.assertTrue(actionReader.read()) + actionReader.getOptions({}) # populate _opts + cmds = actionReader.convert() + self.assertTrue(len(cmds)) - # test if filter has failregex set - self.assertTrue(filterReader._opts.get('failregex', '').strip()) + # all must have some actionban + self.assertTrue(actionReader._opts.get('actionban', '').strip()) - actions = jails.get(jail, 'action') - self.assertTrue(len(actions.strip())) + # Verify that all filters found under config/ have a jail + def testReadStockJailFilterComplete(self): + unittest.F2B.SkipIfCfgMissing(stock=True) + jails = JailsReader(basedir=CONFIG_DIR, force_enable=True, share_config=CONFIG_DIR_SHARE_CFG) + self.assertTrue(jails.read()) # opens fine + self.assertTrue(jails.getOptions()) # reads fine + # grab all filter names + filters = set(os.path.splitext(os.path.split(a)[1])[0] + for a in glob.glob(os.path.join('config', 'filter.d', '*.conf')) + if not (a.endswith('common.conf') or a.endswith('-aggressive.conf'))) + # get filters of all jails (filter names without options inside filter[...]) + filters_jail = set( + extractOptions(jail.options['filter'])[0] for jail in jails.jails + ) + self.maxDiff = None + self.assertTrue(filters.issubset(filters_jail), + "More filters exists than are referenced in stock jail.conf %r" % filters.difference(filters_jail)) + self.assertTrue(filters_jail.issubset(filters), + "Stock jail.conf references non-existent filters %r" % filters_jail.difference(filters)) - # somewhat duplicating here what is done in JailsReader if - # the jail is enabled - for act in actions.split('\n'): - actName, actOpt = extractOptions(act) - self.assertTrue(len(actName)) - self.assertTrue(isinstance(actOpt, dict)) - if actName == 'iptables-multiport': - self.assertIn('port', actOpt) + def testReadStockJailConfForceEnabled(self): + unittest.F2B.SkipIfCfgMissing(stock=True) + # more of a smoke test to make sure that no obvious surprises + # on users' systems when enabling shipped jails + jails = JailsReader(basedir=CONFIG_DIR, force_enable=True, share_config=CONFIG_DIR_SHARE_CFG) # we are running tests from root project dir atm + self.assertTrue(jails.read()) # opens fine + self.assertTrue(jails.getOptions()) # reads fine + comm_commands = jails.convert(allow_no_files=True) - actionReader = ActionReader(actName, jail, {}, - share_config=CONFIG_DIR_SHARE_CFG, basedir=CONFIG_DIR) - self.assertTrue(actionReader.read()) - actionReader.getOptions({}) # populate _opts - cmds = actionReader.convert() - self.assertTrue(len(cmds)) + # by default we have lots of jails ;) + self.assertTrue(len(comm_commands)) - # all must have some actionban - self.assertTrue(actionReader._opts.get('actionban', '').strip()) + # some common sanity checks for commands + for command in comm_commands: + if len(command) >= 3 and [command[0], command[2]] == ['set', 'bantime']: + self.assertTrue(MyTime.str2seconds(command[3]) > 0) + - # Verify that all filters found under config/ have a jail - def testReadStockJailFilterComplete(self): - jails = JailsReader(basedir=CONFIG_DIR, force_enable=True, share_config=CONFIG_DIR_SHARE_CFG) - self.assertTrue(jails.read()) # opens fine - self.assertTrue(jails.getOptions()) # reads fine - # grab all filter names - filters = set(os.path.splitext(os.path.split(a)[1])[0] - for a in glob.glob(os.path.join('config', 'filter.d', '*.conf')) - if not (a.endswith('common.conf') or a.endswith('-aggressive.conf'))) - # get filters of all jails (filter names without options inside filter[...]) - filters_jail = set( - extractOptions(jail.options['filter'])[0] for jail in jails.jails - ) - self.maxDiff = None - self.assertTrue(filters.issubset(filters_jail), - "More filters exists than are referenced in stock jail.conf %r" % filters.difference(filters_jail)) - self.assertTrue(filters_jail.issubset(filters), - "Stock jail.conf references non-existent filters %r" % filters_jail.difference(filters)) + # and we know even some of them by heart + for j in ['sshd', 'recidive']: + # by default we have 'auto' backend ATM, but some distributions can overwrite it, + # (e.g. fedora default is 'systemd') therefore let check it without backend... + self.assertIn(['add', j], + (cmd[:2] for cmd in comm_commands if len(cmd) == 3 and cmd[0] == 'add')) + # and warn on useDNS + self.assertIn(['set', j, 'usedns', 'warn'], comm_commands) + self.assertIn(['start', j], comm_commands) - def testReadStockJailConfForceEnabled(self): - # more of a smoke test to make sure that no obvious surprises - # on users' systems when enabling shipped jails - jails = JailsReader(basedir=CONFIG_DIR, force_enable=True, share_config=CONFIG_DIR_SHARE_CFG) # we are running tests from root project dir atm - self.assertTrue(jails.read()) # opens fine - self.assertTrue(jails.getOptions()) # reads fine - comm_commands = jails.convert(allow_no_files=True) + # last commands should be the 'start' commands + self.assertEqual(comm_commands[-1][0], 'start') - # by default we have lots of jails ;) - self.assertTrue(len(comm_commands)) + for j in jails._JailsReader__jails: + actions = j._JailReader__actions + jail_name = j.getName() + # make sure that all of the jails have actions assigned, + # otherwise it makes little to no sense + self.assertTrue(len(actions), + msg="No actions found for jail %s" % jail_name) - # some common sanity checks for commands - for command in comm_commands: - if len(command) >= 3 and [command[0], command[2]] == ['set', 'bantime']: - self.assertTrue(MyTime.str2seconds(command[3]) > 0) - + # Test for presence of blocktype (in relation to gh-232) + for action in actions: + commands = action.convert() + action_name = action.getName() + if '' in str(commands): + # Verify that it is among cInfo + self.assertIn('blocktype', action._initOpts) + # Verify that we have a call to set it up + blocktype_present = False + target_command = [jail_name, 'action', action_name] + for command in commands: + if (len(command) > 4 and command[0] == 'multi-set' and + command[1:4] == target_command): + blocktype_present = ('blocktype' in [cmd[0] for cmd in command[4]]) + elif (len(command) > 5 and command[0] == 'set' and + command[1:4] == target_command and command[4] == 'blocktype'): # pragma: no cover - because of multi-set + blocktype_present = True + if blocktype_present: + break + self.assertTrue( + blocktype_present, + msg="Found no %s command among %s" + % (target_command, str(commands)) ) - # and we know even some of them by heart - for j in ['sshd', 'recidive']: - # by default we have 'auto' backend ATM, but some distributions can overwrite it, - # (e.g. fedora default is 'systemd') therefore let check it without backend... - self.assertIn(['add', j], - (cmd[:2] for cmd in comm_commands if len(cmd) == 3 and cmd[0] == 'add')) - # and warn on useDNS - self.assertIn(['set', j, 'usedns', 'warn'], comm_commands) - self.assertIn(['start', j], comm_commands) + def testStockConfigurator(self): + unittest.F2B.SkipIfCfgMissing(stock=True) + configurator = Configurator() + configurator.setBaseDir(CONFIG_DIR) + self.assertEqual(configurator.getBaseDir(), CONFIG_DIR) - # last commands should be the 'start' commands - self.assertEqual(comm_commands[-1][0], 'start') + configurator.readEarly() + opts = configurator.getEarlyOptions() + # our current default settings + self.assertEqual(opts['socket'], '/var/run/fail2ban/fail2ban.sock') + self.assertEqual(opts['pidfile'], '/var/run/fail2ban/fail2ban.pid') - for j in jails._JailsReader__jails: - actions = j._JailReader__actions - jail_name = j.getName() - # make sure that all of the jails have actions assigned, - # otherwise it makes little to no sense - self.assertTrue(len(actions), - msg="No actions found for jail %s" % jail_name) + configurator.readAll() + configurator.getOptions() + configurator.convertToProtocol() + commands = configurator.getConfigStream() - # Test for presence of blocktype (in relation to gh-232) - for action in actions: - commands = action.convert() - action_name = action.getName() - if '' in str(commands): - # Verify that it is among cInfo - self.assertIn('blocktype', action._initOpts) - # Verify that we have a call to set it up - blocktype_present = False - target_command = [jail_name, 'action', action_name] - for command in commands: - if (len(command) > 4 and command[0] == 'multi-set' and - command[1:4] == target_command): - blocktype_present = ('blocktype' in [cmd[0] for cmd in command[4]]) - elif (len(command) > 5 and command[0] == 'set' and - command[1:4] == target_command and command[4] == 'blocktype'): # pragma: no cover - because of multi-set - blocktype_present = True - if blocktype_present: - break - self.assertTrue( - blocktype_present, - msg="Found no %s command among %s" - % (target_command, str(commands)) ) + # verify that dbfile comes before dbpurgeage + def find_set(option): + for i, e in enumerate(commands): + if e[0] == 'set' and e[1] == option: + return i + raise ValueError("Did not find command 'set %s' among commands %s" + % (option, commands)) - def testStockConfigurator(self): - configurator = Configurator() - configurator.setBaseDir(CONFIG_DIR) - self.assertEqual(configurator.getBaseDir(), CONFIG_DIR) + # Set up of logging should come first + self.assertEqual(find_set('syslogsocket'), 0) + self.assertEqual(find_set('loglevel'), 1) + self.assertEqual(find_set('logtarget'), 2) + # then dbfile should be before dbpurgeage + self.assertTrue(find_set('dbpurgeage') > find_set('dbfile')) - configurator.readEarly() - opts = configurator.getEarlyOptions() - # our current default settings - self.assertEqual(opts['socket'], '/var/run/fail2ban/fail2ban.sock') - self.assertEqual(opts['pidfile'], '/var/run/fail2ban/fail2ban.pid') + # and there is logging information left to be passed into the + # server + self.assertSortedEqual(commands, + [['set', 'dbfile', + '/var/lib/fail2ban/fail2ban.sqlite3'], + ['set', 'dbpurgeage', '1d'], + ['set', 'loglevel', "INFO"], + ['set', 'logtarget', '/var/log/fail2ban.log'], + ['set', 'syslogsocket', 'auto']]) - configurator.readAll() - configurator.getOptions() - configurator.convertToProtocol() - commands = configurator.getConfigStream() - - # verify that dbfile comes before dbpurgeage - def find_set(option): - for i, e in enumerate(commands): - if e[0] == 'set' and e[1] == option: - return i - raise ValueError("Did not find command 'set %s' among commands %s" - % (option, commands)) - - # Set up of logging should come first - self.assertEqual(find_set('syslogsocket'), 0) - self.assertEqual(find_set('loglevel'), 1) - self.assertEqual(find_set('logtarget'), 2) - # then dbfile should be before dbpurgeage - self.assertTrue(find_set('dbpurgeage') > find_set('dbfile')) - - # and there is logging information left to be passed into the - # server - self.assertSortedEqual(commands, - [['set', 'dbfile', - '/var/lib/fail2ban/fail2ban.sqlite3'], - ['set', 'dbpurgeage', '1d'], - ['set', 'loglevel', "INFO"], - ['set', 'logtarget', '/var/log/fail2ban.log'], - ['set', 'syslogsocket', 'auto']]) - - # and if we force change configurator's fail2ban's baseDir - # there should be an error message (test visually ;) -- - # otherwise just a code smoke test) - configurator._Configurator__jails.setBaseDir('/tmp') - self.assertEqual(configurator._Configurator__jails.getBaseDir(), '/tmp') - self.assertEqual(configurator.getBaseDir(), CONFIG_DIR) + # and if we force change configurator's fail2ban's baseDir + # there should be an error message (test visually ;) -- + # otherwise just a code smoke test) + configurator._Configurator__jails.setBaseDir('/tmp') + self.assertEqual(configurator._Configurator__jails.getBaseDir(), '/tmp') + self.assertEqual(configurator.getBaseDir(), CONFIG_DIR) @with_tmpdir def testMultipleSameAction(self, basedir): diff --git a/fail2ban/tests/fail2banclienttestcase.py b/fail2ban/tests/fail2banclienttestcase.py index e2cb8ab4..8e175006 100644 --- a/fail2ban/tests/fail2banclienttestcase.py +++ b/fail2ban/tests/fail2banclienttestcase.py @@ -43,16 +43,14 @@ from .. import protocol from ..server import server from ..server.mytime import MyTime from ..server.utils import Utils -from .utils import LogCaptureTestCase, logSys as DefLogSys, with_tmpdir, shutil, logging -from .utils import CONFIG_DIR as STOCK_CONF_DIR +from .utils import LogCaptureTestCase, logSys as DefLogSys, with_tmpdir, shutil, logging, \ + STOCK, CONFIG_DIR as STOCK_CONF_DIR from ..helpers import getLogger # Gets the instance of the logger. logSys = getLogger(__name__) -STOCK = exists(pjoin(STOCK_CONF_DIR, 'fail2ban.conf')) - CLIENT = "fail2ban-client" SERVER = "fail2ban-server" BIN = dirname(Fail2banServer.getServerPath()) @@ -1171,6 +1169,7 @@ class Fail2banServerTest(Fail2banClientServerBase): "Jail 'test-jail1' started", all=True) # test action.d/nginx-block-map.conf -- + @unittest.F2B.skip_if_cfg_missing(action="nginx-block-map") @with_foreground_server_thread(startextra={ # create log-file (avoid "not found" errors): 'create_before_start': ('%(tmp)s/blck-failures.log',), diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 40cf51f9..59c7820a 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -24,7 +24,6 @@ __license__ = "GPL" from __builtin__ import open as fopen import unittest -import getpass import os import sys import time, datetime @@ -43,14 +42,12 @@ from ..server.failmanager import FailManagerEmpty from ..server.ipdns import DNSUtils, IPAddr from ..server.mytime import MyTime from ..server.utils import Utils, uni_decode -from .utils import setUpMyTime, tearDownMyTime, mtimesleep, with_tmpdir, LogCaptureTestCase +from .utils import setUpMyTime, tearDownMyTime, mtimesleep, with_tmpdir, LogCaptureTestCase, \ + CONFIG_DIR as STOCK_CONF_DIR from .dummyjail import DummyJail TEST_FILES_DIR = os.path.join(os.path.dirname(__file__), "files") -STOCK_CONF_DIR = "config" -STOCK = os.path.exists(os.path.join(STOCK_CONF_DIR, 'fail2ban.conf')) - # yoh: per Steven Hiscocks's insight while troubleshooting # https://github.com/fail2ban/fail2ban/issues/103#issuecomment-15542836 @@ -445,8 +442,7 @@ class IgnoreIPDNS(LogCaptureTestCase): self.assertFalse(self.filter.inIgnoreIPList("128.178.222.70")) def testIgnoreCmdApacheFakegooglebot(self): - if not STOCK: # pragma: no cover - raise unittest.SkipTest('Skip test because of no STOCK config') + unittest.F2B.SkipIfCfgMissing(stock=True) cmd = os.path.join(STOCK_CONF_DIR, "filter.d/ignorecommands/apache-fakegooglebot") ## below test direct as python module: mod = Utils.load_python_module(cmd) diff --git a/fail2ban/tests/servertestcase.py b/fail2ban/tests/servertestcase.py index 8485aa74..d18b660c 100644 --- a/fail2ban/tests/servertestcase.py +++ b/fail2ban/tests/servertestcase.py @@ -1034,7 +1034,7 @@ class LoggingTests(LogCaptureTestCase): os.remove(f) -from clientreadertestcase import ActionReader, JailsReader, CONFIG_DIR, STOCK +from clientreadertestcase import ActionReader, JailsReader, CONFIG_DIR class ServerConfigReaderTests(LogCaptureTestCase): @@ -1091,814 +1091,816 @@ class ServerConfigReaderTests(LogCaptureTestCase): logSys.debug('# === stop ==='); self.pruneLog() action.stop() - if STOCK: + def testCheckStockJailActions(self): + unittest.F2B.SkipIfCfgMissing(stock=True) + # we are running tests from root project dir atm + jails = JailsReader(basedir=CONFIG_DIR, force_enable=True, share_config=self.__share_cfg) + self.assertTrue(jails.read()) # opens fine + self.assertTrue(jails.getOptions()) # reads fine + stream = jails.convert(allow_no_files=True) - def testCheckStockJailActions(self): - # we are running tests from root project dir atm - jails = JailsReader(basedir=CONFIG_DIR, force_enable=True, share_config=self.__share_cfg) - self.assertTrue(jails.read()) # opens fine - self.assertTrue(jails.getOptions()) # reads fine - stream = jails.convert(allow_no_files=True) + server = TestServer() + transm = server._Server__transm + cmdHandler = transm._Transmitter__commandHandler - server = TestServer() - transm = server._Server__transm - cmdHandler = transm._Transmitter__commandHandler + # for cmd in stream: + # print(cmd) + + # filter all start commands (we want not start all jails): + for cmd in stream: + if cmd[0] != 'start': + # change to the fast init backend: + if cmd[0] == 'add': + cmd[2] = 'polling' + # change log path to test log of the jail + # (to prevent "Permission denied" on /var/logs/ for test-user): + elif len(cmd) > 3 and cmd[0] == 'set' and cmd[2] == 'addlogpath': + fn = os.path.join(TEST_FILES_DIR, 'logs', cmd[1]) + # fallback to testcase01 if jail has no its own test log-file + # (should not matter really): + if not os.path.exists(fn): # pragma: no cover + fn = os.path.join(TEST_FILES_DIR, 'testcase01.log') + cmd[3] = fn + # if fast add dummy regex to prevent too long compile of all regexp + # (we don't use it in this test at all): + elif unittest.F2B.fast and ( + len(cmd) > 3 and cmd[0] in ('set', 'multi-set') and cmd[2] == 'addfailregex' + ): # pragma: no cover + cmd[0] = "set" + cmd[3] = "DUMMY-REGEX " + # command to server, use cmdHandler direct instead of `transm.proceed(cmd)`: + try: + cmdHandler(cmd) + except Exception as e: # pragma: no cover + self.fail("Command %r has failed. Received %r" % (cmd, e)) + + # jails = server._Server__jails + # for j in jails: + # print(j, jails[j]) + + # test default stock actions sepecified in all stock jails: + if not unittest.F2B.fast: + self._testExecActions(server) + + def getDefaultJailStream(self, jail, act): + act = act.replace('%(__name__)s', jail) + actName, actOpt = extractOptions(act) + stream = [ + ['add', jail, 'polling'], + # ['set', jail, 'addfailregex', 'DUMMY-REGEX '], + ] + action = ActionReader( + actName, jail, actOpt, + share_config=self.__share_cfg, basedir=CONFIG_DIR) + self.assertTrue(action.read()) + action.getOptions({}) + stream.extend(action.convert()) + return stream + + def testCheckStockAllActions(self): + unittest.F2B.SkipIfCfgMissing(stock=True) + unittest.F2B.SkipIfFast() + import glob + + server = TestServer() + transm = server._Server__transm + + for actCfg in glob.glob(os.path.join(CONFIG_DIR, 'action.d', '*.conf')): + act = os.path.basename(actCfg).replace('.conf', '') + # transmit artifical jail with each action to the server: + stream = self.getDefaultJailStream('j-'+act, act) + for cmd in stream: + # command to server: + ret, res = transm.proceed(cmd) + self.assertEqual(ret, 0) + # test executing action commands: + self._testExecActions(server) + + + def testCheckStockCommandActions(self): + unittest.F2B.SkipIfCfgMissing(stock=True) + # test cases to check valid ipv4/ipv6 action definition, tuple with (('jail', 'action[params]', 'tests', ...) + # where tests is a dictionary contains: + # 'ip4' - should not be found (logged) on ban/unban of IPv6 (negative test), + # 'ip6' - should not be found (logged) on ban/unban of IPv4 (negative test), + # 'start', 'stop' - should be found (logged) on action start/stop, + # etc. + testJailsActions = ( + # dummy -- + ('j-dummy', 'dummy[name=%(__name__)s, init="==", target="/tmp/fail2ban.dummy"]', { + 'ip4': ('family: inet4',), 'ip6': ('family: inet6',), + 'start': ( + '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- started"`', + ), + 'flush': ( + '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- clear all"`', + ), + 'stop': ( + '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- stopped"`', + ), + 'ip4-check': (), + 'ip6-check': (), + 'ip4-ban': ( + '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- banned 192.0.2.1 (family: inet4)"`', + ), + 'ip4-unban': ( + '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- unbanned 192.0.2.1 (family: inet4)"`', + ), + 'ip6-ban': ( + '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- banned 2001:db8:: (family: inet6)"`', + ), + 'ip6-unban': ( + '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- unbanned 2001:db8:: (family: inet6)"`', + ), + }), + # iptables-multiport -- + ('j-w-iptables-mp', 'iptables-multiport[name=%(__name__)s, bantime="10m", port="http,https", protocol="tcp", chain=""]', { + 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), + 'ip4-start': ( + "`iptables -w -N f2b-j-w-iptables-mp`", + "`iptables -w -A f2b-j-w-iptables-mp -j RETURN`", + "`iptables -w -I INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`", + ), + 'ip6-start': ( + "`ip6tables -w -N f2b-j-w-iptables-mp`", + "`ip6tables -w -A f2b-j-w-iptables-mp -j RETURN`", + "`ip6tables -w -I INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`", + ), + 'flush': ( + "`iptables -w -F f2b-j-w-iptables-mp`", + "`ip6tables -w -F f2b-j-w-iptables-mp`", + ), + 'stop': ( + "`iptables -w -D INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`", + "`iptables -w -F f2b-j-w-iptables-mp`", + "`iptables -w -X f2b-j-w-iptables-mp`", + "`ip6tables -w -D INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`", + "`ip6tables -w -F f2b-j-w-iptables-mp`", + "`ip6tables -w -X f2b-j-w-iptables-mp`", + ), + 'ip4-check': ( + r"""`iptables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-mp[ \t]'`""", + ), + 'ip6-check': ( + r"""`ip6tables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-mp[ \t]'`""", + ), + 'ip4-ban': ( + r"`iptables -w -I f2b-j-w-iptables-mp 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", + ), + 'ip4-unban': ( + r"`iptables -w -D f2b-j-w-iptables-mp -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", + ), + 'ip6-ban': ( + r"`ip6tables -w -I f2b-j-w-iptables-mp 1 -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", + ), + 'ip6-unban': ( + r"`ip6tables -w -D f2b-j-w-iptables-mp -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", + ), + }), + # iptables-allports -- + ('j-w-iptables-ap', 'iptables-allports[name=%(__name__)s, bantime="10m", protocol="tcp", chain=""]', { + 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), + 'ip4-start': ( + "`iptables -w -N f2b-j-w-iptables-ap`", + "`iptables -w -A f2b-j-w-iptables-ap -j RETURN`", + "`iptables -w -I INPUT -p tcp -j f2b-j-w-iptables-ap`", + ), + 'ip6-start': ( + "`ip6tables -w -N f2b-j-w-iptables-ap`", + "`ip6tables -w -A f2b-j-w-iptables-ap -j RETURN`", + "`ip6tables -w -I INPUT -p tcp -j f2b-j-w-iptables-ap`", + ), + 'flush': ( + "`iptables -w -F f2b-j-w-iptables-ap`", + "`ip6tables -w -F f2b-j-w-iptables-ap`", + ), + 'stop': ( + "`iptables -w -D INPUT -p tcp -j f2b-j-w-iptables-ap`", + "`iptables -w -F f2b-j-w-iptables-ap`", + "`iptables -w -X f2b-j-w-iptables-ap`", + "`ip6tables -w -D INPUT -p tcp -j f2b-j-w-iptables-ap`", + "`ip6tables -w -F f2b-j-w-iptables-ap`", + "`ip6tables -w -X f2b-j-w-iptables-ap`", + ), + 'ip4-check': ( + r"""`iptables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-ap[ \t]'`""", + ), + 'ip6-check': ( + r"""`ip6tables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-ap[ \t]'`""", + ), + 'ip4-ban': ( + r"`iptables -w -I f2b-j-w-iptables-ap 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", + ), + 'ip4-unban': ( + r"`iptables -w -D f2b-j-w-iptables-ap -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", + ), + 'ip6-ban': ( + r"`ip6tables -w -I f2b-j-w-iptables-ap 1 -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", + ), + 'ip6-unban': ( + r"`ip6tables -w -D f2b-j-w-iptables-ap -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", + ), + }), + # iptables-ipset-proto6 -- + ('j-w-iptables-ipset', 'iptables-ipset-proto6[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain=""]', { + 'ip4': (' f2b-j-w-iptables-ipset ',), 'ip6': (' f2b-j-w-iptables-ipset6 ',), + 'ip4-start': ( + "`ipset create f2b-j-w-iptables-ipset hash:ip timeout 600`", + "`iptables -w -I INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable`", + ), + 'ip6-start': ( + "`ipset create f2b-j-w-iptables-ipset6 hash:ip timeout 600 family inet6`", + "`ip6tables -w -I INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", + ), + 'flush': ( + "`ipset flush f2b-j-w-iptables-ipset`", + "`ipset flush f2b-j-w-iptables-ipset6`", + ), + 'stop': ( + "`iptables -w -D INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable`", + "`ipset flush f2b-j-w-iptables-ipset`", + "`ipset destroy f2b-j-w-iptables-ipset`", + "`ip6tables -w -D INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", + "`ipset flush f2b-j-w-iptables-ipset6`", + "`ipset destroy f2b-j-w-iptables-ipset6`", + ), + 'ip4-check': (), + 'ip6-check': (), + 'ip4-ban': ( + r"`ipset add f2b-j-w-iptables-ipset 192.0.2.1 timeout 600 -exist`", + ), + 'ip4-unban': ( + r"`ipset del f2b-j-w-iptables-ipset 192.0.2.1 -exist`", + ), + 'ip6-ban': ( + r"`ipset add f2b-j-w-iptables-ipset6 2001:db8:: timeout 600 -exist`", + ), + 'ip6-unban': ( + r"`ipset del f2b-j-w-iptables-ipset6 2001:db8:: -exist`", + ), + }), + # iptables-ipset-proto6-allports -- + ('j-w-iptables-ipset-ap', 'iptables-ipset-proto6-allports[name=%(__name__)s, bantime="10m", chain=""]', { + 'ip4': (' f2b-j-w-iptables-ipset-ap ',), 'ip6': (' f2b-j-w-iptables-ipset-ap6 ',), + 'ip4-start': ( + "`ipset create f2b-j-w-iptables-ipset-ap hash:ip timeout 600`", + "`iptables -w -I INPUT -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", + ), + 'ip6-start': ( + "`ipset create f2b-j-w-iptables-ipset-ap6 hash:ip timeout 600 family inet6`", + "`ip6tables -w -I INPUT -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", + ), + 'flush': ( + "`ipset flush f2b-j-w-iptables-ipset-ap`", + "`ipset flush f2b-j-w-iptables-ipset-ap6`", + ), + 'stop': ( + "`iptables -w -D INPUT -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", + "`ipset flush f2b-j-w-iptables-ipset-ap`", + "`ipset destroy f2b-j-w-iptables-ipset-ap`", + "`ip6tables -w -D INPUT -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", + "`ipset flush f2b-j-w-iptables-ipset-ap6`", + "`ipset destroy f2b-j-w-iptables-ipset-ap6`", + ), + 'ip4-check': (), + 'ip6-check': (), + 'ip4-ban': ( + r"`ipset add f2b-j-w-iptables-ipset-ap 192.0.2.1 timeout 600 -exist`", + ), + 'ip4-unban': ( + r"`ipset del f2b-j-w-iptables-ipset-ap 192.0.2.1 -exist`", + ), + 'ip6-ban': ( + r"`ipset add f2b-j-w-iptables-ipset-ap6 2001:db8:: timeout 600 -exist`", + ), + 'ip6-unban': ( + r"`ipset del f2b-j-w-iptables-ipset-ap6 2001:db8:: -exist`", + ), + }), + # iptables -- + ('j-w-iptables', 'iptables[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain=""]', { + 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), + 'ip4-start': ( + "`iptables -w -N f2b-j-w-iptables`", + "`iptables -w -A f2b-j-w-iptables -j RETURN`", + "`iptables -w -I INPUT -p tcp --dport http -j f2b-j-w-iptables`", + ), + 'ip6-start': ( + "`ip6tables -w -N f2b-j-w-iptables`", + "`ip6tables -w -A f2b-j-w-iptables -j RETURN`", + "`ip6tables -w -I INPUT -p tcp --dport http -j f2b-j-w-iptables`", + ), + 'flush': ( + "`iptables -w -F f2b-j-w-iptables`", + "`ip6tables -w -F f2b-j-w-iptables`", + ), + 'stop': ( + "`iptables -w -D INPUT -p tcp --dport http -j f2b-j-w-iptables`", + "`iptables -w -F f2b-j-w-iptables`", + "`iptables -w -X f2b-j-w-iptables`", + "`ip6tables -w -D INPUT -p tcp --dport http -j f2b-j-w-iptables`", + "`ip6tables -w -F f2b-j-w-iptables`", + "`ip6tables -w -X f2b-j-w-iptables`", + ), + 'ip4-check': ( + r"""`iptables -w -n -L INPUT | grep -q 'f2b-j-w-iptables[ \t]'`""", + ), + 'ip6-check': ( + r"""`ip6tables -w -n -L INPUT | grep -q 'f2b-j-w-iptables[ \t]'`""", + ), + 'ip4-ban': ( + r"`iptables -w -I f2b-j-w-iptables 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", + ), + 'ip4-unban': ( + r"`iptables -w -D f2b-j-w-iptables -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", + ), + 'ip6-ban': ( + r"`ip6tables -w -I f2b-j-w-iptables 1 -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", + ), + 'ip6-unban': ( + r"`ip6tables -w -D f2b-j-w-iptables -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", + ), + }), + # iptables-new -- + ('j-w-iptables-new', 'iptables-new[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain=""]', { + 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), + 'ip4-start': ( + "`iptables -w -N f2b-j-w-iptables-new`", + "`iptables -w -A f2b-j-w-iptables-new -j RETURN`", + "`iptables -w -I INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`", + ), + 'ip6-start': ( + "`ip6tables -w -N f2b-j-w-iptables-new`", + "`ip6tables -w -A f2b-j-w-iptables-new -j RETURN`", + "`ip6tables -w -I INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`", + ), + 'flush': ( + "`iptables -w -F f2b-j-w-iptables-new`", + "`ip6tables -w -F f2b-j-w-iptables-new`", + ), + 'stop': ( + "`iptables -w -D INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`", + "`iptables -w -F f2b-j-w-iptables-new`", + "`iptables -w -X f2b-j-w-iptables-new`", + "`ip6tables -w -D INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`", + "`ip6tables -w -F f2b-j-w-iptables-new`", + "`ip6tables -w -X f2b-j-w-iptables-new`", + ), + 'ip4-check': ( + r"""`iptables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-new[ \t]'`""", + ), + 'ip6-check': ( + r"""`ip6tables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-new[ \t]'`""", + ), + 'ip4-ban': ( + r"`iptables -w -I f2b-j-w-iptables-new 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", + ), + 'ip4-unban': ( + r"`iptables -w -D f2b-j-w-iptables-new -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", + ), + 'ip6-ban': ( + r"`ip6tables -w -I f2b-j-w-iptables-new 1 -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", + ), + 'ip6-unban': ( + r"`ip6tables -w -D f2b-j-w-iptables-new -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", + ), + }), + # iptables-xt_recent-echo -- + ('j-w-iptables-xtre', 'iptables-xt_recent-echo[name=%(__name__)s, bantime="10m", chain=""]', { + 'ip4': ('`iptables ', '/f2b-j-w-iptables-xtre`'), 'ip6': ('`ip6tables ', '/f2b-j-w-iptables-xtre6`'), + 'ip4-start': ( + "`if [ `id -u` -eq 0 ];then iptables -w -I INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre -j REJECT --reject-with icmp-port-unreachable;fi`", + ), + 'ip6-start': ( + "`if [ `id -u` -eq 0 ];then ip6tables -w -I INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre6 -j REJECT --reject-with icmp6-port-unreachable;fi`", + ), + 'stop': ( + "`echo / > /proc/net/xt_recent/f2b-j-w-iptables-xtre`", + "`if [ `id -u` -eq 0 ];then iptables -w -D INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre -j REJECT --reject-with icmp-port-unreachable;fi`", + "`echo / > /proc/net/xt_recent/f2b-j-w-iptables-xtre6`", + "`if [ `id -u` -eq 0 ];then ip6tables -w -D INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre6 -j REJECT --reject-with icmp6-port-unreachable;fi`", + ), + 'ip4-check': ( + r"`test -e /proc/net/xt_recent/f2b-j-w-iptables-xtre`", + ), + 'ip6-check': ( + r"`test -e /proc/net/xt_recent/f2b-j-w-iptables-xtre6`", + ), + 'ip4-ban': ( + r"`echo +192.0.2.1 > /proc/net/xt_recent/f2b-j-w-iptables-xtre`", + ), + 'ip4-unban': ( + r"`echo -192.0.2.1 > /proc/net/xt_recent/f2b-j-w-iptables-xtre`", + ), + 'ip6-ban': ( + r"`echo +2001:db8:: > /proc/net/xt_recent/f2b-j-w-iptables-xtre6`", + ), + 'ip6-unban': ( + r"`echo -2001:db8:: > /proc/net/xt_recent/f2b-j-w-iptables-xtre6`", + ), + }), + # pf default -- multiport on default port (tag set in jail.conf, but not in this test case) + ('j-w-pf', 'pf[name=%(__name__)s, actionstart_on_demand=false]', { + 'ip4': (), 'ip6': (), + 'start': ( + '`echo "table persist counters" | pfctl -a f2b/j-w-pf -f-`', + 'port=""', + '`echo "block quick proto tcp from to any port $port" | pfctl -a f2b/j-w-pf -f-`', + ), + 'flush': ( + '`pfctl -a f2b/j-w-pf -t f2b-j-w-pf -T flush`', + ), + 'stop': ( + '`pfctl -a f2b/j-w-pf -sr 2>/dev/null | grep -v f2b-j-w-pf | pfctl -a f2b/j-w-pf -f-`', + '`pfctl -a f2b/j-w-pf -t f2b-j-w-pf -T flush`', + '`pfctl -a f2b/j-w-pf -t f2b-j-w-pf -T kill`', + ), + 'ip4-check': ("`pfctl -a f2b/j-w-pf -sr | grep -q f2b-j-w-pf`",), + 'ip6-check': ("`pfctl -a f2b/j-w-pf -sr | grep -q f2b-j-w-pf`",), + 'ip4-ban': ("`pfctl -a f2b/j-w-pf -t f2b-j-w-pf -T add 192.0.2.1`",), + 'ip4-unban': ("`pfctl -a f2b/j-w-pf -t f2b-j-w-pf -T delete 192.0.2.1`",), + 'ip6-ban': ("`pfctl -a f2b/j-w-pf -t f2b-j-w-pf -T add 2001:db8::`",), + 'ip6-unban': ("`pfctl -a f2b/j-w-pf -t f2b-j-w-pf -T delete 2001:db8::`",), + }), + # pf multiport with custom ports -- + ('j-w-pf-mp', 'pf[actiontype=][name=%(__name__)s, port="http,https"]', { + 'ip4': (), 'ip6': (), + 'start': ( + '`echo "table persist counters" | pfctl -a f2b/j-w-pf-mp -f-`', + 'port="http,https"', + '`echo "block quick proto tcp from to any port $port" | pfctl -a f2b/j-w-pf-mp -f-`', + ), + 'flush': ( + '`pfctl -a f2b/j-w-pf-mp -t f2b-j-w-pf-mp -T flush`', + ), + 'stop': ( + '`pfctl -a f2b/j-w-pf-mp -sr 2>/dev/null | grep -v f2b-j-w-pf-mp | pfctl -a f2b/j-w-pf-mp -f-`', + '`pfctl -a f2b/j-w-pf-mp -t f2b-j-w-pf-mp -T flush`', + '`pfctl -a f2b/j-w-pf-mp -t f2b-j-w-pf-mp -T kill`', + ), + 'ip4-check': ("`pfctl -a f2b/j-w-pf-mp -sr | grep -q f2b-j-w-pf-mp`",), + 'ip6-check': ("`pfctl -a f2b/j-w-pf-mp -sr | grep -q f2b-j-w-pf-mp`",), + 'ip4-ban': ("`pfctl -a f2b/j-w-pf-mp -t f2b-j-w-pf-mp -T add 192.0.2.1`",), + 'ip4-unban': ("`pfctl -a f2b/j-w-pf-mp -t f2b-j-w-pf-mp -T delete 192.0.2.1`",), + 'ip6-ban': ("`pfctl -a f2b/j-w-pf-mp -t f2b-j-w-pf-mp -T add 2001:db8::`",), + 'ip6-unban': ("`pfctl -a f2b/j-w-pf-mp -t f2b-j-w-pf-mp -T delete 2001:db8::`",), + }), + # pf allports -- test additionally "actionstart_on_demand" was set to true + ('j-w-pf-ap', 'pf[actiontype=, actionstart_on_demand=true][name=%(__name__)s]', { + 'ip4': (), 'ip6': (), + 'ip4-start': ( + '`echo "table persist counters" | pfctl -a f2b/j-w-pf-ap -f-`', + '`echo "block quick proto tcp from to any" | pfctl -a f2b/j-w-pf-ap -f-`', + ), + 'ip6-start': (), # the same as ipv4 + 'flush': ( + '`pfctl -a f2b/j-w-pf-ap -t f2b-j-w-pf-ap -T flush`', + ), + 'stop': ( + '`pfctl -a f2b/j-w-pf-ap -sr 2>/dev/null | grep -v f2b-j-w-pf-ap | pfctl -a f2b/j-w-pf-ap -f-`', + '`pfctl -a f2b/j-w-pf-ap -t f2b-j-w-pf-ap -T flush`', + '`pfctl -a f2b/j-w-pf-ap -t f2b-j-w-pf-ap -T kill`', + ), + 'ip4-check': ("`pfctl -a f2b/j-w-pf-ap -sr | grep -q f2b-j-w-pf-ap`",), + 'ip6-check': ("`pfctl -a f2b/j-w-pf-ap -sr | grep -q f2b-j-w-pf-ap`",), + 'ip4-ban': ("`pfctl -a f2b/j-w-pf-ap -t f2b-j-w-pf-ap -T add 192.0.2.1`",), + 'ip4-unban': ("`pfctl -a f2b/j-w-pf-ap -t f2b-j-w-pf-ap -T delete 192.0.2.1`",), + 'ip6-ban': ("`pfctl -a f2b/j-w-pf-ap -t f2b-j-w-pf-ap -T add 2001:db8::`",), + 'ip6-unban': ("`pfctl -a f2b/j-w-pf-ap -t f2b-j-w-pf-ap -T delete 2001:db8::`",), + }), + # firewallcmd-multiport -- + ('j-w-fwcmd-mp', 'firewallcmd-multiport[name=%(__name__)s, bantime="10m", port="http,https", protocol="tcp", chain=""]', { + 'ip4': (' ipv4 ', 'icmp-port-unreachable'), 'ip6': (' ipv6 ', 'icmp6-port-unreachable'), + 'ip4-start': ( + "`firewall-cmd --direct --add-chain ipv4 filter f2b-j-w-fwcmd-mp`", + "`firewall-cmd --direct --add-rule ipv4 filter f2b-j-w-fwcmd-mp 1000 -j RETURN`", + "`firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports http,https -j f2b-j-w-fwcmd-mp`", + ), + 'ip6-start': ( + "`firewall-cmd --direct --add-chain ipv6 filter f2b-j-w-fwcmd-mp`", + "`firewall-cmd --direct --add-rule ipv6 filter f2b-j-w-fwcmd-mp 1000 -j RETURN`", + "`firewall-cmd --direct --add-rule ipv6 filter INPUT_direct 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports http,https -j f2b-j-w-fwcmd-mp`", + ), + 'stop': ( + "`firewall-cmd --direct --remove-rule ipv4 filter INPUT_direct 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports http,https -j f2b-j-w-fwcmd-mp`", + "`firewall-cmd --direct --remove-rules ipv4 filter f2b-j-w-fwcmd-mp`", + "`firewall-cmd --direct --remove-chain ipv4 filter f2b-j-w-fwcmd-mp`", + "`firewall-cmd --direct --remove-rule ipv6 filter INPUT_direct 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports http,https -j f2b-j-w-fwcmd-mp`", + "`firewall-cmd --direct --remove-rules ipv6 filter f2b-j-w-fwcmd-mp`", + "`firewall-cmd --direct --remove-chain ipv6 filter f2b-j-w-fwcmd-mp`", + ), + 'ip4-check': ( + r"`firewall-cmd --direct --get-chains ipv4 filter | sed -e 's, ,\n,g' | grep -q '^f2b-j-w-fwcmd-mp$'`", + ), + 'ip6-check': ( + r"`firewall-cmd --direct --get-chains ipv6 filter | sed -e 's, ,\n,g' | grep -q '^f2b-j-w-fwcmd-mp$'`", + ), + 'ip4-ban': ( + r"`firewall-cmd --direct --add-rule ipv4 filter f2b-j-w-fwcmd-mp 0 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", + ), + 'ip4-unban': ( + r"`firewall-cmd --direct --remove-rule ipv4 filter f2b-j-w-fwcmd-mp 0 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", + ), + 'ip6-ban': ( + r"`firewall-cmd --direct --add-rule ipv6 filter f2b-j-w-fwcmd-mp 0 -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", + ), + 'ip6-unban': ( + r"`firewall-cmd --direct --remove-rule ipv6 filter f2b-j-w-fwcmd-mp 0 -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", + ), + }), + # firewallcmd-allports -- + ('j-w-fwcmd-ap', 'firewallcmd-allports[name=%(__name__)s, bantime="10m", protocol="tcp", chain=""]', { + 'ip4': (' ipv4 ', 'icmp-port-unreachable'), 'ip6': (' ipv6 ', 'icmp6-port-unreachable'), + 'ip4-start': ( + "`firewall-cmd --direct --add-chain ipv4 filter f2b-j-w-fwcmd-ap`", + "`firewall-cmd --direct --add-rule ipv4 filter f2b-j-w-fwcmd-ap 1000 -j RETURN`", + "`firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 0 -j f2b-j-w-fwcmd-ap`", + ), + 'ip6-start': ( + "`firewall-cmd --direct --add-chain ipv6 filter f2b-j-w-fwcmd-ap`", + "`firewall-cmd --direct --add-rule ipv6 filter f2b-j-w-fwcmd-ap 1000 -j RETURN`", + "`firewall-cmd --direct --add-rule ipv6 filter INPUT_direct 0 -j f2b-j-w-fwcmd-ap`", + ), + 'stop': ( + "`firewall-cmd --direct --remove-rule ipv4 filter INPUT_direct 0 -j f2b-j-w-fwcmd-ap`", + "`firewall-cmd --direct --remove-rules ipv4 filter f2b-j-w-fwcmd-ap`", + "`firewall-cmd --direct --remove-chain ipv4 filter f2b-j-w-fwcmd-ap`", + "`firewall-cmd --direct --remove-rule ipv6 filter INPUT_direct 0 -j f2b-j-w-fwcmd-ap`", + "`firewall-cmd --direct --remove-rules ipv6 filter f2b-j-w-fwcmd-ap`", + "`firewall-cmd --direct --remove-chain ipv6 filter f2b-j-w-fwcmd-ap`", + ), + 'ip4-check': ( + r"`firewall-cmd --direct --get-chains ipv4 filter | sed -e 's, ,\n,g' | grep -q '^f2b-j-w-fwcmd-ap$'`", + ), + 'ip6-check': ( + r"`firewall-cmd --direct --get-chains ipv6 filter | sed -e 's, ,\n,g' | grep -q '^f2b-j-w-fwcmd-ap$'`", + ), + 'ip4-ban': ( + r"`firewall-cmd --direct --add-rule ipv4 filter f2b-j-w-fwcmd-ap 0 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", + ), + 'ip4-unban': ( + r"`firewall-cmd --direct --remove-rule ipv4 filter f2b-j-w-fwcmd-ap 0 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", + ), + 'ip6-ban': ( + r"`firewall-cmd --direct --add-rule ipv6 filter f2b-j-w-fwcmd-ap 0 -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", + ), + 'ip6-unban': ( + r"`firewall-cmd --direct --remove-rule ipv6 filter f2b-j-w-fwcmd-ap 0 -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", + ), + }), + # firewallcmd-ipset (multiport) -- + ('j-w-fwcmd-ipset', 'firewallcmd-ipset[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain=""]', { + 'ip4': (' f2b-j-w-fwcmd-ipset ',), 'ip6': (' f2b-j-w-fwcmd-ipset6 ',), + 'ip4-start': ( + "`ipset create f2b-j-w-fwcmd-ipset hash:ip timeout 600`", + "`firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset src -j REJECT --reject-with icmp-port-unreachable`", + ), + 'ip6-start': ( + "`ipset create f2b-j-w-fwcmd-ipset6 hash:ip timeout 600 family inet6`", + "`firewall-cmd --direct --add-rule ipv6 filter INPUT_direct 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", + ), + 'flush': ( + "`ipset flush f2b-j-w-fwcmd-ipset`", + "`ipset flush f2b-j-w-fwcmd-ipset6`", + ), + 'stop': ( + "`firewall-cmd --direct --remove-rule ipv4 filter INPUT_direct 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset src -j REJECT --reject-with icmp-port-unreachable`", + "`ipset flush f2b-j-w-fwcmd-ipset`", + "`ipset destroy f2b-j-w-fwcmd-ipset`", + "`firewall-cmd --direct --remove-rule ipv6 filter INPUT_direct 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", + "`ipset flush f2b-j-w-fwcmd-ipset6`", + "`ipset destroy f2b-j-w-fwcmd-ipset6`", + ), + 'ip4-check': (), + 'ip6-check': (), + 'ip4-ban': ( + r"`ipset add f2b-j-w-fwcmd-ipset 192.0.2.1 timeout 600 -exist`", + ), + 'ip4-unban': ( + r"`ipset del f2b-j-w-fwcmd-ipset 192.0.2.1 -exist`", + ), + 'ip6-ban': ( + r"`ipset add f2b-j-w-fwcmd-ipset6 2001:db8:: timeout 600 -exist`", + ), + 'ip6-unban': ( + r"`ipset del f2b-j-w-fwcmd-ipset6 2001:db8:: -exist`", + ), + }), + # firewallcmd-ipset (allports) -- + ('j-w-fwcmd-ipset-ap', 'firewallcmd-ipset[name=%(__name__)s, bantime="10m", actiontype=, protocol="tcp", chain=""]', { + 'ip4': (' f2b-j-w-fwcmd-ipset-ap ',), 'ip6': (' f2b-j-w-fwcmd-ipset-ap6 ',), + 'ip4-start': ( + "`ipset create f2b-j-w-fwcmd-ipset-ap hash:ip timeout 600`", + "`firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 0 -p tcp -m set --match-set f2b-j-w-fwcmd-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", + ), + 'ip6-start': ( + "`ipset create f2b-j-w-fwcmd-ipset-ap6 hash:ip timeout 600 family inet6`", + "`firewall-cmd --direct --add-rule ipv6 filter INPUT_direct 0 -p tcp -m set --match-set f2b-j-w-fwcmd-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", + ), + 'flush': ( + "`ipset flush f2b-j-w-fwcmd-ipset-ap`", + "`ipset flush f2b-j-w-fwcmd-ipset-ap6`", + ), + 'stop': ( + "`firewall-cmd --direct --remove-rule ipv4 filter INPUT_direct 0 -p tcp -m set --match-set f2b-j-w-fwcmd-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", + "`ipset flush f2b-j-w-fwcmd-ipset-ap`", + "`ipset destroy f2b-j-w-fwcmd-ipset-ap`", + "`firewall-cmd --direct --remove-rule ipv6 filter INPUT_direct 0 -p tcp -m set --match-set f2b-j-w-fwcmd-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", + "`ipset flush f2b-j-w-fwcmd-ipset-ap6`", + "`ipset destroy f2b-j-w-fwcmd-ipset-ap6`", + ), + 'ip4-check': (), + 'ip6-check': (), + 'ip4-ban': ( + r"`ipset add f2b-j-w-fwcmd-ipset-ap 192.0.2.1 timeout 600 -exist`", + ), + 'ip4-unban': ( + r"`ipset del f2b-j-w-fwcmd-ipset-ap 192.0.2.1 -exist`", + ), + 'ip6-ban': ( + r"`ipset add f2b-j-w-fwcmd-ipset-ap6 2001:db8:: timeout 600 -exist`", + ), + 'ip6-unban': ( + r"`ipset del f2b-j-w-fwcmd-ipset-ap6 2001:db8:: -exist`", + ), + }), + ) + server = TestServer() + transm = server._Server__transm + cmdHandler = transm._Transmitter__commandHandler + + for jail, act, tests in testJailsActions: + stream = self.getDefaultJailStream(jail, act) # for cmd in stream: # print(cmd) - # filter all start commands (we want not start all jails): + # transmit jail to the server: for cmd in stream: - if cmd[0] != 'start': - # change to the fast init backend: - if cmd[0] == 'add': - cmd[2] = 'polling' - # change log path to test log of the jail - # (to prevent "Permission denied" on /var/logs/ for test-user): - elif len(cmd) > 3 and cmd[0] == 'set' and cmd[2] == 'addlogpath': - fn = os.path.join(TEST_FILES_DIR, 'logs', cmd[1]) - # fallback to testcase01 if jail has no its own test log-file - # (should not matter really): - if not os.path.exists(fn): # pragma: no cover - fn = os.path.join(TEST_FILES_DIR, 'testcase01.log') - cmd[3] = fn - # if fast add dummy regex to prevent too long compile of all regexp - # (we don't use it in this test at all): - elif unittest.F2B.fast and ( - len(cmd) > 3 and cmd[0] in ('set', 'multi-set') and cmd[2] == 'addfailregex' - ): # pragma: no cover - cmd[0] = "set" - cmd[3] = "DUMMY-REGEX " - # command to server, use cmdHandler direct instead of `transm.proceed(cmd)`: - try: - cmdHandler(cmd) - except Exception as e: # pragma: no cover - self.fail("Command %r has failed. Received %r" % (cmd, e)) + # command to server: + ret, res = transm.proceed(cmd) + self.assertEqual(ret, 0) - # jails = server._Server__jails - # for j in jails: - # print(j, jails[j]) + jails = server._Server__jails - # test default stock actions sepecified in all stock jails: - if not unittest.F2B.fast: - self._testExecActions(server) + tickets = { + 'ip4': BanTicket('192.0.2.1'), + 'ip6': BanTicket('2001:DB8::'), + } + for jail, act, tests in testJailsActions: + # print(jail, jails[jail]) + for a in jails[jail].actions: + action = jails[jail].actions[a] + logSys.debug('# ' + ('=' * 50)) + logSys.debug('# == %-44s ==', jail + ' - ' + action._name) + logSys.debug('# ' + ('=' * 50)) + self.assertTrue(isinstance(action, _actions.CommandAction)) + # wrap default command processor: + action.executeCmd = self._executeCmd + # test start : + self.pruneLog('# === start ===') + action.start() + if tests.get('start'): + self.assertLogged(*tests['start'], all=True) + else: + self.assertNotLogged(*tests['ip4-start']+tests['ip6-start'], all=True) + ainfo = { + 'ip4': _actions.Actions.ActionInfo(tickets['ip4'], jails[jail]), + 'ip6': _actions.Actions.ActionInfo(tickets['ip6'], jails[jail]), + } + # test ban ip4 : + self.pruneLog('# === ban-ipv4 ===') + action.ban(ainfo['ip4']) + if tests.get('ip4-start'): self.assertLogged(*tests['ip4-start'], all=True) + if tests.get('ip6-start'): self.assertNotLogged(*tests['ip6-start'], all=True) + self.assertLogged(*tests['ip4-check']+tests['ip4-ban'], all=True) + self.assertNotLogged(*tests['ip6'], all=True) + # test unban ip4 : + self.pruneLog('# === unban ipv4 ===') + action.unban(ainfo['ip4']) + self.assertLogged(*tests['ip4-check']+tests['ip4-unban'], all=True) + self.assertNotLogged(*tests['ip6'], all=True) + # test ban ip6 : + self.pruneLog('# === ban ipv6 ===') + action.ban(ainfo['ip6']) + if tests.get('ip6-start'): self.assertLogged(*tests['ip6-start'], all=True) + if tests.get('ip4-start'): self.assertNotLogged(*tests['ip4-start'], all=True) + self.assertLogged(*tests['ip6-check']+tests['ip6-ban'], all=True) + self.assertNotLogged(*tests['ip4'], all=True) + # test unban ip6 : + self.pruneLog('# === unban ipv6 ===') + action.unban(ainfo['ip6']) + self.assertLogged(*tests['ip6-check']+tests['ip6-unban'], all=True) + self.assertNotLogged(*tests['ip4'], all=True) + # test flush for actions should supported this: + if tests.get('flush'): + self.pruneLog('# === flush ===') + action.flush() + self.assertLogged(*tests['flush'], all=True) + # test stop : + self.pruneLog('# === stop ===') + action.stop() + self.assertLogged(*tests['stop'], all=True) - def getDefaultJailStream(self, jail, act): - act = act.replace('%(__name__)s', jail) - actName, actOpt = extractOptions(act) - stream = [ - ['add', jail, 'polling'], - # ['set', jail, 'addfailregex', 'DUMMY-REGEX '], - ] - action = ActionReader( - actName, jail, actOpt, - share_config=self.__share_cfg, basedir=CONFIG_DIR) - self.assertTrue(action.read()) - action.getOptions({}) - stream.extend(action.convert()) - return stream + def _executeMailCmd(self, realCmd, timeout=60): + # replace pipe to mail with pipe to cat: + realCmd = re.sub(r'\)\s*\|\s*mail\b([^\n]*)', + r') | cat; printf "\\n... | "; echo mail \1', realCmd) + # replace abuse retrieving (possible no-network), just replace first occurrence of 'dig...': + realCmd = re.sub(r'\bADDRESSES=\$\(dig\s[^\n]+', + lambda m: 'ADDRESSES="abuse-1@abuse-test-server, abuse-2@abuse-test-server"', + realCmd, 1) + # execute action: + return _actions.CommandAction.executeCmd(realCmd, timeout=timeout) - def testCheckStockAllActions(self): - unittest.F2B.SkipIfFast() - import glob + def testComplexMailActionMultiLog(self): + unittest.F2B.SkipIfCfgMissing(stock=True) + testJailsActions = ( + # mail-whois-lines -- + ('j-mail-whois-lines', + 'mail-whois-lines[' + 'name=%(__name__)s, grepopts="-m 1", grepmax=2, mailcmd="mail -s", ' + + # 2 logs to test grep from multiple logs: + 'logpath="' + os.path.join(TEST_FILES_DIR, "testcase01.log") + '\n' + + ' ' + os.path.join(TEST_FILES_DIR, "testcase01a.log") + '", ' + '_whois_command="echo \'-- information about --\'"' + ']', + { + 'ip4-ban': ( + 'The IP 87.142.124.10 has just been banned by Fail2Ban after', + '100 attempts against j-mail-whois-lines.', + 'Here is more information about 87.142.124.10 :', + '-- information about 87.142.124.10 --', + 'Lines containing failures of 87.142.124.10 (max 2)', + 'testcase01.log:Dec 31 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 87.142.124.10', + 'testcase01a.log:Dec 31 11:55:01 [sshd] error: PAM: Authentication failure for test from 87.142.124.10', + ), + }), + # complain -- + ('j-complain-abuse', + 'complain[' + 'name=%(__name__)s, grepopts="-m 1", grepmax=2, mailcmd="mail -s \'Hostname: , family: \' - ",' + + # test reverse ip: + 'debug=1,' + + # 2 logs to test grep from multiple logs: + 'logpath="' + os.path.join(TEST_FILES_DIR, "testcase01.log") + '\n' + + ' ' + os.path.join(TEST_FILES_DIR, "testcase01a.log") + '", ' + ']', + { + 'ip4-ban': ( + # test reverse ip: + 'try to resolve 10.124.142.87.abuse-contacts.abusix.org', + 'Lines containing failures of 87.142.124.10 (max 2)', + 'testcase01.log:Dec 31 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 87.142.124.10', + 'testcase01a.log:Dec 31 11:55:01 [sshd] error: PAM: Authentication failure for test from 87.142.124.10', + # both abuse mails should be separated with space: + 'mail -s Hostname: test-host, family: inet4 - Abuse from 87.142.124.10 abuse-1@abuse-test-server abuse-2@abuse-test-server', + ), + 'ip6-ban': ( + # test reverse ip: + 'try to resolve 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.abuse-contacts.abusix.org', + 'Lines containing failures of 2001:db8::1 (max 2)', + # both abuse mails should be separated with space: + 'mail -s Hostname: test-host, family: inet6 - Abuse from 2001:db8::1 abuse-1@abuse-test-server abuse-2@abuse-test-server', + ), + }), + ) + server = TestServer() + transm = server._Server__transm + cmdHandler = transm._Transmitter__commandHandler - server = TestServer() - transm = server._Server__transm + for jail, act, tests in testJailsActions: + stream = self.getDefaultJailStream(jail, act) - for actCfg in glob.glob(os.path.join(CONFIG_DIR, 'action.d', '*.conf')): - act = os.path.basename(actCfg).replace('.conf', '') - # transmit artifical jail with each action to the server: - stream = self.getDefaultJailStream('j-'+act, act) - for cmd in stream: - # command to server: - ret, res = transm.proceed(cmd) - self.assertEqual(ret, 0) - # test executing action commands: - self._testExecActions(server) + # for cmd in stream: + # print(cmd) + # transmit jail to the server: + for cmd in stream: + # command to server: + ret, res = transm.proceed(cmd) + self.assertEqual(ret, 0) - def testCheckStockCommandActions(self): - # test cases to check valid ipv4/ipv6 action definition, tuple with (('jail', 'action[params]', 'tests', ...) - # where tests is a dictionary contains: - # 'ip4' - should not be found (logged) on ban/unban of IPv6 (negative test), - # 'ip6' - should not be found (logged) on ban/unban of IPv4 (negative test), - # 'start', 'stop' - should be found (logged) on action start/stop, - # etc. - testJailsActions = ( - # dummy -- - ('j-dummy', 'dummy[name=%(__name__)s, init="==", target="/tmp/fail2ban.dummy"]', { - 'ip4': ('family: inet4',), 'ip6': ('family: inet6',), - 'start': ( - '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- started"`', - ), - 'flush': ( - '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- clear all"`', - ), - 'stop': ( - '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- stopped"`', - ), - 'ip4-check': (), - 'ip6-check': (), - 'ip4-ban': ( - '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- banned 192.0.2.1 (family: inet4)"`', - ), - 'ip4-unban': ( - '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- unbanned 192.0.2.1 (family: inet4)"`', - ), - 'ip6-ban': ( - '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- banned 2001:db8:: (family: inet6)"`', - ), - 'ip6-unban': ( - '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- unbanned 2001:db8:: (family: inet6)"`', - ), - }), - # iptables-multiport -- - ('j-w-iptables-mp', 'iptables-multiport[name=%(__name__)s, bantime="10m", port="http,https", protocol="tcp", chain=""]', { - 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), - 'ip4-start': ( - "`iptables -w -N f2b-j-w-iptables-mp`", - "`iptables -w -A f2b-j-w-iptables-mp -j RETURN`", - "`iptables -w -I INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`", - ), - 'ip6-start': ( - "`ip6tables -w -N f2b-j-w-iptables-mp`", - "`ip6tables -w -A f2b-j-w-iptables-mp -j RETURN`", - "`ip6tables -w -I INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`", - ), - 'flush': ( - "`iptables -w -F f2b-j-w-iptables-mp`", - "`ip6tables -w -F f2b-j-w-iptables-mp`", - ), - 'stop': ( - "`iptables -w -D INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`", - "`iptables -w -F f2b-j-w-iptables-mp`", - "`iptables -w -X f2b-j-w-iptables-mp`", - "`ip6tables -w -D INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`", - "`ip6tables -w -F f2b-j-w-iptables-mp`", - "`ip6tables -w -X f2b-j-w-iptables-mp`", - ), - 'ip4-check': ( - r"""`iptables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-mp[ \t]'`""", - ), - 'ip6-check': ( - r"""`ip6tables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-mp[ \t]'`""", - ), - 'ip4-ban': ( - r"`iptables -w -I f2b-j-w-iptables-mp 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", - ), - 'ip4-unban': ( - r"`iptables -w -D f2b-j-w-iptables-mp -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", - ), - 'ip6-ban': ( - r"`ip6tables -w -I f2b-j-w-iptables-mp 1 -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", - ), - 'ip6-unban': ( - r"`ip6tables -w -D f2b-j-w-iptables-mp -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", - ), - }), - # iptables-allports -- - ('j-w-iptables-ap', 'iptables-allports[name=%(__name__)s, bantime="10m", protocol="tcp", chain=""]', { - 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), - 'ip4-start': ( - "`iptables -w -N f2b-j-w-iptables-ap`", - "`iptables -w -A f2b-j-w-iptables-ap -j RETURN`", - "`iptables -w -I INPUT -p tcp -j f2b-j-w-iptables-ap`", - ), - 'ip6-start': ( - "`ip6tables -w -N f2b-j-w-iptables-ap`", - "`ip6tables -w -A f2b-j-w-iptables-ap -j RETURN`", - "`ip6tables -w -I INPUT -p tcp -j f2b-j-w-iptables-ap`", - ), - 'flush': ( - "`iptables -w -F f2b-j-w-iptables-ap`", - "`ip6tables -w -F f2b-j-w-iptables-ap`", - ), - 'stop': ( - "`iptables -w -D INPUT -p tcp -j f2b-j-w-iptables-ap`", - "`iptables -w -F f2b-j-w-iptables-ap`", - "`iptables -w -X f2b-j-w-iptables-ap`", - "`ip6tables -w -D INPUT -p tcp -j f2b-j-w-iptables-ap`", - "`ip6tables -w -F f2b-j-w-iptables-ap`", - "`ip6tables -w -X f2b-j-w-iptables-ap`", - ), - 'ip4-check': ( - r"""`iptables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-ap[ \t]'`""", - ), - 'ip6-check': ( - r"""`ip6tables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-ap[ \t]'`""", - ), - 'ip4-ban': ( - r"`iptables -w -I f2b-j-w-iptables-ap 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", - ), - 'ip4-unban': ( - r"`iptables -w -D f2b-j-w-iptables-ap -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", - ), - 'ip6-ban': ( - r"`ip6tables -w -I f2b-j-w-iptables-ap 1 -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", - ), - 'ip6-unban': ( - r"`ip6tables -w -D f2b-j-w-iptables-ap -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", - ), - }), - # iptables-ipset-proto6 -- - ('j-w-iptables-ipset', 'iptables-ipset-proto6[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain=""]', { - 'ip4': (' f2b-j-w-iptables-ipset ',), 'ip6': (' f2b-j-w-iptables-ipset6 ',), - 'ip4-start': ( - "`ipset create f2b-j-w-iptables-ipset hash:ip timeout 600`", - "`iptables -w -I INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable`", - ), - 'ip6-start': ( - "`ipset create f2b-j-w-iptables-ipset6 hash:ip timeout 600 family inet6`", - "`ip6tables -w -I INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", - ), - 'flush': ( - "`ipset flush f2b-j-w-iptables-ipset`", - "`ipset flush f2b-j-w-iptables-ipset6`", - ), - 'stop': ( - "`iptables -w -D INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable`", - "`ipset flush f2b-j-w-iptables-ipset`", - "`ipset destroy f2b-j-w-iptables-ipset`", - "`ip6tables -w -D INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", - "`ipset flush f2b-j-w-iptables-ipset6`", - "`ipset destroy f2b-j-w-iptables-ipset6`", - ), - 'ip4-check': (), - 'ip6-check': (), - 'ip4-ban': ( - r"`ipset add f2b-j-w-iptables-ipset 192.0.2.1 timeout 600 -exist`", - ), - 'ip4-unban': ( - r"`ipset del f2b-j-w-iptables-ipset 192.0.2.1 -exist`", - ), - 'ip6-ban': ( - r"`ipset add f2b-j-w-iptables-ipset6 2001:db8:: timeout 600 -exist`", - ), - 'ip6-unban': ( - r"`ipset del f2b-j-w-iptables-ipset6 2001:db8:: -exist`", - ), - }), - # iptables-ipset-proto6-allports -- - ('j-w-iptables-ipset-ap', 'iptables-ipset-proto6-allports[name=%(__name__)s, bantime="10m", chain=""]', { - 'ip4': (' f2b-j-w-iptables-ipset-ap ',), 'ip6': (' f2b-j-w-iptables-ipset-ap6 ',), - 'ip4-start': ( - "`ipset create f2b-j-w-iptables-ipset-ap hash:ip timeout 600`", - "`iptables -w -I INPUT -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", - ), - 'ip6-start': ( - "`ipset create f2b-j-w-iptables-ipset-ap6 hash:ip timeout 600 family inet6`", - "`ip6tables -w -I INPUT -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", - ), - 'flush': ( - "`ipset flush f2b-j-w-iptables-ipset-ap`", - "`ipset flush f2b-j-w-iptables-ipset-ap6`", - ), - 'stop': ( - "`iptables -w -D INPUT -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", - "`ipset flush f2b-j-w-iptables-ipset-ap`", - "`ipset destroy f2b-j-w-iptables-ipset-ap`", - "`ip6tables -w -D INPUT -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", - "`ipset flush f2b-j-w-iptables-ipset-ap6`", - "`ipset destroy f2b-j-w-iptables-ipset-ap6`", - ), - 'ip4-check': (), - 'ip6-check': (), - 'ip4-ban': ( - r"`ipset add f2b-j-w-iptables-ipset-ap 192.0.2.1 timeout 600 -exist`", - ), - 'ip4-unban': ( - r"`ipset del f2b-j-w-iptables-ipset-ap 192.0.2.1 -exist`", - ), - 'ip6-ban': ( - r"`ipset add f2b-j-w-iptables-ipset-ap6 2001:db8:: timeout 600 -exist`", - ), - 'ip6-unban': ( - r"`ipset del f2b-j-w-iptables-ipset-ap6 2001:db8:: -exist`", - ), - }), - # iptables -- - ('j-w-iptables', 'iptables[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain=""]', { - 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), - 'ip4-start': ( - "`iptables -w -N f2b-j-w-iptables`", - "`iptables -w -A f2b-j-w-iptables -j RETURN`", - "`iptables -w -I INPUT -p tcp --dport http -j f2b-j-w-iptables`", - ), - 'ip6-start': ( - "`ip6tables -w -N f2b-j-w-iptables`", - "`ip6tables -w -A f2b-j-w-iptables -j RETURN`", - "`ip6tables -w -I INPUT -p tcp --dport http -j f2b-j-w-iptables`", - ), - 'flush': ( - "`iptables -w -F f2b-j-w-iptables`", - "`ip6tables -w -F f2b-j-w-iptables`", - ), - 'stop': ( - "`iptables -w -D INPUT -p tcp --dport http -j f2b-j-w-iptables`", - "`iptables -w -F f2b-j-w-iptables`", - "`iptables -w -X f2b-j-w-iptables`", - "`ip6tables -w -D INPUT -p tcp --dport http -j f2b-j-w-iptables`", - "`ip6tables -w -F f2b-j-w-iptables`", - "`ip6tables -w -X f2b-j-w-iptables`", - ), - 'ip4-check': ( - r"""`iptables -w -n -L INPUT | grep -q 'f2b-j-w-iptables[ \t]'`""", - ), - 'ip6-check': ( - r"""`ip6tables -w -n -L INPUT | grep -q 'f2b-j-w-iptables[ \t]'`""", - ), - 'ip4-ban': ( - r"`iptables -w -I f2b-j-w-iptables 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", - ), - 'ip4-unban': ( - r"`iptables -w -D f2b-j-w-iptables -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", - ), - 'ip6-ban': ( - r"`ip6tables -w -I f2b-j-w-iptables 1 -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", - ), - 'ip6-unban': ( - r"`ip6tables -w -D f2b-j-w-iptables -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", - ), - }), - # iptables-new -- - ('j-w-iptables-new', 'iptables-new[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain=""]', { - 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), - 'ip4-start': ( - "`iptables -w -N f2b-j-w-iptables-new`", - "`iptables -w -A f2b-j-w-iptables-new -j RETURN`", - "`iptables -w -I INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`", - ), - 'ip6-start': ( - "`ip6tables -w -N f2b-j-w-iptables-new`", - "`ip6tables -w -A f2b-j-w-iptables-new -j RETURN`", - "`ip6tables -w -I INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`", - ), - 'flush': ( - "`iptables -w -F f2b-j-w-iptables-new`", - "`ip6tables -w -F f2b-j-w-iptables-new`", - ), - 'stop': ( - "`iptables -w -D INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`", - "`iptables -w -F f2b-j-w-iptables-new`", - "`iptables -w -X f2b-j-w-iptables-new`", - "`ip6tables -w -D INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`", - "`ip6tables -w -F f2b-j-w-iptables-new`", - "`ip6tables -w -X f2b-j-w-iptables-new`", - ), - 'ip4-check': ( - r"""`iptables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-new[ \t]'`""", - ), - 'ip6-check': ( - r"""`ip6tables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-new[ \t]'`""", - ), - 'ip4-ban': ( - r"`iptables -w -I f2b-j-w-iptables-new 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", - ), - 'ip4-unban': ( - r"`iptables -w -D f2b-j-w-iptables-new -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", - ), - 'ip6-ban': ( - r"`ip6tables -w -I f2b-j-w-iptables-new 1 -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", - ), - 'ip6-unban': ( - r"`ip6tables -w -D f2b-j-w-iptables-new -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", - ), - }), - # iptables-xt_recent-echo -- - ('j-w-iptables-xtre', 'iptables-xt_recent-echo[name=%(__name__)s, bantime="10m", chain=""]', { - 'ip4': ('`iptables ', '/f2b-j-w-iptables-xtre`'), 'ip6': ('`ip6tables ', '/f2b-j-w-iptables-xtre6`'), - 'ip4-start': ( - "`if [ `id -u` -eq 0 ];then iptables -w -I INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre -j REJECT --reject-with icmp-port-unreachable;fi`", - ), - 'ip6-start': ( - "`if [ `id -u` -eq 0 ];then ip6tables -w -I INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre6 -j REJECT --reject-with icmp6-port-unreachable;fi`", - ), - 'stop': ( - "`echo / > /proc/net/xt_recent/f2b-j-w-iptables-xtre`", - "`if [ `id -u` -eq 0 ];then iptables -w -D INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre -j REJECT --reject-with icmp-port-unreachable;fi`", - "`echo / > /proc/net/xt_recent/f2b-j-w-iptables-xtre6`", - "`if [ `id -u` -eq 0 ];then ip6tables -w -D INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre6 -j REJECT --reject-with icmp6-port-unreachable;fi`", - ), - 'ip4-check': ( - r"`test -e /proc/net/xt_recent/f2b-j-w-iptables-xtre`", - ), - 'ip6-check': ( - r"`test -e /proc/net/xt_recent/f2b-j-w-iptables-xtre6`", - ), - 'ip4-ban': ( - r"`echo +192.0.2.1 > /proc/net/xt_recent/f2b-j-w-iptables-xtre`", - ), - 'ip4-unban': ( - r"`echo -192.0.2.1 > /proc/net/xt_recent/f2b-j-w-iptables-xtre`", - ), - 'ip6-ban': ( - r"`echo +2001:db8:: > /proc/net/xt_recent/f2b-j-w-iptables-xtre6`", - ), - 'ip6-unban': ( - r"`echo -2001:db8:: > /proc/net/xt_recent/f2b-j-w-iptables-xtre6`", - ), - }), - # pf default -- multiport on default port (tag set in jail.conf, but not in this test case) - ('j-w-pf', 'pf[name=%(__name__)s, actionstart_on_demand=false]', { - 'ip4': (), 'ip6': (), - 'start': ( - '`echo "table persist counters" | pfctl -a f2b/j-w-pf -f-`', - 'port=""', - '`echo "block quick proto tcp from to any port $port" | pfctl -a f2b/j-w-pf -f-`', - ), - 'flush': ( - '`pfctl -a f2b/j-w-pf -t f2b-j-w-pf -T flush`', - ), - 'stop': ( - '`pfctl -a f2b/j-w-pf -sr 2>/dev/null | grep -v f2b-j-w-pf | pfctl -a f2b/j-w-pf -f-`', - '`pfctl -a f2b/j-w-pf -t f2b-j-w-pf -T flush`', - '`pfctl -a f2b/j-w-pf -t f2b-j-w-pf -T kill`', - ), - 'ip4-check': ("`pfctl -a f2b/j-w-pf -sr | grep -q f2b-j-w-pf`",), - 'ip6-check': ("`pfctl -a f2b/j-w-pf -sr | grep -q f2b-j-w-pf`",), - 'ip4-ban': ("`pfctl -a f2b/j-w-pf -t f2b-j-w-pf -T add 192.0.2.1`",), - 'ip4-unban': ("`pfctl -a f2b/j-w-pf -t f2b-j-w-pf -T delete 192.0.2.1`",), - 'ip6-ban': ("`pfctl -a f2b/j-w-pf -t f2b-j-w-pf -T add 2001:db8::`",), - 'ip6-unban': ("`pfctl -a f2b/j-w-pf -t f2b-j-w-pf -T delete 2001:db8::`",), - }), - # pf multiport with custom ports -- - ('j-w-pf-mp', 'pf[actiontype=][name=%(__name__)s, port="http,https"]', { - 'ip4': (), 'ip6': (), - 'start': ( - '`echo "table persist counters" | pfctl -a f2b/j-w-pf-mp -f-`', - 'port="http,https"', - '`echo "block quick proto tcp from to any port $port" | pfctl -a f2b/j-w-pf-mp -f-`', - ), - 'flush': ( - '`pfctl -a f2b/j-w-pf-mp -t f2b-j-w-pf-mp -T flush`', - ), - 'stop': ( - '`pfctl -a f2b/j-w-pf-mp -sr 2>/dev/null | grep -v f2b-j-w-pf-mp | pfctl -a f2b/j-w-pf-mp -f-`', - '`pfctl -a f2b/j-w-pf-mp -t f2b-j-w-pf-mp -T flush`', - '`pfctl -a f2b/j-w-pf-mp -t f2b-j-w-pf-mp -T kill`', - ), - 'ip4-check': ("`pfctl -a f2b/j-w-pf-mp -sr | grep -q f2b-j-w-pf-mp`",), - 'ip6-check': ("`pfctl -a f2b/j-w-pf-mp -sr | grep -q f2b-j-w-pf-mp`",), - 'ip4-ban': ("`pfctl -a f2b/j-w-pf-mp -t f2b-j-w-pf-mp -T add 192.0.2.1`",), - 'ip4-unban': ("`pfctl -a f2b/j-w-pf-mp -t f2b-j-w-pf-mp -T delete 192.0.2.1`",), - 'ip6-ban': ("`pfctl -a f2b/j-w-pf-mp -t f2b-j-w-pf-mp -T add 2001:db8::`",), - 'ip6-unban': ("`pfctl -a f2b/j-w-pf-mp -t f2b-j-w-pf-mp -T delete 2001:db8::`",), - }), - # pf allports -- test additionally "actionstart_on_demand" was set to true - ('j-w-pf-ap', 'pf[actiontype=, actionstart_on_demand=true][name=%(__name__)s]', { - 'ip4': (), 'ip6': (), - 'ip4-start': ( - '`echo "table persist counters" | pfctl -a f2b/j-w-pf-ap -f-`', - '`echo "block quick proto tcp from to any" | pfctl -a f2b/j-w-pf-ap -f-`', - ), - 'ip6-start': (), # the same as ipv4 - 'flush': ( - '`pfctl -a f2b/j-w-pf-ap -t f2b-j-w-pf-ap -T flush`', - ), - 'stop': ( - '`pfctl -a f2b/j-w-pf-ap -sr 2>/dev/null | grep -v f2b-j-w-pf-ap | pfctl -a f2b/j-w-pf-ap -f-`', - '`pfctl -a f2b/j-w-pf-ap -t f2b-j-w-pf-ap -T flush`', - '`pfctl -a f2b/j-w-pf-ap -t f2b-j-w-pf-ap -T kill`', - ), - 'ip4-check': ("`pfctl -a f2b/j-w-pf-ap -sr | grep -q f2b-j-w-pf-ap`",), - 'ip6-check': ("`pfctl -a f2b/j-w-pf-ap -sr | grep -q f2b-j-w-pf-ap`",), - 'ip4-ban': ("`pfctl -a f2b/j-w-pf-ap -t f2b-j-w-pf-ap -T add 192.0.2.1`",), - 'ip4-unban': ("`pfctl -a f2b/j-w-pf-ap -t f2b-j-w-pf-ap -T delete 192.0.2.1`",), - 'ip6-ban': ("`pfctl -a f2b/j-w-pf-ap -t f2b-j-w-pf-ap -T add 2001:db8::`",), - 'ip6-unban': ("`pfctl -a f2b/j-w-pf-ap -t f2b-j-w-pf-ap -T delete 2001:db8::`",), - }), - # firewallcmd-multiport -- - ('j-w-fwcmd-mp', 'firewallcmd-multiport[name=%(__name__)s, bantime="10m", port="http,https", protocol="tcp", chain=""]', { - 'ip4': (' ipv4 ', 'icmp-port-unreachable'), 'ip6': (' ipv6 ', 'icmp6-port-unreachable'), - 'ip4-start': ( - "`firewall-cmd --direct --add-chain ipv4 filter f2b-j-w-fwcmd-mp`", - "`firewall-cmd --direct --add-rule ipv4 filter f2b-j-w-fwcmd-mp 1000 -j RETURN`", - "`firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports http,https -j f2b-j-w-fwcmd-mp`", - ), - 'ip6-start': ( - "`firewall-cmd --direct --add-chain ipv6 filter f2b-j-w-fwcmd-mp`", - "`firewall-cmd --direct --add-rule ipv6 filter f2b-j-w-fwcmd-mp 1000 -j RETURN`", - "`firewall-cmd --direct --add-rule ipv6 filter INPUT_direct 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports http,https -j f2b-j-w-fwcmd-mp`", - ), - 'stop': ( - "`firewall-cmd --direct --remove-rule ipv4 filter INPUT_direct 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports http,https -j f2b-j-w-fwcmd-mp`", - "`firewall-cmd --direct --remove-rules ipv4 filter f2b-j-w-fwcmd-mp`", - "`firewall-cmd --direct --remove-chain ipv4 filter f2b-j-w-fwcmd-mp`", - "`firewall-cmd --direct --remove-rule ipv6 filter INPUT_direct 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports http,https -j f2b-j-w-fwcmd-mp`", - "`firewall-cmd --direct --remove-rules ipv6 filter f2b-j-w-fwcmd-mp`", - "`firewall-cmd --direct --remove-chain ipv6 filter f2b-j-w-fwcmd-mp`", - ), - 'ip4-check': ( - r"`firewall-cmd --direct --get-chains ipv4 filter | sed -e 's, ,\n,g' | grep -q '^f2b-j-w-fwcmd-mp$'`", - ), - 'ip6-check': ( - r"`firewall-cmd --direct --get-chains ipv6 filter | sed -e 's, ,\n,g' | grep -q '^f2b-j-w-fwcmd-mp$'`", - ), - 'ip4-ban': ( - r"`firewall-cmd --direct --add-rule ipv4 filter f2b-j-w-fwcmd-mp 0 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", - ), - 'ip4-unban': ( - r"`firewall-cmd --direct --remove-rule ipv4 filter f2b-j-w-fwcmd-mp 0 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", - ), - 'ip6-ban': ( - r"`firewall-cmd --direct --add-rule ipv6 filter f2b-j-w-fwcmd-mp 0 -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", - ), - 'ip6-unban': ( - r"`firewall-cmd --direct --remove-rule ipv6 filter f2b-j-w-fwcmd-mp 0 -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", - ), - }), - # firewallcmd-allports -- - ('j-w-fwcmd-ap', 'firewallcmd-allports[name=%(__name__)s, bantime="10m", protocol="tcp", chain=""]', { - 'ip4': (' ipv4 ', 'icmp-port-unreachable'), 'ip6': (' ipv6 ', 'icmp6-port-unreachable'), - 'ip4-start': ( - "`firewall-cmd --direct --add-chain ipv4 filter f2b-j-w-fwcmd-ap`", - "`firewall-cmd --direct --add-rule ipv4 filter f2b-j-w-fwcmd-ap 1000 -j RETURN`", - "`firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 0 -j f2b-j-w-fwcmd-ap`", - ), - 'ip6-start': ( - "`firewall-cmd --direct --add-chain ipv6 filter f2b-j-w-fwcmd-ap`", - "`firewall-cmd --direct --add-rule ipv6 filter f2b-j-w-fwcmd-ap 1000 -j RETURN`", - "`firewall-cmd --direct --add-rule ipv6 filter INPUT_direct 0 -j f2b-j-w-fwcmd-ap`", - ), - 'stop': ( - "`firewall-cmd --direct --remove-rule ipv4 filter INPUT_direct 0 -j f2b-j-w-fwcmd-ap`", - "`firewall-cmd --direct --remove-rules ipv4 filter f2b-j-w-fwcmd-ap`", - "`firewall-cmd --direct --remove-chain ipv4 filter f2b-j-w-fwcmd-ap`", - "`firewall-cmd --direct --remove-rule ipv6 filter INPUT_direct 0 -j f2b-j-w-fwcmd-ap`", - "`firewall-cmd --direct --remove-rules ipv6 filter f2b-j-w-fwcmd-ap`", - "`firewall-cmd --direct --remove-chain ipv6 filter f2b-j-w-fwcmd-ap`", - ), - 'ip4-check': ( - r"`firewall-cmd --direct --get-chains ipv4 filter | sed -e 's, ,\n,g' | grep -q '^f2b-j-w-fwcmd-ap$'`", - ), - 'ip6-check': ( - r"`firewall-cmd --direct --get-chains ipv6 filter | sed -e 's, ,\n,g' | grep -q '^f2b-j-w-fwcmd-ap$'`", - ), - 'ip4-ban': ( - r"`firewall-cmd --direct --add-rule ipv4 filter f2b-j-w-fwcmd-ap 0 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", - ), - 'ip4-unban': ( - r"`firewall-cmd --direct --remove-rule ipv4 filter f2b-j-w-fwcmd-ap 0 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", - ), - 'ip6-ban': ( - r"`firewall-cmd --direct --add-rule ipv6 filter f2b-j-w-fwcmd-ap 0 -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", - ), - 'ip6-unban': ( - r"`firewall-cmd --direct --remove-rule ipv6 filter f2b-j-w-fwcmd-ap 0 -s 2001:db8:: -j REJECT --reject-with icmp6-port-unreachable`", - ), - }), - # firewallcmd-ipset (multiport) -- - ('j-w-fwcmd-ipset', 'firewallcmd-ipset[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain=""]', { - 'ip4': (' f2b-j-w-fwcmd-ipset ',), 'ip6': (' f2b-j-w-fwcmd-ipset6 ',), - 'ip4-start': ( - "`ipset create f2b-j-w-fwcmd-ipset hash:ip timeout 600`", - "`firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset src -j REJECT --reject-with icmp-port-unreachable`", - ), - 'ip6-start': ( - "`ipset create f2b-j-w-fwcmd-ipset6 hash:ip timeout 600 family inet6`", - "`firewall-cmd --direct --add-rule ipv6 filter INPUT_direct 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", - ), - 'flush': ( - "`ipset flush f2b-j-w-fwcmd-ipset`", - "`ipset flush f2b-j-w-fwcmd-ipset6`", - ), - 'stop': ( - "`firewall-cmd --direct --remove-rule ipv4 filter INPUT_direct 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset src -j REJECT --reject-with icmp-port-unreachable`", - "`ipset flush f2b-j-w-fwcmd-ipset`", - "`ipset destroy f2b-j-w-fwcmd-ipset`", - "`firewall-cmd --direct --remove-rule ipv6 filter INPUT_direct 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", - "`ipset flush f2b-j-w-fwcmd-ipset6`", - "`ipset destroy f2b-j-w-fwcmd-ipset6`", - ), - 'ip4-check': (), - 'ip6-check': (), - 'ip4-ban': ( - r"`ipset add f2b-j-w-fwcmd-ipset 192.0.2.1 timeout 600 -exist`", - ), - 'ip4-unban': ( - r"`ipset del f2b-j-w-fwcmd-ipset 192.0.2.1 -exist`", - ), - 'ip6-ban': ( - r"`ipset add f2b-j-w-fwcmd-ipset6 2001:db8:: timeout 600 -exist`", - ), - 'ip6-unban': ( - r"`ipset del f2b-j-w-fwcmd-ipset6 2001:db8:: -exist`", - ), - }), - # firewallcmd-ipset (allports) -- - ('j-w-fwcmd-ipset-ap', 'firewallcmd-ipset[name=%(__name__)s, bantime="10m", actiontype=, protocol="tcp", chain=""]', { - 'ip4': (' f2b-j-w-fwcmd-ipset-ap ',), 'ip6': (' f2b-j-w-fwcmd-ipset-ap6 ',), - 'ip4-start': ( - "`ipset create f2b-j-w-fwcmd-ipset-ap hash:ip timeout 600`", - "`firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 0 -p tcp -m set --match-set f2b-j-w-fwcmd-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", - ), - 'ip6-start': ( - "`ipset create f2b-j-w-fwcmd-ipset-ap6 hash:ip timeout 600 family inet6`", - "`firewall-cmd --direct --add-rule ipv6 filter INPUT_direct 0 -p tcp -m set --match-set f2b-j-w-fwcmd-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", - ), - 'flush': ( - "`ipset flush f2b-j-w-fwcmd-ipset-ap`", - "`ipset flush f2b-j-w-fwcmd-ipset-ap6`", - ), - 'stop': ( - "`firewall-cmd --direct --remove-rule ipv4 filter INPUT_direct 0 -p tcp -m set --match-set f2b-j-w-fwcmd-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", - "`ipset flush f2b-j-w-fwcmd-ipset-ap`", - "`ipset destroy f2b-j-w-fwcmd-ipset-ap`", - "`firewall-cmd --direct --remove-rule ipv6 filter INPUT_direct 0 -p tcp -m set --match-set f2b-j-w-fwcmd-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", - "`ipset flush f2b-j-w-fwcmd-ipset-ap6`", - "`ipset destroy f2b-j-w-fwcmd-ipset-ap6`", - ), - 'ip4-check': (), - 'ip6-check': (), - 'ip4-ban': ( - r"`ipset add f2b-j-w-fwcmd-ipset-ap 192.0.2.1 timeout 600 -exist`", - ), - 'ip4-unban': ( - r"`ipset del f2b-j-w-fwcmd-ipset-ap 192.0.2.1 -exist`", - ), - 'ip6-ban': ( - r"`ipset add f2b-j-w-fwcmd-ipset-ap6 2001:db8:: timeout 600 -exist`", - ), - 'ip6-unban': ( - r"`ipset del f2b-j-w-fwcmd-ipset-ap6 2001:db8:: -exist`", - ), - }), - ) - server = TestServer() - transm = server._Server__transm - cmdHandler = transm._Transmitter__commandHandler + jails = server._Server__jails - for jail, act, tests in testJailsActions: - stream = self.getDefaultJailStream(jail, act) - - # for cmd in stream: - # print(cmd) - - # transmit jail to the server: - for cmd in stream: - # command to server: - ret, res = transm.proceed(cmd) - self.assertEqual(ret, 0) - - jails = server._Server__jails - - tickets = { - 'ip4': BanTicket('192.0.2.1'), - 'ip6': BanTicket('2001:DB8::'), - } - for jail, act, tests in testJailsActions: - # print(jail, jails[jail]) - for a in jails[jail].actions: - action = jails[jail].actions[a] - logSys.debug('# ' + ('=' * 50)) - logSys.debug('# == %-44s ==', jail + ' - ' + action._name) - logSys.debug('# ' + ('=' * 50)) - self.assertTrue(isinstance(action, _actions.CommandAction)) - # wrap default command processor: - action.executeCmd = self._executeCmd - # test start : - self.pruneLog('# === start ===') - action.start() - if tests.get('start'): - self.assertLogged(*tests['start'], all=True) - else: - self.assertNotLogged(*tests['ip4-start']+tests['ip6-start'], all=True) - ainfo = { - 'ip4': _actions.Actions.ActionInfo(tickets['ip4'], jails[jail]), - 'ip6': _actions.Actions.ActionInfo(tickets['ip6'], jails[jail]), - } - # test ban ip4 : - self.pruneLog('# === ban-ipv4 ===') - action.ban(ainfo['ip4']) - if tests.get('ip4-start'): self.assertLogged(*tests['ip4-start'], all=True) - if tests.get('ip6-start'): self.assertNotLogged(*tests['ip6-start'], all=True) - self.assertLogged(*tests['ip4-check']+tests['ip4-ban'], all=True) - self.assertNotLogged(*tests['ip6'], all=True) - # test unban ip4 : - self.pruneLog('# === unban ipv4 ===') - action.unban(ainfo['ip4']) - self.assertLogged(*tests['ip4-check']+tests['ip4-unban'], all=True) - self.assertNotLogged(*tests['ip6'], all=True) - # test ban ip6 : - self.pruneLog('# === ban ipv6 ===') - action.ban(ainfo['ip6']) - if tests.get('ip6-start'): self.assertLogged(*tests['ip6-start'], all=True) - if tests.get('ip4-start'): self.assertNotLogged(*tests['ip4-start'], all=True) - self.assertLogged(*tests['ip6-check']+tests['ip6-ban'], all=True) - self.assertNotLogged(*tests['ip4'], all=True) - # test unban ip6 : - self.pruneLog('# === unban ipv6 ===') - action.unban(ainfo['ip6']) - self.assertLogged(*tests['ip6-check']+tests['ip6-unban'], all=True) - self.assertNotLogged(*tests['ip4'], all=True) - # test flush for actions should supported this: - if tests.get('flush'): - self.pruneLog('# === flush ===') - action.flush() - self.assertLogged(*tests['flush'], all=True) - # test stop : - self.pruneLog('# === stop ===') - action.stop() - self.assertLogged(*tests['stop'], all=True) - - def _executeMailCmd(self, realCmd, timeout=60): - # replace pipe to mail with pipe to cat: - realCmd = re.sub(r'\)\s*\|\s*mail\b([^\n]*)', - r') | cat; printf "\\n... | "; echo mail \1', realCmd) - # replace abuse retrieving (possible no-network), just replace first occurrence of 'dig...': - realCmd = re.sub(r'\bADDRESSES=\$\(dig\s[^\n]+', - lambda m: 'ADDRESSES="abuse-1@abuse-test-server, abuse-2@abuse-test-server"', - realCmd, 1) - # execute action: - return _actions.CommandAction.executeCmd(realCmd, timeout=timeout) - - def testComplexMailActionMultiLog(self): - testJailsActions = ( - # mail-whois-lines -- - ('j-mail-whois-lines', - 'mail-whois-lines[' - 'name=%(__name__)s, grepopts="-m 1", grepmax=2, mailcmd="mail -s", ' + - # 2 logs to test grep from multiple logs: - 'logpath="' + os.path.join(TEST_FILES_DIR, "testcase01.log") + '\n' + - ' ' + os.path.join(TEST_FILES_DIR, "testcase01a.log") + '", ' - '_whois_command="echo \'-- information about --\'"' - ']', - { - 'ip4-ban': ( - 'The IP 87.142.124.10 has just been banned by Fail2Ban after', - '100 attempts against j-mail-whois-lines.', - 'Here is more information about 87.142.124.10 :', - '-- information about 87.142.124.10 --', - 'Lines containing failures of 87.142.124.10 (max 2)', - 'testcase01.log:Dec 31 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 87.142.124.10', - 'testcase01a.log:Dec 31 11:55:01 [sshd] error: PAM: Authentication failure for test from 87.142.124.10', - ), - }), - # complain -- - ('j-complain-abuse', - 'complain[' - 'name=%(__name__)s, grepopts="-m 1", grepmax=2, mailcmd="mail -s \'Hostname: , family: \' - ",' + - # test reverse ip: - 'debug=1,' + - # 2 logs to test grep from multiple logs: - 'logpath="' + os.path.join(TEST_FILES_DIR, "testcase01.log") + '\n' + - ' ' + os.path.join(TEST_FILES_DIR, "testcase01a.log") + '", ' - ']', - { - 'ip4-ban': ( - # test reverse ip: - 'try to resolve 10.124.142.87.abuse-contacts.abusix.org', - 'Lines containing failures of 87.142.124.10 (max 2)', - 'testcase01.log:Dec 31 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 87.142.124.10', - 'testcase01a.log:Dec 31 11:55:01 [sshd] error: PAM: Authentication failure for test from 87.142.124.10', - # both abuse mails should be separated with space: - 'mail -s Hostname: test-host, family: inet4 - Abuse from 87.142.124.10 abuse-1@abuse-test-server abuse-2@abuse-test-server', - ), - 'ip6-ban': ( - # test reverse ip: - 'try to resolve 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.abuse-contacts.abusix.org', - 'Lines containing failures of 2001:db8::1 (max 2)', - # both abuse mails should be separated with space: - 'mail -s Hostname: test-host, family: inet6 - Abuse from 2001:db8::1 abuse-1@abuse-test-server abuse-2@abuse-test-server', - ), - }), - ) - server = TestServer() - transm = server._Server__transm - cmdHandler = transm._Transmitter__commandHandler - - for jail, act, tests in testJailsActions: - stream = self.getDefaultJailStream(jail, act) - - # for cmd in stream: - # print(cmd) - - # transmit jail to the server: - for cmd in stream: - # command to server: - ret, res = transm.proceed(cmd) - self.assertEqual(ret, 0) - - jails = server._Server__jails - - ipv4 = IPAddr('87.142.124.10') - ipv6 = IPAddr('2001:db8::1'); - dmyjail = DummyJail() - for jail, act, tests in testJailsActions: - # print(jail, jails[jail]) - for a in jails[jail].actions: - action = jails[jail].actions[a] - logSys.debug('# ' + ('=' * 50)) - logSys.debug('# == %-44s ==', jail + ' - ' + action._name) - logSys.debug('# ' + ('=' * 50)) - # wrap default command processor: - action.executeCmd = self._executeMailCmd - # test ban : - for (test, ip) in (('ip4-ban', ipv4), ('ip6-ban', ipv6)): - if not tests.get(test): continue - self.pruneLog('# === %s ===' % test) - ticket = BanTicket(ip) - ticket.setAttempt(100) - ticket = _actions.Actions.ActionInfo(ticket, dmyjail) - action.ban(ticket) - self.assertLogged(*tests[test], all=True) + ipv4 = IPAddr('87.142.124.10') + ipv6 = IPAddr('2001:db8::1'); + dmyjail = DummyJail() + for jail, act, tests in testJailsActions: + # print(jail, jails[jail]) + for a in jails[jail].actions: + action = jails[jail].actions[a] + logSys.debug('# ' + ('=' * 50)) + logSys.debug('# == %-44s ==', jail + ' - ' + action._name) + logSys.debug('# ' + ('=' * 50)) + # wrap default command processor: + action.executeCmd = self._executeMailCmd + # test ban : + for (test, ip) in (('ip4-ban', ipv4), ('ip6-ban', ipv6)): + if not tests.get(test): continue + self.pruneLog('# === %s ===' % test) + ticket = BanTicket(ip) + ticket.setAttempt(100) + ticket = _actions.Actions.ActionInfo(ticket, dmyjail) + action.ban(ticket) + self.assertLogged(*tests[test], all=True) diff --git a/fail2ban/tests/utils.py b/fail2ban/tests/utils.py index 0cf40e86..05d8d666 100644 --- a/fail2ban/tests/utils.py +++ b/fail2ban/tests/utils.py @@ -59,6 +59,9 @@ if not CONFIG_DIR: else: CONFIG_DIR = '/etc/fail2ban' +# Indicates that we've stock config: +STOCK = os.path.exists(os.path.join(CONFIG_DIR, 'fail2ban.conf')) + # During the test cases (or setup) use fail2ban modules from main directory: os.putenv('PYTHONPATH', os.path.dirname(os.path.dirname(os.path.dirname( os.path.abspath(__file__))))) @@ -187,6 +190,31 @@ class F2B(DefaultTestOptions): pass def SkipIfNoNetwork(self): pass + + def SkipIfCfgMissing(self, **kwargs): + """Helper to check action/filter config is available + """ + if not STOCK: # pragma: no cover + if kwargs.get('stock'): + raise unittest.SkipTest('Skip test because of missing stock-config files') + for t in ('action', 'filter'): + v = kwargs.get(t) + if v is None: continue + if os.path.splitext(v)[1] == '': v += '.conf' + if not os.path.exists(os.path.join(CONFIG_DIR, t+'.d', v)): + raise unittest.SkipTest('Skip test because of missing %s-config for %r' % (t, v)) + + def skip_if_cfg_missing(self, **decargs): + """Helper decorator to check action/filter config is available + """ + def _deco_wrapper(f): + @wraps(f) + def wrapper(self, *args, **kwargs): + unittest.F2B.SkipIfCfgMissing(**decargs) + return f(self, *args, **kwargs) + return wrapper + return _deco_wrapper + def maxWaitTime(self,wtime): if self.fast: wtime = float(wtime) / 10 From ac9d5f61e7e9151f7518c26db192dd6d0bf56591 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 24 Jan 2018 15:47:05 +0100 Subject: [PATCH 016/189] rewrite keywords reserved in python 3.7 (`async` -> `nonsync`) --- fail2ban/client/fail2banclient.py | 6 +++--- fail2ban/client/fail2banserver.py | 23 +++++++++++++---------- 2 files changed, 16 insertions(+), 13 deletions(-) diff --git a/fail2ban/client/fail2banclient.py b/fail2ban/client/fail2banclient.py index 0a1ae4f1..13ebcdef 100755 --- a/fail2ban/client/fail2banclient.py +++ b/fail2ban/client/fail2banclient.py @@ -228,9 +228,9 @@ class Fail2banClient(Fail2banCmdLine, Thread): return True ## - def configureServer(self, async=True, phase=None): - # if asynchron start this operation in the new thread: - if async: + def configureServer(self, nonsync=True, phase=None): + # if asynchronous start this operation in the new thread: + if nonsync: th = Thread(target=Fail2banClient.configureServer, args=(self, False, phase)) th.daemon = True return th.start() diff --git a/fail2ban/client/fail2banserver.py b/fail2ban/client/fail2banserver.py index 6c57fbf8..afe36cf4 100644 --- a/fail2ban/client/fail2banserver.py +++ b/fail2ban/client/fail2banserver.py @@ -164,21 +164,24 @@ class Fail2banServer(Fail2banCmdLine): cli = self._Fail2banClient() return cli.start(argv) - # Start the server: - from ..server.utils import Utils - # background = True, if should be new process running in background, otherwise start in foreground - # process will be forked in daemonize, inside of Server module. - # async = True, if started from client, should... + # Start the server, corresponding options: + # background = True, if should be new process running in background, otherwise start in + # foreground process will be forked in daemonize, inside of Server module. + # nonsync = True, normally internal call only, if started from client, so configures + # the server via asynchronous thread. background = self._conf["background"] - async = self._conf.get("async", False) + nonsync = self._conf.get("async", False) + # If was started not from the client: - if not async: + if not nonsync: + # Load requirements on demand (we need utils only when asynchronous handling): + from ..server.utils import Utils # Start new thread with client to read configuration and # transfer it to the server: cli = self._Fail2banClient() phase = dict() logSys.debug('Configure via async client thread') - cli.configureServer(async=True, phase=phase) + cli.configureServer(phase=phase) # wait, do not continue if configuration is not 100% valid: Utils.wait_for(lambda: phase.get('ready', None) is not None, self._conf["timeout"], 0.001) logSys.log(5, ' server phase %s', phase) @@ -195,7 +198,7 @@ class Fail2banServer(Fail2banCmdLine): pid = os.getpid() server = Fail2banServer.startServerDirect(self._conf, background) # notify waiting thread server ready resp. done (background execution, error case, etc): - if not async: + if not nonsync: _server_ready() # If forked - just exit other processes if pid != os.getpid(): # pragma: no cover @@ -204,7 +207,7 @@ class Fail2banServer(Fail2banCmdLine): cli._server = server # wait for client answer "done": - if not async and cli: + if not nonsync and cli: Utils.wait_for(lambda: phase.get('done', None) is not None, self._conf["timeout"], 0.001) if not phase.get('done', False): if server: # pragma: no cover From 5f3ba289d6490e8614c1a0f0e4ccab11e44e20c5 Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 24 Jan 2018 20:03:46 +0100 Subject: [PATCH 017/189] restore code coverage (decreased in latest "fixes") - no cover for unreachable cases; --- fail2ban/tests/filtertestcase.py | 4 ++-- fail2ban/tests/utils.py | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 59c7820a..b5877b7f 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -673,10 +673,10 @@ class LogFileMonitor(LogCaptureTestCase): failure_was_logged = self._is_logged('Unable to open %s' % self.name) # verify that we cannot access the file. Checking by name of user is not # sufficient since could be a fakeroot or some other super-user + is_root = True try: - with open(self.name) as f: + with open(self.name) as f: # pragma: no cover - normally no root f.read() - is_root = True except IOError: is_root = False diff --git a/fail2ban/tests/utils.py b/fail2ban/tests/utils.py index 05d8d666..2bcc587b 100644 --- a/fail2ban/tests/utils.py +++ b/fail2ban/tests/utils.py @@ -56,7 +56,7 @@ if not CONFIG_DIR: # Use heuristic to figure out where configuration files are if os.path.exists(os.path.join('config','fail2ban.conf')): CONFIG_DIR = 'config' - else: + else: # pragma: no cover - normally unreachable CONFIG_DIR = '/etc/fail2ban' # Indicates that we've stock config: From f547a7c7b1f9b6e72ad8fae6ec5e18a6bc7bfbcc Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 26 Jan 2018 20:25:11 +0100 Subject: [PATCH 018/189] LogCaptureTestCase: use almost non-blocking handling by getvalue/_is_logged (especially important in tests with waiting for logged via `assertLogged(..., wait=TO)`): - try to acquire lock without blocking, if not possible - return cached/empty (max 5 times, otherwise do lock); - minimized time of the lock of messages list; - avoid sporadic dead-locking during cross lock together with lock within handling of self._strm. --- fail2ban/tests/utils.py | 48 +++++++++++++++++++++++++++-------------- 1 file changed, 32 insertions(+), 16 deletions(-) diff --git a/fail2ban/tests/utils.py b/fail2ban/tests/utils.py index 2bcc587b..f681db76 100644 --- a/fail2ban/tests/utils.py +++ b/fail2ban/tests/utils.py @@ -640,7 +640,9 @@ class LogCaptureTestCase(unittest.TestCase): def __init__(self, lazy=True): self._lock = threading.Lock() self._val = None + self._dirty = True self._recs = list() + self._nolckCntr = 0 self._strm = StringIO() logging.Handler.__init__(self) if lazy: @@ -650,10 +652,11 @@ class LogCaptureTestCase(unittest.TestCase): """Truncate the internal buffer and records.""" if size: raise Exception('invalid size argument: %r, should be None or 0' % size) + self._val = None + self._dirty = True with self._lock: - self._strm.truncate(0) - self._val = None self._recs = list() + self._strm.truncate(0) def __write(self, record): msg = record.getMessage() + '\n' @@ -664,29 +667,42 @@ class LogCaptureTestCase(unittest.TestCase): def getvalue(self): """Return current buffer as whole string.""" - with self._lock: - # cached: - if self._val is not None: - return self._val - # submit already emitted (delivered to handle) records: - for record in self._recs: - self.__write(record) - self._recs = list() - # cache and return: - self._val = self._strm.getvalue() + # if cached (still unchanged/no write operation), we don't need to enter lock: + if not self._dirty: return self._val + # try to lock, if not possible - return cached/empty (max 5 times): + lck = self._lock.acquire(False) + if not lck: # pargma: no cover (may be too sporadic on slow systems) + self._nolckCntr += 1 + if self._nolckCntr <= 5: + return self._val if self._val is not None else '' + self._nolckCntr = 0 + self._lock.acquire() + # minimize time of lock, avoid dead-locking during cross lock within self._strm ... + try: + recs = self._recs + self._recs = list() + finally: + self._lock.release() + # submit already emitted (delivered to handle) records: + for record in recs: + self.__write(record) + # cache and return: + self._val = self._strm.getvalue() + self._dirty = False + return self._val def handle(self, record): # pragma: no cover """Handle the specified record direct (not lazy)""" - with self._lock: - self._val = None - self.__write(record) + self.__write(record) + self._dirty = True def _handle_lazy(self, record): """Lazy handle the specified record on demand""" with self._lock: - self._val = None self._recs.append(record) + # logged - causes changed string buffer (signal by set _dirty): + self._dirty = True def setUp(self): From 435f359a065576f2537c07d24fc1d58277cd190e Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 26 Jan 2018 21:34:10 +0100 Subject: [PATCH 019/189] allow substitute section-related parameters like `` in all config-readers as well as during substitute after supply of init arguments; test cases extended; --- fail2ban/client/configparserinc.py | 76 +++++++++++++++---- fail2ban/client/configreader.py | 2 +- fail2ban/tests/clientreadertestcase.py | 17 ++++- fail2ban/tests/files/filter.d/substition.conf | 3 + 4 files changed, 80 insertions(+), 18 deletions(-) diff --git a/fail2ban/client/configparserinc.py b/fail2ban/client/configparserinc.py index 722f4618..70dfd91b 100644 --- a/fail2ban/client/configparserinc.py +++ b/fail2ban/client/configparserinc.py @@ -33,7 +33,7 @@ if sys.version_info >= (3,2): # SafeConfigParser deprecated from Python 3.2 (renamed to ConfigParser) from configparser import ConfigParser as SafeConfigParser, BasicInterpolation, \ - InterpolationMissingOptionError, NoSectionError + InterpolationMissingOptionError, NoOptionError, NoSectionError # And interpolation of __name__ was simply removed, thus we need to # decorate default interpolator to handle it @@ -63,7 +63,7 @@ if sys.version_info >= (3,2): else: # pragma: no cover from ConfigParser import SafeConfigParser, \ - InterpolationMissingOptionError, NoSectionError + InterpolationMissingOptionError, NoOptionError, NoSectionError # Interpolate missing known/option as option from default section SafeConfigParser._cp_interpolate_some = SafeConfigParser._interpolate_some @@ -112,6 +112,8 @@ after = 1.conf SECTION_NAME = "INCLUDES" + SECTION_OPTNAME_CRE = re.compile(r'^([\w\-]+)/([^\s>]+)$') + SECTION_OPTSUBST_CRE = re.compile(r'%\(([\w\-]+/([^\)]+))\)s') CONDITIONAL_RE = re.compile(r"^(\w+)(\?.+)$") @@ -131,7 +133,36 @@ after = 1.conf SafeConfigParser.__init__(self, *args, **kwargs) self._cfg_share = share_config - def _map_section_options(self, section, option, rest, map): + def get_ex(self, section, option, raw=False, vars={}): + """Get an option value for a given section. + + In opposite to `get`, it differentiate session-related option name like `sec/opt`. + """ + sopt = None + # if option name contains section: + if '/' in option: + sopt = SafeConfigParserWithIncludes.SECTION_OPTNAME_CRE.search(option) + # try get value from named section/option: + if sopt: + sec = sopt.group(1) + opt = sopt.group(2) + seclwr = sec.lower() + if seclwr == 'known': + # try get value firstly from known options, hereafter from current section: + sopt = ('KNOWN/'+section, section) + else: + sopt = (sec,) if seclwr != 'default' else ("DEFAULT",) + for sec in sopt: + try: + v = self.get(sec, opt, raw=raw) + return v + except (NoSectionError, NoOptionError) as e: + pass + # get value of section/option using given section and vars (fallback): + v = self.get(section, option, raw=raw, vars=vars) + return v + + def _map_section_options(self, section, option, rest, defaults): """ Interpolates values of the section options (name syntax `%(section/option)s`). @@ -139,37 +170,54 @@ after = 1.conf """ if '/' not in rest or '%(' not in rest: # pragma: no cover return 0 + rplcmnt = 0 soptrep = SafeConfigParserWithIncludes.SECTION_OPTSUBST_CRE.findall(rest) if not soptrep: # pragma: no cover return 0 for sopt, opt in soptrep: - if sopt not in map: + if sopt not in defaults: sec = sopt[:~len(opt)] seclwr = sec.lower() if seclwr != 'default': + usedef = 0 if seclwr == 'known': # try get raw value from known options: try: v = self._sections['KNOWN/'+section][opt] except KeyError: # fallback to default: - try: - v = self._defaults[opt] - except KeyError: # pragma: no cover - continue + usedef = 1 else: # get raw value of opt in section: - v = self.get(sec, opt, raw=True) + try: + # if section not found - ignore: + try: + sec = self._sections[sec] + except KeyError: # pragma: no cover + continue + v = sec[opt] + except KeyError: # pragma: no cover + # fallback to default: + usedef = 1 else: + usedef = 1 + if usedef: try: v = self._defaults[opt] except KeyError: # pragma: no cover continue - self._defaults[sopt] = v - try: # for some python versions need to duplicate it in map-vars also: - map[sopt] = v - except: pass - return 1 + # replacement found: + rplcmnt = 1 + try: # set it in map-vars (consider different python versions): + defaults[sopt] = v + except: + # try to set in first default map (corresponding vars): + try: + defaults._maps[0][sopt] = v + except: # pragma: no cover + # no way to update vars chain map - overwrite defaults: + self._defaults[sopt] = v + return rplcmnt @property def share_config(self): diff --git a/fail2ban/client/configreader.py b/fail2ban/client/configreader.py index 577a5a16..2248ec34 100644 --- a/fail2ban/client/configreader.py +++ b/fail2ban/client/configreader.py @@ -351,7 +351,7 @@ class DefinitionInitConfigReader(ConfigReader): return self._defCache[optname] except KeyError: try: - v = self.get("Definition", optname, vars=self._pOpts) + v = self._cfg.get_ex("Definition", optname, vars=self._pOpts) except (NoSectionError, NoOptionError, ValueError): v = None self._defCache[optname] = v diff --git a/fail2ban/tests/clientreadertestcase.py b/fail2ban/tests/clientreadertestcase.py index 0472b770..6c0d9226 100644 --- a/fail2ban/tests/clientreadertestcase.py +++ b/fail2ban/tests/clientreadertestcase.py @@ -188,7 +188,7 @@ y = %(jail/y)s self.assertEqual(self.c.get('jail', 'c'), 'def-c,b:"jail-b-test-b-def-b,a:`jail-a-test-a-def-a`"') self.assertEqual(self.c.get('jail', 'd'), 'def-d-b:"def-b,a:`jail-a-test-a-def-a`"') self.assertEqual(self.c.get('test', 'c'), 'def-c,b:"test-b-def-b,a:`test-a-def-a`"') - self.assertEqual(self.c.get('test', 'd'), 'def-d-b:"def-b,a:`test-a-def-a`"') + self.assertEqual(self.c.get('test', 'd'), 'def-d-b:"def-b,a:`test-a-def-a`"') self.assertEqual(self.c.get('DEFAULT', 'c'), 'def-c,b:"def-b,a:`def-a`"') self.assertEqual(self.c.get('DEFAULT', 'd'), 'def-d-b:"def-b,a:`def-a`"') self.assertRaises(Exception, self.c.get, 'test', 'x') @@ -437,9 +437,20 @@ class FilterReaderTest(unittest.TestCase): self.assertSortedEqual(c, output) def testFilterReaderSubstitionKnown(self): - output = [['set', 'jailname', 'addfailregex', 'to=test,sweet@example.com,test2,sweet@example.com fromip=']] + output = [['set', 'jailname', 'addfailregex', '^to=test,sweet@example.com,test2,sweet@example.com fromip=$']] filterName, filterOpt = extractOptions( - 'substition[honeypot=",", sweet="test,,test2"]') + 'substition[failregex="^$", honeypot=",", sweet="test,,test2"]') + filterReader = FilterReader('substition', "jailname", filterOpt, + share_config=TEST_FILES_DIR_SHARE_CFG, basedir=TEST_FILES_DIR) + filterReader.read() + filterReader.getOptions(None) + c = filterReader.convert() + self.assertSortedEqual(c, output) + + def testFilterReaderSubstitionSection(self): + output = [['set', 'jailname', 'addfailregex', '^\s*to=fail2ban@localhost fromip=\s*$']] + filterName, filterOpt = extractOptions( + 'substition[failregex="^\s*\s*$", honeypot=""]') filterReader = FilterReader('substition', "jailname", filterOpt, share_config=TEST_FILES_DIR_SHARE_CFG, basedir=TEST_FILES_DIR) filterReader.read() diff --git a/fail2ban/tests/files/filter.d/substition.conf b/fail2ban/tests/files/filter.d/substition.conf index aaf62eae..862a3cac 100644 --- a/fail2ban/tests/files/filter.d/substition.conf +++ b/fail2ban/tests/files/filter.d/substition.conf @@ -1,3 +1,6 @@ +[DEFAULT] + +honeypot = fail2ban@localhost [Definition] From 03b577d7b92a120e325abe20a99b6956a7e0657c Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 30 Jan 2018 12:27:03 +0100 Subject: [PATCH 020/189] action.d/blocklist_de.conf: fixed tag substitution (in 0.10 it can be variables supplied via shell-arguments), expand `` with trailing newline; tests extended; closes gh-2028 --- config/action.d/blocklist_de.conf | 4 +--- fail2ban/tests/fail2banclienttestcase.py | 30 +++++++++++++++++------- 2 files changed, 22 insertions(+), 12 deletions(-) diff --git a/config/action.d/blocklist_de.conf b/config/action.d/blocklist_de.conf index 2f31d8b9..246f90f7 100644 --- a/config/action.d/blocklist_de.conf +++ b/config/action.d/blocklist_de.conf @@ -54,7 +54,7 @@ actioncheck = # Tags: See jail.conf(5) man page # Values: CMD # -actionban = curl --fail --data-urlencode 'server=' --data 'apikey=' --data 'service=' --data 'ip=' --data-urlencode 'logs=' --data 'format=text' --user-agent "" "https://www.blocklist.de/en/httpreports.html" +actionban = lgm=$(printf 'logs=%%s\n...' ""); curl --fail --data-urlencode "server=" --data "apikey=" --data "service=" --data "ip=" --data-urlencode "$lgm" --data 'format=text' --user-agent "" "https://www.blocklist.de/en/httpreports.html" # Option: actionunban # Notes.: command executed when unbanning an IP. Take care that the @@ -64,8 +64,6 @@ actionban = curl --fail --data-urlencode 'server=' --data 'apikey=\', email="Fail2Ban ", ' + 'apikey="TEST-API-KEY", agent="fail2ban-test-agent", service=]', 'filter =', 'datepattern = ^Epoch', 'failregex = ^ failure "[^"]+" - ', @@ -1219,6 +1223,14 @@ class Fail2banServerTest(Fail2banClientServerBase): self.assertIn('\\125-000-004 1;\n', mp) self.assertIn('\\125-000-005 1;\n', mp) + # check blocklist_de substitution: + self.assertLogged( + "stdout: '*** curl --fail --data-urlencode server=Fail2Ban " + " --data apikey=TEST-API-KEY --data service=nginx-blck-lst ", + "stdout: '... --data format=text --user-agent fail2ban-test-agent", + all=True, wait=MID_WAITTIME + ) + # unban 1, 2 and 5: self.execCmd(SUCCESS, startparams, 'unban', '125-000-001', '125-000-002', '125-000-005') _out_file(mpfn) From 0be0e43d475ab9cec83b747134023279d1c95eb3 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 30 Jan 2018 12:52:26 +0100 Subject: [PATCH 021/189] amend to 03b577d7b92a120e325abe20a99b6956a7e0657c: add new-line after matches via tag `
` without usage of interim variable --- config/action.d/blocklist_de.conf | 2 +- fail2ban/tests/fail2banclienttestcase.py | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/config/action.d/blocklist_de.conf b/config/action.d/blocklist_de.conf index 246f90f7..b9cd0584 100644 --- a/config/action.d/blocklist_de.conf +++ b/config/action.d/blocklist_de.conf @@ -54,7 +54,7 @@ actioncheck = # Tags: See jail.conf(5) man page # Values: CMD # -actionban = lgm=$(printf 'logs=%%s\n...' ""); curl --fail --data-urlencode "server=" --data "apikey=" --data "service=" --data "ip=" --data-urlencode "$lgm" --data 'format=text' --user-agent "" "https://www.blocklist.de/en/httpreports.html" +actionban = curl --fail --data-urlencode "server=" --data "apikey=" --data "service=" --data "ip=" --data-urlencode "logs=
" --data 'format=text' --user-agent "" "https://www.blocklist.de/en/httpreports.html" # Option: actionunban # Notes.: command executed when unbanning an IP. Take care that the diff --git a/fail2ban/tests/fail2banclienttestcase.py b/fail2ban/tests/fail2banclienttestcase.py index 92fdfa5c..e346f09d 100644 --- a/fail2ban/tests/fail2banclienttestcase.py +++ b/fail2ban/tests/fail2banclienttestcase.py @@ -1223,11 +1223,11 @@ class Fail2banServerTest(Fail2banClientServerBase): self.assertIn('\\125-000-004 1;\n', mp) self.assertIn('\\125-000-005 1;\n', mp) - # check blocklist_de substitution: + # check blocklist_de substitution (e. g. new-line after ): self.assertLogged( "stdout: '*** curl --fail --data-urlencode server=Fail2Ban " - " --data apikey=TEST-API-KEY --data service=nginx-blck-lst ", - "stdout: '... --data format=text --user-agent fail2ban-test-agent", + " --data apikey=TEST-API-KEY --data service=nginx-blck-lst ", + "stdout: ' --data format=text --user-agent fail2ban-test-agent", all=True, wait=MID_WAITTIME ) From 0ed11817c197f045ef7a5f82cc9fb33a4d5f1657 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 30 Jan 2018 13:30:31 +0100 Subject: [PATCH 022/189] restore coverage: no cover for normally unreachable scopes (only if test cases failed) --- fail2ban/tests/fail2banclienttestcase.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fail2ban/tests/fail2banclienttestcase.py b/fail2ban/tests/fail2banclienttestcase.py index e346f09d..92e08bfc 100644 --- a/fail2ban/tests/fail2banclienttestcase.py +++ b/fail2ban/tests/fail2banclienttestcase.py @@ -373,7 +373,7 @@ class Fail2banClientServerBase(LogCaptureTestCase): sock = pjoin(tmp, "f2b.sock") # wait for server (socket): ret = Utils.wait_for(lambda: phase.get('end') or exists(sock), MAX_WAITTIME) - if not ret or phase.get('end'): + if not ret or phase.get('end'): # pragma: no cover - test-failure case only raise Exception( 'Unexpected: Socket file does not exists.\nStart failed: %r' % (startparams,) @@ -381,7 +381,7 @@ class Fail2banClientServerBase(LogCaptureTestCase): if ready: # wait for communication with worker ready: ret = Utils.wait_for(lambda: "Server ready" in self.getLog(), MAX_WAITTIME) - if not ret: + if not ret: # pragma: no cover - test-failure case only raise Exception( 'Unexpected: Server ready was not found.\nStart failed: %r' % (startparams,) From 442b0b1c59c904fb6961a605be399f57029cabd5 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 30 Jan 2018 14:41:38 +0100 Subject: [PATCH 023/189] extends date-detector with long epoch (LEPOCH) to parse milliseconds/microseconds posix-dates; provide opportunity to specify own regex-pattern to match epoch date-time, e. g. "^\[{EPOCH}\]"; closes gh-2029 --- fail2ban/server/datedetector.py | 23 +++++++++----- fail2ban/server/datetemplate.py | 27 ++++++++++++++--- fail2ban/tests/datedetectortestcase.py | 42 ++++++++++++++++++++++++++ 3 files changed, 79 insertions(+), 13 deletions(-) diff --git a/fail2ban/server/datedetector.py b/fail2ban/server/datedetector.py index 42df308e..13d70699 100644 --- a/fail2ban/server/datedetector.py +++ b/fail2ban/server/datedetector.py @@ -26,7 +26,8 @@ import time from threading import Lock -from .datetemplate import re, DateTemplate, DatePatternRegex, DateTai64n, DateEpoch +from .datetemplate import re, DateTemplate, DatePatternRegex, DateTai64n, DateEpoch, \ + RE_EPOCH_PATTERN from .strptime import validateTimeZone from .utils import Utils from ..helpers import getLogger @@ -36,7 +37,7 @@ logSys = getLogger(__name__) logLevel = 6 -RE_DATE_PREMATCH = re.compile("\{DATE\}", re.IGNORECASE) +RE_DATE_PREMATCH = re.compile(r"(?(?<=^\[))|(?P(?<=\baudit\()))\d{10,11}\b(?:\.\d{3,6})?)(?:(?(selinux)(?=:\d+\)))|(?(square)(?=\])))" + self._longFrm = longFrm; + epochRE = r"\d{10,11}\b(?:\.\d{3,6})?" + if longFrm: + self.name = "LongEpoch"; + epochRE = r"\d{10,11}(?:\d{3}(?:\d{3})?)?" + if pattern: + regex = RE_EPOCH_PATTERN.sub("(%s)" % epochRE, pattern) + self.setRegex(regex) + elif not lineBeginOnly: + regex = r"((?:^|(?P(?<=^\[))|(?P(?<=\baudit\()))%s)(?:(?(selinux)(?=:\d+\)))|(?(square)(?=\])))" % epochRE self.setRegex(regex, wordBegin=False) ;# already line begin resp. word begin anchored else: - regex = r"((?P(?<=^\[))?\d{10,11}\b(?:\.\d{3,6})?)(?(square)(?=\]))" + regex = r"((?P(?<=^\[))?%s)(?(square)(?=\]))" % epochRE self.setRegex(regex, wordBegin='start', wordEnd=True) def getDate(self, line, dateMatch=None, default_tz=None): @@ -220,8 +231,14 @@ class DateEpoch(DateTemplate): if not dateMatch: dateMatch = self.matchDate(line) if dateMatch: + v = dateMatch.group(1) # extract part of format which represents seconds since epoch - return (float(dateMatch.group(1)), dateMatch) + if self._longFrm and len(v) >= 13: + if len(v) >= 16: + v = float(v) / 1000000 + else: + v = float(v) / 1000 + return (float(v), dateMatch) class DatePatternRegex(DateTemplate): diff --git a/fail2ban/tests/datedetectortestcase.py b/fail2ban/tests/datedetectortestcase.py index 02facf30..69473c9d 100644 --- a/fail2ban/tests/datedetectortestcase.py +++ b/fail2ban/tests/datedetectortestcase.py @@ -77,6 +77,48 @@ class DateDetectorTest(LogCaptureTestCase): log = date + " [sshd] error: PAM: Authentication failure" datelog = self.datedetector.getTime(log) self.assertFalse(datelog) + + def testGetEpochMsTime(self): + self.__datedetector = DateDetector() + self.__datedetector.appendTemplate('LEPOCH') + # correct short/long epoch time, using all variants: + for fact in (1, 1000, 1000000): + for dateUnix in (1138049999, 32535244799): + for date in ("%s", "[%s]", "[%s]", "audit(%s:101)"): + dateLong = dateUnix * fact + date = date % dateLong + log = date + " [sshd] error: PAM: Authentication failure" + datelog = self.datedetector.getTime(log) + self.assertTrue(datelog, "Parse epoch time for %s failed" % (date,)) + ( datelog, matchlog ) = datelog + self.assertEqual(int(datelog), dateUnix) + self.assertEqual(matchlog.group(1), str(dateLong)) + # wrong, no epoch time (< 10 digits, more as 17 digits, begin/end of word) : + for dateUnix in ('123456789', '999999999999999999', '1138049999A', 'A1138049999'): + for date in ("%s", "[%s]", "[%s.555]", "audit(%s.555:101)"): + date = date % dateUnix + log = date + " [sshd] error: PAM: Authentication failure" + datelog = self.datedetector.getTime(log) + self.assertFalse(datelog) + + def testGetEpochPattern(self): + self.__datedetector = DateDetector() + self.__datedetector.appendTemplate('\|\s{LEPOCH}(?=\s\|)') + # correct short/long epoch time, using all variants: + for fact in (1, 1000, 1000000): + for dateUnix in (1138049999, 32535244799): + dateLong = dateUnix * fact + log = "auth-error | %s | invalid password" % dateLong + datelog = self.datedetector.getTime(log) + self.assertTrue(datelog, "Parse epoch time failed: %r" % (log,)) + ( datelog, matchlog ) = datelog + self.assertEqual(int(datelog), dateUnix) + self.assertEqual(matchlog.group(1), str(dateLong)) + # wrong epoch time format (does not match pattern): + for log in ("test%s123", "test-right | %stest", "test%s | test-left"): + log = log % dateLong + datelog = self.datedetector.getTime(log) + self.assertFalse(datelog) def testGetTime(self): log = "Jan 23 21:59:59 [sshd] error: PAM: Authentication failure" From 3e8098d4274b5f00585c201b57bc7165f02c3218 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 30 Jan 2018 15:10:17 +0100 Subject: [PATCH 024/189] python 3.x compatibility: fix replacement string (may fail with errors like `bad escape \d ...`, etc) --- fail2ban/server/datetemplate.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fail2ban/server/datetemplate.py b/fail2ban/server/datetemplate.py index 4c9bb4e3..cd4592c7 100644 --- a/fail2ban/server/datetemplate.py +++ b/fail2ban/server/datetemplate.py @@ -204,7 +204,7 @@ class DateEpoch(DateTemplate): self.name = "LongEpoch"; epochRE = r"\d{10,11}(?:\d{3}(?:\d{3})?)?" if pattern: - regex = RE_EPOCH_PATTERN.sub("(%s)" % epochRE, pattern) + regex = RE_EPOCH_PATTERN.sub(lambda v: "(%s)" % epochRE, pattern) self.setRegex(regex) elif not lineBeginOnly: regex = r"((?:^|(?P(?<=^\[))|(?P(?<=\baudit\()))%s)(?:(?(selinux)(?=:\d+\)))|(?(square)(?=\])))" % epochRE From dcbf9048760d3ada651789a0bb71b18e998ec040 Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 30 Jan 2018 16:40:04 +0100 Subject: [PATCH 025/189] allow to parse milliseconds as float + more test cases; normalize capturing with epoch-pattern match - similar to `{DATE}` should capture and cut out the whole pattern match from the log-line; --- fail2ban/server/datetemplate.py | 11 +++++++---- fail2ban/tests/datedetectortestcase.py | 2 +- fail2ban/tests/fail2banregextestcase.py | 11 +++++++++++ 3 files changed, 19 insertions(+), 5 deletions(-) diff --git a/fail2ban/server/datetemplate.py b/fail2ban/server/datetemplate.py index cd4592c7..49fa0c66 100644 --- a/fail2ban/server/datetemplate.py +++ b/fail2ban/server/datetemplate.py @@ -199,12 +199,15 @@ class DateEpoch(DateTemplate): DateTemplate.__init__(self) self.name = "Epoch" self._longFrm = longFrm; + self._grpIdx = 1 epochRE = r"\d{10,11}\b(?:\.\d{3,6})?" if longFrm: self.name = "LongEpoch"; - epochRE = r"\d{10,11}(?:\d{3}(?:\d{3})?)?" + epochRE = r"\d{10,11}(?:\d{3}(?:\.\d{1,6}|\d{3})?)?" if pattern: - regex = RE_EPOCH_PATTERN.sub(lambda v: "(%s)" % epochRE, pattern) + # pattern should capture/cut out the whole match: + regex = "(" + RE_EPOCH_PATTERN.sub(lambda v: "(%s)" % epochRE, pattern) + ")" + self._grpIdx = 2 self.setRegex(regex) elif not lineBeginOnly: regex = r"((?:^|(?P(?<=^\[))|(?P(?<=\baudit\()))%s)(?:(?(selinux)(?=:\d+\)))|(?(square)(?=\])))" % epochRE @@ -231,10 +234,10 @@ class DateEpoch(DateTemplate): if not dateMatch: dateMatch = self.matchDate(line) if dateMatch: - v = dateMatch.group(1) + v = dateMatch.group(self._grpIdx) # extract part of format which represents seconds since epoch if self._longFrm and len(v) >= 13: - if len(v) >= 16: + if len(v) >= 16 and '.' not in v: v = float(v) / 1000000 else: v = float(v) / 1000 diff --git a/fail2ban/tests/datedetectortestcase.py b/fail2ban/tests/datedetectortestcase.py index 69473c9d..36471489 100644 --- a/fail2ban/tests/datedetectortestcase.py +++ b/fail2ban/tests/datedetectortestcase.py @@ -103,7 +103,7 @@ class DateDetectorTest(LogCaptureTestCase): def testGetEpochPattern(self): self.__datedetector = DateDetector() - self.__datedetector.appendTemplate('\|\s{LEPOCH}(?=\s\|)') + self.__datedetector.appendTemplate('(?<=\|\s){LEPOCH}(?=\s\|)') # correct short/long epoch time, using all variants: for fact in (1, 1000, 1000000): for dateUnix in (1138049999, 32535244799): diff --git a/fail2ban/tests/fail2banregextestcase.py b/fail2ban/tests/fail2banregextestcase.py index f3a51773..148d774c 100644 --- a/fail2ban/tests/fail2banregextestcase.py +++ b/fail2ban/tests/fail2banregextestcase.py @@ -290,6 +290,17 @@ class Fail2banRegexTest(LogCaptureTestCase): self.assertTrue(fail2banRegex.start(args)) self.assertLogged('Lines: 1 lines, 0 ignored, 1 matched, 0 missed') + def testRegexEpochPatterns(self): + (opts, args, fail2banRegex) = _Fail2banRegex( + "-r", "-d", "^\[{LEPOCH}\]\s+", "--maxlines", "5", + "[1516469849] 192.0.2.1 FAIL: failure\n" + "[1516469849551] 192.0.2.2 FAIL: failure\n" + "[1516469849551000] 192.0.2.3 FAIL: failure\n" + "[1516469849551.000] 192.0.2.4 FAIL: failure", + r"^ FAIL\b" + ) + self.assertTrue(fail2banRegex.start(args)) + self.assertLogged('Lines: 4 lines, 0 ignored, 4 matched, 0 missed') def testWrongFilterFile(self): # use test log as filter file to cover eror cases... From 3a1c38695843e37493e88ea7354eccda51a0abcf Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Wed, 31 Jan 2018 12:18:56 +0100 Subject: [PATCH 026/189] Update ChangeLog --- ChangeLog | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/ChangeLog b/ChangeLog index 3813a156..e0e244df 100644 --- a/ChangeLog +++ b/ChangeLog @@ -39,6 +39,10 @@ ver. 0.10.3-dev-1 (20??/??/??) - development edition ### New Features ### Enhancements +* date-detector extended with long epoch (`LEPOCH`) to parse milliseconds/microseconds posix-dates (gh-2029); +* possibility to specify own regex-pattern to match epoch date-time, e. g. `^\[{EPOCH}\]` or `^\[{LEPOCH}\]` (gh-2038); + the epoch-pattern similar to `{DATE}` patterns does the capture and cuts out the match of whole pattern from the log-line, + e. g. date-pattern `^\[{LEPOCH}\]\s+:` will match and cut out `[1516469849551000] :` from begin of the log-line. ver. 0.10.2 (2018/01/18) - nothing-burns-like-the-cold From 2c03f5ad28a1025cf1aed9724d05eb9723428154 Mon Sep 17 00:00:00 2001 From: sebres Date: Thu, 1 Feb 2018 18:04:04 +0100 Subject: [PATCH 027/189] simple syntax change (wrong escape in string): no functional changes --- fail2ban/tests/clientreadertestcase.py | 4 ++-- fail2ban/tests/fail2banregextestcase.py | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/fail2ban/tests/clientreadertestcase.py b/fail2ban/tests/clientreadertestcase.py index 6c0d9226..9e5f0dfe 100644 --- a/fail2ban/tests/clientreadertestcase.py +++ b/fail2ban/tests/clientreadertestcase.py @@ -448,9 +448,9 @@ class FilterReaderTest(unittest.TestCase): self.assertSortedEqual(c, output) def testFilterReaderSubstitionSection(self): - output = [['set', 'jailname', 'addfailregex', '^\s*to=fail2ban@localhost fromip=\s*$']] + output = [['set', 'jailname', 'addfailregex', '^\\s*to=fail2ban@localhost fromip=\\s*$']] filterName, filterOpt = extractOptions( - 'substition[failregex="^\s*\s*$", honeypot=""]') + 'substition[failregex="^\\s*\\s*$", honeypot=""]') filterReader = FilterReader('substition', "jailname", filterOpt, share_config=TEST_FILES_DIR_SHARE_CFG, basedir=TEST_FILES_DIR) filterReader.read() diff --git a/fail2ban/tests/fail2banregextestcase.py b/fail2ban/tests/fail2banregextestcase.py index 148d774c..b1786c03 100644 --- a/fail2ban/tests/fail2banregextestcase.py +++ b/fail2ban/tests/fail2banregextestcase.py @@ -292,7 +292,7 @@ class Fail2banRegexTest(LogCaptureTestCase): def testRegexEpochPatterns(self): (opts, args, fail2banRegex) = _Fail2banRegex( - "-r", "-d", "^\[{LEPOCH}\]\s+", "--maxlines", "5", + "-r", "-d", r"^\[{LEPOCH}\]\s+", "--maxlines", "5", "[1516469849] 192.0.2.1 FAIL: failure\n" "[1516469849551] 192.0.2.2 FAIL: failure\n" "[1516469849551000] 192.0.2.3 FAIL: failure\n" From f1661d35eabdb9700acb28eb46dbeb98983a216c Mon Sep 17 00:00:00 2001 From: sebres Date: Thu, 1 Feb 2018 18:30:00 +0100 Subject: [PATCH 028/189] fix sporadic error, wait for shutdown/exit messages by assert-check, better prevention of dual (parallel) stop --- fail2ban/tests/fail2banclienttestcase.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/fail2ban/tests/fail2banclienttestcase.py b/fail2ban/tests/fail2banclienttestcase.py index 92e08bfc..318a1c50 100644 --- a/fail2ban/tests/fail2banclienttestcase.py +++ b/fail2ban/tests/fail2banclienttestcase.py @@ -312,7 +312,9 @@ def with_foreground_server_thread(startextra={}): # wait for end sign: Utils.wait_for(lambda: phase.get('end', None) is not None, MAX_WAITTIME) self.assertTrue(phase.get('end', None)) - self.assertLogged("Shutdown successful", "Exiting Fail2ban", all=True) + self.assertLogged("Shutdown successful", "Exiting Fail2ban", all=True, wait=MAX_WAITTIME) + # set to NOP: avoid dual call + self.stopAndWaitForServerEnd = lambda *args, **kwargs: None self.stopAndWaitForServerEnd = _stopAndWaitForServerEnd # wait for start thread: Utils.wait_for(lambda: phase.get('start', None) is not None, MAX_WAITTIME) @@ -333,7 +335,6 @@ def with_foreground_server_thread(startextra={}): # so don't kill (same process) - if success, just wait for end of worker: if phase.get('end', None): th.join() - self.stopAndWaitForServerEnd = None return wrapper return _deco_wrapper From 3b8b9c4acf3834643ae97ac2efa54ed251a1cc8d Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Tue, 6 Feb 2018 17:02:23 +0100 Subject: [PATCH 029/189] README: extended systemd-prerequirements: python-systemd package --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 84eb3302..89ada1a1 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,8 @@ Optional: - [pyinotify >= 0.8.3](https://github.com/seb-m/pyinotify) - Linux >= 2.6.13 - [gamin >= 0.0.21](http://www.gnome.org/~veillard/gamin) -- [systemd >= 204](http://www.freedesktop.org/wiki/Software/systemd) +- [systemd >= 204](http://www.freedesktop.org/wiki/Software/systemd) and python bindings: + - [python-systemd package](https://www.freedesktop.org/software/systemd/python-systemd/index.html) - [dnspython](http://www.dnspython.org/) To install, just do: From 98f4d70371c4d168df86af2653f314ea6ad60a0d Mon Sep 17 00:00:00 2001 From: sebres Date: Tue, 6 Feb 2018 18:00:50 +0100 Subject: [PATCH 030/189] fixed cymru-info resolved to answers with multiple data-records (as comma-separated list now) --- fail2ban/server/banmanager.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/fail2ban/server/banmanager.py b/fail2ban/server/banmanager.py index 1275d3a4..a341a7af 100644 --- a/fail2ban/server/banmanager.py +++ b/fail2ban/server/banmanager.py @@ -166,15 +166,21 @@ class BanManager: answers = resolver.query(question, "TXT") if not answers: raise ValueError("No data retrieved") + asns = set() + countries = set() + rirs = set() for rdata in answers: asn, net, country, rir, changed =\ [answer.strip("'\" ") for answer in rdata.to_text().split("|")] asn = self.handleBlankResult(asn) country = self.handleBlankResult(country) rir = self.handleBlankResult(rir) - return_dict["asn"].append(self.handleBlankResult(asn)) - return_dict["country"].append(self.handleBlankResult(country)) - return_dict["rir"].append(self.handleBlankResult(rir)) + asns.add(self.handleBlankResult(asn)) + countries.add(self.handleBlankResult(country)) + rirs.add(self.handleBlankResult(rir)) + return_dict["asn"].append(', '.join(sorted(asns))) + return_dict["country"].append(', '.join(sorted(countries))) + return_dict["rir"].append(', '.join(sorted(rirs))) except dns.resolver.NXDOMAIN: return_dict["asn"].append("nxdomain") return_dict["country"].append("nxdomain") From 3e1534334338e5105cb8503a92df8170ee7da277 Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 9 Feb 2018 11:34:02 +0100 Subject: [PATCH 031/189] fail2ban-regex: bug fixed, if running over systemd-journal and `journalmatch` missing in filter. --- fail2ban/client/fail2banregex.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fail2ban/client/fail2banregex.py b/fail2ban/client/fail2banregex.py index c6338da5..b03c5cf6 100644 --- a/fail2ban/client/fail2banregex.py +++ b/fail2ban/client/fail2banregex.py @@ -614,7 +614,7 @@ class Fail2banRegex(object): self.setDatePattern(None) if journalmatch: flt.addJournalMatch(journalmatch) - output( "Use journal match : %s" % " ".join(journalmatch) ) + output( "Use journal match : %s" % " ".join(journalmatch) ) test_lines = journal_lines_gen(flt, myjournal) else: # if single line parsing (without buffering) From 19a5a2f8c03d2c3b1fd3e7622db12619a1488ade Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 9 Feb 2018 11:43:55 +0100 Subject: [PATCH 032/189] filter.d/murmur.conf: fixed detection of failures reading from journal (systemd-backend only): - extended with optional prefix for the systemd-journal (with second date-pattern as optional match); - added `journalmatch` filtering; closes gh-2043 --- config/filter.d/murmur.conf | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/config/filter.d/murmur.conf b/config/filter.d/murmur.conf index f5f100a6..efdaf8b5 100644 --- a/config/filter.d/murmur.conf +++ b/config/filter.d/murmur.conf @@ -1,11 +1,6 @@ # Fail2Ban filter for murmur/mumble-server # -[INCLUDES] - -before = common.conf - - [Definition] _daemon = murmurd @@ -15,7 +10,13 @@ _daemon = murmurd # variable in your server config file (murmur.ini / mumble-server.ini). _usernameregex = [^>]+ -_prefix = \s+\d+ => <\d+:%(_usernameregex)s\(-1\)> Rejected connection from :\d+: +# Prefix for systemd-journal (with second date-pattern as optional match): +# +__prefix_journal = (?:\S+\s+%(_daemon)s\[\d+\]:(?:\s+\[\d\-]+ [\d:]+.\d+)?) + +__prefix_line = %(__prefix_journal)s? + +_prefix = %(__prefix_line)s\s+\d+ => <\d+:%(_usernameregex)s\(-1\)> Rejected connection from :\d+: prefregex = ^%(_prefix)s .+$ @@ -26,6 +27,8 @@ ignoreregex = datepattern = ^{DATE} +journalmatch = _SYSTEMD_UNIT=murmurd.service + _COMM=murmurd + # DEV Notes: # # Author: Ross Brown From e636567d2305a8be20c869a6b3f9ce7a6e8447ef Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 19 Feb 2018 09:50:46 +0100 Subject: [PATCH 033/189] filter.d/exim.conf: failregex extended with SMTP call dropped: too many syntax or protocol errors. --- config/filter.d/exim.conf | 2 +- fail2ban/tests/files/logs/exim | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/config/filter.d/exim.conf b/config/filter.d/exim.conf index f1e56a75..6a8c12c5 100644 --- a/config/filter.d/exim.conf +++ b/config/filter.d/exim.conf @@ -20,7 +20,7 @@ failregex = ^%(pid)s %(host_info)ssender verify fail for <\S+>: (?:Unknown user| ^%(pid)s \w+ authenticator failed for (?:[^\[\( ]* )?(?:\(\S*\) )?\[\](?::\d+)?(?: I=\[\S+\](:\d+)?)?: 535 Incorrect authentication data( \(set_id=.*\)|: \d+ Time\(s\))?\s*$ ^%(pid)s %(host_info)srejected RCPT [^@]+@\S+: (?:relay not permitted|Sender verify failed|Unknown user|Unrouteable address)\s*$ ^%(pid)s SMTP protocol synchronization error \([^)]*\): rejected (?:connection from|"\S+") %(host_info)s(?:next )?input=".*"\s*$ - ^%(pid)s SMTP call from \S+ %(host_info)sdropped: too many nonmail commands \(last was "\S+"\)\s*$ + ^%(pid)s SMTP call from (?:[^\[\( ]* )?%(host_info)sdropped: too many (?:nonmail commands|syntax or protocol errors) \(last (?:command )?was "[^"]*"\)\s*$ ^%(pid)s SMTP protocol error in "[^"]+(?:"+[^"]*(?="))*?" %(host_info)sAUTH command used when not advertised\s*$ ^%(pid)s no MAIL in SMTP connection from (?:[^\[\( ]* )?(?:\(\S*\) )?%(host_info)sD=\d\S*s(?: C=\S*)?\s*$ ^%(pid)s (?:[\w\-]+ )?SMTP connection from (?:[^\[\( ]* )?(?:\(\S*\) )?%(host_info)sclosed by DROP in ACL\s*$ diff --git a/fail2ban/tests/files/logs/exim b/fail2ban/tests/files/logs/exim index ea3c5dca..79437a90 100644 --- a/fail2ban/tests/files/logs/exim +++ b/fail2ban/tests/files/logs/exim @@ -20,6 +20,8 @@ 2013-06-02 09:05:48 [18505] SMTP protocol synchronization error (next input sent too soon: pipelining was not advertised): rejected "RSET" H=ba77.mx83.fr [82.96.160.77]:58302 I=[1.2.3.4]:25 next input="QUIT\r\n" # failJSON: { "time": "2013-06-02T09:22:05", "match": true , "host": "163.14.21.161" } 2013-06-02 09:22:05 [19591] SMTP call from pc012-6201.spo.scu.edu.tw [163.14.21.161]:3767 I=[1.2.3.4]:25 dropped: too many nonmail commands (last was "RSET") +# failJSON: { "time": "2013-06-02T09:22:06", "match": true , "host": "192.0.2.109" } +2013-06-02 09:22:06 SMTP call from [192.0.2.109] dropped: too many syntax or protocol errors (last command was "AUTH LOGIN") # failJSON: { "time": "2013-06-02T15:06:18", "match": true , "host": "46.20.35.114" } 2013-06-02 15:06:18 H=(VM-WIN2K3-1562) [46.20.35.114] sender verify fail for : Unknown user # failJSON: { "time": "2013-06-07T02:02:09", "match": true , "host": "91.232.21.92" } From 879f580c9a26c2ad0c4d77349ef7b6df0a4cc823 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Mon, 19 Feb 2018 15:59:45 +0100 Subject: [PATCH 034/189] Update ChangeLog --- ChangeLog | 1 + 1 file changed, 1 insertion(+) diff --git a/ChangeLog b/ChangeLog index e0e244df..4bee6288 100644 --- a/ChangeLog +++ b/ChangeLog @@ -35,6 +35,7 @@ ver. 0.10.3-dev-1 (20??/??/??) - development edition ----------- ### Fixes +* `filter.d/exim.conf`: failregex extended - SMTP call dropped: too many syntax or protocol errors (gh-2048); ### New Features From e2665d39fd5e1b108873dca28cbff01c453fea5d Mon Sep 17 00:00:00 2001 From: benrubson Date: Mon, 26 Feb 2018 09:58:37 +0100 Subject: [PATCH 035/189] Use httpS with badips --- config/action.d/badips.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/action.d/badips.py b/config/action.d/badips.py index 473fbf33..03fe7856 100644 --- a/config/action.d/badips.py +++ b/config/action.d/badips.py @@ -81,7 +81,7 @@ class BadIPsAction(ActionBase): # pragma: no cover - may be unavailable """ TIMEOUT = 10 - _badips = "http://www.badips.com" + _badips = "https://www.badips.com" def _Request(self, url, **argv): return Request(url, headers={'User-Agent': self.agent}, **argv) From 9a8add0ef04103428c51145327b0c7fb22ec612f Mon Sep 17 00:00:00 2001 From: benrubson Date: Mon, 26 Feb 2018 10:28:51 +0100 Subject: [PATCH 036/189] changelog --- ChangeLog | 1 + 1 file changed, 1 insertion(+) diff --git a/ChangeLog b/ChangeLog index 4bee6288..b16827a2 100644 --- a/ChangeLog +++ b/ChangeLog @@ -44,6 +44,7 @@ ver. 0.10.3-dev-1 (20??/??/??) - development edition * possibility to specify own regex-pattern to match epoch date-time, e. g. `^\[{EPOCH}\]` or `^\[{LEPOCH}\]` (gh-2038); the epoch-pattern similar to `{DATE}` patterns does the capture and cuts out the match of whole pattern from the log-line, e. g. date-pattern `^\[{LEPOCH}\]\s+:` will match and cut out `[1516469849551000] :` from begin of the log-line. +* badips.py now uses https instead of plain http when requesting badips.com (gh-2057); ver. 0.10.2 (2018/01/18) - nothing-burns-like-the-cold From fce2a501657269b4a191978317d0bc96aa56f518 Mon Sep 17 00:00:00 2001 From: benrubson Date: Mon, 26 Feb 2018 15:55:21 +0100 Subject: [PATCH 037/189] badips.py, solve a str() issue under FreeBSD --- config/action.d/badips.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/action.d/badips.py b/config/action.d/badips.py index 473fbf33..da80017f 100644 --- a/config/action.d/badips.py +++ b/config/action.d/badips.py @@ -368,7 +368,7 @@ class BadIPsAction(ActionBase): # pragma: no cover - may be unavailable Any issues with badips.com request. """ try: - url = "/".join([self._badips, "add", self.category, aInfo['ip']]) + url = "/".join([self._badips, "add", self.category, str(aInfo['ip'])]) if self.key: url = "?".join([url, urlencode({'key': self.key})]) response = urlopen(self._Request(url), timeout=self.timeout) From 8ed892b8bb7637e07b937688f412a5218dffa047 Mon Sep 17 00:00:00 2001 From: benrubson Date: Mon, 26 Feb 2018 16:15:29 +0100 Subject: [PATCH 038/189] Changelog --- ChangeLog | 1 + 1 file changed, 1 insertion(+) diff --git a/ChangeLog b/ChangeLog index 4bee6288..a91f2c68 100644 --- a/ChangeLog +++ b/ChangeLog @@ -44,6 +44,7 @@ ver. 0.10.3-dev-1 (20??/??/??) - development edition * possibility to specify own regex-pattern to match epoch date-time, e. g. `^\[{EPOCH}\]` or `^\[{LEPOCH}\]` (gh-2038); the epoch-pattern similar to `{DATE}` patterns does the capture and cuts out the match of whole pattern from the log-line, e. g. date-pattern `^\[{LEPOCH}\]\s+:` will match and cut out `[1516469849551000] :` from begin of the log-line. +* `action.d/badips.py`: implicit convert IPAddr to str, solves an issue "expected string, IPAddr found" (gh-2059); ver. 0.10.2 (2018/01/18) - nothing-burns-like-the-cold From 095a909c8417958e19174779c27abf1f22ca5a86 Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 26 Feb 2018 17:53:34 +0100 Subject: [PATCH 039/189] action.d/badips.py: increases score in test-cases (5 now, ssh/3 returns to large list, which can cause timeout if badips server to busy); --- fail2ban/tests/action_d/test_badips.py | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/fail2ban/tests/action_d/test_badips.py b/fail2ban/tests/action_d/test_badips.py index 2d08b5df..28228544 100644 --- a/fail2ban/tests/action_d/test_badips.py +++ b/fail2ban/tests/action_d/test_badips.py @@ -52,6 +52,9 @@ if sys.version_info >= (2,7): # pragma: no cover - may be unavailable self.jail.actions.add("badips", pythonModule, initOpts={ 'category': "ssh", 'banaction': "test", + 'score': 5, + 'key': "fail2ban-test-suite", + #'bankey': "fail2ban-test-suite", 'timeout': (3 if unittest.F2B.fast else 30), }) self.action = self.jail.actions["badips"] @@ -80,8 +83,8 @@ if sys.version_info >= (2,7): # pragma: no cover - may be unavailable def testScore(self): self.assertRaises(ValueError, setattr, self.action, "score", -5) - self.action.score = 5 - self.action.score = "5" + self.action.score = 3 + self.action.score = "3" def testBanaction(self): self.assertRaises( @@ -97,11 +100,9 @@ if sys.version_info >= (2,7): # pragma: no cover - may be unavailable self.action.updateperiod = 900 self.action.updateperiod = "900" - def testStart(self): + def testStartStop(self): self.action.start() - self.assertTrue(len(self.action._bannedips) > 10) - - def testStop(self): - self.testStart() + self.assertTrue(len(self.action._bannedips) > 10, + "%s is fewer as 10: %r" % (len(self.action._bannedips), self.action._bannedips)) self.action.stop() self.assertTrue(len(self.action._bannedips) == 0) From f52c67238ada1c22df529142f284bf3677730cc4 Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 26 Feb 2018 18:16:20 +0100 Subject: [PATCH 040/189] action.d/badips.py: code review, ban command covered, debug log-messages, etc; --- config/action.d/badips.py | 2 ++ fail2ban/tests/action_d/test_badips.py | 31 +++++++++++++++++++++----- 2 files changed, 27 insertions(+), 6 deletions(-) diff --git a/config/action.d/badips.py b/config/action.d/badips.py index da80017f..4e3a1cde 100644 --- a/config/action.d/badips.py +++ b/config/action.d/badips.py @@ -186,6 +186,7 @@ class BadIPsAction(ActionBase): # pragma: no cover - may be unavailable urlencode({'age': age})]) if key: url = "&".join([url, urlencode({'key': key})]) + self._logSys.debug('badips.com: get list, url: %r', url) response = urlopen(self._Request(url), timeout=self.timeout) except HTTPError as response: messages = json.loads(response.read().decode('utf-8')) @@ -371,6 +372,7 @@ class BadIPsAction(ActionBase): # pragma: no cover - may be unavailable url = "/".join([self._badips, "add", self.category, str(aInfo['ip'])]) if self.key: url = "?".join([url, urlencode({'key': self.key})]) + self._logSys.debug('badips.com: ban, url: %r', url) response = urlopen(self._Request(url), timeout=self.timeout) except HTTPError as response: messages = json.loads(response.read().decode('utf-8')) diff --git a/fail2ban/tests/action_d/test_badips.py b/fail2ban/tests/action_d/test_badips.py index 28228544..781d0b05 100644 --- a/fail2ban/tests/action_d/test_badips.py +++ b/fail2ban/tests/action_d/test_badips.py @@ -21,15 +21,18 @@ import os import unittest import sys +from ..actiontestcase import CallingMap from ..dummyjail import DummyJail -from ..utils import CONFIG_DIR +from ..servertestcase import IPAddr +from ..utils import LogCaptureTestCase, CONFIG_DIR if sys.version_info >= (2,7): # pragma: no cover - may be unavailable - class BadIPsActionTest(unittest.TestCase): + class BadIPsActionTest(LogCaptureTestCase): available = True, None + pythonModule = None modAction = None - + def setUp(self): """Call before every test case.""" super(BadIPsActionTest, self).setUp() @@ -39,17 +42,20 @@ if sys.version_info >= (2,7): # pragma: no cover - may be unavailable self.jail.actions.add("test") - pythonModule = os.path.join(CONFIG_DIR, "action.d", "badips.py") + pythonModuleName = os.path.join(CONFIG_DIR, "action.d", "badips.py") # check availability (once if not alive, used shorter timeout as in test cases): if BadIPsActionTest.available[0]: if not BadIPsActionTest.modAction: - BadIPsActionTest.modAction = self.jail.actions._load_python_module(pythonModule).Action + if not BadIPsActionTest.pythonModule: + BadIPsActionTest.pythonModule = self.jail.actions._load_python_module(pythonModuleName) + BadIPsActionTest.modAction = BadIPsActionTest.pythonModule.Action + self.jail.actions._load_python_module(pythonModuleName) BadIPsActionTest.available = BadIPsActionTest.modAction.isAvailable(timeout=2 if unittest.F2B.fast else 10) if not BadIPsActionTest.available[0]: raise unittest.SkipTest('Skip test because service is not available: %s' % BadIPsActionTest.available[1]) - self.jail.actions.add("badips", pythonModule, initOpts={ + self.jail.actions.add("badips", pythonModuleName, initOpts={ 'category': "ssh", 'banaction': "test", 'score': 5, @@ -106,3 +112,16 @@ if sys.version_info >= (2,7): # pragma: no cover - may be unavailable "%s is fewer as 10: %r" % (len(self.action._bannedips), self.action._bannedips)) self.action.stop() self.assertTrue(len(self.action._bannedips) == 0) + + def testBanIP(self): + aInfo = CallingMap({ + 'ip': IPAddr('192.0.2.1') + }) + self.action.ban(aInfo) + self.assertLogged('badips.com: ban', wait=True) + self.pruneLog() + # produce an error using wrong category/IP: + self.action._category = 'f2b-this-category-dont-available-test-suite-only' + aInfo['ip'] = '' + self.assertRaises(BadIPsActionTest.pythonModule.HTTPError, self.action.ban, aInfo) + self.assertLogged('IP is invalid', 'invalid category', wait=True, all=False) From 933670d0343b59371b5919b3c0e6c5e2bedbd66f Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 26 Feb 2018 18:35:30 +0100 Subject: [PATCH 041/189] pragma: no cover - availability (once after error case only) --- fail2ban/tests/banmanagertestcase.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fail2ban/tests/banmanagertestcase.py b/fail2ban/tests/banmanagertestcase.py index 0391ae3e..c2c64a35 100644 --- a/fail2ban/tests/banmanagertestcase.py +++ b/fail2ban/tests/banmanagertestcase.py @@ -156,7 +156,7 @@ class StatusExtendedCymruInfo(unittest.TestCase): if tc.available[0]: cymru_info = self.__banManager.getBanListExtendedCymruInfo( timeout=(2 if unittest.F2B.fast else 20)) - else: + else: # pragma: no cover - availability (once after error case only) cymru_info = tc.available[1] if cymru_info.get("error"): # pragma: no cover - availability tc.available = False, cymru_info From 857767f04b66f0b61f1b23c4ed7816439ae6920a Mon Sep 17 00:00:00 2001 From: Ben RUBSON Date: Tue, 27 Feb 2018 10:12:22 +0100 Subject: [PATCH 042/189] Add 'any' badips.py bancategory (#2056) action.d/badips.py: allow `any` as bancategory to retrieve IPs from all categories --- ChangeLog | 1 + config/action.d/badips.py | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/ChangeLog b/ChangeLog index f32f806e..f37ac992 100644 --- a/ChangeLog +++ b/ChangeLog @@ -46,6 +46,7 @@ ver. 0.10.3-dev-1 (20??/??/??) - development edition the epoch-pattern similar to `{DATE}` patterns does the capture and cuts out the match of whole pattern from the log-line, e. g. date-pattern `^\[{LEPOCH}\]\s+:` will match and cut out `[1516469849551000] :` from begin of the log-line. * badips.py now uses https instead of plain http when requesting badips.com (gh-2057); +* add support for "any" badips.py bancategory, to be able to retrieve IPs from all categories with a desired score (gh-2056); ver. 0.10.2 (2018/01/18) - nothing-burns-like-the-cold diff --git a/config/action.d/badips.py b/config/action.d/badips.py index afb58950..0df34c12 100644 --- a/config/action.d/badips.py +++ b/config/action.d/badips.py @@ -220,7 +220,7 @@ class BadIPsAction(ActionBase): # pragma: no cover - may be unavailable @bancategory.setter def bancategory(self, bancategory): - if bancategory not in self.getCategories(incParents=True): + if bancategory != "any" and bancategory not in self.getCategories(incParents=True): self._logSys.error("Category name '%s' not valid. " "see badips.com for list of valid categories", bancategory) From b112250ef0840ddeadec3e59e4cb54a8543d5b7b Mon Sep 17 00:00:00 2001 From: Ben RUBSON Date: Tue, 27 Feb 2018 10:18:59 +0100 Subject: [PATCH 043/189] (Free)BSD IPFW does not allow 2 identical rules (#2054) ipfw actionban fixed to allow same rule added several times (and actionunban to ignore error by deletion of missing rule) --- ChangeLog | 1 + config/action.d/bsd-ipfw.conf | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/ChangeLog b/ChangeLog index f37ac992..e410966a 100644 --- a/ChangeLog +++ b/ChangeLog @@ -37,6 +37,7 @@ ver. 0.10.3-dev-1 (20??/??/??) - development edition ### Fixes * `filter.d/exim.conf`: failregex extended - SMTP call dropped: too many syntax or protocol errors (gh-2048); * `action.d/badips.py`: implicit convert IPAddr to str, solves an issue "expected string, IPAddr found" (gh-2059); +* (Free)BSD ipfw actionban fixed to allow same rule added several times (gh-2054); ### New Features diff --git a/config/action.d/bsd-ipfw.conf b/config/action.d/bsd-ipfw.conf index cbd6a15d..4fbe9195 100644 --- a/config/action.d/bsd-ipfw.conf +++ b/config/action.d/bsd-ipfw.conf @@ -38,7 +38,7 @@ actioncheck = # Values: CMD # # requires an ipfw rule like "deny ip from table(1) to me" -actionban = e=`ipfw table add 2>&1`; x=$?; [ $x -eq 0 -o "$e" = 'ipfw: setsockopt(IP_FW_TABLE_XADD): File exists' ] || { echo "$e" 1>&2; exit $x; } +actionban = e=`ipfw table
add 2>&1`; x=$?; [ $x -eq 0 -o "$e" = 'ipfw: setsockopt(IP_FW_TABLE_XADD): File exists' ] || echo "$e" | grep -q "record already exists" || { echo "$e" 1>&2; exit $x; } # Option: actionunban @@ -47,7 +47,7 @@ actionban = e=`ipfw table
add 2>&1`; x=$?; [ $x -eq 0 -o "$e" = 'ip # Tags: See jail.conf(5) man page # Values: CMD # -actionunban = e=`ipfw table
delete 2>&1`; x=$?; [ $x -eq 0 -o "$e" = 'ipfw: setsockopt(IP_FW_TABLE_XDEL): No such process' ] || { echo "$e" 1>&2; exit $x; } +actionunban = e=`ipfw table
delete 2>&1`; x=$?; [ $x -eq 0 -o "$e" = 'ipfw: setsockopt(IP_FW_TABLE_XDEL): No such process' ] || echo "$e" | grep -q "record not found" || { echo "$e" 1>&2; exit $x; } [Init] # Option: table From 8c291cad384cfe62234246f4a4eebf6bfcec2594 Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 2 Mar 2018 09:15:34 +0100 Subject: [PATCH 044/189] filter.d/asterisk.conf: fixed failregex prefix by log over remote syslog server (gh-2060) --- ChangeLog | 1 + config/filter.d/asterisk.conf | 2 +- fail2ban/tests/files/logs/asterisk | 3 +++ 3 files changed, 5 insertions(+), 1 deletion(-) diff --git a/ChangeLog b/ChangeLog index e410966a..382ffc4c 100644 --- a/ChangeLog +++ b/ChangeLog @@ -35,6 +35,7 @@ ver. 0.10.3-dev-1 (20??/??/??) - development edition ----------- ### Fixes +* `filter.d/asterisk.conf`: fixed failregex prefix by log over remote syslog server (gh-2060); * `filter.d/exim.conf`: failregex extended - SMTP call dropped: too many syntax or protocol errors (gh-2048); * `action.d/badips.py`: implicit convert IPAddr to str, solves an issue "expected string, IPAddr found" (gh-2059); * (Free)BSD ipfw actionban fixed to allow same rule added several times (gh-2054); diff --git a/config/filter.d/asterisk.conf b/config/filter.d/asterisk.conf index 337e9573..6f7ae5d5 100644 --- a/config/filter.d/asterisk.conf +++ b/config/filter.d/asterisk.conf @@ -16,7 +16,7 @@ __pid_re = (?:\s*\[\d+\]) iso8601 = \d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+[+-]\d{4} # All Asterisk log messages begin like this: -log_prefix= (?:NOTICE|SECURITY|WARNING)%(__pid_re)s:?(?:\[C-[\da-f]*\])? [^:]+:\d*(?:(?: in)? \w+:)? +log_prefix= (?:NOTICE|SECURITY|WARNING)%(__pid_re)s:?(?:\[C-[\da-f]*\])?:? [^:]+:\d*(?:(?: in)? [^:]+:)? prefregex = ^%(__prefix_line)s%(log_prefix)s .+$ diff --git a/fail2ban/tests/files/logs/asterisk b/fail2ban/tests/files/logs/asterisk index 0955cfe7..7bd011fc 100644 --- a/fail2ban/tests/files/logs/asterisk +++ b/fail2ban/tests/files/logs/asterisk @@ -106,3 +106,6 @@ Nov 4 18:30:40 localhost asterisk[32229]: NOTICE[32257]: chan_sip.c:23417 in han # #_dis_failJSON: { "time": "2016-05-06T07:08:09", "match": true, "host": "192.0.2.6" } # [2016-05-06 07:08:09] WARNING[6410][C-00000bac] Ext. +012345: Friendly Scanner from 192.0.2.6 # # Yes, this does have quotes around it. + +# failJSON: { "time": "2005-03-01T15:35:53", "match": true , "host": "192.0.2.2", "desc": "log over remote syslog server" } +Mar 1 15:35:53 pbx asterisk[2350]: WARNING[1195][C-00000b43]: Ext. s:6 in @ from-sip-external: "Rejecting unknown SIP connection from 192.0.2.2" From fa520f36c32100ef48c8aba48601997b77eeac47 Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 2 Mar 2018 17:00:01 +0100 Subject: [PATCH 045/189] stability test-cases fix: avoid rare sporadic error on start of server (threaded in foreground); additionally show the log output of the thread-server in case of any error there. --- fail2ban/tests/clientreadertestcase.py | 6 ++-- fail2ban/tests/fail2banclienttestcase.py | 39 +++++++++++++++++++++--- fail2ban/tests/utils.py | 3 -- 3 files changed, 37 insertions(+), 11 deletions(-) diff --git a/fail2ban/tests/clientreadertestcase.py b/fail2ban/tests/clientreadertestcase.py index 9e5f0dfe..184595ab 100644 --- a/fail2ban/tests/clientreadertestcase.py +++ b/fail2ban/tests/clientreadertestcase.py @@ -266,7 +266,7 @@ class JailReaderTest(LogCaptureTestCase): self.assertEqual(('mail--ho_is', {}), extractOptions("mail--ho_is")) self.assertEqual(('mail--ho_is', {}), extractOptions("mail--ho_is['s']")) - #self.printLog() + #print(self.getLog()) #self.assertLogged("Invalid argument ['s'] in ''s''") self.assertEqual(('mail', {'a': ','}), extractOptions("mail[a=',']")) @@ -528,7 +528,7 @@ class JailsReaderTestCache(LogCaptureTestCase): # how many times jail.local was read: cnt = self._getLoggedReadCount('jail.local') # if cnt > 1: - # self.printLog() + # print(self.getLog()) self.assertTrue(cnt == 1, "Unexpected count by reading of jail files, cnt = %s" % cnt) # read whole configuration like a file2ban-client, again ... @@ -648,7 +648,7 @@ class JailsReaderTest(LogCaptureTestCase): ## We should not "read" some bogus jail #old_comm_commands = comm_commands[:] # make a copy #self.assertRaises(ValueError, jails.getOptions, "BOGUS") - #self.printLog() + #print(self.getLog()) #self.assertLogged("No section: 'BOGUS'") ## and there should be no side-effects #self.assertEqual(jails.convert(), old_comm_commands) diff --git a/fail2ban/tests/fail2banclienttestcase.py b/fail2ban/tests/fail2banclienttestcase.py index 318a1c50..ed3d9868 100644 --- a/fail2ban/tests/fail2banclienttestcase.py +++ b/fail2ban/tests/fail2banclienttestcase.py @@ -210,6 +210,12 @@ def _start_params(tmp, use_stock=False, use_stock_cfg=None, "--timeout", str(fail2bancmdline.MAX_WAITTIME), ) +def _inherited_log(startparams): + try: + return startparams[startparams.index('--logtarget')+1] == 'INHERITED' + except ValueError: + return False + def _get_pid_from_file(pidfile): pid = None try: @@ -325,6 +331,13 @@ def with_foreground_server_thread(startextra={}): self.pruneLog() # several commands to server in body of decorated function: return f(self, tmp, startparams, *args, **kwargs) + except Exception as e: # pragma: no cover + print('=== Catch an exception: %s' % e) + log = self.getLog() + if log: + print('=== Error of server, log: ===\n%s===' % log) + self.pruneLog() + raise finally: if th: # wait for server end (if not yet already exited): @@ -369,7 +382,8 @@ class Fail2banClientServerBase(LogCaptureTestCase): else: raise FailExitException() - def _wait_for_srv(self, tmp, ready=True, startparams=None, phase={}): + def _wait_for_srv(self, tmp, ready=True, startparams=None, phase=None): + if not phase: phase = {} try: sock = pjoin(tmp, "f2b.sock") # wait for server (socket): @@ -384,14 +398,17 @@ class Fail2banClientServerBase(LogCaptureTestCase): ret = Utils.wait_for(lambda: "Server ready" in self.getLog(), MAX_WAITTIME) if not ret: # pragma: no cover - test-failure case only raise Exception( - 'Unexpected: Server ready was not found.\nStart failed: %r' - % (startparams,) + 'Unexpected: Server ready was not found, phase %r.\nStart failed: %r' + % (phase, startparams,) ) except: # pragma: no cover + if _inherited_log(startparams): + print('=== Error by wait fot server, log: ===\n%s===' % self.getLog()) + self.pruneLog() log = pjoin(tmp, "f2b.log") if isfile(log): _out_file(log) - else: + elif not _inherited_log(startparams): logSys.debug("No log file %s to examine details of error", log) raise @@ -410,6 +427,7 @@ class Fail2banClientServerBase(LogCaptureTestCase): self.execCmd(SUCCESS, ("-f",) + startparams, "start") finally: # end : + phase['start'] = False phase['end'] = True logSys.debug("end of test worker") @@ -1192,7 +1210,7 @@ class Fail2banServerTest(Fail2banClientServerBase): 'failregex = ^ failure "[^"]+" - ', 'maxretry = 1', # ban by first failure 'enabled = true', - ) + ) }) def testServerActions_NginxBlockMap(self, tmp, startparams): cfg = pjoin(tmp, "config") @@ -1251,3 +1269,14 @@ class Fail2banServerTest(Fail2banClientServerBase): _out_file(mpfn) mp = _read_file(mpfn) self.assertEqual(mp, '') + + # test multiple start/stop of the server (threaded in foreground) -- + if False: # pragma: no cover + @with_foreground_server_thread() + def _testServerStartStop(self, tmp, startparams): + # stop server and wait for end: + self.stopAndWaitForServerEnd(SUCCESS) + + def testServerStartStop(self): + for i in xrange(2000): + self._testServerStartStop() diff --git a/fail2ban/tests/utils.py b/fail2ban/tests/utils.py index f681db76..3eeb8eb4 100644 --- a/fail2ban/tests/utils.py +++ b/fail2ban/tests/utils.py @@ -807,8 +807,5 @@ class LogCaptureTestCase(unittest.TestCase): def getLog(self): return self._log.getvalue() - def printLog(self): - print(self._log.getvalue()) - pid_exists = Utils.pid_exists From 5f021aa648c42ee188d8c31a81937d764982a58a Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 2 Mar 2018 17:08:23 +0100 Subject: [PATCH 046/189] shutdown sockets before close, avoid socket leakage by use of the explicit socket close in async_chat; better error handling with error counting, differentiate special case ([Errno 24] Too many open files), with resulting stop of the server (avoid flood the log file, closes gh-991 and similar issues); restored auto-garbage, because of non-reference-counting python's (like pypy), otherwise it may leak there on objects like unix-socket, etc. --- fail2ban/client/csocket.py | 1 + fail2ban/helpers.py | 4 +- fail2ban/server/asyncserver.py | 75 +++++++++++++++++++++++++++------- 3 files changed, 64 insertions(+), 16 deletions(-) diff --git a/fail2ban/client/csocket.py b/fail2ban/client/csocket.py index 6b478460..86dd17c9 100644 --- a/fail2ban/client/csocket.py +++ b/fail2ban/client/csocket.py @@ -61,6 +61,7 @@ class CSocket: return if sendEnd: self.__csock.sendall(CSPROTO.CLOSE + CSPROTO.END) + self.__csock.shutdown(socket.SHUT_RDWR) self.__csock.close() self.__csock = None diff --git a/fail2ban/helpers.py b/fail2ban/helpers.py index 98d59fa1..6a3ed2fd 100644 --- a/fail2ban/helpers.py +++ b/fail2ban/helpers.py @@ -393,7 +393,9 @@ class BgService(object): self.__count = self.__threshold; if hasattr(gc, 'set_threshold'): gc.set_threshold(0) - gc.disable() + # don't disable auto garbage, because of non-reference-counting python's (like pypy), + # otherwise it may leak there on objects like unix-socket, etc. + #gc.disable() def service(self, force=False, wait=False): self.__count -= 1 diff --git a/fail2ban/server/asyncserver.py b/fail2ban/server/asyncserver.py index e254979d..eb99c69a 100644 --- a/fail2ban/server/asyncserver.py +++ b/fail2ban/server/asyncserver.py @@ -42,21 +42,36 @@ from ..helpers import logging, getLogger, formatExceptionInfo # Gets the instance of the logger. logSys = getLogger(__name__) + ## # Request handler class. # # This class extends asynchat in order to provide a request handler for # incoming query. - class RequestHandler(asynchat.async_chat): def __init__(self, conn, transmitter): asynchat.async_chat.__init__(self, conn) + self.__conn = conn self.__transmitter = transmitter self.__buffer = [] # Sets the terminator. self.set_terminator(CSPROTO.END) + def __close(self): + if self.__conn: + conn = self.__conn + self.__conn = None + try: + conn.shutdown(socket.SHUT_RDWR) + conn.close() + except socket.error: # pragma: no cover - normally unreachable + pass + + def handle_close(self): + self.__close() + asynchat.async_chat.handle_close(self) + def collect_incoming_data(self, data): #logSys.debug("Received raw data: " + str(data)) self.__buffer.append(data) @@ -111,14 +126,15 @@ class RequestHandler(asynchat.async_chat): self.close_when_done() -def loop(active, timeout=None, use_poll=False): +def loop(active, timeout=None, use_poll=False, err_count=None): """Custom event loop implementation Uses poll instead of loop to respect `active` flag, to avoid loop timeout mistake: different in poll and poll2 (sec vs ms), and to prevent sporadic errors like EBADF 'Bad file descriptor' etc. (see gh-161) """ - errCount = 0 + if not err_count: err_count={} + err_count['listen'] = 0 if timeout is None: timeout = Utils.DEFAULT_SLEEP_TIME poll = asyncore.poll @@ -133,22 +149,29 @@ def loop(active, timeout=None, use_poll=False): while active(): try: poll(timeout) - if errCount: - errCount -= 1 + if err_count['listen']: + err_count['listen'] -= 1 except Exception as e: if not active(): break - errCount += 1 - if errCount < 20: + err_count['listen'] += 1 + if err_count['listen'] < 20: # errno.ENOTCONN - 'Socket is not connected' # errno.EBADF - 'Bad file descriptor' if e.args[0] in (errno.ENOTCONN, errno.EBADF): # pragma: no cover (too sporadic) logSys.info('Server connection was closed: %s', str(e)) else: logSys.error('Server connection was closed: %s', str(e)) - elif errCount == 20: + elif err_count['listen'] == 20: logSys.exception(e) logSys.error('Too many errors - stop logging connection errors') + elif err_count['listen'] > 100: # pragma: no cover - normally unreachable + if ( + e.args[0] == errno.EMFILE # [Errno 24] Too many open files + or sum(err_count.itervalues()) > 1000 + ): + logSys.critical("Too many errors - critical count reached %r", err_count) + break ## @@ -165,6 +188,7 @@ class AsyncServer(asyncore.dispatcher): self.__sock = "/var/run/fail2ban/fail2ban.sock" self.__init = False self.__active = False + self.__errCount = {'accept': 0, 'listen': 0} self.onstart = None ## @@ -176,12 +200,25 @@ class AsyncServer(asyncore.dispatcher): def handle_accept(self): try: conn, addr = self.accept() - except socket.error: # pragma: no cover - logSys.warning("Socket error") + except socket.error as e: # pragma: no cover + self.__errCount['accept'] += 1 + if self.__errCount['accept'] < 20: + logSys.warning("Socket error: %s", e) + elif self.__errCount['accept'] == 20: + logSys.error("Too many acceptor errors - stop logging errors") + elif self.__errCount['accept'] > 100: + if ( + e.args[0] == errno.EMFILE # [Errno 24] Too many open files + or sum(self.__errCount.itervalues()) > 1000 + ): + logSys.critical("Too many errors - critical count reached %r", err_count) + self.stop() return - except TypeError: # pragma: no cover - logSys.warning("Type error") + except TypeError as e: # pragma: no cover + logSys.warning("Type error: %s", e) return + if self.__errCount['accept']: + self.__errCount['accept'] -= 1; AsyncServer.__markCloseOnExec(conn) # Creates an instance of the handler class to handle the # request/response on the incoming connection. @@ -219,7 +256,7 @@ class AsyncServer(asyncore.dispatcher): if self.onstart: self.onstart() # Event loop as long as active: - loop(lambda: self.__loop, timeout=timeout, use_poll=use_poll) + loop(lambda: self.__loop, timeout=timeout, use_poll=use_poll, err_count=self.__errCount) self.__active = False # Cleanup all self.stop() @@ -246,13 +283,21 @@ class AsyncServer(asyncore.dispatcher): # Stops the communication server. def stop_communication(self): - logSys.debug("Stop communication") - self.__transmitter = None + if self.__transmitter: + logSys.debug("Stop communication") + self.__transmitter = None + # shutdown socket here: + if self.socket: + try: + self.socket.shutdown(socket.SHUT_RDWR) + except socket.error: # pragma: no cover - normally unreachable + pass ## # Stops the server. def stop(self): + self.stop_communication() self.close() # better remains a method (not a property) since used as a callable for wait_for From 9f5c87352651ed5278d4c27801dc9e70b1166260 Mon Sep 17 00:00:00 2001 From: MatthieuBarbu <30758776+MatthieuBarbu@users.noreply.github.com> Date: Fri, 2 Mar 2018 14:47:16 +0100 Subject: [PATCH 047/189] fix sshd rule just remove the space before ":11" line 52 because don't match on my Debian 9 stretch... I don't know if this is wrong on all OS --- config/filter.d/sshd.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/filter.d/sshd.conf b/config/filter.d/sshd.conf index ab5fd385..71dbee8b 100644 --- a/config/filter.d/sshd.conf +++ b/config/filter.d/sshd.conf @@ -49,7 +49,7 @@ cmnfailre = ^[aA]uthentication (?:failure|error|failed) for .* ^(error: )?maximum authentication attempts exceeded for .* from %(__on_port_opt)s(?: ssh\d*)?%(__suff)s$ ^User .+ not allowed because account is locked%(__suff)s ^Disconnecting: Too many authentication failures(?: for .+?)?%(__suff)s - ^Received disconnect from : 11: + ^Received disconnect from :11: ^Connection closed by %(__suff)s$ ^Accepted publickey for \S+ from (?:\s|$) From 1d7aa2ff217737f5958c95f29fca96389020dd19 Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 2 Mar 2018 17:59:25 +0100 Subject: [PATCH 048/189] filter.d/sshd.conf: rewrite fix (for new ssh log-format) backwards compatible + test-cases extended to cover both cases --- ChangeLog | 1 + config/filter.d/sshd.conf | 2 +- fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf | 2 +- fail2ban/tests/files/logs/sshd | 2 +- 4 files changed, 4 insertions(+), 3 deletions(-) diff --git a/ChangeLog b/ChangeLog index 382ffc4c..4b3de2b7 100644 --- a/ChangeLog +++ b/ChangeLog @@ -37,6 +37,7 @@ ver. 0.10.3-dev-1 (20??/??/??) - development edition ### Fixes * `filter.d/asterisk.conf`: fixed failregex prefix by log over remote syslog server (gh-2060); * `filter.d/exim.conf`: failregex extended - SMTP call dropped: too many syntax or protocol errors (gh-2048); +* `filter.d/sshd.conf`: failregex got an optional space in order to match new log-format (see gh-2061); * `action.d/badips.py`: implicit convert IPAddr to str, solves an issue "expected string, IPAddr found" (gh-2059); * (Free)BSD ipfw actionban fixed to allow same rule added several times (gh-2054); diff --git a/config/filter.d/sshd.conf b/config/filter.d/sshd.conf index 71dbee8b..328ba558 100644 --- a/config/filter.d/sshd.conf +++ b/config/filter.d/sshd.conf @@ -49,7 +49,7 @@ cmnfailre = ^[aA]uthentication (?:failure|error|failed) for .* ^(error: )?maximum authentication attempts exceeded for .* from %(__on_port_opt)s(?: ssh\d*)?%(__suff)s$ ^User .+ not allowed because account is locked%(__suff)s ^Disconnecting: Too many authentication failures(?: for .+?)?%(__suff)s - ^Received disconnect from :11: + ^Received disconnect from :\s*11: ^Connection closed by %(__suff)s$ ^Accepted publickey for \S+ from (?:\s|$) diff --git a/fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf b/fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf index f193009f..a1c55cd3 100644 --- a/fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf +++ b/fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf @@ -44,7 +44,7 @@ cmnfailre = ^%(__prefix_line_sl)s[aA]uthentication (?:failure|error|failed) for ^%(__prefix_line_sl)sUser .+ from not allowed because none of user's groups are listed in AllowGroups\s*%(__suff)s$ ^%(__prefix_line_sl)spam_unix\(sshd:auth\):\s+authentication failure;\s*logname=\S*\s*uid=\d*\s*euid=\d*\s*tty=\S*\s*ruser=\S*\s*rhost=\s.*%(__suff)s$ ^%(__prefix_line_sl)s(error: )?maximum authentication attempts exceeded for .* from %(__on_port_opt)s(?: ssh\d*)? \[preauth\]$ - ^%(__prefix_line_ml1)sUser .+ not allowed because account is locked%(__prefix_line_ml2)sReceived disconnect from : 11: .+%(__suff)s$ + ^%(__prefix_line_ml1)sUser .+ not allowed because account is locked%(__prefix_line_ml2)sReceived disconnect from :\s*11: .+%(__suff)s$ ^%(__prefix_line_ml1)sDisconnecting: Too many authentication failures(?: for .+?)?%(__suff)s%(__prefix_line_ml2)sConnection closed by %(__suff)s$ ^%(__prefix_line_ml1)sConnection from %(__on_port_opt)s%(__prefix_line_ml2)sDisconnecting: Too many authentication failures(?: for .+?)?%(__suff)s$ diff --git a/fail2ban/tests/files/logs/sshd b/fail2ban/tests/files/logs/sshd index b0b760ff..5c7f5cfd 100644 --- a/fail2ban/tests/files/logs/sshd +++ b/fail2ban/tests/files/logs/sshd @@ -110,7 +110,7 @@ May 27 00:16:33 host sshd[2364]: User root not allowed because account is locked # failJSON: { "match": false } May 27 00:16:33 host sshd[2364]: input_userauth_request: invalid user root [preauth] # failJSON: { "time": "2005-05-27T00:16:33", "match": true , "host": "198.51.100.76" } -May 27 00:16:33 host sshd[2364]: Received disconnect from 198.51.100.76: 11: Bye Bye [preauth] +May 27 00:16:33 host sshd[2364]: Received disconnect from 198.51.100.76:11: Bye Bye [preauth] # failJSON: { "time": "2004-09-29T16:28:02", "match": true , "host": "127.0.0.1" } Sep 29 16:28:02 spaceman sshd[16699]: Failed password for dan from 127.0.0.1 port 45416 ssh1 From 6b5516b8512e96952a38749c215e9938bf9bf9f0 Mon Sep 17 00:00:00 2001 From: MatthieuBarbu <30758776+MatthieuBarbu@users.noreply.github.com> Date: Fri, 2 Mar 2018 18:27:06 +0100 Subject: [PATCH 049/189] fix sshd rule #2 in line 58, rule don't match with "%(__suff)s" but work fine if I replace with "%(__on_port_opt)s" Debian 9 stretch : fail2ban 0.10.3 --- config/filter.d/sshd.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/filter.d/sshd.conf b/config/filter.d/sshd.conf index 328ba558..41b76481 100644 --- a/config/filter.d/sshd.conf +++ b/config/filter.d/sshd.conf @@ -55,7 +55,7 @@ cmnfailre = ^[aA]uthentication (?:failure|error|failed) for .* mdre-normal = -mdre-ddos = ^Did not receive identification string from %(__suff)s$ +mdre-ddos = ^Did not receive identification string from %(__on_port_opt)s ^Connection reset by %(__on_port_opt)s%(__suff)s ^SSH: Server;Ltype: (?:Authname|Version|Kex);Remote: -\d+;[A-Z]\w+: ^Read from socket failed: Connection reset by peer%(__suff)s From a3bcbe2d1b7ebe3897894ddc2c400f48806d0a66 Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 2 Mar 2018 19:12:09 +0100 Subject: [PATCH 050/189] backwards-compatibility, test-cases and ChangeLog update --- ChangeLog | 4 +++- config/filter.d/sshd.conf | 2 +- .../tests/config/filter.d/zzz-sshd-obsolete-multiline.conf | 2 +- fail2ban/tests/files/logs/sshd | 2 ++ 4 files changed, 7 insertions(+), 3 deletions(-) diff --git a/ChangeLog b/ChangeLog index 4b3de2b7..abbd254b 100644 --- a/ChangeLog +++ b/ChangeLog @@ -37,7 +37,9 @@ ver. 0.10.3-dev-1 (20??/??/??) - development edition ### Fixes * `filter.d/asterisk.conf`: fixed failregex prefix by log over remote syslog server (gh-2060); * `filter.d/exim.conf`: failregex extended - SMTP call dropped: too many syntax or protocol errors (gh-2048); -* `filter.d/sshd.conf`: failregex got an optional space in order to match new log-format (see gh-2061); +* `filter.d/sshd.conf`: + - failregex got an optional space in order to match new log-format (see gh-2061); + - fixed ddos-mode regex to match refactored message (some versions can contain port now, see gh-2062); * `action.d/badips.py`: implicit convert IPAddr to str, solves an issue "expected string, IPAddr found" (gh-2059); * (Free)BSD ipfw actionban fixed to allow same rule added several times (gh-2054); diff --git a/config/filter.d/sshd.conf b/config/filter.d/sshd.conf index 41b76481..f557223c 100644 --- a/config/filter.d/sshd.conf +++ b/config/filter.d/sshd.conf @@ -55,7 +55,7 @@ cmnfailre = ^[aA]uthentication (?:failure|error|failed) for .* mdre-normal = -mdre-ddos = ^Did not receive identification string from %(__on_port_opt)s +mdre-ddos = ^Did not receive identification string from %(__on_port_opt)s%(__suff)s ^Connection reset by %(__on_port_opt)s%(__suff)s ^SSH: Server;Ltype: (?:Authname|Version|Kex);Remote: -\d+;[A-Z]\w+: ^Read from socket failed: Connection reset by peer%(__suff)s diff --git a/fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf b/fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf index a1c55cd3..7000e627 100644 --- a/fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf +++ b/fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf @@ -50,7 +50,7 @@ cmnfailre = ^%(__prefix_line_sl)s[aA]uthentication (?:failure|error|failed) for mdre-normal = -mdre-ddos = ^%(__prefix_line_sl)sDid not receive identification string from %(__suff)s$ +mdre-ddos = ^%(__prefix_line_sl)sDid not receive identification string from %(__on_port_opt)s%(__suff)s ^%(__prefix_line_sl)sConnection reset by %(__on_port_opt)s%(__suff)s ^%(__prefix_line_ml1)sSSH: Server;Ltype: (?:Authname|Version|Kex);Remote: -\d+;[A-Z]\w+:.*%(__prefix_line_ml2)sRead from socket failed: Connection reset by peer%(__suff)s$ diff --git a/fail2ban/tests/files/logs/sshd b/fail2ban/tests/files/logs/sshd index 5c7f5cfd..524d2d28 100644 --- a/fail2ban/tests/files/logs/sshd +++ b/fail2ban/tests/files/logs/sshd @@ -219,6 +219,8 @@ Apr 27 13:02:04 host sshd[29116]: Received disconnect from 1.2.3.4: 11: Normal S # http://forums.powervps.com/showthread.php?t=1667 # failJSON: { "time": "2005-06-07T01:10:56", "match": true , "host": "69.61.56.114" } Jun 7 01:10:56 host sshd[5937]: Did not receive identification string from 69.61.56.114 +# failJSON: { "time": "2005-06-07T01:11:57", "match": true , "host": "192.0.2.5", "desc": "refactored message (with port now, gh-2062)" } +Jun 7 01:11:57 host sshd[8782]: Did not receive identification string from 192.0.2.5 port 35836 # gh-864(1): # failJSON: { "match": false } From caa2bdfee6afbc78bb42f61633760079bbe472ff Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 2 Mar 2018 19:24:47 +0100 Subject: [PATCH 051/189] amendment for gh-2061: it looks like the port was added here also --- config/filter.d/sshd.conf | 2 +- fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf | 2 +- fail2ban/tests/files/logs/sshd | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/config/filter.d/sshd.conf b/config/filter.d/sshd.conf index f557223c..d8bb5edf 100644 --- a/config/filter.d/sshd.conf +++ b/config/filter.d/sshd.conf @@ -49,7 +49,7 @@ cmnfailre = ^[aA]uthentication (?:failure|error|failed) for .* ^(error: )?maximum authentication attempts exceeded for .* from %(__on_port_opt)s(?: ssh\d*)?%(__suff)s$ ^User .+ not allowed because account is locked%(__suff)s ^Disconnecting: Too many authentication failures(?: for .+?)?%(__suff)s - ^Received disconnect from :\s*11: + ^Received disconnect from %(__on_port_opt)s:\s*11: ^Connection closed by %(__suff)s$ ^Accepted publickey for \S+ from (?:\s|$) diff --git a/fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf b/fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf index 7000e627..5560716d 100644 --- a/fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf +++ b/fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf @@ -44,7 +44,7 @@ cmnfailre = ^%(__prefix_line_sl)s[aA]uthentication (?:failure|error|failed) for ^%(__prefix_line_sl)sUser .+ from not allowed because none of user's groups are listed in AllowGroups\s*%(__suff)s$ ^%(__prefix_line_sl)spam_unix\(sshd:auth\):\s+authentication failure;\s*logname=\S*\s*uid=\d*\s*euid=\d*\s*tty=\S*\s*ruser=\S*\s*rhost=\s.*%(__suff)s$ ^%(__prefix_line_sl)s(error: )?maximum authentication attempts exceeded for .* from %(__on_port_opt)s(?: ssh\d*)? \[preauth\]$ - ^%(__prefix_line_ml1)sUser .+ not allowed because account is locked%(__prefix_line_ml2)sReceived disconnect from :\s*11: .+%(__suff)s$ + ^%(__prefix_line_ml1)sUser .+ not allowed because account is locked%(__prefix_line_ml2)sReceived disconnect from %(__on_port_opt)s:\s*11: .+%(__suff)s$ ^%(__prefix_line_ml1)sDisconnecting: Too many authentication failures(?: for .+?)?%(__suff)s%(__prefix_line_ml2)sConnection closed by %(__suff)s$ ^%(__prefix_line_ml1)sConnection from %(__on_port_opt)s%(__prefix_line_ml2)sDisconnecting: Too many authentication failures(?: for .+?)?%(__suff)s$ diff --git a/fail2ban/tests/files/logs/sshd b/fail2ban/tests/files/logs/sshd index 524d2d28..e80eb30c 100644 --- a/fail2ban/tests/files/logs/sshd +++ b/fail2ban/tests/files/logs/sshd @@ -110,7 +110,7 @@ May 27 00:16:33 host sshd[2364]: User root not allowed because account is locked # failJSON: { "match": false } May 27 00:16:33 host sshd[2364]: input_userauth_request: invalid user root [preauth] # failJSON: { "time": "2005-05-27T00:16:33", "match": true , "host": "198.51.100.76" } -May 27 00:16:33 host sshd[2364]: Received disconnect from 198.51.100.76:11: Bye Bye [preauth] +May 27 00:16:33 host sshd[2364]: Received disconnect from 198.51.100.76 port 58846:11: Bye Bye [preauth] # failJSON: { "time": "2004-09-29T16:28:02", "match": true , "host": "127.0.0.1" } Sep 29 16:28:02 spaceman sshd[16699]: Failed password for dan from 127.0.0.1 port 45416 ssh1 From 29bedd70d533a6186e00fed9a4072785e1062c1d Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 2 Mar 2018 20:56:53 +0100 Subject: [PATCH 052/189] socket stability and coverage: cherry picked from 0.11 version (avoid many sporadic unhandled exceptions) --- fail2ban/client/csocket.py | 12 +-- fail2ban/tests/sockettestcase.py | 139 ++++++++++++++++++++++++++----- 2 files changed, 123 insertions(+), 28 deletions(-) diff --git a/fail2ban/client/csocket.py b/fail2ban/client/csocket.py index 86dd17c9..ce01ae08 100644 --- a/fail2ban/client/csocket.py +++ b/fail2ban/client/csocket.py @@ -45,13 +45,13 @@ class CSocket: def __del__(self): self.close(False) - def send(self, msg): + def send(self, msg, nonblocking=False, timeout=None): # Convert every list member to string obj = dumps(map( lambda m: str(m) if not isinstance(m, (list, dict, set)) else m, msg), HIGHEST_PROTOCOL) self.__csock.send(obj + CSPROTO.END) - return self.receive(self.__csock) + return self.receive(self.__csock, nonblocking, timeout) def settimeout(self, timeout): self.__csock.settimeout(timeout if timeout != -1 else self.__deftout) @@ -66,11 +66,13 @@ class CSocket: self.__csock = None @staticmethod - def receive(sock): + def receive(sock, nonblocking=False, timeout=None): msg = CSPROTO.EMPTY + if nonblocking: sock.setblocking(0) + if timeout: sock.settimeout(timeout) while msg.rfind(CSPROTO.END) == -1: - chunk = sock.recv(6) - if chunk == '': + chunk = sock.recv(512) + if chunk in ('', b''): # python 3.x may return b'' instead of '' raise RuntimeError("socket connection broken") msg = msg + chunk return loads(msg) diff --git a/fail2ban/tests/sockettestcase.py b/fail2ban/tests/sockettestcase.py index 4f9b9d7a..a7c3a43c 100644 --- a/fail2ban/tests/sockettestcase.py +++ b/fail2ban/tests/sockettestcase.py @@ -34,42 +34,70 @@ import unittest from .utils import LogCaptureTestCase from .. import protocol -from ..server.asyncserver import AsyncServer, AsyncServerException, loop +from ..server.asyncserver import asyncore, RequestHandler, loop, AsyncServer, AsyncServerException from ..server.utils import Utils from ..client.csocket import CSocket +from .utils import LogCaptureTestCase -class Socket(unittest.TestCase): + +def TestMsgError(*args): + raise Exception('test unpickle error') +class TestMsg(object): + def __init__(self, unpickle=(TestMsgError, ())): + self.unpickle = unpickle + def __reduce__(self): + return self.unpickle + + +class Socket(LogCaptureTestCase): def setUp(self): """Call before every test case.""" + LogCaptureTestCase.setUp(self) super(Socket, self).setUp() self.server = AsyncServer(self) sock_fd, sock_name = tempfile.mkstemp('fail2ban.sock', 'socket') os.close(sock_fd) os.remove(sock_name) self.sock_name = sock_name + self.serverThread = None def tearDown(self): """Call after every test case.""" + if self.serverThread: + self.server.stop(); # stop if not already stopped + self._stopServerThread() + LogCaptureTestCase.tearDown(self) @staticmethod def proceed(message): """Test transmitter proceed method which just returns first arg""" return message - def testStopPerCloseUnexpected(self): + def _createServerThread(self, force=False): # start in separate thread : - serverThread = threading.Thread( - target=self.server.start, args=(self.sock_name, False)) + self.serverThread = serverThread = threading.Thread( + target=self.server.start, args=(self.sock_name, force)) serverThread.daemon = True serverThread.start() self.assertTrue(Utils.wait_for(self.server.isActive, unittest.F2B.maxWaitTime(10))) + return serverThread + + def _stopServerThread(self): + serverThread = self.serverThread + # wait for end of thread : + Utils.wait_for(lambda: not serverThread.isAlive() + or serverThread.join(Utils.DEFAULT_SLEEP_TIME), unittest.F2B.maxWaitTime(10)) + self.serverThread = None + + def testStopPerCloseUnexpected(self): + # start in separate thread : + serverThread = self._createServerThread() # unexpected stop directly after start: self.server.close() # wait for end of thread : - Utils.wait_for(lambda: not serverThread.isAlive() - or serverThread.join(Utils.DEFAULT_SLEEP_INTERVAL), unittest.F2B.maxWaitTime(10)) + self._stopServerThread() self.assertFalse(serverThread.isAlive()) # clean : self.server.stop() @@ -83,30 +111,99 @@ class Socket(unittest.TestCase): return None def testSocket(self): - serverThread = threading.Thread( - target=self.server.start, args=(self.sock_name, False)) - serverThread.daemon = True - serverThread.start() - self.assertTrue(Utils.wait_for(self.server.isActive, unittest.F2B.maxWaitTime(10))) - time.sleep(Utils.DEFAULT_SLEEP_TIME) - + # start in separate thread : + serverThread = self._createServerThread() client = Utils.wait_for(self._serverSocket, 2) + testMessage = ["A", "test", "message"] self.assertEqual(client.send(testMessage), testMessage) + # test wrong message: + self.assertEqual(client.send([[TestMsg()]]), 'ERROR: test unpickle error') + self.assertLogged("Caught unhandled exception", "test unpickle error", all=True) + + # test good message again: + self.assertEqual(client.send(testMessage), testMessage) + # test close message client.close() # 2nd close does nothing client.close() + # force shutdown: + self.server.stop_communication() + # test send again (should get in shutdown message): + client = Utils.wait_for(self._serverSocket, 2) + self.assertEqual(client.send(testMessage), ['SHUTDOWN']) + self.server.stop() # wait for end of thread : - Utils.wait_for(lambda: not serverThread.isAlive() - or serverThread.join(Utils.DEFAULT_SLEEP_INTERVAL), unittest.F2B.maxWaitTime(10)) + self._stopServerThread() self.assertFalse(serverThread.isAlive()) self.assertFalse(self.server.isActive()) self.assertFalse(os.path.exists(self.sock_name)) + def testSocketConnectBroken(self): + # start in separate thread : + serverThread = self._createServerThread() + client = Utils.wait_for(self._serverSocket, 2) + # unexpected stop during message body: + testMessage = ["A", "test", "message", [protocol.CSPROTO.END]] + + org_handler = RequestHandler.found_terminator + try: + RequestHandler.found_terminator = lambda self: self.close() + self.assertRaisesRegexp(RuntimeError, r"socket connection broken", + lambda: client.send(testMessage, timeout=unittest.F2B.maxWaitTime(10))) + finally: + RequestHandler.found_terminator = org_handler + + def testStopByCommunicate(self): + # start in separate thread : + serverThread = self._createServerThread() + client = Utils.wait_for(self._serverSocket, 2) + + testMessage = ["A", "test", "message"] + self.assertEqual(client.send(testMessage), testMessage) + + org_handler = RequestHandler.found_terminator + try: + RequestHandler.found_terminator = lambda self: TestMsgError() + #self.assertRaisesRegexp(RuntimeError, r"socket connection broken", client.send, testMessage) + self.assertEqual(client.send(testMessage), 'ERROR: test unpickle error') + finally: + RequestHandler.found_terminator = org_handler + + # check errors were logged: + self.assertLogged("Unexpected communication error", "test unpickle error", all=True) + + self.server.stop() + # wait for end of thread : + self._stopServerThread() + self.assertFalse(serverThread.isAlive()) + + def testLoopErrors(self): + # replace poll handler to produce error in loop-cycle: + org_poll = asyncore.poll + err = {'cntr': 0} + def _produce_error(*args): + err['cntr'] += 1 + if err['cntr'] < 50: + raise RuntimeError('test errors in poll') + return org_poll(*args) + + try: + asyncore.poll = _produce_error + serverThread = self._createServerThread() + # wait all-cases processed: + self.assertTrue(Utils.wait_for(lambda: err['cntr'] > 50, unittest.F2B.maxWaitTime(10))) + finally: + # restore: + asyncore.poll = org_poll + # check errors were logged: + self.assertLogged("Server connection was closed: test errors in poll", + "Too many errors - stop logging connection errors", all=True) + def testSocketForce(self): open(self.sock_name, 'w').close() # Create sock file # Try to start without force @@ -114,16 +211,12 @@ class Socket(unittest.TestCase): AsyncServerException, self.server.start, self.sock_name, False) # Try again with force set - serverThread = threading.Thread( - target=self.server.start, args=(self.sock_name, True)) - serverThread.daemon = True - serverThread.start() - self.assertTrue(Utils.wait_for(self.server.isActive, unittest.F2B.maxWaitTime(10))) + serverThread = self._createServerThread(True) self.server.stop() # wait for end of thread : - Utils.wait_for(lambda: not serverThread.isAlive() - or serverThread.join(Utils.DEFAULT_SLEEP_INTERVAL), unittest.F2B.maxWaitTime(10)) + self._stopServerThread() + self.assertFalse(serverThread.isAlive()) self.assertFalse(self.server.isActive()) self.assertFalse(os.path.exists(self.sock_name)) From 96836cb199ed076353cbfe2c02c90948e4235d77 Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 2 Mar 2018 21:30:03 +0100 Subject: [PATCH 053/189] fix several errors (shutdown in test-cases during stop communication, better error handling by unpickle/deserialization, etc) --- fail2ban/server/asyncserver.py | 43 +++++++++++++++++++------------- fail2ban/tests/sockettestcase.py | 5 ++-- 2 files changed, 28 insertions(+), 20 deletions(-) diff --git a/fail2ban/server/asyncserver.py b/fail2ban/server/asyncserver.py index eb99c69a..e3400737 100644 --- a/fail2ban/server/asyncserver.py +++ b/fail2ban/server/asyncserver.py @@ -76,6 +76,10 @@ class RequestHandler(asynchat.async_chat): #logSys.debug("Received raw data: " + str(data)) self.__buffer.append(data) + # exception identifies deserialization errors (exception by load in pickle): + class LoadError(Exception): + pass + ## # Handles a new request. # @@ -93,7 +97,12 @@ class RequestHandler(asynchat.async_chat): self.close_when_done() return # Deserialize - message = loads(message) + try: + message = loads(message) + except Exception as e: + logSys.error('PROTO-error: load message failed: %s', e, + exc_info=logSys.getEffectiveLevel() 100: if ( - e.args[0] == errno.EMFILE # [Errno 24] Too many open files + (isinstance(e, socket.error) and e.args[0] == errno.EMFILE) # [Errno 24] Too many open files or sum(self.__errCount.itervalues()) > 1000 ): - logSys.critical("Too many errors - critical count reached %r", err_count) + logSys.critical("Too many errors - critical count reached %r", self.__errCount) self.stop() return - except TypeError as e: # pragma: no cover - logSys.warning("Type error: %s", e) - return if self.__errCount['accept']: self.__errCount['accept'] -= 1; AsyncServer.__markCloseOnExec(conn) @@ -265,6 +273,13 @@ class AsyncServer(asyncore.dispatcher): stopflg = False if self.__active: self.__loop = False + # shutdown socket here: + if self.socket: + try: + self.socket.shutdown(socket.SHUT_RDWR) + except socket.error: # pragma: no cover - normally unreachable + pass + # close connection: asyncore.dispatcher.close(self) # If not the loop thread (stops self in handler), wait (a little bit) # for the server leaves loop, before remove socket @@ -284,14 +299,8 @@ class AsyncServer(asyncore.dispatcher): def stop_communication(self): if self.__transmitter: - logSys.debug("Stop communication") + logSys.debug("Stop communication, shutdown") self.__transmitter = None - # shutdown socket here: - if self.socket: - try: - self.socket.shutdown(socket.SHUT_RDWR) - except socket.error: # pragma: no cover - normally unreachable - pass ## # Stops the server. diff --git a/fail2ban/tests/sockettestcase.py b/fail2ban/tests/sockettestcase.py index a7c3a43c..4e14ece5 100644 --- a/fail2ban/tests/sockettestcase.py +++ b/fail2ban/tests/sockettestcase.py @@ -54,10 +54,9 @@ class Socket(LogCaptureTestCase): def setUp(self): """Call before every test case.""" - LogCaptureTestCase.setUp(self) super(Socket, self).setUp() self.server = AsyncServer(self) - sock_fd, sock_name = tempfile.mkstemp('fail2ban.sock', 'socket') + sock_fd, sock_name = tempfile.mkstemp('fail2ban.sock', 'f2b-socket') os.close(sock_fd) os.remove(sock_name) self.sock_name = sock_name @@ -120,7 +119,7 @@ class Socket(LogCaptureTestCase): # test wrong message: self.assertEqual(client.send([[TestMsg()]]), 'ERROR: test unpickle error') - self.assertLogged("Caught unhandled exception", "test unpickle error", all=True) + self.assertLogged("PROTO-error: load message failed:", "test unpickle error", all=True) # test good message again: self.assertEqual(client.send(testMessage), testMessage) From 1bdda6c8eb497990b0c000947d6cdd0e3aa4e5cb Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 2 Mar 2018 20:08:48 +0100 Subject: [PATCH 054/189] cache coverage --- fail2ban/server/utils.py | 2 +- fail2ban/tests/filtertestcase.py | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/fail2ban/server/utils.py b/fail2ban/server/utils.py index 58363ff0..8569a3f2 100644 --- a/fail2ban/server/utils.py +++ b/fail2ban/server/utils.py @@ -102,7 +102,7 @@ class Utils(): def unset(self, k): try: del self._cache[k] - except KeyError: # pragme: no cover + except KeyError: pass diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index b5877b7f..2bbfcd9d 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -1640,6 +1640,8 @@ class DNSUtilsTests(unittest.TestCase): c.set(i, i) for i in xrange(5): self.assertEqual(c.get(i), i) + # remove unavailable key: + c.unset('a'); c.unset('a') def testCacheMaxSize(self): c = Utils.Cache(maxCount=5, maxTime=60) From b34ae5999e0d8ee1af8939527305c13152844b3d Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Mon, 5 Mar 2018 19:35:10 +0100 Subject: [PATCH 055/189] action.d/hostdeny.conf: fixes IPv6 syntax differentiate the IPv4 and IPv6 syntax (where it is enclosed in square brackets) --- config/action.d/hostsdeny.conf | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/config/action.d/hostsdeny.conf b/config/action.d/hostsdeny.conf index 5cca6529..4277fed8 100644 --- a/config/action.d/hostsdeny.conf +++ b/config/action.d/hostsdeny.conf @@ -31,7 +31,7 @@ actioncheck = # Tags: See jail.conf(5) man page # Values: CMD # -actionban = IP= && printf %%b ": $IP\n" >> +actionban = printf %%b ": <_ip_value>\n" >> # Option: actionunban # Notes.: command executed when unbanning an IP. Take care that the @@ -39,7 +39,7 @@ actionban = IP= && printf %%b ": $IP\n" >> # Tags: See jail.conf(5) man page # Values: CMD # -actionunban = IP=$(echo | sed 's/\./\\./g') && sed -i "/^: $IP$/d" +actionunban = IP=$(echo "<_ip_value>" | sed 's/[][\.]/\\\0/g') && sed -i "/^: $IP$/d" [Init] @@ -54,3 +54,9 @@ file = /etc/hosts.deny # for hosts.deny/hosts_access. Default is all services. # Values: STR Default: ALL daemon_list = ALL + +# internal variable IP (to differentiate the IPv4 and IPv6 syntax, where it is enclosed in brackets): +_ip_value = + +[Init?family=inet6] +_ip_value = [] From b16aafe2334e7897c2048a9c48205340af9e80c9 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Mon, 5 Mar 2018 19:42:05 +0100 Subject: [PATCH 056/189] Update ChangeLog --- ChangeLog | 1 + 1 file changed, 1 insertion(+) diff --git a/ChangeLog b/ChangeLog index abbd254b..82fb8668 100644 --- a/ChangeLog +++ b/ChangeLog @@ -41,6 +41,7 @@ ver. 0.10.3-dev-1 (20??/??/??) - development edition - failregex got an optional space in order to match new log-format (see gh-2061); - fixed ddos-mode regex to match refactored message (some versions can contain port now, see gh-2062); * `action.d/badips.py`: implicit convert IPAddr to str, solves an issue "expected string, IPAddr found" (gh-2059); +* `action.d/hostsdeny.conf`: fixed IPv6 syntax (enclosed in square brackets, gh-2066); * (Free)BSD ipfw actionban fixed to allow same rule added several times (gh-2054); ### New Features From 5b63ad17c674084a9dd796065307bdda13327625 Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 5 Mar 2018 21:54:18 +0100 Subject: [PATCH 057/189] stability of the test-cases: avoid echoing of server-ready in configure thread, if heavy-debug (only answer from new internal command "server-status"). --- fail2ban/client/fail2banclient.py | 6 +++--- fail2ban/server/transmitter.py | 3 +++ 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/fail2ban/client/fail2banclient.py b/fail2ban/client/fail2banclient.py index 13ebcdef..f80193ee 100755 --- a/fail2ban/client/fail2banclient.py +++ b/fail2ban/client/fail2banclient.py @@ -99,7 +99,7 @@ class Fail2banClient(Fail2banCmdLine, Thread): ret = client.send(c) if ret[0] == 0: logSys.log(5, "OK : %r", ret[1]) - if showRet or c[0] == 'echo': + if showRet or c[0] in ('echo', 'server-status'): output(beautifier.beautify(ret[1])) else: logSys.error("NOK: %r", ret[1].args) @@ -128,7 +128,7 @@ class Fail2banClient(Fail2banCmdLine, Thread): except Exception as e: # pragma: no cover if showRet or self._conf["verbose"] > 1: logSys.debug(e) - if showRet or c[0] == 'echo': + if showRet or c[0] in ('echo', 'server-status'): sys.stdout.flush() return streamRet @@ -186,7 +186,7 @@ class Fail2banClient(Fail2banCmdLine, Thread): logSys.error("Fail2ban seems to be in unexpected state (not running but the socket exists)") return None - stream.append(['echo', 'Server ready']) + stream.append(['server-status']) return stream ## diff --git a/fail2ban/server/transmitter.py b/fail2ban/server/transmitter.py index ecc2a138..1aa7f3e2 100644 --- a/fail2ban/server/transmitter.py +++ b/fail2ban/server/transmitter.py @@ -115,6 +115,9 @@ class Transmitter: return cnt elif command[0] == "echo": return command[1:] + elif command[0] == "server-status": + logSys.debug("Server ready") + return "Server ready" elif command[0] == "sleep": value = command[1] time.sleep(float(value)) From 71b19d9eba8547070c5080023829bc4989405dad Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 7 Mar 2018 15:25:27 +0100 Subject: [PATCH 058/189] stability of time-related test-cases: a bit increased timeouts; code normalization, review and coverage --- fail2ban/tests/fail2banclienttestcase.py | 4 ++-- fail2ban/tests/filtertestcase.py | 15 +++++-------- fail2ban/tests/misctestcase.py | 18 +++++++++++++++ fail2ban/tests/utils.py | 28 ++++++++++++++++-------- 4 files changed, 45 insertions(+), 20 deletions(-) diff --git a/fail2ban/tests/fail2banclienttestcase.py b/fail2ban/tests/fail2banclienttestcase.py index ed3d9868..5dcbdef5 100644 --- a/fail2ban/tests/fail2banclienttestcase.py +++ b/fail2ban/tests/fail2banclienttestcase.py @@ -55,8 +55,8 @@ CLIENT = "fail2ban-client" SERVER = "fail2ban-server" BIN = dirname(Fail2banServer.getServerPath()) -MAX_WAITTIME = 30 if not unittest.F2B.fast else 5 -MID_WAITTIME = MAX_WAITTIME +MAX_WAITTIME = unittest.F2B.maxWaitTime(unittest.F2B.MAX_WAITTIME) +MID_WAITTIME = unittest.F2B.maxWaitTime(unittest.F2B.MID_WAITTIME) ## # Several wrappers and settings for proper testing: diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 2bbfcd9d..ec839585 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -82,10 +82,7 @@ def _killfile(f, name): _killfile(None, name + '.bak') -def _maxWaitTime(wtime): - if unittest.F2B.fast: # pragma: no cover - wtime /= 10.0 - return wtime +_maxWaitTime = unittest.F2B.maxWaitTime class _tmSerial(): @@ -657,12 +654,12 @@ class LogFileMonitor(LogCaptureTestCase): _killfile(self.file, self.name) pass - def isModified(self, delay=2.): + def isModified(self, delay=2): """Wait up to `delay` sec to assure that it was modified or not """ return Utils.wait_for(lambda: self.filter.isModified(self.name), _maxWaitTime(delay)) - def notModified(self, delay=2.): + def notModified(self, delay=2): """Wait up to `delay` sec as long as it was not modified """ return Utils.wait_for(lambda: not self.filter.isModified(self.name), _maxWaitTime(delay)) @@ -817,7 +814,7 @@ class CommonMonitorTestCase(unittest.TestCase): super(CommonMonitorTestCase, self).setUp() self._failTotal = 0 - def waitFailTotal(self, count, delay=1.): + def waitFailTotal(self, count, delay=1): """Wait up to `delay` sec to assure that expected failure `count` reached """ ret = Utils.wait_for( @@ -826,7 +823,7 @@ class CommonMonitorTestCase(unittest.TestCase): self._failTotal += count return ret - def isFilled(self, delay=1.): + def isFilled(self, delay=1): """Wait up to `delay` sec to assure that it was modified or not """ return Utils.wait_for(self.jail.isFilled, _maxWaitTime(delay)) @@ -836,7 +833,7 @@ class CommonMonitorTestCase(unittest.TestCase): """ return Utils.wait_for(self.jail.isEmpty, _maxWaitTime(delay)) - def waitForTicks(self, ticks, delay=2.): + def waitForTicks(self, ticks, delay=2): """Wait up to `delay` sec to assure that it was modified or not """ last_ticks = self.filter.ticks diff --git a/fail2ban/tests/misctestcase.py b/fail2ban/tests/misctestcase.py index fda1a5cd..adf7886b 100644 --- a/fail2ban/tests/misctestcase.py +++ b/fail2ban/tests/misctestcase.py @@ -287,6 +287,20 @@ class TestsUtilsTest(LogCaptureTestCase): self.assertNotLogged('test "xyz"') self.assertNotLogged('test', 'xyz', all=False) self.assertNotLogged('test', 'xyz', 'zyx', all=True) + ## maxWaitTime: + orgfast, unittest.F2B.fast = unittest.F2B.fast, False + self.assertFalse(isinstance(unittest.F2B.maxWaitTime(True), bool)) + self.assertEqual(unittest.F2B.maxWaitTime(lambda: 50)(), 50) + self.assertEqual(unittest.F2B.maxWaitTime(25), 25) + self.assertEqual(unittest.F2B.maxWaitTime(25.), 25.0) + unittest.F2B.fast = True + try: + self.assertEqual(unittest.F2B.maxWaitTime(lambda: 50)(), 50) + self.assertEqual(unittest.F2B.maxWaitTime(25), 2.5) + self.assertEqual(unittest.F2B.maxWaitTime(25.), 25.0) + finally: + unittest.F2B.fast = orgfast + self.assertFalse(unittest.F2B.maxWaitTime(False)) ## assertLogged, assertNotLogged negative case: self.pruneLog() logSys.debug('test "xyz"') @@ -296,8 +310,12 @@ class TestsUtilsTest(LogCaptureTestCase): self.assertNotLogged, 'test', 'xyz', all=True) self._testAssertionErrorRE(r"was not found in the log", self.assertLogged, 'test', 'zyx', all=True) + self._testAssertionErrorRE(r"was not found in the log, waited 1e-06", + self.assertLogged, 'test', 'zyx', all=True, wait=1e-6) self._testAssertionErrorRE(r"None among .* was found in the log", self.assertLogged, 'test_zyx', 'zyx', all=False) + self._testAssertionErrorRE(r"None among .* was found in the log, waited 1e-06", + self.assertLogged, 'test_zyx', 'zyx', all=False, wait=1e-6) self._testAssertionErrorRE(r"All of the .* were found present in the log", self.assertNotLogged, 'test', 'xyz', all=False) ## assertDictEqual: diff --git a/fail2ban/tests/utils.py b/fail2ban/tests/utils.py index 3eeb8eb4..901952eb 100644 --- a/fail2ban/tests/utils.py +++ b/fail2ban/tests/utils.py @@ -180,6 +180,10 @@ def initProcess(opts): class F2B(DefaultTestOptions): + + MAX_WAITTIME = 60 + MID_WAITTIME = 30 + def __init__(self, opts): self.__dict__ = opts.__dict__ if self.fast: @@ -215,8 +219,12 @@ class F2B(DefaultTestOptions): return wrapper return _deco_wrapper - def maxWaitTime(self,wtime): - if self.fast: + def maxWaitTime(self, wtime=True): + if isinstance(wtime, bool) and wtime: + wtime = self.MAX_WAITTIME + # short only integer interval (avoid by conditional wait with callable, and dual + # wrapping in some routines, if it will be called twice): + if self.fast and isinstance(wtime, int): wtime = float(wtime) / 10 return wtime @@ -761,21 +769,24 @@ class LogCaptureTestCase(unittest.TestCase): """ wait = kwargs.get('wait', None) if wait: + wait = unittest.F2B.maxWaitTime(wait) res = Utils.wait_for(lambda: self._is_logged(*s, **kwargs), wait) else: res = self._is_logged(*s, **kwargs) if not kwargs.get('all', False): # at least one entry should be found: - if not res: # pragma: no cover + if not res: logged = self._log.getvalue() - self.fail("None among %r was found in the log: ===\n%s===" % (s, logged)) + self.fail("None among %r was found in the log%s: ===\n%s===" % (s, + ((', waited %s' % wait) if wait else ''), logged)) else: # each entry should be found: - if not res: # pragma: no cover + if not res: logged = self._log.getvalue() for s_ in s: if s_ not in logged: - self.fail("%r was not found in the log: ===\n%s===" % (s_, logged)) + self.fail("%r was not found in the log%s: ===\n%s===" % (s_, + ((', waited %s' % wait) if wait else ''), logged)) def assertNotLogged(self, *s, **kwargs): """Assert that strings were not logged @@ -792,11 +803,10 @@ class LogCaptureTestCase(unittest.TestCase): for s_ in s: if s_ not in logged: return - if True: # pragma: no cover - self.fail("All of the %r were found present in the log: ===\n%s===" % (s, logged)) + self.fail("All of the %r were found present in the log: ===\n%s===" % (s, logged)) else: for s_ in s: - if s_ in logged: # pragma: no cover + if s_ in logged: self.fail("%r was found in the log: ===\n%s===" % (s_, logged)) def pruneLog(self, logphase=None): From a3739bbf27c391d89814abc9cf53aa4f5ca814bf Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 7 Mar 2018 16:25:54 +0100 Subject: [PATCH 059/189] trim name and add one space after padding --- fail2ban/server/server.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fail2ban/server/server.py b/fail2ban/server/server.py index b166b94a..039dd879 100644 --- a/fail2ban/server/server.py +++ b/fail2ban/server/server.py @@ -566,7 +566,7 @@ class Server: self.__logTarget = target return True # set a format which is simpler for console use - fmt = "%(name)-24s[%(process)d]: %(levelname)-7s %(message)s" + fmt = "%(name)-23.23s [%(process)d]: %(levelname)-7s %(message)s" if systarget == "SYSLOG": facility = logOptions.get('facility', 'DAEMON').upper() try: From ce6ca0029a79425976e60cc66aef3761f6c2008e Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 7 Mar 2018 16:27:42 +0100 Subject: [PATCH 060/189] minimize log output in trace case (index instead of full-regexp by "matched" log-line) --- fail2ban/server/filter.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index 9494b869..5b9125ed 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -693,12 +693,12 @@ class Filter(JailThread): # Iterates over all the regular expressions. for failRegexIndex, failRegex in enumerate(self.__failRegex): if logSys.getEffectiveLevel() <= logging.HEAVYDEBUG: # pragma: no cover - logSys.log(5, " Looking for failregex %r", failRegex.getRegex()) + logSys.log(5, " Looking for failregex %d - %r", failRegexIndex, failRegex.getRegex()) failRegex.search(self.__lineBuffer, orgBuffer) if not failRegex.hasMatched(): continue # The failregex matched. - logSys.log(7, " Matched %s", failRegex) + logSys.log(7, " Matched failregex %d: %s", failRegexIndex, failRegex.getGroups()) # Checks if we must ignore this match. if self.ignoreLine(failRegex.getMatchedTupleLines()) \ is not None: From 2e533a3a3a4f9eda488f84d370c4731fa5f54dd0 Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 9 Mar 2018 13:54:04 +0100 Subject: [PATCH 061/189] better handling of default date templates (bounds, replacement using own expressions `...{DATE}...`, etc.) --- fail2ban/server/datedetector.py | 70 +++++++++++++++++++-------------- fail2ban/server/datetemplate.py | 14 ++++++- 2 files changed, 53 insertions(+), 31 deletions(-) diff --git a/fail2ban/server/datedetector.py b/fail2ban/server/datedetector.py index 13d70699..2e85b940 100644 --- a/fail2ban/server/datedetector.py +++ b/fail2ban/server/datedetector.py @@ -109,9 +109,6 @@ class DateDetectorCache(object): """Cache Fail2Ban's default template. """ - if isinstance(template, str): - # exact given template with word begin-end boundary: - template = _getPatternTemplate(template) # if not already line-begin anchored, additional template, that prefers datetime # at start of a line (safety+performance feature): name = template.name @@ -126,60 +123,74 @@ class DateDetectorCache(object): # add template: self.__tmpcache[1].append(template) - def _addDefaultTemplate(self): - """Add resp. cache Fail2Ban's default set of date templates. - """ - self.__tmpcache = [], [] + DEFAULT_TEMPLATES = [ # ISO 8601, simple date, optional subsecond and timezone: # 2005-01-23T21:59:59.981746, 2005-01-23 21:59:59, 2005-01-23 8:59:59 # simple date: 2005/01/23 21:59:59 # custom for syslog-ng 2006.12.21 06:43:20 - self._cacheTemplate("%ExY(?P<_sep>[-/.])%m(?P=_sep)%d(?:T| ?)%H:%M:%S(?:[.,]%f)?(?:\s*%z)?") + "%ExY(?P<_sep>[-/.])%m(?P=_sep)%d(?:T| ?)%H:%M:%S(?:[.,]%f)?(?:\s*%z)?", # asctime with optional day, subsecond and/or year: # Sun Jan 23 21:59:59.011 2005 - self._cacheTemplate("(?:%a )?%b %d %k:%M:%S(?:\.%f)?(?: %ExY)?") + "(?:%a )?%b %d %k:%M:%S(?:\.%f)?(?: %ExY)?", # asctime with optional day, subsecond and/or year coming after day # http://bugs.debian.org/798923 # Sun Jan 23 2005 21:59:59.011 - self._cacheTemplate("(?:%a )?%b %d %ExY %k:%M:%S(?:\.%f)?") + "(?:%a )?%b %d %ExY %k:%M:%S(?:\.%f)?", # simple date too (from x11vnc): 23/01/2005 21:59:59 # and with optional year given by 2 digits: 23/01/05 21:59:59 # (See http://bugs.debian.org/537610) # 17-07-2008 17:23:25 - self._cacheTemplate("%d(?P<_sep>[-/])%m(?P=_sep)(?:%ExY|%Exy) %k:%M:%S") + "%d(?P<_sep>[-/])%m(?P=_sep)(?:%ExY|%Exy) %k:%M:%S", # Apache format optional time zone: # [31/Oct/2006:09:22:55 -0000] # 26-Jul-2007 15:20:52 # named 26-Jul-2007 15:20:52.252 # roundcube 26-Jul-2007 15:20:52 +0200 - self._cacheTemplate("%d(?P<_sep>[-/])%b(?P=_sep)%ExY[ :]?%H:%M:%S(?:\.%f)?(?: %z)?") + "%d(?P<_sep>[-/])%b(?P=_sep)%ExY[ :]?%H:%M:%S(?:\.%f)?(?: %z)?", # CPanel 05/20/2008:01:57:39 - self._cacheTemplate("%m/%d/%ExY:%H:%M:%S") + "%m/%d/%ExY:%H:%M:%S", # 01-27-2012 16:22:44.252 # subseconds explicit to avoid possible %m<->%d confusion # with previous ("%d-%m-%ExY %k:%M:%S" by "%d(?P<_sep>[-/])%m(?P=_sep)(?:%ExY|%Exy) %k:%M:%S") - self._cacheTemplate("%m-%d-%ExY %k:%M:%S(?:\.%f)?") + "%m-%d-%ExY %k:%M:%S(?:\.%f)?", # Epoch - self._cacheTemplate('EPOCH') + "EPOCH", # Only time information in the log - self._cacheTemplate("{^LN-BEG}%H:%M:%S") + "{^LN-BEG}%H:%M:%S", # <09/16/08@05:03:30> - self._cacheTemplate("^<%m/%d/%Exy@%H:%M:%S>") + "^<%m/%d/%Exy@%H:%M:%S>", # MySQL: 130322 11:46:11 - self._cacheTemplate("%Exy%Exm%Exd ?%H:%M:%S") + "%Exy%Exm%Exd ?%H:%M:%S", # Apache Tomcat - self._cacheTemplate("%b %d, %ExY %I:%M:%S %p") + "%b %d, %ExY %I:%M:%S %p", # ASSP: Apr-27-13 02:33:06 - self._cacheTemplate("^%b-%d-%Exy %k:%M:%S") + "^%b-%d-%Exy %k:%M:%S", # 20050123T215959, 20050123 215959, 20050123 85959 - self._cacheTemplate("%ExY%Exm%Exd(?:T| ?)%ExH%ExM%ExS(?:[.,]%f)?(?:\s*%z)?") + "%ExY%Exm%Exd(?:T| ?)%ExH%ExM%ExS(?:[.,]%f)?(?:\s*%z)?", # prefixed with optional named time zone (monit): # PDT Apr 16 21:05:29 - self._cacheTemplate("(?:%Z )?(?:%a )?%b %d %k:%M:%S(?:\.%f)?(?: %ExY)?") + "(?:%Z )?(?:%a )?%b %d %k:%M:%S(?:\.%f)?(?: %ExY)?", # +00:00 Jan 23 21:59:59.011 2005 - self._cacheTemplate("(?:%z )?(?:%a )?%b %d %k:%M:%S(?:\.%f)?(?: %ExY)?") + "(?:%z )?(?:%a )?%b %d %k:%M:%S(?:\.%f)?(?: %ExY)?", # TAI64N - self._cacheTemplate("TAI64N") + "TAI64N", + ] + + @property + def defaultTemplates(self): + if isinstance(DateDetectorCache.DEFAULT_TEMPLATES[0], str): + for i, dt in enumerate(DateDetectorCache.DEFAULT_TEMPLATES): + dt = _getPatternTemplate(dt) + DateDetectorCache.DEFAULT_TEMPLATES[i] = dt + return DateDetectorCache.DEFAULT_TEMPLATES + + def _addDefaultTemplate(self): + """Add resp. cache Fail2Ban's default set of date templates. + """ + self.__tmpcache = [], [] + # cache default templates: + for dt in self.defaultTemplates: + self._cacheTemplate(dt) # self.__templates = self.__tmpcache[0] + self.__tmpcache[1] del self.__tmpcache @@ -269,8 +280,7 @@ class DateDetector(object): self.addDefaultTemplate(flt) return elif "{DATE}" in key: - self.addDefaultTemplate( - lambda template: not template.flags & DateTemplate.LINE_BEGIN, pattern) + self.addDefaultTemplate(preMatch=pattern, allDefaults=False) return else: template = _getPatternTemplate(pattern, key) @@ -283,18 +293,20 @@ class DateDetector(object): logSys.debug(" date pattern regex for %r: %s", getattr(template, 'pattern', ''), template.regex) - def addDefaultTemplate(self, filterTemplate=None, preMatch=None): + def addDefaultTemplate(self, filterTemplate=None, preMatch=None, allDefaults=True): """Add Fail2Ban's default set of date templates. """ ignoreDup = len(self.__templates) > 0 - for template in DateDetector._defCache.templates: + for template in ( + DateDetector._defCache.templates if allDefaults else DateDetector._defCache.defaultTemplates + ): # filter if specified: if filterTemplate is not None and not filterTemplate(template): continue # if exact pattern available - create copy of template, contains replaced {DATE} with default regex: if preMatch is not None: # get cached or create a copy with modified name/pattern, using preMatch replacement for {DATE}: template = _getAnchoredTemplate(template, - wrap=lambda s: RE_DATE_PREMATCH.sub(lambda m: s, preMatch)) + wrap=lambda s: RE_DATE_PREMATCH.sub(lambda m: DateTemplate.unboundPattern(s), preMatch)) # append date detector template (ignore duplicate if some was added before default): self._appendTemplate(template, ignoreDup=ignoreDup) diff --git a/fail2ban/server/datetemplate.py b/fail2ban/server/datetemplate.py index 49fa0c66..606ba6ec 100644 --- a/fail2ban/server/datetemplate.py +++ b/fail2ban/server/datetemplate.py @@ -37,8 +37,10 @@ RE_GROUPED = re.compile(r'(? Date: Fri, 9 Mar 2018 13:56:38 +0100 Subject: [PATCH 062/189] filter.d/recidive.conf: fixed if logging into systemd-journal (SYSLOG) with daemon name in prefix, gh-2069 --- ChangeLog | 1 + config/filter.d/recidive.conf | 12 ++++++------ fail2ban/tests/files/logs/recidive | 5 +++++ 3 files changed, 12 insertions(+), 6 deletions(-) diff --git a/ChangeLog b/ChangeLog index 82fb8668..ceb3c822 100644 --- a/ChangeLog +++ b/ChangeLog @@ -37,6 +37,7 @@ ver. 0.10.3-dev-1 (20??/??/??) - development edition ### Fixes * `filter.d/asterisk.conf`: fixed failregex prefix by log over remote syslog server (gh-2060); * `filter.d/exim.conf`: failregex extended - SMTP call dropped: too many syntax or protocol errors (gh-2048); +* `filter.d/recidive.conf`: fixed if logging into systemd-journal (SYSLOG) with daemon name in prefix, gh-2069; * `filter.d/sshd.conf`: - failregex got an optional space in order to match new log-format (see gh-2061); - fixed ddos-mode regex to match refactored message (some versions can contain port now, see gh-2062); diff --git a/config/filter.d/recidive.conf b/config/filter.d/recidive.conf index e2501cf6..63833cab 100644 --- a/config/filter.d/recidive.conf +++ b/config/filter.d/recidive.conf @@ -21,18 +21,18 @@ before = common.conf [Definition] -_daemon = fail2ban\.actions\s* +_daemon = (?:fail2ban(?:-server|\.actions)\s*) -# The name of the jail that this filter is used for. In jail.conf, name the -# jail using this filter 'recidive', or change this line! +# The name of the jail that this filter is used for. In jail.conf, name the jail using +# this filter 'recidive', or supply another name with `filter = recidive[_jailname="jail"]` _jailname = recidive -failregex = ^(%(__prefix_line)s| %(_daemon)s%(__pid_re)s?:\s+)NOTICE\s+\[(?!%(_jailname)s\])(?:.*)\]\s+Ban\s+\s*$ +failregex = ^%(__prefix_line)s(?:\s*fail2ban\.actions\s*%(__pid_re)s?:\s+)?NOTICE\s+\[(?!%(_jailname)s\])(?:.*)\]\s+Ban\s+\s*$ + +datepattern = ^{DATE} ignoreregex = -[Init] - journalmatch = _SYSTEMD_UNIT=fail2ban.service PRIORITY=5 # Author: Tom Hendrikx, modifications by Amir Caspi diff --git a/fail2ban/tests/files/logs/recidive b/fail2ban/tests/files/logs/recidive index edb3a155..5746a55d 100644 --- a/fail2ban/tests/files/logs/recidive +++ b/fail2ban/tests/files/logs/recidive @@ -12,3 +12,8 @@ Sep 16 00:44:55 spaceman fail2ban.actions: NOTICE [jail] Ban 10.0.0.7 # failJSON: { "time": "2006-02-13T15:52:30", "match": true , "host": "1.2.3.4", "desc": "Extended with [PID] and padding" } 2006-02-13 15:52:30,388 fail2ban.actions [123]: NOTICE [sendmail] Ban 1.2.3.4 + +# failJSON: { "time": "2005-01-16T17:11:25", "match": true , "host": "192.0.2.1", "desc": "SYSLOG / systemd-journal without daemon-name" } +Jan 16 17:11:25 testorg fail2ban.actions[6605]: NOTICE [postfix-auth] Ban 192.0.2.1 +# failJSON: { "time": "2005-03-05T08:41:28", "match": true , "host": "192.0.2.2", "desc": "SYSLOG / systemd-journal with daemon-name" } +Mar 05 08:41:28 test.org fail2ban-server[11524]: fail2ban.actions [11524]: NOTICE [postfix-auth] Ban 192.0.2.2 From e8ffab28fb8c50c576f0422c726b0b1c04829397 Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 19 Mar 2018 13:18:55 +0100 Subject: [PATCH 063/189] filter.d/apache-noscript.conf: extended to match "Primary script unknown", got from php-fpm module. --- config/filter.d/apache-noscript.conf | 9 +++++++-- fail2ban/tests/files/logs/apache-noscript | 2 ++ 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/config/filter.d/apache-noscript.conf b/config/filter.d/apache-noscript.conf index fbc1af64..abc083a6 100644 --- a/config/filter.d/apache-noscript.conf +++ b/config/filter.d/apache-noscript.conf @@ -17,8 +17,13 @@ before = apache-common.conf [Definition] -failregex = ^%(_apache_error_client)s ((AH001(28|30): )?File does not exist|(AH01264: )?script not found or unable to stat): /\S*(php([45]|[.-]cgi)?|\.asp|\.exe|\.pl)(, referer: \S+)?\s*$ - ^%(_apache_error_client)s script '/\S*(php([45]|[.-]cgi)?|\.asp|\.exe|\.pl)\S*' not found or unable to stat(, referer: \S+)?\s*$ +script = /\S*(?:php(?:[45]|[.-]cgi)?|\.asp|\.exe|\.pl) + +prefregex = ^%(_apache_error_client)s .+$ + +failregex = ^(?:(?:AH001(?:28|30): )?File does not exist|(AH01264: )?script not found or unable to stat):