diff --git a/ChangeLog b/ChangeLog index 8365a0a0..b484a1f8 100644 --- a/ChangeLog +++ b/ChangeLog @@ -45,10 +45,25 @@ ver. 0.10.1-dev-1 (2017/??/??) - development current edition ----------- ### Fixes +* jail "pass2allow-ftp" supply blocktype and returntype parameters to the action (gh-1884) +* avoid using "ANSI_X3.4-1968" as preferred encoding (if missing environment variables + 'LANGUAGE', 'LC_ALL', 'LC_CTYPE', and 'LANG', see gh-1587). +* action.d/pf.conf: several fixes for pf-action like anchoring, etc. (see gh-1866, gh-1867); +* fixed ignorself issue "Retrieving own IPs of localhost failed: inet_pton() argument 2 must be string, not int" (see gh-1865); +* fixed tags `` and ``, could be used without ticket (a. g. in `actionstart` etc., gh-1859). + +* setup.py: fixed several setup facilities (gh-1874): + - don't check return code by dry-run: returns 256 on some python/setuptool versions; + - `files/fail2ban.service` renamed as template to `files/fail2ban.service.in`; + - setup process generates `build/fail2ban.service` from `files/fail2ban.service.in` using distribution related bin-path; + - bug-fixing by running setup with option `--dry-run`; ### New Features +* introduced new command-line options `--dp`, `--dump-pretty` to dump the configuration using more + human readable representation (opposite to `-d`); ### Enhancements +* filter.d/dovecot.conf: introduced mode `aggressive` for cases like "disconnected before auth was ready" (gh-1880) ver. 0.10.0 (2017/08/09) - long-awaited 0.10th version @@ -411,13 +426,18 @@ releases. * filter.d/apache-overflows.conf: - Fixes resources greedy expression (see gh-1790); - Rewritten without end-anchor ($), because of potential vulnerability on very long URLs. +* filter.d/apache-badbots.conf - extended to recognize Jorgee Vulnerability Scanner (gh-1882) * filter.d/asterisk.conf - fixed failregex AMI Asterisk authentification failed (see gh-1302) +* filter.d/dovecot.conf - fixed failregex, see gh-1879 (partially cherry-picked from gh-1880) +* filter.d/exim.conf - fixed failregex for case of flood attempts with `D=0s` (gh-1887) ### New Features ### Enhancements * action.d/cloudflare.conf - Cloudflare API v4 implementation (gh-1651) * filter.d/kerio.conf - filter extended with new rules (see gh-1455) +* filter.d/phpmyadmin-syslog.conf - new filter for phpMyAdmin using syslog for auth logging +* filter.d/zoneminder.conf - new filter for ZoneMinder (gh-1376) ver. 0.9.7 (2017/05/11) - awaiting-victory diff --git a/config/filter.d/apache-badbots.conf b/config/filter.d/apache-badbots.conf index f42aa159..12d4105b 100644 --- a/config/filter.d/apache-badbots.conf +++ b/config/filter.d/apache-badbots.conf @@ -7,7 +7,7 @@ [Definition] -badbotscustom = EmailCollector|WebEMailExtrac|TrackBack/1\.02|sogou music spider +badbotscustom = EmailCollector|WebEMailExtrac|TrackBack/1\.02|sogou music spider|(?:Mozilla/\d+\.\d+ )?Jorgee badbots = Atomic_Email_Hunter/4\.0|atSpider/1\.0|autoemailspider|bwh3_user_agent|China Local Browse 2\.6|ContactBot/0\.2|ContentSmartz|DataCha0s/2\.0|DBrowse 1\.4b|DBrowse 1\.4d|Demo Bot DOT 16b|Demo Bot Z 16b|DSurf15a 01|DSurf15a 71|DSurf15a 81|DSurf15a VA|EBrowse 1\.4b|Educate Search VxB|EmailSiphon|EmailSpider|EmailWolf 1\.00|ESurf15a 15|ExtractorPro|Franklin Locator 1\.8|FSurf15a 01|Full Web Bot 0416B|Full Web Bot 0516B|Full Web Bot 2816B|Guestbook Auto Submitter|Industry Program 1\.0\.x|ISC Systems iRc Search 2\.1|IUPUI Research Bot v 1\.9a|LARBIN-EXPERIMENTAL \(efp@gmx\.net\)|LetsCrawl\.com/1\.0 \+http\://letscrawl\.com/|Lincoln State Web Browser|LMQueueBot/0\.2|LWP\:\:Simple/5\.803|Mac Finder 1\.0\.xx|MFC Foundation Class Library 4\.0|Microsoft URL Control - 6\.00\.8xxx|Missauga Locate 1\.0\.0|Missigua Locator 1\.9|Missouri College Browse|Mizzu Labs 2\.2|Mo College 1\.9|MVAClient|Mozilla/2\.0 \(compatible; NEWT ActiveX; Win32\)|Mozilla/3\.0 \(compatible; Indy Library\)|Mozilla/3\.0 \(compatible; scan4mail \(advanced version\) http\://www\.peterspages\.net/?scan4mail\)|Mozilla/4\.0 \(compatible; Advanced Email Extractor v2\.xx\)|Mozilla/4\.0 \(compatible; Iplexx Spider/1\.0 http\://www\.iplexx\.at\)|Mozilla/4\.0 \(compatible; MSIE 5\.0; Windows NT; DigExt; DTS Agent|Mozilla/4\.0 efp@gmx\.net|Mozilla/5\.0 \(Version\: xxxx Type\:xx\)|NameOfAgent \(CMS Spider\)|NASA Search 1\.0|Nsauditor/1\.x|PBrowse 1\.4b|PEval 1\.4b|Poirot|Port Huron Labs|Production Bot 0116B|Production Bot 2016B|Production Bot DOT 3016B|Program Shareware 1\.0\.2|PSurf15a 11|PSurf15a 51|PSurf15a VA|psycheclone|RSurf15a 41|RSurf15a 51|RSurf15a 81|searchbot admin@google\.com|ShablastBot 1\.0|snap\.com beta crawler v0|Snapbot/1\.0|Snapbot/1\.0 \(Snap Shots, \+http\://www\.snap\.com\)|sogou develop spider|Sogou Orion spider/3\.0\(\+http\://www\.sogou\.com/docs/help/webmasters\.htm#07\)|sogou spider|Sogou web spider/3\.0\(\+http\://www\.sogou\.com/docs/help/webmasters\.htm#07\)|sohu agent|SSurf15a 11 |TSurf15a 11|Under the Rainbow 2\.2|User-Agent\: Mozilla/4\.0 \(compatible; MSIE 6\.0; Windows NT 5\.1\)|VadixBot|WebVulnCrawl\.unknown/1\.0 libwww-perl/5\.803|Wells Search II|WEP Search 00 failregex = ^ -.*"(GET|POST|HEAD).*HTTP.*"(?:%(badbots)s|%(badbotscustom)s)"$ diff --git a/config/filter.d/exim.conf b/config/filter.d/exim.conf index 2151a161..3c362490 100644 --- a/config/filter.d/exim.conf +++ b/config/filter.d/exim.conf @@ -22,7 +22,7 @@ failregex = ^%(pid)s %(host_info)ssender verify fail for <\S+>: (?:Unknown user| ^%(pid)s SMTP protocol synchronization error \([^)]*\): rejected (?:connection from|"\S+") %(host_info)s(?:next )?input=".*"\s*$ ^%(pid)s SMTP call from \S+ %(host_info)sdropped: too many nonmail commands \(last was "\S+"\)\s*$ ^%(pid)s SMTP protocol error in "AUTH \S*(?: \S*)?" %(host_info)sAUTH command used when not advertised\s*$ - ^%(pid)s no MAIL in SMTP connection from (?:[^\[\( ]* )?(?:\(\S*\) )?%(host_info)sD=\d\S+s(?: C=\S*)?\s*$ + ^%(pid)s no MAIL in SMTP connection from (?:[^\[\( ]* )?(?:\(\S*\) )?%(host_info)sD=\d\S*s(?: C=\S*)?\s*$ ^%(pid)s (?:[\w\-]+ )?SMTP connection from (?:[^\[\( ]* )?(?:\(\S*\) )?%(host_info)sclosed by DROP in ACL\s*$ ignoreregex = diff --git a/config/filter.d/phpmyadmin-syslog.conf b/config/filter.d/phpmyadmin-syslog.conf new file mode 100644 index 00000000..5b0862bb --- /dev/null +++ b/config/filter.d/phpmyadmin-syslog.conf @@ -0,0 +1,18 @@ +# Fail2Ban fitler for the phpMyAdmin-syslog +# + +[INCLUDES] + +before = common.conf + +[Definition] + +_daemon = phpMyAdmin + +failregex = ^%(__prefix_line)suser denied: (?:\S+|.*?) \(mysql-denied\) from \s*$ + +ignoreregex = + + +# Author: Pavel Mihadyuk +# Regex fixes: Serg G. Brester diff --git a/config/filter.d/zoneminder.conf b/config/filter.d/zoneminder.conf new file mode 100644 index 00000000..cc82755a --- /dev/null +++ b/config/filter.d/zoneminder.conf @@ -0,0 +1,21 @@ +# Fail2Ban filter for Zoneminder login failures + +[INCLUDES] +before = apache-common.conf + +[Definition] + +# pattern: [Wed Apr 27 23:12:07.736196 2016] [:error] [pid 2460] [client 10.1.1.1:47296] WAR [Login denied for user "test"], referer: https://zoneminderurl/index.php +# +# +# Option: failregex +# Notes.: regex to match the password failure messages in the logfile. + +failregex = ^%(_apache_error_client)s WAR \[Login denied for user "[^"]*"\] + +ignoreregex = + +# Notes: +# Tested on Zoneminder 1.29.0 +# +# Author: John Marzella diff --git a/config/jail.conf b/config/jail.conf index cc887fc5..dc385912 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -877,6 +877,7 @@ filter = apache-pass[knocking_url="%(knocking_url)s"] logpath = %(apache_access_log)s blocktype = RETURN returntype = DROP +action = %(action_)s[blocktype=%(blocktype)s, returntype=%(returntype)s] bantime = 1h maxretry = 1 findtime = 1 @@ -911,3 +912,17 @@ logpath = /var/log/slapd.log port = smtp,ssmtp filter = domino-smtp logpath = /home/domino01/data/IBM_TECHNICAL_SUPPORT/console.log + +[phpmyadmin-syslog] +port = http,https +filter = phpmyadmin-syslog +logpath = %(syslog_authpriv)s +backend = %(syslog_backend)s + + +[zoneminder] +# Zoneminder HTTP/HTTPS web interface auth +# Logs auth failures to apache2 error log +port = http,https +logpath = %(apache_error_log)s + diff --git a/fail2ban/client/actionreader.py b/fail2ban/client/actionreader.py index 3d64f701..3ed8204c 100644 --- a/fail2ban/client/actionreader.py +++ b/fail2ban/client/actionreader.py @@ -89,11 +89,11 @@ class ActionReader(DefinitionInitConfigReader): stream.append(head + ["addaction", self._name]) multi = [] for opt, optval in opts.iteritems(): - if opt in self._configOpts: + if opt in self._configOpts and not opt.startswith('known/'): multi.append([opt, optval]) if self._initOpts: for opt, optval in self._initOpts.iteritems(): - if opt not in self._configOpts: + if opt not in self._configOpts and not opt.startswith('known/'): multi.append([opt, optval]) if len(multi) > 1: stream.append(["multi-set", self._jailName, "action", self._name, multi]) diff --git a/fail2ban/client/configparserinc.py b/fail2ban/client/configparserinc.py index b626be9b..d220ed44 100644 --- a/fail2ban/client/configparserinc.py +++ b/fail2ban/client/configparserinc.py @@ -20,8 +20,8 @@ # Author: Yaroslav Halchenko # Modified: Cyril Jaquier -__author__ = 'Yaroslav Halhenko' -__copyright__ = 'Copyright (c) 2007 Yaroslav Halchenko' +__author__ = 'Yaroslav Halhenko, Serg G. Brester (aka sebres)' +__copyright__ = 'Copyright (c) 2007 Yaroslav Halchenko, 2015 Serg G. Brester (aka sebres)' __license__ = 'GPL' import os @@ -150,7 +150,7 @@ after = 1.conf if seclwr == 'known': # try get raw value from known options: try: - v = self._sections['KNOWN'][opt] + v = self._sections['KNOWN/'+section][opt] except KeyError: # fallback to default: try: @@ -297,7 +297,6 @@ after = 1.conf # merge defaults and all sections to self: alld.update(cfg.get_defaults()) for n, s in cfg.get_sections().iteritems(): - curalls = alls # conditional sections cond = SafeConfigParserWithIncludes.CONDITIONAL_RE.match(n) if cond: @@ -313,7 +312,7 @@ after = 1.conf s2 = alls.get(n) if isinstance(s2, dict): # save previous known values, for possible using in local interpolations later: - self.merge_section('KNOWN', s2, '') + self.merge_section('KNOWN/'+n, s2, '') # merge section s2.update(s) else: diff --git a/fail2ban/client/configreader.py b/fail2ban/client/configreader.py index 381af759..577a5a16 100644 --- a/fail2ban/client/configreader.py +++ b/fail2ban/client/configreader.py @@ -20,8 +20,8 @@ # Author: Cyril Jaquier # Modified by: Yaroslav Halchenko (SafeConfigParserWithIncludes) -__author__ = "Cyril Jaquier" -__copyright__ = "Copyright (c) 2004 Cyril Jaquier" +__author__ = "Cyril Jaquier, Yaroslav Halchenko, Serg G. Brester (aka sebres)" +__copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2007 Yaroslav Halchenko, 2015 Serg G. Brester (aka sebres)" __license__ = "GPL" import glob @@ -110,7 +110,7 @@ class ConfigReader(): def sections(self): try: - return (n for n in self._cfg.sections() if n != 'KNOWN') + return (n for n in self._cfg.sections() if not n.startswith('KNOWN/')) except AttributeError: return [] diff --git a/fail2ban/client/configurator.py b/fail2ban/client/configurator.py index e8472ac1..fe735251 100644 --- a/fail2ban/client/configurator.py +++ b/fail2ban/client/configurator.py @@ -76,9 +76,9 @@ class Configurator: self.__fail2ban.getOptions(updateMainOpt) return self.__jails.getOptions(jail, ignoreWrong=ignoreWrong) - def convertToProtocol(self): + def convertToProtocol(self, allow_no_files=False): self.__streams["general"] = self.__fail2ban.convert() - self.__streams["jails"] = self.__jails.convert() + self.__streams["jails"] = self.__jails.convert(allow_no_files=allow_no_files) def getConfigStream(self): cmds = list() diff --git a/fail2ban/client/fail2bancmdline.py b/fail2ban/client/fail2bancmdline.py index f213d037..401aa9b6 100644 --- a/fail2ban/client/fail2bancmdline.py +++ b/fail2ban/client/fail2bancmdline.py @@ -102,6 +102,7 @@ class Fail2banCmdLine(): output(" --logtarget |STDOUT|STDERR|SYSLOG") output(" --syslogsocket auto|") output(" -d dump configuration. For debugging") + output(" --dp, --dump-pretty dump the configuration using more human readable representation") output(" -t, --test test configuration (can be also specified with start parameters)") output(" -i interactive mode") output(" -v increase verbosity") @@ -137,8 +138,8 @@ class Fail2banCmdLine(): self._conf["pidfile"] = opt[1] elif o.startswith("--log") or o.startswith("--sys"): self._conf[ o[2:] ] = opt[1] - elif o == "-d": - self._conf["dump"] = True + elif o in ["-d", "--dp", "--dump-pretty"]: + self._conf["dump"] = True if o == "-d" else 2 elif o == "-t" or o == "--test": self.cleanConfOnly = True self._conf["test"] = True @@ -184,7 +185,8 @@ class Fail2banCmdLine(): # Reads the command line options. try: cmdOpts = 'hc:s:p:xfbdtviqV' - cmdLongOpts = ['loglevel=', 'logtarget=', 'syslogsocket=', 'test', 'async', 'timeout=', 'str2sec=', 'help', 'version'] + cmdLongOpts = ['loglevel=', 'logtarget=', 'syslogsocket=', 'test', 'async', + 'timeout=', 'str2sec=', 'help', 'version', 'dp', '--dump-pretty'] optList, self._args = getopt.getopt(self._argv[1:], cmdOpts, cmdLongOpts) except getopt.GetoptError: self.dispUsage() @@ -240,7 +242,10 @@ class Fail2banCmdLine(): if readcfg: ret, stream = self.readConfig() readcfg = False - self.dumpConfig(stream) + if stream is not None: + self.dumpConfig(stream, self._conf["dump"] == 2) + else: # pragma: no cover + output("ERROR: The configuration stream failed because of the invalid syntax.") if not self._conf.get("test", False): return ret @@ -275,7 +280,8 @@ class Fail2banCmdLine(): self.configurator.readAll() ret = self.configurator.getOptions(jail, self._conf, ignoreWrong=not self.cleanConfOnly) - self.configurator.convertToProtocol() + self.configurator.convertToProtocol( + allow_no_files=self._conf.get("dump", False)) stream = self.configurator.getConfigStream() except Exception as e: logSys.error("Failed during configuration: %s" % e) @@ -283,9 +289,15 @@ class Fail2banCmdLine(): return ret, stream @staticmethod - def dumpConfig(cmd): + def dumpConfig(cmd, pretty=False): + if pretty: + from pprint import pformat + def _output(s): + output(pformat(s, width=1000, indent=2)) + else: + _output = output for c in cmd: - output(c) + _output(c) return True # diff --git a/fail2ban/client/jailreader.py b/fail2ban/client/jailreader.py index 85a68ce4..0210ad15 100644 --- a/fail2ban/client/jailreader.py +++ b/fail2ban/client/jailreader.py @@ -242,9 +242,12 @@ class JailReader(ConfigReader): found_files += 1 stream.append( ["set", self.__name, "addlogpath", p, tail]) - if not (found_files or allow_no_files): - raise ValueError( - "Have not found any log file for %s jail" % self.__name) + if not found_files: + msg = "Have not found any log file for %s jail" % self.__name + if not allow_no_files: + raise ValueError(msg) + logSys.warning(msg) + elif opt == "logencoding": stream.append(["set", self.__name, "logencoding", value]) elif opt == "backend": diff --git a/fail2ban/tests/fail2banclienttestcase.py b/fail2ban/tests/fail2banclienttestcase.py index 260fd3a3..7149001f 100644 --- a/fail2ban/tests/fail2banclienttestcase.py +++ b/fail2ban/tests/fail2banclienttestcase.py @@ -458,7 +458,11 @@ class Fail2banClientTest(Fail2banClientServerBase): startparams = _start_params(tmp, True) self.execSuccess(startparams, "-vvd") self.assertLogged("Loading files") - self.assertLogged("logtarget") + self.assertLogged("['set', 'logtarget',") + self.pruneLog() + # pretty dump: + self.execSuccess(startparams, "--dp") + self.assertLogged("['set', 'logtarget',") @with_tmpdir @with_kill_srv diff --git a/fail2ban/tests/files/logs/exim b/fail2ban/tests/files/logs/exim index 8f3ac1bb..f87d393b 100644 --- a/fail2ban/tests/files/logs/exim +++ b/fail2ban/tests/files/logs/exim @@ -53,7 +53,7 @@ # failJSON: { "time": "2016-03-21T06:38:05", "match": true , "host": "49.212.207.15" } 2016-03-21 06:38:05 [5718] no MAIL in SMTP connection from www3005.sakura.ne.jp [49.212.207.15]:28890 I=[172.89.0.6]:25 D=21s C=EHLO,STARTTLS # failJSON: { "time": "2016-03-21T06:57:36", "match": true , "host": "122.165.71.116" } -2016-03-21 06:57:36 [5908] no MAIL in SMTP connection from [122.165.71.116]:2056 I=[172.89.0.6]:25 D=10s +2016-03-21 06:57:36 [5908] no MAIL in SMTP connection from [122.165.71.116]:2056 I=[172.89.0.6]:25 D=0s # failJSON: { "time": "2016-03-21T06:57:36", "match": true , "host": "122.165.71.116" } 2016-03-21 06:57:36 [5908] no MAIL in SMTP connection from [122.165.71.116] I=[172.89.0.6]:25 D=10s # failJSON: { "time": "2016-03-21T04:07:49", "match": true , "host": "174.137.147.204" } diff --git a/fail2ban/tests/files/logs/phpmyadmin-syslog b/fail2ban/tests/files/logs/phpmyadmin-syslog new file mode 100644 index 00000000..f32a2476 --- /dev/null +++ b/fail2ban/tests/files/logs/phpmyadmin-syslog @@ -0,0 +1,2 @@ +# failJSON: { "time": "2004-08-22T14:50:22", "match": true , "host": "192.0.2.1" } +Aug 22 14:50:22 eurostream phpMyAdmin[16358]: user denied: root (mysql-denied) from 192.0.2.1 diff --git a/fail2ban/tests/files/logs/zoneminder b/fail2ban/tests/files/logs/zoneminder new file mode 100644 index 00000000..abd49869 --- /dev/null +++ b/fail2ban/tests/files/logs/zoneminder @@ -0,0 +1,2 @@ +# failJSON: { "time": "2016-03-28T16:50:49", "match": true , "host": "10.1.1.1" } +[Mon Mar 28 16:50:49.522240 2016] [:error] [pid 1795] [client 10.1.1.1:50700] WAR [Login denied for user "username1"], referer: https://zoneminder/