diff --git a/ChangeLog b/ChangeLog index afb2b883..b4243e5b 100644 --- a/ChangeLog +++ b/ChangeLog @@ -47,8 +47,31 @@ code-review and minor additions from Yaroslav Halchenko. Some filters have been change as required to capture these elements in the right timezone correctly. -ver. 0.8.11 (2013/XX/XXX) - loves-unittests ------------ +ver. 0.8.11 (2013/11/XXX) - loves-unittests and tight, DoS free, filter regexes + +In light of CVE-2013-2178 that triggered our last release we have put a +significant effort into tightening all of the regexs of our filters to avoid +another similar vulnerability. All filters have been updated and some to +include more failure regexs supporting previously unbanned failures and +support for newer application versions too. There are test cases for most log +cases of failures now. + +As usual if you have other examples that demonstrate that a filter is +insufficient please give us an example log line on the github issue tracker +http://github.com/fail2ban/fail2ban/issues and NOT on a random blog in some +obscure corner of the Internet. + +During the tightening of the regexs to avoid DoS vulnerabilities there is the +possibility that we have inadvertently, despite our best intentions, +incorrectly allowed a failure to continue. We will fix this as quickly as +humanly possible. + +IMPORTANT incompatible changes: + Filter name changes: + * 'lighttpd-fastcgi' filter has been renamed to 'suhosin' + * 'sasl' has been renamed to 'postfix-sasl' + These will require changing in jail.{conf,local} if using these filters. + Exim filter has been split into an spam and a relay/auth filter. - Fixes: Daniel Black & Marcel Dopita @@ -66,15 +89,36 @@ ver. 0.8.11 (2013/XX/XXX) - loves-unittests - All backends, possible race condition: do not read from a file initially reported empty. Originally could have lead to accounting for detected log lines multiple times. + - Do not crash if executing a command in fail2ban-client interactive + mode has failed (e.g. due to incorrect syntax). Closes gh-353 Daniel Black & Мернов Георгий * filter.d/dovecot.conf -- Fix when no TLS enabled - line doesn't end in , + Daniel Black & Georgiy Mernov & ftoppi & Мернов Георгий + * filter.d/exim.conf -- regex hardening and extra failure examples in + sample logs + * filter.d/named-refused.conf - BIND 9.9.3 regex changes + Daniel Black & Sebastian Arcus + * filter.d/asterisk -- more regexes Daniel Black * action.d/hostsdeny -- NOTE: new dependancy 'ed'. Switched to use 'ed' across all platforms to ensure permissions are the same before and after a ban - closes gh-266. hostsdeny supports daemon_list now too. + * action.d/bsd-ipfw - action option unsed. Change blocktype to port unreach + instead of deny for consistancy. * filter.d/roundcube-auth - timezone offset can be positive or negative * action.d/bsd-ipfw - action option unsed. Fixed to blocktype for consistency. default to port unreach instead of deny + * filter.d/dropbear - fix regexs to match standard dropbear and the patched + http://www.unchartedbackwaters.co.uk/files/dropbear/dropbear-0.52.patch + and add PAM is it in dropbear-2013.60 source code. + * filter.d/{asterisk,assp,dovecot,proftpd}.conf -- regex hardening + and extra failure examples in sample logs + * filter.d/apache-auth - added expressions for mod_authz, mod_auth and + mod_auth_digest failures. + * filter.d/recidive -- support f2b syslog target and anchor regex at start + * filter.d/mysqld-auth.conf - mysql can use syslog + * filter.d/sshd - regex enhancements to support openssh-6.3. Closes Debian + bug #722970 Rolf Fokkens * action.d/dshield.conf and complain.conf -- reorder mailx arguments. https://bugzilla.redhat.com/show_bug.cgi?id=998020 @@ -85,33 +129,53 @@ ver. 0.8.11 (2013/XX/XXX) - loves-unittests * files/redhat-initd - rewritten to use stock init.d functions thus avoiding problems with getpid. Also $network and iptables moved to Should- rc init fields + Rick Mellor + * filter.d/vsftp - fix capture with tty=ftp + - New Features: + Edgar Hoch + * action.d/firewall-cmd-direct-new.conf - action for firewalld + from https://bugzilla.redhat.com/show_bug.cgi?id=979622 Andy Fragen and Daniel Black - * filter.d/osx-ipfw.conf - ipfw action for OSX based on random rule + * filter.d/osx-ipfw.conf - ipfw action for OSX based on random rule numbers. + Anonymous: + * action.d/osx-afctl - an action based on afctl for osx Daniel Black & ykimon * filter.d/3proxy.conf -- filter added + * fail2ban-regex - now generates http://www.debuggex.com urls for debugging + regular expressions with the -D parameter. Daniel Black * filter.d/exim-spam.conf -- a splitout of exim's spam regexes with additions for greater control over filtering spam. * add date expression for apache-2.4 - milliseconds Christophe Carles & Daniel Black * filter.d/perdition.conf -- filter added + Mark McKinstry + * action.d/apf.conf - add action for Advanced Policy Firewall (apf) + Amir Caspi and kjohnsonecl + * filter.d/uwimap-auth - filter for uwimap-auth IMAP/POP server + Steven Hiscocks and Daniel Black + * filter.d/selinux-{common,ssh} -- add SELinux date and ssh filter - Enhancements: François Boulogne and Frédéric * filter.d/lighttpd - auth regexs for lighttpd-1.4.31 Daniel Black - * filter.d/{asterisk,assp,dovecot,proftpd}.conf -- regex hardening - and extra failure examples in sample logs - * filter.d/apache-auth - added expressions for mod_authz, mod_auth and - mod_auth_digest failures. - Daniel Black & Georgiy Mernov & ftoppi & Мернов Георгий - * filter.d/exim.conf -- regex hardening and extra failure examples in - sample logs - * filter.d/named-refused.conf - BIND 9.9.3 regex changes - Daniel Black & Sebastian Arcus - * filter.d/asterisk -- more regexes + * reorder parsing of jail.conf, jail.d/*.conf, jail.local, jail.d/*.local + and likewise for fail2ban.{conf|local|d/*.conf|d/*.local}. Closes gh-392 + * jail.conf now has asterisk jail - no need for asterisk-tcp and + asterisk-udp. Users should replace existing jails with asterisk to + reduce duplicate parsing of the asterisk log file. + * filter.d/{suhosin,pam-generic,gssftpd,sogo-auth,webmin}- regex anchor at + start + * filter.d/vsftpd - anchored regex at start. disable old pam format regex + * filter.d/pam-generic - added syslog prefix. Disabled support for + linux-pam before version 0.99.2.0 (2005) + * filter.d/postfix-sasl - renamed from sasl, anchor at start and base on + syslog + * filter.d/qmail - rewrote regex to anchor at start. Added regex for + another "in the wild" patch to rblsmtp. Yaroslav Halchenko * fail2ban-regex -- refactored to provide more details (missing and ignored lines, control over logging, etc) while maintaining look&feel @@ -132,8 +196,6 @@ ver. 0.8.11 (2013/XX/XXX) - loves-unittests * filter/named-refused - added refused on zone transfer * filter.d/{courier{login,smtp},proftpd,sieve,wuftpd,xinetd} - General regex impovements - * IMPORTANT: 'lighttpd-fastcgi' filter has been renamed to 'suhosin', which - will require changing in jail.{conf,local} if using this filter. Zurd * filter.d/postfix - add filter for VRFY failures. closes gh-322. Orion Poplawski diff --git a/DEVELOP b/DEVELOP index f9cac432..6010f984 100644 --- a/DEVELOP +++ b/DEVELOP @@ -1,6 +1,6 @@ - __ _ _ ___ _ - / _|__ _(_) |_ ) |__ __ _ _ _ - | _/ _` | | |/ /| '_ \/ _` | ' \ + __ _ _ ___ _ + / _|__ _(_) |_ ) |__ __ _ _ _ + | _/ _` | | |/ /| '_ \/ _` | ' \ |_| \__,_|_|_/___|_.__/\__,_|_||_| ================================================================================ @@ -26,7 +26,7 @@ Pull Requests When submitting pull requests on GitHub we ask you to: * Clearly describe the problem you're solving; -* Don't introduce regressions that will make it hard for systems adminstrators +* Don't introduce regressions that will make it hard for systems administrators to update; * If adding a major feature rebase your changes on master and get to a single commit; * Include test cases (see below); @@ -37,49 +37,307 @@ When submitting pull requests on GitHub we ask you to: Filters ======= -* Include sample logs with 1.2.3.4 used for IP addresses and - example.com/example.org used for DNS names -* Ensure sample log is provided in testcases/files/logs/ with same name as the - filter. Each log line should include match meta data for time & IP above - every line (see other sample log files for examples) -* Ensure regexs start with a ^ and are restrictive as possible. E.g. not .* if - \d+ is sufficient -* Use the functionality of regexs http://docs.python.org/2/library/re.html -* Take a look at the source code of the application. You may see optional or - extra log messages, or parts there of, that need to form part of your regex. +Filters are tricky. They need to: +* work with a variety of the versions of the software that generates the logs; +* work with the range of logging configuration options available in the + software; +* work with multiple operating systems; +* not make assumptions about the log format in excess of the software + (e.g. do not assume a username doesn't contain spaces and use \S+ unless + you've checked the source code); +* account for how future versions of the software will log messages + (e.g. guess what would happen to the log message if different authentication + types are added); +* not be susceptible to DoS vulnerabilities (see Filter Security below); and +* match intended log lines only. -If you only have a basic knowledge of regular repressions read -http://docs.python.org/2/library/re.html first. +Please follow the steps from Filter Test Cases to Developing Filter Regular +Expressions and submit a GitHub pull request (PR) afterwards. If you get stuck, +you can push your unfinished changes and still submit a PR -- describe +what you have done, what is the hurdle, and we'll attempt to help (PR +will be automagically updated with future commits you would push to +complete it). + +Filter test cases +----------------- + +Purpose: + +Start by finding the log messages that the application generates related to +some form of authentication failure. If you are adding to an existing filter +think about whether the log messages are of a similar importance and purpose +to the existing filter. If you were a user of Fail2Ban, and did a package +update of Fail2Ban that started matching new log messages, would anything +unexpected happen? Would the bantime/findtime for the jail be appropriate for +the new log messages? If it doesn't, perhaps it needs to be in a separate +filter definition, for example like exim filter aims at authentication failures +and exim-spam at log messages related to spam. + +Even if it is a new filter you may consider separating the log messages into +different filters based on purpose. + +Cause: + +Are some of the log lines a result of the same action? For example, is a PAM +failure log message, followed by an application specific failure message the +result of the same user/script action? If you add regular expressions for +both you would end up with two failures for a single action. +Therefore, select the most appropriate log message and document the other log +message) with a test case not to match it and a description as to why you chose +one over another. + +With the selected log lines consider what action has caused those log +messages and whether they could have been generated by accident? Could +the log message be occurring due to the first step towards the application +asking for authentication? Could the log messages occur often? If some of +these are true make a note of this in the jail.conf example that you provide. + +Samples: + +It is important to include log file samples so any future change in the regular +expression will still work with the log lines you have identified. + +The sample log messages are provided in a file under testcases/files/logs/ +named identically as the corresponding filter (but without .conf extension). +Each log line should be preceded by a line with failJSON metadata (so the logs +lines are tested in the test suite) directly above the log line. If there is +any specific information about the log message, such as version or an +application configuration option that is needed for the message to occur, +include this in a comment (line beginning with #) above the failJSON metadata. + +Log samples should include only one, definitely not more than 3, examples of +log messages of the same form. If log messages are different in different +versions of the application log messages that show this are encouraged. + +Also attempt to inject an IP into the application (e.g. by specifying +it as a username) so that Fail2Ban possibly detects the IP +from user input rather than the true origin. See the Filter Security section +and the top example in testcases/files/logs/apache-auth as to how to do this. +One you have discovered that this is possible, correct the regex so it doesn't +match and provide this as a test case with "match": false (see failJSON below). + +If the mechanism to create the log message isn't obvious provide a +configuration and/or sample scripts testcases/files/config/{filtername} and +reference these in the comments above the log line. + +FailJSON metadata: + +A failJSON metadata is a comment immediately above the log message. It will +look like: + +# failJSON: { "time": "2013-06-10T10:10:59", "match": true , "host": "93.184.216.119" } + +Time should match the time of the log message. It is in a specific format of +Year-Month-Day'T'Hour:minute:Second. If your log message does not include a +year, like the example below, the year should be listed as 2005, if before Sun +Aug 14 10am UTC, and 2004 if afterwards. Here is an example failJSON +line preceding a sample log line: + +# failJSON: { "time": "2005-03-24T15:25:51", "match": true , "host": "198.51.100.87" } +Mar 24 15:25:51 buffalo1 dropbear[4092]: bad password attempt for 'root' from 198.51.100.87:5543 + +The "host" in failJSON should contain the IP or domain that should be blocked. + +For long lines that you do not want to be matched (e.g. from log injection +attacks) and any log lines to be excluded (see "Cause" section above), set +"match": false in the failJSON and describe the reason in the comment above. + +After developing regexes, the following command will test all failJSON metadata +against the log lines in all sample log files + +./fail2ban-testcases testSampleRegex + +Developing Filter Regular Expressions +------------------------------------- + +Date/Time: + +At the moment, Fail2Ban depends on log lines to have time stamps. That is why +before starting to develop failregex, check if your log line format known to +Fail2Ban. Copy the time component from the log line and append an IP address to +test with following command: + +./fail2ban-regex "2013-09-19 02:46:12 1.2.3.4" "" + +Output of such command should contain something like: + +Date template hits: +|- [# of hits] date format +| [1] Year-Month-Day Hour:Minute:Second + +Ensure that the template description matches time/date elements in your log line +time stamp. If there is no matched format then date template needs to be added +to server/datedetector.py. Ensure that a new template is added in the order +that more specific matches occur first and that there is no confusion between a +Day and a Month. + +Filter file: + +The filter is specified in a config/filter.d/{filtername}.conf file. Filter file +can have sections INCLUDES (optional) and Definition as follows: + +[INCLUDES] + +before = common.conf + +after = filtername.local + +[Definition] + +failregex = .... + +ignoreregex = .... + +This is also documented in the man page jail.conf (section 5). Other definitions +can be added to make failregex's more readable and maintainable to be used +through string Interpolations (see http://docs.python.org/2.7/library/configparser.html) + + +General rules: + +Use "before" if you need to include a common set of rules, like syslog or if +there is a common set of regexes for multiple filters. + +Use "after" if you wish to allow the user to overwrite a set of customisations +of the current filter. This file doesn't need to exist. + +Try to avoid using ignoreregex mainly for performance reasons. The case when you +would use it is if in trying to avoid using it, you end up with an unreadable +failregex. + +Syslog: + +If your application logs to syslog you can take advantage of log line prefix +definitions present in common.conf. So as a base use: + +[INCLUDES] + +before = common.conf + +[Definition] + +_daemon = app + +failregex = ^%(__prefix_line)s + +In this example common.conf defines __prefix_line which also contains the +_daemon name (in syslog terms the service) you have just specified. _daemon +can also be a regex. + +For example, to capture following line _daemon should be set to "dovecot" + +Dec 12 11:19:11 dunnart dovecot: pop3-login: Aborted login (tried to use disabled plaintext auth): rip=190.210.136.21, lip=113.212.99.193 + +and then ^%(__prefix_line)s would match "Dec 12 11:19:11 dunnart dovecot: +". Note it matches the trailing space(s) as well. + +Substitutions (AKA string interpolations): + +We have used string interpolations in above examples. They are useful for +making the regexes more readable, reuse generic patterns in multiple failregex +lines, and also to refer definition of regex parts to specific filters or even +to the user. General principle is that value of a _name variable replaces +occurrences of %(_name)s within the same section or anywhere in the config file +if defined in [DEFAULT] section. + +Regular Expressions: + +Regular expressions (failregex, ignoreregex) assume that the date/time has been +removed from the log line (this is just how fail2ban works internally ATM). + +If the format is like ' error 1.2.3.4 is evil' then you need to match +the < at the start so regex should be similar to '^<> is evil$' using + where the IP/domain name appears in the log line. + +The following general rules apply to regular expressions: + +* ensure regexes start with a ^ and are as restrictive as possible. E.g. do not + use .* if \d+ is sufficient; +* use functionality of Python regexes defined in the standard Python re library + http://docs.python.org/2/library/re.html; +* make regular expressions readable (as much as possible). E.g. + (?:...) represents a non-capturing regex but (...) is more readable, thus + preferred. + +If you have only a basic knowledge of regular repressions we advise to read +http://docs.python.org/2/library/re.html first. It doesn't take long and would +remind you e.g. which characters you need to escape and which you don't. + +Developing/testing a regex: + +You can develop a regex in a file or using command line depending on your +preference. You can also use samples you have already created in the test cases +or test them one at a time. + +The general tool for testing Fail2Ban regexes is fail2ban-regex. To see how to +use it run: + +./fail2ban-regex --help + +Take note of -l heavydebug / -l debug and -v as they might be very useful. + +TIP: Take a look at the source code of the application you are developing + failregex for. You may see optional or extra log messages, or parts there + of, that need to form part of your regex. It may also reveal how some + parts are constrained and different formats depending on configuration or + less common usages. + +TIP: For looking through source code - http://sourcecodebrowser.com/ . It has + call graphs and can browse different versions. + +TIP: Some applications log spaces at the end. If you are not sure add \s*$ as + the end part of the regex. + +If your regex is not matching, http://www.debuggex.com/?flavor=python can help +to tune it: + +* use regex from the ./fail2ban-regex output (to ensure all substitutions are +done) and replace with (?&.ipv4). Make sure that regex type set to +Python; +* for the test data put your log output with the time removed; +- when you have fixed the regex put it back into your filter file. + +Please spread the good word about debuggex - Serge Toarca is kindly continuing +its free availability to Open Source developers. + +Finishing up: + +If you've added a new filter, add a new entry in config/jail.conf. The theory +here is that a user will create a jail.local with [filtername]\nenable=true to +enable your jail. + +So more specifically in the [filter] section in jail.conf: +* ensure that you have "enabled = false" (users will enable as needed); +* use "filter =" set to your filter name; +* use a typical action to disable ports associated with the application; +* set "logpath" to the usual location of application log file; +* if the default findtime or bantime isn't appropriate to the filter, specify + more appropriate choices (possibly with a brief comment line). + +Submit github pull request (See "Pull Requests" above) for +github.com/fail2ban/fail2ban containing your great work. Filter Security --------------- -Poor filter regular expressions are suseptable to DoS attacks. +Poor filter regular expressions are susceptible to DoS attacks. -When a remote user has the ability to introduce text that will match the -filter regex, such that the inserted text matches the part, they have the +When a remote user has the ability to introduce text that would match filter's +failregex, while matching inserted text to the part, they have the ability to deny any host they choose. -So the part must be anchored on text generated by the application, and not -the user, to a sufficient extent that the user cannot insert the entire text. +So the part must be anchored on text generated by the application, and +not the user, to a extent sufficient to prevent user inserting the entire text +matching this or any other failregex. -Filters are matched against the log line with their date removed. - -Ideally filter regex should anchor to the beginning and end of the log line -however as more applications log at the beginning than the end, achoring the +Ideally filter regex should anchor at the beginning and at the end of log line. +However as more applications log at the beginning than the end, anchoring the beginning is more important. If the log file used by the application is shared -with other applications, like system logs, ensure the other application that -use that log file do not log user generated text at the beginning of the line, -or, if they do, ensure the regexs of the filter are sufficient to mitigate the -risk of insertion. +with other applications, like system logs, ensure the other application that use +that log file do not log user generated text at the beginning of the line, or, +if they do, ensure the regexes of the filter are sufficient to mitigate the risk +of insertion. -When creating a regex that extends back to the begining remember the date part -has been removed within fail2ban so theres no need to match that. If the format -is like ' error 1.2.3.4 is evil' then you will need to match the < at -the start so here the regex would start like '^<> is evil$'. - -Some applications log spaces at the end. If you're not sure add \s*$ as the -end part of the regex. Examples of poor filters ------------------------ @@ -96,13 +354,13 @@ We make a failregex Now think evil. The user does the command 'blah from 1.2.3.44' -The program diliently logs: +The program diligently logs: Apr-07-13 07:08:36 Invalid command blah from 1.2.3.44 from 1.2.3.4 And fail2ban matches 1.2.3.44 as the IP that it ban. A DoS attack was successful. -The fix here is that the command can be anything so .* is approprate. +The fix here is that the command can be anything so .* is appropriate. ^Invalid command .* from @@ -121,10 +379,10 @@ banned. 2. Filter regex can match other user injected data -From the apache vulnerability CVE-2013-2178 +From the Apache vulnerability CVE-2013-2178 ( original ref: https://vndh.net/note:fail2ban-089-denial-service ). -An example bad regex for apache: +An example bad regex for Apache: failregex = [[]client []] user .* not found @@ -140,10 +398,10 @@ Now the log line will be: As this log line doesn't match other expressions hence it matches the above regex and blocks 192.168.33.1 as a denial of service from the HTTP requester. -3. Applicaiton generates two identical log messages with different meanings +3. Application generates two identical log messages with different meanings If the application generates the following two messages under different -circmstances: +circumstances: client : authentication failed client : authentication failed @@ -179,7 +437,7 @@ coverage run bin/fail2ban-testcases coverage html Then look at htmlcov/index.html and see how much coverage your test cases -exert over the codebase. Full coverage is a good thing however it may not be +exert over the code base. Full coverage is a good thing however it may not be complete. Try to ensure tests cover as many independent paths through the code. @@ -270,7 +528,7 @@ Design Fail2Ban was initially developed with Python 2.3 (IIRC). It should still be compatible with Python 2.4 and such compatibility assurance makes code ... old-fashioned in many places (RF-Note). In 0.7 the -design went through major refactoring into client/server, +design went through major re-factoring into client/server, a-thread-per-jail design which made it a bit difficult to follow. Below you can find a sketchy description of the main components of the system to orient yourself better. @@ -381,7 +639,7 @@ one way or another provide except FailManagerEmpty: self.failManager.cleanup(MyTime.time()) -thus channeling "ban tickets" from their failManager to the +thus channelling "ban tickets" from their failManager to the corresponding jail. action.py @@ -411,6 +669,61 @@ Releasing * https://bugzilla.redhat.com/buglist.cgi?query_format=advanced&bug_status=NEW&bug_status=ASSIGNED&component=fail2ban&classification=Red%20Hat&classification=Fedora * http://www.freebsd.org/cgi/query-pr-summary.cgi?text=fail2ban +# Make sure the tests pass + + ./fail2ban-testcases-all + +# Ensure the version is correct + + in: + * ./common/version.py + * top of ChangeLog + * README.md + +# Ensure the MANIFEST is complete + +Run: + + python setup.py sdist + +Look for errors like: + 'testcases/files/logs/mysqld.log' not a regular file -- skipping + +Which indicates that testcases/files/logs/mysqld.log has been moved or is a directory + + tar -C /tmp -jxf dist/fail2ban-0.9.0.tar.bz2 + +# clean up current direcory + + diff -rul --exclude \*.pyc . /tmp/fail2ban-0.9.0/ + + # Only differences should be files that you don't want distributed. + +# Ensure the tests work from the tarball + + cd /tmp/fail2ban-0.9.0/ && ./fail2ban-testcases-all + +# Add/finalize the corresponding entry in the ChangeLog + + To generate a list of committers use e.g. + + git shortlog -sn 0.8.10.. | sed -e 's,^[ 0-9\t]*,,g' | tr '\n' '\|' | sed -e 's:|:, :g' + + Ensure the top of the ChangeLog has the right version and current date. + + Ensure the top entry of the ChangeLog has the right version and current date. + +# Update man pages + + (cd man ; ./generate-man ) + git commit -m 'DOC/ENH: update man pages for release' man/* + +# Prepare source and rpm binary distributions + + python setup.py sdist + python setup.py bdist_rpm + python setup.py upload + # Provide a release sample to distributors * Debian: Yaroslav Halchenko @@ -425,41 +738,31 @@ Releasing https://build.opensuse.org/package/users?package=fail2ban&project=openSUSE%3AFactory * Mac Ports: @Malbrouck on github (gh-49) https://trac.macports.org/browser/trunk/dports/security/fail2ban/Portfile + An potentially to the fail2ban-users directory. # Wait for feedback from distributors -# Ensure the version is correct in ./common/version.py +# Prepare a release notice https://github.com/fail2ban/fail2ban/releases/new -# Add/finalize the corresponding entry in the ChangeLog + Upload the source/binaries from the dist directory and tag the release using the URL - To generate a list of committers use e.g. - - git shortlog -sn 0.8.8.. | sed -e 's,^[ 0-9\t]*,,g' | tr '\n' '\|' | sed -e 's:|:, :g' - - Ensure the top of the ChangeLog has the right version and current date. - - Ensure the top entry of the ChangeLog has the right version and current date. - -# Update man pages - - (cd man ; ./generate-man ) - git commit -m 'update man pages for release' man/* - -# Make sure the tests pass - - ./fail2ban-testcases-all - -# Prepare/upload source and rpm binary distributions - - python setup.py check - python setup.py sdist - python setup.py bdist_rpm - python setup.py upload +# Upload source/binaries to sourceforge http://sourceforge.net/projects/fail2ban/ # Run the following and update the wiki with output: - python -c 'import fail2ban.protocol; fail2ban.protocol.printWiki()' + page: http://www.fail2ban.org/wiki/index.php/Commands + +* Update: + http://www.fail2ban.org/wiki/index.php/Downloads + http://www.fail2ban.org/wiki/index.php/ChangeLog + http://www.fail2ban.org/wiki/index.php/Requirements (Check requirement) + http://www.fail2ban.org/wiki/index.php/Main_Page (Add to News) + http://www.fail2ban.org/wiki/index.php/Features + +* See if any filters are upgraded: + http://www.fail2ban.org/wiki/index.php/Special:AllPages + # Email users and development list of release # notify distributors @@ -469,15 +772,17 @@ Post Release Add the following to the top of the ChangeLog -ver. 0.8.12 (2013/XX/XXX) - wanna-be-released +ver. 0.9.1 (2014/XX/XXX) - wanna-be-released ----------- - Fixes: - + - New Features: - + - Enhancements: - + +Alter the git shortlog command in the previous section to refer to the just +released version. and adjust common/version.py to carry .dev suffix to signal a version under development. diff --git a/MANIFEST b/MANIFEST index 1c2b5779..f45c66aa 100644 --- a/MANIFEST +++ b/MANIFEST @@ -91,6 +91,14 @@ fail2ban/exceptions.py fail2ban/helpers.py fail2ban/version.py fail2ban/protocol.py +fail2ban-client +fail2ban-server +fail2ban-testcases +fail2ban-regex +fail2ban-testcases-all +setup.py +setup.cfg +kill-server config/jail.conf config/filter.d/common.conf config/filter.d/apache-auth.conf @@ -111,7 +119,7 @@ config/filter.d/pure-ftpd.conf config/filter.d/qmail.conf config/filter.d/pam-generic.conf config/filter.d/php-url-fopen.conf -config/filter.d/sasl.conf +config/filter.d/postfix-sasl.conf config/filter.d/sieve.conf config/filter.d/sshd.conf config/filter.d/sshd-ddos.conf @@ -126,10 +134,24 @@ config/filter.d/lighttpd-auth.conf config/filter.d/recidive.conf config/filter.d/roundcube-auth.conf config/filter.d/assp.conf -config/filter.d/mysqld-auth.conf config/filter.d/sogo-auth.conf +config/filter.d/mysqld-auth.conf +config/filter.d/selinux-common.conf +config/filter.d/selinux-ssh.conf +config/filter.d/3proxy.conf +config/filter.d/apache-common.conf +config/filter.d/exim-common.conf +config/filter.d/exim-spam.conf +config/filter.d/perdition.conf +config/filter.d/uwimap-auth.conf +config/action.d/apf.conf +config/action.d/osx-afctl.conf +config/action.d/osx-ipfw.conf +config/action.d/sendmail-common.conf config/action.d/bsd-ipfw.conf config/action.d/dummy.conf +config/action.d/firewall-cmd-direct-new.conf +config/action.d/iptables-ipset-proto6-allports.conf config/action.d/iptables-blocktype.conf config/action.d/iptables-ipset-proto4.conf config/action.d/iptables-ipset-proto6.conf @@ -157,6 +179,7 @@ config/action.d/sendmail-whois.conf config/action.d/sendmail-whois-lines.conf config/action.d/shorewall.conf config/fail2ban.conf +doc/run-rootless.txt man/fail2ban-client.1 man/fail2ban.1 man/jail.conf.5 @@ -178,9 +201,8 @@ files/cacti/fail2ban_stats.sh files/cacti/cacti_host_template_fail2ban.xml files/cacti/README files/nagios/check_fail2ban -files/nagios/f2ban.txt +files/nagios/README files/bash-completion files/fail2ban-tmpfiles.conf files/fail2ban.service files/ipmasq-ZZZzzz_fail2ban.rul -files/nagios/README diff --git a/README.md b/README.md index 1b7cf718..129d24f7 100644 --- a/README.md +++ b/README.md @@ -31,8 +31,8 @@ Optional: To install, just do: - tar xvfj fail2ban-0.8.10.tar.bz2 - cd fail2ban-0.8.10 + tar xvfj fail2ban-0.8.11.tar.bz2 + cd fail2ban-0.8.11 python setup.py install This will install Fail2Ban into /usr/share/fail2ban. The executable scripts are diff --git a/THANKS b/THANKS index 2da6614f..e70ca9c9 100644 --- a/THANKS +++ b/THANKS @@ -1,21 +1,26 @@ -Fail2Ban is an open source project with many contributions from its -users community. Below is an alphabetically sorted partial list of the -contributors to the project. If you have been left off, please let us -know (preferably send a pull request on github with the "fix") and you -will be added +Fail2Ban is an open source project which was conceived and originally +developed by Cyril Jaquier until 2010. Since then Fail2Ban grew into +a community-driven project with many contributions from its users. +Below is an alphabetically sorted partial list of the contributors to +the project. If you have been left off, please let us know +(preferably send a pull request on github with the "fix") and you will +be added Adrien Clerc ache +Amir Caspi Andrey G. Grozin Andy Fragen Arturo 'Buanzo' Busleiman Axel Thimm +Beau Raines Bill Heaton Carlos Alberto Lopez Perez Christian Rauch Christophe Carles Christoph Haas Christos Psonis +Cyril Jaquier Daniel B. Cid Daniel Black David Nutter @@ -34,16 +39,19 @@ Joël Bertrand JP Espinosa Justin Shore Kévin Drapel +kjohnsonecl kojiro +Manuel Arostegui Ramirez Marcel Dopita Mark Edgington +Mark McKinstry Markus Hoffmann Marvin Rouge mEDI Мернов Георгий Michael C. Haller Michael Hanselmann -NickMunger +Nick Munger Patrick Börjesson Raphaël Marichez René Berber diff --git a/bin/fail2ban-client b/bin/fail2ban-client index 0f84a15c..6275d420 100755 --- a/bin/fail2ban-client +++ b/bin/fail2ban-client @@ -147,8 +147,9 @@ class Fail2banClient: if showRet: print beautifier.beautify(ret[1]) else: - logSys.debug("NOK: " + `ret[1].args`) - print beautifier.beautifyError(ret[1]) + logSys.error("NOK: " + `ret[1].args`) + if showRet: + print beautifier.beautifyError(ret[1]) return False except socket.error: if showRet: @@ -375,7 +376,10 @@ class Fail2banClient: if cmd == "help": self.dispUsage() elif not cmd == "": - self.__processCommand(shlex.split(cmd)) + try: + self.__processCommand(shlex.split(cmd)) + except Exception, e: + logSys.error(e) except (EOFError, KeyboardInterrupt): print return True diff --git a/bin/fail2ban-regex b/bin/fail2ban-regex index e54659f7..3661ba36 100755 --- a/bin/fail2ban-regex +++ b/bin/fail2ban-regex @@ -23,15 +23,13 @@ and bans the corresponding IP addresses using firewall rules. This tools can test regular expressions for "fail2ban". -Report bugs to https://github.com/fail2ban/fail2ban/issues """ __author__ = "Cyril Jaquier, Yaroslav Halchenko" __copyright__ = "Copyright (c) 2004-2008 Cyril Jaquier, 2012-2013 Yaroslav Halchenko" __license__ = "GPL" -import getopt, sys, time, logging, os, locale, shlex - +import getopt, sys, time, logging, os, locale, shlex, urllib from optparse import OptionParser, Option from ConfigParser import NoOptionError, NoSectionError, MissingSectionHeaderError @@ -51,6 +49,12 @@ from fail2ban.tests.utils import FormatterWithTraceBack # Gets the instance of the logger. logSys = logging.getLogger("fail2ban") +def debuggexURL(sample, regex): + q = urllib.urlencode({ 're': regex.replace('', '(?&.ipv4)'), + 'str': sample, + 'flavor': 'python' }) + return 'http://www.debuggex.com/?' + q + def shortstr(s, l=53): """Return shortened string """ @@ -103,6 +107,15 @@ REGEX: IGNOREREGEX: string a string representing an 'ignoreregex' filename path to a filter file (filter.d/sshd.conf) + +Copyright (c) 2004-2008 Cyril Jaquier, 2008- Fail2Ban Contributors +Copyright of modifications held by their respective authors. +Licensed under the GNU General Public License v2 (GPL). + +Written by Cyril Jaquier . +Many contributions by Yaroslav O. Halchenko and Steven Hiscocks. + +Report bugs to https://github.com/fail2ban/fail2ban/issues """, version="%prog " + version) @@ -116,14 +129,15 @@ IGNOREREGEX: Option("-m", "--journalmatch", help="journalctl style matches overriding filter file. " "\"systemd-journal\" only"), - Option("-v", "--verbose", action='store_true', - help="Be verbose in output"), - Option('-l', "--log-level", type="choice", dest="log_level", choices=('heavydebug', 'debug', 'info', 'warning', 'error', 'fatal'), default=None, help="Log level for the Fail2Ban logger to use"), + Option("-v", "--verbose", action='store_true', + help="Be verbose in output"), + Option("-D", "--debuggex", action='store_true', + help="Produce debuggex.com urls for debugging there"), Option("--print-all-missed", action='store_true', help="Either to print all missed lines"), Option("--print-all-ignored", action='store_true', @@ -132,7 +146,6 @@ IGNOREREGEX: help="Enrich log-messages with compressed tracebacks"), Option("--full-traceback", action='store_true', help="Either to make the tracebacks full, not compressed (as by default)"), - ]) return p @@ -171,7 +184,9 @@ class LineStats(object): def __init__(self): self.tested = self.matched = 0 self.missed_lines = [] + self.missed_lines_timeextracted = [] self.ignored_lines = [] + self.ignored_lines_timeextracted = [] def __str__(self): return "%(tested)d lines, %(ignored)d ignored, %(matched)d matched, %(missed)d missed" % self @@ -195,6 +210,7 @@ class Fail2banRegex(object): def __init__(self, opts): self._verbose = opts.verbose + self._debuggex = opts.debuggex self._print_all_missed = opts.print_all_missed self._print_all_ignored = opts.print_all_ignored self._maxlines_set = False # so we allow to override maxlines in cmdline @@ -306,7 +322,7 @@ class Fail2banRegex(object): orgLineBuffer = self._filter._Filter__lineBuffer fullBuffer = len(orgLineBuffer) >= self._filter.getMaxLines() try: - ret = self._filter.processLine(line, checkAllRegex=True) + line, ret = self._filter.processLine(line, checkAllRegex=True) for match in ret: # Append True/False flag depending if line was matched by # more than one regex @@ -318,13 +334,13 @@ class Fail2banRegex(object): print e return False except IndexError: - print "Sorry, but no found in regex" + print "Sorry, but no found in regex" return False for bufLine in orgLineBuffer[int(fullBuffer):]: if bufLine not in self._filter._Filter__lineBuffer: if self.removeMissedLine(bufLine): self._line_stats.matched += 1 - return len(ret) > 0 + return line, ret def removeMissedLine(self, line): """Remove `line` from missed lines, by comparing without time match""" @@ -350,27 +366,49 @@ class Fail2banRegex(object): # skip comment and empty lines continue is_ignored = fail2banRegex.testIgnoreRegex(line) + line_datetimestripped, ret = fail2banRegex.testRegex(line) + if is_ignored: self._line_stats.ignored_lines.append(line) + self._line_stats.ignored_lines_timeextracted.append(line_datetimestripped) - if fail2banRegex.testRegex(line): + if len(ret) > 0: assert(not is_ignored) self._line_stats.matched += 1 else: if not is_ignored: self._line_stats.missed_lines.append(line) + self._line_stats.missed_lines_timeextracted.append(line_datetimestripped) self._line_stats.tested += 1 if line_no % 10 == 0: self._filter.dateDetector.sortTemplate() + + def printLines(self, ltype): lstats = self._line_stats assert(len(lstats.missed_lines) == lstats.tested - (lstats.matched + lstats.ignored)) l = lstats[ltype + '_lines'] if len(l): header = "%s line(s):" % (ltype.capitalize(),) - if len(l) < 20 or getattr(self, '_print_all_' + ltype): + if self._debuggex: + if ltype == 'missed': + regexlist = self._failregex + else: + regexlist = self._ignoreregex + l = lstats[ltype + '_lines_timeextracted'] + lines = len(l)*len(regexlist) + if lines < 20 or getattr(self, '_print_all_' + ltype): + ans = [[]] + for arg in [l, regexlist]: + ans = [ x + [y] for x in ans for y in arg ] + b = map(lambda a: a[0] + ' | ' + a[1].getFailRegex() + ' | ' + debuggexURL(a[0], a[1].getFailRegex()), ans) + pprint_list([x.rstrip() for x in b], header) + else: + print "%s: too many to print. Use --print-all-%s " \ + "to print all %d lines" % (header, ltype, lines) + elif len(l) < 20 or getattr(self, '_print_all_' + ltype): pprint_list([x.rstrip() for x in l], header) else: print "%s: too many to print. Use --print-all-%s " \ diff --git a/config/action.d/apf.conf b/config/action.d/apf.conf new file mode 100644 index 00000000..9af3066d --- /dev/null +++ b/config/action.d/apf.conf @@ -0,0 +1,43 @@ +# Fail2Ban configuration file +# +# Author: Mark McKinstry +# +[Definition] + +# Option: actionstart +# Notes.: command executed once at the start of Fail2Ban. +# Values: CMD +# +actionstart = + +# Option: actionstop +# Notes.: command executed once at the end of Fail2Ban +# Values: CMD +# +actionstop = + +# Option: actioncheck +# Notes.: command executed once before each actionban command +# Values: CMD +# +actioncheck = + +# Option: actionban +# Notes.: command executed when banning an IP. Take care that the +# command is executed with Fail2Ban user rights. +# Tags: IP address +# number of failures +#