diff --git a/ChangeLog b/ChangeLog index 230ee10d..27e598ef 100644 --- a/ChangeLog +++ b/ChangeLog @@ -4,9 +4,172 @@ |_| \__,_|_|_/___|_.__/\__,_|_||_| ================================================================================ -Fail2Ban (version 0.8.10) 2013/06/12 +Fail2Ban (version 0.8.11.pre1) 2013/10/30 ================================================================================ +ver. 0.8.11 (2013/11/XXX) - loves-unittests-and-tight-DoS-free-filter-regexes +----------- + +In light of CVE-2013-2178 that triggered our last release we have put +a significant effort into tightening all of the regexs of our filters +to avoid another similar vulnerability. All filters have been updated +and some to catch more login/authentication failures and to support +for newer application versions. There are test cases for most log +cases of failures now. + +As usual, if you have other examples that demonstrate that a filter is +insufficient, or if we have inadvertently introduced a regression, +please provide us with example log lines on the github issue tracker +http://github.com/fail2ban/fail2ban/issues and NOT on a random blog in +some obscure corner of the Internet. + +- IMPORTANT incompatible changes: + Filter name changes: + * 'lighttpd-fastcgi' filter has been renamed to 'suhosin' + * 'sasl' has been renamed to 'postfix-sasl' + * 'exim' spam catching failregexes was split out into 'exim-spam' + These changes will require changing jail.{conf,local} if any of + those filters were used. + +- Fixes: + Daniel Black & Marcel Dopita + * filter.d/apache-auth -- fixed and apache auth samples provide. Closes gh-286 + Yaroslav Halchenko + * filter.d/common.conf -- make colon after [daemon] optional. Closes gh-267 + * filter.d/apache-common.conf -- support apache 2.4 more detailed error + log format. Closes gh-268 + * Backends changes detection and parsing. Close gh-223 and gh-103: + - Polling backend: detect changes in the files not only based on + mtime, but also on the size and inode. It should allow for + better detection of changes and log rotations on busy servers, + older python 2.4, and file systems with precision of mtime only + up to a second (e.g. ext3). + - All backends, possible race condition: do not read from a file + initially reported empty. Originally could have lead to + accounting for detected log lines multiple times. + - Do not crash if executing a command in fail2ban-client interactive + mode has failed (e.g. due to incorrect syntax). Closes gh-353 + Daniel Black & Мернов Георгий + * filter.d/dovecot.conf -- Fix when no TLS enabled - line doesn't end in , + Daniel Black & Georgiy Mernov & ftoppi & Мернов Георгий + * filter.d/exim.conf -- regex hardening and extra failure examples in + sample logs + * filter.d/named-refused.conf - BIND 9.9.3 regex changes + Daniel Black & Sebastian Arcus + * filter.d/asterisk -- more regexes + Daniel Black + * action.d/hostsdeny -- NOTE: new dependancy 'ed'. Switched to use 'ed' across + all platforms to ensure permissions are the same before and after a ban. + Closes gh-266. hostsdeny supports daemon_list now too. + * action.d/bsd-ipfw - action option unsed. Change blocktype to port unreach + instead of deny for consistancy. + * filter.d/dovecot - added to support different dovecot failure + "..disallowed plaintext auth". Closes Debian bug #709324 + * filter.d/roundcube-auth - timezone offset can be positive or negative + * action.d/bsd-ipfw - action option unsed. Fixed to blocktype for + consistency. default to port unreach instead of deny + * filter.d/dropbear - fix regexs to match standard dropbear and the patched + http://www.unchartedbackwaters.co.uk/files/dropbear/dropbear-0.52.patch + and add PAM is it in dropbear-2013.60 source code. + * filter.d/{asterisk,assp,dovecot,proftpd}.conf -- regex hardening + and extra failure examples in sample logs + * filter.d/apache-auth - added expressions for mod_authz, mod_auth and + mod_auth_digest failures. + * filter.d/recidive -- support f2b syslog target and anchor regex at start + * filter.d/mysqld-auth.conf - mysql can use syslog + * filter.d/sshd - regex enhancements to support openssh-6.3. Closes Debian + bug #722970. Thanks Colin Watson for the regex analysis. + * filter.d/wuftpd - regex enhancements to support pam and wuftpd. Closes + Debian bug #665925 + Rolf Fokkens + * action.d/dshield.conf and complain.conf -- reorder mailx arguments. + https://bugzilla.redhat.com/show_bug.cgi?id=998020 + John Doe (ache) + * action.d/bsd-ipfw.conf - invert actionstop logic to make exist status 0. + Closes gh-343. + JP Espinosa (Reviewed by O.Poplawski) + * files/redhat-initd - rewritten to use stock init.d functions thus + avoiding problems with getpid. Also $network and iptables moved + to Should- rc init fields + Rick Mellor + * filter.d/vsftp - fix capture with tty=ftp + +- New Features: + Edgar Hoch + * action.d/firewall-cmd-direct-new.conf - action for firewalld + from https://bugzilla.redhat.com/show_bug.cgi?id=979622 + NOTE: requires firewalld-0.3.8+ + Andy Fragen and Daniel Black + * filter.d/osx-ipfw.conf - ipfw action for OSX based on random rule + numbers. + Anonymous: + * action.d/osx-afctl - an action based on afctl for osx + Daniel Black & ykimon + * filter.d/3proxy.conf -- filter added + * fail2ban-regex - now generates http://www.debuggex.com urls for debugging + regular expressions with the -D parameter. + Daniel Black + * filter.d/exim-spam.conf -- a splitout of exim's spam regexes + with additions for greater control over filtering spam. + * add date expression for apache-2.4 - milliseconds + * filter.d/nginx-http-auth -- filter added for http basic authentication + failures in nginx. Partially fulfills gh-405. + Christophe Carles & Daniel Black + * filter.d/perdition.conf -- filter added + Mark McKinstry + * action.d/apf.conf - add action for Advanced Policy Firewall (apf) + Amir Caspi and kjohnsonecl + * filter.d/uwimap-auth - filter for uwimap-auth IMAP/POP server + Steven Hiscocks and Daniel Black + * filter.d/selinux-{common,ssh} -- add SELinux date and ssh filter + +- Enhancements: + François Boulogne and Frédéric + * filter.d/lighttpd - auth regexs for lighttpd-1.4.31 + Daniel Black + * reorder parsing of jail.conf, jail.d/*.conf, jail.local, jail.d/*.local + and likewise for fail2ban.{conf|local|d/*.conf|d/*.local}. Closes gh-392 + * jail.conf now has asterisk jail - no need for asterisk-tcp and + asterisk-udp. Users should replace existing jails with asterisk to + reduce duplicate parsing of the asterisk log file. + * filter.d/{suhosin,pam-generic,gssftpd,sogo-auth,webmin}- regex anchor at + start + * filter.d/vsftpd - anchored regex at start. disable old pam format regex + * filter.d/pam-generic - added syslog prefix. Disabled support for + linux-pam before version 0.99.2.0 (2005) + * filter.d/postfix-sasl - renamed from sasl, anchor at start and base on + syslog + * filter.d/qmail - rewrote regex to anchor at start. Added regex for + another "in the wild" patch to rblsmtp. + Yaroslav Halchenko + * fail2ban-regex -- refactored to provide more details (missing and + ignored lines, control over logging, etc) while maintaining look&feel + * fail2ban-client -- log to standard error. Closes gh-264 + * Fail to configure if not a single log file was found for an + enabled jail. Closes gh-63 + * is now enforced to end with an alphanumeric + * filter.d/roundcube-auth.conf -- anchored version + * date matching - for standard asctime formats prefer more detailed + first (thus use year if available) + * files/gen_badbots was added and filter.d/apache-badbots.conf was + regenerated to get updated (although now still an old) list of + "bad" bots + Alexander Dietrich + * action.d/sendmail-common.conf -- added common sendmail settings file + and made the sender display name configurable + Steven Hiscocks + * filter.d/dovecot - Addition of session, time values and possible blank + user + Zurd and Daniel Black + * filter/named-refused - added refused on zone transfer + * filter.d/{courier{login,smtp},proftpd,sieve,wuftpd,xinetd} - General + regex impovements + Zurd + * filter.d/postfix - add filter for VRFY failures. Closes gh-322. + Orion Poplawski + * fail2ban.d/ and jail.d/ directories are added to etc/fail2ban to facilitate + their use + ver. 0.8.10 (2013/06/12) - wanna-be-secure ----------- diff --git a/DEVELOP b/DEVELOP index 00b7d6d5..9a3be94b 100644 --- a/DEVELOP +++ b/DEVELOP @@ -1,6 +1,6 @@ - __ _ _ ___ _ - / _|__ _(_) |_ ) |__ __ _ _ _ - | _/ _` | | |/ /| '_ \/ _` | ' \ + __ _ _ ___ _ + / _|__ _(_) |_ ) |__ __ _ _ _ + | _/ _` | | |/ /| '_ \/ _` | ' \ |_| \__,_|_|_/___|_.__/\__,_|_||_| ================================================================================ @@ -26,7 +26,7 @@ Pull Requests When submitting pull requests on GitHub we ask you to: * Clearly describe the problem you're solving; -* Don't introduce regressions that will make it hard for systems adminstrators +* Don't introduce regressions that will make it hard for systems administrators to update; * If adding a major feature rebase your changes on master and get to a single commit; * Include test cases (see below); @@ -37,12 +37,383 @@ When submitting pull requests on GitHub we ask you to: Filters ======= -* Include sample logs with 1.2.3.4 used for IP addresses and - example.com/example.org used for DNS names -* Ensure ./fail2ban-regex testcases/files/logs/{samplelog} config/filter.d/{filter}.conf - has matches for EVERY regex -* Ensure regexs end with a $ and are restrictive as possible. E.g. not .* if - [0-9]+ is sufficient +Filters are tricky. They need to: +* work with a variety of the versions of the software that generates the logs; +* work with the range of logging configuration options available in the + software; +* work with multiple operating systems; +* not make assumptions about the log format in excess of the software + (e.g. do not assume a username doesn't contain spaces and use \S+ unless + you've checked the source code); +* account for how future versions of the software will log messages + (e.g. guess what would happen to the log message if different authentication + types are added); +* not be susceptible to DoS vulnerabilities (see Filter Security below); and +* match intended log lines only. + +Please follow the steps from Filter Test Cases to Developing Filter Regular +Expressions and submit a GitHub pull request (PR) afterwards. If you get stuck, +you can push your unfinished changes and still submit a PR -- describe +what you have done, what is the hurdle, and we'll attempt to help (PR +will be automagically updated with future commits you would push to +complete it). + +Filter test cases +----------------- + +Purpose: + +Start by finding the log messages that the application generates related to +some form of authentication failure. If you are adding to an existing filter +think about whether the log messages are of a similar importance and purpose +to the existing filter. If you were a user of Fail2Ban, and did a package +update of Fail2Ban that started matching new log messages, would anything +unexpected happen? Would the bantime/findtime for the jail be appropriate for +the new log messages? If it doesn't, perhaps it needs to be in a separate +filter definition, for example like exim filter aims at authentication failures +and exim-spam at log messages related to spam. + +Even if it is a new filter you may consider separating the log messages into +different filters based on purpose. + +Cause: + +Are some of the log lines a result of the same action? For example, is a PAM +failure log message, followed by an application specific failure message the +result of the same user/script action? If you add regular expressions for +both you would end up with two failures for a single action. +Therefore, select the most appropriate log message and document the other log +message) with a test case not to match it and a description as to why you chose +one over another. + +With the selected log lines consider what action has caused those log +messages and whether they could have been generated by accident? Could +the log message be occurring due to the first step towards the application +asking for authentication? Could the log messages occur often? If some of +these are true make a note of this in the jail.conf example that you provide. + +Samples: + +It is important to include log file samples so any future change in the regular +expression will still work with the log lines you have identified. + +The sample log messages are provided in a file under testcases/files/logs/ +named identically as the corresponding filter (but without .conf extension). +Each log line should be preceded by a line with failJSON metadata (so the logs +lines are tested in the test suite) directly above the log line. If there is +any specific information about the log message, such as version or an +application configuration option that is needed for the message to occur, +include this in a comment (line beginning with #) above the failJSON metadata. + +Log samples should include only one, definitely not more than 3, examples of +log messages of the same form. If log messages are different in different +versions of the application log messages that show this are encouraged. + +Also attempt to inject an IP into the application (e.g. by specifying +it as a username) so that Fail2Ban possibly detects the IP +from user input rather than the true origin. See the Filter Security section +and the top example in testcases/files/logs/apache-auth as to how to do this. +One you have discovered that this is possible, correct the regex so it doesn't +match and provide this as a test case with "match": false (see failJSON below). + +If the mechanism to create the log message isn't obvious provide a +configuration and/or sample scripts testcases/files/config/{filtername} and +reference these in the comments above the log line. + +FailJSON metadata: + +A failJSON metadata is a comment immediately above the log message. It will +look like: + +# failJSON: { "time": "2013-06-10T10:10:59", "match": true , "host": "93.184.216.119" } + +Time should match the time of the log message. It is in a specific format of +Year-Month-Day'T'Hour:minute:Second. If your log message does not include a +year, like the example below, the year should be listed as 2005, if before Sun +Aug 14 10am UTC, and 2004 if afterwards. Here is an example failJSON +line preceding a sample log line: + +# failJSON: { "time": "2005-03-24T15:25:51", "match": true , "host": "198.51.100.87" } +Mar 24 15:25:51 buffalo1 dropbear[4092]: bad password attempt for 'root' from 198.51.100.87:5543 + +The "host" in failJSON should contain the IP or domain that should be blocked. + +For long lines that you do not want to be matched (e.g. from log injection +attacks) and any log lines to be excluded (see "Cause" section above), set +"match": false in the failJSON and describe the reason in the comment above. + +After developing regexes, the following command will test all failJSON metadata +against the log lines in all sample log files + +./fail2ban-testcases testSampleRegex + +Developing Filter Regular Expressions +------------------------------------- + +Date/Time: + +At the moment, Fail2Ban depends on log lines to have time stamps. That is why +before starting to develop failregex, check if your log line format known to +Fail2Ban. Copy the time component from the log line and append an IP address to +test with following command: + +./fail2ban-regex "2013-09-19 02:46:12 1.2.3.4" "" + +Output of such command should contain something like: + +Date template hits: +|- [# of hits] date format +| [1] Year-Month-Day Hour:Minute:Second + +Ensure that the template description matches time/date elements in your log line +time stamp. If there is no matched format then date template needs to be added +to server/datedetector.py. Ensure that a new template is added in the order +that more specific matches occur first and that there is no confusion between a +Day and a Month. + +Filter file: + +The filter is specified in a config/filter.d/{filtername}.conf file. Filter file +can have sections INCLUDES (optional) and Definition as follows: + +[INCLUDES] + +before = common.conf + +after = filtername.local + +[Definition] + +failregex = .... + +ignoreregex = .... + +This is also documented in the man page jail.conf (section 5). Other definitions +can be added to make failregex's more readable and maintainable to be used +through string Interpolations (see http://docs.python.org/2.7/library/configparser.html) + + +General rules: + +Use "before" if you need to include a common set of rules, like syslog or if +there is a common set of regexes for multiple filters. + +Use "after" if you wish to allow the user to overwrite a set of customisations +of the current filter. This file doesn't need to exist. + +Try to avoid using ignoreregex mainly for performance reasons. The case when you +would use it is if in trying to avoid using it, you end up with an unreadable +failregex. + +Syslog: + +If your application logs to syslog you can take advantage of log line prefix +definitions present in common.conf. So as a base use: + +[INCLUDES] + +before = common.conf + +[Definition] + +_daemon = app + +failregex = ^%(__prefix_line)s + +In this example common.conf defines __prefix_line which also contains the +_daemon name (in syslog terms the service) you have just specified. _daemon +can also be a regex. + +For example, to capture following line _daemon should be set to "dovecot" + +Dec 12 11:19:11 dunnart dovecot: pop3-login: Aborted login (tried to use disabled plaintext auth): rip=190.210.136.21, lip=113.212.99.193 + +and then ^%(__prefix_line)s would match "Dec 12 11:19:11 dunnart dovecot: +". Note it matches the trailing space(s) as well. + +Substitutions (AKA string interpolations): + +We have used string interpolations in above examples. They are useful for +making the regexes more readable, reuse generic patterns in multiple failregex +lines, and also to refer definition of regex parts to specific filters or even +to the user. General principle is that value of a _name variable replaces +occurrences of %(_name)s within the same section or anywhere in the config file +if defined in [DEFAULT] section. + +Regular Expressions: + +Regular expressions (failregex, ignoreregex) assume that the date/time has been +removed from the log line (this is just how fail2ban works internally ATM). + +If the format is like ' error 1.2.3.4 is evil' then you need to match +the < at the start so regex should be similar to '^<> is evil$' using + where the IP/domain name appears in the log line. + +The following general rules apply to regular expressions: + +* ensure regexes start with a ^ and are as restrictive as possible. E.g. do not + use .* if \d+ is sufficient; +* use functionality of Python regexes defined in the standard Python re library + http://docs.python.org/2/library/re.html; +* make regular expressions readable (as much as possible). E.g. + (?:...) represents a non-capturing regex but (...) is more readable, thus + preferred. + +If you have only a basic knowledge of regular repressions we advise to read +http://docs.python.org/2/library/re.html first. It doesn't take long and would +remind you e.g. which characters you need to escape and which you don't. + +Developing/testing a regex: + +You can develop a regex in a file or using command line depending on your +preference. You can also use samples you have already created in the test cases +or test them one at a time. + +The general tool for testing Fail2Ban regexes is fail2ban-regex. To see how to +use it run: + +./fail2ban-regex --help + +Take note of -l heavydebug / -l debug and -v as they might be very useful. + +TIP: Take a look at the source code of the application you are developing + failregex for. You may see optional or extra log messages, or parts there + of, that need to form part of your regex. It may also reveal how some + parts are constrained and different formats depending on configuration or + less common usages. + +TIP: For looking through source code - http://sourcecodebrowser.com/ . It has + call graphs and can browse different versions. + +TIP: Some applications log spaces at the end. If you are not sure add \s*$ as + the end part of the regex. + +If your regex is not matching, http://www.debuggex.com/?flavor=python can help +to tune it: + +* use regex from the ./fail2ban-regex output (to ensure all substitutions are +done) and replace with (?&.ipv4). Make sure that regex type set to +Python; +* for the test data put your log output with the time removed; +- when you have fixed the regex put it back into your filter file. + +Please spread the good word about debuggex - Serge Toarca is kindly continuing +its free availability to Open Source developers. + +Finishing up: + +If you've added a new filter, add a new entry in config/jail.conf. The theory +here is that a user will create a jail.local with [filtername]\nenable=true to +enable your jail. + +So more specifically in the [filter] section in jail.conf: +* ensure that you have "enabled = false" (users will enable as needed); +* use "filter =" set to your filter name; +* use a typical action to disable ports associated with the application; +* set "logpath" to the usual location of application log file; +* if the default findtime or bantime isn't appropriate to the filter, specify + more appropriate choices (possibly with a brief comment line). + +Submit github pull request (See "Pull Requests" above) for +github.com/fail2ban/fail2ban containing your great work. + +Filter Security +--------------- + +Poor filter regular expressions are susceptible to DoS attacks. + +When a remote user has the ability to introduce text that would match filter's +failregex, while matching inserted text to the part, they have the +ability to deny any host they choose. + +So the part must be anchored on text generated by the application, and +not the user, to a extent sufficient to prevent user inserting the entire text +matching this or any other failregex. + +Ideally filter regex should anchor at the beginning and at the end of log line. +However as more applications log at the beginning than the end, anchoring the +beginning is more important. If the log file used by the application is shared +with other applications, like system logs, ensure the other application that use +that log file do not log user generated text at the beginning of the line, or, +if they do, ensure the regexes of the filter are sufficient to mitigate the risk +of insertion. + + +Examples of poor filters +------------------------ + +1. Too restrictive + +We find a log message: + + Apr-07-13 07:08:36 Invalid command fial2ban from 1.2.3.4 + +We make a failregex + + ^Invalid command \S+ from + +Now think evil. The user does the command 'blah from 1.2.3.44' + +The program diligently logs: + + Apr-07-13 07:08:36 Invalid command blah from 1.2.3.44 from 1.2.3.4 + +And fail2ban matches 1.2.3.44 as the IP that it ban. A DoS attack was successful. + +The fix here is that the command can be anything so .* is appropriate. + + ^Invalid command .* from + +Here the .* will match until the end of the string. Then realise it has more to +match, i.e. "from " and go back until it find this. Then it will ban +1.2.3.4 correctly. Since the is always at the end, end the regex with a $. + + ^Invalid command .* from $ + +Note if we'd just had the expression: + + ^Invalid command \S+ from $ + +Then provided the user put a space in their command they would have never been +banned. + +2. Filter regex can match other user injected data + +From the Apache vulnerability CVE-2013-2178 +( original ref: https://vndh.net/note:fail2ban-089-denial-service ). + +An example bad regex for Apache: + + failregex = [[]client []] user .* not found + +Since the user can do a get request on: + + GET /[client%20192.168.0.1]%20user%20root%20not%20found HTTP/1.0 +Host: remote.site + +Now the log line will be: + + [Sat Jun 01 02:17:42 2013] [error] [client 192.168.33.1] File does not exist: /srv/http/site/[client 192.168.0.1] user root not found + +As this log line doesn't match other expressions hence it matches the above +regex and blocks 192.168.33.1 as a denial of service from the HTTP requester. + +3. Application generates two identical log messages with different meanings + +If the application generates the following two messages under different +circumstances: + + client : authentication failed + client : authentication failed + + +Then it's obvious that a regex of "^client : authentication +failed$" will still cause problems if the user can trigger the second +log message with a of 123.1.1.1. + +Here there's nothing to do except request/change the application so it logs +messages differently. + Code Testing ============ @@ -66,7 +437,7 @@ coverage run fail2ban-testcases coverage html Then look at htmlcov/index.html and see how much coverage your test cases -exert over the codebase. Full coverage is a good thing however it may not be +exert over the code base. Full coverage is a good thing however it may not be complete. Try to ensure tests cover as many independent paths through the code. @@ -136,6 +507,14 @@ Use the following tags in your commit messages: Multiple tags could be joined with +, e.g. "BF+TST:". +Use the text "closes #333"/"resolves #333 "/"fixes #333" where 333 represents +an issue that is closed. Other text and details in link below. +See: https://help.github.com/articles/closing-issues-via-commit-messages + +If merge resulted in conflicts, clarify what changes were done to +corresponding files in the 'Conflicts:' section of the merge commit +message. See e.g. https://github.com/fail2ban/fail2ban/commit/f5a8a8ac + Adding Actions -------------- @@ -149,7 +528,7 @@ Design Fail2Ban was initially developed with Python 2.3 (IIRC). It should still be compatible with Python 2.4 and such compatibility assurance makes code ... old-fashioned in many places (RF-Note). In 0.7 the -design went through major refactoring into client/server, +design went through major re-factoring into client/server, a-thread-per-jail design which made it a bit difficult to follow. Below you can find a sketchy description of the main components of the system to orient yourself better. @@ -260,7 +639,7 @@ one way or another provide except FailManagerEmpty: self.failManager.cleanup(MyTime.time()) -thus channeling "ban tickets" from their failManager to the +thus channelling "ban tickets" from their failManager to the corresponding jail. action.py @@ -290,30 +669,45 @@ Releasing * https://bugzilla.redhat.com/buglist.cgi?query_format=advanced&bug_status=NEW&bug_status=ASSIGNED&component=fail2ban&classification=Red%20Hat&classification=Fedora * http://www.freebsd.org/cgi/query-pr-summary.cgi?text=fail2ban -# Provide a release sample to distributors +# Make sure the tests pass - * Debian: Yaroslav Halchenko - http://packages.qa.debian.org/f/fail2ban.html - * FreeBSD: Christoph Theis theis@gmx.at>, Nick Hilliard - http://svnweb.freebsd.org/ports/head/security/py-fail2ban/Makefile?view=markup - * Fedora: Axel Thimm - https://apps.fedoraproject.org/packages/fail2ban - * Gentoo: netmon@gentoo.org - http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-analyzer/fail2ban/metadata.xml?view=markup - * openSUSE: Stephan Kulow - https://build.opensuse.org/package/users?package=fail2ban&project=openSUSE%3AFactory - * Mac Ports: @Malbrouck on github (gh-49) - https://trac.macports.org/browser/trunk/dports/security/fail2ban/Portfile + ./fail2ban-testcases-all -# Wait for feedback from distributors +# Ensure the version is correct -# Ensure the version is correct in ./common/version.py + in: + * ./common/version.py + * top of ChangeLog + * README.md + +# Ensure the MANIFEST is complete + +Run: + + python setup.py sdist + +Look for errors like: + 'testcases/files/logs/mysqld.log' not a regular file -- skipping + +Which indicates that testcases/files/logs/mysqld.log has been moved or is a directory + + tar -C /tmp -jxf dist/fail2ban-0.8.11.tar.bz2 + +# clean up current direcory + + diff -rul --exclude \*.pyc . /tmp/fail2ban-0.8.11/ + + # Only differences should be files that you don't want distributed. + +# Ensure the tests work from the tarball + + cd /tmp/fail2ban-0.8.11/ && ./fail2ban-testcases-all # Add/finalize the corresponding entry in the ChangeLog To generate a list of committers use e.g. - git shortlog -sn 0.8.8.. | sed -e 's,^[ 0-9\t]*,,g' | tr '\n' '\|' | sed -e 's:|:, :g' + git shortlog -sn 0.8.10.. | sed -e 's,^[ 0-9\t]*,,g' | tr '\n' '\|' | sed -e 's:|:, :g' Ensure the top of the ChangeLog has the right version and current date. @@ -322,23 +716,59 @@ Releasing # Update man pages (cd man ; ./generate-man ) - git commit -m 'update man pages for release' man/* + git commit -m 'DOC/ENH: update man pages for release' man/* -# Make sure the tests pass +# Prepare source and rpm binary distributions - ./fail2ban-testcases-all - -# Prepare/upload source and rpm binary distributions - - python setup.py check python setup.py sdist python setup.py bdist_rpm python setup.py upload -# Run the following and update the wiki with output: +# Provide a release sample to distributors + * Debian: Yaroslav Halchenko + http://packages.qa.debian.org/f/fail2ban.html + * FreeBSD: Christoph Theis theis@gmx.at>, Nick Hilliard + http://svnweb.freebsd.org/ports/head/security/py-fail2ban/Makefile?view=markup + http://www.freebsd.org/cgi/query-pr-summary.cgi?text=fail2ban + * Fedora: Axel Thimm + https://apps.fedoraproject.org/packages/fail2ban + http://pkgs.fedoraproject.org/cgit/fail2ban.git + https://admin.fedoraproject.org/pkgdb/acls/bugs/fail2ban + * Gentoo: netmon@gentoo.org + http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-analyzer/fail2ban/metadata.xml?view=markup + https://bugs.gentoo.org/buglist.cgi?quicksearch=fail2ban + * openSUSE: Stephan Kulow + https://build.opensuse.org/package/show/openSUSE:Factory/fail2ban + * Mac Ports: @Malbrouck on github (gh-49) + https://trac.macports.org/browser/trunk/dports/security/fail2ban/Portfile + * Mageia: + https://bugs.mageia.org/buglist.cgi?quicksearch=fail2ban + An potentially to the fail2ban-users directory. + +# Wait for feedback from distributors + +# Prepare a release notice https://github.com/fail2ban/fail2ban/releases/new + + Upload the source/binaries from the dist directory and tag the release using the URL + +# Upload source/binaries to sourceforge http://sourceforge.net/projects/fail2ban/ + +# Run the following and update the wiki with output: python -c 'import common.protocol; common.protocol.printWiki()' + page: http://www.fail2ban.org/wiki/index.php/Commands + +* Update: + http://www.fail2ban.org/wiki/index.php/Downloads + http://www.fail2ban.org/wiki/index.php/ChangeLog + http://www.fail2ban.org/wiki/index.php/Requirements (Check requirement) + http://www.fail2ban.org/wiki/index.php/Main_Page (Add to News) + http://www.fail2ban.org/wiki/index.php/Features + +* See if any filters are upgraded: + http://www.fail2ban.org/wiki/index.php/Special:AllPages + # Email users and development list of release # notify distributors @@ -348,10 +778,17 @@ Post Release Add the following to the top of the ChangeLog -ver. 0.8.11 (2013/XX/XXX) - wanna-be-stable -- Fixes -- New Features -- Enhancements +ver. 0.8.12 (2013/XX/XXX) - wanna-be-released +----------- + +- Fixes: + +- New Features: + +- Enhancements: + +Alter the git shortlog command in the previous section to refer to the just +released version. and adjust common/version.py to carry .dev suffix to signal a version under development. diff --git a/MANIFEST b/MANIFEST index 5491c7d5..43927a7a 100644 --- a/MANIFEST +++ b/MANIFEST @@ -5,11 +5,14 @@ TODO THANKS COPYING DEVELOP -doc/run-rootless.txt fail2ban-client fail2ban-server fail2ban-testcases fail2ban-regex +fail2ban-testcases-all +setup.py +setup.cfg +kill-server client/configreader.py client/configparserinc.py client/jailreader.py @@ -43,6 +46,8 @@ server/banmanager.py server/datetemplate.py server/mytime.py server/failregex.py +testcases/actionstestcase.py +testcases/dummyjail.py testcases/files/testcase-usedns.log testcases/files/logs/bsd/syslog-plain.txt testcases/files/logs/bsd/syslog-v.txt @@ -52,21 +57,61 @@ testcases/files/logs/assp testcases/files/logs/asterisk testcases/files/logs/dovecot testcases/files/logs/exim -testcases/files/logs/lighttpd -testcases/files/logs/mysqld.log +testcases/files/logs/suhosin +testcases/files/logs/mysqld-auth testcases/files/logs/named-refused testcases/files/logs/pam-generic testcases/files/logs/postfix testcases/files/logs/proftpd testcases/files/logs/pure-ftpd testcases/files/logs/roundcube-auth -testcases/files/logs/sasl +testcases/files/logs/postfix-sasl testcases/files/logs/sogo-auth testcases/files/logs/sshd testcases/files/logs/sshd-ddos testcases/files/logs/vsftpd testcases/files/logs/webmin-auth -testcases/files/logs/wu-ftpd +testcases/files/logs/wuftpd +testcases/files/logs/3proxy +testcases/files/logs/apache-auth +testcases/files/logs/apache-badbots +testcases/files/logs/apache-nohome +testcases/files/logs/apache-noscript +testcases/files/logs/courierlogin +testcases/files/logs/couriersmtp +testcases/files/logs/cyrus-imap +testcases/files/logs/dropbear +testcases/files/logs/exim-spam +testcases/files/logs/gssftpd +testcases/files/logs/lighttpd-auth +testcases/files/logs/mysqld-auth +testcases/files/logs/perdition +testcases/files/logs/php-url-fopen +testcases/files/logs/qmail +testcases/files/logs/recidive +testcases/files/logs/sieve +testcases/files/logs/selinux-ssh +testcases/files/logs/suhosin +testcases/files/logs/uwimap-auth +testcases/files/logs/wuftpd +testcases/files/logs/xinetd-fail +testcases/files/config/apache-auth/digest/.htaccess +testcases/files/config/apache-auth/digest/.htpasswd +testcases/files/config/apache-auth/digest_time/.htaccess +testcases/files/config/apache-auth/digest_time/.htpasswd +testcases/files/config/apache-auth/basic/authz_owner/.htaccess +testcases/files/config/apache-auth/basic/authz_owner/cant_get_me.html +testcases/files/config/apache-auth/basic/authz_owner/.htpasswd +testcases/files/config/apache-auth/basic/file/.htaccess +testcases/files/config/apache-auth/basic/file/.htpasswd +testcases/files/config/apache-auth/digest.py +testcases/files/config/apache-auth/digest_wrongrelm/.htaccess +testcases/files/config/apache-auth/digest_wrongrelm/.htpasswd +testcases/files/config/apache-auth/digest_anon/.htaccess +testcases/files/config/apache-auth/digest_anon/.htpasswd +testcases/files/config/apache-auth/README +testcases/files/config/apache-auth/noentry/.htaccess +testcases/samplestestcase.py testcases/banmanagertestcase.py testcases/failmanagertestcase.py testcases/clientreadertestcase.py @@ -82,8 +127,6 @@ testcases/files/testcase03.log testcases/files/testcase04.log testcases/misctestcase.py testcases/utils.py -setup.py -setup.cfg common/__init__.py common/exceptions.py common/helpers.py @@ -101,7 +144,7 @@ config/filter.d/couriersmtp.conf config/filter.d/cyrus-imap.conf config/filter.d/exim.conf config/filter.d/gssftpd.conf -config/filter.d/lighttpd-fastcgi.conf +config/filter.d/suhosin.conf config/filter.d/named-refused.conf config/filter.d/postfix.conf config/filter.d/proftpd.conf @@ -109,7 +152,7 @@ config/filter.d/pure-ftpd.conf config/filter.d/qmail.conf config/filter.d/pam-generic.conf config/filter.d/php-url-fopen.conf -config/filter.d/sasl.conf +config/filter.d/postfix-sasl.conf config/filter.d/sieve.conf config/filter.d/sshd.conf config/filter.d/sshd-ddos.conf @@ -124,10 +167,24 @@ config/filter.d/lighttpd-auth.conf config/filter.d/recidive.conf config/filter.d/roundcube-auth.conf config/filter.d/assp.conf -config/filter.d/mysqld-auth.conf config/filter.d/sogo-auth.conf +config/filter.d/mysqld-auth.conf +config/filter.d/selinux-common.conf +config/filter.d/selinux-ssh.conf +config/filter.d/3proxy.conf +config/filter.d/apache-common.conf +config/filter.d/exim-common.conf +config/filter.d/exim-spam.conf +config/filter.d/perdition.conf +config/filter.d/uwimap-auth.conf +config/action.d/apf.conf +config/action.d/osx-afctl.conf +config/action.d/osx-ipfw.conf +config/action.d/sendmail-common.conf config/action.d/bsd-ipfw.conf config/action.d/dummy.conf +config/action.d/firewall-cmd-direct-new.conf +config/action.d/iptables-ipset-proto6-allports.conf config/action.d/iptables-blocktype.conf config/action.d/iptables-ipset-proto4.conf config/action.d/iptables-ipset-proto6.conf @@ -155,6 +212,7 @@ config/action.d/sendmail-whois.conf config/action.d/sendmail-whois-lines.conf config/action.d/shorewall.conf config/fail2ban.conf +doc/run-rootless.txt man/fail2ban-client.1 man/fail2ban.1 man/jail.conf.5 @@ -176,9 +234,8 @@ files/cacti/fail2ban_stats.sh files/cacti/cacti_host_template_fail2ban.xml files/cacti/README files/nagios/check_fail2ban -files/nagios/f2ban.txt +files/nagios/README files/bash-completion files/fail2ban-tmpfiles.conf files/fail2ban.service files/ipmasq-ZZZzzz_fail2ban.rul -files/nagios/README diff --git a/README.Solaris b/README.Solaris index 10a5f88c..e41e3811 100644 --- a/README.Solaris +++ b/README.Solaris @@ -69,27 +69,10 @@ FAIL2BAN CONFIGURATION OPT: Create /etc/fail2ban/fail2ban.local containing: -# Fail2Ban main configuration file -# -# Comments: use '#' for comment lines and ';' (following a space) for inline comments -# -# Changes: in most of the cases you should not modify this -# file, but provide customizations in fail2ban.local file, e.g.: -# -# [Definition] -# loglevel = 4 +# Fail2Ban configuration file for logging fail2ban on Solaris # [Definition] -# Option: logtarget -# Notes.: Set the log target. This could be a file, SYSLOG, STDERR or STDOUT. -# Only one log target can be specified. -# If you change logtarget from the default value and you are -# using logrotate -- also adjust or disable rotation in the -# corresponding configuration file -# (e.g. /etc/logrotate.d/fail2ban on Debian systems) -# Values: STDOUT STDERR SYSLOG file Default: /var/log/fail2ban.log -# logtarget = /var/adm/fail2ban.log @@ -99,13 +82,13 @@ REQ: Create /etc/fail2ban/jail.local containing: enabled = true filter = sshd -action = hostsdeny +action = hostsdeny[daemon_list=sshd] sendmail-whois[name=SSH, dest=you@example.com] ignoreregex = for myuser from logpath = /var/adm/auth.log Set the sendmail dest address to something useful or drop the line to stop it spamming you. -Set 'myuser' to your username to avoid banning yourself or drop it. +Set 'myuser' to your username to avoid banning yourself or remove the line. START (OR RESTART) FAIL2BAN @@ -128,7 +111,7 @@ GOTCHAS AND FIXMES svcadm enable fail2ban * If svcs -xv says that fail2ban failed to start or svcs says it's in maintenance mode - chcek /var/svc/log/network-fail2ban:default.log for clues. + check /var/svc/log/network-fail2ban:default.log for clues. Check permissions on /var/adm, /var/adm/auth.log /var/adm/fail2ban.log and /var/run/fail2ban You may need to: @@ -136,6 +119,4 @@ GOTCHAS AND FIXMES * Fail2ban adds lines like these to /etc/hosts.deny: - ALL: 1.2.3.4 - - wouldn't it be better to just block sshd? + sshd: 1.2.3.4 diff --git a/README.md b/README.md index 05e92fd6..7c00233f 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ / _|__ _(_) |_ ) |__ __ _ _ _ | _/ _` | | |/ /| '_ \/ _` | ' \ |_| \__,_|_|_/___|_.__/\__,_|_||_| - v0.8.10 2013/06/12 + v0.8.11-pre1 2013/10/30 ## Fail2Ban: ban hosts that cause multiple authentication errors @@ -30,8 +30,8 @@ Optional: To install, just do: - tar xvfj fail2ban-0.8.10.tar.bz2 - cd fail2ban-0.8.10 + tar xvfj fail2ban-0.8.11.tar.bz2 + cd fail2ban-0.8.11 python setup.py install This will install Fail2Ban into /usr/share/fail2ban. The executable scripts are diff --git a/THANKS b/THANKS index ba33b766..5f6d1b2b 100644 --- a/THANKS +++ b/THANKS @@ -1,44 +1,65 @@ -Fail2Ban is an open source project with many contributions from its -users community. Below is an alphabetically sorted partial list of the -contributors to the project. If you have been left off, please let us -know (preferably send a pull request on github with the "fix") and you -will be added +Fail2Ban is an open source project which was conceived and originally +developed by Cyril Jaquier until 2010. Since then Fail2Ban grew into +a community-driven project with many contributions from its users. +Below is an alphabetically sorted partial list of the contributors to +the project. If you have been left off, please let us know +(preferably send a pull request on github with the "fix") and you will +be added Adrien Clerc +ache +Amir Caspi Andrey G. Grozin +Andy Fragen Arturo 'Buanzo' Busleiman Axel Thimm +Beau Raines Bill Heaton Carlos Alberto Lopez Perez Christian Rauch +Christophe Carles Christoph Haas Christos Psonis +Cyril Jaquier Daniel B. Cid Daniel Black David Nutter Eric Gerbier Enrico Labedzki +ftoppi +François Boulogne +Frédéric +Georgiy Mernov Guillaume Delvit Hanno 'Rince' Wagner Iain Lea Jonathan Kamens Jonathan Underwood Joël Bertrand +JP Espinosa Justin Shore Kévin Drapel +kjohnsonecl kojiro +Manuel Arostegui Ramirez +Marcel Dopita Mark Edgington +Mark McKinstry Markus Hoffmann Marvin Rouge mEDI +Мернов Георгий Michael C. Haller Michael Hanselmann -NickMunger +Nick Munger Patrick Börjesson Raphaël Marichez +RealRancor René Berber Robert Edeker +Rolf Fokkens Russell Odom +Sebastian Arcus Sireyessire silviogarbes Stephen Gildea @@ -48,5 +69,7 @@ Tyler Vaclav Misek Vincent Deffontaines Yaroslav Halchenko +ykimon Yehuda Katz zugeschmiert +Zurd diff --git a/client/configreader.py b/client/configreader.py index 3d3aff94..96aab5f3 100644 --- a/client/configreader.py +++ b/client/configreader.py @@ -54,16 +54,19 @@ class ConfigReader(SafeConfigParserWithIncludes): % self._basedir) basename = os.path.join(self._basedir, filename) logSys.debug("Reading configs for %s under %s " % (basename, self._basedir)) - config_files = [ basename + ".conf", - basename + ".local" ] - - # choose only existing ones - config_files = filter(os.path.exists, config_files) + config_files = [ basename + ".conf" ] # possible further customizations under a .conf.d directory config_dir = basename + '.d' config_files += sorted(glob.glob('%s/*.conf' % config_dir)) + config_files.append(basename + ".local") + + config_files += sorted(glob.glob('%s/*.local' % config_dir)) + + # choose only existing ones + config_files = filter(os.path.exists, config_files) + if len(config_files): # at least one config exists and accessible logSys.debug("Reading config files: " + ', '.join(config_files)) diff --git a/client/fail2banreader.py b/client/fail2banreader.py index ada88084..0171b457 100644 --- a/client/fail2banreader.py +++ b/client/fail2banreader.py @@ -39,7 +39,7 @@ class Fail2banReader(ConfigReader): ConfigReader.read(self, "fail2ban") def getEarlyOptions(self): - opts = [["string", "socket", "/tmp/fail2ban.sock"], + opts = [["string", "socket", "/var/run/fail2ban/fail2ban.sock"], ["string", "pidfile", "/var/run/fail2ban/fail2ban.pid"]] return ConfigReader.getOptions(self, "Definition", opts) diff --git a/client/jailreader.py b/client/jailreader.py index f8757e26..7fbac423 100644 --- a/client/jailreader.py +++ b/client/jailreader.py @@ -24,7 +24,7 @@ __author__ = "Cyril Jaquier" __copyright__ = "Copyright (c) 2004 Cyril Jaquier" __license__ = "GPL" -import logging, re, glob +import logging, re, glob, os.path from configreader import ConfigReader from filterreader import FilterReader @@ -55,7 +55,23 @@ class JailReader(ConfigReader): def isEnabled(self): return self.__force_enable or self.__opts["enabled"] - + + @staticmethod + def _glob(path): + """Given a path for glob return list of files to be passed to server. + + Dangling symlinks are warned about and not returned + """ + pathList = [] + for p in glob.glob(path): + if not os.path.exists(p): + logSys.warning("File %s doesn't even exist, thus cannot be monitored" % p) + elif not os.path.lexists(p): + logSys.warning("File %s is a dangling link, thus cannot be monitored" % p) + else: + pathList.append(p) + return pathList + def getOptions(self): opts = [["bool", "enabled", "false"], ["string", "logpath", "/var/log/messages"], @@ -103,16 +119,30 @@ class JailReader(ConfigReader): logSys.warn("No actions were defined for %s" % self.__name) return True - def convert(self): + def convert(self, allow_no_files=False): + """Convert read before __opts to the commands stream + + Parameters + ---------- + allow_missing : bool + Either to allow log files to be missing entirely. Primarily is + used for testing + """ + stream = [] for opt in self.__opts: if opt == "logpath": + found_files = 0 for path in self.__opts[opt].split("\n"): - pathList = glob.glob(path) + pathList = JailReader._glob(path) if len(pathList) == 0: - logSys.error("No file found for " + path) + logSys.error("No file(s) found for glob %s" % path) for p in pathList: + found_files += 1 stream.append(["set", self.__name, "addlogpath", p]) + if not (found_files or allow_no_files): + raise ValueError( + "Have not found any log file for %s jail" % self.__name) elif opt == "backend": backend = self.__opts[opt] elif opt == "maxretry": diff --git a/client/jailsreader.py b/client/jailsreader.py index 098b525d..00c63e3c 100644 --- a/client/jailsreader.py +++ b/client/jailsreader.py @@ -18,7 +18,7 @@ # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. # Author: Cyril Jaquier -# +# __author__ = "Cyril Jaquier" __copyright__ = "Copyright (c) 2004 Cyril Jaquier" @@ -32,7 +32,7 @@ from jailreader import JailReader logSys = logging.getLogger("fail2ban.client.config") class JailsReader(ConfigReader): - + def __init__(self, force_enable=False, **kwargs): """ Parameters @@ -44,17 +44,25 @@ class JailsReader(ConfigReader): ConfigReader.__init__(self, **kwargs) self.__jails = list() self.__force_enable = force_enable - + def read(self): return ConfigReader.read(self, "jail") - - def getOptions(self, section = None): + + def getOptions(self, section=None): + """Reads configuration for jail(s) and adds enabled jails to __jails + """ opts = [] self.__opts = ConfigReader.getOptions(self, "Definition", opts) - if section: - # Get the options of a specific jail. - jail = JailReader(section, basedir=self.getBaseDir(), force_enable=self.__force_enable) + if section is None: + sections = self.sections() + else: + sections = [ section ] + + # Get the options of all jails. + for sec in sections: + jail = JailReader(sec, basedir=self.getBaseDir(), + force_enable=self.__force_enable) jail.read() ret = jail.getOptions() if ret: @@ -62,34 +70,30 @@ class JailsReader(ConfigReader): # We only add enabled jails self.__jails.append(jail) else: - logSys.error("Errors in jail '%s'. Skipping..." % section) + logSys.error("Errors in jail %r. Skipping..." % sec) return False - else: - # Get the options of all jails. - for sec in self.sections(): - jail = JailReader(sec, basedir=self.getBaseDir(), force_enable=self.__force_enable) - jail.read() - ret = jail.getOptions() - if ret: - if jail.isEnabled(): - # We only add enabled jails - self.__jails.append(jail) - else: - logSys.error("Errors in jail '" + sec + "'. Skipping...") - return False return True - - def convert(self): + + def convert(self, allow_no_files=False): + """Convert read before __opts and jails to the commands stream + + Parameters + ---------- + allow_missing : bool + Either to allow log files to be missing entirely. Primarily is + used for testing + """ + stream = list() for opt in self.__opts: if opt == "": stream.append([]) # Convert jails for jail in self.__jails: - stream.extend(jail.convert()) + stream.extend(jail.convert(allow_no_files=allow_no_files)) # Start jails for jail in self.__jails: stream.append(["start", jail.getName()]) - + return stream - + diff --git a/common/__init__.py b/common/__init__.py index 2b76f4b6..3eae8ee3 100644 --- a/common/__init__.py +++ b/common/__init__.py @@ -23,3 +23,8 @@ __author__ = "Cyril Jaquier" __copyright__ = "Copyright (c) 2004 Cyril Jaquier" __license__ = "GPL" + +import logging + +# Custom debug level +logging.HEAVYDEBUG = 5 diff --git a/common/helpers.py b/common/helpers.py index c0cf052e..74ea7a7a 100644 --- a/common/helpers.py +++ b/common/helpers.py @@ -17,24 +17,12 @@ # along with Fail2Ban; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -__author__ = "Cyril Jaquier, Arturo 'Buanzo' Busleiman" -__copyright__ = "Copyright (c) 2009 Cyril Jaquier" +__author__ = "Cyril Jaquier, Arturo 'Buanzo' Busleiman, Yaroslav Halchenko" __license__ = "GPL" def formatExceptionInfo(): - """ Author: Arturo 'Buanzo' Busleiman """ + """ Consistently format exception information """ import sys cla, exc = sys.exc_info()[:2] - excName = cla.__name__ - try: - excArgs = exc.__dict__["args"] - # Assure that we always return a string, without unneeded - # 'decorations' with python <= 2.5 where args would be a tuple - if isinstance(excArgs, tuple) and len(excArgs) == 1: - excArgs = excArgs[0] - excArgs = str(excArgs) - except KeyError: - # And always provide a string output - excArgs = str(exc) - return (excName, excArgs) + return (cla.__name__, str(exc)) diff --git a/common/version.py b/common/version.py index fe99f95e..a0cb94ea 100644 --- a/common/version.py +++ b/common/version.py @@ -24,4 +24,4 @@ __author__ = "Cyril Jaquier, Yaroslav Halchenko" __copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2011-2013 Yaroslav Halchenko" __license__ = "GPL" -version = "0.8.10" +version = "0.8.11.pre1" diff --git a/config/action.d/apf.conf b/config/action.d/apf.conf new file mode 100644 index 00000000..9af3066d --- /dev/null +++ b/config/action.d/apf.conf @@ -0,0 +1,43 @@ +# Fail2Ban configuration file +# +# Author: Mark McKinstry +# +[Definition] + +# Option: actionstart +# Notes.: command executed once at the start of Fail2Ban. +# Values: CMD +# +actionstart = + +# Option: actionstop +# Notes.: command executed once at the end of Fail2Ban +# Values: CMD +# +actionstop = + +# Option: actioncheck +# Notes.: command executed once before each actionban command +# Values: CMD +# +actioncheck = + +# Option: actionban +# Notes.: command executed when banning an IP. Take care that the +# command is executed with Fail2Ban user rights. +# Tags: IP address +# number of failures +#