__ _ _ ___ _ / _|__ _(_) |_ ) |__ __ _ _ _ | _/ _` | | |/ /| '_ \/ _` | ' \ |_| \__,_|_|_/___|_.__/\__,_|_||_| ================================================================================ How to develop for Fail2Ban ================================================================================ Fail2Ban uses GIT (http://git-scm.com/) distributed source control. This gives each developer their own complete copy of the entire repository. Developers can add and switch branches and commit changes when ever they want and then ask a maintainer to merge their changes. Fail2Ban uses GitHub (https://github.com/fail2ban/fail2ban) to manage access to the Git repository. GitHub provides free hosting for open-source projects as well as a web-based Git repository browser and an issue tracker. If you are familiar with Python and you have a bug fix or a feature that you would like to add to Fail2Ban, the best way to do so it to use the GitHub Pull Request feature. You can find more details on the Fail2Ban wiki (http://www.fail2ban.org/wiki/index.php/Get_Involved) Pull Requests ============= When submitting pull requests on GitHub we ask you to: * Clearly describe the problem you're solving; * Don't introduce regressions that will make it hard for systems administrators to update; * If adding a major feature rebase your changes on master and get to a single commit; * Include test cases (see below); * Include sample logs (if relevant); * Include a change to the relevant section of the ChangeLog; and * Include yourself in THANKS if not already there. Filters ======= Filters are tricky. They need to: * work with a variety of the versions of the software that generates the logs; * work with the range of logging configuration options available in the software; * work with multiple operating systems; * not make assumptions about the log format in excess of the software (don't assume a username doesn't contain spaces and use \S+ unless you've checked the source code); * make assumptions as to how future versions of the software will log messages (guess what would happen to the log message if different authentication types are added); * not be susceptible to DoS vulnerabilities (see Filter Security below); and * match intended log lines only. Please follow the steps from Filter Test Cases to Developing Filter Regular Expressions and submit a GitHub pull request afterwards. If you get stuck, create a GitHub issue with what you have done and we'll attempt to help. Filter test cases ----------------- Purpose: Start by finding the log messages that the application generates related to some form of authentication failure. If you are adding to an existing filter think about whether the log messages are of a similar importance and purpose to the existing filter. If you are a user of fail2ban, and did a package update of fail2ban that started matching the new log messages, would anything unexpected happen? Would the bantime/findtime for the jail be appropriate for the new log messages. If it doesn't perhaps it needs to be in a separate filter definition, for example like exim is authentication failures and exim-spam contains log messages related to spam. Even if it is a new filter you may consider separating the log messages into different filters based on purpose. Cause: Are some of the log lines a result of the same action? For example is a PAM failure log message, followed by an application specific failure message the result of the same user/script action. The result is if you add regular expressions for both you'll end up with two failures for a single action. Select the most appropriate log message and document the other log message with a test case not to match it and a description as to why you chose one over another. With the log lines selected consider what occurred to generate those log messages and whether they could of been generated by accidental means. Could the log message occur always as this is the first step towards the application asking for authentication? Could the log messages occur often? If some of these are true make a note of this in the jail.conf example that you provide. Samples: Its important to include log file samples so any future change in the regular expression will still work with the log lines you have identified. The sample log messages are provided in testcases/files/logs/ with same name as the filter. Each log line should include a failJSON metadata (so the logs lines are tested in the test suite) directly above the log line. If there is any specific information about the log message, such as version or an application configuration option that is needed for the message to occur, include this in a comment (line beginning with #) above the failJSON metadata. Log samples should include only one, definitely not more than 3, examples of log messages of the same form. If log messages are different in different versions of the application log messages that show this is encouraged. Also attempt inject an IP into the application so that fail2ban detects the IP from user input rather than the true origin. See the Filter Security section and the top example in testcases/files/logs/apache-auth as to how to do this. One you have discovered this correct the regex so it doesn't match and provide this as a test case with match: false (see failJSON below). If the mechanism to create the log message isn't obvious provide a configuration and/or sample scripts testcases/files/config/{filtername} and reference these in the comments above the log line. FailJSON metadata: A failJSON metadata is a comment immediately above the log message. It will look like: # failJSON: { "time": "2013-06-10T10:10:59", "match": true , "host": "193.169.56.211" } Time should match the time of the log message. It is in a specific format of Year-Month-Day'T'Hour:minute:Second. If your log message does not include a year, like the example below, the year will be 2005, if before Sun Aug 14 10am UTC, and 2004 if afterwards. # failJSON: { "time": "2005-03-24T15:25:51", "match": true , "host": "198.51.100.87" } Mar 24 15:25:51 buffalo1 dropbear[4092]: bad password attempt for 'root' from 198.51.100.87:5543 The host will contain the IP or domain that should be blocked. For long lines that you don't want matched, like log injection vulnerabilities and log lines excluded (see "Cause" section above), a "match": false in the failJSON and the reason why in the comment above. After developing the regexs, the following command will test all the failJSON metadata against the log lines: ./fail2ban-testcases testSampleRegex Developing Filter Regular Expressions ------------------------------------- Date/Time: The first step in checking your log line can have a filter is to check that the time format matches an existing regex. To test this copy the time component from the log line and append an IP address. Then test it with: ./fail2ban-regex "2013-09-19 02:46:12 1.2.3.4" "" In the output from this should be something like: Date template hits: |- [# of hits] date format | [1] Year-Month-Day Hour:Minute:Second Ensure that the template description matches of bits in the time format. If there isn't a matched a format and date regex can be added to server/datedetector.py. Ensure this is added in an order that will match make more specific matches occur first and that their is no confusion as to which is the date or month. Filter file: The filter file is in config/filter.d/{filtername}.conf. The format of the filter file has two sections INCLUDES and Definition as follows: [INCLUDES] before = common.conf after = filtername.local [Definition] failregex = .... ignoreregex = .... This is also documented in the man pages as jail.conf (section 5). Other definitions can be added to make failregex's more readable and maintainable. General rules: Use "before" if you need to include a common set of rules, like syslog or if there's a common set of regexs for multiple filters. Use "after" if you wish to allow the user to overwrite a set of customisation's of the current filter. This file doesn't need to exist. Try to avoid using ignoreregex mainly for performance reasons. The case when you would use it is if in trying to avoid using ignoreregex, you end up with an unreadable failregex. Syslog: If your application logs to syslog you can use the following to capture that part. So as a base use: [INCLUDES] before = commmon.conf [Definition] _daemon = app failregex = ^%(__prefix_line)s In this example common.conf defines __prefix_line which also contains the _daemon name, (in syslog terms the service) you specified. _daemon can also be a regex. So the following uses a _daemon set to "dovecot" Dec 12 11:19:11 dunnart dovecot: pop3-login: Aborted login (tried to use disabled plaintext auth): rip=190.210.136.21, lip=113.212.99.193 So now ^%(__prefix_line)s matches "Dec 12 11:19:11 dunnart dovecot: ". Note it matches the trailing space. Putting a space after ^%(__prefix_line)s in the regex will probably not match. Substitutions: Substation's are what the syslog uses. The regex bits of %(_name)s substitute the _name definition into the regex. They are useful for making the regexes more readable and also defining regex parts that occur in multiple log lines. Regular Expressions: The regular expression you will be writing will assume that the date/time has been removed from the log line because this is how fail2ban works internally. If the format is like ' error 1.2.3.4 is evil' then you will need to match the < at the start so regex should be similar to '^<> is evil$'. Use where the IP/domain name appears in the log line. The following general rules apply to regular expressions: * Ensure regexs start with a ^ and are restrictive as possible. E.g. not .* if \d+ is sufficient * Use the functionality of regexs http://docs.python.org/2/library/re.html * Try to make the regular expression readable (as much as possible). E.g. (?:...) represents a non-capturing regex but (...) is more readable. If you only have a basic knowledge of regular repressions read http://docs.python.org/2/library/re.html first. Really. It doesn't take long and will remind you which bits you need to escape and which bits you don't. Developing/testing the regex: You can develop the regex in the file or on the command line depending on your preference. You can also use the samples you've created in the test cases or test them one at a time. The general tool is fail2ban-regex. To see how to use it run: ./fail2ban-regex --help Take note of -l heavydebug / -l debug and -v as they will be most useful. TIP: Take a look at the source code of the application. You may see optional or extra log messages, or parts there of, that need to form part of your regex. It may also show how some parts are con trained and different formats depending on configuration or less common usages. TIP: Some applications log spaces at the end. If you're not sure add \s*$ as the end part of the regex. If your regex isn't matching take a look at http://www.debuggex.com/?flavor=python Using the regex from the ./fail2ban-regex output (to ensure all substitutions are done) and with replaced with (?&.ipv4). Set the regex type to Python. For the test data put your log output with the time removed. When you've fixed the regex put it back into your filter file. Please spread the good word about debuggex - Serge Toarca is kindly continuing its free availability to Open Source developers. Finishing up: If you've created a new filter, add an entry in config/jail.conf. The theory here is that a user will create a jail.conf with [filtername]\nenable=true. So more specifically in the [filter] section in jail.conf: * Ensure that you have "enabled = false", we want people to enable as needed * use "filter =" set to your filter name. * use a action to disable ports associated with the application * set "logpath" to a usual location for the log file for the application. * If the default findtime or bantime isn't appropriate to the filter set a value that is more appropriate. Send the fail2ban a git pull request (See "Pull Requests" above) containing your great work. Filter Security --------------- Poor filter regular expressions are susceptible to DoS attacks. When a remote user has the ability to introduce text that will match the filter regex, such that the inserted text matches the part, they have the ability to deny any host they choose. So the part must be anchored on text generated by the application, and not the user, to a sufficient extent that the user cannot insert the entire text. Ideally filter regex should anchor to the beginning and end of the log line however as more applications log at the beginning than the end, anchoring the beginning is more important. If the log file used by the application is shared with other applications, like system logs, ensure the other application that use that log file do not log user generated text at the beginning of the line, or, if they do, ensure the regexs of the filter are sufficient to mitigate the risk of insertion. Examples of poor filters ------------------------ 1. Too restrictive We find a log message: Apr-07-13 07:08:36 Invalid command fial2ban from 1.2.3.4 We make a failregex ^Invalid command \S+ from Now think evil. The user does the command 'blah from 1.2.3.44' The program diligently logs: Apr-07-13 07:08:36 Invalid command blah from 1.2.3.44 from 1.2.3.4 And fail2ban matches 1.2.3.44 as the IP that it ban. A DoS attack was successful. The fix here is that the command can be anything so .* is appropriate. ^Invalid command .* from Here the .* will match until the end of the string. Then realise it has more to match, i.e. "from " and go back until it find this. Then it will ban 1.2.3.4 correctly. Since the is always at the end, end the regex with a $. ^Invalid command .* from $ Note if we'd just had the expression: ^Invalid command \S+ from $ Then provided the user put a space in their command they would have never been banned. 2. Filter regex can match other user injected data From the Apache vulnerability CVE-2013-2178 ( original ref: https://vndh.net/note:fail2ban-089-denial-service ). An example bad regex for Apache: failregex = [[]client []] user .* not found Since the user can do a get request on: GET /[client%20192.168.0.1]%20user%20root%20not%20found HTTP/1.0 Host: remote.site Now the log line will be: [Sat Jun 01 02:17:42 2013] [error] [client 192.168.33.1] File does not exist: /srv/http/site/[client 192.168.0.1] user root not found As this log line doesn't match other expressions hence it matches the above regex and blocks 192.168.33.1 as a denial of service from the HTTP requester. 3. Application generates two identical log messages with different meanings If the application generates the following two messages under different circumstances: client : authentication failed client : authentication failed Then it's obvious that a regex of "^client : authentication failed$" will still cause problems if the user can trigger the second log message with a of 123.1.1.1. Here there's nothing to do except request/change the application so it logs messages differently. Code Testing ============ Existing tests can be run by executing `fail2ban-testcases`. This has options like --log-level that will probably be useful. `fail2ban-testcases --help` for full options. Test cases should cover all usual cases, all exception cases and all inside / outside boundary conditions. Test cases should cover all branches. The coverage tool will help identify missing branches. Also see http://nedbatchelder.com/code/coverage/branch.html for more details. Install the package python-coverage to visualise your test coverage. Run the following (note: on Debian-based systems, the script is called `python-coverage`): coverage run fail2ban-testcases coverage html Then look at htmlcov/index.html and see how much coverage your test cases exert over the code base. Full coverage is a good thing however it may not be complete. Try to ensure tests cover as many independent paths through the code. Manual Execution. To run in a development environment do: ./fail2ban-client -c config/ -s /tmp/f2b.sock -i start some quick commands: status add test pyinotify status test set test addaction iptables set test actionban iptables echo >> /tmp/ban set test actionunban iptables echo >> /tmp/unban get test actionban iptables get test actionunban iptables set test banip 192.168.2.2 status test Coding Standards ================ Style ----- Please use tabs for now. Keep to 80 columns, at least for readable text. Tests ----- Add tests. They should test all the code you add in a meaning way. Coverage -------- Test coverage should always increase as you add code. You may use "# pragma: no cover" in the code for branches of code that support older versions on python. For all other uses of "pragma: no cover" or "pragma: no branch" document the reason why its not covered. "I haven't written a test case" isn't a sufficient reason. Documentation ------------- Ensure this documentation is up to date after changes. Also ensure that the man pages still are accurate. Ensure that there is sufficient documentation for your new features to be used. Bugs ---- Remove them and don't add any more. Git --- Use the following tags in your commit messages: 'BF:' for bug fixes 'DOC:' for documentation fixes 'ENH:' for enhancements 'TST:' for commits concerning tests only (thus not touching the main code-base) Multiple tags could be joined with +, e.g. "BF+TST:". Use the text "closes #333"/"resolves #333 "/"fixes #333" where 333 represents an issue that is closed. Other text and details in link below. See: https://help.github.com/articles/closing-issues-via-commit-messages If merge resulted in conflicts, clarify what changes were done to corresponding files in the 'Conflicts:' section of the merge commit message. See e.g. https://github.com/fail2ban/fail2ban/commit/f5a8a8ac Adding Actions -------------- If you add an action.d/*.conf file also add a example in config/jail.conf with enabled=false and maxretry=5 for ssh. Design ====== Fail2Ban was initially developed with Python 2.3 (IIRC). It should still be compatible with Python 2.4 and such compatibility assurance makes code ... old-fashioned in many places (RF-Note). In 0.7 the design went through major re-factoring into client/server, a-thread-per-jail design which made it a bit difficult to follow. Below you can find a sketchy description of the main components of the system to orient yourself better. server/ ------ Core classes hierarchy (feel welcome to draw a better/more complete one):: -> inheritance + delegation * storage of multiple instances RF-Note just a note which might be useful to address while doing RF JailThread -> Filter -> FileFilter -> {FilterPoll, FilterPyinotify, ...} | * FileContainer + FailManager + DateDetector + Jail (provided in __init__) which contains this Filter (used for passing tickets from FailManager to Jail's __queue) Server + Jails * Jail + Filter (in __filter) * tickets (in __queue) + Actions (in __action) * Action + BanManager failmanager.py ~~~~~~~~~~~~~~ FailManager Keeps track of failures, recorded as 'tickets'. All operations are done via acquiring a lock FailManagerEmpty(Exception) raised by FailManager.toBan after reaching the list of tickets (RF-Note: asks to become a generator ;) ) filter.py ~~~~~~~~~~ Filter(JailThread) Wraps (non-threaded) FailManager (and proxies to it quite a bit), and provides all primary logic for processing new lines, what IPs to ignore, etc .failManager [FailManager] .dateDetector [DateDetector] .__failRegex [list] .__ignoreRegex [list] Contains regular expressions for failures and ignores .__findTime [numeric] Used in `processLineAndAdd` to skip old lines FileFilter(Filter): Files-aware Filter .__logPath [list] keeps the tracked files (added 1-by-1 using addLogPath) stored as FileContainer's .getFailures actually just returns True if managed to open and get lines (until empty) False if failed to open or absent container matching the filename FileContainer Adapter for a file to deal with log rotation. .open,.close,.readline RF-Note: readline returns "" with handler absent... shouldn't it be None? .__pos Keeps the position pointer dnsutils.py ~~~~~~~~~~~ DNSUtils Utility class for DNS and IP handling filter*.py ~~~~~~~~~~ Implementations of FileFilter's for specific backends. Derived classes should provide an implementation of `run` and usually override `addLogPath`, `delLogPath` methods. In run() method they all one way or another provide try: while True: ticket = self.failManager.toBan() self.jail.putFailTicket(ticket) except FailManagerEmpty: self.failManager.cleanup(MyTime.time()) thus channelling "ban tickets" from their failManager to the corresponding jail. action.py ~~~~~~~~~ Takes care about executing start/check/ban/unban/stop commands Releasing ========= # Check distribution patches and see if they can be included * https://apps.fedoraproject.org/packages/fail2ban/sources * http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-analyzer/fail2ban/ * http://svnweb.freebsd.org/ports/head/security/py-fail2ban/ * https://build.opensuse.org/package/show?package=fail2ban&project=openSUSE%3AFactory * http://sophie.zarb.org/sources/fail2ban (Mageia) * https://trac.macports.org/browser/trunk/dports/security/fail2ban # Check distribution outstanding bugs * https://github.com/fail2ban/fail2ban/issues?sort=updated&state=open * http://bugs.debian.org/cgi-bin/pkgreport.cgi?dist=unstable;package=fail2ban * http://bugs.sabayon.org/buglist.cgi?quicksearch=net-analyzer%2Ffail2ban * https://bugs.gentoo.org/buglist.cgi?query_format=advanced&short_desc=fail2ban&bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=IN_PROGRESS&short_desc_type=allwords * https://bugzilla.redhat.com/buglist.cgi?query_format=advanced&bug_status=NEW&bug_status=ASSIGNED&component=fail2ban&classification=Red%20Hat&classification=Fedora * http://www.freebsd.org/cgi/query-pr-summary.cgi?text=fail2ban # Provide a release sample to distributors * Debian: Yaroslav Halchenko http://packages.qa.debian.org/f/fail2ban.html * FreeBSD: Christoph Theis theis@gmx.at>, Nick Hilliard http://svnweb.freebsd.org/ports/head/security/py-fail2ban/Makefile?view=markup * Fedora: Axel Thimm https://apps.fedoraproject.org/packages/fail2ban * Gentoo: netmon@gentoo.org http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-analyzer/fail2ban/metadata.xml?view=markup * openSUSE: Stephan Kulow https://build.opensuse.org/package/users?package=fail2ban&project=openSUSE%3AFactory * Mac Ports: @Malbrouck on github (gh-49) https://trac.macports.org/browser/trunk/dports/security/fail2ban/Portfile # Wait for feedback from distributors # Ensure the version is correct in ./common/version.py # Add/finalize the corresponding entry in the ChangeLog To generate a list of committers use e.g. git shortlog -sn 0.8.8.. | sed -e 's,^[ 0-9\t]*,,g' | tr '\n' '\|' | sed -e 's:|:, :g' Ensure the top of the ChangeLog has the right version and current date. Ensure the top entry of the ChangeLog has the right version and current date. # Update man pages (cd man ; ./generate-man ) git commit -m 'update man pages for release' man/* # Make sure the tests pass ./fail2ban-testcases-all # Prepare/upload source and rpm binary distributions python setup.py check python setup.py sdist python setup.py bdist_rpm python setup.py upload # Run the following and update the wiki with output: python -c 'import common.protocol; common.protocol.printWiki()' # Email users and development list of release # notify distributors Post Release ============ Add the following to the top of the ChangeLog ver. 0.8.12 (2013/XX/XXX) - wanna-be-released ----------- - Fixes: - New Features: - Enhancements: and adjust common/version.py to carry .dev suffix to signal a version under development.