__ _ _ ___ _ / _|__ _(_) |_ ) |__ __ _ _ _ | _/ _` | | |/ /| '_ \/ _` | ' \ |_| \__,_|_|_/___|_.__/\__,_|_||_| ================================================================================ How to develop for Fail2Ban ================================================================================ Fail2Ban uses GIT (http://git-scm.com/) distributed source control. This gives each developer their own complete copy of the entire repository. Developers can add and switch branches and commit changes when ever they want and then ask a maintainer to merge their changes. Fail2Ban uses GitHub (https://github.com/fail2ban/fail2ban) to manage access to the Git repository. GitHub provides free hosting for open-source projects as well as a web-based Git repository browser and an issue tracker. If you are familiar with Python and you have a bug fix or a feature that you would like to add to Fail2Ban, the best way to do so it to use the GitHub Pull Request feature. You can find more details on the Fail2Ban wiki (http://www.fail2ban.org/wiki/index.php/Get_Involved) Pull Requests ============= When submitting pull requests on GitHub we ask you to: * Clearly describe the problem you're solving; * Don't introduce regressions that will make it hard for systems administrators to update; * If adding a major feature rebase your changes on master and get to a single commit; * Include test cases (see below); * Include sample logs (if relevant); * Include a change to the relevant section of the ChangeLog; and * Include yourself in THANKS if not already there. If you are developing filters see the FILTERS file for documentation. Code Testing ============ Existing tests can be run by executing `fail2ban-testcases`. This has options like --log-level that will probably be useful. `fail2ban-testcases --help` for full options. Test cases should cover all usual cases, all exception cases and all inside / outside boundary conditions. Test cases should cover all branches. The coverage tool will help identify missing branches. Also see http://nedbatchelder.com/code/coverage/branch.html for more details. Install the package python-coverage to visualise your test coverage. Run the following (note: on Debian-based systems, the script is called `python-coverage`): coverage run fail2ban-testcases coverage html Then look at htmlcov/index.html and see how much coverage your test cases exert over the code base. Full coverage is a good thing however it may not be complete. Try to ensure tests cover as many independent paths through the code. Manual Execution. To run in a development environment do: ./fail2ban-client -c config/ -s /tmp/f2b.sock -i start some quick commands: status add test pyinotify status test set test addaction iptables set test actionban iptables echo >> /tmp/ban set test actionunban iptables echo >> /tmp/unban get test actionban iptables get test actionunban iptables set test banip 192.168.2.2 status test Coding Standards ================ Style ----- Please use tabs for now. Keep to 80 columns, at least for readable text. Tests ----- Add tests. They should test all the code you add in a meaning way. Coverage -------- Test coverage should always increase as you add code. You may use "# pragma: no cover" in the code for branches of code that support older versions on python. For all other uses of "pragma: no cover" or "pragma: no branch" document the reason why its not covered. "I haven't written a test case" isn't a sufficient reason. Documentation ------------- Ensure this documentation is up to date after changes. Also ensure that the man pages still are accurate. Ensure that there is sufficient documentation for your new features to be used. Bugs ---- Remove them and don't add any more. Git --- Use the following tags in your commit messages: 'BF:' for bug fixes 'DOC:' for documentation fixes 'ENH:' for enhancements 'TST:' for commits concerning tests only (thus not touching the main code-base) Multiple tags could be joined with +, e.g. "BF+TST:". Use the text "closes #333"/"resolves #333 "/"fixes #333" where 333 represents an issue that is closed. Other text and details in link below. See: https://help.github.com/articles/closing-issues-via-commit-messages If merge resulted in conflicts, clarify what changes were done to corresponding files in the 'Conflicts:' section of the merge commit message. See e.g. https://github.com/fail2ban/fail2ban/commit/f5a8a8ac Adding Actions -------------- If you add an action.d/*.conf file also add a example in config/jail.conf with enabled=false and maxretry=5 for ssh. Design ====== Fail2Ban was initially developed with Python 2.3 (IIRC). It should still be compatible with Python 2.4 and such compatibility assurance makes code ... old-fashioned in many places (RF-Note). In 0.7 the design went through major re-factoring into client/server, a-thread-per-jail design which made it a bit difficult to follow. Below you can find a sketchy description of the main components of the system to orient yourself better. server/ ------ Core classes hierarchy (feel welcome to draw a better/more complete one):: -> inheritance + delegation * storage of multiple instances RF-Note just a note which might be useful to address while doing RF JailThread -> Filter -> FileFilter -> {FilterPoll, FilterPyinotify, ...} | * FileContainer + FailManager + DateDetector + Jail (provided in __init__) which contains this Filter (used for passing tickets from FailManager to Jail's __queue) Server + Jails * Jail + Filter (in __filter) * tickets (in __queue) + Actions (in __action) * Action + BanManager failmanager.py ~~~~~~~~~~~~~~ FailManager Keeps track of failures, recorded as 'tickets'. All operations are done via acquiring a lock FailManagerEmpty(Exception) raised by FailManager.toBan after reaching the list of tickets (RF-Note: asks to become a generator ;) ) filter.py ~~~~~~~~~~ Filter(JailThread) Wraps (non-threaded) FailManager (and proxies to it quite a bit), and provides all primary logic for processing new lines, what IPs to ignore, etc .failManager [FailManager] .dateDetector [DateDetector] .__failRegex [list] .__ignoreRegex [list] Contains regular expressions for failures and ignores .__findTime [numeric] Used in `processLineAndAdd` to skip old lines FileFilter(Filter): Files-aware Filter .__logPath [list] keeps the tracked files (added 1-by-1 using addLogPath) stored as FileContainer's .getFailures actually just returns True if managed to open and get lines (until empty) False if failed to open or absent container matching the filename FileContainer Adapter for a file to deal with log rotation. .open,.close,.readline RF-Note: readline returns "" with handler absent... shouldn't it be None? .__pos Keeps the position pointer dnsutils.py ~~~~~~~~~~~ DNSUtils Utility class for DNS and IP handling filter*.py ~~~~~~~~~~ Implementations of FileFilter's for specific backends. Derived classes should provide an implementation of `run` and usually override `addLogPath`, `delLogPath` methods. In run() method they all one way or another provide try: while True: ticket = self.failManager.toBan() self.jail.putFailTicket(ticket) except FailManagerEmpty: self.failManager.cleanup(MyTime.time()) thus channelling "ban tickets" from their failManager to the corresponding jail. action.py ~~~~~~~~~ Takes care about executing start/check/ban/unban/stop commands Releasing ========= # Check distribution patches and see if they can be included * https://apps.fedoraproject.org/packages/fail2ban/sources * http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-analyzer/fail2ban/ * http://svnweb.freebsd.org/ports/head/security/py-fail2ban/ * https://build.opensuse.org/package/show?package=fail2ban&project=openSUSE%3AFactory * http://sophie.zarb.org/sources/fail2ban (Mageia) * https://trac.macports.org/browser/trunk/dports/security/fail2ban # Check distribution outstanding bugs * https://github.com/fail2ban/fail2ban/issues?sort=updated&state=open * http://bugs.debian.org/cgi-bin/pkgreport.cgi?dist=unstable;package=fail2ban * https://bugs.launchpad.net/ubuntu/+source/fail2ban * http://bugs.sabayon.org/buglist.cgi?quicksearch=net-analyzer%2Ffail2ban * https://bugs.archlinux.org/?project=5&cat%5B%5D=33&string=fail2ban * https://bugs.gentoo.org/buglist.cgi?query_format=advanced&short_desc=fail2ban&bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=IN_PROGRESS&short_desc_type=allwords * https://bugzilla.redhat.com/buglist.cgi?query_format=advanced&bug_status=NEW&bug_status=ASSIGNED&component=fail2ban&classification=Red%20Hat&classification=Fedora * http://www.freebsd.org/cgi/query-pr-summary.cgi?text=fail2ban * https://bugs.mageia.org/buglist.cgi?quicksearch=fail2ban * https://build.opensuse.org/package/requests/openSUSE:Factory/fail2ban # Make sure the tests pass ./fail2ban-testcases-all # Ensure the version is correct in: * ./common/version.py * top of ChangeLog * README.md # Ensure the MANIFEST is complete Run: python setup.py sdist Look for errors like: 'testcases/files/logs/mysqld.log' not a regular file -- skipping Which indicates that testcases/files/logs/mysqld.log has been moved or is a directory tar -C /tmp -jxf dist/fail2ban-0.8.12.tar.bz2 # clean up current direcory diff -rul --exclude \*.pyc . /tmp/fail2ban-0.8.12/ # Only differences should be files that you don't want distributed. # Ensure the tests work from the tarball cd /tmp/fail2ban-0.8.12/ && ./fail2ban-testcases-all # Add/finalize the corresponding entry in the ChangeLog To generate a list of committers use e.g. git shortlog -sn 0.8.11.. | sed -e 's,^[ 0-9\t]*,,g' | tr '\n' '\|' | sed -e 's:|:, :g' Ensure the top of the ChangeLog has the right version and current date. Ensure the top entry of the ChangeLog has the right version and current date. # Update man pages (cd man ; ./generate-man ) git commit -m 'DOC/ENH: update man pages for release' man/* # Prepare source and rpm binary distributions python setup.py sdist python setup.py bdist_rpm python setup.py upload # Provide a release sample to distributors * Arch Linux: https://www.archlinux.org/packages/community/any/fail2ban/ * Debian: Yaroslav Halchenko http://packages.qa.debian.org/f/fail2ban.html * FreeBSD: Christoph Theis theis@gmx.at>, Nick Hilliard http://svnweb.freebsd.org/ports/head/security/py-fail2ban/Makefile?view=markup http://www.freebsd.org/cgi/query-pr-summary.cgi?text=fail2ban * Fedora: Axel Thimm https://apps.fedoraproject.org/packages/fail2ban http://pkgs.fedoraproject.org/cgit/fail2ban.git https://admin.fedoraproject.org/pkgdb/acls/bugs/fail2ban * Gentoo: netmon@gentoo.org http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-analyzer/fail2ban/metadata.xml?view=markup https://bugs.gentoo.org/buglist.cgi?quicksearch=fail2ban * openSUSE: Stephan Kulow https://build.opensuse.org/package/show/openSUSE:Factory/fail2ban * Mac Ports: @Malbrouck on github (gh-49) https://trac.macports.org/browser/trunk/dports/security/fail2ban/Portfile * Mageia: https://bugs.mageia.org/buglist.cgi?quicksearch=fail2ban An potentially to the fail2ban-users directory. # Wait for feedback from distributors # Prepare a release notice https://github.com/fail2ban/fail2ban/releases/new Upload the source/binaries from the dist directory and tag the release using the URL # Upload source/binaries to sourceforge http://sourceforge.net/projects/fail2ban/ # Run the following and update the wiki with output: python -c 'import common.protocol; common.protocol.printWiki()' page: http://www.fail2ban.org/wiki/index.php/Commands * Update: http://www.fail2ban.org/wiki/index.php?title=Template:Fail2ban_Versions&action=edit http://www.fail2ban.org/wiki/index.php?title=Template:Fail2ban_News&action=edit move old bits to: http://www.fail2ban.org/wiki/index.php?title=Template:Fail2ban_OldNews&action=edit http://www.fail2ban.org/wiki/index.php?title=Template:Fail2ban_Versions&action=edit http://www.fail2ban.org/wiki/index.php/ChangeLog http://www.fail2ban.org/wiki/index.php/Requirements (Check requirement) http://www.fail2ban.org/wiki/index.php/Features * See if any filters are upgraded: http://www.fail2ban.org/wiki/index.php/Special:AllPages # Email users and development list of release # notify distributors Post Release ============ Add the following to the top of the ChangeLog ver. 0.8.13 (2014/XX/XXX) - wanna-be-released ----------- - Fixes: - New Features: - Enhancements: Alter the git shortlog command in the previous section to refer to the just released version. and adjust common/version.py to carry .dev suffix to signal a version under development.