fail2ban/DEVELOP

                         __      _ _ ___ _               
                        / _|__ _(_) |_  ) |__  __ _ _ _  
                       |  _/ _` | | |/ /| '_ \/ _` | ' \ 
                       |_| \__,_|_|_/___|_.__/\__,_|_||_|

================================================================================
How to develop for Fail2Ban
================================================================================

Fail2Ban uses GIT (http://git-scm.com/) distributed source control. This gives
each developer their own complete copy of the entire repository. Developers can
add and switch branches and commit changes when ever they want and then ask a
maintainer to merge their changes.

Fail2Ban uses GitHub (https://github.com/fail2ban/fail2ban) to manage access to
the Git repository. GitHub provides free hosting for open-source projects as
well as a web-based Git repository browser and an issue tracker.

If you are familiar with Python and you have a bug fix or a feature that you
would like to add to Fail2Ban, the best way to do so it to use the GitHub Pull
Request feature. You can find more details on the Fail2Ban wiki
(http://www.fail2ban.org/wiki/index.php/Get_Involved)

Pull Requests
=============

When submitting pull requests on GitHub we ask you to:
* Clearly describe the problem you're solving;
* Don't introduce regressions that will make it hard for systems adminstrators 
  to update;
* If adding a major feature rebase your changes on master and get to a single commit;
* Include test cases (see below);
* Include sample logs (if relevant);
* Include a change to the relevant section of the ChangeLog; and
* Include yourself in THANKS if not already there.

Filters
=======

* Include sample logs with 1.2.3.4 used for IP addresses and 
  example.com/example.org used for DNS names
* Ensure sample log is provided in testcases/files/logs/ with same name as the
  filter. Each log line should include match meta data for time & IP above
  every line (see other sample log files for examples)
* Ensure regexs start with a ^ and are restrictive as possible. E.g. not .* if
  \d+ is sufficient
* Use the functionality of regexs http://docs.python.org/2/library/re.html
* Take a look at the source code of the application. You may see optional or
  extra log messages, or parts there of, that need to form part of your regex.

If you only have a basic knowledge of regular repressions read
http://docs.python.org/2/library/re.html first.

Filter Security
---------------

Poor filter regular expressions are suseptable to DoS attacks.

When a remote user has the ability to introduce text that will match the 
filter regex, such that the inserted text matches the <HOST> part, they have the
ability to deny any host they choose.

So the <HOST> part must be anchored on text generated by the application, and not
the user, to a sufficient extent that the user cannot insert the entire text.

Filters are matched against the log line with their date removed.

Ideally filter regex should anchor to the beginning and end of the log line
however as more applications log at the beginning than the end, achoring the
beginning is more important. If the log file used by the application is shared
with other applications, like system logs, ensure the other application that
use that log file do not log user generated text at the beginning of the line,
or, if they do, ensure the regexs of the filter are sufficient to mitigate the
risk of insertion.

When creating a regex that extends back to the begining remember the date part
has been removed within fail2ban so theres no need to match that. If the format
is like '<date...> error 1.2.3.4 is evil' then you will need to match the < at
the start so here the regex would start like '^<> <HOST> is evil$'.

Some applications log spaces at the end. If you're not sure add \s*$ as the
end part of the regex.

Examples of poor filters
------------------------

1. Too restrictive

We find a log message:

    Apr-07-13 07:08:36 Invalid command fial2ban from 1.2.3.4

We make a failregex

    ^Invalid command \S+ from <HOST>

Now think evil. The user does the command 'blah from 1.2.3.44'

The program diliently logs:

    Apr-07-13 07:08:36 Invalid command blah from 1.2.3.44 from 1.2.3.4

And fail2ban matches 1.2.3.44 as the IP that it ban. A DoS attack was successful.

The fix here is that the command can be anything so .* is approprate.

    ^Invalid command .* from <HOST>

Here the .* will match until the end of the string. Then realise it has more to
match, i.e. "from <HOST>" and go back until it find this. Then it will ban
1.2.3.4 correctly. Since the <HOST> is always at the end, end the regex with a $.

    ^Invalid command .* from <HOST>$

Note if we'd just had the expression:

    ^Invalid command \S+ from <HOST>$

Then provided the user put a space in their command they would have never been
banned.

2. Filter regex can match other user injected data

From the apache vulnerability CVE-2013-2178
( original ref: https://vndh.net/note:fail2ban-089-denial-service ).

An example bad regex for apache:

    failregex = [[]client <HOST>[]] user .* not found

Since the user can do a get request on:

    GET /[client%20192.168.0.1]%20user%20root%20not%20found HTTP/1.0
Host: remote.site

Now the log line will be:

    [Sat Jun 01 02:17:42 2013] [error] [client 192.168.33.1] File does not exist: /srv/http/site/[client 192.168.0.1] user root not found

As this log line doesn't match other expressions hence it matches the above
regex and blocks 192.168.33.1 as a denial of service from the HTTP requester.

3. Applicaiton generates two identical log messages with different meanings

If the application generates the following two messages under different
circmstances:

    client <IP>: authentication failed
    client <USER>: authentication failed


Then it's obvious that a regex of "^client <HOST>: authentication
failed$" will still cause problems if the user can trigger the second
log message with a <USER> of 123.1.1.1.

Here there's nothing to do except request/change the application so it logs
messages differently.


Code Testing
============

Existing tests can be run by executing `fail2ban-testcases`. This has options
like --log-level that will probably be useful. `fail2ban-testcases --help` for
full options.

Test cases should cover all usual cases, all exception cases and all inside
/ outside boundary conditions.

Test cases should cover all branches. The coverage tool will help identify
missing branches. Also see http://nedbatchelder.com/code/coverage/branch.html
for more details.

Install the package python-coverage to visualise your test coverage. Run the
following (note: on Debian-based systems, the script is called
`python-coverage`):

coverage run fail2ban-testcases
coverage html

Then look at htmlcov/index.html and see how much coverage your test cases
exert over the codebase. Full coverage is a good thing however it may not be
complete. Try to ensure tests cover as many independent paths through the
code.

Manual Execution. To run in a development environment do:

./fail2ban-client -c config/ -s /tmp/f2b.sock -i start

some quick commands:

status
add test pyinotify
status test
set test addaction iptables
set test actionban iptables echo <ip> <cidr> >> /tmp/ban
set test actionunban iptables echo <ip> <cidr> >> /tmp/unban
get test actionban iptables
get test actionunban iptables
set test banip 192.168.2.2
status test



Coding Standards
================

Style
-----

Please use tabs for now. Keep to 80 columns, at least for readable text.

Tests
-----

Add tests. They should test all the code you add in a meaning way.

Coverage
--------

Test coverage should always increase as you add code.

You may use "# pragma: no cover" in the code for branches of code that support
older versions on python. For all other uses of "pragma: no cover" or
"pragma: no branch" document the reason why its not covered. "I haven't written
a test case" isn't a sufficient reason.

Documentation
-------------

Ensure this documentation is up to date after changes. Also ensure that the man
pages still are accurate. Ensure that there is sufficient documentation for
your new features to be used.

Bugs
----

Remove them and don't add any more.

Git
---

Use the following tags in your commit messages:

'BF:' for bug fixes
'DOC:' for documentation fixes
'ENH:' for enhancements
'TST:' for commits concerning tests only (thus not touching the main code-base)

Multiple tags could be joined with +, e.g. "BF+TST:".

Use the text "closes #333"/"resolves #333 "/"fixes #333" where 333 represents
an issue that is closed. Other text and details in link below.
See: https://help.github.com/articles/closing-issues-via-commit-messages

If merge resulted in conflicts, clarify what changes were done to
corresponding files in the 'Conflicts:' section of the merge commit
message.  See e.g. https://github.com/fail2ban/fail2ban/commit/f5a8a8ac

Adding Actions
--------------

If you add an action.d/*.conf file also add a example in config/jail.conf
with enabled=false and maxretry=5 for ssh.


Design
======

Fail2Ban was initially developed with Python 2.3 (IIRC). It should
still be compatible with Python 2.4 and such compatibility assurance
makes code ... old-fashioned in many places (RF-Note).  In 0.7 the
design went through major refactoring into client/server,
a-thread-per-jail design which made it a bit difficult to follow.
Below you can find a sketchy description of the main components of the
system to orient yourself better.

server/
------

Core classes hierarchy (feel welcome to draw a better/more complete
one)::

 ->   inheritance
 +    delegation
 *    storage of multiple instances

 RF-Note   just a note which might be useful to address while doing RF

 JailThread -> Filter -> FileFilter -> {FilterPoll, FilterPyinotify, ...}
               |         * FileContainer
               + FailManager
               + DateDetector
			   + Jail (provided in __init__) which contains this Filter
                 (used for passing tickets from FailManager to Jail's __queue)
 Server
   + Jails
      * Jail
        + Filter  (in __filter)
        * tickets (in __queue)
        + Actions (in __action)
          * Action
          + BanManager


failmanager.py
~~~~~~~~~~~~~~

FailManager

  Keeps track of failures, recorded as 'tickets'.  All operations are
  done via acquiring a lock

FailManagerEmpty(Exception)

  raised by FailManager.toBan after reaching the list of tickets
  (RF-Note: asks to become a generator ;) )


filter.py
~~~~~~~~~~

Filter(JailThread)

  Wraps (non-threaded) FailManager (and proxies to it quite a bit),
  and provides all primary logic for processing new lines, what IPs to
  ignore, etc

  .failManager  [FailManager]
  .dateDetector [DateDetector]
  .__failRegex  [list]
  .__ignoreRegex [list]
    Contains regular expressions for failures and ignores
  .__findTime   [numeric]
    Used in `processLineAndAdd` to skip old lines

FileFilter(Filter):

  Files-aware Filter

  .__logPath [list]
    keeps the tracked files (added 1-by-1 using addLogPath)
    stored as FileContainer's
  .getFailures
    actually just returns
    True
      if managed to open and get lines (until empty)
    False
      if failed to open or absent container matching the filename

FileContainer

  Adapter for a file to deal with log rotation.

  .open,.close,.readline
     RF-Note: readline returns "" with handler absent... shouldn't it be None?
  .__pos
    Keeps the position pointer


dnsutils.py
~~~~~~~~~~~

DNSUtils

  Utility class for DNS and IP handling


filter*.py
~~~~~~~~~~

Implementations of FileFilter's for specific backends.  Derived
classes should provide an implementation of `run` and usually
override `addLogPath`, `delLogPath` methods.  In run() method they all
one way or another provide

		try:
			while True:
				ticket = self.failManager.toBan()
				self.jail.putFailTicket(ticket)
		except FailManagerEmpty:
			self.failManager.cleanup(MyTime.time())

thus channeling "ban tickets" from their failManager to the
corresponding jail.

action.py
~~~~~~~~~

Takes care about executing start/check/ban/unban/stop commands


Releasing
=========

# Check distribution patches and see if they can be included

  * https://apps.fedoraproject.org/packages/fail2ban/sources
  * http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-analyzer/fail2ban/
  * http://svnweb.freebsd.org/ports/head/security/py-fail2ban/
  * https://build.opensuse.org/package/show?package=fail2ban&project=openSUSE%3AFactory
  * http://sophie.zarb.org/sources/fail2ban (Mageia)
  * https://trac.macports.org/browser/trunk/dports/security/fail2ban

# Check distribution outstanding bugs

  * https://github.com/fail2ban/fail2ban/issues?sort=updated&state=open
  * http://bugs.debian.org/cgi-bin/pkgreport.cgi?dist=unstable;package=fail2ban
  * http://bugs.sabayon.org/buglist.cgi?quicksearch=net-analyzer%2Ffail2ban
  * https://bugs.gentoo.org/buglist.cgi?query_format=advanced&short_desc=fail2ban&bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=IN_PROGRESS&short_desc_type=allwords
  * https://bugzilla.redhat.com/buglist.cgi?query_format=advanced&bug_status=NEW&bug_status=ASSIGNED&component=fail2ban&classification=Red%20Hat&classification=Fedora
  * http://www.freebsd.org/cgi/query-pr-summary.cgi?text=fail2ban

# Provide a release sample to distributors

  * Debian: Yaroslav Halchenko <debian@onerussian.com>
            http://packages.qa.debian.org/f/fail2ban.html
  * FreeBSD: Christoph Theis theis@gmx.at>, Nick Hilliard <nick@foobar.org>
            http://svnweb.freebsd.org/ports/head/security/py-fail2ban/Makefile?view=markup
  * Fedora: Axel Thimm <Axel.Thimm@atrpms.net>
            https://apps.fedoraproject.org/packages/fail2ban
  * Gentoo: netmon@gentoo.org
            http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-analyzer/fail2ban/metadata.xml?view=markup
  * openSUSE: Stephan Kulow <coolo@suse.com>
            https://build.opensuse.org/package/users?package=fail2ban&project=openSUSE%3AFactory
  * Mac Ports: @Malbrouck on github (gh-49)
            https://trac.macports.org/browser/trunk/dports/security/fail2ban/Portfile

# Wait for feedback from distributors

# Ensure the version is correct in ./common/version.py

# Add/finalize the corresponding entry in the ChangeLog

  To generate a list of committers use e.g.

  git shortlog -sn 0.8.8.. | sed -e 's,^[ 0-9\t]*,,g' | tr '\n' '\|' | sed -e 's:|:, :g'

  Ensure the top of the ChangeLog has the right version and current date.

  Ensure the top entry of the ChangeLog has the right version and current date.

# Update man pages

  (cd man ; ./generate-man )
  git commit -m 'update man pages for release' man/*

# Make sure the tests pass

  ./fail2ban-testcases-all

# Prepare/upload source and rpm binary distributions

  python setup.py check
  python setup.py sdist
  python setup.py bdist_rpm
  python setup.py upload

# Run the following and update the wiki with output:

  python -c 'import common.protocol; common.protocol.printWiki()'

# Email users and development list of release

# notify distributors

Post Release
============

Add the following to the top of the ChangeLog

ver. 0.8.12 (2013/XX/XXX) - wanna-be-released
-----------

- Fixes:
  
- New Features:
  
- Enhancements:
  

and adjust common/version.py to carry .dev suffix to signal
a version under development.