fail2ban/DEVELOP

                         __      _ _ ___ _
                        / _|__ _(_) |_  ) |__  __ _ _ _
                       |  _/ _` | | |/ /| '_ \/ _` | ' \
                       |_| \__,_|_|_/___|_.__/\__,_|_||_|

================================================================================
How to develop for Fail2Ban
================================================================================

Fail2Ban uses GIT (http://git-scm.com/) distributed source control. This gives
each developer their own complete copy of the entire repository. Developers can
add and switch branches and commit changes when ever they want and then ask a
maintainer to merge their changes.

Fail2Ban uses GitHub (https://github.com/fail2ban/fail2ban) to manage access to
the Git repository. GitHub provides free hosting for open-source projects as
well as a web-based Git repository browser and an issue tracker.

If you are familiar with Python and you have a bug fix or a feature that you
would like to add to Fail2Ban, the best way to do so it to use the GitHub Pull
Request feature. You can find more details on the Fail2Ban wiki
(http://www.fail2ban.org/wiki/index.php/Get_Involved)

Pull Requests
=============

When submitting pull requests on GitHub we ask you to:
* Clearly describe the problem you're solving;
* Don't introduce regressions that will make it hard for systems administrators
  to update;
* If adding a major feature rebase your changes on master and get to a single commit;
* Include test cases (see below);
* Include sample logs (if relevant);
* Include a change to the relevant section of the ChangeLog; and
* Include yourself in THANKS if not already there.

Filters
=======

Filters are tricky. They need to:
* work with a variety of the versions of the software that generates the logs;
* work with the range of logging configuration options available in the
  software;
* work with multiple operating systems;
* not make assumptions about the log format in excess of the software
  (e.g. do not assume a username doesn't contain spaces and use \S+ unless
  you've checked the source code);
* account for how future versions of the software will log messages
  (e.g. guess what would happen to the log message if different authentication
  types are added);
* not be susceptible to DoS vulnerabilities (see Filter Security below); and
* match intended log lines only.

Please follow the steps from Filter Test Cases to Developing Filter Regular
Expressions and submit a GitHub pull request (PR) afterwards. If you get stuck,
you can push your unfinished changes and still submit a PR -- describe
what you have done, what is the hurdle, and we'll attempt to help (PR
will be automagically updated with future commits you would push to
complete it).

Filter test cases
-----------------

Purpose:

Start by finding the log messages that the application generates related to
some form of authentication failure. If you are adding to an existing filter
think about whether the log messages are of a similar importance and purpose
to the existing filter. If you were a user of Fail2Ban, and did a package
update of Fail2Ban that started matching new log messages, would anything
unexpected happen?  Would the bantime/findtime for the jail be appropriate for
the new log messages?  If it doesn't, perhaps it needs to be in a separate
filter definition, for example like exim filter aims at authentication failures
and exim-spam at log messages related to spam.

Even if it is a new filter you may consider separating the log messages into
different filters based on purpose.

Cause:

Are some of the log lines a result of the same action? For example, is a PAM
failure log message, followed by an application specific failure message the
result of the same user/script action?  If you add regular expressions for
both you would end up with two failures for a single action.
Therefore, select the most appropriate log message and document the other log
message) with a test case not to match it and a description as to why you chose
one over another.

With the selected log lines consider what action has caused those log
messages and whether they could have been generated by accident? Could
the log message be occurring due to the first step towards the application
asking for authentication? Could the log messages occur often? If some of
these are true make a note of this in the jail.conf example that you provide.

Samples:

It is important to include log file samples so any future change in the regular
expression will still work with the log lines you have identified.

The sample log messages are provided in a file under testcases/files/logs/
named identically as the corresponding filter (but without .conf extension).
Each log line should be preceded by a line with failJSON metadata (so the logs
lines are tested in the test suite) directly above the log line. If there is
any specific information about the log message, such as version or an
application configuration option that is needed for the message to occur,
include this in a comment (line beginning with #) above the failJSON metadata.

Log samples should include only one, definitely not more than 3, examples of
log messages of the same form. If log messages are different in different
versions of the application log messages that show this are encouraged.

Also attempt to inject an IP into the application (e.g. by specifying
it as a username) so that Fail2Ban possibly detects the IP
from user input rather than the true origin. See the Filter Security section
and the top example in testcases/files/logs/apache-auth as to how to do this.
One you have discovered that this is possible, correct the regex so it doesn't
match and provide this as a test case with "match": false (see failJSON below).

If the mechanism to create the log message isn't obvious provide a
configuration and/or sample scripts testcases/files/config/{filtername} and
reference these in the comments above the log line.

FailJSON metadata:

A failJSON metadata is a comment immediately above the log message. It will
look like:

# failJSON: { "time": "2013-06-10T10:10:59", "match": true , "host": "93.184.216.119" }

Time should match the time of the log message. It is in a specific format of
Year-Month-Day'T'Hour:minute:Second.  If your log message does not include a
year, like the example below, the year should be listed as 2005, if before Sun
Aug 14 10am UTC, and 2004 if afterwards.  Here is an example failJSON
line preceding a sample log line:

# failJSON: { "time": "2005-03-24T15:25:51", "match": true , "host": "198.51.100.87" }
Mar 24 15:25:51 buffalo1 dropbear[4092]: bad password attempt for 'root' from 198.51.100.87:5543

The "host" in failJSON should contain the IP or domain that should be blocked.

For long lines that you do not want to be matched (e.g. from log injection
attacks) and any log lines to be excluded (see "Cause" section above), set
"match": false in the failJSON and describe the reason in the comment above.

After developing regexes, the following command will test all failJSON metadata
against the log lines in all sample log files

./fail2ban-testcases testSampleRegex

Developing Filter Regular Expressions
-------------------------------------

Date/Time:

At the moment, Fail2Ban depends on log lines to have time stamps.  That is why
before starting to develop failregex, check if your log line format known to
Fail2Ban.  Copy the time component from the log line and append an IP address to
test with following command:

./fail2ban-regex "2013-09-19 02:46:12 1.2.3.4" "<HOST>"

Output of such command should contain something like:

Date template hits:
|- [# of hits] date format
|  [1] Year-Month-Day Hour:Minute:Second

Ensure that the template description matches time/date elements in your log line
time stamp.  If there is no matched format then date template needs to be added
to server/datedetector.py.  Ensure that a new template is added in the order
that more specific matches occur first and that there is no confusion between a
Day and a Month.

Filter file:

The filter is specified in a config/filter.d/{filtername}.conf file. Filter file
can have sections INCLUDES (optional) and Definition as follows:

[INCLUDES]

before = common.conf

after = filtername.local

[Definition]

failregex = ....

ignoreregex = ....

This is also documented in the man page jail.conf (section 5). Other definitions
can be added to make failregex's more readable and maintainable to be used
through string Interpolations (see http://docs.python.org/2.7/library/configparser.html)


General rules:

Use "before" if you need to include a common set of rules, like syslog or if
there is a common set of regexes for multiple filters.

Use "after" if you wish to allow the user to overwrite a set of customisations
of the current filter. This file doesn't need to exist.

Try to avoid using ignoreregex mainly for performance reasons. The case when you
would use it is if in trying to avoid using it, you end up with an unreadable
failregex.

Syslog:

If your application logs to syslog you can take advantage of log line prefix
definitions present in common.conf.  So as a base use:

[INCLUDES]

before = common.conf

[Definition]

_daemon = app

failregex = ^%(__prefix_line)s

In this example common.conf defines __prefix_line which also contains the
_daemon name (in syslog terms the service) you have just specified. _daemon
can also be a regex.

For example, to capture following line _daemon should be set to "dovecot"

Dec 12 11:19:11 dunnart dovecot: pop3-login: Aborted login (tried to use disabled plaintext auth): rip=190.210.136.21, lip=113.212.99.193

and then ^%(__prefix_line)s would match "Dec 12 11:19:11 dunnart dovecot:
". Note it matches the trailing space(s) as well.

Substitutions (AKA string interpolations):

We have used string interpolations in above examples.  They are useful for
making the regexes more readable, reuse generic patterns in multiple failregex
lines, and also to refer definition of regex parts to specific filters or even
to the user.  General principle is that value of a _name variable replaces
occurrences of %(_name)s within the same section or anywhere in the config file
if defined in [DEFAULT] section.

Regular Expressions:

Regular expressions (failregex, ignoreregex) assume that the date/time has been
removed from the log line (this is just how fail2ban works internally ATM).

If the format is like '<date...> error 1.2.3.4 is evil' then you need to match
the < at the start so regex should be similar to '^<> <HOST> is evil$' using
<HOST> where the IP/domain name appears in the log line.

The following general rules apply to regular expressions:

* ensure regexes start with a ^ and are as restrictive as possible. E.g. do not
  use .* if \d+ is sufficient;
* use functionality of Python regexes defined in the standard Python re library
  http://docs.python.org/2/library/re.html;
* make regular expressions readable (as much as possible). E.g.
  (?:...) represents a non-capturing regex but (...) is more readable, thus
  preferred.

If you have only a basic knowledge of regular repressions we advise to read
http://docs.python.org/2/library/re.html first.  It doesn't take long and would
remind you e.g. which characters you need to escape and which you don't.

Developing/testing a regex:

You can develop a regex in a file or using command line depending on your
preference. You can also use samples you have already created in the test cases
or test them one at a time.

The general tool for testing Fail2Ban regexes is fail2ban-regex. To see how to
use it run:

./fail2ban-regex --help

Take note of  -l heavydebug  / -l debug  and -v as they might be very useful.

TIP: Take a look at the source code of the application you are developing
     failregex for. You may see optional or extra log messages, or parts there
     of, that need to form part of your regex.  It may also reveal how some
     parts are constrained and different formats depending on configuration or
     less common usages.

TIP: For looking through source code - http://sourcecodebrowser.com/ . It has
     call graphs and can browse different versions.

TIP: Some applications log spaces at the end. If you are not sure add \s*$ as
     the end part of the regex.

If your regex is not matching, http://www.debuggex.com/?flavor=python can help
to tune it:

* use regex from the ./fail2ban-regex output (to ensure all substitutions are
done) and replace <HOST> with (?&.ipv4). Make sure that regex type set to
Python;
* for the test data put your log output with the time removed;
- when you have fixed the regex put it back into your filter file.

Please spread the good word about debuggex - Serge Toarca is kindly continuing
its free availability to Open Source developers.

Finishing up:

If you've added a new filter, add a new entry in config/jail.conf. The theory
here is that a user will create a jail.local with [filtername]\nenable=true to
enable your jail.

So more specifically in the [filter] section in jail.conf:
* ensure that you have "enabled = false" (users will enable as needed);
* use "filter =" set to your filter name;
* use a typical action to disable ports associated with the application;
* set "logpath" to the usual location of application log file;
* if the default findtime or bantime isn't appropriate to the filter, specify
  more appropriate choices (possibly with a brief comment line).

Submit github pull request (See "Pull Requests" above) for
github.com/fail2ban/fail2ban containing your great work.

Filter Security
---------------

Poor filter regular expressions are susceptible to DoS attacks.

When a remote user has the ability to introduce text that would match filter's
failregex, while matching inserted text to the <HOST> part, they have the
ability to deny any host they choose.

So the <HOST> part must be anchored on text generated by the application, and
not the user, to a extent sufficient to prevent user inserting the entire text
matching this or any other failregex.

Ideally filter regex should anchor at the beginning and at the end of log line.
However as more applications log at the beginning than the end, anchoring the
beginning is more important. If the log file used by the application is shared
with other applications, like system logs, ensure the other application that use
that log file do not log user generated text at the beginning of the line, or,
if they do, ensure the regexes of the filter are sufficient to mitigate the risk
of insertion.


Examples of poor filters
------------------------

1. Too restrictive

We find a log message:

    Apr-07-13 07:08:36 Invalid command fial2ban from 1.2.3.4

We make a failregex

    ^Invalid command \S+ from <HOST>

Now think evil. The user does the command 'blah from 1.2.3.44'

The program diligently logs:

    Apr-07-13 07:08:36 Invalid command blah from 1.2.3.44 from 1.2.3.4

And fail2ban matches 1.2.3.44 as the IP that it ban. A DoS attack was successful.

The fix here is that the command can be anything so .* is appropriate.

    ^Invalid command .* from <HOST>

Here the .* will match until the end of the string. Then realise it has more to
match, i.e. "from <HOST>" and go back until it find this. Then it will ban
1.2.3.4 correctly. Since the <HOST> is always at the end, end the regex with a $.

    ^Invalid command .* from <HOST>$

Note if we'd just had the expression:

    ^Invalid command \S+ from <HOST>$

Then provided the user put a space in their command they would have never been
banned.

2. Filter regex can match other user injected data

From the Apache vulnerability CVE-2013-2178
( original ref: https://vndh.net/note:fail2ban-089-denial-service ).

An example bad regex for Apache:

    failregex = [[]client <HOST>[]] user .* not found

Since the user can do a get request on:

    GET /[client%20192.168.0.1]%20user%20root%20not%20found HTTP/1.0
Host: remote.site

Now the log line will be:

    [Sat Jun 01 02:17:42 2013] [error] [client 192.168.33.1] File does not exist: /srv/http/site/[client 192.168.0.1] user root not found

As this log line doesn't match other expressions hence it matches the above
regex and blocks 192.168.33.1 as a denial of service from the HTTP requester.

3. Application generates two identical log messages with different meanings

If the application generates the following two messages under different
circumstances:

    client <IP>: authentication failed
    client <USER>: authentication failed


Then it's obvious that a regex of "^client <HOST>: authentication
failed$" will still cause problems if the user can trigger the second
log message with a <USER> of 123.1.1.1.

Here there's nothing to do except request/change the application so it logs
messages differently.


Code Testing
============

Existing tests can be run by executing `fail2ban-testcases`. This has options
like --log-level that will probably be useful. `fail2ban-testcases --help` for
full options.

Test cases should cover all usual cases, all exception cases and all inside
/ outside boundary conditions.

Test cases should cover all branches. The coverage tool will help identify
missing branches. Also see http://nedbatchelder.com/code/coverage/branch.html
for more details.

Install the package python-coverage to visualise your test coverage. Run the
following (note: on Debian-based systems, the script is called
`python-coverage`):

coverage run fail2ban-testcases
coverage html

Then look at htmlcov/index.html and see how much coverage your test cases
exert over the code base. Full coverage is a good thing however it may not be
complete. Try to ensure tests cover as many independent paths through the
code.

Manual Execution. To run in a development environment do:

./fail2ban-client -c config/ -s /tmp/f2b.sock -i start

some quick commands:

status
add test pyinotify
status test
set test addaction iptables
set test actionban iptables echo <ip> <cidr> >> /tmp/ban
set test actionunban iptables echo <ip> <cidr> >> /tmp/unban
get test actionban iptables
get test actionunban iptables
set test banip 192.168.2.2
status test



Coding Standards
================

Style
-----

Please use tabs for now. Keep to 80 columns, at least for readable text.

Tests
-----

Add tests. They should test all the code you add in a meaning way.

Coverage
--------

Test coverage should always increase as you add code.

You may use "# pragma: no cover" in the code for branches of code that support
older versions on python. For all other uses of "pragma: no cover" or
"pragma: no branch" document the reason why its not covered. "I haven't written
a test case" isn't a sufficient reason.

Documentation
-------------

Ensure this documentation is up to date after changes. Also ensure that the man
pages still are accurate. Ensure that there is sufficient documentation for
your new features to be used.

Bugs
----

Remove them and don't add any more.

Git
---

Use the following tags in your commit messages:

'BF:' for bug fixes
'DOC:' for documentation fixes
'ENH:' for enhancements
'TST:' for commits concerning tests only (thus not touching the main code-base)

Multiple tags could be joined with +, e.g. "BF+TST:".

Use the text "closes #333"/"resolves #333 "/"fixes #333" where 333 represents
an issue that is closed. Other text and details in link below.
See: https://help.github.com/articles/closing-issues-via-commit-messages

If merge resulted in conflicts, clarify what changes were done to
corresponding files in the 'Conflicts:' section of the merge commit
message.  See e.g. https://github.com/fail2ban/fail2ban/commit/f5a8a8ac

Adding Actions
--------------

If you add an action.d/*.conf file also add a example in config/jail.conf
with enabled=false and maxretry=5 for ssh.


Design
======

Fail2Ban was initially developed with Python 2.3 (IIRC). It should
still be compatible with Python 2.4 and such compatibility assurance
makes code ... old-fashioned in many places (RF-Note).  In 0.7 the
design went through major re-factoring into client/server,
a-thread-per-jail design which made it a bit difficult to follow.
Below you can find a sketchy description of the main components of the
system to orient yourself better.

server/
------

Core classes hierarchy (feel welcome to draw a better/more complete
one)::

 ->   inheritance
 +    delegation
 *    storage of multiple instances

 RF-Note   just a note which might be useful to address while doing RF

 JailThread -> Filter -> FileFilter -> {FilterPoll, FilterPyinotify, ...}
               |         * FileContainer
               + FailManager
               + DateDetector
			   + Jail (provided in __init__) which contains this Filter
                 (used for passing tickets from FailManager to Jail's __queue)
 Server
   + Jails
      * Jail
        + Filter  (in __filter)
        * tickets (in __queue)
        + Actions (in __action)
          * Action
          + BanManager


failmanager.py
~~~~~~~~~~~~~~

FailManager

  Keeps track of failures, recorded as 'tickets'.  All operations are
  done via acquiring a lock

FailManagerEmpty(Exception)

  raised by FailManager.toBan after reaching the list of tickets
  (RF-Note: asks to become a generator ;) )


filter.py
~~~~~~~~~~

Filter(JailThread)

  Wraps (non-threaded) FailManager (and proxies to it quite a bit),
  and provides all primary logic for processing new lines, what IPs to
  ignore, etc

  .failManager  [FailManager]
  .dateDetector [DateDetector]
  .__failRegex  [list]
  .__ignoreRegex [list]
    Contains regular expressions for failures and ignores
  .__findTime   [numeric]
    Used in `processLineAndAdd` to skip old lines

FileFilter(Filter):

  Files-aware Filter

  .__logPath [list]
    keeps the tracked files (added 1-by-1 using addLogPath)
    stored as FileContainer's
  .getFailures
    actually just returns
    True
      if managed to open and get lines (until empty)
    False
      if failed to open or absent container matching the filename

FileContainer

  Adapter for a file to deal with log rotation.

  .open,.close,.readline
     RF-Note: readline returns "" with handler absent... shouldn't it be None?
  .__pos
    Keeps the position pointer


dnsutils.py
~~~~~~~~~~~

DNSUtils

  Utility class for DNS and IP handling


filter*.py
~~~~~~~~~~

Implementations of FileFilter's for specific backends.  Derived
classes should provide an implementation of `run` and usually
override `addLogPath`, `delLogPath` methods.  In run() method they all
one way or another provide

		try:
			while True:
				ticket = self.failManager.toBan()
				self.jail.putFailTicket(ticket)
		except FailManagerEmpty:
			self.failManager.cleanup(MyTime.time())

thus channelling "ban tickets" from their failManager to the
corresponding jail.

action.py
~~~~~~~~~

Takes care about executing start/check/ban/unban/stop commands


Releasing
=========

# Check distribution patches and see if they can be included

  * https://apps.fedoraproject.org/packages/fail2ban/sources
  * http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-analyzer/fail2ban/
  * http://svnweb.freebsd.org/ports/head/security/py-fail2ban/
  * https://build.opensuse.org/package/show?package=fail2ban&project=openSUSE%3AFactory
  * http://sophie.zarb.org/sources/fail2ban (Mageia)
  * https://trac.macports.org/browser/trunk/dports/security/fail2ban

# Check distribution outstanding bugs

  * https://github.com/fail2ban/fail2ban/issues?sort=updated&state=open
  * http://bugs.debian.org/cgi-bin/pkgreport.cgi?dist=unstable;package=fail2ban
  * http://bugs.sabayon.org/buglist.cgi?quicksearch=net-analyzer%2Ffail2ban
  * https://bugs.gentoo.org/buglist.cgi?query_format=advanced&short_desc=fail2ban&bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=IN_PROGRESS&short_desc_type=allwords
  * https://bugzilla.redhat.com/buglist.cgi?query_format=advanced&bug_status=NEW&bug_status=ASSIGNED&component=fail2ban&classification=Red%20Hat&classification=Fedora
  * http://www.freebsd.org/cgi/query-pr-summary.cgi?text=fail2ban

# Provide a release sample to distributors

  * Debian: Yaroslav Halchenko <debian@onerussian.com>
            http://packages.qa.debian.org/f/fail2ban.html
  * FreeBSD: Christoph Theis theis@gmx.at>, Nick Hilliard <nick@foobar.org>
            http://svnweb.freebsd.org/ports/head/security/py-fail2ban/Makefile?view=markup
  * Fedora: Axel Thimm <Axel.Thimm@atrpms.net>
            https://apps.fedoraproject.org/packages/fail2ban
  * Gentoo: netmon@gentoo.org
            http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-analyzer/fail2ban/metadata.xml?view=markup
  * openSUSE: Stephan Kulow <coolo@suse.com>
            https://build.opensuse.org/package/users?package=fail2ban&project=openSUSE%3AFactory
  * Mac Ports: @Malbrouck on github (gh-49)
            https://trac.macports.org/browser/trunk/dports/security/fail2ban/Portfile

# Wait for feedback from distributors

# Ensure the version is correct in ./common/version.py

# Ensure the MANIFEST is complete

Run:

  python setup.py sdist

Look for errors like:
 'testcases/files/logs/mysqld.log' not a regular file -- skipping

Which indicates that testcases/files/logs/mysqld.log has been moved or is a directory

  tar -C /tmp -jxf dist/fail2ban-0.8.10.dev.tar.bz2

# clean up current direcory

  diff -rul --exclude \*.pyc . /tmp/fail2ban-0.8.10.dev/

  # Only differences should be files that you don't want distributed.

  cd /tmp/fail2ban-0.8.10.dev/ && ./fail2ban-testcases-all

# Add/finalize the corresponding entry in the ChangeLog

  To generate a list of committers use e.g.

  git shortlog -sn 0.8.10.. | sed -e 's,^[ 0-9\t]*,,g' | tr '\n' '\|' | sed -e 's:|:, :g'

  Ensure the top of the ChangeLog has the right version and current date.

  Ensure the top entry of the ChangeLog has the right version and current date.

# Update man pages

  (cd man ; ./generate-man )
  git commit -m 'update man pages for release' man/*

# Make sure the tests pass

  ./fail2ban-testcases-all

# Prepare/upload source and rpm binary distributions

  python setup.py check
  python setup.py sdist
  python setup.py bdist_rpm
  python setup.py upload

# Run the following and update the wiki with output:

  python -c 'import common.protocol; common.protocol.printWiki()'

# Email users and development list of release

# notify distributors

Post Release
============

Add the following to the top of the ChangeLog

ver. 0.8.12 (2013/XX/XXX) - wanna-be-released
-----------

- Fixes:

- New Features:

- Enhancements:

Alter the git shortlog command in the previous section to refer to the just
released version.

and adjust common/version.py to carry .dev suffix to signal
a version under development.