DOC: ChangeLog deconflict

pull/362/head
Daniel Black 2013-10-02 09:11:15 +10:00
commit caf284d518
27 changed files with 755 additions and 118 deletions

View File

@ -47,10 +47,13 @@ ver. 0.8.11 (2013/XX/XXX) - loves-unittests
* files/redhat-initd - rewritten to use stock init.d functions thus
avoiding problems with getpid. Also $network and iptables moved
to Should- rc init fields
- New Features:
Andy Fragen and Daniel Black
* filter.d/osx-ipfw.conf - ipfw action for OSX based on random rule
numbers.
Anonymous:
* action.d/osx-afctl - an action based on afctl for osx
Daniel Black & ykimon
* filter.d/3proxy.conf -- filter added
Daniel Black
@ -59,6 +62,10 @@ ver. 0.8.11 (2013/XX/XXX) - loves-unittests
* add date expression for apache-2.4 - milliseconds
Christophe Carles & Daniel Black
* filter.d/perdition.conf -- filter added
Mark McKinstry
* action.d/apf.conf - add action for Advanced Policy Firewall (apf)
Amir Caspi and kjohnsonecl
* filter.d/uwimap-auth - filter for uwimap-auth IMAP/POP server
- Enhancements:
François Boulogne and Frédéric
@ -74,6 +81,8 @@ ver. 0.8.11 (2013/XX/XXX) - loves-unittests
* filter.d/gssftpd - anchored regex at start
* filter.d/mysqld-auth.conf - mysql can use syslog
* fail2ban-regex - now generated http://www.debuggex.com urls for debugging
* filter.d/sshd - regex enhancements to support openssh-6.3. Closes Debian
bug #722970
Daniel Black & Georgiy Mernov & ftoppi & Мернов Георгий
* filter.d/exim.conf -- regex hardening and extra failure examples in
sample logs

339
DEVELOP
View File

@ -26,7 +26,7 @@ Pull Requests
When submitting pull requests on GitHub we ask you to:
* Clearly describe the problem you're solving;
* Don't introduce regressions that will make it hard for systems adminstrators
* Don't introduce regressions that will make it hard for systems administrators
to update;
* If adding a major feature rebase your changes on master and get to a single commit;
* Include test cases (see below);
@ -37,49 +37,304 @@ When submitting pull requests on GitHub we ask you to:
Filters
=======
* Include sample logs with 1.2.3.4 used for IP addresses and
example.com/example.org used for DNS names
* Ensure sample log is provided in testcases/files/logs/ with same name as the
filter. Each log line should include match meta data for time & IP above
every line (see other sample log files for examples)
* Ensure regexs start with a ^ and are restrictive as possible. E.g. not .* if
\d+ is sufficient
* Use the functionality of regexs http://docs.python.org/2/library/re.html
* Take a look at the source code of the application. You may see optional or
extra log messages, or parts there of, that need to form part of your regex.
Filters are tricky. They need to:
* work with a variety of the versions of the software that generates the logs;
* work with the range of logging configuration options available in the
software;
* work with multiple operating systems;
* not make assumptions about the log format in excess of the software
(e.g. do not assume a username doesn't contain spaces and use \S+ unless
you've checked the source code);
* account for how future versions of the software will log messages
(e.g. guess what would happen to the log message if different authentication
types are added);
* not be susceptible to DoS vulnerabilities (see Filter Security below); and
* match intended log lines only.
If you only have a basic knowledge of regular repressions read
http://docs.python.org/2/library/re.html first.
Please follow the steps from Filter Test Cases to Developing Filter Regular
Expressions and submit a GitHub pull request (PR) afterwards. If you get stuck,
you can push your unfinished changes and still submit a PR -- describe
what you have done, what is the hurdle, and we'll attempt to help (PR
will be automagically updated with future commits you would push to
complete it).
Filter test cases
-----------------
Purpose:
Start by finding the log messages that the application generates related to
some form of authentication failure. If you are adding to an existing filter
think about whether the log messages are of a similar importance and purpose
to the existing filter. If you were a user of Fail2Ban, and did a package
update of Fail2Ban that started matching new log messages, would anything
unexpected happen? Would the bantime/findtime for the jail be appropriate for
the new log messages? If it doesn't, perhaps it needs to be in a separate
filter definition, for example like exim filter aims at authentication failures
and exim-spam at log messages related to spam.
Even if it is a new filter you may consider separating the log messages into
different filters based on purpose.
Cause:
Are some of the log lines a result of the same action? For example, is a PAM
failure log message, followed by an application specific failure message the
result of the same user/script action? If you add regular expressions for
both you would end up with two failures for a single action.
Therefore, select the most appropriate log message and document the other log
message) with a test case not to match it and a description as to why you chose
one over another.
With the selected log lines consider what action has caused those log
messages and whether they could have been generated by accident? Could
the log message be occurring due to the first step towards the application
asking for authentication? Could the log messages occur often? If some of
these are true make a note of this in the jail.conf example that you provide.
Samples:
It is important to include log file samples so any future change in the regular
expression will still work with the log lines you have identified.
The sample log messages are provided in a file under testcases/files/logs/
named identically as the corresponding filter (but without .conf extension).
Each log line should be preceded by a line with failJSON metadata (so the logs
lines are tested in the test suite) directly above the log line. If there is
any specific information about the log message, such as version or an
application configuration option that is needed for the message to occur,
include this in a comment (line beginning with #) above the failJSON metadata.
Log samples should include only one, definitely not more than 3, examples of
log messages of the same form. If log messages are different in different
versions of the application log messages that show this are encouraged.
Also attempt to inject an IP into the application (e.g. by specifying
it as a username) so that Fail2Ban possibly detects the IP
from user input rather than the true origin. See the Filter Security section
and the top example in testcases/files/logs/apache-auth as to how to do this.
One you have discovered that this is possible, correct the regex so it doesn't
match and provide this as a test case with "match": false (see failJSON below).
If the mechanism to create the log message isn't obvious provide a
configuration and/or sample scripts testcases/files/config/{filtername} and
reference these in the comments above the log line.
FailJSON metadata:
A failJSON metadata is a comment immediately above the log message. It will
look like:
# failJSON: { "time": "2013-06-10T10:10:59", "match": true , "host": "93.184.216.119" }
Time should match the time of the log message. It is in a specific format of
Year-Month-Day'T'Hour:minute:Second. If your log message does not include a
year, like the example below, the year should be listed as 2005, if before Sun
Aug 14 10am UTC, and 2004 if afterwards. Here is an example failJSON
line preceding a sample log line:
# failJSON: { "time": "2005-03-24T15:25:51", "match": true , "host": "198.51.100.87" }
Mar 24 15:25:51 buffalo1 dropbear[4092]: bad password attempt for 'root' from 198.51.100.87:5543
The "host" in failJSON should contain the IP or domain that should be blocked.
For long lines that you do not want to be matched (e.g. from log injection
attacks) and any log lines to be excluded (see "Cause" section above), set
"match": false in the failJSON and describe the reason in the comment above.
After developing regexes, the following command will test all failJSON metadata
against the log lines in all sample log files
./fail2ban-testcases testSampleRegex
Developing Filter Regular Expressions
-------------------------------------
Date/Time:
At the moment, Fail2Ban depends on log lines to have time stamps. That is why
before starting to develop failregex, check if your log line format known to
Fail2Ban. Copy the time component from the log line and append an IP address to
test with following command:
./fail2ban-regex "2013-09-19 02:46:12 1.2.3.4" "<HOST>"
Output of such command should contain something like:
Date template hits:
|- [# of hits] date format
| [1] Year-Month-Day Hour:Minute:Second
Ensure that the template description matches time/date elements in your log line
time stamp. If there is no matched format then date template needs to be added
to server/datedetector.py. Ensure that a new template is added in the order
that more specific matches occur first and that there is no confusion between a
Day and a Month.
Filter file:
The filter is specified in a config/filter.d/{filtername}.conf file. Filter file
can have sections INCLUDES (optional) and Definition as follows:
[INCLUDES]
before = common.conf
after = filtername.local
[Definition]
failregex = ....
ignoreregex = ....
This is also documented in the man page jail.conf (section 5). Other definitions
can be added to make failregex's more readable and maintainable to be used
through string Interpolations (see http://docs.python.org/2.7/library/configparser.html)
General rules:
Use "before" if you need to include a common set of rules, like syslog or if
there is a common set of regexes for multiple filters.
Use "after" if you wish to allow the user to overwrite a set of customisations
of the current filter. This file doesn't need to exist.
Try to avoid using ignoreregex mainly for performance reasons. The case when you
would use it is if in trying to avoid using it, you end up with an unreadable
failregex.
Syslog:
If your application logs to syslog you can take advantage of log line prefix
definitions present in common.conf. So as a base use:
[INCLUDES]
before = common.conf
[Definition]
_daemon = app
failregex = ^%(__prefix_line)s
In this example common.conf defines __prefix_line which also contains the
_daemon name (in syslog terms the service) you have just specified. _daemon
can also be a regex.
For example, to capture following line _daemon should be set to "dovecot"
Dec 12 11:19:11 dunnart dovecot: pop3-login: Aborted login (tried to use disabled plaintext auth): rip=190.210.136.21, lip=113.212.99.193
and then ^%(__prefix_line)s would match "Dec 12 11:19:11 dunnart dovecot:
". Note it matches the trailing space(s) as well.
Substitutions (AKA string interpolations):
We have used string interpolations in above examples. They are useful for
making the regexes more readable, reuse generic patterns in multiple failregex
lines, and also to refer definition of regex parts to specific filters or even
to the user. General principle is that value of a _name variable replaces
occurrences of %(_name)s within the same section or anywhere in the config file
if defined in [DEFAULT] section.
Regular Expressions:
Regular expressions (failregex, ignoreregex) assume that the date/time has been
removed from the log line (this is just how fail2ban works internally ATM).
If the format is like '<date...> error 1.2.3.4 is evil' then you need to match
the < at the start so regex should be similar to '^<> <HOST> is evil$' using
<HOST> where the IP/domain name appears in the log line.
The following general rules apply to regular expressions:
* ensure regexes start with a ^ and are as restrictive as possible. E.g. do not
use .* if \d+ is sufficient;
* use functionality of Python regexes defined in the standard Python re library
http://docs.python.org/2/library/re.html;
* make regular expressions readable (as much as possible). E.g.
(?:...) represents a non-capturing regex but (...) is more readable, thus
preferred.
If you have only a basic knowledge of regular repressions we advise to read
http://docs.python.org/2/library/re.html first. It doesn't take long and would
remind you e.g. which characters you need to escape and which you don't.
Developing/testing a regex:
You can develop a regex in a file or using command line depending on your
preference. You can also use samples you have already created in the test cases
or test them one at a time.
The general tool for testing Fail2Ban regexes is fail2ban-regex. To see how to
use it run:
./fail2ban-regex --help
Take note of -l heavydebug / -l debug and -v as they might be very useful.
TIP: Take a look at the source code of the application you are developing
failregex for. You may see optional or extra log messages, or parts there
of, that need to form part of your regex. It may also reveal how some
parts are constrained and different formats depending on configuration or
less common usages.
TIP: Some applications log spaces at the end. If you are not sure add \s*$ as
the end part of the regex.
If your regex is not matching, http://www.debuggex.com/?flavor=python can help
to tune it:
* use regex from the ./fail2ban-regex output (to ensure all substitutions are
done) and replace <HOST> with (?&.ipv4). Make sure that regex type set to
Python;
* for the test data put your log output with the time removed;
- when you have fixed the regex put it back into your filter file.
Please spread the good word about debuggex - Serge Toarca is kindly continuing
its free availability to Open Source developers.
Finishing up:
If you've added a new filter, add a new entry in config/jail.conf. The theory
here is that a user will create a jail.local with [filtername]\nenable=true to
enable your jail.
So more specifically in the [filter] section in jail.conf:
* ensure that you have "enabled = false" (users will enable as needed);
* use "filter =" set to your filter name;
* use a typical action to disable ports associated with the application;
* set "logpath" to the usual location of application log file;
* if the default findtime or bantime isn't appropriate to the filter, specify
more appropriate choices (possibly with a brief comment line).
Submit github pull request (See "Pull Requests" above) for
github.com/fail2ban/fail2ban containing your great work.
Filter Security
---------------
Poor filter regular expressions are suseptable to DoS attacks.
Poor filter regular expressions are susceptible to DoS attacks.
When a remote user has the ability to introduce text that will match the
filter regex, such that the inserted text matches the <HOST> part, they have the
When a remote user has the ability to introduce text that would match filter's
failregex, while matching inserted text to the <HOST> part, they have the
ability to deny any host they choose.
So the <HOST> part must be anchored on text generated by the application, and not
the user, to a sufficient extent that the user cannot insert the entire text.
So the <HOST> part must be anchored on text generated by the application, and
not the user, to a extent sufficient to prevent user inserting the entire text
matching this or any other failregex.
Filters are matched against the log line with their date removed.
Ideally filter regex should anchor to the beginning and end of the log line
however as more applications log at the beginning than the end, achoring the
Ideally filter regex should anchor at the beginning and at the end of log line.
However as more applications log at the beginning than the end, anchoring the
beginning is more important. If the log file used by the application is shared
with other applications, like system logs, ensure the other application that
use that log file do not log user generated text at the beginning of the line,
or, if they do, ensure the regexs of the filter are sufficient to mitigate the
risk of insertion.
with other applications, like system logs, ensure the other application that use
that log file do not log user generated text at the beginning of the line, or,
if they do, ensure the regexes of the filter are sufficient to mitigate the risk
of insertion.
When creating a regex that extends back to the begining remember the date part
has been removed within fail2ban so theres no need to match that. If the format
is like '<date...> error 1.2.3.4 is evil' then you will need to match the < at
the start so here the regex would start like '^<> <HOST> is evil$'.
Some applications log spaces at the end. If you're not sure add \s*$ as the
end part of the regex.
Examples of poor filters
------------------------
@ -96,13 +351,13 @@ We make a failregex
Now think evil. The user does the command 'blah from 1.2.3.44'
The program diliently logs:
The program diligently logs:
Apr-07-13 07:08:36 Invalid command blah from 1.2.3.44 from 1.2.3.4
And fail2ban matches 1.2.3.44 as the IP that it ban. A DoS attack was successful.
The fix here is that the command can be anything so .* is approprate.
The fix here is that the command can be anything so .* is appropriate.
^Invalid command .* from <HOST>
@ -121,10 +376,10 @@ banned.
2. Filter regex can match other user injected data
From the apache vulnerability CVE-2013-2178
From the Apache vulnerability CVE-2013-2178
( original ref: https://vndh.net/note:fail2ban-089-denial-service ).
An example bad regex for apache:
An example bad regex for Apache:
failregex = [[]client <HOST>[]] user .* not found
@ -140,10 +395,10 @@ Now the log line will be:
As this log line doesn't match other expressions hence it matches the above
regex and blocks 192.168.33.1 as a denial of service from the HTTP requester.
3. Applicaiton generates two identical log messages with different meanings
3. Application generates two identical log messages with different meanings
If the application generates the following two messages under different
circmstances:
circumstances:
client <IP>: authentication failed
client <USER>: authentication failed
@ -179,7 +434,7 @@ coverage run fail2ban-testcases
coverage html
Then look at htmlcov/index.html and see how much coverage your test cases
exert over the codebase. Full coverage is a good thing however it may not be
exert over the code base. Full coverage is a good thing however it may not be
complete. Try to ensure tests cover as many independent paths through the
code.
@ -270,7 +525,7 @@ Design
Fail2Ban was initially developed with Python 2.3 (IIRC). It should
still be compatible with Python 2.4 and such compatibility assurance
makes code ... old-fashioned in many places (RF-Note). In 0.7 the
design went through major refactoring into client/server,
design went through major re-factoring into client/server,
a-thread-per-jail design which made it a bit difficult to follow.
Below you can find a sketchy description of the main components of the
system to orient yourself better.
@ -381,7 +636,7 @@ one way or another provide
except FailManagerEmpty:
self.failManager.cleanup(MyTime.time())
thus channeling "ban tickets" from their failManager to the
thus channelling "ban tickets" from their failManager to the
corresponding jail.
action.py

3
THANKS
View File

@ -8,6 +8,7 @@ be added
Adrien Clerc
ache
Amir Caspi
Andrey G. Grozin
Andy Fragen
Arturo 'Buanzo' Busleiman
@ -38,10 +39,12 @@ Joël Bertrand
JP Espinosa
Justin Shore
Kévin Drapel
kjohnsonecl
kojiro
Manuel Arostegui Ramirez
Marcel Dopita
Mark Edgington
Mark McKinstry
Markus Hoffmann
Marvin Rouge
mEDI

View File

@ -24,7 +24,7 @@ __author__ = "Cyril Jaquier"
__copyright__ = "Copyright (c) 2004 Cyril Jaquier"
__license__ = "GPL"
import logging, re, glob
import logging, re, glob, os.path
from configreader import ConfigReader
from filterreader import FilterReader
@ -56,6 +56,22 @@ class JailReader(ConfigReader):
def isEnabled(self):
return self.__force_enable or self.__opts["enabled"]
@staticmethod
def _glob(path):
"""Given a path for glob return list of files to be passed to server.
Dangling symlinks are warned about and not returned
"""
pathList = []
for p in glob.glob(path):
if not os.path.exists(p):
logSys.warning("File %s doesn't even exist, thus cannot be monitored" % p)
elif not os.path.lexists(p):
logSys.warning("File %s is a dangling link, thus cannot be monitored" % p)
else:
pathList.append(p)
return pathList
def getOptions(self):
opts = [["bool", "enabled", "false"],
["string", "logpath", "/var/log/messages"],
@ -118,7 +134,7 @@ class JailReader(ConfigReader):
if opt == "logpath":
found_files = 0
for path in self.__opts[opt].split("\n"):
pathList = glob.glob(path)
pathList = JailReader._glob(path)
if len(pathList) == 0:
logSys.error("No file(s) found for glob %s" % path)
for p in pathList:

43
config/action.d/apf.conf Normal file
View File

@ -0,0 +1,43 @@
# Fail2Ban configuration file
#
# Author: Mark McKinstry
#
[Definition]
# Option: actionstart
# Notes.: command executed once at the start of Fail2Ban.
# Values: CMD
#
actionstart =
# Option: actionstop
# Notes.: command executed once at the end of Fail2Ban
# Values: CMD
#
actionstop =
# Option: actioncheck
# Notes.: command executed once before each actionban command
# Values: CMD
#
actioncheck =
# Option: actionban
# Notes.: command executed when banning an IP. Take care that the
# command is executed with Fail2Ban user rights.
# Tags: <ip> IP address
# <failures> number of failures
# <time> unix timestamp of the ban time
# Values: CMD
#
actionban = apf --deny <ip> "banned by Fail2Ban <name>"
# Option: actionunban
# Notes.: command executed when unbanning an IP. Take care that the
# command is executed with Fail2Ban user rights.
# Tags: <ip> IP address
# <failures> number of failures
# <time> unix timestamp of the ban time
# Values: CMD
#
actionunban = apf --remove <ip>

View File

@ -0,0 +1,16 @@
# Fail2Ban configuration file for using afctl on Mac OS X Server 10.5
#
# Anonymous author
# http://www.fail2ban.org/wiki/index.php?title=HOWTO_Mac_OS_X_Server_(10.5)&diff=prev&oldid=4081
#
# Ref: https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man8/afctl.8.html
[Definition]
actionstart =
actionstop =
actioncheck =
actionban = /usr/libexec/afctl -a <ip> -t <bantime>
actionunban = /usr/libexec/afctl -r <ip>
[Init]
bantime = 2880

View File

@ -41,6 +41,9 @@ __kernel_prefix = kernel: \[\d+\.\d+\]
__hostname = \S+
# A MD5 hex
# EXAMPLES: 07:06:27:55:b0:e3:0c:3c:5a:28:2d:7c:7e:4c:77:5f
__md5hex = (?:[\da-f]{2}:){15}[\da-f]{2}
# bsdverbose is where syslogd is started with -v or -vv and results in <4.3> or
# <auth.info> appearing before the host as per testcases/files/logs/bsd/*.

View File

@ -18,6 +18,7 @@ _daemon = dovecot(-auth)?
#
failregex = ^%(__prefix_line)s(pam_unix(\(\S+\))?:)?\s+authentication failure; logname=\S* uid=\S* euid=\S* tty=dovecot ruser=\S* rhost=<HOST>(\s+user=\S*)?\s*$
^%(__prefix_line)s(pop3|imap)-login: (Info: )?(Aborted login|Disconnected)(: Inactivity)? \(((no auth attempts|auth failed, \d+ attempts)( in \d+ secs)?|tried to use disabled \S+ auth)\):( user=<\S*>,)?( method=\S+,)? rip=<HOST>, lip=(\d{1,3}\.){3}\d{1,3}(, session=<\w+>)?(, TLS( handshaking)?(: Disconnected)?)?\s*$
^%(__prefix_line)sdovecot: auth\(\S+\): pam\(\S+,<HOST>\): pam_authenticate\(\) failed: User not known to the underlying authentication module: \d+ Time\(s\)\s*$
# Option: ignoreregex
# Notes.: regex to ignore. If this regex matches, the line is ignored.

View File

@ -4,6 +4,9 @@
# Modified: Yaroslav Halchenko for pure-ftpd
#
#
[INCLUDES]
before = common.conf
[Definition]
@ -18,7 +21,10 @@ __errmsg = (?:Authentication failed for user|Erreur d'authentification pour l'ut
# (?:::f{4,6}:)?(?P<host>[\w\-.^_]+)
# Values: TEXT
#
failregex = pure-ftpd(?:\[\d+\])?: \(.+?@<HOST>\) \[WARNING\] %(__errmsg)s \[.+\]\s*$
#
_daemon = pure-ftpd
failregex = ^%(__prefix_line)s\(.+?@<HOST>\) \[WARNING\] %(__errmsg)s \[.+\]\s*$
# Option: ignoreregex
# Notes.: regex to ignore. If this regex matches, the line is ignored.

View File

@ -22,9 +22,11 @@ _daemon = sshd
# (?:::f{4,6}:)?(?P<host>[\w\-.^_]+)
# Values: TEXT
#
#
failregex = ^%(__prefix_line)s(?:error: PAM: )?[aA]uthentication (?:failure|error) for .* from <HOST>( via \S+)?\s*$
^%(__prefix_line)s(?:error: PAM: )?User not known to the underlying authentication module for .* from <HOST>\s*$
^%(__prefix_line)sFailed \S+ for .* from <HOST>(?: port \d*)?(?: ssh\d*)?\s*$
^%(__prefix_line)sFailed \S+ for .* from <HOST>(?: port \d*)?(?: ssh\d*)?(: (ruser .{0,100}|(\S+ ID \S+ \(serial \d+\) CA )?\S+ %(__md5hex)s(, client user ".{0,100}", client host ".{0,100}")?))?\s*$
^%(__prefix_line)sROOT LOGIN REFUSED.* FROM <HOST>\s*$
^%(__prefix_line)s[iI](?:llegal|nvalid) user .* from <HOST>\s*$
^%(__prefix_line)sUser .+ from <HOST> not allowed because not listed in AllowUsers\s*$

View File

@ -0,0 +1,16 @@
# Fail2Ban configuration file
#
# Author: Amir Caspi
#
[INCLUDES]
before = common.conf
[Definition]
_daemon = (?:ipop3d|imapd)
failregex = ^%(__prefix_line)sLogin (?:failed|excessive login failures|disabled|SYSTEM BREAK-IN ATTEMPT) user=\S* auth=\S* host=.*\[<HOST>\]\s*$
^%(__prefix_line)sFailed .* override of user=.* host=.*\[<HOST>\]\s*$
ignoreregex =

View File

@ -416,8 +416,26 @@ filter = perdition
action = iptables-multiport[name=perdition,port="110,143,993,995"]
logpath = /var/log/maillog
[uwimap-auth]
enabled = false
filter = uwimap-auth
action = iptables-multiport[name=perdition,port="110,143,993,995"]
logpath = /var/log/maillog
[osx-ssh-ipfw]
enabled = false
filter = sshd
action = osx-ipfw
logpath = /var/log/secure.log
[ssh-apf]
enabled = false
filter = sshd
action = apf[name=SSH]
logpath = /var/log/secure
[osx-ssh-afctl]
enabled = false
filter = sshd
action = osx-afctl[bantime=600]
logpath = /var/log/secure.log

View File

@ -29,7 +29,7 @@ import unittest, logging, sys, time, os
if sys.version_info >= (2, 6):
import json
else:
else: # pragma: no cover
try:
import simplejson as json
except ImportError:
@ -49,6 +49,7 @@ if json:
from testcases import samplestestcase
from testcases.utils import FormatterWithTraceBack
from testcases import actionstestcase
from server.mytime import MyTime
from optparse import OptionParser, Option
@ -108,7 +109,7 @@ stdout = logging.StreamHandler(sys.stdout)
fmt = ' %(message)s'
if opts.log_traceback:
if opts.log_traceback or opts.full_traceback:
Formatter = FormatterWithTraceBack
fmt = (opts.full_traceback and ' %(tb)s' or ' %(tbc)s') + fmt
else:
@ -153,6 +154,7 @@ else: # pragma: no cover
tests.addTest(unittest.makeSuite(servertestcase.Transmitter))
tests.addTest(unittest.makeSuite(servertestcase.JailTests))
tests.addTest(unittest.makeSuite(actiontestcase.ExecuteAction))
tests.addTest(unittest.makeSuite(actionstestcase.ExecuteActions))
# FailManager
tests.addTest(unittest.makeSuite(failmanagertestcase.AddFailure))
# BanManager
@ -184,7 +186,7 @@ tests.addTest(unittest.makeSuite(datedetectortestcase.DateDetectorTest))
if json:
# Filter Regex tests with sample logs
tests.addTest(unittest.makeSuite(samplestestcase.FilterSamplesRegex))
else:
else: # pragma: no cover
print "I: Skipping filter samples testing. No simplejson/json module"
#

View File

@ -359,6 +359,10 @@ class Action:
#@staticmethod
def executeCmd(realCmd):
logSys.debug(realCmd)
if not realCmd:
logSys.debug("Nothing to do")
return True
_cmd_lock.acquire()
try: # Try wrapped within another try needed for python version < 2.5
try:

View File

@ -78,8 +78,7 @@ class DateEpoch(DateTemplate):
def __init__(self):
DateTemplate.__init__(self)
# We already know the format for TAI64N
self.setRegex("^\d{10}(\.\d{6})?")
self.setRegex("(?:^|(?P<selinux>(?<=audit\()))\d{10}(?:\.\d{3,6})?(?(selinux)(?=:\d+\)))")
def getDate(self, line):
date = None

View File

@ -0,0 +1,79 @@
# emacs: -*- mode: python; py-indent-offset: 4; indent-tabs-mode: t -*-
# vi: set ft=python sts=4 ts=4 sw=4 noet :
# This file is part of Fail2Ban.
#
# Fail2Ban is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# Fail2Ban is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Fail2Ban; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
# Author: Daniel Black
#
__author__ = "Daniel Black"
__copyright__ = "Copyright (c) 2013 Daniel Black"
__license__ = "GPL"
import unittest, time
import sys, os, tempfile
from server.actions import Actions
from dummyjail import DummyJail
class ExecuteActions(unittest.TestCase):
def setUp(self):
"""Call before every test case."""
self.__jail = DummyJail()
self.__actions = Actions(self.__jail)
self.__tmpfile, self.__tmpfilename = tempfile.mkstemp()
def tearDown(self):
os.remove(self.__tmpfilename)
def defaultActions(self):
self.__actions.addAction('ip')
self.__ip = self.__actions.getAction('ip')
self.__ip.setActionStart('echo ip start 64 >> "%s"' % self.__tmpfilename )
self.__ip.setActionBan('echo ip ban <ip> >> "%s"' % self.__tmpfilename )
self.__ip.setActionUnban('echo ip unban <ip> >> "%s"' % self.__tmpfilename )
self.__ip.setActionCheck('echo ip check <ip> >> "%s"' % self.__tmpfilename )
self.__ip.setActionStop('echo ip stop >> "%s"' % self.__tmpfilename )
def testActionsManipulation(self):
self.__actions.addAction('test')
self.assertTrue(self.__actions.getAction('test'))
self.assertTrue(self.__actions.getLastAction())
self.assertRaises(KeyError,self.__actions.getAction,*['nonexistant action'])
self.__actions.addAction('test1')
self.__actions.delAction('test')
self.__actions.delAction('test1')
self.assertRaises(KeyError, self.__actions.getAction, *['test'])
self.assertRaises(IndexError,self.__actions.getLastAction)
self.__actions.setBanTime(127)
self.assertEqual(self.__actions.getBanTime(),127)
self.assertRaises(ValueError, self.__actions.removeBannedIP, '127.0.0.1')
def testActionsOutput(self):
self.defaultActions()
self.__actions.start()
f = open(self.__tmpfilename)
time.sleep(3)
self.assertEqual(f.read(),"ip start 64\n")
self.__actions.stop()
self.__actions.join()
self.assertEqual(self.__actions.status(),[("Currently banned", 0 ),
("Total banned", 0 ), ("IP list", [] )])

View File

@ -58,6 +58,11 @@ class ExecuteAction(unittest.TestCase):
def _is_logged(self, s):
return s in self._log.getvalue()
def testNameChange(self):
self.assertEqual(self.__action.getName(), "Test")
self.__action.setName("Tricky Test")
self.assertEqual(self.__action.getName(), "Tricky Test")
def testSubstituteRecursiveTags(self):
aInfo = {
'HOST': "192.0.2.0",
@ -101,9 +106,15 @@ class ExecuteAction(unittest.TestCase):
def testExecuteActionBan(self):
self.__action.setActionStart("touch /tmp/fail2ban.test")
self.assertEqual(self.__action.getActionStart(), "touch /tmp/fail2ban.test")
self.__action.setActionStop("rm -f /tmp/fail2ban.test")
self.assertEqual(self.__action.getActionStop(), 'rm -f /tmp/fail2ban.test')
self.__action.setActionBan("echo -n")
self.assertEqual(self.__action.getActionBan(), 'echo -n')
self.__action.setActionCheck("[ -e /tmp/fail2ban.test ]")
self.assertEqual(self.__action.getActionCheck(), '[ -e /tmp/fail2ban.test ]')
self.__action.setActionUnban("true")
self.assertEqual(self.__action.getActionUnban(), 'true')
self.assertFalse(self._is_logged('returned'))
# no action was actually executed yet
@ -112,6 +123,45 @@ class ExecuteAction(unittest.TestCase):
self.assertTrue(self._is_logged('Invariant check failed'))
self.assertTrue(self._is_logged('returned successfully'))
def testExecuteActionEmptyUnban(self):
self.__action.setActionUnban("")
self.assertTrue(self.__action.execActionUnban(None))
self.assertTrue(self._is_logged('Nothing to do'))
def testExecuteActionStartCtags(self):
self.__action.setCInfo("HOST","192.0.2.0")
self.__action.setActionStart("touch /tmp/fail2ban.test.<HOST>")
self.__action.setActionStop("rm -f /tmp/fail2ban.test.<HOST>")
self.__action.setActionCheck("[ -e /tmp/fail2ban.test.192.0.2.0 ]")
self.assertTrue(self.__action.execActionStart())
def testExecuteActionCheckRestoreEnvironment(self):
self.__action.setActionStart("")
self.__action.setActionStop("rm -f /tmp/fail2ban.test")
self.__action.setActionBan("rm /tmp/fail2ban.test")
self.__action.setActionCheck("[ -e /tmp/fail2ban.test ]")
self.assertFalse(self.__action.execActionBan(None))
self.assertTrue(self._is_logged('Unable to restore environment'))
def testExecuteActionChangeCtags(self):
self.__action.setCInfo("ROST","192.0.2.0")
self.assertEqual(self.__action.getCInfo("ROST"),"192.0.2.0")
self.__action.delCInfo("ROST")
self.assertRaises(KeyError, self.__action.getCInfo, "ROST")
def testExecuteActionUnbanAinfo(self):
aInfo = {
'ABC': "123",
}
self.__action.setActionBan("touch /tmp/fail2ban.test.123")
self.__action.setActionUnban("rm /tmp/fail2ban.test.<ABC>")
self.assertTrue(self.__action.execActionBan(None))
self.assertTrue(self.__action.execActionUnban(aInfo))
def testExecuteActionStartEmpty(self):
self.__action.setActionStart("")
self.assertTrue(self.__action.execActionStart())
self.assertTrue(self._is_logged('Nothing to do'))
def testExecuteIncorrectCmd(self):
Action.executeCmd('/bin/ls >/dev/null\nbogusXXX now 2>/dev/null')

View File

@ -22,6 +22,7 @@ __copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2011-2013 Yaroslav Halchenko"
__license__ = "GPL"
import os, tempfile, shutil, unittest
from client.configreader import ConfigReader
from client.jailreader import JailReader
from client.jailsreader import JailsReader
@ -117,6 +118,19 @@ class JailReaderTest(unittest.TestCase):
result = JailReader.splitAction(action)
self.assertEqual(expected, result)
def testGlob(self):
d = tempfile.mkdtemp(prefix="f2b-temp")
# Generate few files
# regular file
open(os.path.join(d, 'f1'), 'w').close()
# dangling link
os.symlink('nonexisting', os.path.join(d, 'f2'))
# must be only f1
self.assertEqual(JailReader._glob(os.path.join(d, '*')), [os.path.join(d, 'f1')])
# since f2 is dangling -- empty list
self.assertEqual(JailReader._glob(os.path.join(d, 'f2')), [])
class JailsReaderTest(unittest.TestCase):
def testProvidingBadBasedir(self):

View File

@ -63,35 +63,45 @@ class DateDetectorTest(unittest.TestCase):
date = [2005, 1, 23, 21, 59, 59, 6, 23, -1]
dateUnix = 1106513999.0
for sdate in (
"Jan 23 21:59:59",
"Sun Jan 23 21:59:59 2005",
"Sun Jan 23 21:59:59",
"2005/01/23 21:59:59",
"2005.01.23 21:59:59",
"23/01/2005 21:59:59",
"23/01/05 21:59:59",
"23/Jan/2005:21:59:59",
"01/23/2005:21:59:59",
"2005-01-23 21:59:59",
"23-Jan-2005 21:59:59",
"23-01-2005 21:59:59",
"01-23-2005 21:59:59.252", # reported on f2b, causes Feb29 fix to break
"@4000000041f4104f00000000", # TAI64N
"2005-01-23T21:59:59.252Z", #ISO 8601
"2005-01-23T21:59:59-05:00Z", #ISO 8601 with TZ
"<01/23/05@21:59:59>",
"050123 21:59:59", # MySQL
"Jan-23-05 21:59:59", # ASSP like
for anchored, sdate in (
(False, "Jan 23 21:59:59"),
(False, "Sun Jan 23 21:59:59 2005"),
(False, "Sun Jan 23 21:59:59"),
(False, "2005/01/23 21:59:59"),
(False, "2005.01.23 21:59:59"),
(False, "23/01/2005 21:59:59"),
(False, "23/01/05 21:59:59"),
(False, "23/Jan/2005:21:59:59"),
(False, "01/23/2005:21:59:59"),
(False, "2005-01-23 21:59:59"),
(False, "23-Jan-2005 21:59:59"),
(False, "23-01-2005 21:59:59"),
(False, "01-23-2005 21:59:59.252"), # reported on f2b, causes Feb29 fix to break
(False, "@4000000041f4104f00000000"), # TAI64N
(False, "2005-01-23T21:59:59.252Z"), #ISO 8601
(False, "2005-01-23T21:59:59-05:00Z"), #ISO 8601 with TZ
(True, "<01/23/05@21:59:59>"),
(True, "050123 21:59:59"), # MySQL
(True, "Jan-23-05 21:59:59"), # ASSP like
(True, "1106513999"), # Regular epoch
(True, "1106513999.123"), # Regular epoch with millisec
(False, "audit(1106513999.123:987)"), # SELinux
):
log = sdate + "[sshd] error: PAM: Authentication failure"
# exclude
for should_match, prefix in ((True, ""),
(not anchored, "bogus-prefix ")):
ldate = prefix + sdate # logged date
log = ldate + "[sshd] error: PAM: Authentication failure"
# exclude
# yoh: on [:6] see in above test
logtime = self.__datedetector.getTime(log)
if should_match:
self.assertNotEqual(logtime, None, "getTime retrieved nothing for %r" % ldate)
self.assertEqual(logtime[:6], date[:6], "getTime comparison failure for %r: \"%s\" is not \"%s\"" % (ldate, logtime[:6], date[:6]))
self.assertEqual(self.__datedetector.getUnixTime(log), dateUnix, "getUnixTime failure for %r: \"%s\" is not \"%s\"" % (ldate, logtime[:6], date[:6]))
else:
self.assertEqual(logtime, None, "getTime should have not matched for %r Got: %s" % (ldate, logtime))
# yoh: on [:6] see in above test
logtime = self.__datedetector.getTime(log)
self.assertNotEqual(logtime, None, "getTime retrieved nothing: failure for %s" % sdate)
self.assertEqual(logtime[:6], date[:6], "getTime comparison failure for %s: \"%s\" is not \"%s\"" % (sdate, logtime[:6], date[:6]))
self.assertEqual(self.__datedetector.getUnixTime(log), dateUnix, "getUnixTime failure for %s: \"%s\" is not \"%s\"" % (sdate, logtime[:6], date[:6]))
def testStableSortTemplate(self):
old_names = [x.getName() for x in self.__datedetector.getTemplates()]
@ -164,6 +174,14 @@ class DateDetectorTest(unittest.TestCase):
print "WARNING: The following date templates overlap:"
pprint.pprint(overlapedTemplates)
def testDateTemplate(self):
t = DateTemplate()
t.setRegex('^a{3,5}b?c*$')
self.assertEqual(t.getRegex(), '^a{3,5}b?c*$')
self.assertRaises(Exception, t.getDate, '')
self.assertEqual(t.matchDate('aaaac').group(), 'aaaac')
# def testDefaultTempate(self):
# self.__datedetector.setDefaultRegex("^\S{3}\s{1,2}\d{1,2} \d{2}:\d{2}:\d{2}")
# self.__datedetector.setDefaultPattern("%b %d %H:%M:%S")

59
testcases/dummyjail.py Normal file
View File

@ -0,0 +1,59 @@
# emacs: -*- mode: python; py-indent-offset: 4; indent-tabs-mode: t -*-
# vi: set ft=python sts=4 ts=4 sw=4 noet :
# This file is part of Fail2Ban.
#
# Fail2Ban is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# Fail2Ban is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Fail2Ban; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
# Fail2Ban developers
__copyright__ = "Copyright (c) 2012 Yaroslav Halchenko"
__license__ = "GPL"
from threading import Lock
class DummyJail(object):
"""A simple 'jail' to suck in all the tickets generated by Filter's
"""
def __init__(self):
self.lock = Lock()
self.queue = []
def __len__(self):
try:
self.lock.acquire()
return len(self.queue)
finally:
self.lock.release()
def putFailTicket(self, ticket):
try:
self.lock.acquire()
self.queue.append(ticket)
finally:
self.lock.release()
def getFailTicket(self):
try:
self.lock.acquire()
try:
return self.queue.pop()
except IndexError:
return False
finally:
self.lock.release()
def getName(self):
return "DummyJail #%s with %d tickets" % (id(self), len(self))

View File

@ -53,8 +53,18 @@ class AddFailure(unittest.TestCase):
def tearDown(self):
"""Call after every test case."""
def testAdd(self):
def testFailManagerAdd(self):
self.assertEqual(self.__failManager.size(), 3)
self.assertEqual(self.__failManager.getFailTotal(), 13)
self.__failManager.setFailTotal(0)
self.assertEqual(self.__failManager.getFailTotal(), 0)
self.__failManager.setFailTotal(13)
def testFailManagerMaxTime(self):
self.assertEqual(self.__failManager.getMaxTime(), 600)
self.__failManager.setMaxTime(13)
self.assertEqual(self.__failManager.getMaxTime(), 13)
self.__failManager.setMaxTime(600)
def _testDel(self):
self.__failManager.delFailure('193.168.0.128')

View File

@ -29,3 +29,6 @@ Jun 23 00:52:43 vhost1-ua dovecot: pop3-login: Disconnected: Inactivity (auth fa
Jul 02 13:49:31 hostname dovecot[442]: pop3-login: Aborted login (auth failed, 1 attempts in 17 secs): user=<test>, method=PLAIN, rip=192.51.100.13, lip=203.0.113.17, session=<YADINsQCDs5BH8Pg>
# failJSON: { "time": "2005-07-02T13:49:32", "match": true , "host": "192.51.100.13" }
Jul 02 13:49:32 hostname dovecot[442]: pop3-login: Disconnected (no auth attempts in 58 secs): user=<>, rip=192.51.100.13, lip=203.0.113.17, session=<LgDINsQCkttVIMPg>
# failJSON: { "time": "2005-07-02T13:49:32", "match": true , "host": "200.76.17.206" }
Jul 02 13:49:32 hostname dovecot[442]: dovecot: auth(default): pam(account@MYSERVERNAME.com,200.76.17.206): pam_authenticate() failed: User not known to the underlying authentication module: 2 Time(s)

View File

@ -1,4 +1,2 @@
# failJSON: { "time": "2005-01-31T16:54:07", "match": true , "host": "24.79.92.194" }
Jan 31 16:54:07 desktop pure-ftpd: (?@24.79.92.194) [WARNING] Authentication failed for user [Administrator]
# failJSON: { "time": "2004-11-05T18:54:02", "match": true , "host": "server202181210195.ixlink.net" }
Nov 5 18:54:02 pure-ftpd: (?@server202181210195.ixlink.net) [WARNING] Authentication failed for user [Administrator]

View File

@ -85,3 +85,12 @@ Mar 26 04:56:27 angel sshd[9739]: User allena from example.com not allowed becau
Feb 7 16:01:07 linux-m899 sshd[5106]: User root from 192.51.100.54 not allowed because a group is listed in DenyGroups
# failJSON: { "time": "2005-01-05T11:15:05", "match": true , "host": "10.0.0.40" }
Jan 5 11:15:05 NAS sshd[1966]: User root from 10.0.0.40 not allowed because none of user's groups are listed in AllowGroups
# failJSON: { "time": "2004-09-29T16:28:02", "match": true , "host": "127.0.0.1" }
Sep 29 16:28:02 spaceman sshd[16699]: Failed password for dan from 127.0.0.1 port 45416 ssh1
# failJSON: { "time": "2004-09-29T17:15:02", "match": true , "host": "127.0.0.1" }
Sep 29 17:15:02 spaceman sshd[12946]: Failed hostbased for dan from 127.0.0.1 port 45785 ssh2: RSA 8c:e3:aa:0f:64:51:02:f7:14:79:89:3f:65:84:7c:30, client user "dan", client host "localhost.localdomain"
# failJSON: { "time": "2004-09-29T17:15:02", "match": true , "host": "127.0.0.1" }
Sep 29 17:15:02 spaceman sshd[12946]: Failed hostbased for dan from 127.0.0.1 port 45785 ssh2: DSA 01:c0:79:41:91:31:9a:7d:95:23:91:ac:b1:6d:59:81, client user "dan", client host "localhost.localdomain"

View File

@ -0,0 +1,22 @@
# failJSON: { "time": "2005-07-03T20:56:53", "match": true , "host": "81.169.154.112" }
Jul 3 20:56:53 Linux2 imapd[666]: Login failed user=lizdy auth=lizdy host=h2066373.stratoserver.net [81.169.154.112]
# failJSON: { "time": "2005-07-29T18:30:19", "match": true , "host": "198.52.115.74" }
Jul 29 18:30:19 Linux2 ipop3d[25745]: Login failed user=info auth=info host=74-115-52-198-dedicated.multacom.com [198.52.115.74]
# http://lists.freebsd.org/pipermail/freebsd-questions/2005-January/072073.html
# failJSON: { "time": "2005-01-14T20:28:07", "match": true , "host": "198.52.115.74" }
Jan 14 20:28:07 grog imapd[19343]: Login excessive login failures user=user auth=user host=74-115-52-198-dedicated.multacom.com [198.52.115.74]
#http://us.generation-nt.com/answer/uw-imapd-doesnt-authenticate-users-help-194297331.html
# failJSON: { "time": "2005-04-08T16:32:01", "match": true , "host": "198.52.115.74" }
Apr 8 16:32:01 abdon imapd[29087]: Login excessive login failures user=brada auth=brada host=xxxxxx [198.52.115.74]
# http://www.howtoforge.com/forums/showthread.php?t=3786
# failJSON: { "time": "2005-04-08T16:32:01", "match": true , "host": "127.0.0.1" }
Apr 8 16:32:01 abdon imapd[21172]: Login disabled user=test auth=test host=localhost.localdomain [127.0.0.1]
# http://mailman2.u.washington.edu/pipermail/imap-uw/2008-February/001889.html
# failJSON: { "time": "2005-02-23T12:36:01", "match": true , "host": "127.0.55.22" }
Feb 23 12:36:01 r2 imapd[3473]: Failed uwmaster override of user=pro1 host=r22.j.de [127.0.55.22]

View File

@ -306,36 +306,7 @@ class LogFileMonitor(unittest.TestCase):
from threading import Lock
class DummyJail(object):
"""A simple 'jail' to suck in all the tickets generated by Filter's
"""
def __init__(self):
self.lock = Lock()
self.queue = []
def __len__(self):
try:
self.lock.acquire()
return len(self.queue)
finally:
self.lock.release()
def putFailTicket(self, ticket):
try:
self.lock.acquire()
self.queue.append(ticket)
finally:
self.lock.release()
def getFailTicket(self):
try:
self.lock.acquire()
return self.queue.pop()
finally:
self.lock.release()
def getName(self):
return "DummyJail #%s with %d tickets" % (id(self), len(self))
from dummyjail import DummyJail
def get_monitor_failures_testcase(Filter_):
"""Generator of TestCase's for different filters/backends

View File

@ -292,12 +292,23 @@ class Transmitter(TransmitterBase):
self.transm.proceed(["set", self.jailName, "dellogpath", value]),
(0, []))
def testJailLogPathInvalidFile(self):
# Invalid file
value = "this_file_shouldn't_exist"
result = self.transm.proceed(
["set", self.jailName, "addlogpath", value])
self.assertTrue(isinstance(result[1], IOError))
def testJailLogPathBrokenSymlink(self):
# Broken symlink
name = tempfile.mktemp(prefix='tmp_fail2ban_broken_symlink')
sname = name + '.slink'
os.symlink(name, sname)
result = self.transm.proceed(
["set", self.jailName, "addlogpath", sname])
self.assertTrue(isinstance(result[1], IOError))
os.unlink(sname)
def testJailIgnoreIP(self):
self.jailAddDelTest(
"ignoreip",