|
|
|
@ -26,7 +26,7 @@ Pull Requests
|
|
|
|
|
|
|
|
|
|
When submitting pull requests on GitHub we ask you to:
|
|
|
|
|
* Clearly describe the problem you're solving;
|
|
|
|
|
* Don't introduce regressions that will make it hard for systems adminstrators
|
|
|
|
|
* Don't introduce regressions that will make it hard for systems administrators
|
|
|
|
|
to update;
|
|
|
|
|
* If adding a major feature rebase your changes on master and get to a single commit;
|
|
|
|
|
* Include test cases (see below);
|
|
|
|
@ -44,12 +44,12 @@ Filters are tricky. They need to:
|
|
|
|
|
* work in multiple operating systems;
|
|
|
|
|
* not make assumptions about the log format in excess of the software;
|
|
|
|
|
* make assumptions as to how future versions of the software will log messages;
|
|
|
|
|
* not be susceptable to DoS vulernabilities; and
|
|
|
|
|
* not be susceptible to DoS vulnerabilities; and
|
|
|
|
|
* match intended log lines only.
|
|
|
|
|
|
|
|
|
|
Please follow the steps from Filter Test Cases to Developing Filter Regular
|
|
|
|
|
Expressions and submit a github pull request afterward. If you get stuck,
|
|
|
|
|
create a github issue with what you have done and we'll attempt to help.
|
|
|
|
|
Expressions and submit a GitHub pull request afterwards. If you get stuck,
|
|
|
|
|
create a GitHub issue with what you have done and we'll attempt to help.
|
|
|
|
|
|
|
|
|
|
Filter test cases
|
|
|
|
|
-----------------
|
|
|
|
@ -58,13 +58,13 @@ Purpose:
|
|
|
|
|
|
|
|
|
|
Start by finding the log messages that the application generates related to
|
|
|
|
|
some form of authentication failure. If you are adding to an existing filter
|
|
|
|
|
think about wheither the log messages are of a simlar importance and purpose
|
|
|
|
|
think about whether the log messages are of a similar importance and purpose
|
|
|
|
|
to the existing filter. If you where a user of fail2ban, and did a package
|
|
|
|
|
update of fail2ban that started matching the new log messages, would anything
|
|
|
|
|
unexpected happen? Would the bantime/findtime for the jail be approprate for
|
|
|
|
|
unexpected happen? Would the bantime/findtime for the jail be appropriate for
|
|
|
|
|
the new log messages. If it doesn't perhaps it needs to be in a separate
|
|
|
|
|
filter defination, for example like exim is authentication failures and
|
|
|
|
|
exim-spam contains log messages replated to spam.
|
|
|
|
|
filter definition, for example like exim is authentication failures and
|
|
|
|
|
exim-spam contains log messages related to spam.
|
|
|
|
|
|
|
|
|
|
Even if its a new filter you may consider separating the log messages into
|
|
|
|
|
different filters based on purpose.
|
|
|
|
@ -75,12 +75,12 @@ Are some of the log lines a result of the same action? For example is a PAM
|
|
|
|
|
failure log message, followed by an application specific failure message the
|
|
|
|
|
result of the same user/script action. The result is if you add regular
|
|
|
|
|
expressions for both you'll end up with two failures for a single action.
|
|
|
|
|
Select the most approprate log message and document the other log message with
|
|
|
|
|
Select the most appropriate log message and document the other log message with
|
|
|
|
|
a test case not to match it and a description as to why you chose one over
|
|
|
|
|
another.
|
|
|
|
|
|
|
|
|
|
With the log lines selected consider what occured to generate those log
|
|
|
|
|
messages and wheither they could of been generated by accidental means. Could
|
|
|
|
|
With the log lines selected consider what occurred to generate those log
|
|
|
|
|
messages and whether they could of been generated by accidental means. Could
|
|
|
|
|
the log message occur always as this is the first step towards the application
|
|
|
|
|
asking for authentication? Could the log messages occur often? If some of
|
|
|
|
|
these are true make a note of this in the jail.conf example that you provide.
|
|
|
|
@ -97,7 +97,7 @@ any specific information about the log message, such as version or an
|
|
|
|
|
application configuration option that is needed for the message to occur,
|
|
|
|
|
include this in a comment (line beginning with #) above the failJSON metadata.
|
|
|
|
|
|
|
|
|
|
Log samples should include only one, definately not more than 3, examples of
|
|
|
|
|
Log samples should include only one, definitely not more than 3, examples of
|
|
|
|
|
log messages of the same form. If log messages are different in different
|
|
|
|
|
versions of the application log messages that show this is encouraged.
|
|
|
|
|
|
|
|
|
@ -122,7 +122,7 @@ Mar 24 15:25:51 buffalo1 dropbear[4092]: bad password attempt for 'root' from 19
|
|
|
|
|
|
|
|
|
|
The host will contain the IP or domain that should be blocked.
|
|
|
|
|
|
|
|
|
|
For long lines that you don't want matched, like log injection vulerabilities
|
|
|
|
|
For long lines that you don't want matched, like log injection vulnerabilities
|
|
|
|
|
and log lines excluded (see "Cause" section above), a "match": false in the
|
|
|
|
|
failJSON and the reason why in the comment above.
|
|
|
|
|
|
|
|
|
@ -157,7 +157,7 @@ is the date or month.
|
|
|
|
|
Filter file:
|
|
|
|
|
|
|
|
|
|
The filter file is in config/filter.d/{filtername}.conf. The format of the
|
|
|
|
|
filter file has two sections INCLUDES and Defination as follows:
|
|
|
|
|
filter file has two sections INCLUDES and Definition as follows:
|
|
|
|
|
|
|
|
|
|
[INCLUDES]
|
|
|
|
|
|
|
|
|
@ -172,7 +172,7 @@ failregex = ....
|
|
|
|
|
ignoreregex = ....
|
|
|
|
|
|
|
|
|
|
This is also documented in the man pages as jail.conf (section 5). Other
|
|
|
|
|
definations can be added to make failregex's more readable and maintainable.
|
|
|
|
|
definitions can be added to make failregex's more readable and maintainable.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
General rules:
|
|
|
|
@ -180,7 +180,7 @@ General rules:
|
|
|
|
|
Use "before" if you need to include a common set of rules, like syslog or if
|
|
|
|
|
there's a common set of regexs for multiple filters.
|
|
|
|
|
|
|
|
|
|
Use "after" if you wish to allow the user to overwrite a set of customisations
|
|
|
|
|
Use "after" if you wish to allow the user to overwrite a set of customisation's
|
|
|
|
|
of the current filter. This file doesn't need to exist.
|
|
|
|
|
|
|
|
|
|
Try to avoid using ignoreregex mainly for performance reasons. The case when
|
|
|
|
@ -214,10 +214,10 @@ So now ^%(__prefix_line)s matches "Dec 12 11:19:11 dunnart dovecot: ". Note it
|
|
|
|
|
matches the trailing space. Putting a space after ^%(__prefix_line)s in the
|
|
|
|
|
regex will probably not match.
|
|
|
|
|
|
|
|
|
|
Substitions:
|
|
|
|
|
Substitutions:
|
|
|
|
|
|
|
|
|
|
Substitions are what the syslog uses. The regex bits of %(_name)s substitute
|
|
|
|
|
the _name defination into the regex. They are useful for making the regexes
|
|
|
|
|
Substation's are what the syslog uses. The regex bits of %(_name)s substitute
|
|
|
|
|
the _name definition into the regex. They are useful for making the regexes
|
|
|
|
|
more readable and also defining regex parts that occur in multiple log lines.
|
|
|
|
|
|
|
|
|
|
Regular Expressions:
|
|
|
|
@ -262,7 +262,7 @@ TIP: Take a look at the source code of the application. You may see optional or
|
|
|
|
|
TIP: Some applications log spaces at the end. If you're not sure add \s*$ as the
|
|
|
|
|
end part of the regex.
|
|
|
|
|
|
|
|
|
|
If your regex isn't matching take a look at http://www.debuggex.com/.
|
|
|
|
|
If your regex isn't matching take a look at http://www.debuggex.com/?flavor=python
|
|
|
|
|
|
|
|
|
|
Using the regex from the ./fail2ban-regex output (to ensure all substitutions
|
|
|
|
|
are done) and with <HOST> replaced with (?&.ipv4). Set the regex type to
|
|
|
|
@ -272,7 +272,8 @@ For the test data put your log output with the time removed.
|
|
|
|
|
|
|
|
|
|
When you've fixed the regex put it back into your filter file.
|
|
|
|
|
|
|
|
|
|
Please give a donation to stoarca for debuggex. Its a great tool isn't it.
|
|
|
|
|
Please spread the good word about debuggex - Serge Toarca is kindly continuing
|
|
|
|
|
its free availability to Open Source developers.
|
|
|
|
|
|
|
|
|
|
Finishing up:
|
|
|
|
|
|
|
|
|
@ -284,8 +285,8 @@ So more specifically in the [filter] section in jail.conf:
|
|
|
|
|
* use "filter =" set to your filter name.
|
|
|
|
|
* use a action to disable ports associated with the application
|
|
|
|
|
* set "logpath" to a usual location for the log file for the application.
|
|
|
|
|
* If the default findtime or bantime isn't approprate to the filter set a value
|
|
|
|
|
that is more approprate.
|
|
|
|
|
* If the default findtime or bantime isn't appropriate to the filter set a value
|
|
|
|
|
that is more appropriate.
|
|
|
|
|
|
|
|
|
|
Send the fail2ban a git pull request (See "Pull Requests" above) containing
|
|
|
|
|
your great work.
|
|
|
|
@ -293,7 +294,7 @@ your great work.
|
|
|
|
|
Filter Security
|
|
|
|
|
---------------
|
|
|
|
|
|
|
|
|
|
Poor filter regular expressions are suseptable to DoS attacks.
|
|
|
|
|
Poor filter regular expressions are susceptible to DoS attacks.
|
|
|
|
|
|
|
|
|
|
When a remote user has the ability to introduce text that will match the
|
|
|
|
|
filter regex, such that the inserted text matches the <HOST> part, they have the
|
|
|
|
@ -303,7 +304,7 @@ So the <HOST> part must be anchored on text generated by the application, and no
|
|
|
|
|
the user, to a sufficient extent that the user cannot insert the entire text.
|
|
|
|
|
|
|
|
|
|
Ideally filter regex should anchor to the beginning and end of the log line
|
|
|
|
|
however as more applications log at the beginning than the end, achoring the
|
|
|
|
|
however as more applications log at the beginning than the end, anchoring the
|
|
|
|
|
beginning is more important. If the log file used by the application is shared
|
|
|
|
|
with other applications, like system logs, ensure the other application that
|
|
|
|
|
use that log file do not log user generated text at the beginning of the line,
|
|
|
|
@ -326,13 +327,13 @@ We make a failregex
|
|
|
|
|
|
|
|
|
|
Now think evil. The user does the command 'blah from 1.2.3.44'
|
|
|
|
|
|
|
|
|
|
The program diliently logs:
|
|
|
|
|
The program diligently logs:
|
|
|
|
|
|
|
|
|
|
Apr-07-13 07:08:36 Invalid command blah from 1.2.3.44 from 1.2.3.4
|
|
|
|
|
|
|
|
|
|
And fail2ban matches 1.2.3.44 as the IP that it ban. A DoS attack was successful.
|
|
|
|
|
|
|
|
|
|
The fix here is that the command can be anything so .* is approprate.
|
|
|
|
|
The fix here is that the command can be anything so .* is appropriate.
|
|
|
|
|
|
|
|
|
|
^Invalid command .* from <HOST>
|
|
|
|
|
|
|
|
|
@ -351,10 +352,10 @@ banned.
|
|
|
|
|
|
|
|
|
|
2. Filter regex can match other user injected data
|
|
|
|
|
|
|
|
|
|
From the apache vulnerability CVE-2013-2178
|
|
|
|
|
From the Apache vulnerability CVE-2013-2178
|
|
|
|
|
( original ref: https://vndh.net/note:fail2ban-089-denial-service ).
|
|
|
|
|
|
|
|
|
|
An example bad regex for apache:
|
|
|
|
|
An example bad regex for Apache:
|
|
|
|
|
|
|
|
|
|
failregex = [[]client <HOST>[]] user .* not found
|
|
|
|
|
|
|
|
|
@ -370,10 +371,10 @@ Now the log line will be:
|
|
|
|
|
As this log line doesn't match other expressions hence it matches the above
|
|
|
|
|
regex and blocks 192.168.33.1 as a denial of service from the HTTP requester.
|
|
|
|
|
|
|
|
|
|
3. Applicaiton generates two identical log messages with different meanings
|
|
|
|
|
3. Application generates two identical log messages with different meanings
|
|
|
|
|
|
|
|
|
|
If the application generates the following two messages under different
|
|
|
|
|
circmstances:
|
|
|
|
|
circumstances:
|
|
|
|
|
|
|
|
|
|
client <IP>: authentication failed
|
|
|
|
|
client <USER>: authentication failed
|
|
|
|
@ -500,7 +501,7 @@ Design
|
|
|
|
|
Fail2Ban was initially developed with Python 2.3 (IIRC). It should
|
|
|
|
|
still be compatible with Python 2.4 and such compatibility assurance
|
|
|
|
|
makes code ... old-fashioned in many places (RF-Note). In 0.7 the
|
|
|
|
|
design went through major refactoring into client/server,
|
|
|
|
|
design went through major re-factoring into client/server,
|
|
|
|
|
a-thread-per-jail design which made it a bit difficult to follow.
|
|
|
|
|
Below you can find a sketchy description of the main components of the
|
|
|
|
|
system to orient yourself better.
|
|
|
|
@ -611,7 +612,7 @@ one way or another provide
|
|
|
|
|
except FailManagerEmpty:
|
|
|
|
|
self.failManager.cleanup(MyTime.time())
|
|
|
|
|
|
|
|
|
|
thus channeling "ban tickets" from their failManager to the
|
|
|
|
|
thus channelling "ban tickets" from their failManager to the
|
|
|
|
|
corresponding jail.
|
|
|
|
|
|
|
|
|
|
action.py
|
|
|
|
|