mirror of https://github.com/fail2ban/fail2ban
DOC: used spellchecker finally. Debuggex needs promotion more than donations
parent
3fac971a5a
commit
2562b2f1bc
71
DEVELOP
71
DEVELOP
|
@ -26,7 +26,7 @@ Pull Requests
|
||||||
|
|
||||||
When submitting pull requests on GitHub we ask you to:
|
When submitting pull requests on GitHub we ask you to:
|
||||||
* Clearly describe the problem you're solving;
|
* Clearly describe the problem you're solving;
|
||||||
* Don't introduce regressions that will make it hard for systems adminstrators
|
* Don't introduce regressions that will make it hard for systems administrators
|
||||||
to update;
|
to update;
|
||||||
* If adding a major feature rebase your changes on master and get to a single commit;
|
* If adding a major feature rebase your changes on master and get to a single commit;
|
||||||
* Include test cases (see below);
|
* Include test cases (see below);
|
||||||
|
@ -44,12 +44,12 @@ Filters are tricky. They need to:
|
||||||
* work in multiple operating systems;
|
* work in multiple operating systems;
|
||||||
* not make assumptions about the log format in excess of the software;
|
* not make assumptions about the log format in excess of the software;
|
||||||
* make assumptions as to how future versions of the software will log messages;
|
* make assumptions as to how future versions of the software will log messages;
|
||||||
* not be susceptable to DoS vulernabilities; and
|
* not be susceptible to DoS vulnerabilities; and
|
||||||
* match intended log lines only.
|
* match intended log lines only.
|
||||||
|
|
||||||
Please follow the steps from Filter Test Cases to Developing Filter Regular
|
Please follow the steps from Filter Test Cases to Developing Filter Regular
|
||||||
Expressions and submit a github pull request afterward. If you get stuck,
|
Expressions and submit a GitHub pull request afterwards. If you get stuck,
|
||||||
create a github issue with what you have done and we'll attempt to help.
|
create a GitHub issue with what you have done and we'll attempt to help.
|
||||||
|
|
||||||
Filter test cases
|
Filter test cases
|
||||||
-----------------
|
-----------------
|
||||||
|
@ -58,13 +58,13 @@ Purpose:
|
||||||
|
|
||||||
Start by finding the log messages that the application generates related to
|
Start by finding the log messages that the application generates related to
|
||||||
some form of authentication failure. If you are adding to an existing filter
|
some form of authentication failure. If you are adding to an existing filter
|
||||||
think about wheither the log messages are of a simlar importance and purpose
|
think about whether the log messages are of a similar importance and purpose
|
||||||
to the existing filter. If you where a user of fail2ban, and did a package
|
to the existing filter. If you where a user of fail2ban, and did a package
|
||||||
update of fail2ban that started matching the new log messages, would anything
|
update of fail2ban that started matching the new log messages, would anything
|
||||||
unexpected happen? Would the bantime/findtime for the jail be approprate for
|
unexpected happen? Would the bantime/findtime for the jail be appropriate for
|
||||||
the new log messages. If it doesn't perhaps it needs to be in a separate
|
the new log messages. If it doesn't perhaps it needs to be in a separate
|
||||||
filter defination, for example like exim is authentication failures and
|
filter definition, for example like exim is authentication failures and
|
||||||
exim-spam contains log messages replated to spam.
|
exim-spam contains log messages related to spam.
|
||||||
|
|
||||||
Even if its a new filter you may consider separating the log messages into
|
Even if its a new filter you may consider separating the log messages into
|
||||||
different filters based on purpose.
|
different filters based on purpose.
|
||||||
|
@ -75,12 +75,12 @@ Are some of the log lines a result of the same action? For example is a PAM
|
||||||
failure log message, followed by an application specific failure message the
|
failure log message, followed by an application specific failure message the
|
||||||
result of the same user/script action. The result is if you add regular
|
result of the same user/script action. The result is if you add regular
|
||||||
expressions for both you'll end up with two failures for a single action.
|
expressions for both you'll end up with two failures for a single action.
|
||||||
Select the most approprate log message and document the other log message with
|
Select the most appropriate log message and document the other log message with
|
||||||
a test case not to match it and a description as to why you chose one over
|
a test case not to match it and a description as to why you chose one over
|
||||||
another.
|
another.
|
||||||
|
|
||||||
With the log lines selected consider what occured to generate those log
|
With the log lines selected consider what occurred to generate those log
|
||||||
messages and wheither they could of been generated by accidental means. Could
|
messages and whether they could of been generated by accidental means. Could
|
||||||
the log message occur always as this is the first step towards the application
|
the log message occur always as this is the first step towards the application
|
||||||
asking for authentication? Could the log messages occur often? If some of
|
asking for authentication? Could the log messages occur often? If some of
|
||||||
these are true make a note of this in the jail.conf example that you provide.
|
these are true make a note of this in the jail.conf example that you provide.
|
||||||
|
@ -97,7 +97,7 @@ any specific information about the log message, such as version or an
|
||||||
application configuration option that is needed for the message to occur,
|
application configuration option that is needed for the message to occur,
|
||||||
include this in a comment (line beginning with #) above the failJSON metadata.
|
include this in a comment (line beginning with #) above the failJSON metadata.
|
||||||
|
|
||||||
Log samples should include only one, definately not more than 3, examples of
|
Log samples should include only one, definitely not more than 3, examples of
|
||||||
log messages of the same form. If log messages are different in different
|
log messages of the same form. If log messages are different in different
|
||||||
versions of the application log messages that show this is encouraged.
|
versions of the application log messages that show this is encouraged.
|
||||||
|
|
||||||
|
@ -122,7 +122,7 @@ Mar 24 15:25:51 buffalo1 dropbear[4092]: bad password attempt for 'root' from 19
|
||||||
|
|
||||||
The host will contain the IP or domain that should be blocked.
|
The host will contain the IP or domain that should be blocked.
|
||||||
|
|
||||||
For long lines that you don't want matched, like log injection vulerabilities
|
For long lines that you don't want matched, like log injection vulnerabilities
|
||||||
and log lines excluded (see "Cause" section above), a "match": false in the
|
and log lines excluded (see "Cause" section above), a "match": false in the
|
||||||
failJSON and the reason why in the comment above.
|
failJSON and the reason why in the comment above.
|
||||||
|
|
||||||
|
@ -157,7 +157,7 @@ is the date or month.
|
||||||
Filter file:
|
Filter file:
|
||||||
|
|
||||||
The filter file is in config/filter.d/{filtername}.conf. The format of the
|
The filter file is in config/filter.d/{filtername}.conf. The format of the
|
||||||
filter file has two sections INCLUDES and Defination as follows:
|
filter file has two sections INCLUDES and Definition as follows:
|
||||||
|
|
||||||
[INCLUDES]
|
[INCLUDES]
|
||||||
|
|
||||||
|
@ -172,7 +172,7 @@ failregex = ....
|
||||||
ignoreregex = ....
|
ignoreregex = ....
|
||||||
|
|
||||||
This is also documented in the man pages as jail.conf (section 5). Other
|
This is also documented in the man pages as jail.conf (section 5). Other
|
||||||
definations can be added to make failregex's more readable and maintainable.
|
definitions can be added to make failregex's more readable and maintainable.
|
||||||
|
|
||||||
|
|
||||||
General rules:
|
General rules:
|
||||||
|
@ -180,7 +180,7 @@ General rules:
|
||||||
Use "before" if you need to include a common set of rules, like syslog or if
|
Use "before" if you need to include a common set of rules, like syslog or if
|
||||||
there's a common set of regexs for multiple filters.
|
there's a common set of regexs for multiple filters.
|
||||||
|
|
||||||
Use "after" if you wish to allow the user to overwrite a set of customisations
|
Use "after" if you wish to allow the user to overwrite a set of customisation's
|
||||||
of the current filter. This file doesn't need to exist.
|
of the current filter. This file doesn't need to exist.
|
||||||
|
|
||||||
Try to avoid using ignoreregex mainly for performance reasons. The case when
|
Try to avoid using ignoreregex mainly for performance reasons. The case when
|
||||||
|
@ -214,10 +214,10 @@ So now ^%(__prefix_line)s matches "Dec 12 11:19:11 dunnart dovecot: ". Note it
|
||||||
matches the trailing space. Putting a space after ^%(__prefix_line)s in the
|
matches the trailing space. Putting a space after ^%(__prefix_line)s in the
|
||||||
regex will probably not match.
|
regex will probably not match.
|
||||||
|
|
||||||
Substitions:
|
Substitutions:
|
||||||
|
|
||||||
Substitions are what the syslog uses. The regex bits of %(_name)s substitute
|
Substation's are what the syslog uses. The regex bits of %(_name)s substitute
|
||||||
the _name defination into the regex. They are useful for making the regexes
|
the _name definition into the regex. They are useful for making the regexes
|
||||||
more readable and also defining regex parts that occur in multiple log lines.
|
more readable and also defining regex parts that occur in multiple log lines.
|
||||||
|
|
||||||
Regular Expressions:
|
Regular Expressions:
|
||||||
|
@ -256,13 +256,13 @@ Take note of -l heavydebug / -l debug and -v as they will be most useful.
|
||||||
|
|
||||||
TIP: Take a look at the source code of the application. You may see optional or
|
TIP: Take a look at the source code of the application. You may see optional or
|
||||||
extra log messages, or parts there of, that need to form part of your regex.
|
extra log messages, or parts there of, that need to form part of your regex.
|
||||||
It may also show how some parts are contrained and different formats
|
It may also show how some parts are con trained and different formats
|
||||||
depending on configuration or less common usages.
|
depending on configuration or less common usages.
|
||||||
|
|
||||||
TIP: Some applications log spaces at the end. If you're not sure add \s*$ as the
|
TIP: Some applications log spaces at the end. If you're not sure add \s*$ as the
|
||||||
end part of the regex.
|
end part of the regex.
|
||||||
|
|
||||||
If your regex isn't matching take a look at http://www.debuggex.com/.
|
If your regex isn't matching take a look at http://www.debuggex.com/?flavor=python
|
||||||
|
|
||||||
Using the regex from the ./fail2ban-regex output (to ensure all substitutions
|
Using the regex from the ./fail2ban-regex output (to ensure all substitutions
|
||||||
are done) and with <HOST> replaced with (?&.ipv4). Set the regex type to
|
are done) and with <HOST> replaced with (?&.ipv4). Set the regex type to
|
||||||
|
@ -272,7 +272,8 @@ For the test data put your log output with the time removed.
|
||||||
|
|
||||||
When you've fixed the regex put it back into your filter file.
|
When you've fixed the regex put it back into your filter file.
|
||||||
|
|
||||||
Please give a donation to stoarca for debuggex. Its a great tool isn't it.
|
Please spread the good word about debuggex - Serge Toarca is kindly continuing
|
||||||
|
its free availability to Open Source developers.
|
||||||
|
|
||||||
Finishing up:
|
Finishing up:
|
||||||
|
|
||||||
|
@ -284,8 +285,8 @@ So more specifically in the [filter] section in jail.conf:
|
||||||
* use "filter =" set to your filter name.
|
* use "filter =" set to your filter name.
|
||||||
* use a action to disable ports associated with the application
|
* use a action to disable ports associated with the application
|
||||||
* set "logpath" to a usual location for the log file for the application.
|
* set "logpath" to a usual location for the log file for the application.
|
||||||
* If the default findtime or bantime isn't approprate to the filter set a value
|
* If the default findtime or bantime isn't appropriate to the filter set a value
|
||||||
that is more approprate.
|
that is more appropriate.
|
||||||
|
|
||||||
Send the fail2ban a git pull request (See "Pull Requests" above) containing
|
Send the fail2ban a git pull request (See "Pull Requests" above) containing
|
||||||
your great work.
|
your great work.
|
||||||
|
@ -293,7 +294,7 @@ your great work.
|
||||||
Filter Security
|
Filter Security
|
||||||
---------------
|
---------------
|
||||||
|
|
||||||
Poor filter regular expressions are suseptable to DoS attacks.
|
Poor filter regular expressions are susceptible to DoS attacks.
|
||||||
|
|
||||||
When a remote user has the ability to introduce text that will match the
|
When a remote user has the ability to introduce text that will match the
|
||||||
filter regex, such that the inserted text matches the <HOST> part, they have the
|
filter regex, such that the inserted text matches the <HOST> part, they have the
|
||||||
|
@ -303,7 +304,7 @@ So the <HOST> part must be anchored on text generated by the application, and no
|
||||||
the user, to a sufficient extent that the user cannot insert the entire text.
|
the user, to a sufficient extent that the user cannot insert the entire text.
|
||||||
|
|
||||||
Ideally filter regex should anchor to the beginning and end of the log line
|
Ideally filter regex should anchor to the beginning and end of the log line
|
||||||
however as more applications log at the beginning than the end, achoring the
|
however as more applications log at the beginning than the end, anchoring the
|
||||||
beginning is more important. If the log file used by the application is shared
|
beginning is more important. If the log file used by the application is shared
|
||||||
with other applications, like system logs, ensure the other application that
|
with other applications, like system logs, ensure the other application that
|
||||||
use that log file do not log user generated text at the beginning of the line,
|
use that log file do not log user generated text at the beginning of the line,
|
||||||
|
@ -326,13 +327,13 @@ We make a failregex
|
||||||
|
|
||||||
Now think evil. The user does the command 'blah from 1.2.3.44'
|
Now think evil. The user does the command 'blah from 1.2.3.44'
|
||||||
|
|
||||||
The program diliently logs:
|
The program diligently logs:
|
||||||
|
|
||||||
Apr-07-13 07:08:36 Invalid command blah from 1.2.3.44 from 1.2.3.4
|
Apr-07-13 07:08:36 Invalid command blah from 1.2.3.44 from 1.2.3.4
|
||||||
|
|
||||||
And fail2ban matches 1.2.3.44 as the IP that it ban. A DoS attack was successful.
|
And fail2ban matches 1.2.3.44 as the IP that it ban. A DoS attack was successful.
|
||||||
|
|
||||||
The fix here is that the command can be anything so .* is approprate.
|
The fix here is that the command can be anything so .* is appropriate.
|
||||||
|
|
||||||
^Invalid command .* from <HOST>
|
^Invalid command .* from <HOST>
|
||||||
|
|
||||||
|
@ -351,10 +352,10 @@ banned.
|
||||||
|
|
||||||
2. Filter regex can match other user injected data
|
2. Filter regex can match other user injected data
|
||||||
|
|
||||||
From the apache vulnerability CVE-2013-2178
|
From the Apache vulnerability CVE-2013-2178
|
||||||
( original ref: https://vndh.net/note:fail2ban-089-denial-service ).
|
( original ref: https://vndh.net/note:fail2ban-089-denial-service ).
|
||||||
|
|
||||||
An example bad regex for apache:
|
An example bad regex for Apache:
|
||||||
|
|
||||||
failregex = [[]client <HOST>[]] user .* not found
|
failregex = [[]client <HOST>[]] user .* not found
|
||||||
|
|
||||||
|
@ -370,10 +371,10 @@ Now the log line will be:
|
||||||
As this log line doesn't match other expressions hence it matches the above
|
As this log line doesn't match other expressions hence it matches the above
|
||||||
regex and blocks 192.168.33.1 as a denial of service from the HTTP requester.
|
regex and blocks 192.168.33.1 as a denial of service from the HTTP requester.
|
||||||
|
|
||||||
3. Applicaiton generates two identical log messages with different meanings
|
3. Application generates two identical log messages with different meanings
|
||||||
|
|
||||||
If the application generates the following two messages under different
|
If the application generates the following two messages under different
|
||||||
circmstances:
|
circumstances:
|
||||||
|
|
||||||
client <IP>: authentication failed
|
client <IP>: authentication failed
|
||||||
client <USER>: authentication failed
|
client <USER>: authentication failed
|
||||||
|
@ -409,7 +410,7 @@ coverage run fail2ban-testcases
|
||||||
coverage html
|
coverage html
|
||||||
|
|
||||||
Then look at htmlcov/index.html and see how much coverage your test cases
|
Then look at htmlcov/index.html and see how much coverage your test cases
|
||||||
exert over the codebase. Full coverage is a good thing however it may not be
|
exert over the code base. Full coverage is a good thing however it may not be
|
||||||
complete. Try to ensure tests cover as many independent paths through the
|
complete. Try to ensure tests cover as many independent paths through the
|
||||||
code.
|
code.
|
||||||
|
|
||||||
|
@ -500,7 +501,7 @@ Design
|
||||||
Fail2Ban was initially developed with Python 2.3 (IIRC). It should
|
Fail2Ban was initially developed with Python 2.3 (IIRC). It should
|
||||||
still be compatible with Python 2.4 and such compatibility assurance
|
still be compatible with Python 2.4 and such compatibility assurance
|
||||||
makes code ... old-fashioned in many places (RF-Note). In 0.7 the
|
makes code ... old-fashioned in many places (RF-Note). In 0.7 the
|
||||||
design went through major refactoring into client/server,
|
design went through major re-factoring into client/server,
|
||||||
a-thread-per-jail design which made it a bit difficult to follow.
|
a-thread-per-jail design which made it a bit difficult to follow.
|
||||||
Below you can find a sketchy description of the main components of the
|
Below you can find a sketchy description of the main components of the
|
||||||
system to orient yourself better.
|
system to orient yourself better.
|
||||||
|
@ -611,7 +612,7 @@ one way or another provide
|
||||||
except FailManagerEmpty:
|
except FailManagerEmpty:
|
||||||
self.failManager.cleanup(MyTime.time())
|
self.failManager.cleanup(MyTime.time())
|
||||||
|
|
||||||
thus channeling "ban tickets" from their failManager to the
|
thus channelling "ban tickets" from their failManager to the
|
||||||
corresponding jail.
|
corresponding jail.
|
||||||
|
|
||||||
action.py
|
action.py
|
||||||
|
|
Loading…
Reference in New Issue