@ -1,4 +1,4 @@
__ _ _ ___ _.. __ _ _ ___ _
/ _|__ _(_) |_ ) |__ __ _ _ _
/ _|__ _(_) |_ ) |__ __ _ _ _
| _/ _` | | |/ /| '_ \/ _` | ' \
| _/ _` | | |/ /| '_ \/ _` | ' \
|_| \__,_|_|_/___|_.__/\__,_|_||_|
|_| \__,_|_|_/___|_.__/\__,_|_||_|
@ -7,10 +7,8 @@
Developing Filters
Developing Filters
================================================================================
================================================================================
Filters
=======
Filters are tricky. They need to:
Filters are tricky. They need to:
* work with a variety of the versions of the software that generates the logs;
* work with a variety of the versions of the software that generates the logs;
* work with the range of logging configuration options available in the
* work with the range of logging configuration options available in the
software;
software;
@ -31,10 +29,11 @@ what you have done, what is the hurdle, and we'll attempt to help (PR
will be automagically updated with future commits you would push to
will be automagically updated with future commits you would push to
complete it).
complete it).
Filter test c ases
Filter Test C ases
----------------- =================
Purpose:
Purpose
-------
Start by finding the log messages that the application generates related to
Start by finding the log messages that the application generates related to
some form of authentication failure. If you are adding to an existing filter
some form of authentication failure. If you are adding to an existing filter
@ -49,7 +48,8 @@ and exim-spam at log messages related to spam.
Even if it is a new filter you may consider separating the log messages into
Even if it is a new filter you may consider separating the log messages into
different filters based on purpose.
different filters based on purpose.
Cause:
Cause
-----
Are some of the log lines a result of the same action? For example, is a PAM
Are some of the log lines a result of the same action? For example, is a PAM
failure log message, followed by an application specific failure message the
failure log message, followed by an application specific failure message the
@ -65,7 +65,8 @@ the log message be occurring due to the first step towards the application
asking for authentication? Could the log messages occur often? If some of
asking for authentication? Could the log messages occur often? If some of
these are true make a note of this in the jail.conf example that you provide.
these are true make a note of this in the jail.conf example that you provide.
Samples:
Samples
-------
It is important to include log file samples so any future change in the regular
It is important to include log file samples so any future change in the regular
expression will still work with the log lines you have identified.
expression will still work with the log lines you have identified.
@ -93,21 +94,22 @@ If the mechanism to create the log message isn't obvious provide a
configuration and/or sample scripts testcases/files/config/{filtername} and
configuration and/or sample scripts testcases/files/config/{filtername} and
reference these in the comments above the log line.
reference these in the comments above the log line.
FailJSON metadata:
FailJSON metadata
-----------------
A failJSON metadata is a comment immediately above the log message. It will
A failJSON metadata is a comment immediately above the log message. It will
look like:
look like::
# failJSON: { "time": "2013-06-10T10:10:59", "match": true , "host": "93.184.216.119" }
# failJSON: { "time": "2013-06-10T10:10:59", "match": true , "host": "93.184.216.119" }
Time should match the time of the log message. It is in a specific format of
Time should match the time of the log message. It is in a specific format of
Year-Month-Day'T'Hour:minute:Second. If your log message does not include a
Year-Month-Day'T'Hour:minute:Second. If your log message does not include a
year, like the example below, the year should be listed as 2005, if before Sun
year, like the example below, the year should be listed as 2005, if before Sun
Aug 14 10am UTC, and 2004 if afterwards. Here is an example failJSON
Aug 14 10am UTC, and 2004 if afterwards. Here is an example failJSON
line preceding a sample log line:
line preceding a sample log line::
# failJSON: { "time": "2005-03-24T15:25:51", "match": true , "host": "198.51.100.87" }
# failJSON: { "time": "2005-03-24T15:25:51", "match": true , "host": "198.51.100.87" }
Mar 24 15:25:51 buffalo1 dropbear[4092]: bad password attempt for 'root' from 198.51.100.87:5543
Mar 24 15:25:51 buffalo1 dropbear[4092]: bad password attempt for 'root' from 198.51.100.87:5543
The "host" in failJSON should contain the IP or domain that should be blocked.
The "host" in failJSON should contain the IP or domain that should be blocked.
@ -116,27 +118,28 @@ attacks) and any log lines to be excluded (see "Cause" section above), set
"match": false in the failJSON and describe the reason in the comment above.
"match": false in the failJSON and describe the reason in the comment above.
After developing regexes, the following command will test all failJSON metadata
After developing regexes, the following command will test all failJSON metadata
against the log lines in all sample log files
against the log lines in all sample log files::
./fail2ban-testcases testSampleRegex
./fail2ban-testcases testSampleRegex
Developing Filter Regular Expressions
Developing Filter Regular Expressions
------------------------------------- =====================================
Date/Time:
Date/Time
---------
At the moment, Fail2Ban depends on log lines to have time stamps. That is why
At the moment, Fail2Ban depends on log lines to have time stamps. That is why
before starting to develop failregex, check if your log line format known to
before starting to develop failregex, check if your log line format known to
Fail2Ban. Copy the time component from the log line and append an IP address to
Fail2Ban. Copy the time component from the log line and append an IP address to
test with following command:
test with following command::
./fail2ban-regex "2013-09-19 02:46:12 1.2.3.4" "<HOST>"
./fail2ban-regex "2013-09-19 02:46:12 1.2.3.4" "<HOST>"
Output of such command should contain something like:
Output of such command should contain something like::
Date template hits:
Date template hits:
|- [# of hits] date format
|- [# of hits] date format
| [1] Year-Month-Day Hour:Minute:Second
| [1] Year-Month-Day Hour:Minute:Second
Ensure that the template description matches time/date elements in your log line
Ensure that the template description matches time/date elements in your log line
time stamp. If there is no matched format then date template needs to be added
time stamp. If there is no matched format then date template needs to be added
@ -144,29 +147,31 @@ to server/datedetector.py. Ensure that a new template is added in the order
that more specific matches occur first and that there is no confusion between a
that more specific matches occur first and that there is no confusion between a
Day and a Month.
Day and a Month.
Filter file:
Filter file
-----------
The filter is specified in a config/filter.d/{filtername}.conf file. Filter file
The filter is specified in a config/filter.d/{filtername}.conf file. Filter file
can have sections INCLUDES (optional) and Definition as follows:
can have sections INCLUDES (optional) and Definition as follows::
[INCLUDES]
[INCLUDES]
before = common.conf
before = common.conf
after = filtername.local
after = filtername.local
[Definition]
[Definition]
failregex = ....
failregex = ....
ignoreregex = ....
ignoreregex = ....
This is also documented in the man page jail.conf (section 5). Other definitions
This is also documented in the man page jail.conf (section 5). Other definitions
can be added to make failregex's more readable and maintainable to be used
can be added to make failregex's more readable and maintainable to be used
through string Interpolations (see http://docs.python.org/2.7/library/configparser.html)
through string Interpolations (see http://docs.python.org/2.7/library/configparser.html)
General rules:
General rules
-------------
Use "before" if you need to include a common set of rules, like syslog or if
Use "before" if you need to include a common set of rules, like syslog or if
there is a common set of regexes for multiple filters.
there is a common set of regexes for multiple filters.
@ -178,33 +183,35 @@ Try to avoid using ignoreregex mainly for performance reasons. The case when you
would use it is if in trying to avoid using it, you end up with an unreadable
would use it is if in trying to avoid using it, you end up with an unreadable
failregex.
failregex.
Syslog:
Syslog
------
If your application logs to syslog you can take advantage of log line prefix
If your application logs to syslog you can take advantage of log line prefix
definitions present in common.conf. So as a base use:
definitions present in common.conf. So as a base use::
[INCLUDES]
[INCLUDES]
before = common.conf
before = common.conf
[Definition]
[Definition]
_daemon = app
_daemon = app
failregex = ^%(__prefix_line)s
failregex = ^%(__prefix_line)s
In this example common.conf defines __prefix_line which also contains the
In this example common.conf defines __prefix_line which also contains the
_daemon name (in syslog terms the service) you have just specified. _daemon
_daemon name (in syslog terms the service) you have just specified. _daemon
can also be a regex.
can also be a regex.
For example, to capture following line _daemon should be set to "dovecot"
For example, to capture following line _daemon should be set to "dovecot"::
Dec 12 11:19:11 dunnart dovecot: pop3-login: Aborted login (tried to use disabled plaintext auth): rip=190.210.136.21, lip=113.212.99.193
Dec 12 11:19:11 dunnart dovecot: pop3-login: Aborted login (tried to use disabled plaintext auth): rip=190.210.136.21, lip=113.212.99.193
and then ^%(__prefix_line)s would match "Dec 12 11:19:11 dunnart dovecot:
and then `` ^%(__prefix_line)s`` would match "Dec 12 11:19:11 dunnart dovecot:
". Note it matches the trailing space(s) as well.
". Note it matches the trailing space(s) as well.
Substitutions (AKA string interpolations):
Substitutions (AKA string interpolations)
-----------------------------------------
We have used string interpolations in above examples. They are useful for
We have used string interpolations in above examples. They are useful for
making the regexes more readable, reuse generic patterns in multiple failregex
making the regexes more readable, reuse generic patterns in multiple failregex
@ -213,7 +220,8 @@ to the user. General principle is that value of a _name variable replaces
occurrences of %(_name)s within the same section or anywhere in the config file
occurrences of %(_name)s within the same section or anywhere in the config file
if defined in [DEFAULT] section.
if defined in [DEFAULT] section.
Regular Expressions:
Regular Expressions
-------------------
Regular expressions (failregex, ignoreregex) assume that the date/time has been
Regular expressions (failregex, ignoreregex) assume that the date/time has been
removed from the log line (this is just how fail2ban works internally ATM).
removed from the log line (this is just how fail2ban works internally ATM).
@ -236,29 +244,33 @@ If you have only a basic knowledge of regular repressions we advise to read
http://docs.python.org/2/library/re.html first. It doesn't take long and would
http://docs.python.org/2/library/re.html first. It doesn't take long and would
remind you e.g. which characters you need to escape and which you don't.
remind you e.g. which characters you need to escape and which you don't.
Developing/testing a regex:
Developing/testing a regex
--------------------------
You can develop a regex in a file or using command line depending on your
You can develop a regex in a file or using command line depending on your
preference. You can also use samples you have already created in the test cases
preference. You can also use samples you have already created in the test cases
or test them one at a time.
or test them one at a time.
The general tool for testing Fail2Ban regexes is fail2ban-regex. To see how to
The general tool for testing Fail2Ban regexes is fail2ban-regex. To see how to
use it run:
use it run::
./fail2ban-regex --help
./fail2ban-regex --help
Take note of -l heavydebug / -l debug and -v as they might be very useful.
Take note of -l heavydebug / -l debug and -v as they might be very useful.
TIP: Take a look at the source code of the application you are developing
.. TIP::
Take a look at the source code of the application you are developing
failregex for. You may see optional or extra log messages, or parts there
failregex for. You may see optional or extra log messages, or parts there
of, that need to form part of your regex. It may also reveal how some
of, that need to form part of your regex. It may also reveal how some
parts are constrained and different formats depending on configuration or
parts are constrained and different formats depending on configuration or
less common usages.
less common usages.
TIP: For looking through source code - http://sourcecodebrowser.com/ . It has
.. TIP::
For looking through source code - http://sourcecodebrowser.com/ . It has
call graphs and can browse different versions.
call graphs and can browse different versions.
TIP: Some applications log spaces at the end. If you are not sure add \s*$ as
.. TIP::
Some applications log spaces at the end. If you are not sure add \s*$ as
the end part of the regex.
the end part of the regex.
If your regex is not matching, http://www.debuggex.com/?flavor=python can help
If your regex is not matching, http://www.debuggex.com/?flavor=python can help
@ -277,13 +289,15 @@ When you have fixed the regex put it back into your filter file.
Please spread the good word about Debuggex - Serge Toarca is kindly continuing
Please spread the good word about Debuggex - Serge Toarca is kindly continuing
its free availability to Open Source developers.
its free availability to Open Source developers.
Finishing up:
Finishing up
------------
If you've added a new filter, add a new entry in config/jail.conf. The theory
If you've added a new filter, add a new entry in config/jail.conf. The theory
here is that a user will create a jail.local with [filtername]\nenable=true to
here is that a user will create a jail.local with [filtername]\nenable=true to
enable your jail.
enable your jail.
So more specifically in the [filter] section in jail.conf:
So more specifically in the [filter] section in jail.conf:
* ensure that you have "enabled = false" (users will enable as needed);
* ensure that you have "enabled = false" (users will enable as needed);
* use "filter =" set to your filter name;
* use "filter =" set to your filter name;
* use a typical action to disable ports associated with the application;
* use a typical action to disable ports associated with the application;
@ -295,7 +309,7 @@ Submit github pull request (See "Pull Requests" above) for
github.com/fail2ban/fail2ban containing your great work.
github.com/fail2ban/fail2ban containing your great work.
Filter Security
Filter Security
--------------- ===============
Poor filter regular expressions are susceptible to DoS attacks.
Poor filter regular expressions are susceptible to DoS attacks.
@ -321,33 +335,33 @@ Examples of poor filters
1. Too restrictive
1. Too restrictive
We find a log message:
We find a log message::
Apr-07-13 07:08:36 Invalid command fial2ban from 1.2.3.4
Apr-07-13 07:08:36 Invalid command fial2ban from 1.2.3.4
We make a failregex
We make a failregex::
^Invalid command \S+ from <HOST>
^Invalid command \S+ from <HOST>
Now think evil. The user does the command 'blah from 1.2.3.44'
Now think evil. The user does the command 'blah from 1.2.3.44'
The program diligently logs:
The program diligently logs::
Apr-07-13 07:08:36 Invalid command blah from 1.2.3.44 from 1.2.3.4
Apr-07-13 07:08:36 Invalid command blah from 1.2.3.44 from 1.2.3.4
And fail2ban matches 1.2.3.44 as the IP that it ban. A DoS attack was successful.
And fail2ban matches 1.2.3.44 as the IP that it ban. A DoS attack was successful.
The fix here is that the command can be anything so .* is appropriate.
The fix here is that the command can be anything so .* is appropriate::
^Invalid command .* from <HOST>
^Invalid command .* from <HOST>
Here the .* will match until the end of the string. Then realise it has more to
Here the .* will match until the end of the string. Then realise it has more to
match, i.e. "from <HOST>" and go back until it find this. Then it will ban
match, i.e. "from <HOST>" and go back until it find this. Then it will ban
1.2.3.4 correctly. Since the <HOST> is always at the end, end the regex with a $.
1.2.3.4 correctly. Since the <HOST> is always at the end, end the regex with a $::
^Invalid command .* from <HOST>$
^Invalid command .* from <HOST>$
Note if we'd just had the expression:
Note if we'd just had the expression::
^Invalid command \S+ from <HOST>$
^Invalid command \S+ from <HOST>$
@ -359,16 +373,16 @@ banned.
From the Apache vulnerability CVE-2013-2178
From the Apache vulnerability CVE-2013-2178
( original ref: https://vndh.net/note:fail2ban-089-denial-service ).
( original ref: https://vndh.net/note:fail2ban-089-denial-service ).
An example bad regex for Apache:
An example bad regex for Apache::
failregex = [[]client <HOST>[]] user .* not found
failregex = [[]client <HOST>[]] user .* not found
Since the user can do a get request on:
Since the user can do a get request on::
GET /[client%20192.168.0.1]%20user%20root%20not%20found HTTP/1.0
GET /[client%20192.168.0.1]%20user%20root%20not%20found HTTP/1.0
Host: remote.site
Host: remote.site
Now the log line will be:
Now the log line will be::
[Sat Jun 01 02:17:42 2013] [error] [client 192.168.33.1] File does not exist: /srv/http/site/[client 192.168.0.1] user root not found
[Sat Jun 01 02:17:42 2013] [error] [client 192.168.33.1] File does not exist: /srv/http/site/[client 192.168.0.1] user root not found
@ -379,27 +393,27 @@ regex and blocks 192.168.33.1 as a denial of service from the HTTP requester.
From: https://github.com/fail2ban/fail2ban/pull/426
From: https://github.com/fail2ban/fail2ban/pull/426
An example ssh log (simplified)
An example ssh log (simplified)::
Sep 29 17:15:02 spaceman sshd[12946]: Failed password for user from 127.0.0.1 port 20000 ssh1: ruser remoteuser
Sep 29 17:15:02 spaceman sshd[12946]: Failed password for user from 127.0.0.1 port 20000 ssh1: ruser remoteuser
As we assume username can include anything including spaces its prudent to put
As we assume username can include anything including spaces its prudent to put
.* here. The remote user can also exist as anything so lets not make assumptions again.
.* here. The remote user can also exist as anything so lets not make assumptions again::
failregex = ^%(__prefix_line)sFailed \S+ for .* from <HOST>( port \d*)?( ssh\d+)?(: ruser .*)?$
failregex = ^%(__prefix_line)sFailed \S+ for .* from <HOST>( port \d*)?( ssh\d+)?(: ruser .*)?$
So this works. The problem is if the .* after remote user is injected by the
So this works. The problem is if the .* after remote user is injected by the
user to be 'from 1.2.3.4'. The resultant log line is.
user to be 'from 1.2.3.4'. The resultant log line is::
Sep 29 17:15:02 spaceman sshd[12946]: Failed password for user from 127.0.0.1 port 20000 ssh1: ruser from 1.2.3.4
Sep 29 17:15:02 spaceman sshd[12946]: Failed password for user from 127.0.0.1 port 20000 ssh1: ruser from 1.2.3.4
Testing with:
Testing with::
fail2ban-regex -v 'Sep 29 17:15:02 Failed password for user from 127.0.0.1 port 20000 ssh1: ruser from 1.2.3.4' '^ Failed \S+ for .* from <HOST>( port \d*)?( ssh\d+)?(: ruser .*)?$'
fail2ban-regex -v 'Sep 29 17:15:02 Failed password for user from 127.0.0.1 port 20000 ssh1: ruser from 1.2.3.4' '^ Failed \S+ for .* from <HOST>( port \d*)?( ssh\d+)?(: ruser .*)?$'
TIP: I've removed the bit that matches __prefix_line from the regex and log.
.. TIP: : I've removed the bit that matches __prefix_line from the regex and log.
Shows:
Shows::
1) [1] ^ Failed \S+ for .* from <HOST>( port \d*)?( ssh\d+)?(: ruser .*)?$
1) [1] ^ Failed \S+ for .* from <HOST>( port \d*)?( ssh\d+)?(: ruser .*)?$
1.2.3.4 Sun Sep 29 17:15:02 2013
1.2.3.4 Sun Sep 29 17:15:02 2013
@ -412,14 +426,14 @@ The result was that 1.2.3.4 was matched, injected by the user, and the wrong IP
was banned.
was banned.
The solution here is to make the first .* non-greedy with .*?. Here it matches
The solution here is to make the first .* non-greedy with .*?. Here it matches
as little as required and the fail2ban-regex tool shows the output:
as little as required and the fail2ban-regex tool shows the output::
fail2ban-regex -v 'Sep 29 17:15:02 Failed password for user from 127.0.0.1 port 20000 ssh1: ruser from 1.2.3.4' '^ Failed \S+ for .*? from <HOST>( port \d*)?( ssh\d+)?(: ruser .*)?$'
fail2ban-regex -v 'Sep 29 17:15:02 Failed password for user from 127.0.0.1 port 20000 ssh1: ruser from 1.2.3.4' '^ Failed \S+ for .*? from <HOST>( port \d*)?( ssh\d+)?(: ruser .*)?$'
1) [1] ^ Failed \S+ for .*? from <HOST>( port \d*)?( ssh\d+)?(: ruser .*)?$
1) [1] ^ Failed \S+ for .*? from <HOST>( port \d*)?( ssh\d+)?(: ruser .*)?$
127.0.0.1 Sun Sep 29 17:15:02 2013
127.0.0.1 Sun Sep 29 17:15:02 2013
So the general case here is a log line that contains:
So the general case here is a log line that contains::
(fixed_data_1)<HOST>(fixed_data_2)(user_injectable_data)
(fixed_data_1)<HOST>(fixed_data_2)(user_injectable_data)
@ -427,20 +441,21 @@ Where the regex that matches fixed_data_1 is gready and matches the entire
string, before moving backwards and user_injectable_data can match the entire
string, before moving backwards and user_injectable_data can match the entire
string.
string.
Another case:
Another case
------------
ref: https://www.debuggex.com/r/CtAbeKMa2sDBEfA2/0
ref: https://www.debuggex.com/r/CtAbeKMa2sDBEfA2/0
A webserver logs the following without URL escaping:
A webserver logs the following without URL escaping::
[error] 2865#0: *66647 user "xyz" was not found in "/file", client: 1.2.3.1, server: www.host.com, request: "GET ", client: 3.2.1.1, server: fake.com, request: "GET exploited HTTP/3.3", host: "injected.host", host: "www.myhost.com"
[error] 2865#0: *66647 user "xyz" was not found in "/file", client: 1.2.3.1, server: www.host.com, request: "GET ", client: 3.2.1.1, server: fake.com, request: "GET exploited HTTP/3.3", host: "injected.host", host: "www.myhost.com"
regex:
regex::
failregex = ^ \[error\] \d+#\d+: \*\d+ user "\S+":? (?:password mismatch|was not found in ".*"), client: <HOST>, server: \S+, request: "\S+ .+ HTTP/\d+\.\d+", host: "\S+"
failregex = ^ \[error\] \d+#\d+: \*\d+ user "\S+":? (?:password mismatch|was not found in ".*"), client: <HOST>, server: \S+, request: "\S+ .+ HTTP/\d+\.\d+", host: "\S+"
The .* matches to the end of the string. Finds that it can't continue to match
The .* matches to the end of the string. Finds that it can't continue to match
", client ... so it moves from the back and find that the user injected web URL:
", client ... so it moves from the back and find that the user injected web URL::
", client: 3.2.1.1, server: fake.com, request: "GET exploited HTTP/3.3", host: "injected.host
", client: 3.2.1.1, server: fake.com, request: "GET exploited HTTP/3.3", host: "injected.host
@ -453,14 +468,14 @@ beyond <HOST>.
4. Application generates two identical log messages with different meanings
4. Application generates two identical log messages with different meanings
If the application generates the following two messages under different
If the application generates the following two messages under different
circumstances:
circumstances::
client <IP>: authentication failed
client <IP>: authentication failed
client <USER>: authentication failed
client <USER>: authentication failed
Then it's obvious that a regex of " ^client <HOST>: authentication
Then it's obvious that a regex of `` ^client <HOST>: authentication
failed$" will still cause problems if the user can trigger the second
failed$`` will still cause problems if the user can trigger the second
log message with a <USER> of 123.1.1.1.
log message with a <USER> of 123.1.1.1.
Here there's nothing to do except request/change the application so it logs
Here there's nothing to do except request/change the application so it logs
Steven Hiscocks
11 years ago