From 2562b2f1bc2a82372ccd9ff62a677f4c1457f846 Mon Sep 17 00:00:00 2001 From: Daniel Black Date: Thu, 19 Sep 2013 19:18:04 +1000 Subject: [PATCH] DOC: used spellchecker finally. Debuggex needs promotion more than donations --- DEVELOP | 71 +++++++++++++++++++++++++++++---------------------------- 1 file changed, 36 insertions(+), 35 deletions(-) diff --git a/DEVELOP b/DEVELOP index 74bd8ff3..710ff145 100644 --- a/DEVELOP +++ b/DEVELOP @@ -26,7 +26,7 @@ Pull Requests When submitting pull requests on GitHub we ask you to: * Clearly describe the problem you're solving; -* Don't introduce regressions that will make it hard for systems adminstrators +* Don't introduce regressions that will make it hard for systems administrators to update; * If adding a major feature rebase your changes on master and get to a single commit; * Include test cases (see below); @@ -44,12 +44,12 @@ Filters are tricky. They need to: * work in multiple operating systems; * not make assumptions about the log format in excess of the software; * make assumptions as to how future versions of the software will log messages; -* not be susceptable to DoS vulernabilities; and +* not be susceptible to DoS vulnerabilities; and * match intended log lines only. Please follow the steps from Filter Test Cases to Developing Filter Regular -Expressions and submit a github pull request afterward. If you get stuck, -create a github issue with what you have done and we'll attempt to help. +Expressions and submit a GitHub pull request afterwards. If you get stuck, +create a GitHub issue with what you have done and we'll attempt to help. Filter test cases ----------------- @@ -58,13 +58,13 @@ Purpose: Start by finding the log messages that the application generates related to some form of authentication failure. If you are adding to an existing filter -think about wheither the log messages are of a simlar importance and purpose +think about whether the log messages are of a similar importance and purpose to the existing filter. If you where a user of fail2ban, and did a package update of fail2ban that started matching the new log messages, would anything -unexpected happen? Would the bantime/findtime for the jail be approprate for +unexpected happen? Would the bantime/findtime for the jail be appropriate for the new log messages. If it doesn't perhaps it needs to be in a separate -filter defination, for example like exim is authentication failures and -exim-spam contains log messages replated to spam. +filter definition, for example like exim is authentication failures and +exim-spam contains log messages related to spam. Even if its a new filter you may consider separating the log messages into different filters based on purpose. @@ -75,12 +75,12 @@ Are some of the log lines a result of the same action? For example is a PAM failure log message, followed by an application specific failure message the result of the same user/script action. The result is if you add regular expressions for both you'll end up with two failures for a single action. -Select the most approprate log message and document the other log message with +Select the most appropriate log message and document the other log message with a test case not to match it and a description as to why you chose one over another. -With the log lines selected consider what occured to generate those log -messages and wheither they could of been generated by accidental means. Could +With the log lines selected consider what occurred to generate those log +messages and whether they could of been generated by accidental means. Could the log message occur always as this is the first step towards the application asking for authentication? Could the log messages occur often? If some of these are true make a note of this in the jail.conf example that you provide. @@ -97,7 +97,7 @@ any specific information about the log message, such as version or an application configuration option that is needed for the message to occur, include this in a comment (line beginning with #) above the failJSON metadata. -Log samples should include only one, definately not more than 3, examples of +Log samples should include only one, definitely not more than 3, examples of log messages of the same form. If log messages are different in different versions of the application log messages that show this is encouraged. @@ -122,7 +122,7 @@ Mar 24 15:25:51 buffalo1 dropbear[4092]: bad password attempt for 'root' from 19 The host will contain the IP or domain that should be blocked. -For long lines that you don't want matched, like log injection vulerabilities +For long lines that you don't want matched, like log injection vulnerabilities and log lines excluded (see "Cause" section above), a "match": false in the failJSON and the reason why in the comment above. @@ -157,7 +157,7 @@ is the date or month. Filter file: The filter file is in config/filter.d/{filtername}.conf. The format of the -filter file has two sections INCLUDES and Defination as follows: +filter file has two sections INCLUDES and Definition as follows: [INCLUDES] @@ -172,7 +172,7 @@ failregex = .... ignoreregex = .... This is also documented in the man pages as jail.conf (section 5). Other -definations can be added to make failregex's more readable and maintainable. +definitions can be added to make failregex's more readable and maintainable. General rules: @@ -180,7 +180,7 @@ General rules: Use "before" if you need to include a common set of rules, like syslog or if there's a common set of regexs for multiple filters. -Use "after" if you wish to allow the user to overwrite a set of customisations +Use "after" if you wish to allow the user to overwrite a set of customisation's of the current filter. This file doesn't need to exist. Try to avoid using ignoreregex mainly for performance reasons. The case when @@ -214,10 +214,10 @@ So now ^%(__prefix_line)s matches "Dec 12 11:19:11 dunnart dovecot: ". Note it matches the trailing space. Putting a space after ^%(__prefix_line)s in the regex will probably not match. -Substitions: +Substitutions: -Substitions are what the syslog uses. The regex bits of %(_name)s substitute -the _name defination into the regex. They are useful for making the regexes +Substation's are what the syslog uses. The regex bits of %(_name)s substitute +the _name definition into the regex. They are useful for making the regexes more readable and also defining regex parts that occur in multiple log lines. Regular Expressions: @@ -256,13 +256,13 @@ Take note of -l heavydebug / -l debug and -v as they will be most useful. TIP: Take a look at the source code of the application. You may see optional or extra log messages, or parts there of, that need to form part of your regex. - It may also show how some parts are contrained and different formats + It may also show how some parts are con trained and different formats depending on configuration or less common usages. TIP: Some applications log spaces at the end. If you're not sure add \s*$ as the end part of the regex. -If your regex isn't matching take a look at http://www.debuggex.com/. +If your regex isn't matching take a look at http://www.debuggex.com/?flavor=python Using the regex from the ./fail2ban-regex output (to ensure all substitutions are done) and with replaced with (?&.ipv4). Set the regex type to @@ -272,7 +272,8 @@ For the test data put your log output with the time removed. When you've fixed the regex put it back into your filter file. -Please give a donation to stoarca for debuggex. Its a great tool isn't it. +Please spread the good word about debuggex - Serge Toarca is kindly continuing +its free availability to Open Source developers. Finishing up: @@ -284,8 +285,8 @@ So more specifically in the [filter] section in jail.conf: * use "filter =" set to your filter name. * use a action to disable ports associated with the application * set "logpath" to a usual location for the log file for the application. -* If the default findtime or bantime isn't approprate to the filter set a value - that is more approprate. +* If the default findtime or bantime isn't appropriate to the filter set a value + that is more appropriate. Send the fail2ban a git pull request (See "Pull Requests" above) containing your great work. @@ -293,7 +294,7 @@ your great work. Filter Security --------------- -Poor filter regular expressions are suseptable to DoS attacks. +Poor filter regular expressions are susceptible to DoS attacks. When a remote user has the ability to introduce text that will match the filter regex, such that the inserted text matches the part, they have the @@ -303,7 +304,7 @@ So the part must be anchored on text generated by the application, and no the user, to a sufficient extent that the user cannot insert the entire text. Ideally filter regex should anchor to the beginning and end of the log line -however as more applications log at the beginning than the end, achoring the +however as more applications log at the beginning than the end, anchoring the beginning is more important. If the log file used by the application is shared with other applications, like system logs, ensure the other application that use that log file do not log user generated text at the beginning of the line, @@ -326,13 +327,13 @@ We make a failregex Now think evil. The user does the command 'blah from 1.2.3.44' -The program diliently logs: +The program diligently logs: Apr-07-13 07:08:36 Invalid command blah from 1.2.3.44 from 1.2.3.4 And fail2ban matches 1.2.3.44 as the IP that it ban. A DoS attack was successful. -The fix here is that the command can be anything so .* is approprate. +The fix here is that the command can be anything so .* is appropriate. ^Invalid command .* from @@ -351,10 +352,10 @@ banned. 2. Filter regex can match other user injected data -From the apache vulnerability CVE-2013-2178 +From the Apache vulnerability CVE-2013-2178 ( original ref: https://vndh.net/note:fail2ban-089-denial-service ). -An example bad regex for apache: +An example bad regex for Apache: failregex = [[]client []] user .* not found @@ -370,10 +371,10 @@ Now the log line will be: As this log line doesn't match other expressions hence it matches the above regex and blocks 192.168.33.1 as a denial of service from the HTTP requester. -3. Applicaiton generates two identical log messages with different meanings +3. Application generates two identical log messages with different meanings If the application generates the following two messages under different -circmstances: +circumstances: client : authentication failed client : authentication failed @@ -409,7 +410,7 @@ coverage run fail2ban-testcases coverage html Then look at htmlcov/index.html and see how much coverage your test cases -exert over the codebase. Full coverage is a good thing however it may not be +exert over the code base. Full coverage is a good thing however it may not be complete. Try to ensure tests cover as many independent paths through the code. @@ -500,7 +501,7 @@ Design Fail2Ban was initially developed with Python 2.3 (IIRC). It should still be compatible with Python 2.4 and such compatibility assurance makes code ... old-fashioned in many places (RF-Note). In 0.7 the -design went through major refactoring into client/server, +design went through major re-factoring into client/server, a-thread-per-jail design which made it a bit difficult to follow. Below you can find a sketchy description of the main components of the system to orient yourself better. @@ -611,7 +612,7 @@ one way or another provide except FailManagerEmpty: self.failManager.cleanup(MyTime.time()) -thus channeling "ban tickets" from their failManager to the +thus channelling "ban tickets" from their failManager to the corresponding jail. action.py