mirror of https://github.com/fail2ban/fail2ban
added TL;DR up-front.
Egbert
parent
0a35688424
commit
1b9effd9fd
|
|
@ -4,6 +4,78 @@ If the `fail2ban` couldn’t match anything … regardless of whether it is stan
|
|||
|
||||
That is what this page offers, specifically developing as well as troubleshooting Regex used by `fail2ban`.
|
||||
|
||||
TL;DR
|
||||
=====
|
||||
Basic Test Iteration
|
||||
--------------------
|
||||
Basically, you will be armed on going back-and-forth with the following commands:
|
||||
```bash
|
||||
fail2ban-regex \
|
||||
-l HEAVYDEBUG \
|
||||
--print-no-missed \
|
||||
/tmp/query-errors.log named-refused.conf
|
||||
```
|
||||
then execute
|
||||
```bash
|
||||
$EDITOR /etc/fail2ban/filter.d/named-refused.local
|
||||
```
|
||||
Add more pattern to:
|
||||
* `datapattern` as needed until it pass
|
||||
* `prefregex` as needed until it pass
|
||||
* `failregex` until it pass
|
||||
|
||||
Rinse, lather, repeat often as needed.
|
||||
|
||||
Date Pattern
|
||||
------------
|
||||
You first tests for a working `datapattern`:
|
||||
...
|
||||
Date template hits:
|
||||
|- [# of hits] date format
|
||||
| [6] {^LN-BEG}Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
|
||||
...
|
||||
```
|
||||
Note the '`[6]`' output which matches my examples given in the next top-level section below.
|
||||
|
||||
Pre-Filtering
|
||||
-------------
|
||||
After clearing/setting the `prefregex`, test it again and watch for `Pre-filter matched` in your output:
|
||||
```console
|
||||
H: Looking for prefregex '^(?P<content>.+)$'
|
||||
T: Pre-filter matched {'content': ' query-errors: info: client @0x7f01e00004e0 123.123.123.123#80 (sl): view red: query failed (REFUSED) for sl/IN/ANY at query.c:5445'}
|
||||
```
|
||||
Ensure that the {'content': '<your-target-pattern'} is viewable and it's all there.
|
||||
|
||||
Fail Pattern
|
||||
------------
|
||||
After ensuring a success `datepattern` (and optionally the `prefregex`) match, set your `failregex` to a generic pattern of `.+ <HOST>`.
|
||||
|
||||
Then watch for `Results` tabulation in your output:
|
||||
```console
|
||||
Results
|
||||
=======
|
||||
|
||||
Failregex: 6 total
|
||||
|- #) [# of hits] regular expression
|
||||
| 1) [6] ^<your-target-pattern>
|
||||
`-
|
||||
```
|
||||
During your many iterative test cycle, pay attention to the following output for the match count (of our example '`[6]`' lines):
|
||||
|
||||
At any time during your edit cycle of expanding pattern, that you get '`[0]`' match,
|
||||
```console
|
||||
Failregex: 0 total tabulation.
|
||||
```
|
||||
that is A screw-up, go back and revert the `failregex` back to last-known working pattern then resume test cycle.
|
||||
|
||||
When no more patterns can be added to `failregex`, end it with a '`$`' pattern.
|
||||
|
||||
Restart `fail2ban`.
|
||||
```bash
|
||||
fail2ban-client reload
|
||||
```
|
||||
For the hoary and gory details, read on...
|
||||
|
||||
WHAT ARE THE STAGES OF REGEX?
|
||||
=============================
|
||||
|
||||
|
|
@ -35,7 +107,7 @@ Sadly, latest Bind9 daemon has no configurable field to suppress these false DNS
|
|||
|
||||
FIRST PATTERN, FIRST
|
||||
====================
|
||||
Hopefully you got a ‘date’ hit. Something like from your `fail2ban-regex -v` (please note the important '`-v`' command line option):
|
||||
Hopefully you got that ‘date’ hit. `fail2ban` already provided MANY `datepattern` found in many log files. Execute something like `fail2ban-regex -v` (please note the important '`-v`' command line option):
|
||||
```bash
|
||||
fail2ban-regex -v /tmp/captured.log /etc/fail2ban/filter.d/named-refused.conf
|
||||
```
|
||||
|
|
@ -47,9 +119,11 @@ Date template hits:
|
|||
| [6] {^LN-BEG}Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
|
||||
...
|
||||
```
|
||||
whose output shows '`[6]`' lines that have matched the date timestamp at the beginning of each line. That’s an excellent good start for troubleshooting.
|
||||
whose output shows '`[6]`' lines that have matched the date timestamp at the beginning of each line. That’s an excellent good start for troubleshooting. `fail2ban` tries to get all known `datepattern`s.
|
||||
|
||||
In the sad case of '`[0]`' match for a date pattern hit, use the '`--VD`' option along with '`-l HEAVYDEBUG`' option in your `fail2ban-regex`. Having a [0] means you are dealing with a log text whose `datepattern` that fail2ban has never dealt with before. Then, you’ll need to craft your own `datepattern`.
|
||||
NOTE: Common `rsyslog` syslog daemon may output regular `datepattern` or high-precision `datepattern` (via `$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat` setting in `/etc/rsyslog.conf`) and fail2ban `datepattern` gets both of these date format right.
|
||||
|
||||
In the rare event (and sad case) of '`[0]`' match for a date pattern hit, you can develop a new `datepattern by usng the '`--VD`' option along with '`-l HEAVYDEBUG`' option in your `fail2ban-regex`. Having a [0] means you are dealing with a log text whose `datepattern` that fail2ban has never dealt with before; you’ll need to craft your own `datepattern`.
|
||||
|
||||
Such unknown `datepattern` shall be a subject for another blog, not here.
|
||||
|
||||
|
|
@ -57,27 +131,31 @@ PRE-FILTER MATCHED
|
|||
==================
|
||||
If you have a single-line pattern, skip this section and leave `prefregex` empty or undefined.
|
||||
|
||||
`prefregex` is ideally for a pattern of the entire log file that is inherently on all of the lines within same log file. Such common pattern found in line-by-line log file are:
|
||||
* date (pretty much always)
|
||||
`prefregex` is ideally for a pattern that is found (after the `datepattern` portion) in each and every line of the entire log file. Such common pattern found after `datepattern` MAY contain any 1 or more of the following:
|
||||
* daemon name (optional)
|
||||
* subroutine name and/or line number (optional)
|
||||
* process ID (optional)
|
||||
* severity level (optional)
|
||||
* many more...
|
||||
|
||||
So, the ideal `prefregex` would be highly dependent on proper supporting of this combinatorial of the above list of patterns (some always there, and others mostly optional) in order to make it work for everyone that uses the application which generates the logs.
|
||||
So, an ideal `prefregex` would be highly dependent on a regex that properly supports a combinatorial of the above list of patterns (some always there, and others mostly optional) in order to make it work for everyone that uses the application (often `rsyslogd` but its `named` here) which generates the logs.
|
||||
|
||||
Secondary benefit of `prefregex` is to ensure that `failregex` is left with the most dynamic (and interesting) part of the regex line. `prefregex` takes that most common parts (see above list) of the line.
|
||||
|
||||
```console
|
||||
<--- prefregex -->|<-- failregex ->
|
||||
3-Jan-2020 myscript: Dynamic error message part
|
||||
3-Jan-2020 myscript[12512]: Dynamic error message part
|
||||
```
|
||||
|
||||
Furthermore, the really good reason to support `prefregex`, single pattern or not, is to accommodate whichever daemon/script is writing into that log file as each user of your filter has to support the many user-defined but different configuration settings to include or exclude certain things into each line of the log file. If the daemon/script of that log file doesn't have any logging parameters that would affect its log file, then `prefregex` may not be for you to use.
|
||||
There's more: the really good reason to support `prefregex` is this: your end-user may tweak that daemon/script configuration file that results in a different line output. Your finished filter will surely break. So track those configuration settings related to log output.
|
||||
|
||||
That said, log line having different but additional info(s) after such `datepattern`. `prefregex` can be made flexible, for different end-users' daemon configuration settings.
|
||||
|
||||
No log settings? Then `prefregex` may not be for you to use.
|
||||
|
||||
To Pre-Filter or Not To Pre-Filter
|
||||
-----------------------------------
|
||||
This section only applies if you have (or will have) multiple patterns within this same filter file that you are creating or modifying.
|
||||
This section only applies if you have (or will have) multiple patterns. This section also applies if log line becomes different with its daemon's configuration settings related to log output.
|
||||
|
||||
If a pre-defined `prefregex` already existed and you know it works, then you can move on to the next section. If you are creating one, read on.
|
||||
|
||||
|
|
@ -190,7 +268,7 @@ Execute the command:
|
|||
fail2ban-regex \
|
||||
-l HEAVYDEBUG \
|
||||
--print-no-missed \
|
||||
/tmp/query-errors.log named-refused.local
|
||||
/tmp/query-errors.log named-refused.conf
|
||||
```
|
||||
and notice the output:
|
||||
```console
|
||||
|
|
|
|||
Loading…
Reference in New Issue