Created Developing Regex in Fail2ban (markdown)

Developing-Regex-in-Fail2ban.md

@@ -0,0 +1,248 @@
+So, you’re eager to write a new fail2ban filter and it failed … miserably or you have a new case but unsure how to get the best regex ... fastest.
+
+If the `fail2ban` couldn’t match anything … regardless of whether it is standard fail2ban config or your highly, purportedly, hapzardly-concoted filter config file: this page is for you.
+
+That is what this page offers, specifically developing as well as troubleshooting Regex used by `fail2ban`.
+
+WHAT ARE THE STAGES OF REGEX?
+=============================
+
+Fail2ban has several components of regex in which to apply toward the log text, these components/subcomponents are:
+
+* datepattern
+* prefregex
+    * failregex
+    * ignoreregex
+
+Usually, the date starts at the beginning of each log line that `fail2ban` searches against. For this article, we shall assume that date comes firstly before anything.
+
+ACTUAL EXAMPLES!
+================
+The actual examples were obtained during a DDOS against my Bind9 master nameserver. And a regex is needed ... fast.
+
+Actual log file is (after privacy redactions):
+```log
+19-Sep-2020 11:47:00.116 query-errors: info: client @0x7f0410000e40 123.123.123.123#80 (sl): view red: query failed (REFUSED) for sl/IN/ANY at query.c:5445
+19-Sep-2020 11:47:01.120 query-errors: info: client @0x7f0410000e40 123.123.123.123#80 (sl): view red: query failed (REFUSED) for sl/IN/ANY at query.c:5445
+19-Sep-2020 11:47:02.020 query-errors: info: client @0x7f0410000e40 123.123.123.123#80 (sl): view red: query failed (REFUSED) for sl/IN/ANY at query.c:5445
+19-Sep-2020 11:47:03.356 query-errors: info: client @0x7f0410000e40 123.123.123.123#80 (sl): view red: query failed (REFUSED) for sl/IN/ANY at query.c:5445
+19-Sep-2020 11:47:04.988 query-errors: info: client @0x7f0410000e40 123.123.123.123#80 (sl): view red: query failed (REFUSED) for sl/IN/ANY at query.c:5445
+19-Sep-2020 11:47:05.576 query-errors: info: client @0x7f0410000e40 123.123.123.123#80 (sl): view red: query failed (REFUSED) for sl/IN/ANY at query.c:5445
+```
+Note: A little history, the sl TLD went off-line and IoTs were spraying invalid DNS-QUERY records with falsified source IP address toward selected DNS servers, resulting in a mild DNS amplification attack via DNS-QUERY-REFUSED error message all being sent to the target victim.
+
+Sadly, latest Bind9 daemon has no configurable field to suppress these false DNS-QUERY-REFUSED acknowledgement messages (ISC Bind team claim it is not kosher to do this, but I still have a problem and intend fail2ban to deal with it).
+
+FIRST PATTERN, FIRST
+====================
+Hopefully you got a ‘date’ hit. Something like from your `fail2ban-regex -v` (please note the important -v command line option):
+```bash
+fail2ban-regex -v /tmp/captured.log /etc/fail2ban/filter.d/named-refused.conf
+```
+which outputted the following:
+```console
+...
+Date template hits:
+|- [# of hits] date format
+|  [6] {^LN-BEG}Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
+...
+```
+whose output shows `[6]` lines that have matched the date timestamp at the beginning of each line. That’s an excellent good start for troubleshooting.
+
+In the sad case of [0] match for a date pattern hit, use the `--VD` option along with `-l HEAVYDEBUG` option in your `fail2ban-regex`. Having a [0] means you are dealing with a log text whose `datepattern` that fail2ban has never dealt with before. You’ll need to craft your own `datepattern`.
+
+Such unknown `datepattern` shall be a subject for another blog, not here.
+
+PRE-FILTER MATCHED
+==================
+In every filter file, `prefregex` defaults to `^(?P<content>.+)$`. If you haven’t touch or set the `prefregex`, move on to the next section.
+
+Otherwise, `prefregex` becomes your focus in troubleshooting.
+
+You can tell that the (default or customized) `prefregex` actually works if you added `-l HEAVYDEBUG` to your `fail2ban-regex` command line:
+```bash
+fail2ban-regex \
+    -v \
+    -l HEAVYDEBUG \
+    /tmp/captured.log \
+    /etc/fail2ban/filter.d/named-refused.conf
+``
+Remember the above command; we are going to use it each time we modified the filter configuration file: and quite very often. Use your bash history buffer and recall that command, over and over again. Remember.
+
+and its output shows a line starting with `T: Pre-filter matched`:
+
+H:   Looking for prefregex '^(?P<content>.+)$'
+T:   Pre-filter matched {'content': ' query-errors: info: client @0x7f01e00004e0 123.123.123.123#80 (sl): view red: query failed (REFUSED) for sl/IN/ANY at query.c:5445'}
+
+and note the value of `'content:'`. This content comes after the `datepattern`; we have successfully parse the date timestamp.  Next, remaining content is then fed into the `failregex` patterns.
+
+Note: Please note in 'content': value that there is an extra space at the beginning of that value so be careful with the ‘`^`‘ and make sure it starts with ‘`^ `‘ (note a space after caret symbol.)
+
+But, with regard to that extra space char, do what I do; incorporate that space into your `prefregex`. Your customized `prefregex` will take away that beginning but lone space character from all your future (and current) `failregex` filter patterns. This makes for an easier-to-read `failpregex` pattern(s).
+```ini
+prefregex = ^ <F-CONTENT>.+</F-CONTENT>$
+```
+The above custom `prefregex` will ensure that that beginning space character is removed before sending the remaining content to the `failregex`.  This new `prefregex` returns just the interesting '`<F-CONTENT>.+</F-CONTENT>$`' which is basically everything after that lone (but unwanted) space char.
+
+Running that fail2ban-regex with the '`-l HEAVYDEBUG`', the new output shows:
+```console
+T:   Pre-filter matched {'content': 'query-errors: info: client @0x7f0410000e40 123.123.123.123#80 (sl): view red: query failed (REFUSED) for sl/IN/ANY at query.c:5445'}
+```
+Notice that a space no longer exist before '`query-errors`'.
+
+Everything from the beginning of the first non-space to the end of the line can then be dealt with by our yet-to-be-defined `failregex`.
+
+FAILREGEX MATCHED
+==================
+Focus on the `failregex` portion of the filter config file.  They're under `[Definition]`. 
+
+The catch of using `failregex` is that there MUST be at least one regex group match such as '`<HOST>`', '`<ADDR>`', or '`<F-USER>`'.
+
+So, do what I do… Make a generic failregex` in your filter confing file like this:
+```ini
+failregex = query.+<HOST>
+```
+Notice that there is no '`$`' to catch end-of-line match condition? We’ll do those '`$`' lastly because we’re trying to just match … ANYTHING!
+
+Re-run fail2ban-regex with -l HEAVYDEBUG and notice the '`T: Matched FailRegex part`':
+```console
+T:   Matched FailRegex('query.+(?:(?:::f{4,6}:)?(?P<ip4>(?:\\d{1,3}\\.){3}\\d{1,3})|\\[?(?P<ip6>(?:[0-9a-fA-F]{1,4}::?|::){1,7}(?:[0-9a-fA-F]{1,4}|(?<=:):))\\]?|(?P<dns>[\\w\\-.^_]*\\w))')
+```
+Now I am matching SOMETHING! 
+
+Notice the convoluted patterns after '`query.+?`' These long patterns represent '`<HOST>`' part. We can safely ignore that for now.
+
+Most importantly, I am MATCHING something that starts with '`^query`'! Yippee!
+GYRATING TOWARD FULL MATCH
+
+With a working matching pattern (albeit a failed but overly-broad pattern), we can then work toward a full-blown concise (yet flexible) pattern.
+
+Let’s start by adding more static pattern. I am pretty sure from my intensive examination of that line 5445 in Bind9 `query.c` source file that '`query-error: info:`' is something that will not change for my target condition. This log output may have other variance like '`query-error: warn`' or '`query-error: debug`' but I am ignoring those.
+
+First iteration of `failregex` expansion:
+```ini
+failregex = ^query-errors: info: .+<HOST>
+```
+Execute the command:
+```bash
+fail2ban-regex \
+    -l HEAVYDEBUG \
+    --print-no-missed \
+    /tmp/query-errors.log named-refused.local
+``
+and notice the output:
+```console
+Results
+=======
+
+Failregex: 6 total
+|-  #) [# of hits] regular expression
+|   1) [6] ^query-errors: info: .+<HOST>
+`-
+```
+See the '`[6]`'? I have six matches out of 6 lines give in log text file. I am getting close to a full-blown pattern! Don’t forget, we have to close that pattern out with a $ but not yet, save that for the end of this tutorial.
+
+CAUTION: Every time you make a change to your filter file, PAY VERY CLOSE ATTENTION to this part of the output:
+```console
+Failregex: X total tabulation. 
+```
+Once you get that '`Failregex: 0 total`', you know you have done something HORRIBLE, busted and broke your pattern, so roll that pattern back to its simpler pattern and start again.
+
+INCREMENTS, INCREMENTS, INCREMENTS
+==================================
+As we add more and more increments of pattern and ensuring that '`Failregex: 6 matches`' still appears:
+```console
+    failregex: query-errors: info: client.+<HOST>
+    failregex: query-errors: info: client @0x[0-9a-fA-F]{8,12}.+<HOST>
+```
+Whoa, my pattern is getting too long… so I made a variable to contain this entire pattern and called it '`_client`'.
+```ini
+_client = query-error: info: client @0x[0-9a-f]{8,12}
+```
+Now I can shorten the '`failregex`' a bit:
+```ini
+failregex = ^%(_client)s <HOST>
+```
+It’s the same thing, but oh it so readable, onward to matching the rest of the line.
+
+NOTE: You are running `fail2ban-regex` between each modification, aren’t you?
+
+NOW FOR THE ENDING PART
+=======================
+We have FINALLY reached the '`<HOST>`' part of the failregex/log text.
+
+Now it is closing time! Let’s race to the '`$`' (end).
+
+Add that port number after the host:
+```ini
+failregex = ^%(_client)s <HOST>#\d{1,5}
+```
+NOTE: You are still re-running fail2ban-regex between each modification, aren’t you?
+
+REPETITION
+==========
+Notice that the domain name '`sl`' got used twice in each of the same log line?
+
+Let us make a pattern called '`_domain`' to reduce our typing errors a bit.
+```ini
+_domain = [0-9a-zA-Z\._\-]{2,256}
+```
+Our new `failregex` becomes:
+```ini
+failregex = ^%(_client)s <HOST>#\d{1,5} \(%(_domain)s\):
+```
+NOTE: You are running `fail2ban-regex` between each modification, still getting that exact same match '`[6]`' (or whatever count you’re aiming for.)
+
+SIMPLIFICATION
+==============
+Now for the view part of nameserver error output where `ISC Bind9` handles the view name.
+View name is optional: We may not get a view name on some nameserver installation.
+```ini
+_view_name = [0-9a-zA-Z\._\-]{1,64}
+_view = ( \%(_domain)s\))?: view %(_view_name)s
+```
+Our latest `failregex` becomes:
+```ini
+failregex = ^%(_client)s <HOST>#\d{1,5}%(_view)s
+```
+Still have a long way to go before we add that '`$`' ending pattern.
+
+NOTE: `fail2ban-regex` between each modification still?
+
+FINAL STRETCH
+=============
+We have the remaining of log text left to go:
+```console
+query failed (REFUSED) for example.tld/IN/ANY at query.c:5445
+```
+We’re impatient lot, aren’t we? Rush it up with:
+```ini
+_query_refused = query failed \(REFUSED\) for %(_dns_tuple)s at %(_codeloc)s$
+```
+and supply missing defines:
+```ini
+_domain = [0-9a-zA-Z\._\-]{1,254}
+_dns_tuple = %(_domain)s\/IN\/ANY
+
+_filespec = [0-9a-zA-Z\._\-]{1,254}
+_codeloc = %(_filespec)s:\d{1,6}
+```
+
+NOTE: Guess? You are running fail2ban-regex between each modification? You still getting that non-zero '`Failregex: 6 total match`' under Results?
+```console
+Failregex: 6 total
+|-  #) [# of hits] regular expression
+|   1) [6] ^query-errors: info: client @0x[0-9a-f]{8,12} <HOST>#\d{1,5}( \([0-9a-zA-Z\._\-]{1,254}\))?: view [0-9a-zA-Z\._\-]{1,64}: query failed \(REFUSED\) for [0-9a-zA-Z\._\-]{1,254}\/IN\/ANY at [0-9a-zA-Z\._\-]{1,254}:\d{1,6}$
+`-
+```
+Ok, you could have paid attention to the last line of the output:
+```console
+Lines: 6 lines, 0 ignored, 6 matched, 0 missed
+```
+But I’ve find this to be easily overlooked hence the focal point within the Failregex as being more informative.
+
+CONCLUSION
+==========
+Now we can add the `'$'` to the end of `failregex`.
+
+Execute `fail2ban-client reload` and watch the blocking begin.