From 21e9bd76abd0e9902e78eba5e3104a84560c7096 Mon Sep 17 00:00:00 2001 From: Egbert <10352354+egberts@users.noreply.github.com> Date: Sun, 20 Sep 2020 18:25:15 -0400 Subject: [PATCH] Created Developing Regex in Fail2ban (markdown) --- Developing-Regex-in-Fail2ban.md | 248 ++++++++++++++++++++++++++++++++ 1 file changed, 248 insertions(+) create mode 100644 Developing-Regex-in-Fail2ban.md diff --git a/Developing-Regex-in-Fail2ban.md b/Developing-Regex-in-Fail2ban.md new file mode 100644 index 0000000..ad850b3 --- /dev/null +++ b/Developing-Regex-in-Fail2ban.md @@ -0,0 +1,248 @@ +So, you’re eager to write a new fail2ban filter and it failed … miserably or you have a new case but unsure how to get the best regex ... fastest. + +If the `fail2ban` couldn’t match anything … regardless of whether it is standard fail2ban config or your highly, purportedly, hapzardly-concoted filter config file: this page is for you. + +That is what this page offers, specifically developing as well as troubleshooting Regex used by `fail2ban`. + +WHAT ARE THE STAGES OF REGEX? +============================= + +Fail2ban has several components of regex in which to apply toward the log text, these components/subcomponents are: + +* datepattern +* prefregex + * failregex + * ignoreregex + +Usually, the date starts at the beginning of each log line that `fail2ban` searches against. For this article, we shall assume that date comes firstly before anything. + +ACTUAL EXAMPLES! +================ +The actual examples were obtained during a DDOS against my Bind9 master nameserver. And a regex is needed ... fast. + +Actual log file is (after privacy redactions): +```log +19-Sep-2020 11:47:00.116 query-errors: info: client @0x7f0410000e40 123.123.123.123#80 (sl): view red: query failed (REFUSED) for sl/IN/ANY at query.c:5445 +19-Sep-2020 11:47:01.120 query-errors: info: client @0x7f0410000e40 123.123.123.123#80 (sl): view red: query failed (REFUSED) for sl/IN/ANY at query.c:5445 +19-Sep-2020 11:47:02.020 query-errors: info: client @0x7f0410000e40 123.123.123.123#80 (sl): view red: query failed (REFUSED) for sl/IN/ANY at query.c:5445 +19-Sep-2020 11:47:03.356 query-errors: info: client @0x7f0410000e40 123.123.123.123#80 (sl): view red: query failed (REFUSED) for sl/IN/ANY at query.c:5445 +19-Sep-2020 11:47:04.988 query-errors: info: client @0x7f0410000e40 123.123.123.123#80 (sl): view red: query failed (REFUSED) for sl/IN/ANY at query.c:5445 +19-Sep-2020 11:47:05.576 query-errors: info: client @0x7f0410000e40 123.123.123.123#80 (sl): view red: query failed (REFUSED) for sl/IN/ANY at query.c:5445 +``` +Note: A little history, the sl TLD went off-line and IoTs were spraying invalid DNS-QUERY records with falsified source IP address toward selected DNS servers, resulting in a mild DNS amplification attack via DNS-QUERY-REFUSED error message all being sent to the target victim. + +Sadly, latest Bind9 daemon has no configurable field to suppress these false DNS-QUERY-REFUSED acknowledgement messages (ISC Bind team claim it is not kosher to do this, but I still have a problem and intend fail2ban to deal with it). + +FIRST PATTERN, FIRST +==================== +Hopefully you got a ‘date’ hit. Something like from your `fail2ban-regex -v` (please note the important -v command line option): +```bash +fail2ban-regex -v /tmp/captured.log /etc/fail2ban/filter.d/named-refused.conf +``` +which outputted the following: +```console +... +Date template hits: +|- [# of hits] date format +| [6] {^LN-BEG}Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? +... +``` +whose output shows `[6]` lines that have matched the date timestamp at the beginning of each line. That’s an excellent good start for troubleshooting. + +In the sad case of [0] match for a date pattern hit, use the `--VD` option along with `-l HEAVYDEBUG` option in your `fail2ban-regex`. Having a [0] means you are dealing with a log text whose `datepattern` that fail2ban has never dealt with before. You’ll need to craft your own `datepattern`. + +Such unknown `datepattern` shall be a subject for another blog, not here. + +PRE-FILTER MATCHED +================== +In every filter file, `prefregex` defaults to `^(?P.+)$`. If you haven’t touch or set the `prefregex`, move on to the next section. + +Otherwise, `prefregex` becomes your focus in troubleshooting. + +You can tell that the (default or customized) `prefregex` actually works if you added `-l HEAVYDEBUG` to your `fail2ban-regex` command line: +```bash +fail2ban-regex \ + -v \ + -l HEAVYDEBUG \ + /tmp/captured.log \ + /etc/fail2ban/filter.d/named-refused.conf +`` +Remember the above command; we are going to use it each time we modified the filter configuration file: and quite very often. Use your bash history buffer and recall that command, over and over again. Remember. + +and its output shows a line starting with `T: Pre-filter matched`: + +H: Looking for prefregex '^(?P.+)$' +T: Pre-filter matched {'content': ' query-errors: info: client @0x7f01e00004e0 123.123.123.123#80 (sl): view red: query failed (REFUSED) for sl/IN/ANY at query.c:5445'} + +and note the value of `'content:'`. This content comes after the `datepattern`; we have successfully parse the date timestamp. Next, remaining content is then fed into the `failregex` patterns. + +Note: Please note in 'content': value that there is an extra space at the beginning of that value so be careful with the ‘`^`‘ and make sure it starts with ‘`^ `‘ (note a space after caret symbol.) + +But, with regard to that extra space char, do what I do; incorporate that space into your `prefregex`. Your customized `prefregex` will take away that beginning but lone space character from all your future (and current) `failregex` filter patterns. This makes for an easier-to-read `failpregex` pattern(s). +```ini +prefregex = ^ .+$ +``` +The above custom `prefregex` will ensure that that beginning space character is removed before sending the remaining content to the `failregex`. This new `prefregex` returns just the interesting '`.+$`' which is basically everything after that lone (but unwanted) space char. + +Running that fail2ban-regex with the '`-l HEAVYDEBUG`', the new output shows: +```console +T: Pre-filter matched {'content': 'query-errors: info: client @0x7f0410000e40 123.123.123.123#80 (sl): view red: query failed (REFUSED) for sl/IN/ANY at query.c:5445'} +``` +Notice that a space no longer exist before '`query-errors`'. + +Everything from the beginning of the first non-space to the end of the line can then be dealt with by our yet-to-be-defined `failregex`. + +FAILREGEX MATCHED +================== +Focus on the `failregex` portion of the filter config file. They're under `[Definition]`. + +The catch of using `failregex` is that there MUST be at least one regex group match such as '``', '``', or '``'. + +So, do what I do… Make a generic failregex` in your filter confing file like this: +```ini +failregex = query.+ +``` +Notice that there is no '`$`' to catch end-of-line match condition? We’ll do those '`$`' lastly because we’re trying to just match … ANYTHING! + +Re-run fail2ban-regex with -l HEAVYDEBUG and notice the '`T: Matched FailRegex part`': +```console +T: Matched FailRegex('query.+(?:(?:::f{4,6}:)?(?P(?:\\d{1,3}\\.){3}\\d{1,3})|\\[?(?P(?:[0-9a-fA-F]{1,4}::?|::){1,7}(?:[0-9a-fA-F]{1,4}|(?<=:):))\\]?|(?P[\\w\\-.^_]*\\w))') +``` +Now I am matching SOMETHING! + +Notice the convoluted patterns after '`query.+?`' These long patterns represent '``' part. We can safely ignore that for now. + +Most importantly, I am MATCHING something that starts with '`^query`'! Yippee! +GYRATING TOWARD FULL MATCH + +With a working matching pattern (albeit a failed but overly-broad pattern), we can then work toward a full-blown concise (yet flexible) pattern. + +Let’s start by adding more static pattern. I am pretty sure from my intensive examination of that line 5445 in Bind9 `query.c` source file that '`query-error: info:`' is something that will not change for my target condition. This log output may have other variance like '`query-error: warn`' or '`query-error: debug`' but I am ignoring those. + +First iteration of `failregex` expansion: +```ini +failregex = ^query-errors: info: .+ +``` +Execute the command: +```bash +fail2ban-regex \ + -l HEAVYDEBUG \ + --print-no-missed \ + /tmp/query-errors.log named-refused.local +`` +and notice the output: +```console +Results +======= + +Failregex: 6 total +|- #) [# of hits] regular expression +| 1) [6] ^query-errors: info: .+ +`- +``` +See the '`[6]`'? I have six matches out of 6 lines give in log text file. I am getting close to a full-blown pattern! Don’t forget, we have to close that pattern out with a $ but not yet, save that for the end of this tutorial. + +CAUTION: Every time you make a change to your filter file, PAY VERY CLOSE ATTENTION to this part of the output: +```console +Failregex: X total tabulation. +``` +Once you get that '`Failregex: 0 total`', you know you have done something HORRIBLE, busted and broke your pattern, so roll that pattern back to its simpler pattern and start again. + +INCREMENTS, INCREMENTS, INCREMENTS +================================== +As we add more and more increments of pattern and ensuring that '`Failregex: 6 matches`' still appears: +```console + failregex: query-errors: info: client.+ + failregex: query-errors: info: client @0x[0-9a-fA-F]{8,12}.+ +``` +Whoa, my pattern is getting too long… so I made a variable to contain this entire pattern and called it '`_client`'. +```ini +_client = query-error: info: client @0x[0-9a-f]{8,12} +``` +Now I can shorten the '`failregex`' a bit: +```ini +failregex = ^%(_client)s +``` +It’s the same thing, but oh it so readable, onward to matching the rest of the line. + +NOTE: You are running `fail2ban-regex` between each modification, aren’t you? + +NOW FOR THE ENDING PART +======================= +We have FINALLY reached the '``' part of the failregex/log text. + +Now it is closing time! Let’s race to the '`$`' (end). + +Add that port number after the host: +```ini +failregex = ^%(_client)s #\d{1,5} +``` +NOTE: You are still re-running fail2ban-regex between each modification, aren’t you? + +REPETITION +========== +Notice that the domain name '`sl`' got used twice in each of the same log line? + +Let us make a pattern called '`_domain`' to reduce our typing errors a bit. +```ini +_domain = [0-9a-zA-Z\._\-]{2,256} +``` +Our new `failregex` becomes: +```ini +failregex = ^%(_client)s #\d{1,5} \(%(_domain)s\): +``` +NOTE: You are running `fail2ban-regex` between each modification, still getting that exact same match '`[6]`' (or whatever count you’re aiming for.) + +SIMPLIFICATION +============== +Now for the view part of nameserver error output where `ISC Bind9` handles the view name. +View name is optional: We may not get a view name on some nameserver installation. +```ini +_view_name = [0-9a-zA-Z\._\-]{1,64} +_view = ( \%(_domain)s\))?: view %(_view_name)s +``` +Our latest `failregex` becomes: +```ini +failregex = ^%(_client)s #\d{1,5}%(_view)s +``` +Still have a long way to go before we add that '`$`' ending pattern. + +NOTE: `fail2ban-regex` between each modification still? + +FINAL STRETCH +============= +We have the remaining of log text left to go: +```console +query failed (REFUSED) for example.tld/IN/ANY at query.c:5445 +``` +We’re impatient lot, aren’t we? Rush it up with: +```ini +_query_refused = query failed \(REFUSED\) for %(_dns_tuple)s at %(_codeloc)s$ +``` +and supply missing defines: +```ini +_domain = [0-9a-zA-Z\._\-]{1,254} +_dns_tuple = %(_domain)s\/IN\/ANY + +_filespec = [0-9a-zA-Z\._\-]{1,254} +_codeloc = %(_filespec)s:\d{1,6} +``` + +NOTE: Guess? You are running fail2ban-regex between each modification? You still getting that non-zero '`Failregex: 6 total match`' under Results? +```console +Failregex: 6 total +|- #) [# of hits] regular expression +| 1) [6] ^query-errors: info: client @0x[0-9a-f]{8,12} #\d{1,5}( \([0-9a-zA-Z\._\-]{1,254}\))?: view [0-9a-zA-Z\._\-]{1,64}: query failed \(REFUSED\) for [0-9a-zA-Z\._\-]{1,254}\/IN\/ANY at [0-9a-zA-Z\._\-]{1,254}:\d{1,6}$ +`- +``` +Ok, you could have paid attention to the last line of the output: +```console +Lines: 6 lines, 0 ignored, 6 matched, 0 missed +``` +But I’ve find this to be easily overlooked hence the focal point within the Failregex as being more informative. + +CONCLUSION +========== +Now we can add the `'$'` to the end of `failregex`. + +Execute `fail2ban-client reload` and watch the blocking begin. \ No newline at end of file