From df94ec4c5267d96b09ae9696c35aed3239e5a053 Mon Sep 17 00:00:00 2001 From: Vladimir Varlamov Date: Fri, 22 Mar 2024 00:16:41 +0300 Subject: [PATCH] filter.d/exim.conf: rewrite host line regex for all varied exim's log_selector states Depending on Exim's log_selector settings, log lines may contain additional information about the connection. And also the line itself with the address of the remote host can vary greatly. But fortunately, all states can be found in the Exim code itself and taken into account. Makes it easier to add new regexps. Closes #3263 --- ChangeLog | 4 +- config/filter.d/exim-common.conf | 70 ++++++++++++++++++++++++++++---- config/filter.d/exim-spam.conf | 14 +++---- config/filter.d/exim.conf | 27 ++++++------ fail2ban/tests/files/logs/exim | 5 +++ 5 files changed, 89 insertions(+), 31 deletions(-) diff --git a/ChangeLog b/ChangeLog index e0358d03..19a44460 100644 --- a/ChangeLog +++ b/ChangeLog @@ -28,9 +28,11 @@ ver. 1.0.3-dev-1 (20??/??/??) - development nightly edition (value read from `/proc/sys/net/ipv6/conf/all/disable_ipv6`) if available, otherwise seeks over local IPv6 from network interfaces if available for platform and uses DNS to find local IPv6 as a fallback only * improve `ignoreself` by considering all local addresses from network interfaces additionally to IPs from hostnames (gh-3132) +* `filter.d/exim.conf`: + - rewrite host line regex for all varied exim's log_selector states (gh-3263) + - fixed "dropped: too many ..." regex, also matching unrecognized commands now (gh-3502) * `action.d/mikrotik.conf` - new action for mikrotik routerOS, adds and removes entries from address lists on the router (gh-2860) * `action.d/smtp.py` - added optional support for TLS connections via the `ssl` arg. -* `filter.d/exim.conf` - fixed "dropped: too many ..." regex, also matching unrecognized commands now (gh-3502) * `filter.d/nginx-forbidden.conf` - new filter to ban forbidden locations, e. g. using `deny` directive (gh-2226) * `filter.d/sshd.conf`: - avoid double counting for "maximum authentication attempts exceeded" (gh-3502) diff --git a/config/filter.d/exim-common.conf b/config/filter.d/exim-common.conf index 36644e94..a4c0491c 100644 --- a/config/filter.d/exim-common.conf +++ b/config/filter.d/exim-common.conf @@ -9,12 +9,68 @@ after = exim-common.local [Definition] -host_info_pre = (?:H=([\w.-]+ )?(?:\(\S+\) )?)? -host_info_suf = (?::\d+)?(?: I=\[\S+\](:\d+)?)?(?: U=\S+)?(?: P=e?smtp)?(?: F=(?:<>|[^@]+@\S+))?\s -host_info = %(host_info_pre)s\[\]%(host_info_suf)s +cleaner = (?: (?:C|Ci|CV|D|DKIM|DN|DS|DT|F|I|K|L|M8S|P|PRDR|PRX|Q|QT|R|RT|S|SNI|ST|T|TFO|U|X)(?:=\S+)?)* +host_info = %(cleaner)s (?:H=)?(?:[\w.-]+)? ?(?:\(\S+\))? ?\[\](?::\d+)?%(cleaner)s pid = (?: \[\d+\]| \w+ exim\[\d+\]:)? -# DEV Notes: -# From exim source code: ./src/receive.c:add_host_info_for_log -# -# Author: Daniel Black + +# DEV Notes +# ------------ +# Host string happens: +# H=[ip address] +# H=(helo_name) [ip address] +# H=host_name [ip address] +# H=host_name (helo_name) [ip address] +# flags H=host_name (helo_name) [ip address] flags +# where only [ip address] always visible, ignore ident +# From exim source code: +# src/src/host.c:host_and_ident() +# src/receive.c:add_host_info_for_log() + +# Cleaner removing all flags but H +# Summary of Fields in Log Lines depending on log_selector +# https://www.exim.org/exim-html-current/doc/html/spec_html/ch-log_files.html +# at version exim-4.97.1 +# --- +# A authenticator name (and optional id and sender) +# C SMTP confirmation on delivery +# Ci connection identifier +# command list for “no mail in SMTP session” +# CV certificate verification status +# D duration of “no mail in SMTP session” +# DKIM domain verified in incoming message +# DN distinguished name from peer certificate +# DS DNSSEC secured lookups +# DT on =>, == and ** lines: time taken for, or to attempt, a delivery +# F sender address (on delivery lines) +# H host name and IP address +# I local interface used +# id message id (from header) for incoming message +# K CHUNKING extension used +# L on <= and => lines: PIPELINING extension used +# M8S 8BITMIME status for incoming message +# P on <= lines: protocol used +# on => and ** lines: return path +# PRDR PRDR extension used +# PRX on <= and => lines: proxy address +# Q alternate queue name +# QT on => lines: time spent on queue so far +# on “Completed” lines: time spent on queue +# R on <= lines: reference for local bounce +# on => >> ** and == lines: router name +# RT on <= lines: time taken for reception +# S size of message in bytes +# SNI server name indication from TLS client hello +# ST shadow transport name +# T on <= lines: message subject (topic) +# TFO connection took advantage of TCP Fast Open +# on => ** and == lines: transport name +# U local user or RFC 1413 identity +# X TLS cipher suite +# --- + +# Authors: +# Cyril Jaquier +# Daniel Black (rewrote with strong regexs) +# Martin O'Neal (added additional regexs to detect authentication failures, protocol errors, and drops) +# Varlamov Vladimir (host line definition) diff --git a/config/filter.d/exim-spam.conf b/config/filter.d/exim-spam.conf index 1038cc0b..0a283f46 100644 --- a/config/filter.d/exim-spam.conf +++ b/config/filter.d/exim-spam.conf @@ -26,9 +26,9 @@ before = exim-common.conf [Definition] -failregex = ^%(pid)s \S+ F=(<>|\S+@\S+) %(host_info)srejected by local_scan\(\): .{0,256}$ - ^%(pid)s %(host_info)sF=(<>|[^@]+@\S+) rejected RCPT [^@]+@\S+: .*dnsbl.*\s*$ - ^%(pid)s \S+ %(host_info)sF=(<>|[^@]+@\S+) rejected after DATA: This message contains a virus \(\S+\)\.\s*$ +failregex = ^%(pid)s \S+%(host_info)s rejected by local_scan\(\): .{0,256}$ + ^%(pid)s%(host_info)s rejected RCPT [^@]+@\S+: .*dnsbl.*\s*$ + ^%(pid)s \S+%(host_info)s rejected after DATA: This message contains a virus \(\S+\)\.\s*$ ^%(pid)s \S+ SA: Action: flagged as Spam but accepted: score=\d+\.\d+ required=\d+\.\d+ \(scanned in \d+/\d+ secs \| Message-Id: \S+\)\. From \S+ \(host=\S+ \[\]\) for $ ^%(pid)s \S+ SA: Action: silently tossed message: score=\d+\.\d+ required=\d+\.\d+ trigger=\d+\.\d+ \(scanned in \d+/\d+ secs \| Message-Id: \S+\)\. From \S+ \(host=(\S+ )?\[\]\) for \S+$ @@ -43,8 +43,6 @@ ignoreregex = honeypot = trap@example.com -# DEV Notes: -# The %(host_info) definition contains a match -# -# Author: Cyril Jaquier -# Daniel Black (rewrote with strong regexs) +# DEV Notes +# ----------- +# The %(host_info) definition contains a match. No space before. See exim-common.conf diff --git a/config/filter.d/exim.conf b/config/filter.d/exim.conf index 854c8ba7..7d4144a7 100644 --- a/config/filter.d/exim.conf +++ b/config/filter.d/exim.conf @@ -14,16 +14,16 @@ before = exim-common.conf [Definition] # Fre-filter via "prefregex" is currently inactive because of too different failure syntax in exim-log (testing needed): -#prefregex = ^%(pid)s \b(?:\w+ authenticator failed|([\w\-]+ )?SMTP (?:(?:call|connection) from|protocol(?: synchronization)? error)|no MAIL in|(?:%(host_info_pre)s\[[^\]]+\]%(host_info_suf)s(?:sender verify fail|rejected RCPT|dropped|AUTH command))).+$ +#prefregex = ^%(pid)s \b(?:\w+ authenticator failed|([\w\-]+ )?SMTP (?:(?:call|connection) from|protocol(?: synchronization)? error)|no MAIL in|(?:%(host_info)s(?:sender verify fail|rejected RCPT|dropped|AUTH command))).+$ -failregex = ^%(pid)s %(host_info)ssender verify fail for <\S+>: (?:Unknown user|Unrouteable address|all relevant MX records point to non-existent hosts)\s*$ - ^%(pid)s \w+ authenticator failed for (?:[^\[\( ]* )?(?:\(\S*\) )?\[\](?::\d+)?(?: I=\[\S+\](:\d+)?)?: 535 Incorrect authentication data( \(set_id=.*\)|: \d+ Time\(s\))?\s*$ - ^%(pid)s %(host_info)srejected RCPT [^@]+@\S+: (?:relay not permitted|Sender verify failed|Unknown user|Unrouteable address)\s*$ - ^%(pid)s SMTP protocol synchronization error \([^)]*\): rejected (?:connection from|"\S+") %(host_info)s(?:next )?input=".*"\s*$ - ^%(pid)s SMTP call from (?:[^\[\( ]* )?%(host_info)sdropped: too many (?:(?:nonmail|unrecognized) commands|syntax or protocol errors) - ^%(pid)s SMTP protocol error in "[^"]+(?:"+[^"]*(?="))*?" %(host_info)sAUTH command used when not advertised\s*$ - ^%(pid)s no MAIL in SMTP connection from (?:[^\[\( ]* )?(?:\(\S*\) )?%(host_info)sD=\d\S*s(?: C=\S*)?\s*$ - ^%(pid)s (?:[\w\-]+ )?SMTP connection from (?:[^\[\( ]* )?(?:\(\S*\) )?%(host_info)sclosed by DROP in ACL\s*$ +failregex = ^%(pid)s%(host_info)s sender verify fail for <\S+>: (?:Unknown user|Unrouteable address|all relevant MX records point to non-existent hosts)\s*$ + ^%(pid)s \w+ authenticator failed for%(host_info)s: 535 Incorrect authentication data(?: \(set_id=.*\)|: \d+ Time\(s\))?\s*$ + ^%(pid)s%(host_info)s rejected RCPT [^@]+@\S+: (?:relay not permitted|Sender verify failed|Unknown user|Unrouteable address)\s*$ + ^%(pid)s SMTP protocol synchronization error \([^)]*\): rejected (?:connection from|"\S+")%(host_info)s (?:next )?input=".*"\s*$ + ^%(pid)s SMTP call from%(host_info)s dropped: too many (?:(?:nonmail|unrecognized) commands|syntax or protocol errors) + ^%(pid)s SMTP protocol error in "[^"]+(?:"+[^"]*(?="))*?"%(host_info)s AUTH command used when not advertised\s*$ + ^%(pid)s no MAIL in SMTP connection from%(host_info)s + ^%(pid)s (?:[\w\-]+ )?SMTP connection from%(host_info)s closed by DROP in ACL\s*$ > mdre-aggressive = ^%(pid)s no host name found for IP address $ @@ -42,13 +42,10 @@ mode = normal ignoreregex = -# DEV Notes: -# The %(host_info) definition contains a match +# DEV Notes +# ----------- +# The %(host_info) definition contains a match. No space before. See exim-common.conf # # SMTP protocol synchronization error \([^)]*\) <- This needs to be non-greedy # to void capture beyond ")" to avoid a DoS Injection vulnerability as input= is # user injectable data. -# -# Author: Cyril Jaquier -# Daniel Black (rewrote with strong regexs) -# Martin O'Neal (added additional regexs to detect authentication failures, protocol errors, and drops) diff --git a/fail2ban/tests/files/logs/exim b/fail2ban/tests/files/logs/exim index 04a47af7..94bca085 100644 --- a/fail2ban/tests/files/logs/exim +++ b/fail2ban/tests/files/logs/exim @@ -106,3 +106,8 @@ 2017-12-03 08:32:00 no host name found for IP address 192.0.2.8 # failJSON: { "time": "2017-12-03T08:51:35", "match": true , "host": "192.0.2.9", "desc": "no IP found for host" } 2017-12-03 08:51:35 no IP address found for host test.example.com (during SMTP connection from [192.0.2.9]) + +# failJSON: { "time": "2024-03-21T19:26:06", "match": true , "host": "194.169.175.1" } +2024-03-21 19:26:06 dovecot_login authenticator failed for (User) [194.169.175.1]:21298 I=[22.33.44.55]:465 Ci=30416: 535 Incorrect authentication data (set_id=uaf589@example.com) +# failJSON: { "time": "2024-03-21T09:18:51", "match": true , "host": "9.12.1.21" } +2024-03-21 09:18:51 H=m05.horp.tld [9.12.1.21]:43030 I=[194.169.175.2]:25 Ci=7326 CV=no SNI=mail.leone.tld F= rejected RCPT : relay not permitted