update changelog and documentation (new features and handling)

ChangeLog

@@ -11,6 +11,12 @@ ver. 1.1.1-dev-1 (20??/??/??) - development nightly edition
 -----------
 
 ### Fixes
+* fixes `systemd` bug with missing journal descriptor after rotation by reopening of journal if it is recognized as not alive (gh-3929)
+* improve threaded clean-up of all filters, new thread functions `afterStop` (to force clean-up after stop) and `done`, invoking `afterStop` once
+* ensure journal-reader is always closed (additional prevention against leaks and "too many open files"), thereby avoid sporadic segfault
+  in systemd module (see https://github.com/systemd/python-systemd/issues/143)
+* fixes `systemd` causing "too many open files" error for a lot of journal files and large amout of systemd jails 
+  (see new parameter `rotated` below, gh-3391);
 * `jail.conf`:
   - default banactions need to be specified in `paths-*.conf` (maintainer level) now
   - since stock fail2ban includes `paths-debian.conf` by default, banactions are `nftables`
@@ -43,8 +49,17 @@ ver. 1.1.1-dev-1 (20??/??/??) - development nightly edition
 * `filter.d/vsftpd.conf` - fixed regex (if failures generated by systemd-journal, gh-3954)
 
 ### New Features and Enhancements
+* backend `systemd` extended with new parameter `rotated` (default `false`, as prevention against "too many open files"),
+  that allows to monitor only actual journals and ignore now a lot of rotated files by default; so can drastically reduce
+  amount of used file descriptors, normally to 1 or 2 descriptors per jail (gh-3391)
 * new jail option `skip_if_nologs` to ignore jail if no `logpath` matches found, fail2ban continue to start with warnings/errors,
   thus other jails become running (gh-2756)
+* implements automatic switch `backend = auto` to backend `systemd`, when the following is true (RFE gh-3768):
+  - no files matching `logpath` found for this jail;
+  - no `systemd_if_nologs = false` is specified for the jail (`true` by default);
+  - option `journalmatch` is set for the jail or its filter (otherwise it'd be too heavy to allow all auto-jails,
+    even if they have never been foreseen for journal monitoring);
+  (option `skip_if_nologs` will be ignored if we could switch backend to `systemd`)
 * configuration `ignoreip` and fail2ban-client commands `addignoreip`/`delignoreip` extended with `file:...` syntax 
   to ignore IPs from file-ip-set (containing IP, subnet, dns/fqdn or raw strings); the file would be read lazy on demand,
   by first ban (and automatically reloaded by update after small latency to avoid expensive stats check on every compare);

man/jail.conf.5

@@ -182,6 +182,8 @@ This specifies the stack size (in KiB) to be used for subsequently created threa
 
 .SH "JAIL CONFIGURATION FILE(S) (\fIjail.conf\fB)"
 The following options are applicable to any jail. They appear in a section specifying the jail name or in the \fI[DEFAULT]\fR section which defines default values to be used if not specified in the individual section.
+.sp
+It is also possible to specify or to overwrite any option of filter file directly in jail (see section FILTER FILES).
 .TP
 .B filter
 name of the filter -- filename of the filter in /etc/fail2ban/filter.d/ without the .conf/.local extension.
@@ -198,7 +200,10 @@ Optional space separated option 'tail' can be added to the end of the path to ca
 Ensure syslog or the program that generates the log file isn't configured to compress repeated log messages to "\fI*last message repeated 5 time*s\fR" otherwise it will fail to detect. This is called \fIRepeatedMsgReduction\fR in rsyslog and should be \fIOff\fR.
 .TP
 .B skip_if_nologs
-if no logpath matches found, skip the jail by start of fail2ban if \fIskip_if_nologs\fR set to true, otherwise (default: false) start of fail2ban will fail with an error "Have not found any log file".
+if no logpath matches found, skip the jail by start of fail2ban if \fIskip_if_nologs\fR set to true, otherwise (default: false) start of fail2ban will fail with an error "Have not found any log file", unless the backend is \fIauto\fR and the jail is able to swith backend to \fIsystemd\fR (see \fIauto\fR in section \fBBackends\fR below).
+.TP
+.B systemd_if_nologs
+if no logpath matches found, switch backend \fIauto\fR to \fIsystemd\fR (see \fBBackends\fR section), unless disabled with \fBsystemd_if_nologs = false\fR (default \fBtrue\fR).
 .TP
 .B logencoding
 encoding of log files used for decoding. Default value of "auto" uses current system locale.
@@ -280,7 +285,7 @@ number of failures that have to occur in the last \fBfindtime\fR seconds to ban
 .B backend
 backend to be used to detect changes in the logpath.
 .br
-It defaults to "auto" which will try "pyinotify", "systemd" before "polling". Any of these can be specified. "pyinotify" is only valid on Linux systems with the "pyinotify" Python libraries.
+It defaults to "auto" which will try "pyinotify" before "polling" and may switch to "systemd" if no files matching \fBlogpath\fR will be found (see section \fBBackends\fR below). Any of these can be specified. "pyinotify" is only valid on Linux systems with the "pyinotify" Python libraries.
 .TP
 .B usedns
 use DNS to resolve HOST names that appear in the logs. By default it is "warn" which will resolve hostnames to IPs however it will also log a warning. If you are using DNS here you could be blocking the wrong IPs due to the asymmetric nature of reverse DNS (that the application used to write the domain name to log) compared to forward DNS that fail2ban uses to resolve this back to an IP (but not necessarily the same one). Ideally you should configure your applications to log a real IP. This can be set to "yes" to prevent warnings in the log or "no" to disable DNS resolution altogether (thus ignoring entries where hostname, not an IP is logged)..
@@ -300,21 +305,46 @@ max number of matched log-lines the jail would hold in memory per ticket. By def
 .SS Backends
 Available options are listed below.
 .TP
+.B auto
+automatically selects best suitable \fBbackend\fR, starting with file-based like \fIpyinotify\fR or \fIpolling\fR to monitor the \fBlogpath\fR matching files, but can also automatically switch to backend \fIsystemd\fR, when the following is true:
+.RS
+.IP \(bu 4n
+no files matching \fBlogpath\fR found for this jail;
+.IP \(bu 4n
+no \fBsystemd_if_nologs = false\fR is specified for the jail;
+.IP \(bu 4n
+option \fBjournalmatch\fR is set for the jail or its filter (otherwise it'd be too heavy to allow all auto-jails, even if they have never been foreseen for journal monitoring);
+.TP
+.br
+Option \fBskip_if_nologs\fR will be ignored if we could switch \fBbackend\fR to \fIsystemd\fR.
+.RE
+.TP
 .B pyinotify
-requires pyinotify (a file alteration monitor) to be installed. If pyinotify is not installed, Fail2ban will use auto.
+requires pyinotify (a file alteration monitor) to be installed. The backend would receive modification events from a built-in Linux kernel \fIinotify\fR feature used to watch for changes on tracking files and directories, and therefore is better suitable for monitoring of logfiles than \fIpolling\fR.
 .TP
 .B polling
-uses a polling algorithm which does not require external libraries.
+uses a polling algorithm which does not require additional libraries.
 .TP
 .B systemd
-uses systemd python library to access the systemd journal. Specifying \fBlogpath\fR is not valid for this backend and instead utilises \fBjournalmatch\fR from the jails associated filter config. Multiple systemd-specific flags can be passed to the backend, including \fBjournalpath\fR and \fBjournalfiles\fR, to explicitly set the path to a directory or set of files. \fBjournalflags\fR, which by default is 4 and excludes user session files, can be set to include them with \fBjournalflags=1\fR, see the python-systemd documentation for other settings and further details. Examples:
+uses systemd python library to access the systemd journal. Specifying \fBlogpath\fR is not valid for this backend and instead utilises \fBjournalmatch\fR from the jails associated filter config. Multiple systemd-specific flags can be passed to the backend, including \fBjournalpath\fR and \fBjournalfiles\fR, to explicitly set the path to a directory or set of files, \fBjournalflags\fR, which by default is 1 (LOCAL_ONLY) and opens journal on local machine only, can be set to 4 (SYSTEM_ONLY) with \fBjournalflags=4\fR to exclude user session files, or \fBnamespace\fR.
+.br
+Note that \fBjournalflags\fR, \fBjournalpath\fR, \fBjournalfiles\fR and \fBnamespace\fR are exclusive. See the python-systemd documentation for other settings and further details.
+.sp 1
+Examples:
 .PP
 .RS
 .nf
-backend = systemd[journalpath=/run/log/journal/machine-1]
-backend = systemd[journalfiles="/path/to/system.journal, /path/to/user.journal"]
-backend = systemd[journalflags=1]
+        backend = systemd[journalpath=/run/log/journal/machine-1]
+        backend = systemd[journalfiles="/path/to/system.journal, /path/to/user.journal"]
+        backend = systemd[journalflags=4, rotated=on]
 .fi
+.sp 1
+To avoid "too many open files" situation (descriptors exhaustion), fail2ban will ignore rotated journal files by default and has own specific parameter \fBrotated\fR (default \fBfalse\fR), so it'd automatically retrieve non-rotated set of \fBjournalfiles\fR corresponding \fBjournalflags\fR (and \fBjournalpath\fR if set). 
+Thus \fBsystemd\fR backend works by default similar to file-based backends and can find only actual (not rotated) messages and could seek (findtime etc) maximally to the time point of last rotation only.
+.br
+The same is valid for \fBfail2ban-regex systemd-journal ...\fR, so it will ignore messages from rotated journal files by default. To search across whole journal one shall use \fBfail2ban-regex systemd-journal[rotated=on] ...\fR.
+.RE
+
 
 .SS Actions
 Each jail can be configured with only a single filter, but may have multiple actions. By default, the name of a action is the action filename, and in the case of Python actions, the ".py" file extension is stripped. Where multiple of the same action are to be used, the \fBactname\fR option can be assigned to the action to avoid duplication e.g.:
@@ -493,11 +523,11 @@ is the regex (\fBreg\fRular \fBex\fRpression) that will match failed attempts. T
 \fI<F-ID>...</F-ID>\fR - free regex capturing group targeting identifier used for ban (instead of IP address or hostname).
 .IP
 \fI<F-*>...</F-*>\fR - free regex capturing named group stored in ticket, which can be used in action.
-.nf
+.br
 For example \fI<F-USER>[^@]+</F-USER>\fR matches and stores a user name, that can be used in action with interpolation tag \fI<F-USER>\fR.
 .IP
 \fI<F-ALT_*n>...</F-ALT_*n>\fR - free regex capturing alternative named group stored in ticket.
-.nf
+.br
 For example first found matched value defined in regex as \fI<F-ALT_USER>\fR, \fI<F-ALT_USER1>\fR or \fI<F-ALT_USER2>\fR would be stored as <F-USER> (if direct match is not found or empty).
 .PP
 Every of abovementioned tags can be specified in \fBprefregex\fR and in \fBfailregex\fR, thereby if specified in both, the value matched in \fBfailregex\fR overwrites a value matched in \fBprefregex\fR.
@@ -518,10 +548,10 @@ This is an obsolete handling and if the lines contain some common identifier, be
 is the regex to identify log entries that should be ignored by Fail2Ban, even if they match failregex.
 
 .TP
-\fBmaxlines\fR
+.B maxlines
 specifies the maximum number of lines to buffer to match multi-line regexs. For some log formats this will not required to be changed. Other logs may require to increase this value if a particular log file is frequently written to.
 .TP
-\fBdatepattern\fR
+.B datepattern
 specifies a custom date pattern/regex as an alternative to the default date detectors e.g. %%Y-%%m-%%d %%H:%%M(?::%%S)?.
 For a list of valid format directives, see Python library documentation for strptime behaviour.
 .br
@@ -544,8 +574,10 @@ There are several prefixes and words with special meaning that could be specifie
 \fI{NONE}\fR - value would allow one to find failures totally without date-time in log message. Filter will use now as a timestamp (or last known timestamp from previous line with timestamp).
 .RE
 .TP
-\fBjournalmatch\fR
-specifies the systemd journal match used to filter the journal entries. See \fBjournalctl(1)\fR and \fBsystemd.journal-fields(7)\fR for matches syntax and more details on special journal fields. This option is only valid for the \fIsystemd\fR backend.
+.B journalmatch
+specifies the systemd journal match used to filter the journal entries. See \fBjournalctl(1)\fR and \fBsystemd.journal-fields(7)\fR for matches syntax and more details on special journal fields. This option is only applied by the \fIsystemd\fR and \fIauto\fR backends and it is mandatory for automatical switch to \fIsystemd\fR by \fIauto\fR backend.
+
+.RE
 .PP
 Similar to actions, filters may have an [Init] section also (optional since v.0.10). All parameters of both sections [Definition] and [Init] can be overridden (redefined or extended) in \fIjail.conf\fR or \fIjail.local\fR (or in related \fIfilter.d/filter-name.local\fR).
 Every option supplied in the jail to the filter overwrites the value specified in [Init] section, which in turn would overwrite the value in [Definition] section.