Merge branch 'systemd-review'

Large set of fixes and enhancements for `systemd` and `auto` backends:
* fixes `systemd` bug with missing journal descriptor after rotation by reopening of journal if it is recognized as not alive (gh-3929)
* improve threaded clean-up of all filters, new thread functions `afterStop` (to force clean-up after stop) and `done`, invoking `afterStop` once
* ensure journal-reader is always closed (additional prevention against leaks and "too many open files"), thereby avoid sporadic segfault in systemd module (see https://github.com/systemd/python-systemd/issues/143)
* fixes `systemd` causing "too many open files" error for a lot of journal files and large amout of systemd jails (see new parameter `rotated` below, gh-3391);
* backend `systemd` extended with new parameter `rotated` (default `false`, as prevention against "too many open files"),
  that allows to monitor only actual journals and ignore now a lot of rotated files by default; so can drastically reduce
  amount of used file descriptors, normally to 1 or 2 descriptors per jail (gh-3391)
* implements automatic switch `backend = auto` to backend `systemd`, when the following is true (RFE gh-3768):
  - no files matching `logpath` found for this jail;
  - no `systemd_if_nologs = false` is specified for the jail (`true` by default);
  - option `journalmatch` is set for the jail or its filter (otherwise it'd be too heavy to allow all auto-jails,
    even if they have never been foreseen for journal monitoring);
  (option `skip_if_nologs` will be ignored if we could switch backend to `systemd`)

ChangeLog

@@ -11,6 +11,12 @@ ver. 1.1.1-dev-1 (20??/??/??) - development nightly edition
 -----------
 
 ### Fixes
+* fixes `systemd` bug with missing journal descriptor after rotation by reopening of journal if it is recognized as not alive (gh-3929)
+* improve threaded clean-up of all filters, new thread functions `afterStop` (to force clean-up after stop) and `done`, invoking `afterStop` once
+* ensure journal-reader is always closed (additional prevention against leaks and "too many open files"), thereby avoid sporadic segfault
+  in systemd module (see https://github.com/systemd/python-systemd/issues/143)
+* fixes `systemd` causing "too many open files" error for a lot of journal files and large amout of systemd jails 
+  (see new parameter `rotated` below, gh-3391);
 * `jail.conf`:
   - default banactions need to be specified in `paths-*.conf` (maintainer level) now
   - since stock fail2ban includes `paths-debian.conf` by default, banactions are `nftables`
@@ -43,8 +49,17 @@ ver. 1.1.1-dev-1 (20??/??/??) - development nightly edition
 * `filter.d/vsftpd.conf` - fixed regex (if failures generated by systemd-journal, gh-3954)
 
 ### New Features and Enhancements
+* backend `systemd` extended with new parameter `rotated` (default `false`, as prevention against "too many open files"),
+  that allows to monitor only actual journals and ignore now a lot of rotated files by default; so can drastically reduce
+  amount of used file descriptors, normally to 1 or 2 descriptors per jail (gh-3391)
 * new jail option `skip_if_nologs` to ignore jail if no `logpath` matches found, fail2ban continue to start with warnings/errors,
   thus other jails become running (gh-2756)
+* implements automatic switch `backend = auto` to backend `systemd`, when the following is true (RFE gh-3768):
+  - no files matching `logpath` found for this jail;
+  - no `systemd_if_nologs = false` is specified for the jail (`true` by default);
+  - option `journalmatch` is set for the jail or its filter (otherwise it'd be too heavy to allow all auto-jails,
+    even if they have never been foreseen for journal monitoring);
+  (option `skip_if_nologs` will be ignored if we could switch backend to `systemd`)
 * configuration `ignoreip` and fail2ban-client commands `addignoreip`/`delignoreip` extended with `file:...` syntax 
   to ignore IPs from file-ip-set (containing IP, subnet, dns/fqdn or raw strings); the file would be read lazy on demand,
   by first ban (and automatically reloaded by update after small latency to avoid expensive stats check on every compare);

fail2ban/client/jailreader.py

@@ -117,12 +117,13 @@ class JailReader(ConfigReader):
 		"logencoding": ["string", None],
 		"logpath": ["string", None],
 		"skip_if_nologs": ["bool", False],
+		"systemd_if_nologs": ["bool", True],
 		"action": ["string", ""]
 	}
 	_configOpts.update(FilterReader._configOpts)
 
 	_ignoreOpts = set(
-		['action', 'filter', 'enabled', 'backend', 'skip_if_nologs'] +
+		['action', 'filter', 'enabled', 'backend', 'skip_if_nologs', 'systemd_if_nologs'] +
 		list(FilterReader._configOpts.keys())
 	)
 
@@ -239,7 +240,7 @@ class JailReader(ConfigReader):
 			return self.__opts
 		return _merge_dicts(self.__opts, self.__filter.getCombined())
 
-	def convert(self, allow_no_files=False):
+	def convert(self, allow_no_files=False, systemd_if_nologs=True):
 		"""Convert read before __opts to the commands stream
 
 		Parameters
@@ -277,14 +278,25 @@ class JailReader(ConfigReader):
 						stream2.append(
 							["set", self.__name, "addlogpath", p, tail])
 				if not found_files:
-					msg = "Have not found any log file for %s jail" % self.__name
+					msg = "Have not found any log file for '%s' jail." % self.__name
 					skip_if_nologs = self.__opts.get('skip_if_nologs', False)
-					if not allow_no_files and not skip_if_nologs:
+					# if auto and we can switch to systemd backend (only possible if jail have journalmatch):
+					if backend.startswith("auto") and systemd_if_nologs and (
+					  self.__opts.get('systemd_if_nologs', True) and
+					  self.__opts.get('journalmatch', None) is not None
+					):
+						# switch backend to systemd:
+						backend = 'systemd'
+						msg += " Jail will monitor systemd journal."
+						skip_if_nologs = False
+					elif not allow_no_files and not skip_if_nologs:
 						raise ValueError(msg)
 					logSys.warning(msg)
 					if skip_if_nologs:
-						self.__opts['config-error'] = msg
-						stream = [['config-error', "Jail '%s' skipped, because of missing log files." % (self.__name,)]]
+						self.__opts['runtime-error'] = msg
+						msg = "Jail '%s' skipped, because of missing log files." % (self.__name,)
+						logSys.warning(msg)
+						stream = [['config-error', msg]]
 						return stream
 			elif opt == "ignoreip":
 				stream.append(["set", self.__name, "addignoreip"] + splitwords(value))

fail2ban/client/jailsreader.py

@@ -88,7 +88,7 @@ class JailsReader(ConfigReader):
 				parse_status |= 2
 		return ((ignoreWrong and parse_status & 1) or not (parse_status & 2))
 
-	def convert(self, allow_no_files=False):
+	def convert(self, allow_no_files=False, systemd_if_nologs=True):
 		"""Convert read before __opts and jails to the commands stream
 
 		Parameters
@@ -101,11 +101,14 @@ class JailsReader(ConfigReader):
 		stream = list()
 		# Convert jails
 		for jail in self.__jails:
-			stream.extend(jail.convert(allow_no_files=allow_no_files))
+			stream.extend(jail.convert(allow_no_files, systemd_if_nologs))
 		# Start jails
 		for jail in self.__jails:
-			if not jail.options.get('config-error'):
+			if not jail.options.get('config-error') and not jail.options.get('runtime-error'):
 				stream.append(["start", jail.getName()])
+			else:
+				# just delete rtm-errors (to check next time if cached)
+				jail.options.pop('runtime-error', None)
 
 		return stream
 

fail2ban/server/filter.py

@@ -1288,24 +1288,15 @@ class FileFilter(Filter):
 				break
 			db.updateLog(self.jail, log)
 			
-	def onStop(self):
+	def afterStop(self):
 		"""Stop monitoring of log-file(s). Invoked after run method.
 		"""
-		# ensure positions of pending logs are up-to-date:
-		if self._pendDBUpdates and self.jail.database:
-			self._updateDBPending()
 		# stop files monitoring:
 		for path in list(self.__logs.keys()):
 			self.delLogPath(path)
-
-	def stop(self):
-		"""Stop filter
-		"""
-		# normally onStop will be called automatically in thread after its run ends, 
-		# but for backwards compatibilities we'll invoke it in caller of stop method.
-		self.onStop()
-		# stop thread:
-		super(Filter, self).stop()
+		# ensure positions of pending logs are up-to-date:
+		if self._pendDBUpdates and self.jail.database:
+			self._updateDBPending()
 
 ##
 # FileContainer class.

fail2ban/server/filterpyinotify.py

@@ -367,19 +367,18 @@ class FilterPyinotify(FileFilter):
 				self.commonError("unhandled", e)
 			
 		logSys.debug("[%s] filter exited (pyinotifier)", self.jailName)
-		self.__notifier = None
+		self.done()
 
 		return True
 
 	##
-	# Call super.stop() and then stop the 'Notifier'
+	# Clean-up: then stop the 'Notifier'
 
-	def stop(self):
-		# stop filter thread:
-		super(FilterPyinotify, self).stop()
+	def afterStop(self):
 		try:
 			if self.__notifier: # stop the notifier
 				self.__notifier.stop()
+				self.__notifier = None
 		except AttributeError: # pragma: no cover
 			if self.__notifier: raise
 

fail2ban/server/filtersystemd.py

@@ -25,18 +25,61 @@ __license__ = "GPL"
 import os
 import time
 
+from glob import glob
 from systemd import journal
 
 from .failmanager import FailManagerEmpty
 from .filter import JournalFilter, Filter
 from .mytime import MyTime
 from .utils import Utils
-from ..helpers import getLogger, logging, splitwords, uni_decode
+from ..helpers import getLogger, logging, splitwords, uni_decode, _as_bool
 
 # Gets the instance of the logger.
 logSys = getLogger(__name__)
 
 
+_systemdPathCache = Utils.Cache()
+def _getSystemdPath(path):
+	"""Get systemd path using systemd-path command (cached)"""
+	p = _systemdPathCache.get(path)
+	if p: return p
+	p = Utils.executeCmd('systemd-path %s' % path, timeout=10, shell=True, output=True)
+	if p and p[0]:
+		p = str(p[1].decode('utf-8')).split('\n')[0]
+		_systemdPathCache.set(path, p)
+		return p
+	p = '/var/log' if path == 'system-state-logs' else ('/run/log' if path == 'system-runtime-logs' else None)
+	_systemdPathCache.set(path, p)
+	return p
+
+def _globJournalFiles(flags=None, path=None):
+	"""Get journal files without rotated files."""
+	filesSet = set()
+	_join = os.path.join
+	def _addJF(filesSet, p, flags):
+		"""add journal files to set corresponding path and flags (without rotated *@*.journal)"""
+		# system journal:
+		if (flags is None) or (flags & journal.SYSTEM_ONLY):
+			filesSet |= set(glob(_join(p,'system.journal'))) - set(glob(_join(p,'system*@*.journal')))
+		# current user-journal:
+		if (flags is not None) and (flags & journal.CURRENT_USER):
+			uid = os.getuid()
+			filesSet |= set(glob(_join(p,('user-%s.journal' % uid)))) - set(glob(_join(p,('user-%s@*.journal' % uid))))
+		# all local journals:
+		if (flags is None) or not (flags & (journal.SYSTEM_ONLY|journal.CURRENT_USER)):
+			filesSet |= set(glob(_join(p,'*.journal'))) - set(glob(_join(p,'*@*.journal')))
+	if path:
+		# journals relative given path only:
+		_addJF(filesSet, path, flags)
+	else:
+		# persistent journals corresponding flags:
+		if (flags is None) or not (flags & journal.RUNTIME_ONLY):
+			_addJF(filesSet, _join(_getSystemdPath('system-state-logs'), 'journal/*'), flags)
+		# runtime journals corresponding flags:
+		_addJF(filesSet, _join(_getSystemdPath('system-runtime-logs'), 'journal/*'), flags)
+	return filesSet
+
+
 ##
 # Journal reader class.
 #
@@ -52,12 +95,13 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 	# @param jail the jail object
 
 	def __init__(self, jail, **kwargs):
-		jrnlargs = FilterSystemd._getJournalArgs(kwargs)
+		self.__jrnlargs = FilterSystemd._getJournalArgs(kwargs)
 		JournalFilter.__init__(self, jail, **kwargs)
 		self.__modified = 0
 		# Initialise systemd-journal connection
-		self.__journal = journal.Reader(**jrnlargs)
+		self.__journal = journal.Reader(**self.__jrnlargs)
 		self.__matches = []
+		self.__bypassInvalidateMsg = 0
 		self.setDatePattern(None)
 		logSys.debug("Created FilterSystemd")
 
@@ -74,31 +118,86 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 		except KeyError:
 			pass
 		else:
-			import glob
 			p = args['files']
 			if not isinstance(p, (list, set, tuple)):
 				p = splitwords(p)
 			files = []
 			for p in p:
-				files.extend(glob.glob(p))
+				files.extend(glob(p))
 			args['files'] = list(set(files))
 
-		# Default flags is SYSTEM_ONLY(4). This would lead to ignore user session files,
-		# so can prevent "Too many open files" errors on a lot of user sessions (see gh-2392):
+		rotated = _as_bool(kwargs.pop('rotated', 0))
+		# Default flags is SYSTEM_ONLY(4) or LOCAL_ONLY(1), depending on rotated parameter. 
+		# This could lead to ignore user session files, so together with ignoring rotated
+		# files would prevent "Too many open files" errors on a lot of user sessions (see gh-2392):
 		try:
 			args['flags'] = int(kwargs.pop('journalflags'))
 		except KeyError:
 			# be sure all journal types will be opened if files/path specified (don't set flags):
-			if ('files' not in args or not len(args['files'])) and ('path' not in args or not args['path']):
-				args['flags'] = int(os.getenv("F2B_SYSTEMD_DEFAULT_FLAGS", 4))
-				
+			if (not args.get('files') and not args.get('path')):
+				args['flags'] = os.getenv("F2B_SYSTEMD_DEFAULT_FLAGS", None)
+				if args['flags'] is not None:
+					args['flags'] = int(args['flags'])
+				elif rotated:
+					args['flags'] = journal.SYSTEM_ONLY
+
 		try:
 			args['namespace'] = kwargs.pop('namespace')
 		except KeyError:
 			pass
 
+		# To avoid monitoring rotated logs, as prevention against "Too many open files",
+		# set the files to system.journal and user-*.journal (without rotated *@*.journal):
+		if not rotated and not args.get('files') and not args.get('namespace'):
+			args['files'] = _globJournalFiles(
+				args.get('flags', journal.LOCAL_ONLY), args.get('path'))
+			if args['files']:
+				args['files'] = list(args['files'])
+				args['path'] = None; # cannot be cannot be specified simultaneously with files
+			else:
+				args['files'] = None
+
 		return args
 
+	@property
+	def _journalAlive(self):
+		"""Checks journal is online.
+		"""
+		try:
+			# open?
+			if self.__journal.closed: # pragma: no cover
+				return False
+			# has cursor? if it is broken (e. g. no descriptor) - it'd raise this:
+			# OSError: [Errno 99] Cannot assign requested address
+			if self.__journal._get_cursor():
+				return True
+		except OSError: # pragma: no cover
+			pass
+		return False
+
+	def _reopenJournal(self): # pragma: no cover
+		"""Reopen journal (if it becomes offline after rotation)
+		"""
+		if self.__journal.closed:
+			# recreate reader:
+			self.__journal = journal.Reader(**self.__jrnlargs)
+		else:
+			try:
+				# workaround for gh-3929 (no journal descriptor after rotation),
+				# to reopen journal we'd simply invoke inherited init again:
+				self.__journal.close()
+				ja = self.__jrnlargs
+				super(journal.Reader, self.__journal).__init__(
+					ja.get('flags', 0), ja.get('path'), ja.get('files'), ja.get('namespace'))
+			except:
+				# cannot reopen in that way, so simply recreate reader:
+				self.closeJournal()
+				self.__journal = journal.Reader(**self.__jrnlargs)
+		# restore journalmatch specified for the jail:
+		self.resetJournalMatches()
+		# just to avoid "Invalidate signaled" happening again after reopen:
+		self.__bypassInvalidateMsg = MyTime.time() + 1
+
 	##
 	# Add a journal match filters from list structure
 	#
@@ -257,6 +356,8 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 	def inOperationMode(self):
 		self.inOperation = True
 		logSys.info("[%s] Jail is in operation now (process new journal entries)", self.jailName)
+		# just to avoid "Invalidate signaled" happening often at start:
+		self.__bypassInvalidateMsg = MyTime.time() + 1
 
 	##
 	# Main loop.
@@ -314,6 +415,14 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 		while self.active:
 			# wait for records (or for timeout in sleeptime seconds):
 			try:
+				if self.idle:
+					# because journal.wait will returns immediately if we have records in journal,
+					# just wait a little bit here for not idle, to prevent hi-load:
+					if not Utils.wait_for(lambda: not self.active or not self.idle, 
+						self.sleeptime * 10, self.sleeptime
+					):
+						self.ticks += 1
+						continue
 				## wait for entries using journal.wait:
 				if wcode == journal.NOP and self.inOperation:
 					## todo: find better method as wait_for to break (e.g. notify) journal.wait(self.sleeptime),
@@ -328,8 +437,10 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 					## if invalidate (due to rotation, vacuuming or journal files added/removed etc):
 					if self.active and wcode == journal.INVALIDATE:
 						if self.ticks:
-							logSys.log(logging.DEBUG, "[%s] Invalidate signaled, take a little break (rotation ends)", self.jailName)
+							if not self.__bypassInvalidateMsg or MyTime.time() > self.__bypassInvalidateMsg:
+								logSys.log(logging.MSG, "[%s] Invalidate signaled, take a little break (rotation ends)", self.jailName)
 							time.sleep(self.sleeptime * 0.25)
+							self.__bypassInvalidateMsg = 0
 						Utils.wait_for(lambda: not self.active or \
 							self.__journal.wait(Utils.DEFAULT_SLEEP_INTERVAL) != journal.INVALIDATE,
 							self.sleeptime * 3, 0.00001)
@@ -340,14 +451,11 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 								if self.__journal.get_previous(): self.__journal.get_next()
 							except OSError:
 								pass
-				if self.idle:
-					# because journal.wait will returns immediately if we have records in journal,
-					# just wait a little bit here for not idle, to prevent hi-load:
-					if not Utils.wait_for(lambda: not self.active or not self.idle, 
-						self.sleeptime * 10, self.sleeptime
-					):
-						self.ticks += 1
-						continue
+						# if it is not alive - reopen:
+						if not self._journalAlive:
+							logSys.log(logging.MSG, "[%s] Journal reader seems to be offline, reopen journal", self.jailName)
+							self._reopenJournal()
+							wcode = journal.NOP
 				self.__modified = 0
 				while self.active:
 					logentry = None
@@ -408,8 +516,8 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 
 		logSys.debug("[%s] filter terminated", self.jailName)
 
-		# close journal:
-		self.closeJournal()
+		# call afterStop once (close journal, etc):
+		self.done()
 
 		logSys.debug("[%s] filter exited (systemd)", self.jailName)
 		return True
@@ -443,12 +551,10 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 				break
 			db.updateJournal(self.jail, log, *args)
 
-	def onStop(self):
-		"""Stop monitoring of journal. Invoked after run method.
-		"""
+	def afterStop(self):
+		"""Cleanup"""
 		# close journal:
 		self.closeJournal()
 		# ensure positions of pending logs are up-to-date:
 		if self._pendDBUpdates and self.jail.database:
 			self._updateDBPending()
-

fail2ban/server/jail.py

@@ -335,7 +335,9 @@ class Jail(object):
 			try:
 				## signal to stop filter / actions:
 				if stop:
-					obj.stop()
+					if obj.isAlive():
+						obj.stop()
+					obj.done(); # and clean-up everything
 				## wait for end of threads:
 				if join:
 					obj.join()

fail2ban/server/jailthread.py

@@ -103,7 +103,21 @@ class JailThread(Thread):
 	def stop(self):
 		"""Sets `active` property to False, to flag run method to return.
 		"""
-		self.active = False
+		if self.active: self.active = False
+		# normally onStop will be called automatically in thread after its run ends,
+		# but for backwards compatibilities we'll invoke it in caller of stop method.
+		self.onStop()
+		self.onStop = lambda:()
+		self.done()
+
+	def done(self):
+		self.done = lambda:()
+		# if still runniung - wait a bit before initiate clean-up:
+		if self.is_alive():
+			Utils.wait_for(lambda: not self.is_alive(), 5)
+		# now clean-up everything:
+		self.afterStop()
+
 
 	@abstractmethod
 	def run(self): # pragma: no cover - absract
@@ -111,11 +125,15 @@ class JailThread(Thread):
 		"""
 		pass
 
+	def afterStop(self):
+		"""Cleanup resources."""
+		pass
+
 	def join(self):
 		""" Safer join, that could be called also for not started (or ended) threads (used for cleanup).
 		"""
 		## if cleanup needed - create derivative and call it before join...
-
+		self.done()
 		## if was really started - should call join:
 		if self.active is not None:
 			super(JailThread, self).join()

fail2ban/tests/clientreadertestcase.py

@@ -719,51 +719,67 @@ class JailsReaderTest(LogCaptureTestCase):
 		jails = JailsReader(basedir=IMPERFECT_CONFIG, share_config=IMPERFECT_CONFIG_SHARE_CFG)
 		self.assertTrue(jails.read())
 		self.assertFalse(jails.getOptions(ignoreWrong=False))
-		self.assertRaises(ValueError, jails.convert)
-		comm_commands = jails.convert(allow_no_files=True)
-		self.maxDiff = None
-		self.assertSortedEqual(comm_commands,
-			[['add', 'emptyaction', 'auto'],
-			 ['add', 'test-known-interp', 'auto'],
-			 ['multi-set', 'test-known-interp', 'addfailregex', [
-			   'failure test 1 (filter.d/test.conf) <HOST>',
-			   'failure test 2 (filter.d/test.local) <HOST>',
-			   'failure test 3 (jail.local) <HOST>'
-			 ]],
-			 ['start', 'test-known-interp'],
-			 ['add', 'missinglogfiles', 'auto'],
-			 ['set', 'missinglogfiles', 'addfailregex', '<IP>'],
-			 ['config-error', "Jail 'missinglogfiles_skip' skipped, because of missing log files."],
-			 ['add', 'brokenaction', 'auto'],
-			 ['set', 'brokenaction', 'addfailregex', '<IP>'],
-			 ['set', 'brokenaction', 'addaction', 'brokenaction'],
-			 ['multi-set', 'brokenaction', 'action', 'brokenaction', [
-				 ['actionban', 'hit with big stick <ip>'],
-				 ['actname', 'brokenaction'],
-				 ['name', 'brokenaction']
-			 ]],
-			 ['add', 'parse_to_end_of_jail.conf', 'auto'],
-			 ['set', 'parse_to_end_of_jail.conf', 'addfailregex', '<IP>'],
-			 ['set', 'tz_correct', 'addfailregex', '<IP>'],
-			 ['set', 'tz_correct', 'logtimezone', 'UTC+0200'],
-			 ['start', 'emptyaction'],
-			 ['start', 'missinglogfiles'],
-			 ['start', 'brokenaction'],
-			 ['start', 'parse_to_end_of_jail.conf'],
-		         ['add', 'tz_correct', 'auto'],
-			 ['start', 'tz_correct'],
-			 ['config-error',
-				"Jail 'brokenactiondef' skipped, because of wrong configuration: Invalid action definition 'joho[foo': unexpected option syntax"],
-			 ['config-error',
-				"Jail 'brokenfilterdef' skipped, because of wrong configuration: Invalid filter definition 'flt[test': unexpected option syntax"],
-			 ['config-error',
-				"Jail 'missingaction' skipped, because of wrong configuration: Unable to read action 'noactionfileforthisaction'"],
-			 ['config-error',
-				"Jail 'missingbitsjail' skipped, because of wrong configuration: Unable to read the filter 'catchallthebadies'"],
-			 ])
+		self.assertRaises(ValueError, lambda: jails.convert(systemd_if_nologs=False))
 		self.assertLogged("Errors in jail 'missingbitsjail'.")
 		self.assertNotLogged("Skipping...")
+		# check with allow no files (just to cover other jail problems), but without switch to systemd:
+		self.pruneLog('[test-phase] allow no files, no switch to systemd ...')
+		comm_commands = jails.convert(allow_no_files=True, systemd_if_nologs=False)
+		self.maxDiff = None
+		def _checkStream(comm_commands, backend='auto'):
+			self.assertSortedEqual(comm_commands,
+				[['add', 'emptyaction', 'auto'],
+				 ['add', 'test-known-interp', 'auto'],
+				 ['multi-set', 'test-known-interp', 'addfailregex', [
+				   'failure test 1 (filter.d/test.conf) <HOST>',
+				   'failure test 2 (filter.d/test.local) <HOST>',
+				   'failure test 3 (jail.local) <HOST>'
+				 ]],
+				 ['start', 'test-known-interp'],
+				 ['add', 'missinglogfiles', backend], # can switch backend because have journalmatch
+				 ['set', 'missinglogfiles', 'addfailregex', '<IP>'],
+				 ['set', 'missinglogfiles', 'addjournalmatch', '_COMM=test'],
+				 ['config-error', "Jail 'missinglogfiles_skip' skipped, because of missing log files."],
+				 ['add', 'brokenaction', 'auto'],
+				 ['set', 'brokenaction', 'addfailregex', '<IP>'],
+				 ['set', 'brokenaction', 'addaction', 'brokenaction'],
+				 ['multi-set', 'brokenaction', 'action', 'brokenaction', [
+					 ['actionban', 'hit with big stick <ip>'],
+					 ['actname', 'brokenaction'],
+					 ['name', 'brokenaction']
+				 ]],
+				 ['add', 'parse_to_end_of_jail.conf', 'auto'],
+				 ['set', 'parse_to_end_of_jail.conf', 'addfailregex', '<IP>'],
+				 ['set', 'tz_correct', 'addfailregex', '<IP>'],
+				 ['set', 'tz_correct', 'logtimezone', 'UTC+0200'],
+				 ['start', 'emptyaction'],
+				 ['start', 'missinglogfiles'],
+				 ['start', 'brokenaction'],
+				 ['start', 'parse_to_end_of_jail.conf'],
+			   ['add', 'tz_correct', 'auto'],
+				 ['start', 'tz_correct'],
+				 ['config-error',
+					"Jail 'brokenactiondef' skipped, because of wrong configuration: Invalid action definition 'joho[foo': unexpected option syntax"],
+				 ['config-error',
+					"Jail 'brokenfilterdef' skipped, because of wrong configuration: Invalid filter definition 'flt[test': unexpected option syntax"],
+				 ['config-error',
+					"Jail 'missingaction' skipped, because of wrong configuration: Unable to read action 'noactionfileforthisaction'"],
+				 ['config-error',
+					"Jail 'missingbitsjail' skipped, because of wrong configuration: Unable to read the filter 'catchallthebadies'"],
+				 ])
+		_checkStream(comm_commands, backend='auto')
+		self.assertNotLogged("Have not found any log file for 'missinglogfiles' jail. Jail will monitor systemd journal.")
 		self.assertLogged("No file(s) found for glob /weapons/of/mass/destruction")
+		self.assertLogged("Jail 'missinglogfiles_skip' skipped, because of missing log files.")
+		# switch backend auto to systemd if no files found, note that jail "missinglogfiles_skip" will be skipped yet,
+		# because for this jail configured skip_if_nologs = true, all other jails shall switch to systemd with warning
+		self.pruneLog('[test-phase] auto -> systemd')
+		comm_commands = jails.convert(allow_no_files=True)
+		_checkStream(comm_commands, backend='systemd')
+		self.assertNotLogged("Errors in jail 'missingbitsjail'.")
+		self.assertLogged("Have not found any log file for 'missinglogfiles' jail. Jail will monitor systemd journal.")
+		self.assertLogged("No file(s) found for glob /weapons/of/mass/destruction")
+		self.assertLogged("Jail 'missinglogfiles_skip' skipped, because of missing log files.")
 
 	def testReadStockActionConf(self):
 		unittest.F2B.SkipIfCfgMissing(stock=True)

fail2ban/tests/config/jail.conf

@@ -21,6 +21,7 @@ failregex = %(known/failregex)s
 
 [missinglogfiles]
 enabled = true
+journalmatch = _COMM=test ;# allow to switch to systemd (by backend = `auto` and no logs found)
 logpath = /weapons/of/mass/destruction
 
 [missinglogfiles_skip]

fail2ban/tests/filtertestcase.py

@@ -32,7 +32,7 @@ import tempfile
 import uuid
 
 try:
-	from systemd import journal
+	from ..server.filtersystemd import journal, _globJournalFiles
 except ImportError:
 	journal = None
 
@@ -1477,7 +1477,7 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover
 			self.filter.addFailRegex(r"(?:(?:Authentication failure|Failed [-/\w+]+) for(?: [iI](?:llegal|nvalid) user)?|[Ii](?:llegal|nvalid) user|ROOT LOGIN REFUSED) .*(?: from|FROM) <HOST>")
 
 		def tearDown(self):
-			if self.filter and self.filter.active:
+			if self.filter and (self.filter.active or self.filter.active is None):
 				self.filter.stop()
 				self.filter.join()		  # wait for the thread to terminate
 			super(MonitorJournalFailures, self).tearDown()
@@ -1510,6 +1510,34 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover
 				return MonitorJournalFailures._runtimeJournal
 			raise unittest.SkipTest('systemd journal seems to be not available (e. g. no rights to read)')
 		
+		def testGlobJournal_System(self):
+			if not journal: # pragma: no cover
+				raise unittest.SkipTest("systemd python interface not available")
+			jrnlfile = self._getRuntimeJournal()
+			jrnlpath = os.path.dirname(jrnlfile)
+			self.assertIn(jrnlfile, _globJournalFiles(journal.SYSTEM_ONLY))
+			self.assertIn(jrnlfile, _globJournalFiles(journal.SYSTEM_ONLY, jrnlpath))
+			self.assertIn(jrnlfile, _globJournalFiles(journal.LOCAL_ONLY))
+			self.assertIn(jrnlfile, _globJournalFiles(journal.LOCAL_ONLY, jrnlpath))
+
+		@with_tmpdir
+		def testGlobJournal(self, tmp):
+			if not journal: # pragma: no cover
+				raise unittest.SkipTest("systemd python interface not available")
+			# no files yet in temp-path:
+			self.assertFalse(_globJournalFiles(None, tmp))
+			# test against temp-path, shall ignore all rotated files:
+			tsysjrnl = os.path.join(tmp, 'system.journal')
+			tusrjrnl = os.path.join(tmp, 'user-%s.journal' % os.getuid())
+			def touch(fn): os.close(os.open(fn, os.O_CREAT|os.O_APPEND))
+			touch(tsysjrnl);
+			touch(tusrjrnl);
+			touch(os.path.join(tmp, 'system@test-rotated.journal'));
+			touch(os.path.join(tmp, 'user-%s@test-rotated.journal' % os.getuid()));
+			self.assertSortedEqual(_globJournalFiles(None, tmp), {tsysjrnl, tusrjrnl})
+			self.assertSortedEqual(_globJournalFiles(journal.SYSTEM_ONLY, tmp), {tsysjrnl})
+			self.assertSortedEqual(_globJournalFiles(journal.CURRENT_USER, tmp), {tusrjrnl})
+
 		def testJournalFilesArg(self):
 			# retrieve current system journal path
 			jrnlfile = self._getRuntimeJournal()
@@ -1533,9 +1561,16 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover
 			self.assertTrue(self.isEmpty(1))
 			self.assertEqual(len(self.jail), 0)
 			self.assertRaises(FailManagerEmpty, self.filter.failManager.toBan)
+		def testJournalPath_RotatedArg(self):
+			# retrieve current system journal path
+			jrnlpath = self._getRuntimeJournal()
+			jrnlpath = os.path.dirname(jrnlpath)
+			self._initFilter(journalpath=jrnlpath, rotated=1)
 
 		def testJournalFlagsArg(self):
-			self._initFilter(journalflags=0) # e. g. 2 - journal.RUNTIME_ONLY
+			self._initFilter(journalflags=0)
+		def testJournalFlags_RotatedArg(self):
+			self._initFilter(journalflags=0, rotated=1)
 
 		def assert_correct_ban(self, test_ip, test_attempts):
 			self.assertTrue(self.waitFailTotal(test_attempts, 10)) # give Filter a chance to react

fail2ban/tests/misctestcase.py

@@ -350,7 +350,7 @@ class TestsUtilsTest(LogCaptureTestCase):
 		unittest.F2B.fast = True
 		try:
 			self.assertEqual(unittest.F2B.maxWaitTime(lambda: 50)(), 50)
-			self.assertEqual(unittest.F2B.maxWaitTime(25), 2.5)
+			self.assertEqual(unittest.F2B.maxWaitTime(25), 5)
 			self.assertEqual(unittest.F2B.maxWaitTime(25.), 25.0)
 		finally:
 			unittest.F2B.fast = orgfast

fail2ban/tests/utils.py

@@ -219,7 +219,7 @@ class F2B(DefaultTestOptions):
 		# short only integer interval (avoid by conditional wait with callable, and dual 
 		# wrapping in some routines, if it will be called twice):
 		if self.fast and isinstance(wtime, int):
-			wtime = float(wtime) / 10
+			wtime = float(wtime) / (10 if wtime < 10 else 5)
 		return wtime
 
 

man/jail.conf.5

@@ -182,6 +182,8 @@ This specifies the stack size (in KiB) to be used for subsequently created threa
 
 .SH "JAIL CONFIGURATION FILE(S) (\fIjail.conf\fB)"
 The following options are applicable to any jail. They appear in a section specifying the jail name or in the \fI[DEFAULT]\fR section which defines default values to be used if not specified in the individual section.
+.sp
+It is also possible to specify or to overwrite any option of filter file directly in jail (see section FILTER FILES).
 .TP
 .B filter
 name of the filter -- filename of the filter in /etc/fail2ban/filter.d/ without the .conf/.local extension.
@@ -198,7 +200,10 @@ Optional space separated option 'tail' can be added to the end of the path to ca
 Ensure syslog or the program that generates the log file isn't configured to compress repeated log messages to "\fI*last message repeated 5 time*s\fR" otherwise it will fail to detect. This is called \fIRepeatedMsgReduction\fR in rsyslog and should be \fIOff\fR.
 .TP
 .B skip_if_nologs
-if no logpath matches found, skip the jail by start of fail2ban if \fIskip_if_nologs\fR set to true, otherwise (default: false) start of fail2ban will fail with an error "Have not found any log file".
+if no logpath matches found, skip the jail by start of fail2ban if \fIskip_if_nologs\fR set to true, otherwise (default: false) start of fail2ban will fail with an error "Have not found any log file", unless the backend is \fIauto\fR and the jail is able to swith backend to \fIsystemd\fR (see \fIauto\fR in section \fBBackends\fR below).
+.TP
+.B systemd_if_nologs
+if no logpath matches found, switch backend \fIauto\fR to \fIsystemd\fR (see \fBBackends\fR section), unless disabled with \fBsystemd_if_nologs = false\fR (default \fBtrue\fR).
 .TP
 .B logencoding
 encoding of log files used for decoding. Default value of "auto" uses current system locale.
@@ -280,7 +285,7 @@ number of failures that have to occur in the last \fBfindtime\fR seconds to ban
 .B backend
 backend to be used to detect changes in the logpath.
 .br
-It defaults to "auto" which will try "pyinotify", "systemd" before "polling". Any of these can be specified. "pyinotify" is only valid on Linux systems with the "pyinotify" Python libraries.
+It defaults to "auto" which will try "pyinotify" before "polling" and may switch to "systemd" if no files matching \fBlogpath\fR will be found (see section \fBBackends\fR below). Any of these can be specified. "pyinotify" is only valid on Linux systems with the "pyinotify" Python libraries.
 .TP
 .B usedns
 use DNS to resolve HOST names that appear in the logs. By default it is "warn" which will resolve hostnames to IPs however it will also log a warning. If you are using DNS here you could be blocking the wrong IPs due to the asymmetric nature of reverse DNS (that the application used to write the domain name to log) compared to forward DNS that fail2ban uses to resolve this back to an IP (but not necessarily the same one). Ideally you should configure your applications to log a real IP. This can be set to "yes" to prevent warnings in the log or "no" to disable DNS resolution altogether (thus ignoring entries where hostname, not an IP is logged)..
@@ -300,21 +305,46 @@ max number of matched log-lines the jail would hold in memory per ticket. By def
 .SS Backends
 Available options are listed below.
 .TP
+.B auto
+automatically selects best suitable \fBbackend\fR, starting with file-based like \fIpyinotify\fR or \fIpolling\fR to monitor the \fBlogpath\fR matching files, but can also automatically switch to backend \fIsystemd\fR, when the following is true:
+.RS
+.IP \(bu 4n
+no files matching \fBlogpath\fR found for this jail;
+.IP \(bu 4n
+no \fBsystemd_if_nologs = false\fR is specified for the jail;
+.IP \(bu 4n
+option \fBjournalmatch\fR is set for the jail or its filter (otherwise it'd be too heavy to allow all auto-jails, even if they have never been foreseen for journal monitoring);
+.TP
+.br
+Option \fBskip_if_nologs\fR will be ignored if we could switch \fBbackend\fR to \fIsystemd\fR.
+.RE
+.TP
 .B pyinotify
-requires pyinotify (a file alteration monitor) to be installed. If pyinotify is not installed, Fail2ban will use auto.
+requires pyinotify (a file alteration monitor) to be installed. The backend would receive modification events from a built-in Linux kernel \fIinotify\fR feature used to watch for changes on tracking files and directories, and therefore is better suitable for monitoring of logfiles than \fIpolling\fR.
 .TP
 .B polling
-uses a polling algorithm which does not require external libraries.
+uses a polling algorithm which does not require additional libraries.
 .TP
 .B systemd
-uses systemd python library to access the systemd journal. Specifying \fBlogpath\fR is not valid for this backend and instead utilises \fBjournalmatch\fR from the jails associated filter config. Multiple systemd-specific flags can be passed to the backend, including \fBjournalpath\fR and \fBjournalfiles\fR, to explicitly set the path to a directory or set of files. \fBjournalflags\fR, which by default is 4 and excludes user session files, can be set to include them with \fBjournalflags=1\fR, see the python-systemd documentation for other settings and further details. Examples:
+uses systemd python library to access the systemd journal. Specifying \fBlogpath\fR is not valid for this backend and instead utilises \fBjournalmatch\fR from the jails associated filter config. Multiple systemd-specific flags can be passed to the backend, including \fBjournalpath\fR and \fBjournalfiles\fR, to explicitly set the path to a directory or set of files, \fBjournalflags\fR, which by default is 1 (LOCAL_ONLY) and opens journal on local machine only, can be set to 4 (SYSTEM_ONLY) with \fBjournalflags=4\fR to exclude user session files, or \fBnamespace\fR.
+.br
+Note that \fBjournalflags\fR, \fBjournalpath\fR, \fBjournalfiles\fR and \fBnamespace\fR are exclusive. See the python-systemd documentation for other settings and further details.
+.sp 1
+Examples:
 .PP
 .RS
 .nf
-backend = systemd[journalpath=/run/log/journal/machine-1]
-backend = systemd[journalfiles="/path/to/system.journal, /path/to/user.journal"]
-backend = systemd[journalflags=1]
+        backend = systemd[journalpath=/run/log/journal/machine-1]
+        backend = systemd[journalfiles="/path/to/system.journal, /path/to/user.journal"]
+        backend = systemd[journalflags=4, rotated=on]
 .fi
+.sp 1
+To avoid "too many open files" situation (descriptors exhaustion), fail2ban will ignore rotated journal files by default and has own specific parameter \fBrotated\fR (default \fBfalse\fR), so it'd automatically retrieve non-rotated set of \fBjournalfiles\fR corresponding \fBjournalflags\fR (and \fBjournalpath\fR if set). 
+Thus \fBsystemd\fR backend works by default similar to file-based backends and can find only actual (not rotated) messages and could seek (findtime etc) maximally to the time point of last rotation only.
+.br
+The same is valid for \fBfail2ban-regex systemd-journal ...\fR, so it will ignore messages from rotated journal files by default. To search across whole journal one shall use \fBfail2ban-regex systemd-journal[rotated=on] ...\fR.
+.RE
+
 
 .SS Actions
 Each jail can be configured with only a single filter, but may have multiple actions. By default, the name of a action is the action filename, and in the case of Python actions, the ".py" file extension is stripped. Where multiple of the same action are to be used, the \fBactname\fR option can be assigned to the action to avoid duplication e.g.:
@@ -493,11 +523,11 @@ is the regex (\fBreg\fRular \fBex\fRpression) that will match failed attempts. T
 \fI<F-ID>...</F-ID>\fR - free regex capturing group targeting identifier used for ban (instead of IP address or hostname).
 .IP
 \fI<F-*>...</F-*>\fR - free regex capturing named group stored in ticket, which can be used in action.
-.nf
+.br
 For example \fI<F-USER>[^@]+</F-USER>\fR matches and stores a user name, that can be used in action with interpolation tag \fI<F-USER>\fR.
 .IP
 \fI<F-ALT_*n>...</F-ALT_*n>\fR - free regex capturing alternative named group stored in ticket.
-.nf
+.br
 For example first found matched value defined in regex as \fI<F-ALT_USER>\fR, \fI<F-ALT_USER1>\fR or \fI<F-ALT_USER2>\fR would be stored as <F-USER> (if direct match is not found or empty).
 .PP
 Every of abovementioned tags can be specified in \fBprefregex\fR and in \fBfailregex\fR, thereby if specified in both, the value matched in \fBfailregex\fR overwrites a value matched in \fBprefregex\fR.
@@ -518,10 +548,10 @@ This is an obsolete handling and if the lines contain some common identifier, be
 is the regex to identify log entries that should be ignored by Fail2Ban, even if they match failregex.
 
 .TP
-\fBmaxlines\fR
+.B maxlines
 specifies the maximum number of lines to buffer to match multi-line regexs. For some log formats this will not required to be changed. Other logs may require to increase this value if a particular log file is frequently written to.
 .TP
-\fBdatepattern\fR
+.B datepattern
 specifies a custom date pattern/regex as an alternative to the default date detectors e.g. %%Y-%%m-%%d %%H:%%M(?::%%S)?.
 For a list of valid format directives, see Python library documentation for strptime behaviour.
 .br
@@ -544,8 +574,10 @@ There are several prefixes and words with special meaning that could be specifie
 \fI{NONE}\fR - value would allow one to find failures totally without date-time in log message. Filter will use now as a timestamp (or last known timestamp from previous line with timestamp).
 .RE
 .TP
-\fBjournalmatch\fR
-specifies the systemd journal match used to filter the journal entries. See \fBjournalctl(1)\fR and \fBsystemd.journal-fields(7)\fR for matches syntax and more details on special journal fields. This option is only valid for the \fIsystemd\fR backend.
+.B journalmatch
+specifies the systemd journal match used to filter the journal entries. See \fBjournalctl(1)\fR and \fBsystemd.journal-fields(7)\fR for matches syntax and more details on special journal fields. This option is only applied by the \fIsystemd\fR and \fIauto\fR backends and it is mandatory for automatical switch to \fIsystemd\fR by \fIauto\fR backend.
+
+.RE
 .PP
 Similar to actions, filters may have an [Init] section also (optional since v.0.10). All parameters of both sections [Definition] and [Init] can be overridden (redefined or extended) in \fIjail.conf\fR or \fIjail.local\fR (or in related \fIfilter.d/filter-name.local\fR).
 Every option supplied in the jail to the filter overwrites the value specified in [Init] section, which in turn would overwrite the value in [Definition] section.