Commit Graph

4034 Commits (a1e9cc552c1f8c2afaed42954b58dc4afc12b70f)

Author SHA1 Message Date
sebres 189e70d99c processLine etc. rewritten:
- normalize calling parameters (persistent parameters moved from function arguments to filter member variables)
- save last line as lambda instead of return it as string (lazy convert of process line tuple to string on demand, needed in fail2ban-regex only)
2016-11-18 17:02:00 +01:00
sebres a2cf34a64e code review: added endpos to found tuple, just to be safe by unpack 2016-11-17 21:12:23 +01:00
sebres b5433f48b7 amend after code review of merge gh-1581 2016-11-11 11:09:46 +01:00
sebres bee6e7376b Merge branch 'aclindsa:master' 2016-11-11 10:58:40 +01:00
sebres ea4c1f6356 Merge branch 'master' into 0.10 2016-11-11 10:29:45 +01:00
sebres dab5f56609 Merge branch 'fix-gh-1477' 2016-11-11 10:17:07 +01:00
Alex 8ac28e5dcb Make changes and add test file 2016-11-10 13:09:32 +01:00
Alex 8c40766511 Add Mongodb-auth filter and jail 2016-11-10 12:48:24 +01:00
Serg G. Brester 4e252be76f Update FILTERS
closes #1591
2016-10-25 11:01:32 +02:00
sebres 58717c1854 fail2ban-testcases: persistently set (python) time zone to CET during test cases process (used in zone-related test-cases) 2016-10-17 13:26:24 +02:00
sebres c8b036456d changelog entries 2016-10-17 12:47:42 +02:00
sebres ffa9705412 fixed UTC/GMT named time zone using `%Z` and `%z` patterns (special case with 0 zone offset);
Currently still ignores another named zones, because fail2ban assumes that the given date is in the current default zone.
Closes gh-1575
2016-10-17 12:09:53 +02:00
sebres faee5f1fdc better caching (thereby better performance), better recognition of similar regex 2016-10-17 11:20:30 +02:00
sebres ae7297e16b more precise date template handling (WARNING: this commit creates possible incompatibilities):
- datedetector rewritten more strict as earlier;
  - default templates can be specified exacter using prefix/suffix syntax (via `datepattern`);
  - more as one date pattern can be specified using option `datepattern` now (new-line separated);
  - some default options like `datepattern` can be specified directly in section `[Definition]`, that avoids contrary usage of unnecessarily `[Init]` section, because of performance (each extra section costs time);
  - option `datepattern` can be specified in jail also (jails without filters);
  - if first group specified, only this will be cut out from search log-line (e. g.: `^date:[({DATE})]` will cut out only datetime match pattern, and leaves `date:[] failure ip...` for searching in filter);
  - faster match and fewer searching of appropriate templates (DateDetector.matchTime calls rarer DateTemplate.matchDate now);
  - standard filters extended with exact prefixed or anchored date templates;

template cache introduced (in opposition to default template cache, holds custom templates cached by pattern for possible common usage of same template/regex);
2016-10-17 11:20:27 +02:00
sebres bd1eb70c52 speedup template first time selection through pre-sorted template list by template hits 2016-10-17 11:18:35 +02:00
sebres 0bed91b3c2 speedup SeekToTime test cases using exact date pattern... 2016-10-17 11:18:33 +02:00
sebres e735f8f568 default non-unicode and case-sensitive matching (by pattern templates automatically add `(?iu)` for "ignore case" and "unicode" if expected) 2016-10-17 11:18:32 +02:00
sebres ab0ac2111c added possibility to specify more precise default date pattern:
- `datepattern = {^LN-BEG}` - only line-begin anchored default patterns
     (matches date only at begin of line, or with max distance up to 2 non-alphanumeric characters from line-begin);
  - `datepattern = {*WD-BEG}` - only word-begin anchored default patterns;
  - `datepattern = ^prefix{DATE}suffix` - exact specified default patterns (using prefix and suffix);
common filter configs gets a more precise, line-begin anchored (datepattern = {^LN-BEG}) resp. custom anchoring default date-patterns;
2016-10-17 11:18:30 +02:00
sebres f56ff5f48b optimized to better usage of the last time template (parse part of line at the same place as last time, if enclosed in the same boundaries)
thereby follow rule "shortest distance to datetime should win", so possible collision causes search though all templates;
speedup it a little bit (not possible collision if distance <= 1 or if line-begin anchoring, so break search if such template found)
2016-10-17 11:18:27 +02:00
sebres b9033d004e amend distance collision check - always find template with shortest distance, also first time (test-case extended);
datedetector property template without lock, initially placed start-anchored templates at begin of template list, small optimization of strptime
2016-10-17 11:18:26 +02:00
sebres 75a5440acf extends date detector template with distance (position of match in log-line), to prevent grave collision using (re)ordered template list (e.g. find-spot of wrong date-match inside foreign input, misleading date patterns by ambiguous formats, etc.);
By change of the distance (e.g. another format found), the pattern with smallest distance will be always preferred now.
To speedup (template lookup) resp. minimize of list reorder counts, the distance will be used as divider factor of the template weight by the templates comparison.
2016-10-17 11:18:24 +02:00
sebres 84fe55b99b [temp commit] 2nd try to optimize datedetector/datetemplate functionality (almost ready, needs fine tuning) 2016-10-17 11:18:19 +02:00
sebres a7d9de8c52 [temp commit] 1st try to optimize datedetector/datetemplate functionality (fix ambiguous resp. misleading date detection if several formats used in log resp. by format switch after restart of some services):
* Misleading date patterns defined more precisely (using extended syntax %E[mdHMS]
  for exact two-digit match)
* `filter.d/freeswitch.conf`
    - Optional prefixes (server, daemon, dual time) if systemd daemon logs used (gh-1548)
    - User part rewritten to accept IPv6 resp. domain after "@" (gh-1548)
2016-10-17 11:16:20 +02:00
Aaron Lindsay 7805f9972d filter.d/sshd.conf: Match 'Invalid user' with 'port \d*' 2016-10-15 15:52:19 -04:00
Yaroslav Halchenko 5502e47486 Merge pull request #1579 from sebres/fix-gh-1578
filter.d/sendmail-reject.conf: double space (should be by missing dns-host only)
2016-10-15 13:18:52 -04:00
sebres 10bdadaef2 fixed sporadically (multi-threading) errors by reload/stop/start of polling filter inside getModified (so prevents to stop running main cycle) 2016-10-15 19:12:11 +02:00
sebres 519e355bf2 ChangeLog entry added 2016-10-15 14:59:36 +02:00
sebres 84c3eb3e0e filter.d/sendmail-reject.conf: double space (should be by missing dns-host only)
Closes #1578
2016-10-15 14:53:45 +02:00
Serg G. Brester c1174d7935 Merge pull request #1577 from sebres/_0.10/code-review-fix-log-fmt-auto-verbosity
0.10/code review + fix log format by auto verbosity
2016-10-15 13:01:19 +02:00
sebres 7f8c48d59e code review (e. g. remove code duplication) and coverage 2016-10-15 00:39:25 +02:00
sebres 5b40309052 code coverage of server module: switch backend 2016-10-14 23:57:52 +02:00
sebres 57a7795282 code coverage of server module: multiple ignoreregex 2016-10-14 23:57:35 +02:00
sebres 1ef367e77a fixes log format by starting server with `--loglevel=debug` without specifying of verbosity level 2016-10-14 23:30:12 +02:00
sebres 98f87a1a52 better server-ready event: notify waiting thread if server really ready (communication ready) or failed to start 2016-10-14 22:50:30 +02:00
sebres c809c3e61e Merge branch 'master' into 0.10 2016-10-13 19:01:13 +02:00
sebres 15dc2db8bb Merge pull request #1498 from ahpnils:npf to master:
This new action files adds support for the NPF packet filter, available on NetBSD since version 6.0.
Closes #1498
2016-10-13 19:00:54 +02:00
Nils f7df6026a3 Update Changelog to reflect the new np.conf action 2016-10-13 18:53:16 +02:00
Nils d08db22b92 Create npf.conf for the NPF packet filter
This file adds support for the NPF packet filter, available on NetBSD since version 6.0
2016-10-13 18:50:54 +02:00
Serg G. Brester 77f2dcfdb6 Merge pull request #1576 from sebres/_0.10/fail2ban-regex-coverage
tests of fail2ban-regex extended to cover exec_command_line also
2016-10-13 18:42:07 +02:00
sebres 44f93bfbff increase coverage, better test and output of errors 2016-10-13 18:27:59 +02:00
sebres 7e8575cc56 tests of fail2ban-regex extended to cover exec_command_line also;
Closes #1573
2016-10-13 17:32:38 +02:00
Serg G. Brester 733d0ef596 Merge pull request #1569 from sebres/_0.10/fix-fakegooglebot
fixes deprecated DNSUtils.IsValidIP in fakegooglebot ignore command
2016-10-06 12:15:49 +02:00
sebres 2d2d4cf185 amend to c2d2e79b0d48bf66b04c3772c2419f30a4b1f9db: fixed sporadically bug in getBanListExtendedCymruInfo:
except dns.resolver.NXDOMAIN:
UnboundLocalError: local variable 'dns' referenced before assignment
2016-10-05 15:43:39 +02:00
sebres 0ae932ba5e setup fix for python3, bypass directories (__pycache__) created after ignore command was tested 2016-10-05 15:29:56 +02:00
sebres fa8184d4cc fixes deprecated DNSUtils.IsValidIP in fakegooglebot ignore command + test covered now;
Closes #1559
2016-10-05 15:01:33 +02:00
sebres 973ac9a45c amend to c2d2e79b0d48bf66b04c3772c2419f30a4b1f9db: missing error variable in import block;
additionally fixes forgotten skip for cymru server case, if --no-network specified);
Closes #1568
2016-10-05 12:18:55 +02:00
sebres c2d2e79b0d ExtendedCymruInfo: better availability check (code review and timeout's);
max sleep time check of too long sleep increased to 1 second (typo fix)
2016-09-30 17:01:06 +02:00
sebres ee1727ecca Merge pull request #1563 from niklasf/fix-lazy-ipv6-regex (and sebres/fix-lazy-ipv6-regex) into 0.10 2016-09-30 13:34:54 +02:00
sebres 276759b6c2 ExtendedCymruInfo code review and availability check in test cases;
max sleep time check of too long sleep increased to 1 second
2016-09-30 13:19:00 +02:00
sebres 9bf8985e2a nginx-limit-req.conf: more precise failregex (word-boundary if `<HOST>` should be non-greedy for some reasons) 2016-09-30 12:33:43 +02:00