forked from fail2ban/fail2ban
-
Notifications
You must be signed in to change notification settings - Fork 0
/
DEVELOP
426 lines (298 loc) · 13.1 KB
/
DEVELOP
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
__ _ _ ___ _
/ _|__ _(_) |_ ) |__ __ _ _ _
| _/ _` | | |/ /| '_ \/ _` | ' \
|_| \__,_|_|_/___|_.__/\__,_|_||_|
================================================================================
How to develop for Fail2Ban
================================================================================
Fail2Ban uses GIT (http://git-scm.com/) distributed source control. This gives
each developer their own complete copy of the entire repository. Developers can
add and switch branches and commit changes when ever they want and then ask a
maintainer to merge their changes.
Fail2Ban uses GitHub (https://github.com/fail2ban/fail2ban) to manage access to
the Git repository. GitHub provides free hosting for open-source projects as
well as a web-based Git repository browser and an issue tracker.
If you are familiar with Python and you have a bug fix or a feature that you
would like to add to Fail2Ban, the best way to do so it to use the GitHub Pull
Request feature. You can find more details on the Fail2Ban wiki
(http://www.fail2ban.org/wiki/index.php/Get_Involved)
Pull Requests
=============
When submitting pull requests on GitHub we ask you to:
* Clearly describe the problem you're solving;
* Don't introduce regressions that will make it hard for systems administrators
to update;
* If adding a major feature rebase your changes on master and get to a single commit;
* Include test cases (see below);
* Include sample logs (if relevant);
* Include a change to the relevant section of the ChangeLog; and
* Include yourself in THANKS if not already there.
If you are developing filters see the FILTERS file for documentation.
Code Testing
============
Existing tests can be run by executing `fail2ban-testcases`. This has options
like --log-level that will probably be useful. `fail2ban-testcases --help` for
full options.
Test cases should cover all usual cases, all exception cases and all inside
/ outside boundary conditions.
Test cases should cover all branches. The coverage tool will help identify
missing branches. Also see http://nedbatchelder.com/code/coverage/branch.html
for more details.
Install the package python-coverage to visualise your test coverage. Run the
following (note: on Debian-based systems, the script is called
`python-coverage`):
coverage run fail2ban-testcases
coverage html
Then look at htmlcov/index.html and see how much coverage your test cases
exert over the code base. Full coverage is a good thing however it may not be
complete. Try to ensure tests cover as many independent paths through the
code.
Manual Execution. To run in a development environment do:
./fail2ban-client -c config/ -s /tmp/f2b.sock -i start
some quick commands:
status
add test pyinotify
status test
set test addaction iptables
set test actionban iptables echo <ip> <cidr> >> /tmp/ban
set test actionunban iptables echo <ip> <cidr> >> /tmp/unban
get test actionban iptables
get test actionunban iptables
set test banip 192.168.2.2
status test
Coding Standards
================
Style
-----
Please use tabs for now. Keep to 80 columns, at least for readable text.
Tests
-----
Add tests. They should test all the code you add in a meaning way.
Coverage
--------
Test coverage should always increase as you add code.
You may use "# pragma: no cover" in the code for branches of code that support
older versions on python. For all other uses of "pragma: no cover" or
"pragma: no branch" document the reason why its not covered. "I haven't written
a test case" isn't a sufficient reason.
Documentation
-------------
Ensure this documentation is up to date after changes. Also ensure that the man
pages still are accurate. Ensure that there is sufficient documentation for
your new features to be used.
Bugs
----
Remove them and don't add any more.
Git
---
Use the following tags in your commit messages:
'BF:' for bug fixes
'DOC:' for documentation fixes
'ENH:' for enhancements
'TST:' for commits concerning tests only (thus not touching the main code-base)
Multiple tags could be joined with +, e.g. "BF+TST:".
Use the text "closes #333"/"resolves #333 "/"fixes #333" where 333 represents
an issue that is closed. Other text and details in link below.
See: https://help.github.com/articles/closing-issues-via-commit-messages
If merge resulted in conflicts, clarify what changes were done to
corresponding files in the 'Conflicts:' section of the merge commit
message. See e.g. https://github.com/fail2ban/fail2ban/commit/f5a8a8ac
Adding Actions
--------------
If you add an action.d/*.conf file also add a example in config/jail.conf
with enabled=false and maxretry=5 for ssh.
Design
======
Fail2Ban was initially developed with Python 2.3 (IIRC). It should
still be compatible with Python 2.4 and such compatibility assurance
makes code ... old-fashioned in many places (RF-Note). In 0.7 the
design went through major re-factoring into client/server,
a-thread-per-jail design which made it a bit difficult to follow.
Below you can find a sketchy description of the main components of the
system to orient yourself better.
server/
------
Core classes hierarchy (feel welcome to draw a better/more complete
one)::
-> inheritance
+ delegation
* storage of multiple instances
RF-Note just a note which might be useful to address while doing RF
JailThread -> Filter -> FileFilter -> {FilterPoll, FilterPyinotify, ...}
| * FileContainer
+ FailManager
+ DateDetector
+ Jail (provided in __init__) which contains this Filter
(used for passing tickets from FailManager to Jail's __queue)
Server
+ Jails
* Jail
+ Filter (in __filter)
* tickets (in __queue)
+ Actions (in __action)
* Action
+ BanManager
failmanager.py
~~~~~~~~~~~~~~
FailManager
Keeps track of failures, recorded as 'tickets'. All operations are
done via acquiring a lock
FailManagerEmpty(Exception)
raised by FailManager.toBan after reaching the list of tickets
(RF-Note: asks to become a generator ;) )
filter.py
~~~~~~~~~~
Filter(JailThread)
Wraps (non-threaded) FailManager (and proxies to it quite a bit),
and provides all primary logic for processing new lines, what IPs to
ignore, etc
.failManager [FailManager]
.dateDetector [DateDetector]
.__failRegex [list]
.__ignoreRegex [list]
Contains regular expressions for failures and ignores
.__findTime [numeric]
Used in `processLineAndAdd` to skip old lines
FileFilter(Filter):
Files-aware Filter
.__logPath [list]
keeps the tracked files (added 1-by-1 using addLogPath)
stored as FileContainer's
.getFailures
actually just returns
True
if managed to open and get lines (until empty)
False
if failed to open or absent container matching the filename
FileContainer
Adapter for a file to deal with log rotation.
.open,.close,.readline
RF-Note: readline returns "" with handler absent... shouldn't it be None?
.__pos
Keeps the position pointer
dnsutils.py
~~~~~~~~~~~
DNSUtils
Utility class for DNS and IP handling
filter*.py
~~~~~~~~~~
Implementations of FileFilter's for specific backends. Derived
classes should provide an implementation of `run` and usually
override `addLogPath`, `delLogPath` methods. In run() method they all
one way or another provide
try:
while True:
ticket = self.failManager.toBan()
self.jail.putFailTicket(ticket)
except FailManagerEmpty:
self.failManager.cleanup(MyTime.time())
thus channelling "ban tickets" from their failManager to the
corresponding jail.
action.py
~~~~~~~~~
Takes care about executing start/check/ban/unban/stop commands
Releasing
=========
# Check distribution patches and see if they can be included
* https://apps.fedoraproject.org/packages/fail2ban/sources
* http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-analyzer/fail2ban/
* http://svnweb.freebsd.org/ports/head/security/py-fail2ban/
* https://build.opensuse.org/package/show?package=fail2ban&project=openSUSE%3AFactory
* http://sophie.zarb.org/sources/fail2ban (Mageia)
* https://trac.macports.org/browser/trunk/dports/security/fail2ban
# Check distribution outstanding bugs
* https://github.com/fail2ban/fail2ban/issues?sort=updated&state=open
* http://bugs.debian.org/cgi-bin/pkgreport.cgi?dist=unstable;package=fail2ban
* https://bugs.launchpad.net/ubuntu/+source/fail2ban
* http://bugs.sabayon.org/buglist.cgi?quicksearch=net-analyzer%2Ffail2ban
* https://bugs.archlinux.org/?project=5&cat%5B%5D=33&string=fail2ban
* https://bugs.gentoo.org/buglist.cgi?query_format=advanced&short_desc=fail2ban&bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=IN_PROGRESS&short_desc_type=allwords
* https://bugzilla.redhat.com/buglist.cgi?query_format=advanced&bug_status=NEW&bug_status=ASSIGNED&component=fail2ban&classification=Red%20Hat&classification=Fedora
* http://www.freebsd.org/cgi/query-pr-summary.cgi?text=fail2ban
* https://bugs.mageia.org/buglist.cgi?quicksearch=fail2ban
* https://build.opensuse.org/package/requests/openSUSE:Factory/fail2ban
# Make sure the tests pass
./fail2ban-testcases-all
# Ensure the version is correct
in:
* ./common/version.py
* top of ChangeLog
* README.md
# Ensure the MANIFEST is complete
Run:
python setup.py sdist
Look for errors like:
'testcases/files/logs/mysqld.log' not a regular file -- skipping
Which indicates that testcases/files/logs/mysqld.log has been moved or is a directory
tar -C /tmp -jxf dist/fail2ban-0.8.12.tar.bz2
# clean up current direcory
diff -rul --exclude \*.pyc . /tmp/fail2ban-0.8.12/
# Only differences should be files that you don't want distributed.
# Ensure the tests work from the tarball
cd /tmp/fail2ban-0.8.12/ && ./fail2ban-testcases-all
# Add/finalize the corresponding entry in the ChangeLog
To generate a list of committers use e.g.
git shortlog -sn 0.8.11.. | sed -e 's,^[ 0-9\t]*,,g' | tr '\n' '\|' | sed -e 's:|:, :g'
Ensure the top of the ChangeLog has the right version and current date.
Ensure the top entry of the ChangeLog has the right version and current date.
# Update man pages
(cd man ; ./generate-man )
git commit -m 'DOC/ENH: update man pages for release' man/*
# Prepare source and rpm binary distributions
python setup.py sdist
python setup.py bdist_rpm
python setup.py upload
# Provide a release sample to distributors
* Arch Linux:
https://www.archlinux.org/packages/community/any/fail2ban/
* Debian: Yaroslav Halchenko <[email protected]>
http://packages.qa.debian.org/f/fail2ban.html
* FreeBSD: Christoph Theis [email protected]>, Nick Hilliard <[email protected]>
http://svnweb.freebsd.org/ports/head/security/py-fail2ban/Makefile?view=markup
http://www.freebsd.org/cgi/query-pr-summary.cgi?text=fail2ban
* Fedora: Axel Thimm <[email protected]>
https://apps.fedoraproject.org/packages/fail2ban
http://pkgs.fedoraproject.org/cgit/fail2ban.git
https://admin.fedoraproject.org/pkgdb/acls/bugs/fail2ban
* Gentoo: [email protected]
http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-analyzer/fail2ban/metadata.xml?view=markup
https://bugs.gentoo.org/buglist.cgi?quicksearch=fail2ban
* openSUSE: Stephan Kulow <[email protected]>
https://build.opensuse.org/package/show/openSUSE:Factory/fail2ban
* Mac Ports: @Malbrouck on github (gh-49)
https://trac.macports.org/browser/trunk/dports/security/fail2ban/Portfile
* Mageia:
https://bugs.mageia.org/buglist.cgi?quicksearch=fail2ban
An potentially to the fail2ban-users directory.
# Wait for feedback from distributors
# Prepare a release notice https://github.com/fail2ban/fail2ban/releases/new
Upload the source/binaries from the dist directory and tag the release using the URL
# Upload source/binaries to sourceforge http://sourceforge.net/projects/fail2ban/
# Run the following and update the wiki with output:
python -c 'import common.protocol; common.protocol.printWiki()'
page: http://www.fail2ban.org/wiki/index.php/Commands
* Update:
http://www.fail2ban.org/wiki/index.php?title=Template:Fail2ban_Versions&action=edit
http://www.fail2ban.org/wiki/index.php?title=Template:Fail2ban_News&action=edit
move old bits to:
http://www.fail2ban.org/wiki/index.php?title=Template:Fail2ban_OldNews&action=edit
http://www.fail2ban.org/wiki/index.php?title=Template:Fail2ban_Versions&action=edit
http://www.fail2ban.org/wiki/index.php/ChangeLog
http://www.fail2ban.org/wiki/index.php/Requirements (Check requirement)
http://www.fail2ban.org/wiki/index.php/Features
* See if any filters are upgraded:
http://www.fail2ban.org/wiki/index.php/Special:AllPages
# Email users and development list of release
# notify distributors
Post Release
============
Add the following to the top of the ChangeLog
ver. 0.8.13 (2014/XX/XXX) - wanna-be-released
-----------
- Fixes:
- New Features:
- Enhancements:
Alter the git shortlog command in the previous section to refer to the just
released version.
and adjust common/version.py to carry .dev suffix to signal
a version under development.