-
Notifications
You must be signed in to change notification settings - Fork 2
/
Copy pathTODO
49 lines (39 loc) · 2.5 KB
/
TODO
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
Look into using LUA Scripting in Redis:
http://www.redisgreen.net/blog/2013/03/18/intro-to-lua-for-redis-programmers/
Replace the current deduplication logic with a Redis/LUA script.
Message De-duplication:
- De-dupe should be opt-in: if you're only using 1 broker, there is no point
De-dupe Key Store / Lock Manager
- in order to not have unbounded growth in the # of keys
redis must store, implement an expiration policy
- key expiration policy based on age K[a]
- purge keys
- consumers route to dead letter queue / table if/when message being processed has
an age of M[a] >= 0.9 * K[a]
Publishing:
- do not use a constant ordering for the brokers to publish to.
Eg, for [1,2,3] where publishing to 2 of, use different permutations [1,2] [2,3], [3,2], [2,1], ...
This will help with message ordering (when there are 2 brokers) and spread when there are >2 brokers
- support N/M publishing constraints: eg: "at least 2(N) of 5(M) brokers"
- support 2 types of publish calls: 1 that uses redundancy (slower) and one that doesn't
for the non-redundant, it can be faster and doesn't need to consult the lock manager
- move 250ms publisher retry delay into configuration
- do same for other delays, defaults and all other parameters - put into 1 var that can be overridden
Configuration:
- allow teporingo (Clojure rabbit client) publisher configuration to be pulled from config/teporingo.yml
AMQP Utilities
- functions for clearing out: vhsots, exchanges, queues and bindings
- functions for creating users, removing users, setting permissions?
Consumer Reliability
- in the case when a message is poision to the consumer, IOW it throws an exception, it
should be pulled from the queue so that processing can continue - this raises the
requestion of what should be done with it. One idea is to route it to a dead letter store,
either another exchange/queue, a database, a file-system, etc. Should this be the responsibility
of teporingo? Or is this up to users to implement their own?
At this time poision messages cause undesireable behavior: they cause a busy
loop where the client is delivered the message, it fails with an exception and
the consumer dies. The message is not acked, so rabbit puts it back on the
queue. Teporingo detects that the consumer died and restarts it (for me this
is desireable behavior to increase the reliability for transient errors). Then
the process repeats itself, spinning, using up resources even though the message
will never actually be processed.