Notes

General Notes

• Maybe StoreClass (on Broker) should receive logger instance
• Have broker as a command that only needs a configuration file to run (specify a package name for workers, for example)
• Use Travis-CI as CI?
• Load balancing based on Broker (number of workers should be mutable based on machine resources, not number of workers working/not working)
• Should be able to deal with big queues (~30k jobs)
• GitHub's backend (using reque) processes 280k jobs per day
• Job/Pipeline priorities
• Job/Pipeline persistence (for Router and Pipeliner)
• Should we restart workers processes from time to time?
• Should monitor Pipeliner and Router?
• How we can add another Router in the same cluster? If the Router hangs, all the system will stop, it's a single point of failure.
• Option to "tag" brokers (certain jobs need to be run on certain machines)

URLs

• http://pypi.python.org/pypi/multido/0.1.0
• http://packages.python.org/joblib/
• https://github.com/saga-project/BigJob/wiki
• http://blog.pythonisito.com/2012/08/zeromq-flow-control-and-other-options.html
• http://blog.pythonisito.com/2012/08/distributed-systems-with-zeromq.html
• https://github.com/getsentry/sentry
• http://www.gevent.org/
• http://www.python.org/dev/peps/pep-3153/
• https://github.com/nvie/rq
• https://github.com/pydata/pyrallel
• https://github.com/bombela/gevent_subprocess
• https://github.com/ask/multiprocessing2
• https://github.com/alonho/pystuck
• https://github.com/apresta/simple-mapreduce
• https://github.com/cobrateam/stardust
• https://github.com/dustinlacewell/loom
• https://github.com/concurrence/concurrence
• https://github.com/garcia-marc/viri
• http://discoproject.org/
• http://celeryproject.org/
• http://seeknuance.com/
• https://github.com/samrushing/shrike
• https://github.com/defunkt/resque - https://github.com/blog/542-introducing-resque
• https://github.com/sebastien/watchdog
• https://github.com/gitpython-developers/async
• https://github.com/rick446/MonQ
• https://github.com/melor/poni

Resque features

• See what's pending
• Modify pending jobs in-place
• Tags
• Priorities
• Fast pushing and popping
• See what workers are doing
• See what workers have done
• See failed jobs
• Kill fat workers
• Kill stale workers
• Kill workers that are running too long
• Keep Rails loaded / persistent workers
• Distributed workers (run them on multiple machines)
• Workers can watch multiple (or all) tags
• Don't retry failed jobs
• Don't "release" failed jobs