These locks provide a similar interface to those in the threading and multiprocessing modules with a different set of tradeoffs. These locks are useable by multiple different processes when those processes agree on a naming scheme. This can be used in order to allow multiple processes running on the same machine to semi-coordinate or in the case of the redis or dynamo backed locks multiple processes running on different machines to coordinate work. This provides a different type of coordination than multiple workers consuming from a single queue and can allow quickly prototyping a solution where workers attempt to grab a job and take a lock on it, grabbing another job if they fail at getting the lock. One benefit of this type of solution is that it allows running on spot hardware in the cloud since if a single job is dropped before it is completed the lock will soon expire and another worker will be able to grab that same piece of work.
Much like the locks provided by the threading/multiprocessing modules, these can (and probably should) be used as context managers.
SocketLock requires no additional third party libs, and should work well on *nix OS's.
Advantages:
- if a process dies the lock is released ~instantly
- no lockfiles polluting the filesystem
Disadvantages:
- requires all processes to be on the same host OS
- only works on nix-based os's (and maybe not even mac)
from locking import SocketLock
import time
with SocketLock(lockname="uniquename") as mylock:
# at this point we're holding the lock and can
# safely perform operations without worrying about
# other threads/process holding a lock with this name
# interfering
time.sleep(1)FileLock requires no additional third party libs and should work on most OS's, with the disclaimer that I only have access to *nix OS's.
from locking import FileLock
import time
with FileLock(lockname="foolock") as mylock:
time.sleep(1)RedisLock requires redis and obviously a redis server.
The advantage of RedisLock over SocketLock or FileLock is that you don't need to be on the same host as other processes.
This can be useful if you want one of N hosts to perform some action.
from locking import RedisLock
import time
with RedisLock(lockname="some_process_identifier", hosts=["myredis.com"]):
time.sleep(1)DynamoLock doesn't require an always on redis like RedisLock, however it does require dynamodb access on AWS.
In theory this should be pretty cheap.
from locking import DynamoLock
import time
with DynamoLock(lockname="some_process_identifier", table="locks", checkpoint_frequency=2, ttl=5):
time.sleep(1)python -m pip install --upgrade lockingpython -m pip install --upgrade git+https://[email protected]/jbylund/locking.git