celery-pubsub 2.0.0-beta3

Publish and Subscribe with Celery

Supported dependencies

Python	Celery 3	Celery 4	Celery 5
3.7
3.8
3.9
3.10
3.11
3.12
pypy 3.9
pypy 3.10

Basic usage

import celery
import celery_pubsub

@celery.shared_task
def my_task_1(*args, **kwargs):
    return "task 1 done"


@celery.shared_task
def my_task_2(*args, **kwargs):
    return "task 2 done"


# First, let's subscribe
celery_pubsub.subscribe('some.topic', my_task_1)
celery_pubsub.subscribe('some.topic', my_task_2)

# Or subscribe with decorator + task decorator
@celery_pubsub.subscribe_to(topic="some.topic")
@celery.shared_task
def my_task_3(*args, **kwargs):
    return "task 3 done"

# Or use only `subscribe_to` decorator
@celery_pubsub.subscribe_to(topic="some.topic")
def my_task_4(*args, **kwargs):
    return "task 4 done"

# Now, let's publish something
res = celery_pubsub.publish('some.topic', data='something', value=42)

# We can get the results if we want to (and if the tasks returned something)
# But in pub/sub, usually, there's no result.
print(res.get())

# This will get nowhere, as no task subscribed to this topic
res = celery_pubsub.publish('nowhere', data='something else', value=23)

Advanced usage

Wildcards can be used in topic names:

• * matches any one group

  • some.*.test will match some.awesome.test, some.random.test but not some.pretty.cool.test, elsewhere or here.some.up.test
  • some.* will match some.test and some.thing but it won't match some or some.testy.test

• # matches any number of groups

  • some.#.test will match some.awesome.test, some.random.test, some.pretty.cool.test but not elsewhere or here.some.up.test
  • some.# will match anything that starts with some., such as some.very.specific.topic.indeed
  • # will match anything

# Let's subscribe
celery_pubsub.subscribe('some.*', my_task_1)
celery_pubsub.subscribe('some.*.test', my_task_2)
celery_pubsub.subscribe('some.#', my_task_3)
celery_pubsub.subscribe('#', my_task_4)
celery_pubsub.subscribe('some.beep', my_task_5)
# it's okay to have more than one task on the same topic
celery_pubsub.subscribe('some.beep', my_task_6)

# or subscribe directly with decorator
@celery_pubsub.subscribe_to(topic="some.*")
def my_task_7(*args, **kwargs): ...

@celery_pubsub.subscribe_to(topic="some.*.test")
def my_task_8(*args, **kwargs): ...

# Let's publish
celery_pubsub.publish('nowhere', 4)               # task 4 only
celery_pubsub.publish('some', 8)                  # task 4 only
celery_pubsub.publish('some.thing', 15)           # tasks 1, 3, 4 and 7
celery_pubsub.publish('some.true.test', 16)       # tasks 2, 3, 4 and 8
celery_pubsub.publish('some.beep', 23)            # tasks 1, 3, 4, 5, 6 and 7
celery_pubsub.publish('some.very.good.test', 42)  # tasks 3 and 4

# And if you want to publish synchronously:
celery_pubsub.publish_now('some.very.good.test', 42)  # tasks 3 and 4

# You can unsubscribe too
celery_pubsub.unsubscribe('#', my_task_4)

# Now, task 4 will not be called anymore
celery_pubsub.publish('some.very.good.test', 42)  # task 3 only

Changelog

• 2.0.0
  • Add new decorator: subscribe_to
  • Drop support for CPython 2.7, 3.4, 3.5, 3.6
  • Drop support for Pypy 2.7 and 3.6.
  • Drop support for Celery 3.
  • Add support for Pypy 3.9 and 3.10.
  • Add support for CPython 3.11 and 3.12.
  • Type hints are now directly in the code. No more stubs files.
• 1.0.2
  • Add stubs file for type hinting.
• 1.0.1
  • Changed README.rst to README.md.
  • Added better badges to show the supported status for each Celery & Python version.
• 1.0.0
  • Flexible requirements (no more pinned). Better to support a wide range of environments.
  • Changed test framework from nose to pytest.
  • Flake8
  • Support for Python 3.9 and 3.10.
• 0.2.1
  • Performance: Internally uses a set to store the subscribed tasks.
  • Updated Codeclimate as the old reporter <https://github.com/codeclimate/python-test-reporter>_ is deprecated.
  • Pinned requirements' dependency versions.
    • celery 4.3.0 -> 4.4.0
    • kombu 4.6.4 -> 4.6.7
    • billiard 3.6.1.0 -> 3.6.2.0
    • codeclimate-test-reporter 0.2.3 -> removed!
• 0.2.0
  • Removed Python 3.4 support. Reason: no longer supported by Kombu 4.6+.
  • Officially supported by Python 3.8.
  • Pinned requirements' dependency versions.
    • celery 4.2.1 -> 4.3.0
    • kombu 4.2.1 -> 4.6.4
    • billiard 3.5.0.4 -> 3.6.1.0
    • vine 1.1.4 -> 1.3.0
• 0.1.9
  • Added Python 3.4, 3.5, 3.7, and multiple branches of pypy
  • Pinned requirements' dependency versions.
    • celery 4.1.0 -> 4.2.1
    • kombu 4.1.0 -> 4.2.1
    • billiard 3.5.0 -> 3.5.0.4
    • nose pinned to 1.3.7
    • coverage pinned to 4.3.4 (was already 4.3.4 but fuzzy)
    • codeclimate-test-reported pinned to 0.2.3
  • Extra badge to show the number of downloads (thanks to PePy)
• 0.1.8
  • Fixup for broken pip install celery_pubsub==0.1.7
• 0.1.7
  • PyPI long description fixed
  • Removed README.md and fixed README.rst
  • Added command python setup.py test to run unit tests with coverage
  • pypy support
• 0.1.5
  • Python 3 support
• 0.1.1
  • Added README
  • Refined setup
  • No need to access celery_pubsub.pubsub anymore. Direct access in celery_pubsub.
  • Tests moved out of package
  • Added Travis for CI
• 0.1
  • Initial version