Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add subscription option to disable json parsing #229

Open
csaroff opened this issue Sep 28, 2022 · 2 comments
Open

Add subscription option to disable json parsing #229

csaroff opened this issue Sep 28, 2022 · 2 comments

Comments

@csaroff
Copy link

csaroff commented Sep 28, 2022

Currently rele's subscription model only supports consuming messages that are json deserializable.

Unfortunately some of GCP's core services publish non-json messages. For example, the gcp dicom store publishes notifications to a topic when a new dicom instance was inserted, but the message field is just the dicom instance path. When attempting to consume these messages through rele, we get the following error.

Exception raised while processing message for dicom_uploads - handle_upload: JSONDecodeError
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/rele/subscription.py", line 112, in __call__
    data = json.loads(message.data.decode("utf-8"))
  File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Can we add an option to disable deserialization of these messages? Something like:

from rele import sub

@sub(topic='my-topic', deserialize_message=False):
def handle_upload(data, **kwargs):
    print('Handling data', data)
@andrewgy8
Copy link
Contributor

andrewgy8 commented Sep 29, 2022

Another extension to this is something where the Callback deserialize function could be configurable.

Something like:

def deserialize_func(message):
	return message

@sub(topic='my-topic', deserialize=deserialize_func):
def handle_upload(data, **kwargs):
    print('Handling data', data)

By default, I would expect the current behaviour. But for some use cases, like @csaroff 's , you may want to control the deserialization logic yourself.

@csaroff
Copy link
Author

csaroff commented Sep 29, 2022

Yeah, agreed. That's much better

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants