decode_responses for python3

[charlez]

Notes:

• Noted the following comment a few places in the code: "decode_responses must be set to True when used with python3"
• Is setting decode_responses=True an absolute requirement for python3 environments? If yes, please explain why, thanks.
• FYI: I've encountered a UnicodeDecodeError when setting decode_responses=True, for example: "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 56: invalid start byte"
• Following is a stack trace for the aforementioned case:

 Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/anaconda3/lib/python3.8/site-packages/redis/client.py", line 2484, in xinfo_stream
    return self.execute_command('XINFO STREAM', name)
  File "/opt/anaconda3/lib/python3.8/site-packages/rediscluster/client.py", line 551, in execute_command
    return self._execute_command(*args, **kwargs)
  File "/opt/anaconda3/lib/python3.8/site-packages/rediscluster/client.py", line 626, in _execute_command
    return self.parse_response(connection, command, **kwargs)
  File "/opt/anaconda3/lib/python3.8/site-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/opt/anaconda3/lib/python3.8/site-packages/redis/connection.py", line 739, in read_response
    response = self._parser.read_response()
  File "/opt/anaconda3/lib/python3.8/site-packages/redis/connection.py", line 363, in read_response
    response = [self.read_response() for i in xrange(length)]
  File "/opt/anaconda3/lib/python3.8/site-packages/redis/connection.py", line 363, in <listcomp>
    response = [self.read_response() for i in xrange(length)]
  File "/opt/anaconda3/lib/python3.8/site-packages/redis/connection.py", line 363, in read_response
    response = [self.read_response() for i in xrange(length)]
  File "/opt/anaconda3/lib/python3.8/site-packages/redis/connection.py", line 363, in <listcomp>
    response = [self.read_response() for i in xrange(length)]
  File "/opt/anaconda3/lib/python3.8/site-packages/redis/connection.py", line 363, in read_response
    response = [self.read_response() for i in xrange(length)]
  File "/opt/anaconda3/lib/python3.8/site-packages/redis/connection.py", line 363, in <listcomp>
    response = [self.read_response() for i in xrange(length)]
  File "/opt/anaconda3/lib/python3.8/site-packages/redis/connection.py", line 365, in read_response
    response = self.encoder.decode(response)
  File "/opt/anaconda3/lib/python3.8/site-packages/redis/connection.py", line 131, in decode
    value = value.decode(self.encoding, self.encoding_errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 56: invalid start byte

• With decode_responses=False the UnicodeDecodeError isn't raised.
• What are the risks of continuing to set decode_responses=False in the python3 environment?

Your comments, suggestions, and/or work-arounds are appreciated.

[Grokzen]

@charlez This is not something that i enforce from my lib per say, but it is something that upstream package redis-py enforces and requires the user to make a choice.

Taken from their README.md file

...
By default, all responses are returned as bytes in Python 3 and str in Python 2. The user is responsible for decoding to Python 3 strings or Python 2 unicode objects.

If all string responses from a client should be decoded, the user can specify decode_responses=True to Redis.__init__. In this case, any Redis command that returns a string type will be decoded with the encoding specified.

The default encoding is "utf-8", but this can be customized with the encoding argument to the redis.Redis class. The encoding will be used to automatically encode any strings passed to commands, such as key names and values. When decode_responses=True, string data returned from commands will be decoded with the same encoding.
...

The rule of thumb is that when using python3 this should be set to decode_responses=True and nothing else. I even think that when python 2.7 support is eventually dropped, that flag will be hardcoded to True or maybe even removed and logic will assume it is True.

There is no hard requirements that you really really really must do this, but most data and almost all situations have some kind of requirements for having utf-8 in some way shape or form so just use utf-8 anyway and it should not be an issue for you at any point. If you turn the question around, i see no real reason why you would not want utf-8 everywhere anyway.

[charlez]

OK, understood @Grokzen, thanks.

Please consider my vote to retain the ability to set decode_responses=False in future versions, as I don't control what users elect to put into their streams; but the code must be able to handle whatever format data users elect to stream.

[Grokzen]

@charlez If that is sucha big concern for your use-case please make a plea to the upstream package redis-py instead. This lib will support whatever they support, if they remove it so will it here, if it is still around so will it be here so please open up a case there towards them.

[andymccurdy]

There are no plans to remove decode_responses or change its default value. Allowing decode_responses=False is required for storing binary data such as JSON, pickled Python objects and other miscellaneous data.