Bug: cluster generator fails to parse non-ascii subject and sender

[sebbASF]

The cluster generator uses fields of the msg assuming that they will be strings.

However that is not the case if non-ascii characters have been used.

In such cases, code such as msg.get('subject') will return an email.header.Header object.
This causes code such as bytes(subject, encoding = 'ascii') to fail with

TypeError: encoding without a string argument

In turn, this causes the archiver to revert to a very basic fallback mid:

    mid = hashlib.sha224(str("%s-%s" % (lid, msg_metadata['archived-at'])).encode('utf-8')).hexdigest() + "@" + (lid if lid else "none")

Unless archived-at is defined, this will be constant for a given list id

This is relatively easy to fix; the generator should use the msg_metadata dict which
the archiver has already set up.

HOWEVER, to ensure that it's possible to regenerate the same Permalinks, any fix MUST be implemented as a new generator type, with a new syntax (i.e. change the 'r' prefix).

There are probably some other changes that need to be made to the cluster generator.
For example, Message-Id should be canonicalised.

Note that the fallback mid cannot be changed, as that would affect all the generators.

[Humbedooh]

Let's bump the 'r' prefix then. I'd suggest moving to next char, 'v' :)
So, am I understanding this right in that the generators should be passed msg_metadata instead of msg in the first variable slot, and that would address the main issue? If so, we'd probably be best off by accessing them with msg.get('subject', '') as they may not have been present in the email and this would avoid a key error.

What do you mean by message-id being canonicalized?

[sebbASF]

No, it's not possible to replace msg by msg_metadata, because that will change the output from the existing generators (at least with some input).

I was thinking of adding the metadata as another parameter.
It would only be used by the new generator(s).
This would preserve the existing behaviour.

As to canonicalisation, headers may be wrapped in transit.
For maximum portability, they should be unwrapped before use.