implement encrypted national identities

[chadwhitacre]

⬑ #3994

⬑ #3976

← #3999 — #4001 →

Punchlist

• require a valid email before saving—implement encrypted national identities #4000 (comment)
• clear identities in Participant.clear_personal_information
• add rekeying—implement encrypted national identities #4000 (comment)
• use cursor in add_event
• deal with identities during take_over
• remake against master once add schema and Python for countries #3999 lands → Encrypted identities #4028

[chadwhitacre]

Redid the commit for very minor doc and whitespace tweaks. Previous commit was c55614d.

[chadwhitacre]

Per #3976 (comment), we should require a verified email before storing an identity.

[chadwhitacre]

Started a punchlist in the description.

[chadwhitacre]

I wrote some code to rekey participant_identities.info as part of gratipay/inside.gratipay.com#606. The field doesn't exist yet in #3998, so I'll stub out ./bin/rekey.py there. I'm filing this code here since this is where we will be introducing the participant_identities table.

def rekey(db, packer):
    """Rekey the encrypted data in our database.

    :param GratipayDB db: a database object
    :param EncryptingPacker packer:

    This script features prominently in our procedure for rekeying our
    encrypted data, as documented in the "[Keep
    Secrets](http://inside.gratipay.com/howto/keep-secrets)" howto. It operates
    by loading records from `participant_identities` in batches of 100 based on
    a `_last_keyed` field. It updates `_last_keyed` atomically with the rekeyed
    `info`, so it can safely be rerun in the face of network failure, etc.

    """
    n = 0
    while 1:
        m = _rekey_one_batch(db, packer)
        if m == 0:
            break
        n += m
    return n


def _rekey_one_batch(db, packer):
    batch = db.all("""

        SELECT id, info
          FROM participant_identities
         WHERE _last_keyed < date_trunc('month', now())
         LIMIT 100

    """)
    if not batch:
        return 0

    for rec in batch:
        plaintext = packer.unpack(rec.info)
        new_token = packer.pack(plaintext)
        db.run( "UPDATE participant_identities SET info=%s, _last_keyed=now() WHERE id=%s"
              , (new_token, rec.id)
               )

    return True

[chadwhitacre]

Rebased on #3999 at f366481, previous head was 7422191.

[chadwhitacre]

Old head was 8f22a0e.