New message format for the IRMA proofs (keyshare JWTs and improved consistency)

[ivard]

We are currently looking into the message format of the proofs response. We have to make a change here, because we want to add the responses of the keyshare server to the actual proof. In #240 we change the keyshare JWT format and remove all information from the JWT that verifiers don't need to know. This enables us to make this change.

Currently, there is a difference in message format between regular disclosure sessions and attribute-based signature sessions. In short, the key differences are:

• In regular disclosure sessions the proofs signatures are named "proofs" and in attribute-based signatures "signature".
• In attribute based signatures the "message" and a "timestamp" fields are included extra. As a consequence, the 'realnonce' value is a hash of message, timestamp and nonce, whereas in regular disclosure sessions the realnonce is equal to just the "nonce".
• In attribute based signatures a JSON-LD context is present ("@context" field)
• In attribute based signatures, the nonce and the Idemix context ("context" field) is included explicitly

The keyshare JWTs that we add also includes a iat, so for sessions that use a keyshare server two timestamps might be involved (one in the keyshare JWT and one in the "timestamp" field). The "timestamp" is meant to realize non-repudiation in attribute based signature sessions, and the keyshare JWT is meant to verify that the wallet was authentic at the time of the session. So, both timestamps have a different purposes.

Below I made a proposal how the new format looks like in the most simple form. Maybe we can improve it even more.

Ideas:

• Also include "nonce" and "context" for regular disclosure sessions (Q: how do we then clearly explain that the "nonce" in the regular disclosure session needs to be used as-is, whereas in attribute based signature sessions it needs to be combined with other values?)
• Also include the timestamp for regular disclosure sessions and fully remove the message format difference (Q: is the extra consistency worth the extra complexity and the extra network requests to the atumd timestamp server?)

I'd like to hear your opinions.

Regular disclosure sessions:

{
   "@context": "https://irma.app/ld/disclosure/v1",
   "signature": [
      {
         "c": "PwhkA1sYUfdh0s4PwO53shtxUbeFQB4HQwlQc4+Ekhw=",
         "A": "btD2Fy4D0fSSffyMSUW+g1t3wpoCzVZs9h1KIkNnRm7hk/1RLLgfzL8UDCzlbR+IaFreA1oUka9HzDl8TWLd7q+hw8BYK8yWsxmROInXUZAaNkXm7wHIsoKFMd3fk5Tqtgw5HtbAir4yYsGkYY9REh1KbnjeZ1AFexk0vj0ab1E=",
         "e_response": "hS/wYp6rZEF8ReP3JqWvGlKQHeO2UPzPgXjXm3L7xVNUXDfAI117OpPObrVuS3FoeMUIxt4kn5Lu",
         "v_response": "BREvFtXCn4ZX/xUChMFEhY+Zb2dbtYay3bg+ioe7F/7ek5RdtNXWHU6doY4xvFvp6ZGFL5jhEGmPLPY28TicY85VeswM0bhG3TgwXTb4jhCozO2m6yPnREXjEO7HKKyqMCzM5PSL+09Hn8QuY//8/CUOpdx+bdd8JH7N9QzM6oEqHV7u6wMkzVpWXn0vxD6Oexe130dpeutjKmasETnkQfI5XMcj1tfUQuwAi3SwJR2h1GTRV8Xayg8IHP768WDtNy4pfeUrbHnBdqm8ZInnKWX327+raBR74qQdKZ9FFkPOESR8msaGu6prppvregRjuDBF/q6/LtrQv/kSRZwK",
         "a_responses": {
            "0": "jiMQkufXH9k4S9zysBq1Do6e6RwCrh5ykKBM/229ChjCaGRmivQwkaqLD/un6gb1g+tsHN585qAjZx4Jmd9IZeNvv9VyzK0udTk=",
            "2": "Vml7tPyn30rSz4tiM2lP1aX4kNDUitPJMMqoL0RX46/nVCIh3ggDLaRl6j2z2U1ceBQk/VPz42X7NyqKnLoPZJ/89TIwhel/A7k=",
            "3": "5dIYc3y+9T1n9FQjtnZ6/7dzScy4iRpQ00MKoQeGIgQG2U3ls4/1jaL9A10amkutYzaMpYK4lBove9JQWDt2+tvqyHozOFclyuE=",
            "5": "1e5C23QrEUOIa2kE8tJZncdvi3pR81viDyT9ywAADVjPkx9SFzamFMLHGEH5M2aCMV0MaReYVGAzHB4fWyoyN3xUkwYC4IyvT4k="
         },
         "a_disclosed": {
            "1": "AwAKOQIBAALWy2qU9p3l52l9LU1rVT4M",
            "4": "aGpt"
         }
      },
      {
         "c": "PwhkA1sYUfdh0s4PwO53shtxUbeFQB4HQwlQc4+Ekhw=",
         "A": "nrEY/yFSQOW3j5aMDsEnXtXBxl39hdNYWKTXkbJqrp005RiTHshebwDZFjDhcXz8LyZM7JoGAYpRTmukFK6CebrpGYfdrQ2o3Y+6MSL+4WsDysaRuAV/K9vBeCAHtlx+8lIURkrDg9fhNYE6EszIaLj7FsbLBSOXsmAqUQxmg+kC2MXvQPL9d1tgswAAy1cDLZispVKKzurqwlmaXM2g3JlkSPVYehEg0LKwHNr3EEKD6VKXYsxn2biAF0s7s3M3Vk8WEU+mCHXzK+Tsy886qZLqJZR569slY8viPEiMiTHq5cmBDECrC+1tcJw+PFSg/DOKxpspy7mL2JU5He0Smw==",
         "e_response": "i6wuZTlUJ5xKPlQVQZjSeY2nc8zyoCrlMEk6OLHyp9k4jAy+2C5Is4JwgJxmJH+V4z4Dhm/4P7XSCXek8XKe",
         "v_response": "Bv7Ayf8u0GfY7kMQb2nQ2BhlPPIKutExYW6kctCpF63rY9MSvXIh/hVq5eGmAuN3WlyrlrE5abKLivBed3ERyG5jnxS2d+JYPf4blgFiY3CVSDLSEUTtaZ6y9N8Zub0H2WBeGyF/rEiZcGRN3SJcX56OSOSklnFuXDVY79BD6gXuhRI8B5l6LJ1HmwbrJvFmn9OAn6AJ+UcbXGsMl936HZtl7Y7DS9VceCAQpCvD2mdrjzjK9Q4z5lZPk9vQhG0E/gBAcl+rfrbro29yAHn7CuTsJUc6OPRVjksiJVJfbDX6uH1mUal8vfTHZML0kvloTHz1STRz07la9lUGLTYPLw77rwdaoTfKQhPr9QN33beI1FL5+e9iDo7Ydw2MXjyyx4b4+uBNKAp2T3yOOI142hD2wBOQzwm2sri4QUCi4axQN4awwNzUx1kbJ8gq5lND5BfgPjYHNEadC+Q/c0jzHnPuZq9e8gq8MgUMcKTlBW0EK5IRDCBIvx6AMQhuxdAcw5i2f/kNgzaIRXPuQKI7TgI=",
         "a_responses": {
            "0": "nM8G0zLmL/u5xf/1zQ5DgVjc2kUh+K4m2wTNaSC1YxuHhAoHOB8YenmdyzW190TrwJUlHC3JxGXKCjwJcTEakcs2n3fyshjN/qxptCY5qKU="
         },
         "a_disclosed": {
            "1": "AwAK8AAaAANPk8AhXLlPUlSw2hYiAvCI",
            "2": "6Mrm6OrmyuTcwtrL"
         }
      }
   ],
   "indices": [
      [
         {
            "cred": 0,
            "attr": 4
         }
      ],
      [
         {
            "cred": 1,
            "attr": 2
         }
      ]
   ],
   "keyshare_jwts": {
      "pbdf-1": "..."
   }
}

Attribute based signature:

{
   "@context": "https://irma.app/ld/signature/v2",
   "signature": [
      {
         "c": "PwhkA1sYUfdh0s4PwO53shtxUbeFQB4HQwlQc4+Ekhw=",
         "A": "btD2Fy4D0fSSffyMSUW+g1t3wpoCzVZs9h1KIkNnRm7hk/1RLLgfzL8UDCzlbR+IaFreA1oUka9HzDl8TWLd7q+hw8BYK8yWsxmROInXUZAaNkXm7wHIsoKFMd3fk5Tqtgw5HtbAir4yYsGkYY9REh1KbnjeZ1AFexk0vj0ab1E=",
         "e_response": "hS/wYp6rZEF8ReP3JqWvGlKQHeO2UPzPgXjXm3L7xVNUXDfAI117OpPObrVuS3FoeMUIxt4kn5Lu",
         "v_response": "BREvFtXCn4ZX/xUChMFEhY+Zb2dbtYay3bg+ioe7F/7ek5RdtNXWHU6doY4xvFvp6ZGFL5jhEGmPLPY28TicY85VeswM0bhG3TgwXTb4jhCozO2m6yPnREXjEO7HKKyqMCzM5PSL+09Hn8QuY//8/CUOpdx+bdd8JH7N9QzM6oEqHV7u6wMkzVpWXn0vxD6Oexe130dpeutjKmasETnkQfI5XMcj1tfUQuwAi3SwJR2h1GTRV8Xayg8IHP768WDtNy4pfeUrbHnBdqm8ZInnKWX327+raBR74qQdKZ9FFkPOESR8msaGu6prppvregRjuDBF/q6/LtrQv/kSRZwK",
         "a_responses": {
            "0": "jiMQkufXH9k4S9zysBq1Do6e6RwCrh5ykKBM/229ChjCaGRmivQwkaqLD/un6gb1g+tsHN585qAjZx4Jmd9IZeNvv9VyzK0udTk=",
            "2": "Vml7tPyn30rSz4tiM2lP1aX4kNDUitPJMMqoL0RX46/nVCIh3ggDLaRl6j2z2U1ceBQk/VPz42X7NyqKnLoPZJ/89TIwhel/A7k=",
            "3": "5dIYc3y+9T1n9FQjtnZ6/7dzScy4iRpQ00MKoQeGIgQG2U3ls4/1jaL9A10amkutYzaMpYK4lBove9JQWDt2+tvqyHozOFclyuE=",
            "5": "1e5C23QrEUOIa2kE8tJZncdvi3pR81viDyT9ywAADVjPkx9SFzamFMLHGEH5M2aCMV0MaReYVGAzHB4fWyoyN3xUkwYC4IyvT4k="
         },
         "a_disclosed": {
            "1": "AwAKOQIBAALWy2qU9p3l52l9LU1rVT4M",
            "4": "aGpt"
         }
      },
      {
         "c": "PwhkA1sYUfdh0s4PwO53shtxUbeFQB4HQwlQc4+Ekhw=",
         "A": "nrEY/yFSQOW3j5aMDsEnXtXBxl39hdNYWKTXkbJqrp005RiTHshebwDZFjDhcXz8LyZM7JoGAYpRTmukFK6CebrpGYfdrQ2o3Y+6MSL+4WsDysaRuAV/K9vBeCAHtlx+8lIURkrDg9fhNYE6EszIaLj7FsbLBSOXsmAqUQxmg+kC2MXvQPL9d1tgswAAy1cDLZispVKKzurqwlmaXM2g3JlkSPVYehEg0LKwHNr3EEKD6VKXYsxn2biAF0s7s3M3Vk8WEU+mCHXzK+Tsy886qZLqJZR569slY8viPEiMiTHq5cmBDECrC+1tcJw+PFSg/DOKxpspy7mL2JU5He0Smw==",
         "e_response": "i6wuZTlUJ5xKPlQVQZjSeY2nc8zyoCrlMEk6OLHyp9k4jAy+2C5Is4JwgJxmJH+V4z4Dhm/4P7XSCXek8XKe",
         "v_response": "Bv7Ayf8u0GfY7kMQb2nQ2BhlPPIKutExYW6kctCpF63rY9MSvXIh/hVq5eGmAuN3WlyrlrE5abKLivBed3ERyG5jnxS2d+JYPf4blgFiY3CVSDLSEUTtaZ6y9N8Zub0H2WBeGyF/rEiZcGRN3SJcX56OSOSklnFuXDVY79BD6gXuhRI8B5l6LJ1HmwbrJvFmn9OAn6AJ+UcbXGsMl936HZtl7Y7DS9VceCAQpCvD2mdrjzjK9Q4z5lZPk9vQhG0E/gBAcl+rfrbro29yAHn7CuTsJUc6OPRVjksiJVJfbDX6uH1mUal8vfTHZML0kvloTHz1STRz07la9lUGLTYPLw77rwdaoTfKQhPr9QN33beI1FL5+e9iDo7Ydw2MXjyyx4b4+uBNKAp2T3yOOI142hD2wBOQzwm2sri4QUCi4axQN4awwNzUx1kbJ8gq5lND5BfgPjYHNEadC+Q/c0jzHnPuZq9e8gq8MgUMcKTlBW0EK5IRDCBIvx6AMQhuxdAcw5i2f/kNgzaIRXPuQKI7TgI=",
         "a_responses": {
            "0": "nM8G0zLmL/u5xf/1zQ5DgVjc2kUh+K4m2wTNaSC1YxuHhAoHOB8YenmdyzW190TrwJUlHC3JxGXKCjwJcTEakcs2n3fyshjN/qxptCY5qKU="
         },
         "a_disclosed": {
            "1": "AwAK8AAaAANPk8AhXLlPUlSw2hYiAvCI",
            "2": "6Mrm6OrmyuTcwtrL"
         }
      }
   ],
   "indices": [
      [
         {
            "cred": 0,
            "attr": 4
         }
      ],
      [
         {
            "cred": 1,
            "attr": 2
         }
      ]
   ],
   "keyshare_jwts": {
      "pbdf-1": "..."
   },
   "nonce": "SG9VA+rlP3uCT1H/zGcNJg==",
   "context": "AQ==",
   "message": "test",
   "timestamp": {
      "Time": 1694012229,
      "ServerUrl": "https://irma.sidn.nl/atumd",
      "Sig": {
         "Alg": "ed25519",
         "Data": "NHwUO/7CecRfJ8bs/spjZ+Giz+w+EHyVXjQxS7FcGpiJ3Dl8kS3wC6iyrTw8JZ8PxOVsqrPD6yfhVj1Hv+OXCQ==",
         "PublicKey": "XqKPue9V0YMZuDxo6/gcaFiy314myUQECDbw1qmePYY="
      }
   }
}

[stenreijers]

There is a pull request open (#245) that we currently deploy. It includes more information on a new proof endpoint /session/{requestorToken}/result-extended that allows us to better integrate Yivi in our infrastructure.

Perhaps that pull request, which has not yet been accepted in the master branch, could serve as an initial basis for the discussion here.

[ivard]

If I understand you correctly, you propose to also use your result-extended format for the response that the app sends to the IRMA server to make it more transparant for people who would like to communicate with the Yivi app directly (without an IRMA server in between).

The data format you propose looks quite nice and clear. It's a bit difficult that the credential metadata is included twice in this format, once in the JSON object and once in attribute 1.

irma meta AwAKvAEFAAODVwoDVGgGat2KoqbPp64A
Identifier      : pbdf.gemeente.personalData
Signed          : 2022-09-01 02:00:00 +0200 CEST
Expires         : 2027-09-02 02:00:00 +0200 CEST
IsValid         : true
Version         : 3
KeyCounter      : 3
KeyExpires      : 2023-05-17 16:35:53 +0200 CEST
KeyModulusBitlen: 2048

This is a bit annoying for validation since it would require that we check that all the metadata attributes are in sync with the plain values in the JSON object. Wouldn't it be a nicer solution to change the encoding of the a_disclosed values in such a way that the value becomes human readable in the JSON object itself? That approach is a bit more universal.

A concern I still have is that major change in the data format of the proofs may be quite some work. It would require quite some refactoring, since the proof validation code in irmago should be changed a bit to support the new format and it should remain backwards compatible with the old format for a smooth migration process. My original idea was to keep the changes as small as possible to prevent a lot of work.

[stenreijers]

Could you clarify a bit more on what you mean with "a lot of work". The pull request has that endpoint already implemented.

I agree that the duplication of attribute 0 is not nice. However, in JSON format is better since it does not require any additional processing.

[ivard]

My proposal concerns a change of the /proofs endpoint that the Yivi app uses to post its proofs to the IRMA server. For a smooth migration process we need old Yivi apps to communicate with new IRMA servers and new Yivi apps to communicate with old IRMA servers. So irmaclient should be able to generate the proof in both the old and the new formats, and irmaserver needs to be able to verify both proof structures. That's not covered yet by your implementation.

Furthermore, we don't have full focus for this type of changes. We have a small team that needs to multitask a lot. That's why we want to give things a clear scope. Otherwise, we have no time anymore for our operational responsibilities.

[stenreijers]

This is a bit annoying for validation since it would require that we check that all the metadata attributes are in sync with the plain values in the JSON object.

In our use-case there is no third party in between the consumer of the JSON object, so we do not experience these trust issues.

It would require quite some refactoring, since the proof validation code in irmago should be changed a bit to support the new format and it should remain backwards compatible with the old format for a smooth migration process

This is indeed something that could be some refactoring. Although you could put this out of scope and do not mix the proof validation process with a specific proof format with an extended JSON endpoint that just provides a lot more information. You can also write a small piece of code that just transforms the extended JSON endpoint to the compact proof format. Internally, that would require little to no work.

That way you have two proof formats: extended and compact, where the compact is just the "legacy" JSON object that only contains the necessary material to validate the proof and does not contain all extended fields. A simple transformation between extended and compact proof formats can be made quite fast, right?

[bobhageman]

@context:
It seems valuable to have the @context in all messages. However, there could be confusion due to the existing context property in attribute-based signatures. Since context is always a base64 encoded number 1 and doesn't seem to be used, can it be removed? This would also require removing context from the session request for completeness (https://irma.app/docs/irma-protocol/#get-irma-session-clienttoken-request).

nonce:
Ideally, the nonce property should always have the same structure (mixing apples and pears in the same property is error-prone). So, simplistically thinking, can we standardize the structure of the nonce? Meaning, either keep it as-is everywhere or construct it from a combination of multiple values everywhere? An alternative could be to keep the difference intact but determine how it's calculated and verified in code based on @context (and document it thoroughly).

timestamp:
I believe it's valuable to make the messages as consistent as possible. This allows for a consistent way to check if a message has the correct format, such as in a test suite like: https://github.com/w3c/vc-test-suite in the future. The additional network request load should be manageable. However, the question is whether this adds significant complexity to the code? In my opinion, that could be a reason not to do it.

[ivard]

I think it makes sense to remove the context from the message format indeed. It's something from the Idemix specification and we need that to create and validate proofs. We chose a fixed constant for this. So, you could argue that the context value is kind of bound to the @context value in this way. Given the @context you can determine what the context should be for Idemix.

Concerning the nonce, I agree that standardising would be nice. If we use the exact nonce mechanism that we currently use for signatures, then there is not much work I think. Both mechanisms are already in place, so it's just a matter of calling another function at some places.

Concerning the timestamp, I think the code change is not a major issue here. It is now a conditional piece of code, so removing the if clause is practically all that needs to be done. The only consideration here is consistency vs. performance.

[sietseringers]

For what it's worth, I always had in the back of my mind to at some point unify the distinction between the two session types as follows:

• Remove the attribute-based signature session request type altogether.
• Add a message *string field to the DisclosureRequest type, which if present gets signed the way the message of an attribute based signature request does (also enabling the nonce -> "real nonce" transformation).
• Add a boolean to the DisclosureRequest type requesting the presence of a timestamp. If no message is present then it defaults to false, and otherwise to true, but in either case the requestor can override it.

Such a setup would offer maximum flexibility to the requestor, by offering the possiblity of signing a message and getting a timestamp as orthogonal features, while simultaneously simplifying the session request types.

I think I would not enable timestamps across every session, not only for the extra load on the timstamp server, but also because in the majority of the sessions, which are disclosure sessions, the timestamp is just not necessary: in such sessions the disclosure resulting out of a session is useful only during the session, and not afterwards. In such a scenario the timestamp is not necessary, because the time of creation of the disclosure (now) and the time of verifying it (now) coincide. I would say that that argument is more important than the consistency that always including it would bring (but your opinion may differ).

As to the context (without the @): that is indeed nowhere used in IRMA and it should be fine to remove it. Hypothetical other Idemix implementations might use it, however, so that you lose compatibility with them, but then there already are various minor places where IRMA deviates from the Identity Mixer specification anyway.

[sietseringers]

As to the nonce sometimes being an ordinary nonce and sometimes being more complicated. If you're considering abandoning backwards compatibility with existing attribute-based signatures, then a more graceful approach to take the to-be-signed message and the timestamp into account would be the following. Normally, the challenge (i.e. the field called c in each of the signatures) is computed as follows:

c = SHA256(1, ..., nonce)

in which I have denoted the context with its fixed value 1 and in which ... represents elements that are not relevant for this discussion.

If a message needs to be included, you could adapt this as follows: keep the nonce as it is and compute c instead with:

c = SHA256(2, ..., nonce, message, timestamp)

The context has a different value here to achieve domain separation, i.e., to fully and verifiably separate proofs that have a message from those that don't. I actually think that this is the purpose of that context that we never used. Note that you still don't need to include it in the various structs: if a message is present or expected or required then you can infer its value to be 2.

This would be a more sensible approach to implementing attribute-based signatures than the transformation of the nonce (I'm not actually sure why we originally implemented it the way it is). It would probably also remove the necessity for the isSig parameter present throughout parts of (the API of) irmago and gabi, which always looked a bit like a code smell to me.

[ivard]

If the disclosure and signature request are being merged as well, then I agree that of both names disclosure should be the preferred one. I did not consider merging the request side too. That solves the issue that we need different @context values for disclosure and signatures, so that's nice. I think that should be do-able.

Concerning the nonce calculation, it would have been nice if we would have included the message and timestamp in the c directly instead of having the current nonce-in-nonce approach. I'm not convinced yet that addressing this issue is worth the effort now though.

[ivard]

Besides, I was wondering, why would a verifier request for a disclosure (without a message) that includes a timestamp? I don't really see the use case there yet.

[sietseringers]

Concerning the nonce calculation, it would have been nice if we would have included the message and timestamp in the c directly instead of having the current nonce-in-nonce approach. I'm not convinced yet that addressing this issue is worth the effort now though.

But it would be the most thorough solution that you can get for the nonce inconsistency, I think. It would be a bit of an effort, yes, but not a huge amount I expect. I also think that it would only concern gabi, with the exception of removing that isSig parameter from the APIs (again, assuming backwards compatibilty would be dropped).

Besides, I was wondering, why would an verifier request for a disclosure (without a message) that includes a timestamp? I don't really see the use case there yet.

It would enable usage of a disclosure in more ways:

• Normally, i.e. during the session for authentication purposes;
• Later, e.g. when retrieving it at some later moment from a database or audit log or something, it can still be verified that the disclosure was valid at the time of creation of the timestamp;
• Suppose that verifier A receives the disclosure and verifier B has no trust relationship with verifier A (so it cannot rely on the nonce generated by verifier A). If verifier B somehow obtains the disclosure, during the session or at some later time (e.g. because verifier A sends it or makes it available to verifier B), then verifier B can still verify the disclosure as long as it trusts the timestamp server. I think this might also be useful to @stenreijers.

Note that a single disclosure proof could be used in all three ways.

[woutslakhorst]

If you drop backwards compatibility, would it then not be a good idea to move to the international standards currently mentioned by the ARF?

[ivard]

Do you mean using the message format of VC/VP? That's important of course and @bobhageman is looking into this, but I don't want to already put that into scope here. You do have a point in here that it might not be the highest priority to start improving our old data format.

[stevenvegt]

but I don't want to already put that into scope here

I see that you run into some very specific problems, but since you already giving up backwards compatibility, why not try to put the new requirements into the VP format? Yivi currently being a completely custom non-standard wallet is giving us a lot of headaches. For demo purposes we are now using the Sphereon wallet because a Yivi demo does not prove compatibility with the upcoming European standards.

If it is a knowledge and capacity thing, I'm willing to spend time on this. We have quite some experience with these standards already.

[stenreijers]

Unfortunately even when Yivi implements W3C VC, it is still not compliant with the ARF. The Idemix cryptography scheme, used in Yivi, is not part of the standard.

[ivard]

That's also a good point indeed about Idemix.

For the record, I never said that backwards compatibility is not an issue. I don't mind changing the format if that's necessary, but there has to be a smooth migration process from the old data format to the new.

[woutslakhorst]

The question would be: do you want the product Yivi to succeed or the technology Idemix? Yivi as a product can still survive when it also supports "boring" crypto.

[ivard]

I was thinking, maybe we should clarify which type of keyshare JWT we include. Otherwise that might be confusing. So "keyshare_proofp_jwts" or proofp_jwts. I prefer the first one, but the string length is a bit of a downside.