ngclient feature: Add option to only update metadata if needed

[jku]

This is not full thought out but I'm filing so it's not forgotten.

https://docs.google.com/document/d/1QWBvpwYxOy9njAmd8vpizNQpPti9rd5ugVhji0r3T4c

Sigstore client wants to use TUF to download files but wants to avoid the two mandatory requests (root N+1, timestamp) on startup if it's not necessary. This seems to be a totally valid feature request: if this is feasible we could offer that as an option.

The WIP sigstore client checks the timestamp expiry before creating an Updater: if timestamp is not expired, the client then decides update is not needed. There could be some issues with this:

1. timestamp metadata validity is not checked
2. cached target file validity is not checked
3. root expiry is not checked

these may be partly theoretical worries but it still feels like a hack...

Maybe it is possible to

• offer a (non-spec-compliant) Updater option that verifies
  • that metadata is valid
  • that targets are verified by the metadata
  • without hitting the network if possible
• while still keeping updater implementation readable

?

[jku]

Alternatively if this is acceptable to these users: we could start supporting things like If-None-Match in our http requests... this definitely has some security implications as well, and does not eliminate the request itself)

[jku]

from an API perspective this is not an issue. One option is:

updater = Updater(...)
try:
  # try to load with local metadata only
  updater.refresh(local_cache_only=True)
except ExpiredMetadataError:
  # load normally then
  updater.refresh()

the other choice would be

# lazy_fetch will fetch data from remote but only if the local metadata is not valid
config = UpdaterConfig(lazy_fetch=True)
updater = Updater(..., config=config)

the implementation details of either version are unclear, and there are edge cases (like maybe the top-level metadata is valid but a delegated metadata has expired -- and updating timestamp would have fixed that but we don't find out before we actually try to get_targetinfo()). I think these sort of edge cases may be acceptable: the repository just has to acknowledge that it has a duty to keep renewing expiries earlier if it knows clients are going to lazy_fetch...

[jku]

So I've de done two design attempts so far...

1. client config lazy_refresh=True -- major issue here is that it gives timestamp signer way too much power: a single timestamp compromise would let attacker prevent updates until end of time
2. client config lazy_refresh_max=N -- significantly more complex and now client makes decisions that affect how the repository should be maintained (how long a specific metadata version should be publicly available and when they need to be resigned)

Maybe there is still a third one:

• new client config lazy_refresh=True: if this is not set, then client always updates timestamp
• new optional root attribute max_timestamp_validity=N (or max_lazy_refresh=N): repository takes the responsibility that clients are allowed to use cached timestamps as long as expiry is within max_timestamp_validity seconds. This means two things:
  • compromised timestamp with eternal expiry period won't trick clients
  • repository has now promised that it is making an effort to ensure the validity of all current metadata for max_timestamp_validity seconds -- so get_targetinfo() should not fail unexpectedly even if we use cached timestamp