client secret reset without disruption

[stevehu]

For some customers, they have security policies to reset functional id password on a yearly basis and client secret is considered the same. Given the special usage of client id in OAuth 2.0 and its presence of logging, auditing and metrics, we cannot create a new client id for every reset. So the only option is to reset the client secret. When resetting secret, we cannot shut down the client instances all together to update the secret in the configuration at the same time. It has to be updated in a rotation which means we have to support multiple secrets for one client. The design would be that one client can have multiple secrets and each secret has a time slot that is effective. During rotation, there are two valid secrets at the same time so that some instances use the new secret to authenticate itself. The existing instances are still using the old secret until they are all updated. The old secret then needs to be retired.