invalidating cached credentials for all users after 4.x and verify shows medic-user-<uname>-meta db's as never passing checks

[Hareet]

I'm having issues trying to upgrade the same set of data from this issue

Differences from that issue, I'm not moving data around.

• Mounting data disk.

• Launching docker compose (with medic-os and haproxy) pointed at data disk.

• Running couchdb-migration container to pre-index-views 4.5.2

• Stopping containers

• Launching single-node cht-core couchdb:4.5.2 image that mounts storage/medic-core/couchdb/data/:/opt/couchdb/data

• Run couchdb-migration container move-node command

• Checking couchdb logs:

[debug] 2024-04-09T23:45 [email protected]  -------- Invalidating cached credentials for user3721
[debug] 2024-04-09T23:45: [email protected]  -------- Invalidating cached credentials for user8215

• Running couchdb-migration verify command gives you errors:

Verifying medic-user-user3721-meta
      Views of database medic-user-user3721-meta are not indexed. 
      This can be caused by a migration failure or by the the views functions not indexing any documents.

Are both output to be expected?

[Hareet]

Okay, I re-ran another upgrade using an original copy of that dataset in the above post.

I can verify never moving data, not touching it at all and when launching 4.5.2, cht-couchdb:4.5.2 on bootup shows the invalidating cached credentials log entry for all users, and when logging in as medic admin user, I'm unable to retrieve messages tab, or users from the admin panel. Both are downloading docs.

I'm looking to clarify if this is expected behavior when jumping to 4.5.2 from 3.x and what the user impact will be

[dianabarsan]

I wouldn't read anything into Views of database medic-user-user3721-meta are not indexed., this message can appear if there are no docs that need to be indexed or anything like that.

I'm not sure what Invalidating cached credentials for user3721 means, it's a debug message, not an error.

and when logging in as medic admin user, I'm unable to retrieve messages tab, or users from the admin panel. Both are downloading docs.

I'm not sure what this means. Can you log in? It would be helpful to provide more information about the behavior that you are seeing.

[Hareet]

I'm not sure what this means. Can you log in? It would be helpful to provide more information about the behavior that you are seeing.

Yeah, i could log in and eventually everything would load. Partner has finished their testing and are preparing for production, so I'm going to close this issue.

[1yuv]

I've opened this issue as after upgrades existing user credentials can not login the user except medic user.

Edit: couch2pg and admin users can't login with their existing credentials.

[Hareet]

I've opened this issue as after upgrades existing user credentials can not login the user except medic user.

Edit: couch2pg and admin users can't login with their existing credentials.

Actual non-admin users can still login right?

We recreate the couch config in cht-4.x, so the medic user and session hashes are re-entered, but not the other admin users. @dianabarsan I think I remember a discussion sometime back about multiple admins and having to potentially sync passwords across clusters was difficult, so we limited to only 1 admin. Is that accurate? I'm unable to find the issue where the discussion happened

[1yuv]

Actual non-admin users can still login right?

This is right @Hareet . For couch2pg user, I can't even see the user on the users list. For admin user, I am able to see this user. I tried to update the password for this user and tried to log in, but that doesn't work. Message from console:

docId
: 
"_local/_2Hd9uEzhGWwzmceOcwYLA%3D%3D"
error
: 
"forbidden"
message
: 
"You are not allowed to access this db."
name
: 
"forbidden"
reason
: 
"You are not allowed to access this db."

Screen.Recording.2024-04-24.at.12.45.04.PM.mov

[dianabarsan]

Hi @Hareet
I think you are correct, we don't recreate admin users when migrating to a cluster, except for the main user. I don't think it's even possible to migrate, because those settings don't get reused.
These users should be created again. Unfortunately, I don't believe we have an endpoint for that anymore, and the last of the code that changed admin passwords was removed as part of this commit: medic/cht-core@15f96b2

The way to add admins is to use the default couchdb endpoints. To add admins on a cluster, you would need to add the admin one one node, and then copy the hashed password onto the other nodes. This is a bit of a complication.

May I ask, do you require these users to be db admins or can your workflow work with a regular online user?