Allow vault efs resource acquisition to operate on multiple vaults in parallel

[aryanjassal]

Description

For handlers like VaultsSecretsCopy and VaultsSecretsMove, we need to acquire a transaction across multiple vaults. This allows for all the given operations to be completed in a single transaction, avoiding polluting the commit space.

This PR creates such pattern, to acquire the resources from a vault dynamically in a way which works across multiple vaults at the same time.

Issues Fixed

• Fixes Manage multiple transactions dynamically in the vault domain #821 (FIX ENG-423)
• Fixes VaultsSecretsRemove incorrectly reorders directories to be removed #837 (FIX ENG-450)

Tasks

• 1. Add VaultInternal.acquireRead
• 2. Add VaultInternal.acquireWrite
• 3. Add tests
• 4. Update VaultsSecretsRemove to use this instead of grouping paths by vault name.

Final checklist

• Domain specific tests
• Full tests
• Updated inline-comment documentation
• Lint fixed
• Squash and rebased
• Sanity check the final build

[linear]

ENG-423 Manage multiple transactions dynamically in the vault domain

[linear]

ENG-450 `VaultsSecretsRemove` incorrectly reorders directories to be removed

[aryanjassal]

The tasks in this PR have been completed except for deciding upon a proper fix for the RPC wrapper type. For now, I have written a custom type which adds metadata field while allowing for better type narrowing. (see #847 (comment)). However, this is a temporary fix and not final.

I will need to have a discussion if there are any obvious caveats or things I am missing. Once that is no longer an issue, either the type update can be made local to Polykey (not recommended), or made available globally to js-rpc (recommended, as this type might be optionally used by Polykey CLI, and keeping a RPC type inside Polykey doesn't make sense)

[tegefaulkes]

If the RPC type problem is just a usage problem for now then just deal with it using the as keyword and comment why. Move discussion to a new issue directly addressing it.

[aryanjassal]

If the RPC type problem is just a usage problem for now then just deal with it using the as keyword and comment why. Move discussion to a new issue directly addressing it.

I believe we discussed why this issue can't be resolved by usage of the as keyword. I am asking because I have updated the RPC type to the new one which works in this case, but might not be consistent with other RPCs. SO, I just want to find a solution to this which can be applied globally instead of a targeted fix.

[aryanjassal]

I ran a bunch of tests to see if there was a difference between the new type and the old type.

// Original type currently in RPC
type ParamsTypeA<
  T extends JSONObject = JSONObject,
  M extends JSONObject = JSONObject,
> = {
  metadata?: JSONObject &
    RPCMetadata &
    Omit<T["metadata"] & M, keyof RPCMetadata>;
} & Omit<T, "metadata">;

// My type updated type which works on complex types
type ParamsTypeB<
  T extends JSONObject = JSONObject,
  M extends JSONObject = JSONObject,
> = T & {
  metadata?: RPCMetadata &
    M &
    (T extends { metadata: infer U } ? U : JSONObject);
};

// Types being tested

type NoMetadata = {
  text1: string;
  text2: number;
};

type NonConflictingMetadata = {
  text1: boolean;
  metadata: {
    available: boolean;
  };
};

type ConflictingMetadataDifferentType = {
  text1: number;
  metadata: {
    authorization: number;
  };
};

type ConflictingMetadataSameType = {
  text1: number;
  metadata: {
    authorization: string;
  };
};

type TA = {
  type: true;
  field: string;
};

type TB = {
  type: false;
  value: number;
};

type CombinedType = TA | TB;

// Actual tests

// UsingTypeA is old type currently in RPC
// UsingTypeB is new type I'm working on for this PR

let a1: UsingTypeA<NoMetadata> = { text1: "123", text2: 456 };
let a2: UsingTypeB<NoMetadata> = { text1: "123", text2: 456 };

let b1: UsingTypeA<NonConflictingMetadata> = {
  text1: true,
  metadata: { available: true },
};
let b2: UsingTypeB<NonConflictingMetadata> = {
  text1: true,
  metadata: { available: true },
};

let c1: UsingTypeA<ConflictingMetadataDifferentType> = {
  text1: 1,
  metadata: { authorization: "bearer" },  // Type string is not assignable to never
};
let c2: UsingTypeB<ConflictingMetadataDifferentType> = {
  text1: 1,
  metadata: { authorization: "bearer" },  // Type string is not assignable to never
};

let d1: UsingTypeA<ConflictingMetadataSameType> = {
  text1: 1,
  metadata: { authorization: "bearer" },
};
let d2: UsingTypeB<ConflictingMetadataSameType> = {
  text1: 1,
  metadata: { authorization: "bearer" },
};

let e11: UsingTypeA<CombinedType> = { type: true, field: "123" };  // Fails because type narrowing fails
let e12: UsingTypeA<CombinedType> = { type: false, value: 123 };  // Fails because type narrowing fails
let e21: UsingTypeB<CombinedType> = { type: true, field: "123" };
let e22: UsingTypeB<CombinedType> = { type: false, value: 123 };

The results of all the tests are the same (ie, *1 and *2 match in their outputs) except for e* tests. As we can see here, the behaviour is basically similar except for having complex unified types, which is when UsingTypeB (which is the new implementation) shines.

This is the full test file for reference
test.ts

[aryanjassal]

I ran a local update on js-rpc and tried building it. It built just fine. I tried to modify the types.d.ts in the distributable in both Polykey and Polykey CLI, and they both also built just fine.

Side note, however, is that Brian's IDE doesn't use the simple TS compiler for warnings/errors. As such, some errors are able to fall through. These errors also don't block the building, so the package is built and run just fine, but the errors does exist.

polykey • feature-multiple-vault-resource • nix-shell-env
-> tsc
tests/audit/utils.test.ts:50:48 - error TS2345: Argument of type 'string[][]' is not assignable to parameter of type 'TopicSubPath[]'.
  Type 'string[]' is not assignable to type 'TopicSubPath'.
    Type 'string[]' is not assignable to type '["discovery", "checkRediscovery"]'.
      Target requires 2 element(s) but source may have fewer.

50     const filtered = auditUtils.filterSubPaths(data).map((v) => v.join('.'));
                                                  ~~~~

tests/client/handlers/nodes.test.ts:828:11 - error TS2345: Argument of type '{ nodeGraph: NodeGraph; keyRing: KeyRing; }' is not assignable to parameter of type '{ nodeGraph: NodeGraph; }'.
  Object literal may only specify known properties, and 'keyRing' does not exist in type '{ nodeGraph: NodeGraph; }'.

828           keyRing,
              ~~~~~~~

tests/git/utils.ts:128:85 - error TS2339: Property 'type' does not exist on type 'never'.

128         `data.type must be "want", "have", "SEPARATOR", "done", "none", got "${data.type}"`,
                                                                                        ~~~~


Found 3 errors in 3 files.

Errors  Files
     1  tests/audit/utils.test.ts:50
     1  tests/client/handlers/nodes.test.ts:828
     1  tests/git/utils.ts:128

polykey • feature-multiple-vault-resource • nix-shell-env
-> npm run build

> [email protected] build
> shx rm -rf ./dist && tsc -p ./tsconfig.build.json


> [email protected] postbuild
> shx cp -f src/status/*.json dist/status/


polykey • feature-multiple-vault-resource • nix-shell-env
-> 

When I open the relevant files, I do see the errors just fine. This can be replicated on both Neovim and VS Code. We might need another step in our development scripts which runs tsc as a step to confirm no warnings or errors exist.

@tegefaulkes @CMCDragonkai

[aryanjassal]

Oh shoot, I completely forgot js-rpc does not have automatic publishing for its CI. @tegefaulkes you need to do a manual release on Monday for this.

[CMCDragonkai]

Why doesn't it? Get it sorted!

[aryanjassal]

Oh shoot, I completely forgot js-rpc does not have automatic publishing for its CI. @tegefaulkes you need to do a manual release on Monday for this.

MatrixAI/js-rpc#70

The CI isn't working properly yet, as some tests were failing, so they were commented out. This needs to be resolved on high priority. Moreover, the CI isn't automatically publishing a release. This needs to be investigated, @brynblack.

[CMCDragonkai]

Status of this? If @brynblack is not available, see if you can do the work to get it published.

[aryanjassal]

Status of this? If @brynblack is not available, see if you can do the work to get it published.

Well, the js-rpc was merged and released by Brian, so this isn't an issue anymore. I can't do a release as I don't have the relevant secrets or knowledge of making a release.