Turn on test sharding for k8s and docker-build #1058
Conversation
| run: 'for f in *.tar.gz; do tar -zxf "$f" -C "${f%.tar.gz}"; done', | ||
| "working-directory": "${{ github.workspace}}/sdk", |
There was a problem hiding this comment.
This relaxes the logic to unpack any SDK tarballs that happen to be present, not just a language-specific tarball.
| if (sharded) { | ||
| return ["nodejs", "go", "dotnet", "python", "java"].map((language) => { | ||
| return { | ||
| name: `Download ${language} SDK`, | ||
| uses: action.downloadArtifact, | ||
| with: { | ||
| name: `${language}-sdk.tar.gz`, | ||
| path: "${{ github.workspace}}/sdk/", | ||
| }, | ||
| }; | ||
| }); | ||
| } |
There was a problem hiding this comment.
Each shard now needs every SDK.
| @@ -314,7 +314,7 @@ export function InstallPythonDeps(): Step { | |||
| export function InstallSDKDeps(): Step { | |||
| return { | |||
| name: "Install dependencies", | |||
| run: "make install_${{ matrix.language}}_sdk", | |||
| run: "make install_sdks", | |||
There was a problem hiding this comment.
I think this is supported by our native providers, but I could also add a conditional here to be safe.
There was a problem hiding this comment.
I don't think neither pulumi-kubernetes nor pulumi-docker-build have a install_sdks entrypoint currently?
| @@ -314,7 +314,7 @@ export function InstallPythonDeps(): Step { | |||
| export function InstallSDKDeps(): Step { | |||
| return { | |||
| name: "Install dependencies", | |||
| run: "make install_${{ matrix.language}}_sdk", | |||
| run: "make install_sdks", | |||
There was a problem hiding this comment.
I don't think neither pulumi-kubernetes nor pulumi-docker-build have a install_sdks entrypoint currently?
| name: "Run tests", | ||
| run: testCmd, | ||
| }; | ||
| const shardCmd = `echo go test -v -count=1 -cover -timeout 2h -parallel 4${shortMode} "$(go run github.com/blampe/shard@b251cf8da6a83022628496125f285095772ebe86 --total \${{ strategy.job-total }} --index \${{ strategy.job-index }})" > test-command`; |
There was a problem hiding this comment.
Do we plan on movinggithub.com/blampe/shard to a Pulumi controlled repo?
There was a problem hiding this comment.
Perhaps this should be an inline script so it's versioned per-provider via ci-mgmt? If not, then we should pin it.
There was a problem hiding this comment.
Following up on this following the work on the generic provider template (#1140) and to clear this off my review backlog 😅 ...
I think we should hold off on adding the sharding here and instead focus on the generic template and moving the native providers to that.
Just to mirror the comments from elsewhere:
- In order to get this live, we should create a version of the shard program in the pulumi organisation.
- How are we planning to upgrade the version of the shard tool that we depend on?
- In the implementation of the shard tool - is there any way to leverage Go's own test discovery process so we're sure the tool behaves consistently with running
go test? - Related: I think there's potentially an issue here around when there's modules within modules.
This turns on test sharding for the docker-build and p-k providers.
The actual logic for how we discover tests and assign them is here, along with some test cases if there are correctness concerns.
Results for p-k:
Total billable time increased by 35m, which roughly lines up with the fixed setup time we use on the 5 additional shards.
In terms of onboarding, if a provider wants to use this they should:
make install_sdkstarget exists, because each shard now needs all the SDKs.go run github.com/blampe/shard@latest --total 1 --index 0will emit a warning for any tests with name collisions. Due to the way we're invokinggo test -run (names...) packages..., it's possible for a test to get re-run on multiple shards if its name isn't unique, but this isn't the end of the world.Refs #676