Race condition after first deploy for new app with heroku run

[schneems]

From time to time calls to app.run("ls") will return with an empty result. This is somewhat of a known behavior but I've never been to isolate or reproduce the cause leading me to believe it had to do with some kind of race condition.

@dzuelke proposed that this would happen if heroku run ls was being performed against an app without a release yet in which case the /app directory would be empty.

He suggests that there's a race condition between when git push heroku returns and when a new slug is a available for heroku run. Essentially:

git push heroku && heroku run <cmd>

Does not guarantee that the heroku run will execute against whatever was generated via git push heroku. In talking with the #lifecycle team they confirmed this suspicion. https://heroku.slack.com/archives/CA5BNBLDR/p1597250243189200

When you deploy via git, the git push is picked up by endosome, this then translates to a build via vacuole and codon. Once the build is done, the success is recorded via API, but the actual process of booting a new dyno is not done syncronously, it's performed by enqueuing an action where #runtime will pick it up and then boot the new release.

The heroku release phase suffers from this same race condition. The release phase is implemented essentially as a heroku run against a specific release that is not "current". It does this by manually making an API request:

• https://github.com/heroku/api/blob/3b01662e7aca7b01f549d28f050e77e263cab8fc/spec/endpoints/api_v3/dynos_spec.rb#L133-L148

You can also do this manually via the CLI with the api plugin:

$ heroku api POST /apps/stormy-scrubland-58269/dynos --body='{"command": "ls", "attach": true, "release": 8}' --version=3.dyno-release

When a release is manually specified like this then a status of 409 will indicate that the release exists, but it is not ready for use yet. The request can then be retried while the race condition resolves.

This behavior exists in regular heroku run calls as well: https://github.com/heroku/cli/blob/b74b0e6fd838622b01f3ff3d7e016cab5296d8e9/packages/run/src/lib/dyno.ts#L128-L136. However this retry behavior does not solve our race condition because this endpoint uses the latest release instead of a specific release.

When you deploy an app multiple releases are created before the actual deploy:

$ heroku releases
=== lit-hamlet-40699 Releases - Current: v2
v2  Enable Logplex   [email protected]  2020/08/12 13:51:36 -0500 (~ 5s ago)
v1  Initial release  [email protected]  2020/08/12 13:51:36 -0500 (~ 5s ago)

That's why the race condition exists and how the release phase works around it.

Opportunities

We have the opportunity to fix this race condition in the same way that the release phase does this by manually specifying a release version target when we heroku run and retrying if we get a 409 response.

This was deeply investigated by @dzuelke, @Malax, and me. We determined that we want to move forwards by creating a heroku CLI plugin that allows for specifying a release version such as heroku run:release ls --release=5

[Malax]

We determined that we want to move forwards by creating a heroku CLI plugin that allows for specifying a release version such as heroku run:release ls --release=5

The preliminary version of that plugin can be found here: https://github.com/Malax/heroku-cli-run-release