dev scriptworkers sometimes run stale versions of images

[bhearsum]

I sometimes notice that a dev version of a scriptworker will encounter errors that are not possible if they are running the expected version of the code. Sure enough, doing an additional deployment (with no added code changes) will fix the issue, which leads me to believe that there are circumstances that can cause us to be running outdated versions of images.

I've never seen this in production, so I'm suspicious that there's something different between the two either here, in cloudops-infra, or perhaps even in k8s-autoscale.

I've noticed this a couple of times this week, and @gabrielBusta reports that he's seen it as well.

[bhearsum]

Via @jcristau:

remember chatting about it with jbuck, we figured something might be caching the image, and because we use the dev tag rather than a versioned one, could get an old image from e.g. docker hub

(And presumably this doesn't happen with prod because we rarely (if ever) deploy in rapid succession there.)