Support remote persistent workers

[meroton-benjamin]

This ticket gathers information about remote persistent workers and their requirements for a buildbarn implementation.

Bazel has a concept of persistent workers since 2015 these workers help alleviate the issue that certain compilers such as the java compiler has a long startup and warmup time.

As an optimization, Bazel would wrap the javac instance with a small program that takes WorkRequests to feed the compiler and returns WorkResponses describing the next item to compile keeping the jit warm and fast.

Other languages such as Haskell suffers of the same issue and has therefore also implemented such wrapper programs. Previously this feature was only available for local execution but recently Bazel introduced --experimental_remote_mark_tool_inputs flag for annotating remote actions so that they too can recognize a remote persistent work request according to this design document.

An implementation in Buildbarn could be similar to a hardware runner. The scheduler would be configured to utilize worker stickiness in order to reuse persistent workers. And a custom worker which is persistent worker aware would have custom logic for:

• Startup - Start the tool in a tool workspace if necessary
• Action invocation - Feed the tool with WorkRequests through stdin and use WorkResponse to create action outputs
• Cleanup - Shut down the tool and discard the tool workspace

The exact details would be part of the implementation.

Todo:

• Key extraction for tool key in scheduler
• Configurable stickiness in scheduler for specific keys
• Persistent worker aware worker/runner that invokes actions via stdin
• Cleanup logic for stale persistent workers
• Upload a specification to the Remote Execution API project

[moroten]

See also the Bazel persistent workers presentation from BazelCon 2020.