Distributed Build Options

[Cjewett]

Hi,

An issue we have at our company is the lack of cloud backed distributed build caching is brutal. Build times for our monorepo continues to increase and while we've introduced solutions like reading the git history to determine which projects should be built and tested it only gets you so far. I figured this is a problem many others have hit so it might make sense to ask around first.

I've been looking over multiple ways to tackle the problem and they all have their pros and cons.

Integrate as an MSBuild Task: Difficult to test and maintain from our perspective but you're doing it all as close to the source as possible which seems like a huge plus.

nx-dotnet: Looks promising and we've tested the version specific to web frameworks. It was impressive but tickets like this are worrisome since we like to leverage the dotnet ecosystem to its fullest with custom Prop and Target files, etc.

Docker: Write efficient Dockerfiles and leverage tools like Buildah. Has its own downsides trying to maintain efficient Dockerfiles (we've been floating the idea of automating the creation), having to call many different docker build commands efficiently, etc.

BuildXL: Seems like Microsoft is using this internally but the lack of engagement and Docs to support non-Microsoft employees makes it seem like it's not meant to be used publicly.

I also see references to CloudBuild and /graph related commands but haven't messed around with it and don't know if that is something coming down the pipeline. https://github.com/microsoft/MSBuildPrediction/blob/main/documentation/design.md#qa That section hasn't been updated in 3 years but mentions CloudBuild at the bottom.

Ultimately all of these other solutions exist because msbuild doesn't provide its own cloud backed distributed cache solution, so I'm curious if this has been something that has been brought up? If so, what was the outcome of those talks? If we could just toggle on a property like we do for CPVM and then assign some domain to a storage provider and magic happened behind the scenes that would be amazing.

[JanKrivanek]

There is currently support for pluging-in 3rd pty cache providers. The updated documentation is yet to be merged, but you can check it here: #9329

FYI @dfederm - as there is an external ask for the feature. Can we maybe point to any off-the-shelve cache plugin to be used externally?

[Cjewett]

Thanks for the update @JanKrivanek ! I actually have a thread with some folks including @dfederm in an Issue because I tried using the cache plugin functionality locally and ran into issues. I'll update that thread with the issues I ran into because I'm coming back to this now that the holidays are over.

Relevant issue: #9441

[dfederm]

The flagship/reference plugins are here: https://github.com/microsoft/MSBuildCache. The intent is that they're out-of- box plugins, as well as a library which you can use to write your own plugin with a different storage backend.

@Cjewett I'd love to work with you on getting caching integrated into your builds. Feel free to reach out over email via my GitHub handle [at] microsoft.com

[Cjewett]

This looks absolutely awesome! Didn't realize a repository with defaults was already spun up but was hoping for it. I'll take a look and experiment with some simple repositories and reach out. Thanks!