Build Performance Results

This is a comparison between Visual Studio 15.8.4 and what was shipped in 15.9 Preview 3.

Our core focus here is improving the developer loop, in cases such as:

• You have a currently built/deployed Xamarin.Forms app.
• You change one line of XAML in your NetStandard Xamarin.Forms project.
• You build/deploy again.

Along the way if we happened to find something to improve initial builds, that would be a bonus.

Projects Used

• Hello Forms: File | New Xamarin.Forms project with a NetStandard library. Using project template from 15.8.4.
• SmartHotel360: A sample Xamarin.Forms app with 99 NuGet packages, using a fork so we have updated Xamarin.Forms to 3.0.x.
• Xamarin.Forms Control Gallery: The "control gallery" sample project has around 7 Xamarin.Android library projects and includes building Xamarin.Forms itself.

To see specific details about changes made see what was shipped in 15.9 P3 here.

MSBuild Targets Timed

• Build or build
• SignAndroidPackage or package, this creates the Android APK file and signs it
• Install or deploy, this deploys the APK to the attached device

I performed all of these timings with a USB-connected Pixel 2 device.

From the logs, what are first/second/third builds?

• First: simulates a build after an initial checkout, after nuget restore.
• Second: a build with no changes
• Third: modify a XAML file, build

Improvements to Incremental Build Times

App	15.8	15.8 Log	15.9	15.9 Log
Hello Forms (no changes)	00:02.99	binlog	00:02.75	binlog
Hello Forms (XAML change)	00:08.03	binlog	00:06.63	binlog
SmartHotel360 (no changes)	00:03.74	binlog	00:03.54	binlog
SmartHotel360 (XAML change)	00:10.62	binlog	00:08.34	binlog
Xamarin.Forms (no changes)	00:16.48	binlog	00:11.20	binlog
Xamarin.Forms (XAML change)	00:52.53	binlog	00:35.34	binlog

Improvements to Deploy Times

App	15.8	15.8 Log	15.9	15.9 Log
Hello Forms (first)	00:15.22	binlog	00:12.90	binlog
Hello Forms (no changes)	00:03.07	binlog	00:02.83	binlog
Hello Forms (XAML change)	00:07.06	binlog	00:06.63	binlog
SmartHotel360 (first)	00:17.39	binlog	00:15.65	binlog
SmartHotel360 (no changes)	00:03.88	binlog	00:03.69	binlog
SmartHotel360 (XAML change)	00:09.03	binlog	00:05.10	binlog
Xamarin.Forms (first)	00:19.30	binlog	00:13.76	binlog
Xamarin.Forms (no changes)	00:16.82	binlog	00:11.90	binlog
Xamarin.Forms (XAML change)	00:21.21	binlog	00:15.07	binlog

Initial Build Times

We regressed very slightly on the initial build time here. Reviewing the build logs, I believe this was due to #deletebinobj related fixes. These are fixes for "correctness" of our build, where developers could get a strange error that is resolved by deleting their bin and obj directories. I believe these fixes have added time to the build, but it is important to be correct before optimizing things. We also improved the first deployment times, so there is certainly some tradeoff here.

App	15.8	15.8 Log	15.9	15.9 Log
Hello Forms (build)	00:35.93	binlog	00:37.33	binlog
Hello Forms (package)	00:07.74	binlog	00:07.33	binlog
Hello Forms (deploy)	00:15.22	binlog	00:12.90	binlog
SmartHotel360 (build)	01:04.20	binlog	01:05.90	binlog
SmartHotel360 (package)	00:10.65	binlog	00:10.53	binlog
SmartHotel360 (deploy)	00:17.39	binlog	00:15.65	binlog
Xamarin.Forms (build)	02:51.11	binlog	02:59.56	binlog
Xamarin.Forms (package)	00:18.70	binlog	00:18.19	binlog
Xamarin.Forms (deploy)	00:19.30	binlog	00:13.76	binlog

Other Notes

A build with no changes is generally 2-3 seconds, which is certainly improved from what it was 6 months ago. We are still working on this, as there are still a few MSBuild targets running on every build that could be skipped (or improved).

Since these timings run build, package, and deploy in three distinct steps, this overhead is present in all three builds.

There are some additional changes in master that will improve initial build times. These are still in progress, but should be coming soon! I'll update the numbers as they are available.

Lessons learned

In PR #2105, I was seeing a massive improvement to first builds of "Hello World" apps.

I realized since I was timing builds with only the OSS xamarin-android, I was basically timing builds without Fast Deployment. This feature sideloads assemblies (and is enabled by proprietary Xamarin.Android code), and so .NET assemblies are not embedded in the APK by default.

This meant that the PCL facade assemblies were being embedded in the APK during my timing. That explains the huge difference in build times I measured, since making Java.Interop a non-PCL enabled OSS xamarin-android to not embed as many assemblies.

However, this changes was still great, since it improved initial deployment times, as seen in the times above. We have also found developers common disable Fast Deployment for various reasons (they hit a bug?), so build times will be improved there.

Conclusion

Some notes on what we should always do when timing builds.

• Always start with a clean working copy! git clean -dxf
• Time Release builds of Xamarin.Android.Build.Tasks.dll.
• Time changes within the context of the proprietary Xamarin.Android code.
• If possible, potentially even deploy the changes to the system Xamarin.Android install.
• Time more than one project. Record everyting. Put it in the commit message!

It is possible we should look into some kind of CI setup / scripting to make these easier.