-
Notifications
You must be signed in to change notification settings - Fork 532
Build Performance Results
This is a comparison between:
- Visual Studio 15.8.4, Xamarin.Android 9.0.x
- Visual Studio 15.9 Preview 3, Xamarin.Android 9.1.0.x - released as Visual Studio 2017 15.9.0 stable
- Visual Studio 16.0 Preview 2, Xamarin.Android 9.1.100.x - soon to be released as Visual Studio 2019* or the next version of Visual Studio "Dev16".
Our core focus here is improving the developer loop, in cases such as:
- You have a currently built/deployed Xamarin.Forms app.
- You change one line of XAML in your NetStandard Xamarin.Forms project.
- You build/deploy again.
Along the way if we happened to find something to improve initial builds, that would be a bonus.
- Hello Forms: File | New Xamarin.Forms project with a NetStandard library. Using project template from 15.8.4.
- SmartHotel360: A sample Xamarin.Forms app with 99 NuGet packages, using a fork so we have updated Xamarin.Forms to 3.0.x.
- Xamarin.Forms Control Gallery: The "control gallery" sample project has around 7 Xamarin.Android library projects and includes building Xamarin.Forms itself.
Specific details about changes mades made in each version of Visual Studio can be found here.
-
Build
or build -
SignAndroidPackage
or package, this creates the Android APK file and signs it -
Install
or deploy, this deploys the APK to the attached device
I performed all of these timings with a USB-connected Pixel 2 device.
From the logs, what are first/second/third builds?
- First: simulates a build after an initial checkout, after nuget restore.
- Second: a build with no changes.
- Third: modify a XAML file, build.
App | 15.8 | 15.8 Log | 15.9 | 15.9 Log | 16.0 | 16.0 Log |
---|---|---|---|---|---|---|
Hello Forms (no changes) | 00:02.99 | binlog | 00:02.75 | binlog | 00:02.60 | binlog |
Hello Forms (XAML change) | 00:08.03 | binlog | 00:06.63 | binlog | 00:04.45 | binlog |
SmartHotel360 (no changes) | 00:03.74 | binlog | 00:03.54 | binlog | 00:03.44 | binlog |
SmartHotel360 (XAML change) | 00:10.62 | binlog | 00:08.34 | binlog | 00:07.47 | binlog |
Xamarin.Forms (no changes) | 00:16.48 | binlog | 00:11.20 | binlog | 00:08.27 | binlog |
Xamarin.Forms (XAML change) | 00:52.53 | binlog | 00:35.34 | binlog | 00:27.92 | binlog |
App | 15.8 | 15.8 Log | 15.9 | 15.9 Log | 16.0 | 16.0 Log |
---|---|---|---|---|---|---|
Hello Forms (first) | 00:15.22 | binlog | 00:12.90 | binlog | 00:05.48 | binlog |
Hello Forms (no changes) | 00:03.07 | binlog | 00:02.83 | binlog | 00:02.64 | binlog |
Hello Forms (XAML change) | 00:07.06 | binlog | 00:06.63 | binlog | 00:03.41 | binlog |
SmartHotel360 (first) | 00:17.39 | binlog | 00:15.65 | binlog | 00:08.88 | binlog |
SmartHotel360 (no changes) | 00:03.88 | binlog | 00:03.69 | binlog | 00:03.44 | binlog |
SmartHotel360 (XAML change) | 00:09.03 | binlog | 00:05.10 | binlog | 00:04.44 | binlog |
Xamarin.Forms (first) | 00:19.30 | binlog | 00:13.76 | binlog | 00:10.92 | binlog |
Xamarin.Forms (no changes) | 00:16.82 | binlog | 00:11.90 | binlog | 00:08.32 | binlog |
Xamarin.Forms (XAML change) | 00:21.21 | binlog | 00:15.07 | binlog | 00:11.23 | binlog |
We regressed very slightly on the initial build time in 15.9, but improved significantly in 16.0.
Reviewing the build logs from 15.9, I believe this was due to
#deletebinobj
related fixes. These are fixes for "correctness" of
our build, where developers could get a strange error that is resolved
by deleting their bin
and obj
directories. I believe these fixes
have added time to the build, but it is important to be correct
before optimizing things. We also improved the first deployment
times, so there is certainly some tradeoff here.
App | 15.8 | 15.8 Log | 15.9 | 15.9 Log | 16.0 | 16.0 Log |
---|---|---|---|---|---|---|
Hello Forms (build) | 00:35.93 | binlog | 00:37.33 | binlog | 00:24.37 | binlog |
Hello Forms (package) | 00:07.74 | binlog | 00:07.33 | binlog | 00:06.62 | binlog |
Hello Forms (deploy) | 00:15.22 | binlog | 00:12.90 | binlog | 00:05.48 | binlog |
SmartHotel360 (build) | 01:04.20 | binlog | 01:05.90 | binlog | 00:50.13 | binlog |
SmartHotel360 (package) | 00:10.65 | binlog | 00:10.53 | binlog | 00:10.81 | binlog |
SmartHotel360 (deploy) | 00:17.39 | binlog | 00:15.65 | binlog | 00:08.88 | binlog |
Xamarin.Forms (build) | 02:51.11 | binlog | 02:59.56 | binlog | 01:55.04 | binlog |
Xamarin.Forms (package) | 00:18.70 | binlog | 00:18.19 | binlog | 00:13.43 | binlog |
Xamarin.Forms (deploy) | 00:19.30 | binlog | 00:13.76 | binlog | 00:10.92 | binlog |
Xamarin.Android has support starting in 16.0 Preview 2 for Google's
new d8 and r8 dex compiler. In addition to supporting the
feature set of d8, in some cases the _CompileToDalvik
MSBuild target
will have a 15-20% speed increase. This will add up to better initial
build times.
The following times were using Xamarin.Android 9.1.99, with initial D8/R8 support:
App | Before | Before Log | d8 / r8 | d8 / r8 Log |
---|---|---|---|---|
Hello Forms (debug / default) | 00:27.81 | binlog | 00:23.49 | binlog |
Hello Forms (release / proguard) | 00:33.38 | binlog | 00:32.61 | binlog |
SmartHotel360 (debug / default) | 00:53.30 | binlog | 00:48.76 | binlog |
SmartHotel360 (release / proguard) | 01:06.06 | binlog | 01:03.79 | binlog |
Xamarin.Forms (debug / default) | 02:04.28 | binlog | 01:57.71 | binlog |
Xamarin.Forms (release / proguard) | 02:24.06 | binlog | 02:40.94 | binlog |
Xamarin.Android has had initial support for aapt2
starting in 15.9.
The prospect is that using aapt2
will make incremental builds
involving AndroidResource
files much faster. This is a bigger
benefit to "classic" Xamarin.Android apps, since they heavily rely on
AndroidResource
files. Xamarin.Forms apps will still see a benefit
when AndroidResource
files are modified in the Xamarin.Android head
project.
In order to test this, I modified my scripts slightly to do the following:
- Time
SignAndroidPackage
, soBuild
and the APK creation happen at the same time. - Incremental builds modify an
AndroidResource
file, instead of C# code/XAML.
Results from 16.0 Preview 2:
App | Aapt | Aapt Log | Aapt2 | Aapt2 Log |
---|---|---|---|---|
Xamarin.Forms (initial) | 02:16.04 | binlog | 02:04.67 | binlog |
Xamarin.Forms (no changes) | 00:10.76 | binlog | 00:10.65 | binlog |
Xamarin.Forms (XML change) | 00:40.98 | binlog | 00:38.42 | binlog |
NOTE: Further exploration is needed here, since all the projects tested in this document are Xamarin.Forms projects.
A build with no changes is generally 2-3 seconds, which is certainly improved from what it was in 15.7. We are still working on this, as there are still a few MSBuild targets running on every build that could be skipped (or improved).
Since these timings run build, package, and deploy in three distinct steps, this overhead is present in all three builds.
In PR #2105, I was seeing a massive improvement to first builds of "Hello World" apps.
I realized since I was timing builds with only the OSS
xamarin-android, I was basically timing builds without Fast Deployment
. This feature sideloads assemblies (and is enabled by
proprietary Xamarin.Android code), and so .NET assemblies are not
embedded in the APK by default.
This meant that the PCL facade assemblies were being embedded in the APK during my timing. That explains the huge difference in build times I measured, since making Java.Interop a non-PCL enabled OSS xamarin-android to not embed as many assemblies.
However, this changes was still great, since it improved initial
deployment times, as seen in the times above. We have also found
developers common disable Fast Deployment
for various reasons (they
hit a bug?), so build times will be improved there.
Some notes on what we should always do when timing builds.
- Always start with a clean working copy!
git clean -dxf
- Time Release builds of
Xamarin.Android.Build.Tasks.dll
. - Time changes within the context of the proprietary Xamarin.Android code.
- If possible, potentially even deploy the changes to the system Xamarin.Android install.
- Time more than one project. Record everyting. Put it in the commit message!
It is possible we should look into some kind of CI setup / scripting to make these easier.
- APK Tests on the Hyper V Emulator
- Design Time Build System
- Profile MSBuild Tasks
- Diagnose Fast Deployment Issues
- Preview layout XML files with Android Studio
- Documentation