Skip to content

0000

Jump to bottom
Brendan Zagaeski edited this page Jun 18, 2019 · 2 revisions

Complications debugging native libraries in arm Xamarin apps on arm64 devices using the C++ debugger in Visual Studio

Update June 2019: The problem I originally saw with starting the 32-bit gdbserver looks like it might have been specific to the arm64-v8a Google APIs emulator image. On a hardware Google Pixel 3 device, the 32-bit gdbserver runs successfully and debugs 32-bit native code as expected, so hard-coding MIEngine to use TargetArchitecture.ARM allowed me to debug 32-bit apps in Visual Studio.

Original information from September 2018:

It turns out that the way MIEngine probes for the gdbserver location is only the first of several complications. Long story short, if the native code must remain 32-bit, then I would recommend debugging it on a 32-bit device. (Once it is debugged, it can still be published to a 64-bit device for end-user use.) Based on my tests so far, it is not clear that gdbserver itself can even be made to work correctly to debug armeabi-v7a apps on arm64-v8a devices (see the last section below). If I were to take a guess about what's going wrong with gdbserver itself, I would say that maybe the 64-bit gdbserver cannot find or load the 32-bit symbol information as expected. And unfortunately it seems that arm64-v8a devices do not allow running the 32-bit gdbserver.

To illustrate the various complications I found, here are some of tests I tried along with their results.

./lib/gdbserver gets killed on launch, but ./gdbserver runs successfully

I tried the following 2 commands in the adb shell command prompt:

run-as com.example.csharpandroidapp1 /data/data/com.example.csharpandroidapp1/lib/gdbserver

Result: Killed is shown on the console output. That is, the gdbserver process gets killed (receives SIGKILL) and isn't allowed to run.

run-as com.example.csharpandroidapp1 /data/data/com.example.csharpandroidapp1/gdbserver

Result: gdbserver runs successfully.

I later figured out that this difference in behavior was due to the fact that ./gdbserver was the arm64 version of gdbserver while ./lib/gdbserver was the arm version.

Forcing use of the "good" ./gdbserver on device is not sufficient because the .gdb directory on the development machine is still set up with app_process64

I added a target at the bottom of my C# .csproj file (just before the closing </Project>) to prevent gdbserver from being included in the .apk archive at all:

<Target Name="NoGdbServerInApk" BeforeTargets="_CompileDex">
  <ItemGroup>
    <AndroidNativeLibrary Remove="@(NativeLibraryPaths)" Condition="'%(FileName)' == 'gdbserver'" />
  </ItemGroup>
</Target>

This meant that MIEngine would never find ./lib/gdbserver and would always use ./gdbserver.

Result: Unable to start debugging. Unexpected GDB output from command "-target-select remote :5039". Reply contains invalid hex digit 59

Additional messages from MIEngine log output:

warning: Selected architecture aarch64 is not compatible with reported target architecture arm
warning: Architecture rejected target-supplied description
Reply contains invalid hex digit 59

Explanation: The Visual Studio Tools for Xamarin extension is hard-coded to run adb shell getprop ro.product.cpu.abi on each launch and pass the result to MIEngine as _launchOptions.TargetArchitecture. Since the target device in this case is ARM64, that is what gets sent to MIEngine. So when MIEngine prepares the .gdb directory, it downloads the 64-bit files (like app_process64) from the device rather than the 32-bit files. This means that MIEngine tells gdb to run -file-exec-and-symbols C:\\CsharpAndroidApp1\\CsharpAndroidApp1\\bin\\Debug\\.gdb\\app_process64, which puts gdb in the incorrect aarch64 architecture mode. This in turn causes gdb to fail when it tries to connect to the remote gdbserver (attached to an arm process).

Hard-coding MIEngine to use TargetArchitecture.ARM doesn't quite work because then ./gdbserver fails like ./lib/gdbserver did

I tried hard-coding MIEngine to use TargetArchitecture.ARM by changing a line in the launch options setup code to this.TargetArchitecture = TargetArchitecture.ARM;.

Result: Unable to start debugging. GDBServer failed to start or attach to the target application.

Additional messages from the MIEngine log output:

MS_MIDebug: 2: (17324) ADB<-run-as com.example.csharpandroidapp1 /data/data/com.example.csharpandroidapp1/gdbserver +debug-socket --attach 7810; echo gdbserver exited with code $?
MS_MIDebug: 2: (17649) GDB SERVER: Killed \n
MS_MIDebug: 2: (17654) GDB SERVER: gdbserver exited with code 137\n

This is the same problem I had seen originally when running ./lib/gdbserver by hand, but now it was happening for the ./gdbserver too. I double-checked the run-as command by hand in adb shell and got the same "Killed" result.

gdbserver is always killed if it is a 32-bit executable

I was initially confused about why ./gdbserver was now getting killed when that path was working fine before. After trying a variety of things to check if I had messed up the permissions or device environment somehow, I eventually thought to double-check the contents of the gdbserver executable.

Result: The ./gdbserver file was now a 32-bit executable. When I went back to launching with the unmodified MIEngine, I discovered that in that case the deployed ./gdbserver was a 64-bit executable.

Explanation: Apparently on arm64-v8a devices it is required to use the 64-bit gdbserver regardless of whether the app being debugged is itself armeabi-v7a or arm64-v8a.

The 64-bit gdbserver returns level="-1" for some frames in arm apps on arm64 devices, and MIEngine does not expect a negative number there

Based on the earlier results, I now understood that I needed to mix together the 64-bit and 32-bit behaviors so that MIEngine would launch the 64-bit gdbserver on device but set up the .gdb directory with the 32-bit binaries. So I reverted back to an unmodified version of MIEngine and changed each occurrence of _launchOptions.TargetArchitecture in just the Launcher code to TargetArchitecture.ARM. (The exact changes needed might be different with a different revision of MIEngine.)

(Another approach would be to keep the original one-line change to the launch options setup code, remove the custom target from the .csproj so gdbserver would be included in the .apk, and force the .vcxproj to use the arm64 gdbserver by adding <GdbServerPath>C:\Microsoft\AndroidNDK64\android-ndk-r15c\prebuilt\android-arm64\gdbserver\gdbserver</GdbServerPath> to the <PropertyGroup Label="Globals"> section. This would mean that the ./lib/gdbserver binary would be 64-bit, so MIEngine would be able to launch it successfully and wouldn't fall back to the ./gdbserver version.)

Result: Unable to start debugging. Unrecognized format of field "level" in result: {level=-1,addr=0xea6078d4,func=??,args=[]}

Additional messages from the MIEngine log output:

MS_MIDebug: 1: (25130) <-1025-thread-info
MS_MIDebug: 1: (26896) ->1025^done,threads=[
    {id="1",target-id="Thread 8719.8719",name="csharpandroidapp1",frame={level="0",addr="0xea5d6418",func="??",args=[]},state="stopped",core="0"},
    {id="2",target-id="Thread 8719.8724",name="Jit thread pool",frame={level="0",addr="0xea5d6418",func="??",args=[]},state="stopped",core="0"},
    {id="3",target-id="Thread 8719.8725",name="Signal Catcher",frame={level="0",addr="0xea607b00",func="??",args=[]},state="stopped",core="0"},
    {id="4",target-id="Thread 8719.8726",name="JDWP",frame={level="0",addr="0xea608a14",func="??",args=[]},state="stopped",core="0"},
    {id="5",target-id="Thread 8719.8727",name="ReferenceQueueD",frame={level="0",addr="0xea5d6418",func="??",args=[]},state="stopped",core="0"},
    {id="6",target-id="Thread 8719.8728",name="FinalizerDaemon",frame={level="0",addr="0xea5d6418",func="??",args=[]},state="stopped",core="0"},
    {id="7",target-id="Thread 8719.8729",name="FinalizerWatchd",frame={level="0",addr="0xea5d6418",func="??",args=[]},state="stopped",core="0"},
    {id="8",target-id="Thread 8719.8730",name="HeapTaskDaemon",frame={level="0",addr="0xea5d6418",func="??",args=[]},state="stopped",core="0"},
    {id="9",target-id="Thread 8719.8731",name="Binder:8719_1",frame={level="-1",addr="0xea6078d4",func="??",args=[]},state="stopped",core="0"},
    {id="10",target-id="Thread 8719.8732",name="Binder:8719_2",frame={level="-1",addr="0xea6078d4",func="??",args=[]},state="stopped",core="0"}
], current-thread-id="1"

Explanation: For some reason, gdbserver returns level="-1" for certain frames in the result from the -thread-info command. I also tested this outside of Visual Studio using adb shell, gdbserver, and gdb by hand, and I got the same level="-1" result for those frames. I am not sure if this is an expected result or perhaps a bug in gdbserver, but in any case the logic in MIEngine that parses that output is not expecting a -1 in that field, so it fails.

Adjusting MIEngine to parse negative numbers as 0 for the level field doesn't fully solve the problem either

I changed the line that parses level as a uint to uint level = (uint)(Math.Max(frame.FindInt("level"), 0));. This allows MIEngine to attach to the app, and I could then pause and step through the disassembly successfully, but the debug symbols apparently did not work as expected so breakpoints in my .cpp files still didn't work.

Note also how the func fields in the output from -thread-info all show ?? for the function names. If I switch to an armeabi-v7a device or emulator and run the same test, the func fields all show function names at that step.

If I were to take a guess about what's going wrong with gdbserver itself, I would say that maybe the 64-bit gdbserver cannot find or load the 32-bit symbol information as expected.

Clone this wiki locally