How C/C++ Debugging Works on Android

How debugging of C/C++ code works on Android? Nothing special actually. Gdb itself has a feature for remote debugging.

Note: For this article I am using Android NDK, r5. The behavior of ndk-build and ndk-gdb commands can be different in other versions.

Basic setup

You simply run gdbserver on the device and let it attach to some process. Gdbserver acts as a remote debugger which is commanded by gdb itself. This step is pretty easy and is done by some variant of

gdbserver :5039 --attach 123

where 5039 is a TCP port number to which gdb will connect and 123 is a PID of process we want to debug.

The next step is to run gdb client and connect it to the gdbserver on the device. You need to start gdb with command

gdb app_process

where app_process is Android binary which you need to copy from Android device to your PC. This binary is some kind of loader and lies in /system/bin/app_process file on the device (or in the emulator).

After you start gdb you connect it with the gdbserver by this command (ran from gdb shell)

target remote :5039

Because you are usually running Android device connected through USB and not normal TCP/IP network (or you are running Android emulator) you have to set up TCP port forwarding in advance. This is done by command

adb forward tcp:5039 tcp:5039

which you have to execute before starting gdb. This command will handle all connection requests to port 5039 on PC to the device on the same port.

And this is all.

Debug symbols

You typically compile your .so library with debug symbols. But it makes good sense to strip those symbols off before uploading your binary to the Android device. Binary will be smaller (which means faster installation time of the apk package), will occupy less memory and will run faster. So you upload stripped version of your libraries to the Android device and keep unstripped versions on your PC.

To tell gdb the path to the unstripped libraries run this command from gdb shell

set solib-search-path obj/local/armeabi

You can enter more paths and separate them by colon (:) character.

There is a difference between debug symbols and code optimization. Code optimization allows changing the code flow or removing some dead-code and unused variables. While debug symbols bind machine code together with appropriate lines in the source code. Usually you build a binary with debug symbols and no code optimization for debugging and another binary without debug symbols and with some code optimization for release. But you can use debug symbols together with code optimization. In this case some variables will be missing and some lines of code will be probably jumped over but in general you can debug this way.

Android ndk-build command always produce binaries (your libnative.so files) with debug symbols and place them in obj/local/armeabi subdirectory of your project. Then it strips off debug symbols and place stripped version in the final .apk (and thus in the device). ndk-gdb script uses this set solib-search-path command to point gdb to the unstripped binary so you can see your position in the source code. You can find the set solib-search-path command in obj/local/armeabi/gdb.setup file.

If you have android:debuggable=”true” in your AndroidManifest.xml or if you run ndk-build with NDK_DEBUG=1 argument then ndk-build will produce binaries with no code optimization. Otherwise code optimization will be used.

Sources

Gdb needs to have access to your source files. Otherwise it cannot show you the source code as it is executed. You can configure directory with your source files by command

directory jni

You can enter more paths and separate them by space. Take a look into obj/local/armeabi/gdb.setup file for example.

How ndk-gdb and friends works

If you are using Android NDK, r5 and Android 2.3 device you can use ndk-gdb script which does everything for you.

If you have android:debuggable=”true” in your AndroidManifest.xml then ndk-build will add the gdbserver into your .apk package. This gdbserver will be started by ndk-gdb and also all the other steps will be set up by this script.

Important caveat

Because gdbserver is attached to the already running process (as opposed to situation where process would be started by gdbserver) it can miss some code execution which take place soon after the application start. There is no easy solution to this.

I usually write some endless while loop and then change the control variable after gdb is fully started. For example

int i = 0
while (!i) {
  a++;
}

After ndk-gdb starts gdb session I can set breakpoints appropriately and change value of i variable by command

set var i=1

and then continue in application execution by gdb command c.

Multithread debug problem

It is well-known problem that ndk-gdb shipped with Android NDK, r4b used on Android 2.2 device was only able to hit the breakpoint on the main thread. If you set breakpoint on any other thread gdb session will crash.

Every time new thread is created libc will call _thread_created_hook() function (which has empty body). The purpose of this function is that gdb will set breakpoint on this function and thus will know that new thread was created. Gdb will then keep database of existing threads. Because of this gdb needs to have access to the libc.so file to figure out address of the _thread_created_hook() function.

Android NDK, r5 contains ndk-gdb which copies the file libc.so from the device (or emulator) and stores it in obj/local/armeabi subdirectory of your project. Also gdbserver included in this ndk is compiled with libthread_db support. Both those two things are missing in Android NDK, r4b.

Links

Description of multithreading debug bug.
http://code.google.com/p/android/issues/detail?id=9713

Android NDK, r5
http://dl.google.com/android/ndk/android-ndk-r5-linux-x86.tar.bz2

Android ndk main page
http://developer.android.com/sdk/ndk/index.html