In Android 2.3 Google ships fixed setup so you are now finally able to debug multithread application.

In fact multithread debugging is possible at least from Android 1.5. But you have to set up the environment yourself. Due to permissions limitation it is only possible on rooted devices. But it is possible on emulator as emulator is always rooted!

Here is a small shell script which does everything for you. It will start gdbserver on the device, gdb on PC and connect them together. It will also set up paths correctly so gdb can find its unstripped libraries on PC while having stripped libraries on the device.

Run this script from your project root directory. You have to modify APP_PACKAGE and ANDROID_TARGET variables to match your project settings. You also need to have you project compiled in debug mode. This can be done by running ndk-build NDK_DEBUG=1 or by setting android:debuggable=”true” in AndroidManifest.xml and compiling normally.

#!/bin/bash
# Martin Hejna (c) 2011
# martin.hejna@gmail.com
# CC-by


## Comment out for quiet run
set -x

# Set those variables to match your application
[ "$APP_PACKAGE" ] || APP_PACKAGE="com.example"
[ "$ANDROID_TARGET" ] || ANDROID_TARGET="android-9"

# Set installation paths of SDK and NDK
[ "$ANDROID_SDK" ] || ANDROID_SDK="/home/martin/Android/android-sdk-linux_86"
[ "$ANDROID_NDK" ] || ANDROID_NDK="/home/martin/Android/android-ndk-r5"

# adb and gdb executable paths
ADB="$ANDROID_SDK"/platform-tools/adb
GDB=$ANDROID_NDK/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin/arm-linux-androideabi-gdb

# Check we are in application root path (the one with AndroidManifest.xml in it)
APP_ROOT=`dirname $0`
if [ ! -f $APP_ROOT/AndroidManifest.xml ]
then
    echo This script must be run from package root directory
    echo This is the directory with AndroidManifest.xml in it
    exit 1
fi

# Path to unstripped versions of native .so libraries
UNSTRIPPED_SO_LIB="$APP_ROOT/obj/local/armeabi"
APP_PROCESS=$UNSTRIPPED_SO_LIB/app_process
APP_LIBC_SO=$UNSTRIPPED_SO_LIB/libc.so
if [ ! -d $UNSTRIPPED_SO_LIB ]; then
    echo Error: Directory $UNSTRIPPED_SO_LIB does not exist
    exit 1
fi

# Use cgdb if available
if which cgdb > /dev/null
then GDB_CMD="cgdb -d $GDB --"
else GDB_CMD=$GDB
fi

# We use same port number on pc and on device
DEBUG_PORT=5039

# File with gdb setup instructions (executed when gdb starts)
[ "$GDB_SETUP" ] || GDB_SETUP="/tmp/gdb.setup"


# Generate setup file for gdb with appropriate paths to symbols
function gdb_setup()
{
cat <<EOF >$GDB_SETUP
  set solib-search-path $UNSTRIPPED_SO_LIB:$ANDROID_NDK/platforms/$ANDROID_TARGET/arch-arm/usr/lib
  directory $ANDROID_NDK/platforms/$ANDROID_TARGET/arch-arm/usr/include $APP_ROOT/jni $ANDROID_NDK/sources/cxx-stl/system
  ## set solib-absolute-prefix $ANDROID_NDK/platforms/$ANDROID_TARGET/arch-arm/usr/lib

  file $APP_PROCESS
  target remote :$DEBUG_PORT

  # Probably not needed. Uncomment if you have problems with watchdogs
  ## set variable gDvm.nativeDebuggerActive=true
EOF
}

# Get pid of the process (first one if there are more of them)
function get_pid_on_device()
{
    $ADB shell ps | grep "$@" | awk '{print $2; exit}'
}


#############################################################################
## Main entry
############################################################################


# Restart adb server. Uncomment if you have problems with adb being stuck
## $ADB kill-server

# Get the app_process and libc.so from the device
# Always download and overwrite those files to be sure we have correct ones
$ADB pull /system/bin/app_process $APP_PROCESS
$ADB pull /system/lib/libc.so $APP_LIBC_SO

# Set up port forwarding for gdb and gdbserver
if $ADB forward tcp:$DEBUG_PORT tcp:$DEBUG_PORT
then :
else
    echo Failed to setup tcp forwarding for port $DEBUG_PORT
    echo Is the device running?
    exit 1
fi

# Get pid of the process
APP_PID=`get_pid_on_device "$APP_PACKAGE"`
if [ -z $APP_PID ]
then
    echo "$APP_PACKAGE is not running"
    exit 1
fi

# Kill any prior gdbserver
GDBSERV_PID=`get_pid_on_device gdbserver`
if [ -n "$GDBSERV_PID" ]
then 
    $ADB shell kill $GDBSERV_PID
    sleep 1
fi

# Start gdbserver and attach it to our application
$ADB shell gdbserver :$DEBUG_PORT --attach $APP_PID &
sleep 2

# Invoke gdb and feed it with setup commands to set up path to libraries
gdb_setup
$GDB_CMD -x $GDB_SETUP
rm -f $GDB_SETUP

Have fun!

Note: For this article I am using Android NDK, r5. The behavior of ndk-build and ndk-gdb commands can be different in other versions.

Basic setup

You simply run gdbserver on the device and let it attach to some process. Gdbserver acts as a remote debugger which is commanded by gdb itself. This step is pretty easy and is done by some variant of

gdbserver :5039 --attach 123

where 5039 is a TCP port number to which gdb will connect and 123 is a PID of process we want to debug.

The next step is to run gdb client and connect it to the gdbserver on the device. You need to start gdb with command

gdb app_process

where app_process is Android binary which you need to copy from Android device to your PC. This binary is some kind of loader and lies in /system/bin/app_process file on the device (or in the emulator).

After you start gdb you connect it with the gdbserver by this command (ran from gdb shell)

target remote :5039

Because you are usually running Android device connected through USB and not normal TCP/IP network (or you are running Android emulator) you have to set up TCP port forwarding in advance. This is done by command

adb forward tcp:5039 tcp:5039

which you have to execute before starting gdb. This command will handle all connection requests to port 5039 on PC to the device on the same port.

And this is all.

Debug symbols

You typically compile your .so library with debug symbols. But it makes good sense to strip those symbols off before uploading your binary to the Android device. Binary will be smaller (which means faster installation time of the apk package), will occupy less memory and will run faster. So you upload stripped version of your libraries to the Android device and keep unstripped versions on your PC.

To tell gdb the path to the unstripped libraries run this command from gdb shell

set solib-search-path obj/local/armeabi

You can enter more paths and separate them by colon (:) character.

There is a difference between debug symbols and code optimization. Code optimization allows changing the code flow or removing some dead-code and unused variables. While debug symbols bind machine code together with appropriate lines in the source code. Usually you build a binary with debug symbols and no code optimization for debugging and another binary without debug symbols and with some code optimization for release. But you can use debug symbols together with code optimization. In this case some variables will be missing and some lines of code will be probably jumped over but in general you can debug this way.

Android ndk-build command always produce binaries (your libnative.so files) with debug symbols and place them in obj/local/armeabi subdirectory of your project. Then it strips off debug symbols and place stripped version in the final .apk (and thus in the device). ndk-gdb script uses this set solib-search-path command to point gdb to the unstripped binary so you can see your position in the source code. You can find the set solib-search-path command in obj/local/armeabi/gdb.setup file.

If you have android:debuggable=”true” in your AndroidManifest.xml or if you run ndk-build with NDK_DEBUG=1 argument then ndk-build will produce binaries with no code optimization. Otherwise code optimization will be used.

Sources

Gdb needs to have access to your source files. Otherwise it cannot show you the source code as it is executed. You can configure directory with your source files by command

directory jni

You can enter more paths and separate them by space. Take a look into obj/local/armeabi/gdb.setup file for example.

How ndk-gdb and friends works

If you are using Android NDK, r5 and Android 2.3 device you can use ndk-gdb script which does everything for you.

If you have android:debuggable=”true” in your AndroidManifest.xml then ndk-build will add the gdbserver into your .apk package. This gdbserver will be started by ndk-gdb and also all the other steps will be set up by this script.

Important caveat

Because gdbserver is attached to the already running process (as opposed to situation where process would be started by gdbserver) it can miss some code execution which take place soon after the application start. There is no easy solution to this.

I usually write some endless while loop and then change the control variable after gdb is fully started. For example

int i = 0
while (!i) {
  a++;
}

After ndk-gdb starts gdb session I can set breakpoints appropriately and change value of i variable by command

set var i=1

and then continue in application execution by gdb command c.

Multithread debug problem

It is well-known problem that ndk-gdb shipped with Android NDK, r4b used on Android 2.2 device was only able to hit the breakpoint on the main thread. If you set breakpoint on any other thread gdb session will crash.

Every time new thread is created libc will call _thread_created_hook() function (which has empty body). The purpose of this function is that gdb will set breakpoint on this function and thus will know that new thread was created. Gdb will then keep database of existing threads. Because of this gdb needs to have access to the libc.so file to figure out address of the _thread_created_hook() function.

Android NDK, r5 contains ndk-gdb which copies the file libc.so from the device (or emulator) and stores it in obj/local/armeabi subdirectory of your project. Also gdbserver included in this ndk is compiled with libthread_db support. Both those two things are missing in Android NDK, r4b.

Links

Description of multithreading debug bug.
http://code.google.com/p/android/issues/detail?id=9713

Android NDK, r5
http://dl.google.com/android/ndk/android-ndk-r5-linux-x86.tar.bz2

Android ndk main page
http://developer.android.com/sdk/ndk/index.html

See my previous spot how to set up cgdb debugger if you think it will suit you.

See my blog spot how to set up Eclipse for compiling and editing C/C++ code.

0) Prerequisities

You need Eclipse with CDT installed.
See my previous spot how to do it.

You also need Android ndk. Download it from http://developer.android.com/sdk/ndk/index.html and unpack it somewhere.

1) In Eclipse open your Android project which contains C/C++ code that you want to debug.

For this tutorial I’ve created simple MyAndroidProject.

2) Set android:debuggable=”true”. Set android:targetSdkVersion=”9″.

android:debuggable is a property of <application> tag in your AndroidManifest.xml. You can set it either directly in xml or in Application tab as in the screenshot.

android:targetSdkVersion=”9″ is a property of <uses-sdk> tag in your AndroidManifest.xml. You can set it either directly in xml or in Manifest tab as in the screenshot.

3) Run your application in debug mode and try to run ndk-gdb from console

To run application in debug mode press debug button (green bug/spider button in toolbox). In console go to your project directory and run ndk-gdb. It should succeed. If it fails you have to resolve the problem. Running ndk-gdb does not only ensure us that we are doing everything right so far, but also creates app_process, gdb.setup and libc.so files in obj/local/armeabi/ subdirectory of our project. Those files will be needed in later steps.

4) Create C/C++ debug configuration

Click on combo-box like down arrow next to debug button. Pop-up menu will appear and then select Debug Configurations…

In Debug Configurations window select C/C++ Application and press New button as advised on the right pane.

5) Set name of the debug configuration, and fill information on Main tab.

Select Standard Create Process Launcher by clicking on the blue Select other… link at the bottom of the window.

In C/C++ Application entry fill the path to the app_process binary which is located in obj/local/armeabi/ subdirectory of your project.

6) Click on Debugger tab and fill information about debugger.

Choose gdbserver Debugger as a Debugger.

It’s good idea to set initial breakpoint to some function but Android projects do not contain main function so fill some appropriate function name in Stop on startup field (I filled Java_com_example_map_MyAndroidProject_doSomething but you can see only Java_com_exam on the screenshot).

Set path to GDB debugger. The debugger is distributed with the Android ndk. Its located at toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin/arm-linux-androideabi-gdb.

Set path to GDB command line. This path should point to obj/local/armeabi/gdb2.setup file inside your project. You don’t have gdb2.setup file there yet but you will create one in a while.

7) Click on the Connection tab (so you are in Debugger->Connection section of your C/C++ debug configuration) and fill information about connecting gdb with gdbserver.

Choose TCP as a type of connection and choose 5039 as a Port number.

8) Finally click Apply to save all the information about your C/C++ debug configuration.

This will save your new C/C++ debug configuration. Later, when running your application in debug mode you can choose in combo box associated with Debug button what debug configuration you want to use. Now you have two debug configurations. Android Java one which was created automatically for you when you’ve created Android project. And C/C++ one you’ve just created.

9) Go to the obj/local/armeabi/ subdirectory of your project and copy gdb.setup file to gdb2.setup file. Remove target remote :5039 line from gdb2.setup.

Eclipse don’t like target remote :5039 line in gdb setup file because it wants to enter this command internally (that is why you configured port 5039 in the previous step). Because the gdb.setup file is recreated by ndk scripts you have to copy it to the gdb2.setup and point Eclipse to the gdb2.setup file (we did this in step 6).

10) Go to the directory with Android ndk and copy ndk-gdb to ndk-gdb-eclipse. Remove execution of gdb command from ndk-gdb-eclipse.

Original content of ndk-gdb (Android NDK, r5):

if [ -n "$OPTION_EXEC" ] ; then
    cat $OPTION_EXEC >> $GDBSETUP
fi
$GDBCLIENT -x `native_path $GDBSETUP`

Content of ndk-gdb-eclipse:

if [ -n "$OPTION_EXEC" ] ; then
    cat $OPTION_EXEC >> $GDBSETUP
fi
## $GDBCLIENT -x `native_path $GDBSETUP`

Eclipse will run the gdb binary itself. So we have to remove the execution of gdb from ndk-gdb. To save original content it is good idea to copy the ndk-gdb to ndk-gdb-eclipse.

11) Now you are done. Put breakpoint in Java code after the execution of System.loadLibrary() and start your application in Debug mode.

Start your application in debug mode by clicking on Debug button. It will automatically choose Android debug mode. Later you will have to take care to choose Android Java debugging configuration by clicking on combo arrow associated with the debug button.

The reason that breakpoint should be after System.loadLibrary() call is that our C/C++ code will be already loaded in that point and we can set breakpoints in it.

12) When execution reach the breakpoint run ndk-gdb-eclipse from your project directory and start debugging in C/C++ debug mode.

Go to the directory with your project and run ndk-gdb-eclipse. This will start server-part of the debugging infrastructure. Now click on the combo arrow associated with the debug button in Eclipse and choose Debug Configurations… Choose your C/C++ debug configuration and click Debug button.

The C/C++ debug configuration will be added to recently used debug configurations so later you don’t have to walk into the Debug Configurations… window again and you can choose between Android Java debug configuration and C/C++ debug configuration by just clicking on the combo arrow associated with debug button in Eclipse. However always make sure which debug configuration you are about to execute as this is where mistakes happen very often (at least for me).

After you’ve started C/C++ debug configuration click Resume button (or press F8). The application should resume its run and stop on C/C++ breakpoint (if you have one).

13) Now you are debugging the C/C++ code. Have fun!

Try to set some breakpoints in C/C++ code, etc.

Note on running ndk-gdb-eclipse

You have to run ndk-gdb-eclipse every time before starting C/C++ debug session. This script starts the gdbserver binary on device/emulator so gdb (run by Eclipse) can connect to it.

I made several attempts to force Eclipse to run ndk-gdb-eclipse script itself on start of C/C++ debug session. The most logical point is to write small script which will run ndk-gdb-eclipse without parameters (or with –force parameter) and then run gdb from Android ndk toolchain with all the parameters. This script can be used as GDB debugger command in the Debugger tab. But even if this script (and ndk-gdb-eclipse inside) was run successfully, the resulting connection of gdb and gdbserver always broke apart.

Final note and thanks!

This tutorial is heavily inspired by Sequoyah Project native debug tutorial. See http://www.eclipse.org/sequoyah/documentation/native_debug.php

Many thanks to you guys.

cgdb (http://cgdb.sourceforge.net/) is superior console front-end to gdb so it seems logical to use it for debugging of Android applications. Following modification of ndk-gdb script will cause that cgdb will be invoked instead of gdb.

Original ndk-gdb (Android NDK, r5):

# Now launch the appropriate gdb client with the right init commands
#
GDBCLIENT=${TOOLCHAIN_PREFIX}gdb
GDBSETUP=$APP_OUT/gdb.setup

Modified ndk-gdb (originally from Android NDK, r5) which uses cgdb instead of gdb:

# Now launch the appropriate gdb client with the right init commands
#
GDBCLIENT="cgdb -d ${TOOLCHAIN_PREFIX}gdb --"
GDBSETUP=$APP_OUT/gdb.setup

Now if you run ndk-gdb you will get cgdb front-end.

Small cgdb tutorial

In cgdb you can have focus either on source window (part with the source text) or gdb window.
To switch from source window to gdb press i.
To switch from gdb window to source press esc.

Commands you can use in source window:
arrows — scroll the text
space — set or delete breakpoint
o — open source file
i — switch focus to gdb window
- — make source window 1 line smaller
= — make source window 1 line bigger

Commands you can use in gdb window:
n — next instruction. Will not dive into subfunctions
s — step. Will dive into subfunctions
c — continue
b — set breakpoint
bt — show call stack (backtrace)
info threads — show information about running threads
info breakpoints — show information about breakpoints
pgup, pgdown — scroll the content of gdb window
esc — switch focus to source windows

Full cgdb documentation is at http://cgdb.sourceforge.net/docs/cgdb-no-split.html
Full gdb documentation is at http://www.gnu.org/software/gdb/documentation/

Have fun!

0) Prerequisities

You need to have Google ADT (Android Development Tools) installed. See http://developer.android.com/sdk/eclipse-adt.html how to do it.

You also need Android ndk. Download it from http://developer.android.com/sdk/ndk/index.html and unpack it somewhere.

1) Install CDT (C/C++ Development Tools) into Eclipse.

Choose Help->Install New Software… from the main menu.

Choose http://download.eclipse.org/releases/galileo as the source site. If you have another Eclipse release than Galileo choose the appropriate url.

Click Next, Accept licences and finish the installation process.

2) In Eclipse create Android project to which you want to add C/C++ code (if you already don’t have one).

For this tutorial I’ve created simple MyAndroidProject.

3) In file manager create jni/ directory in your project directory and place your C/C++ sources file here. Also put here Android.mk file which is a makefile that tells Android build-system how to build your files.

Take a look into Android ndk docs/ANDROID-MK.html file how to create one.

Simple example of Android.mk file:

LOCAL_PATH := $(call my-dir)

include $(CLEAR_VARS)

LOCAL_LDLIBS    := -llog

LOCAL_MODULE    := native
LOCAL_SRC_FILES := native.c

include $(BUILD_SHARED_LIBRARY)

4)  Refresh (F5) directories in Package Explorer to see jni directory here. Open your .c/.cpp file.

Your .c/.cpp file (native.c in my case) contains a lot of syntax errors which are not truly syntax errors. This is because Eclipse threats the project as a pure Java project. We have to convert the project into mixed Java & C/C++ project.

5) Press Ctrl+n (or choose File->New->Other… from main menu) and select Convert to a C/C++ Project.

This will convert your project into a mixed Java & C/C++ project rather than into pure C/C++ project (the name of the function is misleading).

Click Next. Then choose your project and below choose Makefile project and – Other Toolchain –. Click Finish.

After doing this Eclipse will ask you if you want to switch to C/C++ perspective. Choose Yes because otherwise you wouldn’t be able to set C/C++ build preferences.

6) Click on your project with right button and select Properties or press Alt+Enter

Properties windows will appear. Here you have to configure use of ndk-build instead of make all command and set proper include paths.

7) Choose C/C++ Build and configure ndk-build as a build command

In Builder settings fill ndk-build into Build command entry. You have to uncheck Use default build command. You also need to have ndk-build script in your PATH.

In Behaviour setting uncheck clean (ndk-build cleans project automatically on build and does not support separate clean command) and clear all text from build (ndk-build does not accept all as a parameter.

Click Apply to save settings.

8) Choose C/C++ General->Paths and Symbols and configure include path

In Includes tab choose GNU C or GNU C++ and click Add… button. Add path to include directory which is located in platforms/android-4/arch/arm/usr/include subdirectory of place where you’ve unpacked Android ndk. Include path depends on target for which you are compiling (android-4 in my case — i.e. Android 1.6).

Finally click Apply and OK and that is all. Now you can use all Eclipse power for editing your C/C++ sources. If you click Run or Debug Eclipse will compile C/C++ code as well as Java code and run it on device/emulator. However you will not be able to debug C/C++ code.