profiling with Breakpad on android x86 devices

[maks]

When attempting to get breakpad profiling (via Mobile USB option) for fennec running on a android x86 device (pc type hardware running androix-x86.org ics rc2 build) I'm not able to get any stack profiling information back.

This is using current version of the addon with latest fennec x86 nightly.

When the addon first connects to the device, I do see it successfully copy across all the system libs on to my laptop, into /tmp.

I tried using android-ndk-r8e/toolchains/x86-4.6/prebuilt/linux-x86_64/bin/i686-linux-android-addr2line instead of arm-eabi-addr2line (via aliasing on PATH) but that didn't seem to change anything and strangely I could still use it to see (full stack function info) with traces made on a arm device.

When using with fennec on x86, I'm able to get as far as clicking analyze but then I just get an empty trace in the cleopatra ui and in terminal running ff all I see is:

Fetching js source
Resolve symbol
2013-09-13 19:53:20: dump_symbols.cc:584: INFO: 
2013-09-13 19:53:20: dump_symbols.cc:585: INFO: LoadSymbols: BEGIN   /usr/lib/x86_64-linux-gnu/libxml2.so.2
2013-09-13 19:53:20: dump_symbols.cc:685: INFO: LoadSymbols:   read CFI from .eh_frame
2013-09-13 19:53:20: dump_symbols.cc:788: INFO: LoadSymbols: SUCCESS /usr/lib/x86_64-linux-gnu/libxml2.so.2
2013-09-13 19:53:20: dump_symbols.cc:584: INFO: 
2013-09-13 19:53:20: dump_symbols.cc:585: INFO: LoadSymbols: BEGIN   /usr/lib/x86_64-linux-gnu/librsvg-2.so.2
2013-09-13 19:53:20: dump_symbols.cc:685: INFO: LoadSymbols:   read CFI from .eh_frame
2013-09-13 19:53:20: dump_symbols.cc:788: INFO: LoadSymbols: SUCCESS /usr/lib/x86_64-linux-gnu/librsvg-2.so.2
2013-09-13 19:53:20: dump_symbols.cc:584: INFO: 
2013-09-13 19:53:20: dump_symbols.cc:585: INFO: LoadSymbols: BEGIN   /usr/lib/x86_64-linux-gnu/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-svg.so
2013-09-13 19:53:20: dump_symbols.cc:685: INFO: LoadSymbols:   read CFI from .eh_frame
2013-09-13 19:53:20: dump_symbols.cc:788: INFO: LoadSymbols: SUCCESS /usr/lib/x86_64-linux-gnu/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-svg.so
2013-09-13 19:53:20: dump_symbols.cc:584: INFO: 
2013-09-13 19:53:20: dump_symbols.cc:585: INFO: LoadSymbols: BEGIN   /usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so.0
2013-09-13 19:53:20: dump_symbols.cc:685: INFO: LoadSymbols:   read CFI from .eh_frame
2013-09-13 19:53:20: dump_symbols.cc:788: INFO: LoadSymbols: SUCCESS /usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so.0

[bgirard]

We have not even looked into x86 unwinding on Android so I'm not surprised it doesn't work. I don't know how much addr2line across configurations but I imagine it's not much so I'm not surprised it would work with ARM. Feel free to tinker with it but I can't provide any cycles to help for now. For now I recommend profiling without breakpad and instrumenting the code with PROFILER_LABEL incrementally in hot spots. Note that we should support the PC addresses when sampling even on x86.

[maks]

Building Fennec to make the profiling work (with leaf nodes but not full stack unwinding) can be found in: https://bugzilla.mozilla.org/show_bug.cgi?id=926255

BUT I had to modify my path when running FF nightly (with the addon) to have android-ndk-r8e/toolchains/x86-4.6/prebuilt/linux-x86_64/bin/i686-linux-android-addr2line be linked as arm-eabi-addr2line, because "arm-eabi-addr2line" is hard-coded somewhere in the addon I think. Would be good to have a drop-down perhaps in settings to choose between arm and x86 device targets?

[maks]

I should also add that it turns out the initial problem I saw with an empty trace was to do with not killing the running fennec process after rebuilding and reinstalling a new apk on to the android device I was testing on.

[AaronMT]

@maks just ran into your 'but' comment too, nothing major, just had to symlink

[maks]

@AaronMT yes though its a bit annoying having to remember to change the symlink back and forward each time I swap between x86 and arm

[bgirard]

Ohh the addr2line are not compatible? Argg that's a pain. Ok ideally we can support both at the same time.

[maks]

@bgirard I'm not sure but I don't think so, since they are diff files in the NDK and even diff files for diff versions on same platform, eg:

find android-ndk-r8e/ -name "addr2line"

android-ndk-r8e/toolchains/mipsel-linux-android-4.4.3/prebuilt/linux-x86_64/bin/mipsel-linux-android-addr2line
android-ndk-r8e/toolchains/x86-4.7/prebuilt/linux-x86_64/bin/i686-linux-android-addr2line
android-ndk-r8e/toolchains/mipsel-linux-android-4.7/prebuilt/linux-x86_64/bin/mipsel-linux-android-addr2line
android-ndk-r8e/toolchains/arm-linux-androideabi-4.6/prebuilt/linux-x86_64/bin/arm-linux-androideabi-addr2line
android-ndk-r8e/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86_64/bin/arm-linux-androideabi-addr2line
android-ndk-r8e/toolchains/x86-4.6/prebuilt/linux-x86_64/bin/i686-linux-android-addr2line
android-ndk-r8e/toolchains/arm-linux-androideabi-4.7/prebuilt/linux-x86_64/bin/arm-linux-androideabi-addr2line
android-ndk-r8e/toolchains/mipsel-linux-android-4.6/prebuilt/linux-x86_64/bin/mipsel-linux-android-addr2line
android-ndk-r8e/toolchains/x86-4.4.3/prebuilt/linux-x86_64/bin/i686-linux-android-addr2line

and I had to use i686-linux-android-addr2line to get it to work with fennec built for i386