Skip to main content.

18. Debugging

18.1. BDM

BDM is way cool, especially if you are bringing up a custom board and don't want to invest in an In Circuit Emulator. You should definitely include a BDM connector on any custom hardware, and all off-the-shelf boards have it. For a broad introduction, see http://www.macraigor.com/zenofbdm.pdf

Be careful to ensure that the CPU watchdog is disabled in the configuration file for your BDM probes, otherwise it will continually reset the CPU and nothing will work. In particular, the watchdog is enabled at reset and must be disabled if you wish to single step from reset. Also check that DER is zero'd when running with the debugger. Leaving the BDM probe connected can interfere with the target if it is not configured absolutely correctly, which usually involves the BDM probe being completely passive once the kernel is running. Symptoms are that the kernel crashes with the BDM probe connected, but runs fine without it. If you are using BDM and experience unexpected kernel crashes, try disconnecting the BDM probe.

If you're using an off-the-shelf board which already has a working rommonitor?, you generally won't need to use BDM at all, as you can get by fine with just the serialconsole?. Many developers, particularly those bringing up a custom board from scratch, find BDM invaluable; to others it can be more trouble than it's worth. One advantage of a BDM interface over a kernel debugger is that you can really "freeze" the CPU including all timers, interrupts etc.

18.1.1. MMU Support

Some BDM debuggers are capable of performing MMU table walks, which is essential for debugging in virtual memory environments such as Linux because otherwise the BDM port merely deals with raw physical addresses. However, many BDM systems can't do this, so check with the vendor that they support MMU table walks on the particular CPU you're interested in before committing to one; otherwise it will be almost impossible to use once the kernel has turned the MMU on. See http://lists.linuxppc.org/listarcs/linuxppc-embedded/199909/msg00021.html

Once the kernel is running you can use*gdb*remotely or even run it natively on the target hardware if you have enough RAM. It may be worth providing extra RAM on some of your development boards to allow for this.

18.1.2. GDB on BDM

See "Using GDB for Remote Debugging: BDM Support" in the CrossGCC? FAQ at http://www.objsw.com/CrossGCC/FAQ-7.html

BDM debugging under gdb is not supported by all BDM hardware vendors. See the thread http://www.oarcorp.com/rtems/maillistArchives/rtems-users/1999/july/msg00057.html

To support BDM debugging under gdb requires the appropriate remote-* device in gdb, and possibly a kernel driver. If you are cross-developing, you must configure gdb as follows to include the appropriate devices:

    % configure --target=powerpc-linux

The following systems currently support BDM debugging on gdb:

18.1.2.1. BDM4GDB

http://bdm4gdb.sourceforge.net/

An open user-supported project to build your own PowerPC BDM interface at extremely low cost. Supports extensible Flash programming and implements software-tablewalk so you have MMUsupport. You can use gdb to debug kernel code (single stepping, breakpoints, etc.) and/or inspect kernel data.

If you don't want to build it yourself, the adapter is available at cost from Denx, subject to availability.

18.1.2.2. 1MPCBDM

http://www.vas-gmbh.de/software/mpcbdm/

The precursor to the BDM4GDB project, courtesy of Frank Przybylski.

18.1.2.3. PPCBDM

http://cyclone.parad.ru/ppcbdm/

The ultra low hardware cost precursor to MPCBDM and BDM4GDB, courtesy of Sergey Drazhnikov.

18.1.2.4. 1Abatron

http://www.abatron.ch/BDI/bdiGDB.html

Has an ideal BDM/JTAG emulation solution for Linux hosting, with Linux MMUsupport specifically in mind. It is an Ethernet-based unit which has a telnet interface with which one can program various popular flash parts. This is ideal for Linux since one can build a kernel and expect script the programming process automagically.

Their external box implements the standard gdb remote protocol allowing you to host debugging on any platform that can host gdb.

18.1.2.5. Avocet Systems (formerly Huntsville Microsystems)

http://www.hmi.com/bmd.htm

Patches to add support to gdb are available in ftp://ftp.hmi.com/pub/gdb/

18.1.2.6. Macraigor Systems

http://www.macraigor.com/

Currently only support Windows platform, but say Linux support for the Kestrel device (but not the Wiggler, Raven, etc) should be available sometime in September 2000.

OCD Commander is a free assembly level debugger for Windows, which may work under VMWare. I couldn't get it to work with the Raven/Blackbird interface on the Embedded Planet CLLF under Windows; it wouldn't single-step correctly.

Beware that the 1999 version of OCD Commander silently truncates S-records longer than 20 bytes during download, so you'll need to patch objcopy or write your own S-record generator to work around this.

They have flash programming software available for download, but the Web site doesn't mention that it isn't actually free: you have to pay for the key to use it.

If you're prepared to do your debugging under Windows, you can use the Cygwin tools from http://sources.redhat.com/cygwin/ to build a gdb cross-debugger which runs under Windows and uses the Wigglers.dll to talk to the target. Note that Wigglers.dll also loads other DLLs, so copy all the DLLs that come with OCD Commander from http://www.macraigor.com/" into your gdb directory and use the command:

    (gdb) target ocd wiggler

You can also do this remotely, by using rproxy,describedbelow.

18.1.2.7. RISCWatch Debugger for PowerPC Processors

http://www.chips.ibm.com/products/powerpc/riscwtch/

RISCWatch is a hardware and software development tool for the PowerPC 600/700 Family of microprocessors and the PowerPC 400Series of Embedded Controllers. The source-level debugger and processor-control features provide developers with the tools needed to develop and debug hardware and software.

18.1.2.8. WindRiver? (formerly EST Corporation)

http://www.windriver.com/products/html/embed_dev_tools.html

They've ported their VisionXD? UNIX debugger to Linux and it is available with their parallel port of Ethernet-based probe. It can program flash parts and do source level debugging. It costs $6k or $8k depending on parport or Ethernet connection. The Windows version is only $4k. :-/

Beware that the EST tools don't handle the Linux zImage properly. See http://lists.linuxppc.org/listarcs/linuxppc-embedded/200002/msg00073.html

18.1.3. Other debuggers

There are many other sources of BDM hardware probes and debuggers, but many vendors still lag way behind as far as Linux support. Commercial solutions which do not support a Linux development host include:

18.1.3.1. Mentor (X-ray)

http://www.mentor.com/embedded/xray/index.html

18.1.3.2. Avocet Systems (SourceGate?)

http://www.hmi.com/bmd.htm

18.1.3.3. Tasking (CrossView? Pro)

http://www.tasking.com/technology/xvwp.html

18.1.3.4. WindRiver? (SingleStep?)

http://www.windriver.com/products/html/singlestep_suite_ds.html

18.1.3.5. Applied Microsystems (PowerTap?)

http://www.amc.com/products/powertap.html

18.1.3.6. Lauterbach

http://www.lauterbach.com/

18.2. Serial Console

Virtually all boards use a serial console on SMC1 for boot messages and general debugging. Connect it to a serial port on your Linux development machine, where you can run minicom to interact with the board.

18.3. GDB

Once the kernel is running, you can use gdb in several different ways to debug user space programs:

18.3.1. gdbserver

You can run gdbserver on your target and run gdb back on your development machine, even if you're cross-developing. This requires far less resources than running all of gdb on your target. See http://qslinux.org/docs/cross/gdb/index.html

If you're cross-developing, remember to configure your gdb as described earlier.

18.3.2. rproxy

http://www.std.com/qqi/labslave/rproxy.html

This in an extended/enhanced gdbserver, which can also run on Windows and talk to BDM devices not yet supported by Linux.

18.3.3. native

If you have lots of RAM, you can run gdb directly on your target. If you are cross-developing, you need to configure gdb with:

% configure --host=powerpc-linux

18.4. Kernel

Some kernels include kgdb support for using gdb for kernel debugging, enabled by configuring with CONFIG_KGDB.

18.5. Oops Messages

You get these whenever something truly bad happens in the kernel. Learn to know and respect them -- they are your friends, not your enemies. For general info on how to understand them, see the file Documentation/oops-tracing.txt in the kernel source tree.

You'll get a long way just by looking up the instructions at the address indicated by NIP on the first line of the Oops in the output of:

objdump --disassemble vmlinux
This will show you the instruction causing the fault. Work backwards to find the line of C source code associated with it, and add printk's around it to find what is going wrong.

For more help with decoding kernel panic messages, see http://lists.linuxppc.org/listarcs/linuxppc-embedded/199912/msg00090.html

18.6. printk

printk is an indispensible tool. You can use it to add checkpointing, print kernel values that you can't get to via /proc, etc. It can be called anywhere, including interrupt routines, provided you're prepared for some interesting output.

Note that during the boot process, the kernel "prints" lots of stuff, and it all goes into a buffer, to emerge quite late in the boot process when the serial console port is initialized with the call to console_init. This eventually calls register_console which will dump out the logged messages. So you can't necessarily assume that the kernel didn't get to your checkpoint just because the printk message didn't appear on the serial port during this part of the boot sequence.

17. Applications 1. Introduction 19. Performance
Prev Home Next