The Breakpad processor library is an open-source framework to access the the information contained within crash dumps for multiple platforms, and to use that information to produce stack traces showing the call chain of each thread in a process. After processing, this data is made available to users of the library.
The Breakpad processor is intended to sit at the core of a comprehensive crash-reporting system that does not require debugging information to be provided to those running applications being monitored. Some existing crash-reporting systems, such as GNOME’s Bug-Buddy and Apple’s CrashReporter, require symbolic information to be present on the end user’s computer; in the case of CrashReporter, the reports are transmitted only to Apple, not to third-party developers. Other systems, such as Microsoft’s Windows Error Reporting and SupportSoft’s Talkback, transmit only a snapshot of a crashed process’ state, which can later be combined with symbolic debugging information without the need for it to be present on end users’ computers. Because symbolic debugging information consumes a large amount of space and is otherwise not needed during the normal operation of software, and because some developers are reluctant to release debugging symbols to their customers, Breakpad follows the latter approach.
We know of no currently-maintained crash-reporting systems that meet our requirements, which are to: * allow for symbols to be separate from the application, * handle crash reports from multiple platforms, * allow developers to operate their own crash-reporting platform, and to * be open-source. Windows Error Reporting only functions for Microsoft products, and requires the involvement of Microsoft’s servers. Talkback, while cross-platform, has not been maintained and at this point does not support Mac OS X on x86, which we consider to be a significant platform. Talkback is also closed-source commercial software, and has very specific requirements for its server platform.
We are aware of Windows-only crash-reporting systems that leverage Microsoft’s debugging interfaces. Such systems, even if extended to support dumps from other platforms, are tied to using Windows for at least a portion of the processor platform.
The Breakpad processor itself is written in standard C++ and will work on a variety of platforms. The dumps it accepts may also have been created on a variety of systems. The library is able to combine dumps with symbolic debugging information to create stack traces that include function signatures. The processor library includes simple command-line tools to examine dumps and process them, producing stack traces. It also exposes several layers of APIs enabling crash-reporting systems to be built around the Breakpad processor.
In the processor, the dump data is of primary significance. Dumps typically contain:
.so, .exe, .dll, etc.) which provides the code.Ordinarily, dumps are produced as a result of a crash, but other triggers may be set to produce dumps at any time a developer deems appropriate. The Breakpad processor can handle dumps in the minidump format, either generated by an Breakpad client “handler” implementation, or by another implementation that produces dumps in this format. The DbgHelp.dll!MiniDumpWriteDump function on Windows produces dumps in this format, and is the basis for the Breakpad handler implementation on that platform.
The minidump format is essentially a simple container format, organized as a series of streams. Each stream contains some type of data relevant to the crash. A typical “normal” minidump contains streams for the thread list, the module list, the CPU context at the time of the crash, and various bits of additional system information. Other types of minidump can be generated, such as a full-memory minidump, which in addition to stack memory contains snapshots of all of a process’ mapped memory regions.
The minidump format was chosen as Breakpad’s dump format because it has an established track record on Windows, and it can be adapted to meet the needs of the other platforms that Breakpad supports. Most other operating systems use “core” files as their native dump formats, but the capabilities of core files vary across platforms, and because core files are usually presented in a platform’s native executable format, there are complications involved in accessing the data contained therein without the benefit of the header files that define an executable format’s entire structure. Because minidumps are leaner than a typical executable format, a redefinition of the format in a cross-platform header file, minidump_format.h, was a straightforward task. Similarly, the capabilities of the minidump format are understood, and because it provides an extensible container, any of Breakpad’s needs that could not be met directly by the standard minidump format could likely be met by extending it as needed. Finally, using this format means that the dump file is compatible with native debugging tools at least on Windows. A possible future avenue for exploration is the conversion of minidumps to core files, to enable this same benefit on other platforms.
We have already provided an extension to the minidump format that allows it to carry dumps generated on systems with PowerPC processors. The format already allows for variable CPUs, so our work in this area was limited to defining a context structure sufficient to represent the execution state of a PowerPC. We have also defined an extension that allows minidumps to indicate which thread of execution requested a dump be produced for non-crash dumps.
Often, the information contained within a dump alone is sufficient to produce a full stack backtrace for each thread. Certain optimizations that compilers employ in producing code frustrate this process. Specifically, the “frame pointer omission” optimization of x86 compilers can make it impossible to produce useful stack traces given only a stack snapshot and CPU context. In these cases, however, compiler-emitted debugging information can aid in producing useful stack traces. The Breakpad processor is able to take advantage of this debugging information as supplied by Microsoft’s C/C++ compiler, the only compiler to apply such optimizations by default. As a result, the Breakpad processor can produce useful stack traces even from code with frame pointer omission optimizations as produced by this compiler.
The symbol files that the Breakpad processor accepts allow for frame pointer omission data, but this is only one of their capabilities. Each symbol file also includes information about the functions, source files, and source code line numbers for a single module of code. A module is an individually-loadble chunk of code: these can be executables containing a main program (exe files on Windows) or shared libraries (.so files on Linux, .dylib files, frameworks, and bundles on Mac OS X, and .dll files on Windows). Dumps contain information about which of these modules were loaded at the time the dump was produced, and given this information, the Breakpad processor attempts to locate debugging symbols for the module through a user-supplied function embodied in a “symbol supplier.” Breakpad includes a sample symbol supplier, called SimpleSymbolSupplier, that is used by its command-line tools; this supplier locates symbol files by pathname. SimpleSymbolSupplier is also available to other users of the Breakpad processor library. This allows for the use of a simple reference implementation, but preserves flexibility for users who may have more demanding symbol file storage needs.
Breakpad’s symbol file format is text-based, and was defined to be fairly human-readable and to encompass the needs of multiple platforms. The Breakpad processor itself does not operate directly with native symbol formats (DWARF and STABS on most Unix-like systems, .pdb files on Windows), because of the complications in accessing potentially complex symbol formats with slight variations between platforms, stored within different types of binary formats. In the case of .pdb files, the debugging format is not even documented. Instead, Breakpad’s symbol files are produced on each platform, using specific debugging APIs where available, to convert native symbols to Breakpad’s cross-platform format.
Most commonly, a developer will enable an application to use Breakpad by building it with a platform-specific client “handler” library. After building the application, the developer will create symbol files for Breakpad’s use using the included dump_syms or symupload tools, or another suitable tool, and place the symbol files where the processor’s symbol supplier will be able to locate them.
When a dump file is given to the processor’s MinidumpProcessor class, it will read it using its included minidump reader, contained in the Minidump family of classes. It will collect information about the operating system and CPU that produced the dump, and determine whether the dump was produced as a result of a crash or at the direct request of the application itself. It then loops over all of the threads in a process, attempting to walk the stack associated with each thread. This process is achieved by the processor’s Stackwalker components, of which there are a slightly different implementations for each CPU type that the processor is able to handle dumps from. Beginning with a thread’s context, and possibly using debugging data, the stackwalker produces a list of stack frames, containing each instruction executed in the chain. These instructions are matched up with the modules that contributed them to a process, and the SymbolSupplier is invoked to locate a symbol file. The symbol file is given to a SourceLineResolver, which matches the instruction up with a specific function name, source file, and line number, resulting in a representation of a stack frame that can easily be used to identify which code was executing.
The results of processing are made available in a ProcessState object, which contains a vector of threads, each containing a vector of stack frames.
For small-scale use of the Breakpad processor, and for testing and debugging, the minidump_stackwalk tool is provided. It invokes the processor and displays the full results of processing, optionally allowing symbols to be provided to the processor by a pathname-based symbol supplier, SimpleSymbolSupplier.
For lower-level testing and debugging, the processor library also includes a minidump_dump tool, which walks through an entire minidump file and displays its contents in somewhat readable form.
The Breakpad processor library is able to process dumps produced on Mac OS X systems running on x86, x86-64, and PowerPC processors, on Windows and Linux systems running on x86 or x86-64 processors, and on Android systems running ARM or x86 processors. The processor library itself is written in standard C++, and should function properly in most Unix-like environments. It has been tested on Linux and Mac OS X.
There are currently no firm plans or timetables to implement any of these features, although they are possible avenues for future exploration.
The symbol file format can be extended to carry information about the locations of parameters and local variables as stored in stack frames and registers, and the processor can use this information to provide enhanced stack traces showing function arguments and variable values.
On Mac OS X and Linux, we can provide tools to convert files from the minidump format into the native core format. This will enable developers to open dump files in a native debugger, just as they are presently able to do with minidumps on Windows.