How to Build Google Crashpad
Overview
Crashpad is a cross-platform system for end-to-end crash reporting. Crashpad supports reporting of native crashes on a variety of operating systems including Windows, macOS, Linux, Android, and iOS. Crashpad also provides tools such as dump_syms
, symupload
and minidump_stackwalk
that provide developers with function names, file names, and line numbers in their crash reports. Integrating with Crashpad helps software engineers find and fix program crashes in order to develop more stable applications.
Installing depot_tools
The Chromium depot_tools are a set of tools that are required to build Crashpad. Run the following terminal commands below to clone depot_tools
and add the tools to your system PATH
variable. Be sure to change /path/to/depot_tools
to the path where you cloned depot_tools.
macOS
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
sudo echo "export PATH=/path/to/depot_tools:$PATH" >> ~/.zshrc
Linux
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
sudo echo "export PATH=/path/to/depot_tools:$PATH" >> ~/.bashrc
Windows
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
setx path "%path%;C:\path\to\depot_tools"
Getting the Crashpad Source
The Crashpad source code can be found here. Crashpad’s dependencies are managed by gclient
instead of git submodules, so it is best to use fetch
to get the source code.
Initial Checkout
mkdir ~/crashpad
cd ~/crashpad
fetch crashpad
Subsequent Checkouts
cd ~/crashpad/crashpad
git pull -r
gclient sync
Building Crashpad
Crashpad uses gn
to generate ninja
build files. You'll need to ensure you've installed a C++ compiler such as MSVC, gcc, or clang. For Linux, you'll also need to install build-essential
and libcurl4-openssl-dev
.
Generating Build Configuration
cd ~/crashpad/crashpad
gn gen out/Default
You can supply multiple arguments to customize your build, depending on the desired target OS and processor architecture. If you omit these values, gn will use values that match the current system.
For Windows, you will likely want to build Crashpad as a dynamic library. You can do this by passing extra_cflags="/MD"
for release configurations and extra_cflags="/MDd"
for debug configurations. Be sure to double-check that you're escaping special characters correctly, as the following command syntax can vary depending on whether you're using CMD or PowerShell.
gn gen out/Default --args="extra_cflags=\"/MD\""
Building with Ninja
Once you've generated your build config, use ninja to build your Crashpad configuration.
ninja -C out/Default
Integrating Crashpad
The Crashpad build generates several libraries that must be specified in the arguments passed to your application's linker.
macOS & Linux
At a minimum, macOS and Linux applications need to be linked with out/Default/obj/client/libcommon.a
, out/Default/obj/client/libclient.a
, out/Default/obj/util/libutil.a
, and out/Default/obj/third_party/mini_chromium/mini_chromium/base/libbase.a
.
A macOS application will need to link with out/Default/obj/util/libmig_output.a
as well.
When building Linux applications, libbase.a
must be the last Crashpad file specified in the build arguments, or the application will not build.
Additionally, ~/crashpad/crashpad
and ~/crashpad/crashpad/third_party/mini_chromium/mini_chromium
need to be added as include directories.
Finally, out/Default/crashpad_handler
needs to be deployed with the application and accessible at runtime.
Windows
At a minimum, Windows applications need to be linked with out\Default\obj\client\common.lib
, out\Default\obj\client\client.lib
, out\Default\obj\util\util.lib
, and out\Default\obj\third_party\mini_chromium\mini_chromium\base\base.lib.
Finally, out\Default\crashpad_handler.exe
needs to be deployed with the application and accessible at runtime.
Configuring Crashpad
Add the following includes to your application's entry point.
#include "client/crashpad_client.h"
#include "client/crash_report_database.h"
#include "client/settings.h"
Add a typedef for StringType
, add using statements for the base
, crashpad
and std
namespace and declare the following methods at the top of the entry point of the application.
#if defined(__APPLE__)
typedef std::string StringType;
#elif defined(__linux__)
typedef std::string StringType;
#elif defined(_MSC_VER)
typedef std::wstring StringType;
#endif
using namespace base;
using namespace crashpad;
using namespace std;
bool initializeCrashpad(void);
StringType getExecutableDir(void);
Implement the initializeCrashpad
method replacing the file paths with valid values.
bool initializeCrashpad() {
// Get directory where the exe lives so we can pass a full path to handler, reportsDir, metricsDir and attachments
StringType exeDir = getExecutableDir();
// Ensure that handler is shipped with your application
FilePath handler(exeDir + "/path/to/crashpad_handler");
// Directory where reports will be saved. Important! Must be writable or crashpad_handler will crash.
FilePath reportsDir(exeDir + "/path/to/crashpad");
// Directory where metrics will be saved. Important! Must be writable or crashpad_handler will crash.
FilePath metricsDir(exeDir + "/path/to/crashpad");
// Configure url with BugSplat’s public fred database. Replace 'fred' with the name of your BugSplat database.
StringType url = "https://fred.bugsplat.com/post/bp/crash/crashpad.php";
// Metadata that will be posted to the server with the crash report map
map<StringType, StringType> annotations;
annotations["format"] = "minidump"; // Required: Crashpad setting to save crash as a minidump
annotations["database"] = "fred"; // Required: BugSplat appName
annotations["product"] = "myCrashpadCrasher"; // Required: BugSplat appName
annotations["version"] = "1.0.0"; // Required: BugSplat appVersion
annotations["key"] = "Sample key"; // Optional: BugSplat key field
annotations["user"] = "[email protected]"; // Optional: BugSplat user email
annotations["list_annotations"] = "Sample comment"; // Optional: BugSplat crash description
// Disable crashpad rate limiting so that all crashes have dmp files
vector<StringType> arguments;
arguments.push_back("--no-rate-limit");
// Initialize Crashpad database
unique_ptr<CrashReportDatabase> database = CrashReportDatabase::Initialize(reportsDir);
if (database == NULL) return false;
// File paths of attachments to be uploaded with the minidump file at crash time - default upload limit is 2MB
vector<FilePath> attachments;
FilePath attachment(exeDir + "/path/to/attachment.txt");
attachments.push_back(attachment);
// Enable automated crash uploads
Settings *settings = database->GetSettings();
if (settings == NULL) return false;
settings->SetUploadsEnabled(true);
// Start crash handler
CrashpadClient *client = new CrashpadClient();
bool status = client->StartHandler(handler, reportsDir, metricsDir, url, annotations, arguments, true, true, attachments);
return status;
}
Next, implement the platform-specific getExecutableDir
method.
macOS
#include <mach-o/dyld.h>
#include <vector>
StringType getExecutableDir() {
unsigned int bufferSize = 512;
vector<char> buffer(bufferSize + 1);
if (_NSGetExecutablePath(&buffer[0], &bufferSize)) {
buffer.resize(bufferSize);
_NSGetExecutablePath(&buffer[0], &bufferSize);
}
char *lastForwardSlash = strrchr(&buffer[0], '/');
if (lastForwardSlash == NULL) return NULL;
*lastForwardSlash = 0;
return &buffer[0];
}
Linux
#include <stdio.h>
#include <unistd.h>
#define MIN(x, y) (((x) < (y)) ? (x) : (y))
StringType getExecutableDir() {
char pBuf[FILENAME_MAX];
int len = sizeof(pBuf);
int bytes = MIN(readlink("/proc/self/exe", pBuf, len), len - 1);
if (bytes >= 0) {
pBuf[bytes] = '\0';
}
char* lastForwardSlash = strrchr(&pBuf[0], '/');
if (lastForwardSlash == NULL) return NULL;
*lastForwardSlash = '\0';
return pBuf;
}
Windows
StringType getExecutableDir() {
HMODULE hModule = GetModuleHandleW(NULL);
WCHAR path[MAX_PATH];
DWORD retVal = GetModuleFileNameW(hModule, path, MAX_PATH);
if (retVal == 0) return NULL;
wchar_t *lastBackslash = wcsrchr(path, '\\');
if (lastBackslash == NULL) return NULL;
*lastBackslash = 0;
return path;
}
Finally, call the initializeCrashpad
method at the entry point of the application.
int main(int argc, char *argv[]) {
initializeCrashpad();
}
Generating Symbols
Generating sym files requires the dump_syms
tool from the repository of Crashpad’s predecessor, Breakpad. Dump_syms creates sym files from executable binaries, which allow minidumps to be symbolicated with function names, file names, and line numbers in the call stack.
mkdir ~/breakpad
cd breakpad
fetch breakpad
macOS
Build the application with symbolic information (preferably in a separate dSYM file) in order to get fully symbolicated crash reports.
Next, build the Xcode project located at src/src/tools/mac/dump_syms/dump_syms.xcodeproj
. Switch the configuration to dump_syms and build the project. The report navigator tab (icon looks like a chat bubble in Xcode 11) will show the file system location with the compiled executable. Run the dump_syms executable.
dump_syms -g "/path/to/myApp.app.dSYM" "/path/to/myApp.app/Contents/MacOS/myApp" > myApp.sym
Linux
Build the application with symbolic information and a build identifier. Using clang
this means building with the -g
and -Wl,--build-id
arguments.
Next, run ./configure && make
in the Breakpad directory. This will generate dump_syms
, symupload
and minidump_stackwalk
.
dump_syms /path/to/myApp.out > myApp.out.sym
Windows
The dump_syms
functionality is built into the symupload
utility and can be skipped if the application will be symbolicated remotely. Run dump_syms
only if the application will be symbolicated locally or the sym file will be uploaded to a remote server via some means other than symupload
.
For symupload.exe
to generate the correct output, applications must be built with symbolic information so that each exe
and dll
file generates a corresponding pdb
file. Generated pdb
files must contain full debug information. With Visual Studio, full debug information can be generated with the /Zi
compiler argument and the /DEBUG:FULL
linker argument. Failure to specify either the /Zi
or /DEBUG:FULL
arguments will result in dump_syms outputting incorrect sym file data. Additionally, the output pdb
file must be in the same folder as the corresponding exe
or dll
file otherwise dump_syms.exe
will fail.
To run symupload.exe
a copy of msdia140.dll
must be placed in the same folder. If Visual Studio is installed this file can be found at [VisualStudioFolder]\DIA SDK\bin\amd64\msdia140.dll
. Copy msdia140.dll
into the same folder as symupload.exe
and run symupload.exe
.
symupload.exe "path/to/myApp.exe" "https://fred.bugsplat.com/post/bp/symbol/breakpadsymbols.php?appName=myApp&appVer=1.0.0"
Dump_syms can be built from source so that the debugger can be used for troubleshooting. To build dump_syms
clone the gyp repository and run python setup.py install
. Next, run gyp ~\breakpad\src\tools\windows\dump_syms\dump_syms.gyp
and use Visual Studio to build the sln file generated by gyp. If the build fails with an error that unique_pointer is not part of std add #include <memory>
to the top of the file that contains the error and rebuild.
Uploading Symbols
The symupload
tool is also part of the Breakpad repository and can be used to upload sym files to your server. Symbols need to be uploaded to a server in order for it to correctly symbolicate a minidump file.
macOS
Build the Xcode project located at src/src/tools/mac/symupload/symupload.xcodeproj
. The report navigator tab (icon looks like a chat bubble in Xcode 11) will show you the file system location with the compiled executable. Copy the symupload file into your project and run the executable wrapping the arguments to symupload in quotes.
symupload "/path/to/myApp.sym" "https://fred.bugsplat.com/post/bp/symbol/breakpadsymbols.php?appName=myApp&appVer=1.0.0"
Windows
To use symupload
, applications must be built with symbolic information so that each exe
and dll
file generates a corresponding pdb
file. Generated pdb
files must contain full debug information. Full debug information can be generated with the /Zi
compiler argument and the /DEBUG:FULL
linker argument. Failure to specify either the /Zi
or /DEBUG:FULL
arguments will result in symupload
failing. The output pdb
file must be in the same folder as the corresponding exe
or dll
file.
To run symupload.exe
a copy of msdia140.dll
must be placed in the same folder. If Visual Studio is installed, this file can be found at [VisualStudioFolder]\DIA SDK\bin\amd64\msdia140.dll
. Copy msdia140.dll into the same folder as symupload.exe and run symupload.exe.
symupload.exe "/path/to/myApp.exe" "https://fred.bugsplat.com/post/bp/symbol/breakpadsymbols.php?appName=myApp&appVer=1.0.0"
Symupload can be built from source so that the debugger can be used for troubleshooting. To build symupload
clone the gyp repository and run python setup.py install
. Next, run gyp ~\breakpad\src\tools\windows\symupload\symupload.gyp
and use Visual Studio to build the sln file generated by gyp
. If the build fails with an error that unique_pointer is not part of std add #include <memory>
to the top of the file that contains the error and rebuild.
Symbolicating Crash Reports
Minidump_stackwalk
is another tool in the Breakpad repository that is responsible for the symbolication of minidump files. In order to correctly symbolicate minidumps, sym files need to be nested at least 2 folders deep. The topmost parent folder’s name must equal the sym files module name. The first child folder’s name must equal the module id. Additionally, the sym file name must also match the module name. The module id and module name can be found in the module record of the sym file.
For example the module myApp
with the module id 1A67F3DEAACA3B209D9992871B2620AA0
must be located at /path/to/symbols/myApp/1A67F3DEAACA3B209D9992871B2620AA0/myApp.sym
.
macOS & Linux
Minidump_stackwalk is built when building Breakpad.
cd ~/breakpad/src && ./configure && make
Windows
Minidump_stackwalk can be built on Windows with the help of MinGW-w64 and TDM-GCC. First, download and install TDM-GCC. Next, open a MINGw64 shell, such as Git Bash.
cd ~/breakpad/src && ./configure --disable-dependency-tracking && make
If make doesn't work, try removing all instances of /usr/bin
from Makefile
.
Run minidump_stackwalk
passing it a path to a .dmp
file and a symbols directory. The folders in the symbols directory need to be laid out following the pattern module_name/module_id/module_name.sym
:
minidump_stackwalk -m "/path/to/minidump.dmp" "/path/to/symbols"
Additionally, minidumps generated by the Crashpad library can also be symbolicated via Debugging Tools for Windows given the .exe
, .dll
and .pdb
files are made available to WinDBG.
Troubleshooting
Most issues symbolicating dump files can be traced back to mismatched module ids. Anything that modifies a given executable (code-signing, anti-cheat) must be performed before dump_syms
and symupload
are run. Every time an executable is modified dump_syms
and symupload
need to be re-run. The name and id of the module loaded at runtime can be found in the minidump_stackwalk
output. The module name and id must match the module name and id of the generated sym file in order for minidump_stackwalk
to correctly symbolicate the minidump file.
References
Last updated
Was this helpful?