# How to Build Google Crashpad ## Overview Crashpad is a cross-platform system for end-to-end crash reporting. Crashpad supports reporting of native crashes on a variety of operating systems including Windows, macOS, Linux, Android, and iOS. Crashpad also provides tools such as `dump_syms`, `symupload` and `minidump_stackwalk` that provide developers with function names, file names, and line numbers in their crash reports. Integrating with Crashpad helps software engineers find and fix program crashes in order to develop more stable applications. {% embed url="" %} ## Installing depot\_tools The Chromium [depot\_tools](https://chromium.googlesource.com/chromium/tools/depot_tools.git) are a set of tools that are required to build Crashpad. Run the following terminal commands below to clone `depot_tools` and add the tools to your system `PATH` variable. Be sure to change `/path/to/depot_tools` to the path where you cloned depot\_tools. ### **macOS** ```bash git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git sudo echo "export PATH=/path/to/depot_tools:$PATH" >> ~/.zshrc ``` ### **Linux** ``` git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git sudo echo "export PATH=/path/to/depot_tools:$PATH" >> ~/.bashrc ``` ### **Windows** ```bash git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git setx path "%path%;C:\path\to\depot_tools" ``` ## Getting the Crashpad Source The Crashpad source code can be found [here](https://chromium.googlesource.com/crashpad/crashpad). Crashpad’s dependencies are managed by `gclient` instead of git submodules, so it is best to use `fetch` to get the source code. ### **Initial Checkout** ```bash mkdir ~/crashpad cd ~/crashpad fetch crashpad ``` ### **Subsequent Checkouts** ```bash cd ~/crashpad/crashpad git pull -r gclient sync ``` ## Building Crashpad Crashpad uses `gn` to generate `ninja` build files. You'll need to ensure you've installed a C++ compiler such as [MSVC](https://visualstudio.microsoft.com/), [gcc](https://gcc.gnu.org/), or [clang](https://clang.llvm.org/). For Linux, you'll also need to install `build-essential` and `libcurl4-openssl-dev` . ### **Generating Build Configuration** ```bash cd ~/crashpad/crashpad gn gen out/Default ``` You can supply multiple arguments to customize your build, depending on the desired target OS and processor architecture. If you omit these values, [gn](https://gn.googlesource.com/gn/+/main/docs/quick_start.md) will use values that match the current system. For Windows, you will likely want to build Crashpad as a dynamic library. You can do this by passing `extra_cflags="/MD"` for release configurations and `extra_cflags="/MDd"` for debug configurations. Be sure to double-check that you're escaping special characters correctly, as the following command syntax can vary depending on whether you're using CMD or PowerShell. ```bash gn gen out/Default --args="extra_cflags=\"/MD\"" ``` ### **Building with Ninja** Once you've generated your build config, use [ninja](https://ninja-build.org/) to build your Crashpad configuration. ```bash ninja -C out/Default ``` ## Integrating Crashpad The Crashpad build generates several libraries that must be specified in the arguments passed to your application's linker. ### **macOS & Linux** At a minimum, macOS and Linux applications need to be linked with `out/Default/obj/client/libcommon.a`, `out/Default/obj/client/libclient.a`, `out/Default/obj/util/libutil.a`, and `out/Default/obj/third_party/mini_chromium/mini_chromium/base/libbase.a`. A macOS application will need to link with `out/Default/obj/util/libmig_output.a` as well. When building Linux applications, `libbase.a` must be the last Crashpad file specified in the build arguments, or the application will not build. Additionally, `~/crashpad/crashpad` and `~/crashpad/crashpad/third_party/mini_chromium/mini_chromium` need to be added as include directories. Finally, `out/Default/crashpad_handler` needs to be deployed with the application and accessible at runtime. ### **Windows** At a minimum, Windows applications need to be linked with `out\Default\obj\client\common.lib`, `out\Default\obj\client\client.lib`, `out\Default\obj\util\util.lib`, and `out\Default\obj\third_party\mini_chromium\mini_chromium\base\base.lib.` Finally, `out\Default\crashpad_handler.exe` needs to be deployed with the application and accessible at runtime. ## Configuring Crashpad Add the following includes to your application's entry point. ```cpp #include "client/crashpad_client.h" #include "client/crash_report_database.h" #include "client/settings.h" ``` Add a typedef for `StringType`, add using statements for the `base`, `crashpad` and `std` namespace and declare the following methods at the top of the entry point of the application. ```bash #if defined(__APPLE__) typedef std::string StringType; #elif defined(__linux__) typedef std::string StringType; #elif defined(_MSC_VER) typedef std::wstring StringType; #endif using namespace base; using namespace crashpad; using namespace std; bool initializeCrashpad(void); StringType getExecutableDir(void); ``` Implement the `initializeCrashpad` method replacing the file paths with valid values. ```cpp bool initializeCrashpad() { // Get directory where the exe lives so we can pass a full path to handler, reportsDir, metricsDir and attachments StringType exeDir = getExecutableDir(); // Ensure that handler is shipped with your application FilePath handler(exeDir + "/path/to/crashpad_handler"); // Directory where reports will be saved. Important! Must be writable or crashpad_handler will crash. FilePath reportsDir(exeDir + "/path/to/crashpad"); // Directory where metrics will be saved. Important! Must be writable or crashpad_handler will crash. FilePath metricsDir(exeDir + "/path/to/crashpad"); // Configure url with BugSplat’s public fred database. Replace 'fred' with the name of your BugSplat database. StringType url = "https://fred.bugsplat.com/post/bp/crash/crashpad.php"; // Metadata that will be posted to the server with the crash report map map annotations; annotations["format"] = "minidump"; // Required: Crashpad setting to save crash as a minidump annotations["database"] = "fred"; // Required: BugSplat appName annotations["product"] = "myCrashpadCrasher"; // Required: BugSplat appName annotations["version"] = "1.0.0"; // Required: BugSplat appVersion annotations["key"] = "Sample key"; // Optional: BugSplat key field annotations["user"] = "fred@bugsplat.com"; // Optional: BugSplat user email annotations["list_annotations"] = "Sample comment"; // Optional: BugSplat crash description // Disable crashpad rate limiting so that all crashes have dmp files vector arguments; arguments.push_back("--no-rate-limit"); // Initialize Crashpad database unique_ptr database = CrashReportDatabase::Initialize(reportsDir); if (database == NULL) return false; // File paths of attachments to be uploaded with the minidump file at crash time - default upload limit is 2MB vector attachments; FilePath attachment(exeDir + "/path/to/attachment.txt"); attachments.push_back(attachment); // Enable automated crash uploads Settings *settings = database->GetSettings(); if (settings == NULL) return false; settings->SetUploadsEnabled(true); // Start crash handler CrashpadClient *client = new CrashpadClient(); bool status = client->StartHandler(handler, reportsDir, metricsDir, url, annotations, arguments, true, true, attachments); return status; } ``` Next, implement the platform-specific `getExecutableDir` method. ### **macOS** ```cpp #include #include StringType getExecutableDir() { unsigned int bufferSize = 512; vector buffer(bufferSize + 1); if (_NSGetExecutablePath(&buffer[0], &bufferSize)) { buffer.resize(bufferSize); _NSGetExecutablePath(&buffer[0], &bufferSize); } char *lastForwardSlash = strrchr(&buffer[0], '/'); if (lastForwardSlash == NULL) return NULL; *lastForwardSlash = 0; return &buffer[0]; } ``` ### **Linux** ```cpp #include #include #define MIN(x, y) (((x) < (y)) ? (x) : (y)) StringType getExecutableDir() { char pBuf[FILENAME_MAX]; int len = sizeof(pBuf); int bytes = MIN(readlink("/proc/self/exe", pBuf, len), len - 1); if (bytes >= 0) { pBuf[bytes] = '\0'; } char* lastForwardSlash = strrchr(&pBuf[0], '/'); if (lastForwardSlash == NULL) return NULL; *lastForwardSlash = '\0'; return pBuf; } ``` ### Windows ```cpp StringType getExecutableDir() { HMODULE hModule = GetModuleHandleW(NULL); WCHAR path[MAX_PATH]; DWORD retVal = GetModuleFileNameW(hModule, path, MAX_PATH); if (retVal == 0) return NULL; wchar_t *lastBackslash = wcsrchr(path, '\\'); if (lastBackslash == NULL) return NULL; *lastBackslash = 0; return path; } ``` Finally, call the `initializeCrashpad` method at the entry point of the application. ```cpp int main(int argc, char *argv[]) { initializeCrashpad(); } ``` ## Generating Symbols Generating sym files requires the `dump_syms` tool from the repository of Crashpad’s predecessor, [Breakpad](https://chromium.googlesource.com/breakpad/breakpad/). Dump\_syms creates [sym files](https://github.com/google/breakpad/blob/master/docs/symbol_files.md) from executable binaries, which allow minidumps to be symbolicated with function names, file names, and line numbers in the call stack. ```bash mkdir ~/breakpad cd breakpad fetch breakpad ``` ### **macOS** Build the application with symbolic information (preferably in a separate dSYM file) in order to get fully symbolicated crash reports. Next, build the Xcode project located at `src/src/tools/mac/dump_syms/dump_syms.xcodeproj`. Switch the configuration to dump\_syms and build the project. The report navigator tab (icon looks like a chat bubble in Xcode 11) will show the file system location with the compiled executable. Run the dump\_syms executable. ```bash dump_syms -g "/path/to/myApp.app.dSYM" "/path/to/myApp.app/Contents/MacOS/myApp" > myApp.sym ``` ### **Linux** Build the application with symbolic information and a build identifier. Using `clang` this means building with the `-g` and `-Wl,--build-id` arguments. Next, run `./configure && make` in the Breakpad directory. This will generate `dump_syms`, `symupload` and `minidump_stackwalk`. ```bash dump_syms /path/to/myApp.out > myApp.out.sym ``` ### **Windows** The `dump_syms` functionality is built into the `symupload` utility and can be skipped if the application will be symbolicated remotely. Run `dump_syms` only if the application will be symbolicated locally or the sym file will be uploaded to a remote server via some means other than `symupload`. For `symupload.exe` to generate the correct output, applications must be built with symbolic information so that each `exe` and `dll` file generates a corresponding `pdb` file. Generated `pdb` files must contain full debug information. With Visual Studio, full debug information can be generated with the `/Zi` compiler argument and the `/DEBUG:FULL` linker argument. Failure to specify either the `/Zi` or `/DEBUG:FULL` arguments will result in dump\_syms outputting incorrect sym file data. Additionally, the output `pdb` file must be in the same folder as the corresponding `exe` or `dll` file otherwise `dump_syms.exe` will fail. To run `symupload.exe` a copy of `msdia140.dll` must be placed in the same folder. If [Visual Studio](https://docs.microsoft.com/en-us/visualstudio/productinfo/2017-redistribution-vs#dia-sdk) is installed this file can be found at `[VisualStudioFolder]\DIA SDK\bin\amd64\msdia140.dll`. Copy `msdia140.dll` into the same folder as `symupload.exe` and run `symupload.exe`. ```bash symupload.exe "path/to/myApp.exe" "https://fred.bugsplat.com/post/bp/symbol/breakpadsymbols.php?appName=myApp&appVer=1.0.0" ``` Dump\_syms can be built from source so that the debugger can be used for troubleshooting. To build `dump_syms` clone the [gyp repository](https://github.com/chromium/gyp) and run `python setup.py install`. Next, run `gyp ~\breakpad\src\tools\windows\dump_syms\dump_syms.gyp` and use Visual Studio to build the sln file generated by gyp. If the build fails with an error that unique\_pointer is not part of std add `#include ` to the top of the file that contains the error and rebuild. ## Uploading Symbols The `symupload` tool is also part of the [Breakpad](https://chromium.googlesource.com/breakpad/breakpad/) repository and can be used to upload sym files to your server. Symbols need to be uploaded to a server in order for it to correctly symbolicate a minidump file. ### **macOS** Build the Xcode project located at `src/src/tools/mac/symupload/symupload.xcodeproj`. The report navigator tab (icon looks like a chat bubble in Xcode 11) will show you the file system location with the compiled executable. Copy the symupload file into your project and run the executable wrapping the arguments to symupload in quotes. ```bash symupload "/path/to/myApp.sym" "https://fred.bugsplat.com/post/bp/symbol/breakpadsymbols.php?appName=myApp&appVer=1.0.0" ``` ### **Windows** To use `symupload` , applications must be built with symbolic information so that each `exe` and `dll` file generates a corresponding `pdb` file. Generated `pdb` files must contain full debug information. Full debug information can be generated with the `/Zi` compiler argument and the `/DEBUG:FULL` linker argument. Failure to specify either the `/Zi` or `/DEBUG:FULL` arguments will result in `symupload` failing. The output `pdb` file must be in the same folder as the corresponding `exe` or `dll` file. To run `symupload.exe` a copy of `msdia140.dll` must be placed in the same folder. If [Visual Studio](https://docs.microsoft.com/en-us/visualstudio/productinfo/2017-redistribution-vs#dia-sdk) is installed, this file can be found at `[VisualStudioFolder]\DIA SDK\bin\amd64\msdia140.dll`. Copy msdia140.dll into the same folder as symupload.exe and run symupload.exe. ```bash symupload.exe "/path/to/myApp.exe" "https://fred.bugsplat.com/post/bp/symbol/breakpadsymbols.php?appName=myApp&appVer=1.0.0" ``` Symupload can be built from source so that the debugger can be used for troubleshooting. To build `symupload` clone the [gyp repository](https://github.com/chromium/gyp) and run `python setup.py install`. Next, run `gyp ~\breakpad\src\tools\windows\symupload\symupload.gyp` and use Visual Studio to build the sln file generated by `gyp`. If the build fails with an error that unique\_pointer is not part of std add `#include ` to the top of the file that contains the error and rebuild. ## Symbolicating Crash Reports `Minidump_stackwalk` is another tool in the [Breakpad](https://chromium.googlesource.com/breakpad/breakpad/) repository that is responsible for the symbolication of minidump files. In order to correctly symbolicate minidumps, sym files need to be nested at least 2 folders deep. The topmost parent folder’s name must equal the sym files module name. The first child folder’s name must equal the module id. Additionally, the sym file name must also match the module name. The module id and module name can be found in the [module record](https://github.com/google/breakpad/blob/master/docs/symbol_files.md#module-records) of the sym file. For example the module `myApp` with the module id `1A67F3DEAACA3B209D9992871B2620AA0` must be located at `/path/to/symbols/myApp/1A67F3DEAACA3B209D9992871B2620AA0/myApp.sym`. ### **macOS & Linux** Minidump\_stackwalk is built when building [Breakpad](https://chromium.googlesource.com/breakpad/breakpad#getting-started-from-master). ``` cd ~/breakpad/src && ./configure && make ``` ### **Windows** Minidump\_stackwalk can be built on Windows with the help of [MinGW-w64](https://www.mingw-w64.org/) and [TDM-GCC](https://jmeubank.github.io/tdm-gcc/). First, download and install [TDM-GCC](https://jmeubank.github.io/tdm-gcc/download/). Next, open a MINGw64 shell, such as [Git Bash](https://gitforwindows.org/). ```bash cd ~/breakpad/src && ./configure --disable-dependency-tracking && make ``` If make doesn't work, try removing all instances of `/usr/bin` from `Makefile`. Run `minidump_stackwalk` passing it a path to a `.dmp` file and a symbols directory. The folders in the symbols directory need to be laid out following the pattern `module_name/module_id/module_name.sym`: ``` minidump_stackwalk -m "/path/to/minidump.dmp" "/path/to/symbols" ``` Additionally, minidumps generated by the Crashpad library can also be symbolicated via [Debugging Tools](https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/) for Windows given the `.exe`, `.dll` and `.pdb` files are made available to WinDBG. ## Troubleshooting Most issues symbolicating dump files can be traced back to mismatched module ids. Anything that modifies a given executable (code-signing, anti-cheat) must be performed before `dump_syms` and `symupload` are run. Every time an executable is modified `dump_syms` and `symupload` need to be re-run. The name and id of the module loaded at runtime can be found in the `minidump_stackwalk` output. The module name and id must match the module name and id of the generated sym file in order for `minidump_stackwalk` to correctly symbolicate the minidump file. ## References 1. 2. 3. 4. 5. 6. 7.