mod_whatkilledus and mod_backtrace

What are they for, briefly?

Log information which can help determine the cause of a web server child process crash and allow the circumstances of different crashes to be quickly compared in order to determine if the crashes need to be investigated individually.

What's new with version 2.00?

Version 2.00 is a complete rewrite of earlier modules of the same names which provided similar functionality.

The original versions of the modules were written by me and donated to the Apache HTTP Server Project back in 2004, and committed to the Apache HTTP Server 1.3 source tree. Similar modules for Apache HTTP Server 2.x were not accepted into the 2.x tree but instead have been distributed from http://people.apache.org/~trawick/exception_hook.html for many years. They have also been available from other commercial and non-commercial distributions.

Basic differences with mod_whatkilledus and mod_backtrace v2.00 from the original versions

• mod_whatkilledus and mod_backtrace now support the Windows platform. mod_backtrace now supports Mac OS X and Solaris.
• mod_backtrace can be used to annotate error log messages with a backtrace if certain configurable conditions are met.
• mod_whatkilledus always writes crash reports to its own log file; writing to the server error log is not an option.
• mod_whatkilledus allows sensitive information in the request to be obscured in the crash report in order to protect that information in cases where access to the log files is not carefully restricted.   More information...
• Where applicable (Unix), mod_whatkilledus ensures that the EnableExceptionHook directive is properly set to allow crash reports.
• mod_whatkilledus no longer makes assumptions about the connection handled by the current thread, which could be incorrect when using the event MPM.

In addition there are a number of implementation differences intended to improve various aspects of the software in less user-visible ways.

Quick start

• Quick start guide

Download

• Download wku_bt-2.00.zip

mod_whatkilledus v2.00 overview

mod_whatkilledus generates crash reports when a web server thread crashes. The crash reports include:

• technical information about the crash
• backtrace (if mod_backtrace is available and loaded)
• details of the client request which triggered the crash

mod_whatkilledus documentation

mod_backtrace v2.00 overview

mod_backtrace can be used in two ways:

• Annotate error log with backtraces when certain conditions are met.
• Provide an API to other module for obtaining a backtrace. mod_whatkilledus uses this API, which is how crash reports may include a backtrace of the crashing thread when mod_backtrace is loaded.

mod_backtrace documentation

General caveats with v2.00

Building from source

Users of older or non-mainstream Unix-like systems may encounter unanticipated build failures. Binaries are provided for Windows, but Windows users who compile the source themselves may also encounter unanticipated build failures based on the level of SDK or compiler in use. Report any build issues.

Runtime behavior

Defects or different behaviors in the OS-provided code may differ on levels of the OS which I have not tested, and may require modifications to the software to utilize that OS level effectively. A verification procedure is provided for both mod_whatkilledus and mod_backtrace to ensure that they are functioning properly on your system, and I recommend following that procedure if your system differs from the test platforms described below. Report any operational problems.

Test platforms

Please report other successfully tested platforms so that I can list them here.

Special testing notes

• Mac OS X 10.6.8 — 2.2.latest with worker MPM
  The modules apparently did what they were supposed to, but while load testing this build I frequently encountered conditions where httpd child processes would lock up in spin loops with or without these modules, so testing was inconclusive.

Are you still using Apache httpd 2.0.x?

I'm not. I haven't tested the modules with 2.0.x, though I don't think there would be significant changes required.

Cost of use or distribution

This software is free — there is no cost to use or distribute this software either as-is or after modification to suit your purposes. At the same time, the author provides this software as-is, without any warranty. The license for this software is the Apache License, Version 2.0, and the license contains details about your rights and obligations as a user or distributor of this software. In particular, redistributors must comply with certain reasonable obligations described in the license.

• http://www.apache.org/licenses/LICENSE-2.0.html

Groups wishing to incorporate this software into their product for redistribution should not rely on my advice but instead should work with their legal counsel to make a final determination. In the event that specific questions need to be discussed with the author, use this contact information.

Credit

mod_whatkilledus currently contains a small bit of code from mod_log_forensic as well as the contents of test_char.h, both part of Apache HTTP Server.

Future plans

Here are some possible work items for the future:

Windows

• Prepare 64-bit binaries for Apache httpd 2.2 and 2.4.
• Add more information about the exception to the mod_whatkilledus log.
• Obtain the module name for each frame in the backtrace.

Unix

• Implement low-level APIs to control core dumps, and exploit those APIs in mod_whatkilledus in order to provide such features as informing about core dump enablement, limiting the number of core dumps created, etc.
• Update low-level APIs to include more information about the cause of the crash. These can be enabled in the modules for httpd after future changes to the httpd APIs.

General

• Clean up and formalize the low-level diag APIs for handling the system-specific details (crash description, backtrace) so that other programs may use them.
• Clean up and formalize the mod_backtrace APIs so that other Apache httpd modules can use them.
• Allow the mod_whatkilledus log file to be configured.
• Add more configurability for error log backtrace selection.
• Add more granular control over obscuring of request details that can be logged, such as obscuring an individual cookie or query argument and leaving others visible.

Desired changes to Apache httpd

• Windows: Implement the exception hook API. Currently mod_whatkilledus handles this internally on the Windows platform, but an exception hook would provide a framework for multiple modules to provide insight into crashes or clean up critical resources without conflicting with each other.
• Unix: Update httpd's signal handler setup and signal handler implementation to provide more information to exception hooks. Structures like siginfo_t and ucontext_t could provide more information about the cause of the crash and, in some limited circumstances, allow a better backtrace to be generated.

If you'd like to sponsor specific enhancements whether or not they are listed above, please let me know.

Contact

Reports of problems or success or other inquiries about the software may be sent to the following e-mail address:

Jeff Trawick
info@emptyhammock.com

Please include Diagnostic modules in the subject of your e-mail and provide the following information:

• operating system and version
• CPU architecture
• version of Apache httpd, or name and version of other web server based on Apache httpd
• any other third-party modules in use
• compiler used for this software, or where binaries were obtained
• compiler used for web server, or where binaries were obtained
• for build failures, show invocation and complete output of build
• for operational failures, show any incorrect output or other failures

This is a free service, so I may not be able to reply individually; however, any information you are able to provide about problems will be used to improve this software in the future.

If you'd like to sponsor specific enhancements or resolution of specific problems, please let me know.