mod_whatkilledus and mod_backtrace

Code on GitHub   

What are they for, briefly?

Log information which can help determine the cause of a web server child process crash and allow the circumstances of different crashes to be quickly compared in order to determine if the crashes need to be investigated individually.

What's new with version 2.01? (August 1, 2014)

mod_whatkilledus improvements

• The name of the log file can now be configured with the WKULogfile directive.
• On Windows: Add directive WKUDisableWindowsErrorBox to allow disabling the pop-up error handling dialog after a child process crash.
• On Windows: Handle exceptions in 64-bit builds.
• With Event MPM on Unix: When building with httpd 2.4.10 or later, properly track request processing across threads with the Event MPM. Previously, the wrong client and/or request might be logged as the trigger for a crash (with the Event MPM only). The change is not effective when building with httpd 2.4.9 or earlier.

mod_backtrace improvements

• Support libunwind on Linux, FreeBSD, and OS X. This provides better resolution of function names, so a raw address will be provided in lieu of the function name in fewer situations. (Add LIBUNWIND=yes to the make invocation.) OS X note: A libunwind build silently fails to capture a backtrace with 32-bit builds, so use the default mechanism with 32-bit builds on OS X.
• Filter out internal functions from backtraces so that only the caller's stack frames are reported.
• Improve formatting of a backtrace for the error log field.

Windows binaries

64-bit binaries are now included for use with httpd 2.2 and 2.4 on Windows. (32-bit binaries are still included.)

Refer to the Quick Start guide for information about a required software dependency.

Build improvements

• Support make APXS=/path/to/apxs for building with non-default install layouts.
• Support Clang on FreeBSD 10.
• Add -funwind-tables on ARM platform for better backtraces.
• mod_whatkilledus fails to build in a less mysterious way when using an httpd build that doesn't have exception hooks enabled.

Quick start

• Quick start guide

Download

• Download wku_bt-2.01.zip

mod_whatkilledus v2.01 overview

mod_whatkilledus generates crash reports when a web server thread crashes. The crash reports include:

• technical information about the crash
• backtrace (if mod_backtrace is available and loaded)
• details of the client request which triggered the crash

mod_whatkilledus documentation

mod_backtrace v2.01 overview

mod_backtrace can be used in two ways:

• Annotate error log with backtraces when certain conditions are met.
• Provide an API to other module for obtaining a backtrace. mod_whatkilledus uses this API, which is how crash reports may include a backtrace of the crashing thread when mod_backtrace is loaded.

mod_backtrace documentation

Are you still using Apache httpd 2.0.x?

I'm not. I haven't tested the modules with 2.0.x, though I don't think there would be significant changes required.

Cost of use or distribution

This software is free — there is no cost to use or distribute this software either as-is or after modification to suit your purposes. At the same time, the author provides this software as-is, without any warranty. The license for this software is the Apache License, Version 2.0, and the license contains details about your rights and obligations as a user or distributor of this software. In particular, redistributors must comply with certain reasonable obligations described in the license.

• http://www.apache.org/licenses/LICENSE-2.0.html

Groups wishing to incorporate this software into their product for redistribution should not rely on my advice but instead should work with their legal counsel to make a final determination. In the event that specific questions need to be discussed with the author, use this contact information.

Credit

mod_whatkilledus currently contains a small bit of code from mod_log_forensic as well as the contents of test_char.h, both part of Apache HTTP Server.

Contact

Reports of problems or success or other inquiries about the software may be sent to the following e-mail address:

Jeff Trawick
info@emptyhammock.com

Please include Diagnostic modules in the subject of your e-mail and provide the following information:

• operating system and version
• CPU architecture
• version of Apache httpd, or name and version of other web server based on Apache httpd
• any other third-party modules in use
• compiler used for this software, or where binaries were obtained
• compiler used for web server, or where binaries were obtained
• for build failures, show invocation and complete output of build
• for operational failures, show any incorrect output or other failures

This is a free service, so I may not be able to reply individually; however, any information you are able to provide about problems will be used to improve this software in the future.

If you'd like to sponsor specific enhancements or resolution of specific problems, please let me know.

---

What was new with version 2.00? (September 2012)

Version 2.00 is a complete rewrite of earlier modules of the same names which provided similar functionality.

The original versions of the modules were written by me and donated to the Apache HTTP Server Project back in 2004, and committed to the Apache HTTP Server 1.3 source tree. Similar modules for Apache HTTP Server 2.x were not accepted into the 2.x tree but instead have been distributed from http://people.apache.org/~trawick/exception_hook.html for many years. They have also been available from other commercial and non-commercial distributions.

Basic differences with mod_whatkilledus and mod_backtrace v2.00 from the original versions

• mod_whatkilledus and mod_backtrace now support the Windows platform. mod_backtrace now supports Mac OS X and Solaris.
• mod_backtrace can be used to annotate error log messages with a backtrace if certain configurable conditions are met.
• mod_whatkilledus always writes crash reports to its own log file; writing to the server error log is not an option.
• mod_whatkilledus allows sensitive information in the request to be obscured in the crash report in order to protect that information in cases where access to the log files is not carefully restricted.   More information...
• Where applicable (Unix), mod_whatkilledus ensures that the EnableExceptionHook directive is properly set to allow crash reports.
• mod_whatkilledus no longer makes assumptions about the connection handled by the current thread, which could be incorrect when using the event MPM.

In addition there are a number of implementation differences intended to improve various aspects of the software in less user-visible ways.