Emptyhammock — Apache httpd 2.x exception hook, mod_whatkilledus, mod

This document was hosted from my apache.org directory for many years. Latest information: https://emptyhammock.com/projects/httpd/diag/

a couple of modules that use the exception hook added in Apache 2.0.49

These can be used in conjunction or individually or simply cut up into pieces as you see fit. They should work fine on Apache 2.0 >= 2.0.49, but mod_whatkilledus currently needs the test_char.h file from an Apache 2.1-dev build.

You must specify the --enable-exception-hook configure option and also add EnableExceptionHook on to your httpd conf file in order to allow modules to run after fatal exceptions (synchronous signals on Unix).

mod_backtrace is limited to Linux and FreeBSD at present, unfortunately, but mod_whatkilledus has been used on a variety of Unix-like systems.

mod_whatkilledus

inspired by an ugly problem in 1.3 where we couldn't get a core dump and had no idea what type of request led to the segfault... only thing I could think to do was give them a module that wrote a log message near the beginning of the request to give all sorts of information about the request (url, ssl?, virtual host, etc.) in hopes of finding a pattern... but of course this filled up a big file

mod_whatkilledus keeps a little bit of state on each active connection, which allows it to know what a thread was handling in case the thread segfaults. If that happens, it writes a report to the error log

Here is an example report:

[Tue Mar 16 19:11:44 2004] pid 29412 mod_whatkilledus sig 11 crash
[Tue Mar 16 19:11:44 2004] pid 29412 mod_whatkilledus active connection: 9.65.120.97:2035->9.27.177.26:8080 (conn_rec 434b90)
[Tue Mar 16 19:11:44 2004] pid 29412 mod_whatkilledus active request (request_rec 438b48):
GET /silly?fn=sigsegv HTTP/1.1|Host:aplinux.raleigh.ibm.com%3a8080|
User-Agent:Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv%3a1.5)
Gecko/20031007|Accept:text/xml,application/xml,application/xhtml+xml,
text/html;q=0.9,text/plain;q=0.8,image/png,image/jpeg,image/gif;q=0.2
,*/*;q=0.1|Accept-Language:en-us,en;q=0.5|Accept-Encoding:gzip,deflat
e|Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.7|Keep-Alive:300|Connec
tion:keep-alive|Cache-Control:max-age=0
[Tue Mar 16 19:11:44 2004] pid 29412 mod_whatkilledus end of report

It might be interesting to know what other threads in the process were doing too. They probably didn't cause the problem, but the server crashed in the middle of that activity, which could conceivably have future repercussions. Hopefully a future revision of mod_whatkilledus will have a configuration directive that specifies that information for all threads should be logged, not just information for the thread that crashed.

mod_backtrace

here's another trick for situations where you can't get a core dump and/or can't get a backtrace

mod_backtrace is a module (currently just for Linux and FreeBSD) which uses system functions to format a backtrace to show what code did the dirty deed. Unfortunately the exact format and unfortunately the usefulness varies among glibc versions. Here is how it can look on Redhat Advanced Server 2.1:

[Tue Mar 16 19:11:44 2004] pid 29412 mod_backtrace backtrace for sig 11 (thread "pid" 29438)
[Tue Mar 16 19:11:44 2004] pid 29412 mod_backtrace main() is at 8071048
/home/trawick/20471fvt/modules/mod_backtrace.so[0x4006faee]
/home/trawick/20471fvt/bin/httpd(ap_run_ihs_fatal_exception_received+0x2e)[0x8069fc6]
/home/trawick/20471fvt/bin/httpd[0x806a1d0]
/lib/i686/libpthread.so.0[0x4007aac5]
/lib/i686/libc.so.6[0x40132958]
/home/trawick/20471fvt/modules/mod_silly2.so[0x402b7c4c]
/home/trawick/20471fvt/bin/httpd(ap_run_handler+0x2a)[0x806d3e6]
/home/trawick/20471fvt/bin/httpd(ap_invoke_handler+0xc1)[0x806d901]
/home/trawick/20471fvt/bin/httpd(ap_process_request+0x37)[0x806929b]
/home/trawick/20471fvt/bin/httpd(apr_vformatter+0x72c)[0x80652e8]
/home/trawick/20471fvt/bin/httpd(ap_run_process_connection+0x2a)[0x80751ee]
/home/trawick/20471fvt/bin/httpd[0x806a64f]
/home/trawick/20471fvt/bin/httpd[0x806ac22]
/home/trawick/20471fvt/lib/libapr-0.so.0[0x4005bf34]
/lib/i686/libpthread.so.0[0x40077c6f]
/lib/i686/libc.so.6(__clone+0x3a)[0x401f1afa]
[Tue Mar 16 19:11:44 2004] pid 29412 mod_backtrace end of backtrace

I've seen just a series of hex addresses on an older SuSE distro on zSeries. The address of main() is logged too, so enterprising individuals will be able to make sense of those hex addresses by using "nm -n httpd" to display httpd symbols. For functions which are part of httpd, the hex addresses in the backtrace will correspond to instructions within the range of a function displayed by nm. (Enterprising zSeries folks will of course need to ignore the high-order bit of the addresses in the backtrace.)

I don't know how DSOs are laid out in storage, and mod_backtrace doesn't display any known addresses for the DSOs to give a hint. Maybe mod_backtrace needs to have an option to walk through the loaded DSOs and spit out the module name and some associated address to make this straightforward.