Emptyhammock — Apache httpd 1.3 exception hook, mod_whatkilledus, mod

This document was hosted from my apache.org directory for many years. Latest information: https://emptyhammock.com/projects/httpd/diag/

an exception hook for Apache 1.3 and a couple of modules that use it

These can be used in conjunction or individually or simply cut up into pieces as you see fit *on test systems*. It does appear to work :) If you find a problem, please let me know.

activating the hook

The required changes to Apache httpd 1.3 are in 1.3.30-dev. Wait for 1.3.30 to be released or check it out from cvs or pick up a daily snapshot.

You'll need to build Apache with the symbol AP_ENABLE_EXCEPTION_HOOK defined. This will allow plug-in modules to implement a call-back which will be called when a synchronous signal is generated.

Note: The symbol is already defined for AIX, Linux, and Solaris.

OPTIM=-DAP_ENABLE_EXCEPTION_HOOK ./configure --other-opts

The web server configuration file must also be changed to include

EnableExceptionHook On

at global scope. By default, exception hooks don't get to run just because the required support is compiled into the server.

mod_whatkilledus

Note: This is now in Apache 1.3's modules/experimental directory. See the file there for directions on how to activate it.

inspired by an ugly problem in 1.3 where we couldn't get a core dump and had no idea what type of request led to the segfault... only thing I could think to do was give them a module that wrote a log message near the beginning of the request to give all sorts of information about the request (url, ssl?, virtual host, etc.) in hopes of finding a pattern... but of course this filled up a big file

mod_whatkilledus keeps a little bit of state on each active connection, which allows it to know what a thread was handling in case the thread segfaults. If that happens, it writes a report to a log file.

Required configuration:

Here is an example entry written after a segfault:

(The data written by the latest level of code is actually a little different, but you get the idea ;))

[Wed Jan 21 10:38:32 2004] pid 2211 sig 11 crash
[Wed Jan 21 10:38:32 2004] pid 2211 active connection:
192.168.1.11:32894->192.168.1.11:8080
[Wed Jan 21 10:38:32 2004] pid 2211 active request:
GET /silly?sigsegv
HTTP/1.1|Accept:text/xml,application/xml,application/xh
tml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,imag
e/jpeg,image/gif;q=0.2,*/*;q=0.1|Accept-Charset:ISO-885
9-1,utf-8;q=0.7,*;q=0.7|Accept-Encoding:gzip,deflate|Ac
cept-Language:en-us,en;q=0.5|Connection:keep-alive|Host
:sol9:8080|Keep-Alive:300|User-Agent:Mozilla/5.0 (X11; 
U; SunOS i86pc; en-US; rv:1.5) Gecko/20031020
-----

mod_backtrace

Note: This is now in Apache 1.3's modules/experimental directory. See the file there for directions on how to activate it.

This is another diagnostic trick for situations where you can't get a core dump and/or can't get a backtrace.

mod_backtrace is a a module currently limited to Linux and FreeBSD which uses system library functions to format a backtrace to show what code did the dirty deed. By default the information is written to the error log, but a directive is provided for setting the name of a log file to which the backtrace will be appended.

Unfortunately the exact format and unfortunately the usefulness of the backtrace varies among library versions. Here is how it can look on Fedora Core 1. (Some experimentation with removing the httpd symbol stripping from the Apache install would be useful.)

[Mon Jan 19 17:09:10 2004] pid 10735 backtrace for signal 11
[Mon Jan 19 17:09:10 2004] pid 10735 main() is at 8076b93
/tmp/13/libexec/mod_backtrace_13.so[0x6f7ca2]
/tmp/13/libexec/mod_backtrace_13.so[0x6f7d3f]
/tmp/13/bin/httpd[0x8074618]
/tmp/13/bin/httpd[0x8074658]
/lib/tls/libc.so.6[0x15aa18]
/tmp/13/bin/httpd(ap_invoke_handler+0x102)[0x806a043]
/tmp/13/bin/httpd[0x807eaf7]
/tmp/13/bin/httpd(ap_process_request+0x30)[0x807eb56]
/tmp/13/bin/httpd[0x8075d5a]
/tmp/13/bin/httpd[0x8075fba]
/tmp/13/bin/httpd[0x8076060]
/tmp/13/bin/httpd[0x80766fd]
/tmp/13/bin/httpd(main+0x388)[0x8076f1b]
/lib/tls/libc.so.6(__libc_start_main+0xf0)[0x148750]
/tmp/13/bin/httpd(dlopen+0x41)[0x804f215]
[Mon Jan 19 17:09:10 2004] pid 10735 end of backtrace

I've seen just a series of hex addresses on an older SuSE distro on zSeries. The address of main() is logged too, so enterprising individuals will be able to make sense of those hex addresses by using "nm -n httpd" to display httpd symbols. For functions which are part of httpd, the hex addresses in the backtrace will correspond to instructions within the range of a function displayed by nm. (Enterprising zSeries folks will of course need to ignore the high-order bit of the addresses in the backtrace.)

I don't know how DSOs are laid out in storage, and mod_backtrace_13 doesn't display any known addresses for the DSOs to give a hint. Maybe mod_backtrace_13 needs to have an option to walk through the loaded DSOs and spit out the module name and some associated address to make this straightforward.

Required configuration:

LoadModule backtrace_13_module libexec/mod_backtrace_13.so
BTFile /tmp/mod_backtrace_log

HP-UX offers a U_STACK_TRACE() function but I have not been able to get it to work from mod_backtrace_13. My notes:

The U_STACK_TRACE() function lives in libcl.sl in 11i 1.0. That library, or a shared library that links to it directly, cannot be loaded into Apache 1.3:

/usr/lib/dld.sl: Can't shl_load() a library containing Thread Local Storage:   
    /usr/lib/libcl.2
/usr/lib/dld.sl: Exec format error
Syntax error on line 1025 of /home/trawick/inst/13/conf/httpd.conf:
Cannot load /home/trawick/inst/13/libexec/mod_backtrace_13.so into server:
    Exec format error

Thus mod_backtrace_13 and HP-UX is on hold for now. Possibly the support can be implemented in the mod_backtrace for Apache 2, but I have not investigated that yet. Also, it may be possible to dynamically load U_STACK_TRACE() using a function other than shl_load(), but I have not investigated that yet either.