httpd improvements made as part of this project

  • mod_proxy_scgi: minor improvements to documentation
    trunk: r1589413, 2.4.x: r1589417
  • mod_proxy_fcgi: don't crash when connect to the backend fails
    trunk: r1590437, 2.4.10: r1592998
  • mod_proxy_fcgi: remove wasted memset() calls and other historical bits
    Just a minor performance improvement.
    trunk: r1591472, 2.4.10: r1594537
  • mod_proxy_fcgi: respect iobuffersize parameter on worker definition
    Not to be used indiscriminately (extra memory for I/O buffers may reduce overall performance).
    trunk: r1591508+r1603027+r1603029+1603122, 2.4.10: r1601749+1603140
  • mod_proxy_fcgi: better error messages
    trunk: r1592032, 2.4.10: r1604369
  • mod_proxy_fcgi: fix occasional high CPU when handling request bodies
    trunk: r1592037, 2.4.10: r1605642
  • proxy_util: code cleanups
    trunk: r1592511+r1592514, 2.4.10: r1598605
  • mod_proxy_scgi: support Unix sockets
    trunk: r1592529, 2.4.10: r1593004

uWSGI improvements made as part of this project

  • Fix uWSGI FastCGI protocol handler to read the FCGI_STDIN eof record
    Fixed with this commit.
  • Eventually fix uWSGI FastCGI protocol handler (and maybe others) to implement lingering close over TCP
    Look at the fifth and later posts in this thread.
    See this short-term fix. (I think this fix consumes uWSGI threads longer than is necessary. Hopefully the application architecture supports reading the data and finally closing the socket asynchronously.)

Problems to investigate

These might be bugs in the implementations of the web server or uWSGI or whatever else, or configuration issues.

1. uWSGI doesn't always read FCGI_STDIN

At the end of writing the request body (if any) to the application, the gateway writes an empty FCGI_STDIN record, which indicates that no more bytes are available. AFICT uWSGI doesn't read this unless the application tries to read the request body.

httpd with mod_proxy_fcgi gets bit by this problem because it writes the empty FCGI_STDIN in a separate syscall, and uWSGI may have already closed the socket. When that timing scenario occurs, mod_proxy_fcgi can get EPIPE from the writev() and fail the request, returning a gateway error. The problem occurs much more frequently with Unix sockets than with TCP sockets, and more frequently with multiple threads configured for the WSGI application.

This doesn't affect nginx because it sends the empty FCGI_STDIN in the same buffer as the FastCGI parameters (envvars), so uWSGI is always still waiting for that.

See this blog post for a tiny mod_proxy_fcgi patch to work around this.

2. nginx returns 200 response with empty body after I/O error with uWSGI

This seems odd, but maybe it is on purpose.

If you send a request body that causes the data sent by nginx to uWSGI to exceed 4K (i.e., multiple writes) and the application doesn't try to read the data, nginx sees an error writing the rest but the client sees a 200 (OK) response with empty body. (With the uWSGI problem and httpd, the client sees a gateway error.) I've seen other error scenarios where nginx sent a 200 response with an empty body.

httpd's proxy will take the drastic action of sending no response in some error cases, but that is to get the browser to retry.

3. uWSGI doesn't implement lingering close, resulting in intermittent TCP errors

Look at the fifth and later posts in this thread. Hopefully this pull request will result in the uWSGI folks doing a proper implementation.

4. nginx not reading all of app response over slow Wi-Fi LAN

This is weird. strace on uWSGI (on the other side of a slow Wi-Fi LAN from nginx) says it wrote 71544 bytes, and strace on nginx says it read 71544 bytes, but the client got a short response, and nginx logged this:

2014/05/01 15:35:38 [error] 32092#0: *3 upstream timed out (110: Connection timed out)
while reading upstream, client: 127.0.0.1, server: , request: "POST / HTTP/1.0",
upstream: "fastcgi://192.168.1.211:2003", host: "127.0.0.1"
      

I did not see nginx try another read (after reading as many bytes as uWSGI sent). Maybe I'll find time to play with this more.

5. frequent crashes with third-party module mod_proxy_uwsgi

not investigated yet

At present, I don't know that I want to get too involved with mod_proxy_uwsgi, in case I want to start with mod_proxy_scgi and write an ASL-licensed version myself for potential inclusion in the httpd project. But for now it doesn't seem very important.