ruby passenger taken out by httpd logrotate?!?

I’m having a problem where running logrotate on apache httpd causes Passenger to sometimes (but not every time) become unresponsive (with errors in the apache logs I’ll paste below), in such a way that even running `httpd restart` after that point isn’t enough to bring passenger back — have to find and hard kill the passenger processes, and/or just reboot the server.

I’ll paste more complete details below — but I’m a bit surprised to have this problem in otherwise very stable and robust adn reliable Passenger. Especially because searching in this list and googling around, it’s a problem at least some people have been reporting (although with different error messages than I’m seeing) for years. I’m surprised Phusion hasn’t fixed it yet. Is this a problem that actually only reports a tiny minority of people in as of yet undetermined edge cases, and I’m just one of the unlucky ones, perhaps due to my specific setup? ¬†Or is this really a problem waiting for anyone, that probably ought to be documented/advertised better?

While I haven’t been through enough logrotates to be sure yet, I suspect changing logrotate to a ‘copytruncate’ _without_ an httpd restart will keep the problem from occuring.

My situation:

CentOS 5.8
Apache/2.2.3

ruby used by passenger is one installed by rbenv in a system location:

LoadModule passenger_module /usr/local/rbenv/versions/1.9.3-p125/lib/ruby/gems/1.9.1/gems/passenger-3.0.12/ext/apache2/mod_passenger.so
PassengerRoot /usr/local/rbenv/versions/1.9.3-p125/lib/ruby/gems/1.9.1/gems/passenger-3.0.12
PassengerRuby /usr/local/rbenv/versions/1.9.3-p125/bin/ruby

logrotate config that causes the problems:

/opt/my_app/current/log/*.log {
missingok
compress
weekly
rotate 5
postrotate /sbin/service httpd restart > /dev/null 2>/dev/null || true
endscript
}

Sometimes but not always after log rotate (however as often as 25% of the time I’d think), passenger becomes unresponsive after logrotate, http requests to passenger-supported URLs just timeout with no response. And the httpd log:

[Sun Nov 25 04:02:03 2012] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations

followed by ONE of these:

/usr/bin/env: ruby: No such file or directory

Followed by a bunch of these:

[ pid=18821 thr=47690455569248 file=ext/apache2/Hooks.cpp:862 time=2012-11-25 04:02:41.202 ]: Unexpected error in mod_passenger: Cannot connect to Unix socket '/tmp/passenger.1.0.12709/generation-1/socket': No such file or directory (2)
Backtrace:
in 'Passenger::ApplicationPool::Client* Passenger::ApplicationPool::Client::connect(const std::string&, const std::string&, const Passenger::StaticString&)' (Client.h:438)
in 'Passenger::ApplicationPool::Client* Hooks::getApplicationPool()' (Hooks.cpp:268)
in 'Passenger::SessionPtr Hooks::getSession(const Passenger::PoolOptions&)' (Hooks.cpp:294)
in 'int Hooks::handleRequest(request_rec*)' (Hooks.cpp:563)

I don’t know if the `/usr/bin/env ruby` thing is at the heart of the problem (somehow when logrotate restarts apache passenger loses it’s ruby, in a way that ordinary manual restarts don’t? No clue in the log where that line is coming from!), or is an unrelated thing.

Once this problem happens, manual httpd restart isn’t even enough to recover, one needs to reboot the server (or presumably find and kill all the passenger processes?)

What I assume will fix it

We have another apache/passenger on another server that uses a more conservative logrotate that does copytruncate only, with no httpd restart, and does not seem to be having the problem. I figure using the same approach on the server having the problem will fix things (how could it not?) — but am still puzzled by the out-of-character bug in passenger — the restart logrotate seems to be more typically used for apache, and it’s odd if passenger has bugs with it.

/opt/my_app/current/log/*.log {
        missingok
        compress
        weekly
        rotate 5
        copytruncate
}
About these ads
This entry was posted in General. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s