On 16 Jun 2014, at 12:16, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
On 16/06/14 11:51, Phil Mayers wrote:
We've got a version of 3.0.x HEAD running on our test server. It's sporadically hanging - stopping answering requests.
I've caught a process doing it - backtrace here:
https://gist.github.com/philmayers/5582ca9fb89f373919b5
Looks like it's actually faulted, but the fault handler stuff is stuck on a mutex - inside talloc?
Huh, looks like backtrace() is actually calling malloc().
Yes.
Oh dear:
What a complete and utter shithead.
I'd love to say I'm surprised by the tone in the comments of that bug, but I've read enough glibc bugs to know that it was a common problem in the mid-2000s...
See however the far more reasonable and clueful:
https://sourceware.org/bugzilla/show_bug.cgi?id=16159
Basically - backtrace() not safe to call from signal handlers by the looks of it? Glumness.
I've added the suggested workaround for __GLIBC__ only, with the assumption that other versions of backtrace (OSX and FreeBSD have their own) are sane and don't allocate heap memory... because that's a spectacularly stupid thing to do. That's why we use backtrace_symbols_fd FFS (to avoid allocating strings for symbol names).
Still not clear why the process is SEGVing - vp->length is 3, talloc_array should not fail. I guess memory corruption of some sort?
Maybe a bad parent, which is weird... valgrind? Could you also try reverting 6075844989ca8301a2743aed06ddd95382a6e78b Which i'd say is the most likely candidate for introducing this bug, if it's only started appearing relatively recently. Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS Development Team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2