3.0.x HEAD hanging

[Arran Cudbard-Bell]

On 16 Jun 2014, at 12:16, Phil Mayers <p.mayers at imperial.ac.uk> wrote:

> On 16/06/14 11:51, Phil Mayers wrote:
>> We've got a version of 3.0.x HEAD running on our test server. It's
>> sporadically hanging - stopping answering requests.
>> 
>> I've caught a process doing it - backtrace here:
>> 
>> https://gist.github.com/philmayers/5582ca9fb89f373919b5
>> 
>> Looks like it's actually faulted, but the fault handler stuff is stuck
>> on a mutex - inside talloc?
> 
> Huh, looks like backtrace() is actually calling malloc().

Yes.

> Oh dear:
> 
>  https://sourceware.org/bugzilla/show_bug.cgi?id=956

What a complete and utter shithead.

> 
> I'd love to say I'm surprised by the tone in the comments of that bug, but I've read enough glibc bugs to know that it was a common problem in the mid-2000s...
> 
> See however the far more reasonable and clueful:
> 
>  https://sourceware.org/bugzilla/show_bug.cgi?id=16159
> 
> Basically - backtrace() not safe to call from signal handlers by the looks of it? Glumness.

I've added the suggested workaround for __GLIBC__ only, with the assumption
that other versions of backtrace (OSX and FreeBSD have their own) are sane and don't 
allocate heap memory... because that's a spectacularly stupid thing to do.

That's why we use backtrace_symbols_fd FFS (to avoid allocating strings for symbol 
names).

> Still not clear why the process is SEGVing - vp->length is 3, talloc_array should not fail. I guess memory corruption of some sort?

Maybe a bad parent, which is weird... valgrind? 

Could you also try reverting 6075844989ca8301a2743aed06ddd95382a6e78b

Which i'd say is the most likely candidate for introducing this bug, if it's only started appearing relatively recently.

Arran Cudbard-Bell <a.cudbardb at freeradius.org>
FreeRADIUS Development Team

FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 881 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freeradius.org/pipermail/freeradius-devel/attachments/20140616/aca95c37/attachment.pgp>