On 18 Nov 2015, at 07:47, Matthew Newton <mcn4@leicester.ac.uk> wrote:
Hi,
Just put our first real FR3 deployment live yesterday (3.0.x HEAD 46fb0cd312f0) :) Then quickly hit a crash :(
Initial outputs (was running -X to start so these are single threaded) gave:
... (294) eap_tls: ocsp: Cert status: good (294) eap_tls: ocsp: Certificate is valid (294) eap_tls: TLS_accept: SSLv3 read client certificate A (294) eap_tls: <<< recv TLS 1.0 Handshake [length 0046], ClientKeyExchange *** glibc detected *** freeradius: free(): invalid pointer: 0x0000000001c7a140 *** ======= Backtrace: ========= ...
and
... Waking up in 0.1 seconds. (4563) Cleaning up request packet ID 8 with timestamp +1917 (4564) Cleaning up request packet ID 9 with timestamp +1917 Waking up in 0.2 seconds. (4566) Cleaning up request packet ID 207 with timestamp +1917 *** glibc detected *** /usr/sbin/freeradius: free(): invalid next size (fast): 0x00000000019f4830 *** ======= Backtrace: ========= ...
Haven't manage to get a core dump yet, but the panic action is working. Three crashes so far are the same, one slightly different. All are in _talloc_free: The three same ones error in the request_free (process.c:806) at the end of request_done. The other is in eappeap_process (peap.c:828).
The fact it manages to get all the way through request_free without crashing, with all the checks that are done in there, before bombing out in the free seem slightly weird to me, as the data in *request must be sane.
System is Debian wheezy with talloc 2.0.7. I think my next plan is to upgrade the system to jessie which has version 2.1.1 available and see if it's a bug in that old version of talloc that's being hit. Weird nobody else has reported it before, though; it's not a busy server and crashed four times so far this morning (though not at all overnight when less lightly loaded).
Full startup output from -X and four gdb panic action dumps are at (yeah I know there are a few warnings in the config at the moment):
https://gist.github.com/mcnewton/63a01685f3748e202e81
One snippet of panic_action output below.
Any ideas?
Valgrind :/ Sounds like malloc memory header has been corrupted. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2