3.1.x consistency check
Hi, So trying 3.1.x to see if it fixes the 3.0.x error.... :) Packaged build (so disable-developer) segfaults in rlm_ldap (ldap.c:1431). Trying to debug that I built a new version which has enable-developer set. That bombs out in a consistency check in SoH, as below. Looks like it doesn't like that pair_make_request is called with a NULL value? But can't imagine that is right as that's normally what happens to create an attribute? About to go home now so thought I'd shoot out a quick e-mail in case someone else knows the immediate answer without me having to stare at code for a while :) (Update on the original issue; I'm going to build against latest talloc and see if that fixes it - may be just a bug in the older talloc version in Debian wheezy. That will have to wait until next week now.) Cheers, Matthew (14) # Executing group from file /srv/radius/sites-enabled/default (14) authenticate { (14) outer-eap: Peer sent packet with method EAP PEAP (25) (14) outer-eap: Calling submodule eap_peap to process data (14) eap_peap: Continuing EAP-TLS (14) eap_peap: Got complete TLS record (661 bytes) (14) eap_peap: [eap-tls verify] = ok (14) eap_peap: [eap-tls process] = ok (14) eap_peap: Session established. Decoding tunneled attributes (14) eap_peap: PEAP state WAITING FOR SOH RESPONSE (14) eap_peap: EAP method unknown (254) (14) eap_peap: SoH System-Health-ID vendor 00000137 component=0 (14) eap_peap: SoH MS type-value payload CONSISTENCY CHECK FAILED src/lib/pair.c[674]: VALUE_PAIR "SoH-MS-Machine-Name" char buffer not \0 terminated SOFT ASSERT FAILED src/lib/pair.c[2531]: 0 RAISING SIGNAL: Aborted Program received signal SIGABRT, Aborted. 0x00007ffff627f79b in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/pt-raise.c:37 37 ../nptl/sysdeps/unix/sysv/linux/pt-raise.c: No such file or directory. (gdb) (gdb) bt #0 0x00007ffff627f79b in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/pt-raise.c:37 #1 0x00007ffff79504ce in fr_fault (sig=6) at src/lib/debug.c:648 #2 0x00007ffff7950fdb in fr_assert_cond (file=0x7ffff798205b "src/lib/pair.c", line=2531, expr=0x7ffff79824e6 "0", cond=false) at src/lib/debug.c:1090 #3 0x00007ffff7963154 in fr_pair_verify (file=0x7ffff798205b "src/lib/pair.c", line=674, vp=0xafe780) at src/lib/pair.c:2531 #4 0x00007ffff795ff49 in fr_pair_add (head=0xb11d10, add=0xafea10) at src/lib/pair.c:674 #5 0x00007ffff795fc5f in fr_pair_make (ctx=0xb11c90, vps=0xb11d10, attribute=0x464afc "SoH-MS-Machine-Role", value=0x0, op=T_OP_SET) at src/lib/pair.c:536 #6 0x00000000004357fe in eap_peap_soh_mstlv (request=0xb121f0, p=0xae054f "\001", data_len=5) at src/main/soh.c:299 #7 0x000000000043603b in soh_verify (request=0xb121f0, data=0xae04ee "", data_len=543) at src/main/soh.c:502 #8 0x00007ffff1553779 in eap_peap_soh_verify (request=0xb121f0, packet=0xafe020, data=0xae04a8 "", data_len=621) at src/modules/rlm_eap/types/rlm_eap_peap/peap.c:198 #9 0x00007ffff15550a6 in eap_peap_process (eap_session=0xadafa0, tls_session=0xadc350) at src/modules/rlm_eap/types/rlm_eap_peap/peap.c:814 #10 0x00007ffff1552ff5 in mod_process (arg=0xaa3820, eap_session=0xadafa0) at src/modules/rlm_eap/types/rlm_eap_peap/rlm_eap_peap.c:293 #11 0x00007ffff1b6361f in eap_module_call (module=0xaa3590, eap_session=0xadafa0) at src/modules/rlm_eap/eap.c:189 #12 0x00007ffff1b63ea8 in eap_method_select (inst=0xa3d570, eap_session=0xadafa0) at src/modules/rlm_eap/eap.c:425 #13 0x00007ffff1b622fb in mod_authenticate (instance=0xa3d570, request=0xb121f0) at src/modules/rlm_eap/rlm_eap.c:202 #14 0x0000000000429051 in call_modsingle (component=MOD_AUTHENTICATE, sp=0xaba740, request=0xb121f0) at src/main/modcall.c:327 #15 0x0000000000429749 in modcall_recurse (request=0xb121f0, component=MOD_AUTHENTICATE, depth=1, entry=0x7fffffffd818, do_next_sibling=true) at src/main/modcall.c:610 #16 0x0000000000429221 in modcall_child (request=0xb121f0, component=MOD_AUTHENTICATE, depth=1, entry=0x7fffffffd800, c=0xaba740, result=0x7fffffffd6ec, do_next_sibling=true) at src/main/modcall.c:434 #17 0x000000000042a2ea in modcall_recurse (request=0xb121f0, component=MOD_AUTHENTICATE, depth=0, entry=0x7fffffffd800, do_next_sibling=true) at src/main/modcall.c:834 #18 0x000000000042b00a in modcall (component=MOD_AUTHENTICATE, c=0xaba640, request=0xb121f0) at src/main/modcall.c:1179 #19 0x00000000004266a8 in indexed_modcall (comp=MOD_AUTHENTICATE, idx=1368977, request=0xb121f0) at src/main/modules.c:956 #20 0x000000000042895f in process_authenticate (auth_type=1368977, request=0xb121f0) at src/main/modules.c:2099 #21 0x000000000040f32b in rad_check_password (request=0xb121f0) at src/main/auth.c:251 #22 0x000000000040fa25 in rad_authenticate (request=0xb121f0) at src/main/auth.c:516 #23 0x000000000043a089 in request_running (request=0xb121f0, action=1) at src/main/process.c:1482 #24 0x0000000000438d97 in request_queue_or_run (request=0xb121f0, process=0x439f27 <request_running>) at src/main/process.c:953 #25 0x000000000043a98c in request_receive (ctx=0xb11c30, listener=0xa09090, packet=0xb11c90, client=0x9905e0, fun=0x40f4d0 <rad_authenticate>) at src/main/process.c:1740 #26 0x00000000004189cb in auth_socket_recv (listener=0xa09090) at src/main/listen.c:1903 #27 0x00000000004415a1 in event_socket_handler (xel=0xac79d0, fd=15, ctx=0xa09090) at src/main/process.c:4544 #28 0x00007ffff797aa53 in fr_event_loop (el=0xac79d0) at src/lib/event.c:637 #29 0x00000000004432e6 in radius_event_process () at src/main/process.c:5587 #30 0x0000000000431cdd in main (argc=4, argv=0x7fffffffe6e8) at src/main/radiusd.c:578 (gdb) -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On 20 Nov 2015, at 11:57, Matthew Newton <mcn4@leicester.ac.uk> wrote:
Hi,
So trying 3.1.x to see if it fixes the 3.0.x error.... :)
Packaged build (so disable-developer) segfaults in rlm_ldap (ldap.c:1431). Trying to debug that I built a new version which has enable-developer set. That bombs out in a consistency check in SoH, as below.
Looks like it doesn't like that pair_make_request is called with a NULL value? But can't imagine that is right as that's normally what happens to create an attribute?
About to go home now so thought I'd shoot out a quick e-mail in case someone else knows the immediate answer without me having to stare at code for a while :)
Pushed a fix for your SoH issue. Buffer was being underallocated. Should have been t + 1, but we have proper functions for assigning values to attributes now, so just used that instead :) Will look at LDAP. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On 20 Nov 2015, at 12:07, Arran Cudbard-Bell <a.cudbardb@freeradius.org> wrote:
On 20 Nov 2015, at 11:57, Matthew Newton <mcn4@leicester.ac.uk> wrote:
Hi,
So trying 3.1.x to see if it fixes the 3.0.x error.... :)
Packaged build (so disable-developer) segfaults in rlm_ldap (ldap.c:1431). Trying to debug that I built a new version which has enable-developer set. That bombs out in a consistency check in SoH, as below.
Looks like it doesn't like that pair_make_request is called with a NULL value? But can't imagine that is right as that's normally what happens to create an attribute?
About to go home now so thought I'd shoot out a quick e-mail in case someone else knows the immediate answer without me having to stare at code for a while :)
Pushed a fix for your SoH issue.
Buffer was being underallocated. Should have been t + 1, but we have proper functions for assigning values to attributes now, so just used that instead :)
Will look at LDAP.
Pushed a fix for LDAP. Looks like that array is NULL if no extensions are present. Expected array with one NULL element, but... OpenLDAP. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On 20 Nov 2015, at 12:13, Arran Cudbard-Bell <a.cudbardb@freeradius.org> wrote:
On 20 Nov 2015, at 12:07, Arran Cudbard-Bell <a.cudbardb@freeradius.org> wrote:
On 20 Nov 2015, at 11:57, Matthew Newton <mcn4@leicester.ac.uk> wrote:
Hi,
So trying 3.1.x to see if it fixes the 3.0.x error.... :)
Packaged build (so disable-developer) segfaults in rlm_ldap (ldap.c:1431). Trying to debug that I built a new version which has enable-developer set. That bombs out in a consistency check in SoH, as below.
Looks like it doesn't like that pair_make_request is called with a NULL value? But can't imagine that is right as that's normally what happens to create an attribute?
About to go home now so thought I'd shoot out a quick e-mail in case someone else knows the immediate answer without me having to stare at code for a while :)
Pushed a fix for your SoH issue.
Buffer was being underallocated. Should have been t + 1, but we have proper functions for assigning values to attributes now, so just used that instead :)
Will look at LDAP.
Pushed a fix for LDAP. Looks like that array is NULL if no extensions are present.
Expected array with one NULL element, but... OpenLDAP.
Interestingly the SoH issue was there in 3.0.x too, so you must be the first person to use it with the v3.0.x branch :/ -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On Fri, Nov 20, 2015 at 12:36:22PM -0500, Arran Cudbard-Bell wrote:
Pushed a fix for your SoH issue.
:-)
Buffer was being underallocated. Should have been t + 1, but we have proper functions for assigning values to attributes now, so just used that instead :)
Will look at LDAP.
Pushed a fix for LDAP. Looks like that array is NULL if no extensions are present.
:-) :-)
Expected array with one NULL element, but... OpenLDAP.
Interestingly the SoH issue was there in 3.0.x too, so you must be the first person to use it with the v3.0.x branch :/
:-) :-) :-) :-/ <runs off to recompile 3.0.x and see if it fixes the random crashing...!> That would have taken me much longer to solve... :-S Thanks!! Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
:-) :-) :-) :-/
<runs off to recompile 3.0.x and see if it fixes the random crashing...!>
Maybe, it would have been zeroing out a byte directly after that buffer in the heap. If you were doing non developer builds then the validity checks wouldn't have caught it. It may be worth doing a dev build of v3.0.x (if you're not already) just to see if the validity checks catch the issue.
That would have taken me much longer to solve... :-S
I've fixed a fair few of those already ;)
Thanks!!
No problem! -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On Fri, Nov 20, 2015 at 05:20:50PM -0500, Arran Cudbard-Bell wrote:
:-) :-) :-) :-/
<runs off to recompile 3.0.x and see if it fixes the random crashing...!>
Maybe, it would have been zeroing out a byte directly after that buffer in the heap.
Yes
If you were doing non developer builds then the validity checks wouldn't have caught it. It may be worth doing a dev build of v3.0.x (if you're not already) just to see if the validity checks catch the issue.
That's a good idea. It'll probably be after the weekend now. I've just rebuilt 3.0.x and pushed it to the live server, so I'll leave that for the weekend and see how it holds out. If it crashes like before it probably won't be until 9am Monday morning anyway. Though I could return Session-Timeout:=30 to create higher load... ;-)
That would have taken me much longer to solve... :-S
I've fixed a fair few of those already ;)
:) Probably just shows I've got too much other stuff going on :( Cheers, Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On Fri, Nov 20, 2015 at 10:52:28PM +0000, Matthew Newton wrote:
It may be worth doing a dev build of v3.0.x (if you're not already) just to see if the validity checks catch the issue.
That's a good idea. It'll probably be after the weekend now. I've just rebuilt 3.0.x and pushed it to the live server, so I'll leave that for the weekend and see how it holds out. If it crashes like before it probably won't be until 9am Monday morning anyway.
Follow on from this. The (non-dev) build from Friday with SoH bugfix has run all weekend and through both the busy time of yesterday morning and this morning without crashing, so I think that patch has fixed it. Thanks! Previously would crash within a few minutes or hours at the most. I've just put a dev build on the other server and about to push all load on to it, but it's not crashed on the first packet like 3.1 did so hopefully the consistency checks are consistent :) I'll probably try bumping up to 3.1 again after this, "just because"... Cheers, Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On 24 Nov 2015, at 09:01, Matthew Newton <mcn4@LEICESTER.AC.UK> wrote:
On Fri, Nov 20, 2015 at 10:52:28PM +0000, Matthew Newton wrote:
It may be worth doing a dev build of v3.0.x (if you're not already) just to see if the validity checks catch the issue.
That's a good idea. It'll probably be after the weekend now. I've just rebuilt 3.0.x and pushed it to the live server, so I'll leave that for the weekend and see how it holds out. If it crashes like before it probably won't be until 9am Monday morning anyway.
Follow on from this.
The (non-dev) build from Friday with SoH bugfix has run all weekend and through both the busy time of yesterday morning and this morning without crashing, so I think that patch has fixed it. Thanks! Previously would crash within a few minutes or hours at the most.
I've just put a dev build on the other server and about to push all load on to it, but it's not crashed on the first packet like 3.1 did so hopefully the consistency checks are consistent :)
I'll probably try bumping up to 3.1 again after this, "just because"...
In v3.1.x just took a pass through to convert most direct writes to vp_strvalue and vp_octets to use the value assignment API, which will always ensure proper \0 termination and correct vp_length. Hopefully there won't be any more incidents of this. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On Tue, Dec 01, 2015 at 12:04:28PM -0500, Arran Cudbard-Bell wrote:
I'll probably try bumping up to 3.1 again after this, "just because"...
In v3.1.x just took a pass through to convert most direct writes to vp_strvalue and vp_octets to use the value assignment API, which will always ensure proper \0 termination and correct vp_length. Hopefully there won't be any more incidents of this.
Cool. :) FWIW I've been running one server on 3.0 and the other on 3.1 for a week now and all fine. Current plan is to leave them diverse like that. Cheers, Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On 1 Dec 2015, at 12:19, Matthew Newton <mcn4@leicester.ac.uk> wrote:
On Tue, Dec 01, 2015 at 12:04:28PM -0500, Arran Cudbard-Bell wrote:
I'll probably try bumping up to 3.1 again after this, "just because"...
In v3.1.x just took a pass through to convert most direct writes to vp_strvalue and vp_octets to use the value assignment API, which will always ensure proper \0 termination and correct vp_length. Hopefully there won't be any more incidents of this.
Cool. :)
FWIW I've been running one server on 3.0 and the other on 3.1 for a week now and all fine. Current plan is to leave them diverse like that.
Guess it gives you the option to go either way :) -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
participants (2)
-
Arran Cudbard-Bell -
Matthew Newton