Moving to 3.0.x series and crashes
Hi, I've been trying to upgrade our eduroam radius server 2.1.12 to 3.0.10 or newer. Our wifi is configured to authenticate 802.1X PEAP/mschapv2 -> freeradius -> Microsoft ad. In test environment freeradius works ok but in production it randomly crashes. With 3.0.10 I got some crash but did not catch any info about it and I just decided to try directly git version without any crash investigation. So last night (2nd dec) I pulled 3.0.x from git, compiled and put it on production. It survide about 12 hours and crashed with messages below Thu Dec 3 08:16:21 2015 : Auth: (22577) Login OK: [host/tatk032054.univ.yo.oulu.fi] (from client cisco_aps port 13 cli f8-16-54-f1-5d-33) Thu Dec 3 08:16:25 2015 : ERROR: (22589) mschap: ERROR: Program returned code (1) and output 'Password expired (0xc0000071)' Thu Dec 3 08:16:25 2015 : ERROR: (22589) mschap: ERROR: Program returned code (1) and output 'Logon failure (0xc000006d)' Thu Dec 3 08:16:25 2015 : Error: ASSERT FAILED src/modules/rlm_mschap/rlm_mschap.c[1751]: nt_password CAUGHT SIGNAL: Aborted Backtrace of last 42 frames: /usr/local/lib/libfreeradius-radius.so(fr_fault+0x115)[0x7faabee6c4c2] /usr/local/lib/libfreeradius-server.so(rad_assert_fail+0x46)[0x7faabf0cbe39] /usr/local/lib/rlm_mschap.so(+0x5624)[0x7faab9905624] /usr/local/sbin/radiusd[0x4283e4] /usr/local/sbin/radiusd[0x428aa4] /usr/local/sbin/radiusd[0x4285ba] /usr/local/sbin/radiusd[0x4294d8] /usr/local/sbin/radiusd(modcall+0xa2)[0x42a21d] /usr/local/sbin/radiusd(indexed_modcall+0x363)[0x425c2f] /usr/local/sbin/radiusd(process_authenticate+0x22)[0x427cd6] /usr/local/lib/rlm_eap_mschapv2.so(+0x25ff)[0x7faab62215ff] /usr/local/lib/rlm_eap.so(+0x3f55)[0x7faaba72bf55] /usr/local/lib/rlm_eap.so(eap_method_select+0x421)[0x7faaba72c888] /usr/local/lib/rlm_eap.so(+0x2c07)[0x7faaba72ac07] /usr/local/sbin/radiusd[0x4283e4] /usr/local/sbin/radiusd[0x428aa4] /usr/local/sbin/radiusd[0x4285ba] /usr/local/sbin/radiusd[0x4294d8] /usr/local/sbin/radiusd(modcall+0xa2)[0x42a21d] /usr/local/sbin/radiusd(indexed_modcall+0x363)[0x425c2f] /usr/local/sbin/radiusd(process_authenticate+0x22)[0x427cd6] /usr/local/sbin/radiusd[0x40f583] /usr/local/sbin/radiusd(rad_authenticate+0x51e)[0x40fe9c] /usr/local/sbin/radiusd(rad_virtual_server+0x110)[0x4104b9] /usr/local/lib/rlm_eap_peap.so(eappeap_process+0xe68)[0x7faab6428a12] /usr/local/lib/rlm_eap_peap.so(+0x1fd3)[0x7faab6425fd3] /usr/local/lib/rlm_eap.so(+0x3f55)[0x7faaba72bf55] /usr/local/lib/rlm_eap.so(eap_method_select+0x421)[0x7faaba72c888] /usr/local/lib/rlm_eap.so(+0x2c07)[0x7faaba72ac07] /usr/local/sbin/radiusd[0x4283e4] /usr/local/sbin/radiusd[0x428aa4] /usr/local/sbin/radiusd[0x4285ba] /usr/local/sbin/radiusd[0x4294d8] /usr/local/sbin/radiusd(modcall+0xa2)[0x42a21d] /usr/local/sbin/radiusd(indexed_modcall+0x363)[0x425c2f] /usr/local/sbin/radiusd(process_authenticate+0x22)[0x427cd6] /usr/local/sbin/radiusd[0x40f583] /usr/local/sbin/radiusd(rad_authenticate+0x51e)[0x40fe9c] /usr/local/sbin/radiusd[0x43cca9] /usr/local/sbin/radiusd[0x438939] /lib64/libpthread.so.0(+0x7df5)[0x7faabd943df5] /lib64/libc.so.6(clone+0x6d)[0x7faabd43a1ad] No panic action set _EXIT(1) CALLED src/lib/debug.c[745]. Last error was: Internal sanity check 3352 [root@pc184005 ~]# I know I may should provide more info, but should I set panic action or compile fr with debug etc? Looks like it has something to do with mschapv password change so heres my mschap.conf http://cc.oulu.fi/~mtr/radius/mschap
On Dec 3, 2015, at 7:27 AM, Miika Räisänen <Miika.Raisanen@oulu.fi> wrote:
I've been trying to upgrade our eduroam radius server 2.1.12 to 3.0.10 or newer. Our wifi is configured to authenticate 802.1X PEAP/mschapv2 -> freeradius -> Microsoft ad.
In test environment freeradius works ok but in production it randomly crashes. With 3.0.10 I got some crash but did not catch any info about it and I just decided to try directly git version without any crash investigation.
So last night (2nd dec) I pulled 3.0.x from git, compiled and put it on production. It survide about 12 hours and crashed with messages below
Thu Dec 3 08:16:21 2015 : Auth: (22577) Login OK: [host/tatk032054.univ.yo.oulu.fi] (from client cisco_aps port 13 cli f8-16-54-f1-5d-33) Thu Dec 3 08:16:25 2015 : ERROR: (22589) mschap: ERROR: Program returned code (1) and output 'Password expired (0xc0000071)' Thu Dec 3 08:16:25 2015 : ERROR: (22589) mschap: ERROR: Program returned code (1) and output 'Logon failure (0xc000006d)' Thu Dec 3 08:16:25 2015 : Error: ASSERT FAILED src/modules/rlm_mschap/rlm_mschap.c[1751]: nt_password CAUGHT SIGNAL: Aborted Backtrace of last 42 frames: /usr/local/lib/libfreeradius-radius.so(fr_fault+0x115)[0x7faabee6c4c2] /usr/local/lib/libfreeradius-server.so(rad_assert_fail+0x46)[0x7faabf0cbe39] /usr/local/lib/rlm_mschap.so(+0x5624)[0x7faab9905624]
That helps a bit... please follow the instructions in https://github.com/FreeRADIUS/freeradius-server/blob/v3.0.x/doc/bugs to get a backtrace with symbols. That will tell us exactly where the problem occurs.
I know I may should provide more info, but should I set panic action or compile fr with debug etc?
Yes, please compile with debugging.
Looks like it has something to do with mschapv password change so heres my mschap.conf http://cc.oulu.fi/~mtr/radius/mschap
You have all of the password change config disabled. Alan DeKok.
On 3 Dec 2015, at 08:49, Alan DeKok <aland@deployingradius.com> wrote:
On Dec 3, 2015, at 7:27 AM, Miika Räisänen <Miika.Raisanen@oulu.fi> wrote:
I've been trying to upgrade our eduroam radius server 2.1.12 to 3.0.10 or newer. Our wifi is configured to authenticate 802.1X PEAP/mschapv2 -> freeradius -> Microsoft ad.
In test environment freeradius works ok but in production it randomly crashes. With 3.0.10 I got some crash but did not catch any info about it and I just decided to try directly git version without any crash investigation.
So last night (2nd dec) I pulled 3.0.x from git, compiled and put it on production. It survide about 12 hours and crashed with messages below
Thu Dec 3 08:16:21 2015 : Auth: (22577) Login OK: [host/tatk032054.univ.yo.oulu.fi] (from client cisco_aps port 13 cli f8-16-54-f1-5d-33) Thu Dec 3 08:16:25 2015 : ERROR: (22589) mschap: ERROR: Program returned code (1) and output 'Password expired (0xc0000071)' Thu Dec 3 08:16:25 2015 : ERROR: (22589) mschap: ERROR: Program returned code (1) and output 'Logon failure (0xc000006d)' Thu Dec 3 08:16:25 2015 : Error: ASSERT FAILED src/modules/rlm_mschap/rlm_mschap.c[1751]: nt_password CAUGHT SIGNAL: Aborted Backtrace of last 42 frames: /usr/local/lib/libfreeradius-radius.so(fr_fault+0x115)[0x7faabee6c4c2] /usr/local/lib/libfreeradius-server.so(rad_assert_fail+0x46)[0x7faabf0cbe39] /usr/local/lib/rlm_mschap.so(+0x5624)[0x7faab9905624]
That helps a bit... please follow the instructions in
https://github.com/FreeRADIUS/freeradius-server/blob/v3.0.x/doc/bugs
to get a backtrace with symbols. That will tell us exactly where the problem occurs.
I know I may should provide more info, but should I set panic action or compile fr with debug etc?
Yes, please compile with debugging.
Looks like it has something to do with mschapv password change so heres my mschap.conf http://cc.oulu.fi/~mtr/radius/mschap
You have all of the password change config disabled.
mod_authenticate doesn't check for that is just checks for the presence of PW_MSCHAP2_CPW. I'm confused as to how this should be working TBH. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On Dec 3, 2015, at 9:17 AM, Arran Cudbard-Bell <a.cudbardb@freeradius.org> wrote:
You have all of the password change config disabled.
mod_authenticate doesn't check for that is just checks for the presence of PW_MSCHAP2_CPW.
I'm confused as to how this should be working TBH.
mod_authenticate() calls do_mschap_cpw() which calls ntlm_auth from the entries in the configuration file. Alan DeKok.
On Dec 3, 2015, at 10:04 AM, Alan DeKok <aland@deployingradius.com> wrote:
On Dec 3, 2015, at 9:17 AM, Arran Cudbard-Bell <a.cudbardb@freeradius.org> wrote:
You have all of the password change config disabled.
mod_authenticate doesn't check for that is just checks for the presence of PW_MSCHAP2_CPW.
I'm confused as to how this should be working TBH.
mod_authenticate() calls do_mschap_cpw() which calls ntlm_auth from the entries in the configuration file.
Yes, but it also checks for the presence of the above attribute in the body of mod_authenticate, and that's where the assert is being triggered. -Arran
-----Original Message----- From: Freeradius-Users [mailto:freeradius-users- bounces+miika.raisanen=oulu.fi@lists.freeradius.org] On Behalf Of Alan DeKok Sent: 3. joulukuuta 2015 15:50 To: FreeRadius users mailing list Subject: Re: Moving to 3.0.x series and crashes
On Dec 3, 2015, at 7:27 AM, Miika Räisänen <Miika.Raisanen@oulu.fi> wrote:
I've been trying to upgrade our eduroam radius server 2.1.12 to 3.0.10 or
newer. Our wifi is configured to authenticate 802.1X PEAP/mschapv2 -> freeradius -> Microsoft ad.
In test environment freeradius works ok but in production it randomly
crashes. With 3.0.10 I got some crash but did not catch any info about it and I just decided to try directly git version without any crash investigation.
So last night (2nd dec) I pulled 3.0.x from git, compiled and put it on
production. It survide about 12 hours and crashed with messages below
Thu Dec 3 08:16:21 2015 : Auth: (22577) Login OK:
[host/tatk032054.univ.yo.oulu.fi] (from client cisco_aps port 13 cli f8-16-54- f1-5d-33)
Thu Dec 3 08:16:25 2015 : ERROR: (22589) mschap: ERROR: Program returned code (1) and output 'Password expired (0xc0000071)' Thu Dec 3 08:16:25 2015 : ERROR: (22589) mschap: ERROR: Program returned code (1) and output 'Logon failure (0xc000006d)' Thu Dec 3 08:16:25 2015 : Error: ASSERT FAILED src/modules/rlm_mschap/rlm_mschap.c[1751]: nt_password CAUGHT SIGNAL: Aborted Backtrace of last 42 frames: /usr/local/lib/libfreeradius-radius.so(fr_fault+0x115)[0x7faabee6c4c2] /usr/local/lib/libfreeradius- server.so(rad_assert_fail+0x46)[0x7faabf0cbe39] /usr/local/lib/rlm_mschap.so(+0x5624)[0x7faab9905624]
That helps a bit... please follow the instructions in
https://github.com/FreeRADIUS/freeradius-server/blob/v3.0.x/doc/bugs
to get a backtrace with symbols. That will tell us exactly where the problem occurs.
I know I may should provide more info, but should I set panic action or compile fr with debug etc?
Yes, please compile with debugging.
Looks like it has something to do with mschapv password change so heres my mschap.conf http://cc.oulu.fi/~mtr/radius/mschap
You have all of the password change config disabled.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
So I ran it in screen for a while and here we go: http://cc.oulu.fi/~mtr/radius/gdb_radiusd System is: Linux xxx.xxx.xxx.xx 3.10.0-229.11.1.el7.x86_64 #1 SMP Thu Aug 6 01:06:18 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux gcc-4.8.3-9.el7.x86_64 libgcc-4.8.3-9.el7.x86_64 samba-winbind-clients-4.1.12-23.el7_1.x86_64 samba-winbind-4.1.12-23.el7_1.x86_64 samba-winbind-modules-4.1.12-23.el7_1.x86_64 samba-libs-4.1.12-23.el7_1.x86_64 samba-common-4.1.12-23.el7_1.x86_64 samba-4.1.12-23.el7_1.x86_64 Need something more?
-----Original Message----- From: Freeradius-Users [mailto:freeradius-users- bounces+miika.raisanen=oulu.fi@lists.freeradius.org] On Behalf Of Alan DeKok Sent: 7. joulukuuta 2015 14:37 To: FreeRadius users mailing list Subject: Re: Moving to 3.0.x series and crashes
On Dec 7, 2015, at 7:22 AM, Miika Räisänen <Miika.Raisanen@oulu.fi> wrote:
Need something more?
That's fine.
Please grab the v3.0.x branch from github. It has a fix.
Alan DeKok.
Grabbed, compiled and put to production about a week ago. Still no crashes so I think it works. Thanks.
participants (4)
-
A.L.M.Buxey＠lboro.ac.uk -
Alan DeKok -
Arran Cudbard-Bell -
Miika Räisänen