Hello, OS: CentOS 7 Kernel: 3.10.0-693.2.2.el7.x86_64 radiusd: FreeRADIUS Version 3.0.13, for host x86_64-redhat-linux-gnu, built on Aug 23 2017 at 15:18:22 Recently upgraded freeradius and noticed that it is crashing at random times. Dont see anything interesting in the logs. $ radiusd -X starts normally. But do see segfaults in $ dmesg [ 2886.030026] perf: interrupt took too long (2511 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 [ 6420.898264] perf: interrupt took too long (3143 > 3138), lowering kernel.perf_event_max_sample_rate to 63000 [14198.608930] radiusd[32759]: segfault at e1a32313a ip 00007f06a7f9f6e4 sp 00007fffe181ea20 error 4 in libc-2.17.so[7f06a7f57000+1b8000] [14975.220890] perf: interrupt took too long (4010 > 3928), lowering kernel.perf_event_max_sample_rate to 49000 [21863.497234] perf: interrupt took too long (5298 > 5012), lowering kernel.perf_event_max_sample_rate to 37000 [24292.736627] radiusd[12730]: segfault at 7fc6e4000d80 ip 00007fc6e4000d80 sp 00007fffe6524f28 error 15 [29848.535819] radiusd[26477]: segfault at 18 ip 00007fe03d119418 sp 00007ffcc24e1dd0 error 4 in libc-2.17.so[7fe03d09d000+1b8000] [36113.258489] traps: radiusd[7203] general protection ip:7f06d80cd27f sp:7f06cfda2e40 error:0 in libc-2.17.so[7f06d8051000+1b8000] [40207.985076] traps: radiusd[4354] general protection ip:7fee629196e4 sp:7ffcd65ffcd0 error:0 in libc-2.17.so[7fee628d1000+1b8000] [56725.833141] IPv6: udp checksum is 0 for [2804:1f06:1:2048:aa9d:21ff: fe47:876e]:1645->[2403:7300:101:36::12]:1812 [56726.856901] IPv6: udp checksum is 0 for [2804:1f06:1:2048:aa9d:21ff: fe47:876e]:1645->[2403:7300:101:36::12]:1812 [60021.572076] in:imjournal[1692]: segfault at 0 ip 00007f5afccb8983 sp 00007f5af97c8b50 error 4 in imjournal.so[7f5afccb6000+5000] [68735.882196] IPv6: udp checksum is 0 for [2607:2400:2:2187:23a:7dff: fed4:5d92]:1645->[2403:7300:101:36::12]:1812 [68736.906013] IPv6: udp checksum is 0 for [2607:2400:2:2187:23a:7dff: fed4:5d92]:1645->[2403:7300:101:36::12]:1812 [69014.260230] IPv6: udp checksum is 0 for [2607:2400:5:2151:207:7dff: fe1d:b706]:1645->[2403:7300:101:36::12]:1812 [69015.308755] IPv6: udp checksum is 0 for [2607:2400:5:2151:207:7dff: fe1d:b706]:1645->[2403:7300:101:36::12]:1812 [73005.257649] IPv6: udp checksum is 0 for [2a03:6940:1:2068:1ae7:28ff: fef7:61bd]:1645->[2403:7300:101:36::12]:1812 [73006.311233] IPv6: udp checksum is 0 for [2a03:6940:1:2068:1ae7:28ff: fef7:61bd]:1645->[2403:7300:101:36::12]:1812 [110480.127950] IPv6: udp checksum is 0 for [2607:2400:1:2100:206:f6ff: fed2:61f8]:1645->[2403:7300:101:36::12]:1812 [110481.209600] IPv6: udp checksum is 0 for [2607:2400:1:2100:206:f6ff: fed2:61f8]:1645->[2403:7300:101:36::12]:1812 [117945.839480] in:imjournal[14521]: segfault at 0 ip 00007f5cc505b983 sp 00007f5cc136bb50 error 4 in imjournal.so[7f5cc5059000+5000] [122742.076476] Key type dns_resolver registered [122742.118795] Key type cifs.spnego registered [122742.118799] Key type cifs.idmap registered [167924.574413] in:imjournal[9288]: segfault at 0 ip 00007f973987f983 sp 00007f973538fb50 error 4 in imjournal.so[7f973987d000+5000] [210134.352360] conntrack: generic helper won't handle protocol 47. Please consider loading the specific helper module. [218406.756306] in:imjournal[7538]: segfault at 0 ip 00007fed6c091983 sp 00007fed673a1b50 error 4 in imjournal.so[7fed6c08f000+5000] [268886.722800] in:imjournal[24594]: segfault at 0 ip 00007f9abd926983 sp 00007f9ab8436b50 error 4 in imjournal.so[7f9abd924000+5000] [305921.081321] IPv6: udp checksum is 0 for [2607:2400:1:2116:e22f:6dff: fe91:f6e1]:1645->[2403:7300:101:36::12]:1812 [305922.162078] IPv6: udp checksum is 0 for [2607:2400:1:2116:e22f:6dff: fe91:f6e1]:1645->[2403:7300:101:36::12]:1812 [316036.147874] IPv6: udp checksum is 0 for [2607:2400:1:2188:6273:5cff: fe2f:4207]:1645->[2403:7300:101:36::12]:1812 [316037.196177] IPv6: udp checksum is 0 for [2607:2400:1:2188:6273:5cff: fe2f:4207]:1645->[2403:7300:101:36::12]:1812 [337566.944167] in:imjournal[10826]: segfault at 0 ip 00007f3bc891c983 sp 00007f3bc2c2cb50 error 4 in imjournal.so[7f3bc891a000+5000] [355474.169434] radiusd[5184]: segfault at f5594a28 ip 00007f30d0129b00 sp 00007f30c537f6e0 error 4 in libtalloc.so.2.1.9[7f30d0126000+f000] [362624.748526] radiusd[32532]: segfault at 18 ip 00007f554e2112ce sp 00007ffdec151d90 error 4 in libc-2.17.so[7f554e195000+1b8000] [366369.197821] traps: radiusd[32147] general protection ip:7f824fdaaad5 sp:7f8247a7d290 error:0 in libc-2.17.so[7f824fd2d000+1b8000] [368382.126990] radiusd[15692]: segfault at 18 ip 00007f5c5c5b052d sp 00007f5c51280d10 error 4 in libc-2.17.so[7f5c5c535000+1b8000] [368415.327048] traps: radiusd[18044] general protection ip:7f3975b0881d sp:7f396b7cc6a8 error:0 in libc-2.17.so[7f3975a7f000+1b8000] [369617.422068] radiusd[25252]: segfault at 7f4dfef4a9d8 ip 00007f4e078c5418 sp 00007ffcf89273e0 error 4 in libc-2.17.so[7f4e07849000+ 1b8000] [374905.093166] traps: radiusd[605] general protection ip:7fb7254ef81d sp:7fb7189abc28 error:0 in libc-2.17.so[7fb725466000+1b8000] [376801.247227] radiusd[15436]: segfault at 7fbe5299e5d8 ip 00007fbe7d72c418 sp 00007ffc52397530 error 4 in libc-2.17.so[7fbe7d6b0000+ 1b8000] [377761.161832] radiusd[21228]: segfault at 7fee80002e10 ip 00007fee80002e10 sp 00007ffdee783a68 error 15 [379843.294430] radiusd[6923]: segfault at 18 ip 00007f4518b36418 sp 00007ffeb09da9e0 error 4 in libc-2.17.so[7f4518aba000+1b8000] [389971.741324] radiusd[15204]: segfault at 7f07c54771d8 ip 00007f07adc8b52d sp 00007f07a315cb20 error 4 in libc-2.17.so[7f07adc10000+ 1b8000] [393581.655644] radiusd[12099]: segfault at 18 ip 00007f7a9f2a82ce sp 00007ffeabae65e0 error 4 in libc-2.17.so[7f7a9f22c000+1b8000] [397222.777713] in:imjournal[4812]: segfault at 0 ip 00007f5fe8cef983 sp 00007f5fe27ffb50 error 4 in imjournal.so[7f5fe8ced000+5000] [397225.678506] radiusd[9651]: segfault at 18 ip 00007f20c6ede418 sp 00007f20a7ffdd40 error 4 in libc-2.17.so[7f20c6e62000+1b8000] [398893.363679] IPv6: udp checksum is 0 for [2a03:c9c0:4:2134:5e83:8fff: fee3:9e06]:1645->[2403:7300:101:36::12]:1812 [398894.419431] IPv6: udp checksum is 0 for [2a03:c9c0:4:2134:5e83:8fff: fee3:9e06]:1645->[2403:7300:101:36::12]:1812 [402638.895409] radiusd[18160]: segfault at 18 ip 00007fac35bcc52d sp 00007fac2a89cd10 error 4 in libc-2.17.so[7fac35b51000+1b8000] [405040.177274] IPv6: udp checksum is 0 for [2806:210:1:2089:8a5a:92ff: fef7:4208]:1645->[2403:7300:101:36::12]:1812 [405041.200663] IPv6: udp checksum is 0 for [2806:210:1:2089:8a5a:92ff: fef7:4208]:1645->[2403:7300:101:36::12]:1812 [405071.980878] radiusd[2973]: segfault at 8 ip 00007f8a7ae76cb9 sp 00007ffd8eb385f0 error 4 in libtalloc.so.2.1.9[7f8a7ae74000+f000] [407617.663020] traps: radiusd[25365] general protection ip:7f1d269a2ad5 sp:7f1d1d6733b0 error:0 in libc-2.17.so[7f1d26925000+1b8000] [411156.319552] radiusd[18616]: segfault at 43bfc68 ip 00007f03651f5b00 sp 00007f0342ffbc40 error 4 in libtalloc.so.2.1.9[7f03651f2000+f000] [413030.189572] radiusd[30421]: segfault at 18 ip 00007f6f058529f6 sp 00007f6ef951d450 error 4 in libc-2.17.so[7f6f057d5000+1b8000] [413156.407921] traps: radiusd[31701] general protection ip:7f666dcdb6e4 sp:7ffc87dab4a0 error:0 in libc-2.17.so[7f666dc93000+1b8000] [413605.949866] radiusd[1624]: segfault at 0 ip 00007f7b648974c9 sp 00007f7b557eee80 error 4 in libfreeradius-radius.so[7f7b6488c000+3c000] [460136.289414] in:imjournal[10498]: segfault at 0 ip 00007f7c42347983 sp 00007f7c3b657b50 error 4 in imjournal.so[7f7c42345000+5000] [474167.794387] IPv6: udp checksum is 0 for [2607:2400:3:2109:aa9d:21ff: fe03:274e]:1645->[2403:7300:101:36::12]:1812 [474168.850441] IPv6: udp checksum is 0 for [2607:2400:3:2109:aa9d:21ff: fe03:274e]:1645->[2403:7300:101:36::12]:1812 We have 3 radius servers displaying the same behavior. Anything to try? Any input is appreciated! thank you, -dave
First thing would be to upgrade to latest version to see if the issue has already been fixed On 5 Oct 2017 7:01 pm, "Dave Macias" <davama@gmail.com> wrote:
Hello,
OS: CentOS 7 Kernel: 3.10.0-693.2.2.el7.x86_64 radiusd: FreeRADIUS Version 3.0.13, for host x86_64-redhat-linux-gnu, built on Aug 23 2017 at 15:18:22
Recently upgraded freeradius and noticed that it is crashing at random times. Dont see anything interesting in the logs. $ radiusd -X starts normally. But do see segfaults in $ dmesg
[ 2886.030026] perf: interrupt took too long (2511 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 [ 6420.898264] perf: interrupt took too long (3143 > 3138), lowering kernel.perf_event_max_sample_rate to 63000 [14198.608930] radiusd[32759]: segfault at e1a32313a ip 00007f06a7f9f6e4 sp 00007fffe181ea20 error 4 in libc-2.17.so[7f06a7f57000+1b8000] [14975.220890] perf: interrupt took too long (4010 > 3928), lowering kernel.perf_event_max_sample_rate to 49000 [21863.497234] perf: interrupt took too long (5298 > 5012), lowering kernel.perf_event_max_sample_rate to 37000 [24292.736627] radiusd[12730]: segfault at 7fc6e4000d80 ip 00007fc6e4000d80 sp 00007fffe6524f28 error 15 [29848.535819] radiusd[26477]: segfault at 18 ip 00007fe03d119418 sp 00007ffcc24e1dd0 error 4 in libc-2.17.so[7fe03d09d000+1b8000] [36113.258489] traps: radiusd[7203] general protection ip:7f06d80cd27f sp:7f06cfda2e40 error:0 in libc-2.17.so[7f06d8051000+1b8000] [40207.985076] traps: radiusd[4354] general protection ip:7fee629196e4 sp:7ffcd65ffcd0 error:0 in libc-2.17.so[7fee628d1000+1b8000] [56725.833141] IPv6: udp checksum is 0 for [2804:1f06:1:2048:aa9d:21ff: fe47:876e]:1645->[2403:7300:101:36::12]:1812 [56726.856901] IPv6: udp checksum is 0 for [2804:1f06:1:2048:aa9d:21ff: fe47:876e]:1645->[2403:7300:101:36::12]:1812 [60021.572076] in:imjournal[1692]: segfault at 0 ip 00007f5afccb8983 sp 00007f5af97c8b50 error 4 in imjournal.so[7f5afccb6000+5000] [68735.882196] IPv6: udp checksum is 0 for [2607:2400:2:2187:23a:7dff: fed4:5d92]:1645->[2403:7300:101:36::12]:1812 [68736.906013] IPv6: udp checksum is 0 for [2607:2400:2:2187:23a:7dff: fed4:5d92]:1645->[2403:7300:101:36::12]:1812 [69014.260230] IPv6: udp checksum is 0 for [2607:2400:5:2151:207:7dff: fe1d:b706]:1645->[2403:7300:101:36::12]:1812 [69015.308755] IPv6: udp checksum is 0 for [2607:2400:5:2151:207:7dff: fe1d:b706]:1645->[2403:7300:101:36::12]:1812 [73005.257649] IPv6: udp checksum is 0 for [2a03:6940:1:2068:1ae7:28ff: fef7:61bd]:1645->[2403:7300:101:36::12]:1812 [73006.311233] IPv6: udp checksum is 0 for [2a03:6940:1:2068:1ae7:28ff: fef7:61bd]:1645->[2403:7300:101:36::12]:1812 [110480.127950] IPv6: udp checksum is 0 for [2607:2400:1:2100:206:f6ff: fed2:61f8]:1645->[2403:7300:101:36::12]:1812 [110481.209600] IPv6: udp checksum is 0 for [2607:2400:1:2100:206:f6ff: fed2:61f8]:1645->[2403:7300:101:36::12]:1812 [117945.839480] in:imjournal[14521]: segfault at 0 ip 00007f5cc505b983 sp 00007f5cc136bb50 error 4 in imjournal.so[7f5cc5059000+5000] [122742.076476] Key type dns_resolver registered [122742.118795] Key type cifs.spnego registered [122742.118799] Key type cifs.idmap registered [167924.574413] in:imjournal[9288]: segfault at 0 ip 00007f973987f983 sp 00007f973538fb50 error 4 in imjournal.so[7f973987d000+5000] [210134.352360] conntrack: generic helper won't handle protocol 47. Please consider loading the specific helper module. [218406.756306] in:imjournal[7538]: segfault at 0 ip 00007fed6c091983 sp 00007fed673a1b50 error 4 in imjournal.so[7fed6c08f000+5000] [268886.722800] in:imjournal[24594]: segfault at 0 ip 00007f9abd926983 sp 00007f9ab8436b50 error 4 in imjournal.so[7f9abd924000+5000] [305921.081321] IPv6: udp checksum is 0 for [2607:2400:1:2116:e22f:6dff: fe91:f6e1]:1645->[2403:7300:101:36::12]:1812 [305922.162078] IPv6: udp checksum is 0 for [2607:2400:1:2116:e22f:6dff: fe91:f6e1]:1645->[2403:7300:101:36::12]:1812 [316036.147874] IPv6: udp checksum is 0 for [2607:2400:1:2188:6273:5cff: fe2f:4207]:1645->[2403:7300:101:36::12]:1812 [316037.196177] IPv6: udp checksum is 0 for [2607:2400:1:2188:6273:5cff: fe2f:4207]:1645->[2403:7300:101:36::12]:1812 [337566.944167] in:imjournal[10826]: segfault at 0 ip 00007f3bc891c983 sp 00007f3bc2c2cb50 error 4 in imjournal.so[7f3bc891a000+5000] [355474.169434] radiusd[5184]: segfault at f5594a28 ip 00007f30d0129b00 sp 00007f30c537f6e0 error 4 in libtalloc.so.2.1.9[7f30d0126000+f000] [362624.748526] radiusd[32532]: segfault at 18 ip 00007f554e2112ce sp 00007ffdec151d90 error 4 in libc-2.17.so[7f554e195000+1b8000] [366369.197821] traps: radiusd[32147] general protection ip:7f824fdaaad5 sp:7f8247a7d290 error:0 in libc-2.17.so[7f824fd2d000+1b8000] [368382.126990] radiusd[15692]: segfault at 18 ip 00007f5c5c5b052d sp 00007f5c51280d10 error 4 in libc-2.17.so[7f5c5c535000+1b8000] [368415.327048] traps: radiusd[18044] general protection ip:7f3975b0881d sp:7f396b7cc6a8 error:0 in libc-2.17.so[7f3975a7f000+1b8000] [369617.422068] radiusd[25252]: segfault at 7f4dfef4a9d8 ip 00007f4e078c5418 sp 00007ffcf89273e0 error 4 in libc-2.17.so[7f4e07849000+ 1b8000] [374905.093166] traps: radiusd[605] general protection ip:7fb7254ef81d sp:7fb7189abc28 error:0 in libc-2.17.so[7fb725466000+1b8000] [376801.247227] radiusd[15436]: segfault at 7fbe5299e5d8 ip 00007fbe7d72c418 sp 00007ffc52397530 error 4 in libc-2.17.so[7fbe7d6b0000+ 1b8000] [377761.161832] radiusd[21228]: segfault at 7fee80002e10 ip 00007fee80002e10 sp 00007ffdee783a68 error 15 [379843.294430] radiusd[6923]: segfault at 18 ip 00007f4518b36418 sp 00007ffeb09da9e0 error 4 in libc-2.17.so[7f4518aba000+1b8000] [389971.741324] radiusd[15204]: segfault at 7f07c54771d8 ip 00007f07adc8b52d sp 00007f07a315cb20 error 4 in libc-2.17.so[7f07adc10000+ 1b8000] [393581.655644] radiusd[12099]: segfault at 18 ip 00007f7a9f2a82ce sp 00007ffeabae65e0 error 4 in libc-2.17.so[7f7a9f22c000+1b8000] [397222.777713] in:imjournal[4812]: segfault at 0 ip 00007f5fe8cef983 sp 00007f5fe27ffb50 error 4 in imjournal.so[7f5fe8ced000+5000] [397225.678506] radiusd[9651]: segfault at 18 ip 00007f20c6ede418 sp 00007f20a7ffdd40 error 4 in libc-2.17.so[7f20c6e62000+1b8000] [398893.363679] IPv6: udp checksum is 0 for [2a03:c9c0:4:2134:5e83:8fff: fee3:9e06]:1645->[2403:7300:101:36::12]:1812 [398894.419431] IPv6: udp checksum is 0 for [2a03:c9c0:4:2134:5e83:8fff: fee3:9e06]:1645->[2403:7300:101:36::12]:1812 [402638.895409] radiusd[18160]: segfault at 18 ip 00007fac35bcc52d sp 00007fac2a89cd10 error 4 in libc-2.17.so[7fac35b51000+1b8000] [405040.177274] IPv6: udp checksum is 0 for [2806:210:1:2089:8a5a:92ff: fef7:4208]:1645->[2403:7300:101:36::12]:1812 [405041.200663] IPv6: udp checksum is 0 for [2806:210:1:2089:8a5a:92ff: fef7:4208]:1645->[2403:7300:101:36::12]:1812 [405071.980878] radiusd[2973]: segfault at 8 ip 00007f8a7ae76cb9 sp 00007ffd8eb385f0 error 4 in libtalloc.so.2.1.9[7f8a7ae74000+f000] [407617.663020] traps: radiusd[25365] general protection ip:7f1d269a2ad5 sp:7f1d1d6733b0 error:0 in libc-2.17.so[7f1d26925000+1b8000] [411156.319552] radiusd[18616]: segfault at 43bfc68 ip 00007f03651f5b00 sp 00007f0342ffbc40 error 4 in libtalloc.so.2.1.9[7f03651f2000+f000] [413030.189572] radiusd[30421]: segfault at 18 ip 00007f6f058529f6 sp 00007f6ef951d450 error 4 in libc-2.17.so[7f6f057d5000+1b8000] [413156.407921] traps: radiusd[31701] general protection ip:7f666dcdb6e4 sp:7ffc87dab4a0 error:0 in libc-2.17.so[7f666dc93000+1b8000] [413605.949866] radiusd[1624]: segfault at 0 ip 00007f7b648974c9 sp 00007f7b557eee80 error 4 in libfreeradius-radius.so[7f7b6488c000+3c000] [460136.289414] in:imjournal[10498]: segfault at 0 ip 00007f7c42347983 sp 00007f7c3b657b50 error 4 in imjournal.so[7f7c42345000+5000] [474167.794387] IPv6: udp checksum is 0 for [2607:2400:3:2109:aa9d:21ff: fe03:274e]:1645->[2403:7300:101:36::12]:1812 [474168.850441] IPv6: udp checksum is 0 for [2607:2400:3:2109:aa9d:21ff: fe03:274e]:1645->[2403:7300:101:36::12]:1812
We have 3 radius servers displaying the same behavior. Anything to try?
Any input is appreciated!
thank you, -dave - List info/subscribe/unsubscribe? See http://www.freeradius.org/ list/users.html
radiusd: FreeRADIUS Version 3.0.13, for host x86_64-redhat-linux-
Recently upgraded freeradius and noticed that it is crashing at random times. Dont see anything interesting in the logs. $ radiusd -X starts normally. But do see segfaults in $ dmesg
...
We have 3 radius servers displaying the same behavior. Anything to try?
https://github.com/FreeRADIUS/freeradius-server/blob/v4.0.x/doc/bugs.md has some information about gdb, which can show where the segfault occurs. And try to reproduce them with 3.0.x-HEAD from git, they might have been fixed already. -- Herwin Weststrate
https://github.com/FreeRADIUS/freeradius-server/blob/v4.0.x/doc/bugs.md has some information about gdb, which can show where the segfault occurs. And try to reproduce them with 3.0.x-HEAD from git, they might have been fixed already.
was hoping to stay with the provided repo version so that $ yum can control it but i'll give it a try and see what results from it. Looking at the repository in git, ldap module is already incorporated, yes? We have these from freeradius installed freeradius-ldap-3.0.13-8.el7_4.x86_64 freeradius-3.0.13-8.el7_4.x86_64 freeradius-utils-3.0.13-8.el7_4.x86_64 Thank you for the quick responses!
participants (3)
-
Alan Buxey -
Dave Macias -
Herwin Weststrate