Random segfault crash

[Dave Macias]

Hello,

OS: CentOS 7
Kernel: 3.10.0-693.2.2.el7.x86_64
radiusd: FreeRADIUS Version 3.0.13, for host x86_64-redhat-linux-gnu, built
on Aug 23 2017 at 15:18:22

Recently upgraded freeradius and noticed that it is crashing at random
times. Dont see anything interesting in the logs. $ radiusd -X starts
normally. But do see segfaults in $ dmesg


[ 2886.030026] perf: interrupt took too long (2511 > 2500), lowering
kernel.perf_event_max_sample_rate to 79000
[ 6420.898264] perf: interrupt took too long (3143 > 3138), lowering
kernel.perf_event_max_sample_rate to 63000
[14198.608930] radiusd[32759]: segfault at e1a32313a ip 00007f06a7f9f6e4 sp
00007fffe181ea20 error 4 in libc-2.17.so[7f06a7f57000+1b8000]
[14975.220890] perf: interrupt took too long (4010 > 3928), lowering
kernel.perf_event_max_sample_rate to 49000
[21863.497234] perf: interrupt took too long (5298 > 5012), lowering
kernel.perf_event_max_sample_rate to 37000
[24292.736627] radiusd[12730]: segfault at 7fc6e4000d80 ip 00007fc6e4000d80
sp 00007fffe6524f28 error 15
[29848.535819] radiusd[26477]: segfault at 18 ip 00007fe03d119418 sp
00007ffcc24e1dd0 error 4 in libc-2.17.so[7fe03d09d000+1b8000]
[36113.258489] traps: radiusd[7203] general protection ip:7f06d80cd27f
sp:7f06cfda2e40 error:0 in libc-2.17.so[7f06d8051000+1b8000]
[40207.985076] traps: radiusd[4354] general protection ip:7fee629196e4
sp:7ffcd65ffcd0 error:0 in libc-2.17.so[7fee628d1000+1b8000]
[56725.833141] IPv6: udp checksum is 0 for [2804:1f06:1:2048:aa9d:21ff:
fe47:876e]:1645->[2403:7300:101:36::12]:1812
[56726.856901] IPv6: udp checksum is 0 for [2804:1f06:1:2048:aa9d:21ff:
fe47:876e]:1645->[2403:7300:101:36::12]:1812
[60021.572076] in:imjournal[1692]: segfault at 0 ip 00007f5afccb8983 sp
00007f5af97c8b50 error 4 in imjournal.so[7f5afccb6000+5000]
[68735.882196] IPv6: udp checksum is 0 for [2607:2400:2:2187:23a:7dff:
fed4:5d92]:1645->[2403:7300:101:36::12]:1812
[68736.906013] IPv6: udp checksum is 0 for [2607:2400:2:2187:23a:7dff:
fed4:5d92]:1645->[2403:7300:101:36::12]:1812
[69014.260230] IPv6: udp checksum is 0 for [2607:2400:5:2151:207:7dff:
fe1d:b706]:1645->[2403:7300:101:36::12]:1812
[69015.308755] IPv6: udp checksum is 0 for [2607:2400:5:2151:207:7dff:
fe1d:b706]:1645->[2403:7300:101:36::12]:1812
[73005.257649] IPv6: udp checksum is 0 for [2a03:6940:1:2068:1ae7:28ff:
fef7:61bd]:1645->[2403:7300:101:36::12]:1812
[73006.311233] IPv6: udp checksum is 0 for [2a03:6940:1:2068:1ae7:28ff:
fef7:61bd]:1645->[2403:7300:101:36::12]:1812
[110480.127950] IPv6: udp checksum is 0 for [2607:2400:1:2100:206:f6ff:
fed2:61f8]:1645->[2403:7300:101:36::12]:1812
[110481.209600] IPv6: udp checksum is 0 for [2607:2400:1:2100:206:f6ff:
fed2:61f8]:1645->[2403:7300:101:36::12]:1812
[117945.839480] in:imjournal[14521]: segfault at 0 ip 00007f5cc505b983 sp
00007f5cc136bb50 error 4 in imjournal.so[7f5cc5059000+5000]
[122742.076476] Key type dns_resolver registered
[122742.118795] Key type cifs.spnego registered
[122742.118799] Key type cifs.idmap registered
[167924.574413] in:imjournal[9288]: segfault at 0 ip 00007f973987f983 sp
00007f973538fb50 error 4 in imjournal.so[7f973987d000+5000]
[210134.352360] conntrack: generic helper won't handle protocol 47. Please
consider loading the specific helper module.
[218406.756306] in:imjournal[7538]: segfault at 0 ip 00007fed6c091983 sp
00007fed673a1b50 error 4 in imjournal.so[7fed6c08f000+5000]
[268886.722800] in:imjournal[24594]: segfault at 0 ip 00007f9abd926983 sp
00007f9ab8436b50 error 4 in imjournal.so[7f9abd924000+5000]
[305921.081321] IPv6: udp checksum is 0 for [2607:2400:1:2116:e22f:6dff:
fe91:f6e1]:1645->[2403:7300:101:36::12]:1812
[305922.162078] IPv6: udp checksum is 0 for [2607:2400:1:2116:e22f:6dff:
fe91:f6e1]:1645->[2403:7300:101:36::12]:1812
[316036.147874] IPv6: udp checksum is 0 for [2607:2400:1:2188:6273:5cff:
fe2f:4207]:1645->[2403:7300:101:36::12]:1812
[316037.196177] IPv6: udp checksum is 0 for [2607:2400:1:2188:6273:5cff:
fe2f:4207]:1645->[2403:7300:101:36::12]:1812
[337566.944167] in:imjournal[10826]: segfault at 0 ip 00007f3bc891c983 sp
00007f3bc2c2cb50 error 4 in imjournal.so[7f3bc891a000+5000]
[355474.169434] radiusd[5184]: segfault at f5594a28 ip 00007f30d0129b00 sp
00007f30c537f6e0 error 4 in libtalloc.so.2.1.9[7f30d0126000+f000]
[362624.748526] radiusd[32532]: segfault at 18 ip 00007f554e2112ce sp
00007ffdec151d90 error 4 in libc-2.17.so[7f554e195000+1b8000]
[366369.197821] traps: radiusd[32147] general protection ip:7f824fdaaad5
sp:7f8247a7d290 error:0 in libc-2.17.so[7f824fd2d000+1b8000]
[368382.126990] radiusd[15692]: segfault at 18 ip 00007f5c5c5b052d sp
00007f5c51280d10 error 4 in libc-2.17.so[7f5c5c535000+1b8000]
[368415.327048] traps: radiusd[18044] general protection ip:7f3975b0881d
sp:7f396b7cc6a8 error:0 in libc-2.17.so[7f3975a7f000+1b8000]
[369617.422068] radiusd[25252]: segfault at 7f4dfef4a9d8 ip
00007f4e078c5418 sp 00007ffcf89273e0 error 4 in libc-2.17.so[7f4e07849000+
1b8000]
[374905.093166] traps: radiusd[605] general protection ip:7fb7254ef81d
sp:7fb7189abc28 error:0 in libc-2.17.so[7fb725466000+1b8000]
[376801.247227] radiusd[15436]: segfault at 7fbe5299e5d8 ip
00007fbe7d72c418 sp 00007ffc52397530 error 4 in libc-2.17.so[7fbe7d6b0000+
1b8000]
[377761.161832] radiusd[21228]: segfault at 7fee80002e10 ip
00007fee80002e10 sp 00007ffdee783a68 error 15
[379843.294430] radiusd[6923]: segfault at 18 ip 00007f4518b36418 sp
00007ffeb09da9e0 error 4 in libc-2.17.so[7f4518aba000+1b8000]
[389971.741324] radiusd[15204]: segfault at 7f07c54771d8 ip
00007f07adc8b52d sp 00007f07a315cb20 error 4 in libc-2.17.so[7f07adc10000+
1b8000]
[393581.655644] radiusd[12099]: segfault at 18 ip 00007f7a9f2a82ce sp
00007ffeabae65e0 error 4 in libc-2.17.so[7f7a9f22c000+1b8000]
[397222.777713] in:imjournal[4812]: segfault at 0 ip 00007f5fe8cef983 sp
00007f5fe27ffb50 error 4 in imjournal.so[7f5fe8ced000+5000]
[397225.678506] radiusd[9651]: segfault at 18 ip 00007f20c6ede418 sp
00007f20a7ffdd40 error 4 in libc-2.17.so[7f20c6e62000+1b8000]
[398893.363679] IPv6: udp checksum is 0 for [2a03:c9c0:4:2134:5e83:8fff:
fee3:9e06]:1645->[2403:7300:101:36::12]:1812
[398894.419431] IPv6: udp checksum is 0 for [2a03:c9c0:4:2134:5e83:8fff:
fee3:9e06]:1645->[2403:7300:101:36::12]:1812
[402638.895409] radiusd[18160]: segfault at 18 ip 00007fac35bcc52d sp
00007fac2a89cd10 error 4 in libc-2.17.so[7fac35b51000+1b8000]
[405040.177274] IPv6: udp checksum is 0 for [2806:210:1:2089:8a5a:92ff:
fef7:4208]:1645->[2403:7300:101:36::12]:1812
[405041.200663] IPv6: udp checksum is 0 for [2806:210:1:2089:8a5a:92ff:
fef7:4208]:1645->[2403:7300:101:36::12]:1812
[405071.980878] radiusd[2973]: segfault at 8 ip 00007f8a7ae76cb9 sp
00007ffd8eb385f0 error 4 in libtalloc.so.2.1.9[7f8a7ae74000+f000]
[407617.663020] traps: radiusd[25365] general protection ip:7f1d269a2ad5
sp:7f1d1d6733b0 error:0 in libc-2.17.so[7f1d26925000+1b8000]
[411156.319552] radiusd[18616]: segfault at 43bfc68 ip 00007f03651f5b00 sp
00007f0342ffbc40 error 4 in libtalloc.so.2.1.9[7f03651f2000+f000]
[413030.189572] radiusd[30421]: segfault at 18 ip 00007f6f058529f6 sp
00007f6ef951d450 error 4 in libc-2.17.so[7f6f057d5000+1b8000]
[413156.407921] traps: radiusd[31701] general protection ip:7f666dcdb6e4
sp:7ffc87dab4a0 error:0 in libc-2.17.so[7f666dc93000+1b8000]
[413605.949866] radiusd[1624]: segfault at 0 ip 00007f7b648974c9 sp
00007f7b557eee80 error 4 in libfreeradius-radius.so[7f7b6488c000+3c000]
[460136.289414] in:imjournal[10498]: segfault at 0 ip 00007f7c42347983 sp
00007f7c3b657b50 error 4 in imjournal.so[7f7c42345000+5000]
[474167.794387] IPv6: udp checksum is 0 for [2607:2400:3:2109:aa9d:21ff:
fe03:274e]:1645->[2403:7300:101:36::12]:1812
[474168.850441] IPv6: udp checksum is 0 for [2607:2400:3:2109:aa9d:21ff:
fe03:274e]:1645->[2403:7300:101:36::12]:1812


We have 3 radius servers displaying the same behavior. Anything to try?

Any input is appreciated!

thank you,
-dave