We have recently attempted to upgrade from 3.0.15 to 3.0.21 but are seeing very frequent crashes. Config unchanged from 3.0.15. Any ideas appreciated. Regards, Adrian Smith Examples from the radius.log: CONSISTENCY CHECK FAILED src/lib/cursor.c[219]: VALUE_PAIR "BTOpenzone-Roaming-Direction" length 42696464 is greater than char buffer length 7 SOFT ASSERT FAILED src/lib/pair.c[2341]: 0 CAUGHT SIGNAL: Aborted Backtrace of last 15 frames: /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_fault+0x115)[0x7fab75ae7490] /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_assert_cond+0x4c)[0x7fab75ae7f50] /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_pair_verify+0x3af)[0x7fab75af8c52] /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_cursor_next+0x58)[0x7fab75ae641f] /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_pair_list_verify+0x151)[0x7fab75af9169] /opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(+0x25482)[0x7fab75d4c482] /opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(verify_request+0x16c)[0x7fab75d4c5f8] /opt/freeradius-server-3.0.21/sbin/radiusd[0x43d3f0] /opt/freeradius-server-3.0.21/sbin/radiusd[0x43d1f6] /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_event_run+0x144)[0x7fab75b0f7d4] /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_event_loop+0x509)[0x7fab75b10117] /opt/freeradius-server-3.0.21/sbin/radiusd(radius_event_process+0x26)[0x4492d5] /opt/freeradius-server-3.0.21/sbin/radiusd(main+0xcbb)[0x43274b] /lib64/libc.so.6(__libc_start_main+0xfd)[0x3a0081ed1d] /opt/freeradius-server-3.0.21/sbin/radiusd[0x40f489] No panic action set and Mon Aug 24 01:20:40 2020 : Error: CONSISTENCY CHECK FAILED src/main/process.c[639]: Expected RADIUS_PACKET request to be parented by 0xd1f5d0 (REQUEST), but parented by 0xd04680 (null_context) Talloc chunk lineage: 0xd1f230 (RADIUS_PACKET) < 0xd04680 (null_context) Talloc context level 0: full talloc report on 'null_context' (total 466556 bytes in 3189 blocks) session-state contains 14 bytes in 2 blocks (ref 0) 0xd8c690 .name contains 14 bytes in 1 blocks (ref 0) 0xd79b20 acct_listener_pool contains 15485 bytes in 101 blocks (ref 0) 0xd77ab0 REQUEST contains 7277 bytes in 100 blocks (ref 0) 0xd77e30 RADIUS_PACKET contains 3340 bytes in 48 blocks (ref 0) 0x7f090001f090 uint8_t contains 481 bytes in 1 blocks (ref 0) 0x7f090000fd40 <snip> Mon Aug 24 01:20:40 2020 : Error: ASSERT FAILED src/main/util.c[1095]: 0 CAUGHT SIGNAL: Aborted Backtrace of last 12 frames: /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_fault+0x115)[0x7f092d5aa490] /opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(rad_assert_fail+0x46)[0x7f092d80e438] /opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(+0x25437)[0x7f092d80f437] /opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(verify_request+0x10a)[0x7f092d80f596] /opt/freeradius-server-3.0.21/sbin/radiusd[0x43d3f0] /opt/freeradius-server-3.0.21/sbin/radiusd[0x43d1f6] /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_event_run+0x144)[0x7f092d5d27d4] /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_event_loop+0x509)[0x7f092d5d3117] /opt/freeradius-server-3.0.21/sbin/radiusd(radius_event_process+0x26)[0x4492d5] /opt/freeradius-server-3.0.21/sbin/radiusd(main+0xcbb)[0x43274b] /lib64/libc.so.6(__libc_start_main+0xfd)[0x3a0081ed1d] /opt/freeradius-server-3.0.21/sbin/radiusd[0x40f489] No panic action set and CONSISTENCY CHECK FAILED src/lib/cursor.c[219]: VALUE_PAIR "NAS-Identifier" char buffer is not parented by VALUE_PAIR 0x7fc3f0014480, instead parented by 0x231dee0 (null_context) SOFT ASSERT FAILED src/lib/pair.c[2358]: 0 CAUGHT SIGNAL: Aborted Backtrace of last 15 frames: /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_fault+0x115)[0x7fc46fbd8490] /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_assert_cond+0x4c)[0x7fc46fbd8f50] /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_pair_verify+0x4cc)[0x7fc46fbe9d6f] /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_cursor_next+0x58)[0x7fc46fbd741f] /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_pair_list_verify+0x151)[0x7fc46fbea169] /opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(+0x25482)[0x7fc46fe3d482] /opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(verify_request+0x16c)[0x7fc46fe3d5f8] /opt/freeradius-server-3.0.21/sbin/radiusd[0x43d3f0] /opt/freeradius-server-3.0.21/sbin/radiusd[0x43d1f6] /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_event_run+0x144)[0x7fc46fc007d4] /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_event_loop+0x509)[0x7fc46fc01117] /opt/freeradius-server-3.0.21/sbin/radiusd(radius_event_process+0x26)[0x4492d5] /opt/freeradius-server-3.0.21/sbin/radiusd(main+0xcbb)[0x43274b] /lib64/libc.so.6(__libc_start_main+0xfd)[0x33d121ed1d] /opt/freeradius-server-3.0.21/sbin/radiusd[0x40f489] No panic action set and Mon Aug 24 01:20:49 2020 : Error: CONSISTENCY CHECK FAILED src/main/process.c[2919]: Expected RADIUS_PACKET request to be parented by 0x1878490 (REQUEST), but parented by 0x184bee0 (null_context) Talloc chunk lineage: 0x1878170 (RADIUS_PACKET) < 0x184bee0 (null_context) Talloc context level 0: full talloc report on 'null_context' (total 64956 bytes in 465 blocks) session-state contains 14 bytes in 2 blocks (ref 0) 0x1873070 .name contains 14 bytes in 1 blocks (ref 0) 0x187e190 acct_listener_pool contains 15323 bytes in 94 blocks (ref 0) 0x196f200 <snip> Mon Aug 24 01:20:49 2020 : Error: ASSERT FAILED src/main/util.c[1095]: 0 CAUGHT SIGNAL: Aborted Backtrace of last 9 frames: /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_fault+0x115)[0x7f74c5e88490] /opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(rad_assert_fail+0x46)[0x7f74c60ec438] /opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(+0x25437)[0x7f74c60ed437] /opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(verify_request+0x10a)[0x7f74c60ed596] /opt/freeradius-server-3.0.21/sbin/radiusd[0x44221c] /opt/freeradius-server-3.0.21/sbin/radiusd[0x43f5a3] /opt/freeradius-server-3.0.21/sbin/radiusd[0x43af5c] /lib64/libpthread.so.0(+0x33d1607aa1)[0x7f74c5840aa1] /lib64/libc.so.6(clone+0x6d)[0x33d12e8bcd] No panic action set And Mon Aug 24 01:22:08 2020 : Error: CONSISTENCY CHECK FAILED src/main/process.c[1159]: Expected RADIUS_PACKET request to be parented by 0xe77dc0 (REQUEST), but parented by 0xd94ee0 (null_context) Talloc chunk lineage: 0xe77ab0 (RADIUS_PACKET) < 0xd94ee0 (null_context) Talloc context level 0: full talloc report on 'null_context' (total 594675 bytes in 3880 blocks) session-state contains 14 bytes in 2 blocks (ref 0) 0xe21380 .name contains 14 bytes in 1 blocks (ref 0) 0xda9720 acct_listener_pool contains 16303 bytes in 114 blocks (ref 0) 0xda3b10 REQUEST contains 8095 bytes in 113 blocks (ref 0) 0xda3ed0 RADIUS_PACKET contains 258 bytes in 4 blocks (ref 0) 0xe18100 VALUE_PAIR contains 82 bytes in 2 blocks (ref 0) 0x7f55f00076f0 uint8_t contains 2 bytes in 1 blocks (ref 0) 0x7f55f0006170 src/lib/radius.c:495 contains 24 bytes in 1 blocks (ref 0) 0xda9370 RADIUS_PACKET contains 3610 bytes in 52 blocks (ref 0) 0x7f5610028b50n and Mon Aug 24 01:22:11 2020 : Error: ASSERT FAILED src/main/util.c[566]: !request->in_request_hash CAUGHT SIGNAL: Aborted Backtrace of last 17 frames: /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_fault+0x115)[0x7f31e3310490] /opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(rad_assert_fail+0x46)[0x7f31e3574438] /opt/freeradius-server-3.0.21/lib/libfreeradius-server.so(+0x24481)[0x7f31e3574481] /usr/lib64/libtalloc.so.2(+0x2dbe)[0x7f31e2ee2dbe] /usr/lib64/libtalloc.so.2(+0x2c4b)[0x7f31e2ee2c4b] /usr/lib64/libtalloc.so.2(+0x2c4b)[0x7f31e2ee2c4b] /usr/lib64/libtalloc.so.2(+0x2c4b)[0x7f31e2ee2c4b] /usr/lib64/libtalloc.so.2(_talloc_free+0xfb)[0x7f31e2ee32db] /opt/freeradius-server-3.0.21/sbin/radiusd[0x43d2e2] /opt/freeradius-server-3.0.21/sbin/radiusd[0x43db8c] /opt/freeradius-server-3.0.21/sbin/radiusd[0x43d1f6] /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_event_run+0x144)[0x7f31e33387d4] /opt/freeradius-server-3.0.21/lib/libfreeradius-radius.so(fr_event_loop+0x509)[0x7f31e3339117] /opt/freeradius-server-3.0.21/sbin/radiusd(radius_event_process+0x26)[0x4492d5] /opt/freeradius-server-3.0.21/sbin/radiusd(main+0xcbb)[0x43274b] /lib64/libc.so.6(__libc_start_main+0xfd)[0x33d121ed1d] /opt/freeradius-server-3.0.21/sbin/radiusd[0x40f489] No panic action set
On Aug 24, 2020, at 4:31 AM, adrian.p.smith@bt.com wrote:
We have recently attempted to upgrade from 3.0.15 to 3.0.21 but are seeing very frequent crashes. Config unchanged from 3.0.15.
Are you running custom modules?
CONSISTENCY CHECK FAILED src/lib/cursor.c[219]: VALUE_PAIR "BTOpenzone-Roaming-Direction" length 42696464 is greater than char buffer length 7
That's bad. As background, we have some additional sanity checks we run on developer builds. These checks verify that the internal data structures are OK. The checks slow down the server substantially, but they are enormously useful for debugging. If the checks fail, then something is going seriously wrong. Even worse, if you have many such errors spread all over the server, then *everything* is wrong. My guess is that you built 3.0.21 on a system which already had FreeRADIUS installed. Maybe the headers got confused, as some data structures have changed. Build 3.0.21 on a clean system, or use the packages at http://packages.networkradius.com That release works in production without any of those errors. Alan DeKok.
fuck off dickweed On Monday, August 24, 2020, 7:28:09 AM EDT, Alan DeKok <aland@deployingradius.com> wrote: On Aug 24, 2020, at 4:31 AM, adrian.p.smith@bt.com wrote:
We have recently attempted to upgrade from 3.0.15 to 3.0.21 but are seeing very frequent crashes. Config unchanged from 3.0.15.
Are you running custom modules?
CONSISTENCY CHECK FAILED src/lib/cursor.c[219]: VALUE_PAIR "BTOpenzone-Roaming-Direction" length 42696464 is greater than char buffer length 7
That's bad. As background, we have some additional sanity checks we run on developer builds. These checks verify that the internal data structures are OK. The checks slow down the server substantially, but they are enormously useful for debugging. If the checks fail, then something is going seriously wrong. Even worse, if you have many such errors spread all over the server, then *everything* is wrong. My guess is that you built 3.0.21 on a system which already had FreeRADIUS installed. Maybe the headers got confused, as some data structures have changed. Build 3.0.21 on a clean system, or use the packages at http://packages.networkradius.com That release works in production without any of those errors. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Aug 24, 2020, at 9:26 AM, thepete10 <thepete10@bellsouth.net> wrote: ... That's good for a ban. Alan DeKok.
-----Original Message----- From: Freeradius-Users <freeradius-users-bounces+adrian.p.smith=bt.com@lists.freeradius.org> On Behalf Of Alan DeKok Sent: 24 August 2020 12:27 To: FreeRadius users mailing list <freeradius-users@lists.freeradius.org> Subject: Re: 3.0.21 consistency check failed On Aug 24, 2020, at 4:31 AM, adrian.p.smith@bt.com wrote:
We have recently attempted to upgrade from 3.0.15 to 3.0.21 but are seeing very frequent crashes. Config unchanged from 3.0.15.
Are you running custom modules?
CONSISTENCY CHECK FAILED src/lib/cursor.c[219]: VALUE_PAIR "BTOpenzone-Roaming-Direction" length 42696464 is greater than char buffer length 7
That's bad. As background, we have some additional sanity checks we run on developer builds. These checks verify that the internal data structures are OK. The checks slow down the server substantially, but they are enormously useful for debugging. If the checks fail, then something is going seriously wrong. Even worse, if you have many such errors spread all over the server, then *everything* is wrong. My guess is that you built 3.0.21 on a system which already had FreeRADIUS installed. Maybe the headers got confused, as some data structures have changed. Build 3.0.21 on a clean system, or use the packages at http://packages.networkradius.com That release works in production without any of those errors. Alan DeKok. Thanks Alan, will check our build process and give that a go. Adrian
participants (3)
-
adrian.p.smith＠bt.com -
Alan DeKok -
thepete10