Crash due to fr_packet_cmp
Hello , I have a big problem with my freeradius ... After some time with load the freeradius crashes We first noticed this crash with our plugins activated, but then we deactivated all plugins and used "default" configuration: /usr/local/etc/raddb/users: DEFAULT Auth-Type := Accept Reply-Message = "test freeradius" /usr/local/etc/raddb/sites-enabled/default authorize { files } accounting { ok } and crashes still occured. This crash has been observed while load testing, so there is no debug logs. Crash time appears to be dependant on load: 1400 req/s >> 4h - 6h 700 req/s >> 8h - 10h An analyse of core files shows the same error in accounting or authentication: core.8555 Program terminated with signal 11, Segmentation fault. (gdb) bt #0 fr_packet_cmp (a=0xb4897cd8, b=0x0) at packet.c:139 #1 0x00c80d98 in packet_entry_cmp (one=0xbf89cf94, two=0x8e21a80) at packet.c:400 #2 0x00c71b39 in list_find (ht=0x8d49520, head=<value optimized out>, reversed=285904539, data=0xbf89cf94) at hash.c:191 #3 0x00c71c43 in fr_hash_table_find (ht=0x8d49520, data=0xbf89cf94) at hash.c:461 #4 0x00c71c71 in fr_hash_table_finddata (ht=0x8d49520, data=0xbf89cf94) at hash.c:491 #5 0x00c80a29 in fr_packet_list_find (pl=0xc80d70, request=0xb4897cd8) at packet.c:536 #6 0x0806cf83 in received_request (listener=0x8d4f608, packet=0xb4897cd8, prequest=0xbf89d0dc, client=0x8d2fc80) at event.c:2723 #7 0x0805ab24 in acct_socket_recv (listener=0x8d4f608, pfun=0xbf89d0e0, prequest=0xbf89d0dc) at listen.c:889 #8 0x08068e00 in event_socket_handler (xel=0x8d49d70, fd=6, ctx=0xc80d70) at event.c:3288 #9 0x00c81b05 in fr_event_loop (el=0x8d49d70) at event.c:400 #10 0x080698c6 in radius_event_process () at event.c:3632 #11 0x080620ca in main (argc=1, argv=0xbf89d334) at radiusd.c:398 === core.21300 (gdb) bt #0 fr_packet_cmp (a=0xb2d7a858, b=0x0) at packet.c:139 #1 0x00457d98 in packet_entry_cmp (one=0xbf8e0fe4, two=0xb2f316e8) at packet.c:400 #2 0x00448b39 in list_find (ht=0x8e3d590, head=<value optimized out>, reversed=640351498, data=0xbf8e0fe4) at hash.c:191 #3 0x00448c43 in fr_hash_table_find (ht=0x8e3d590, data=0xbf8e0fe4) at hash.c:461 #4 0x00448c71 in fr_hash_table_finddata (ht=0x8e3d590, data=0xbf8e0fe4) at hash.c:491 #5 0x00457a29 in fr_packet_list_find (pl=0x457d70, request=0xb2d7a858) at packet.c:536 #6 0x0806cf83 in received_request (listener=0x89819a0, packet=0xb2d7a858, prequest=0xbf8e112c, client=0x896ac98) at event.c:2723 #7 0x0805af38 in auth_socket_recv (listener=0x89819a0, pfun=0xbf8e1130, prequest=0xbf8e112c) at listen.c:801 #8 0x08068e00 in event_socket_handler (xel=0x8e3b920, fd=9, ctx=0x457d70) at event.c:3288 #9 0x00458b05 in fr_event_loop (el=0x8e3b920) at event.c:400 #10 0x080698c6 in radius_event_process () at event.c:3632 #11 0x080620ca in main (argc=1, argv=0xbf8e1384) at radiusd.c:398 I need help :( have a fun Fabiana
fabiana marvani wrote:
After some time with load the freeradius crashes
We first noticed this crash with our plugins activated, but then we deactivated all plugins and used "default" configuration:
There have been a few reports similar to this. They all require sending the server many 10s of millions of packets over a long time. This makes it hard to reproduce && debug. It's likely a race condition. But it's hard to say where, or why.
core.8555 Program terminated with signal 11, Segmentation fault. (gdb) bt #0 fr_packet_cmp (a=0xb4897cd8, b=0x0) at packet.c:139 ... #6 0x0806cf83 in received_request (listener=0x8d4f608, packet=0xb4897cd8, prequest=0xbf89d0dc, client=0x8d2fc80) at event.c:2723
The server keeps all packets in a hash, to ensure it catches duplicates, etc. The hash is keyed by the source packet (src/dst ip/port). The crash comes because the "request" structure is still in the hash, though the "packet" entry in that structure has become NULL. The only problem is... the packet entry is *only* set to NULL after the request has been deleted from hash. And *only* the main thread adds/deletes entries to the hash. And *only* the main thread allocates or free's request data structures. So this is a problem that should be avoided completely by the design of the server. Some questions: - which OS && CPU (32 / 64-bit) - which version of the server - which command line was used to run the server - is this reproducible in non-threaded mode (radiusd -fs) If you are using an older version of the server, please also try with the current git "stable" branch (see git.freeradius.org). It has some changes which give it only one code path for doing certain kinds of request mangling. This makes it less likely for there to be errors, race conditions, etc. Alan DeKok.
participants (2)
-
Alan DeKok -
fabiana marvani