SEVERE! radiusd 2.0 and 1.1.4 dying! Segmentation fault
Hi, Freeradius 2.0 alpha was working correctly since November 1st. Then, this month, suddenly the server started to die, complaining of "Info: rlm_sql (sql): There are no DB handles to use! skipped 0, tried to connect 0". The server runs threaded with max_servers = 32 and num_sql_socks = 32 (there are 5 reqs per seconds, no more than that). Ok so I've tried to run it single threaded (-X), but then, it's slow and it missess some access requests, due to processing the accounting. I've uninstalled it and installed 1.1.4, but the same occurs! Restarting radiusd when it fails gives another 15 minutes before it dies again. Also, disabling accounting helps prolong the server lifetime. Any clue on that? Thanks. Sat Jan 27 19:13:16 2007 : Debug: modsingle[accounting]: returned from detail (rlm_detail) for request 108 Sat Jan 27 19:13:16 2007 : Debug: modcall[accounting]: module "detail" returns ok for request 108 Sat Jan 27 19:13:16 2007 : Debug: modsingle[accounting]: calling ippool (rlm_sqlippool) for request 108 Sat Jan 27 19:13:16 2007 : Debug: rlm_sql (sql_postgresql): Reserving sql socket id: 11 Sat Jan 27 19:13:16 2007 : Debug: radius_xlat: 'BEGIN' ********** Internal heap ERROR 17177 addr=(nil) ********* ****************************************************** HEAP DUMP heap name="Alloc statemen" desc=0x77e2b8 extent sz=0x1040 alt=32767 het=32767 rec=0 flg=2 opc=3 parent=0x77adb0 owner=(nil) nex=(nil) xsz=0x1040 EXTENT 0 addr=0x788818 Chunk 000788828 sz= 3752 free " " Chunk 0007896d0 sz= 312 freeable assoc with mark prv=(nil) nxt=(nil) Chunk 000789808 sz= 80 freeable assoc with mark prv=(nil) nxt=(nil) EXTENT 1 addr=0x77d3e8 Chunk 00077d3f8 sz= 2448 perm "perm " alo=32 Total heap size = 6592 FREE LISTS: Bucket 0 size=160 Bucket 1 size=288 Bucket 2 size=544 Bucket 3 size=1056 Bucket 4 size=2080 Chunk 000788828 sz= 3752 free " " Bucket 5 size=4128 Bucket 6 size=16416 Bucket 7 size=32800 Total free space = 3752 UNPINNED RECREATABLE CHUNKS (lru first): PERMANENT CHUNKS: Chunk 00077d3f8 sz= 2448 perm "perm " alo=32 Permanent space = 2448 ****************************************************** Hla: 255 ORA-21500: internal error code, arguments: [17177], [0x000000000], [], [], [], [], [], [] Errors in file : ORA-21500: internal error code, arguments: [17177], [0x000000000], [], [], [], [], [], [] ----- Call Stack Trace ----- calling call entry argument values in hex location type point (? means dubious value) -------------------- -------- -------------------- ---------------------------- Cannot seek to string table section header in /proc/11022/exe. Cannot seek to string table section header in /proc/11022/exe. 9688CDEF CALL 9660C588 0 ? 0 ? 774EC8 ? 0 ? 1 ? 0 ? 96DA64D8 CALLr 00000000 0 ? 0 ? 655680 ? 0 ? 4FA13060 ? 0 ? 96DA6CD4 CALL 965ED0E8 Sat Jan 27 19:13:16 2007 : Debug: radius_xlat: 'SELECT id,UserName,Attribute,Value,op FROM radreply WHERE Username = 'alfbericoz@foo.com' ORDER BY id' 0 ? 0 ? FF ? 0 ? 0 ? 0 ? 96DA6898 CALL 96605AC8 4FA13090 ? 0 ? 655680 ? 0 ? FF ? 0 ? 96D75B7F CALL 965FF0C8 0 ? 0 ? 0 ? 0 ? 788ED0 ? 0 ? 96D9135D CALL 96D757AA 0 ? 0 ? 0 ? 0 ? 0 ? 0 ? 966CA4FA CALL 96607898 0 ? 0 ? 0 ? 0 ? 0 ? 0 ? 966DF8CE CALL 966070F8 77DDA8 ? 0 ? 781BF0 ? 0 ? 4FA15E50 ? 0 ? 966DF582 CALL 965F7D68 0 ? 0 ? B0D0A8C0 ? 3E ? B0D0AE20 ? 3E ? 966DBF1E CALL 965FEC88 0 ? 0 ? 0 ? 0 ? 0 ? 0 ? 9678A292 CALL 9660F088 0 ? 0 ? 4FA161D0 ? 0 ? 772E10 ?Sat Jan 27 19:13:16 2007 : Debug: radius_xlat: 'SELECT id,UserName,Attribute,Value,op FROM radreply WHERE Username = 'john@foo.com' ORDER BY id' 0 ? 962BB4F6 CALL 962BAD60 4FA160C0 ? 0 ? 6536A4 ? 0 ? 4FA162E2 ? 0 ? 0077C450 CALLs 00000000 ----- Argument/Register Address Dump ----- Argument/Register addr=774ec8. Dump of memory from 0x000774E88 to 0x000774FC8 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 FFFFFFFF 00000000 007773F0 00000000 0077AB38 00000000 0077AEE0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 007748D0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000001 00000000 973DC7A0 0000002A 00001000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00FA865A 000000FF 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 Argument/Register addr=655680. Dump of memory from 0x000655640 to 0x000655780 00000000 00000000 00000000 00000001 00000000 7273752F 636F6C2F 6C2F6C61 722F6269 00000000 735F6D6C 6F5F6C71 6C636172 2E312D65 2E342E31 00006F73 00000481 00000000 00000000 963BD000 0000002A 00654CC0 00000000 97657028 0000002A 00655B00 00000000 00000000 006551F0 00000000 00655680 00000000 00000000 00000000 00655AC8 00000000 00000000 00000000 00000000 97657078 0000002A 97657128 0000002A 97657118 0000002A 00000000 976570C8 0000002A 976570D8 0000002A 976570E8 0000002A 97657158 0000002A 00000000 97657168 0000002A 97657178 0000002A 976570F8 0000002A 97657108 0000002A 00000000 976570A8 0000002A 976570B8 0000002A 97657088 0000002A 97657098 0000002A 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 97657138 0000002A 00000000 00000000 97657188 0000002A 97657148 0000002A Argument/Register addr=4fa13060. Dump of memory from 0x04FA13020 to 0x04FA13160 00000000 4FA13060 00000000 96DA6CD9 0000002A 00774EC8 00000000 9688CBFE 0000002A 00000000 00000000 00000000 000000FF 00000000 00000000 00000000 00655680 00000000 00000000 4FA13140 00000000 96DA689D 0000002A 00000028 00000030 4FA13150 00000000 00000000 4FA13090 00000000 00655680 00000000 000000FF 00000000 97663E70 0000002A 00000000 000000FF 00000000 00000002 00000000 00000000 00000000 B0E2D800 0000003E 00000000 4FA131C0 00000000 B0A0A68A 0000003E 00000001 00000000 00000000 00000000 00000000 00774EC8 00000000 96411030 0000002A 00000000 00000000 B0A0A552 0000003E 00000000 00000000 00000000 00000002 00000000 00004319 00000000 00000000 00000000 00000000 00774EC8 00000000 00000000 00000000 B0E2D800 0000003E 00655680 00000000 00000000 4FA131C0 00000000 96D75B84 0000002A 00000000 0000003E 4FA13198 00000000 Argument/Register addr=788ed0. Dump of memory from 0x000788E90 to 0x000788FD0 00000000 00000000 00000000 00040000 00000000 00000000 00050001 00000029 009B51C0 00000000 009F97A6 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000105 00000029 00000000 00000000 00000000 00000000 00000000 00000000 00010001 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000029 00000000 00000002 00000002 00000002 00000000 00000000 00000000 00788E38 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00788B28 00000000 00000000 00000000 00000029 00000191 20B38F00 00788E18 00000000 00000000 00000000 00000000 00000000 00000000 F8E9DACB 00000604 Argument/Register addr=77dda8. Dump of memory from 0x00077DD68 to 0x00077DEA8 00000000 00000000 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000661 30B38F00 0077D3B8 00000000 00000000 00000000 00000000 00000000 00000000 F8E9DACB 00000401 007748D0 00000000 007748D0 00000000 00000000 40000400 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0077E2B8 00000000 00000000 00000000 00000000 00000000 0000007C 00000BFF 00000000 00000000 00000000 00000002 00000060 00000000 00000002 00000000 007896F0 00000000 00000112 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 Argument/Register addr=781bf0. Dump of memory from 0x000781BB0 to 0x000781CF0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 007846C0 00000000 00000000 00000007 00000000 00000000 00000000 00028000 00000002 00000000 00000000 00000000 00000000 00000000 007834E4 00000000 0000000D 00000000 00000000 00000000 00000000 00783518 00000000 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0078351A 00000000 0077DDA8 00000000 00000000 00000000 00000000 00000000 00000000 00783530 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 Repeat 1 times 00000000 00000000 00000000 00000000 00000000 Argument/Register addr=4fa15e50. Dump of memory from 0x04FA15E10 to 0x04FA15F50 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00783530 00000000 00781BF0 00000000 007834C8 00000000 00000000 00781BE8 00000000 9734DE24 0000002A 4FA15F90 00000000 966DBF23 0000002A 00000000 0077DDA8 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00774960 00000000 00000000 00000000 00000000 00000001 00000000 00000000 00000000 0077DDA8 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0077C450 00000000 0077D128 00000000 00000000 00000000 00000000 4FA161D0 00000000 00772DB0 00000000 00653200 00000000 00772DB0 00000000 00000000 00000001 00000000 4FA15FE0 00000000 00000000 00000000 00000000 00000000 00000000 0077DDA8 00000000 0077EFB8 00000000 0077DE28 00000000 0077E178 00000000 00000000 0077E180 00000000 0077C8E8 00000000 Argument/Register addr=4fa161d0. Dump of memory from 0x04FA16190 to 0x04FA162D0 00000000 00000000 00000000 4FA16840 00000000 00000000 955729B2 0000002A 00000000 00000000 00000000 00000000 00000000 00000001 00000000 00000020 00000000 00000000 0000000A 454C4553 72205443 72676461 7270756F 00000000 796C7065 2C64692E 67646172 70756F72 6C706572 72472E79 4E70756F 2C656D61 00000000 67646172 70756F72 6C706572 74412E79 62697274 2C657475 67646172 70756F72 00000000 6C706572 61562E79 2C65756C 67646172 70756F72 6C706572 706F2E79 52462020 00000000 72204D4F 72676461 7270756F 796C7065 6573752C 6F726772 57207075 45524548 Sat Jan 27 19:13:16 2007 : Debug: radius_xlat: 'SELECT radgroupreply.id,radgroupreply.GroupName,radgroupreply.Attribute,radgroupreply.Value,radgroupreply.op FROM radgroupreply,usergroup WHERE usergroup.Username = 'alfbericoz@foo.com' AND usergroup.GroupName = radgroupreply.GroupName ORDER BY radgroupreply.id' Sat Jan 27 19:13:16 2007 : Debug: rlm_sql_postgresql: Status: PGRES_COMMAND_OK Sat Jan 27 19:13:16 2007 : Debug: rlm_sql_postgresql: affected rows = Sat Jan 27 19:13:16 2007 : Debug: radius_xlat: 'UPDATE radippool SET nasipaddress = '', pool_key = 0, callingstationid = '', expiry_time = 'now'::timestamp(0) - '1 second'::interval WHERE nasipaddress = '100.125.1.1' AND pool_key = '2952792152' AND username = 'manoel@foo.com' AND callingstationid = '#BRAS-01#E110#2136'' Sat Jan 27 19:13:16 2007 : Debug: rlm_sql_postgresql: Status: PGRES_COMMAND_OK Sat Jan 27 19:13:16 2007 : Debug: rlm_sql_postgresql: Status: PGRES_COMMAND_OK Sat Jan 27 19:13:16 2007 : Debug: rlm_sql_postgresql: Status: PGRES_COMMAND_OK 00000000 65737520 6F726772 552E7075 6E726573 20656D61 6D27203D 656C6C75 72616D72 Sat Jan 27 19:13:16 2007 : Debug: rlm_sql_postgresql: affected rows = 0 Sat Jan 27 19:13:16 2007 : Debug: rlm_sql_postgresql: affected rows = Sat Jan 27 19:13:16 2007 : Debug: rlm_sql_postgresql: affected rows = 00000000 736E6974 72657340 746D6F63 632E6C65 622E6D6F 41202772 7520444E 67726573 Sat Jan 27 19:13:16 2007 : Debug: radius_xlat: 'UPDATE radippool SET nasipaddress = '', pool_key = 0, callingstationid = '', expiry_time = 'now'::timestamp(0) - '1 second'::interval WHERE nasipaddress = '100.125.1.1' AND pool_key = '2147485784' AND username = 'manoel@foo.com' AND callingstationid = '#BRAS-01#E80#2136'' Sat Jan 27 19:13:16 2007 : Debug: radius_xlat: 'UPDATE radippool SET nasipaddress = '', pool_key = 0, callingstationid = '', expiry_time = 'now'::timestamp(0) - '1 second'::interval WHERE nasipaddress = '100.125.1.1' AND pool_key = '2147485849' AND username = 'cezar@foo.com' AND callingstationid = '#BRAS-01#E80#2201'' Sat Jan 27 19:13:16 2007 : Debug: radius_xlat: 'COMMIT' 00000000 70756F72 6F72472E 614E7075 3D20656D 64617220 756F7267 70657270 472E796C 00000000 70756F72 656D614E 44524F20 42205245 Argument/Register addr=772e10. Dump of memory from 0x000772DD0 to 0x000772F10 00000000 00000000 00000000 00000000 00000000 00000000 00748250 00000000 00000000 00000000 00772E10 00000000 00000000 00000000 00000000 00000000 00000000 00000051 00000000 007748D0 00000000 0077C450 00000000 00000000 0077D128 00000000 0077DDA8 00000000 009CB560 00000000 009E1E80 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000071 00000000 00000000 00656480 00000000 00000000 00000000 B0E306B8 0000003E 0076D7A0 00000000 00000000 0076D370 00000000 00000000 00000000 0076D480 00000000 00000008 00000040 00000000 00772ED0 00000000 00000000 00000000 0076D690 00000000 00000007 00000020 00000000 00000000 00000000 00000091 00000000 B0E30828 0000003E B0E30828 0000003E 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 Argument/Register addr=4fa160c0. Dump of memory from 0x04FA16080 to 0x04FA161C0 00000000 006528B0 00000000 984025B0 0000002A 0091DFA0 00000000 00772DB0 00000000 00000000 4FA161D0 00000000 961B5744 0000002A 00000000 00000000 4FA160D0 00000000 00000000 00000000 00000000 00A02350 00000000 6C6C756D 616D7265 6E697472 65734073 00000000 6D6F6372 2E6C6574 2E6D6F63 00007262 955728DF 0000002A 4FA167B0 00000000 00000000 4FA16776 00000000 00000002 73000000 00000320 00000000 00000000 00000000 00000000 00000000 00000000 4FA16360 00000000 FFFFFFFF 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000002 4FA16698 00000000 955729B2 0000002A 00000000 00000020 0000000E FFFFFFFF FFFFFFFF 00000000 00000000 4FA16840 00000000 00000000 955729B2 0000002A 00000000 00000000 00000000 00000000 00000000 00000001 Argument/Register addr=6536a4. Dump of memory from 0x000653664 to 0x0006537A4 00000000 472E7075 70756F72 656D614E 72203D20 72676461 7270756F 796C7065 00000000 6F72472E 614E7075 4F20656D 52454452 20594220 67646172 70756F72 6C706572 00000000 64692E79 00000000 000002D1 00000000 41445055 72204554 63616461 53207463 00000000 41205445 53746363 54706F74 3D656D69 445F4F54 28455441 27532527 7979272C 00000000 6D2D7979 64642D6D 32686820 696D3A34 2773733A 41202C29 53746363 69737365 00000000 69546E6F 283D656D 74786528 74636172 63657328 20646E6F 6D6F7266 75632820 00000000 6E657272 69745F74 7473656D 20706D61 5428202D 41445F4F 27284554 2C275325 00000000 79797927 6D6D2D79 2064642D 34326868 3A696D3A 29277373 20292929 7865202B 00000000 63617274 696D2874 6574756E 6F726620 6328206D 65727275 745F746E 73656D69 00000000 706D6174 28202D20 445F4F54 28455441 27532527 7979272C 6D2D7979 64642D6D 00000000 32686820 Argument/Register addr=4fa162e2. Dump of memory from 0x04FA162A2 to 0x04FA163E2 00000000 70756F72 6F72472E 614E7075 3D20656D 64617220 756F7267 70657270 472E796C 00000000 70756F72 656D614E 44524F20 42205245 61722059 6F726764 65727075 2E796C70 00000000 00006469 00000000 009F7BBC 00000000 4FA16440 00000000 955728D4 0000002A 00000000 00000006 00000000 984025B0 0000002A 009F7BBC 00000000 B0C5CFB9 0000003E 00000000 FBAD8001 00000000 009F7BBC 00000000 009F7BBC 00000000 009F7BBC 00000000 00000000 009F7BBC 00000000 009F7BC7 00000000 FFFFFFFF FFFFFFFF 009F7BBC 00000000 00000000 FFFFFFFF FFFFFFFF 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 4FA164F0 00000000 009CCBB0 00000000 00000000 4FA10000 00000000 00000000 00000000 000003F2 00000000 B0C48276 0000003E 00000000 00000018 00000030 00000000 00000000 4FA163E0 00000000 00521080 00000000 00000000 FFFFFFFF ----- End of Call Stack Trace ----- Segmentation fault
Guilherme Franco wrote:
Hi,
Freeradius 2.0 alpha was working correctly since November 1st.
Then, this month, suddenly the server started to die, complaining of "Info: rlm_sql (sql): There are no DB handles to use! skipped 0, tried to connect 0".
This normally means your database is slow. Clean out old accounting records (maybe move them to another table) and execute a "vacuum analyze".
The server runs threaded with max_servers = 32 and num_sql_socks = 32 (there are 5 reqs per seconds, no more than that).
Ok so I've tried to run it single threaded (-X), but then, it's slow and it missess some access requests, due to processing the accounting.
...indicating a high load, supporting the hypothesis.
I've uninstalled it and installed 1.1.4, but the same occurs!
Restarting radiusd when it fails gives another 15 minutes before it dies again.
Also, disabling accounting helps prolong the server lifetime.
Probably because it reduces the load, again supporting the hypothesis. However - you also say it is segfaulting? Which I would not expect. I don't really understand the format of the crash dump - can you supply one from "gdb" as documented in doc/bugs?
Thanks Mr. Mayers, The database is Oracle on a powerful machine which only do acct/ auth. All the relevant auth/ accounting queries are indexed to speed things up. There's a PostgreSQL database to take care of the sqlippool module. The strange thing is that even when the accounting is off (with low load then) the error appears randomly. Also, if the proxy realm dies the problem occurs too. That segfault was captured by running "radiusd -xxx", which pinpoints to an Oracle OCI error in this case (with acct on). I can't give you a gdb because the server is running fine now, but who knows when it may happen... That setup was running fine for almost 3 months. All indicates a resource starving problem, but the load is low :( Thank you very much.
Guilherme Franco wrote:
The strange thing is that even when the accounting is off (with low load then) the error appears randomly.
Also, if the proxy realm dies the problem occurs too. ... I can't give you a gdb because the server is running fine now, but who knows when it may happen...
Then please set up a test server, with a test DB so that you can reproduce the problem, and not affect your users. If you're running Linux, "valgrind" is a good tool. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
participants (3)
-
Alan DeKok -
Guilherme Franco -
Phil Mayers