SEVERE! radiusd 2.0 and 1.1.4 dying! Segmentation fault

27 Jan 2007

      Hi,

Freeradius 2.0 alpha was working correctly since November 1st.

Then, this month, suddenly the server started to die, complaining of
"Info: rlm_sql (sql): There are no DB handles to use! skipped 0, tried
to connect 0".

The server runs threaded with max_servers = 32 and num_sql_socks = 32
(there are 5 reqs per seconds, no more than that).

Ok so I've tried to run it single threaded (-X), but then, it's slow
and it missess some access requests, due to processing the accounting.

I've uninstalled it and installed 1.1.4, but the same occurs!

Restarting radiusd when it fails gives another 15 minutes before it dies again.

Also, disabling accounting helps prolong the server lifetime.

Any clue on that? Thanks.

Sat Jan 27 19:13:16 2007 : Debug:   modsingle[accounting]: returned
from detail (rlm_detail) for request 108
Sat Jan 27 19:13:16 2007 : Debug:   modcall[accounting]: module
"detail" returns ok for request 108
Sat Jan 27 19:13:16 2007 : Debug:   modsingle[accounting]: calling
ippool (rlm_sqlippool) for request 108
Sat Jan 27 19:13:16 2007 : Debug: rlm_sql (sql_postgresql): Reserving
sql socket id: 11
Sat Jan 27 19:13:16 2007 : Debug: radius_xlat:  'BEGIN'
********** Internal heap ERROR 17177 addr=(nil) *********

******************************************************
HEAP DUMP heap name="Alloc statemen"  desc=0x77e2b8
 extent sz=0x1040 alt=32767 het=32767 rec=0 flg=2 opc=3
 parent=0x77adb0 owner=(nil) nex=(nil) xsz=0x1040
EXTENT 0 addr=0x788818
  Chunk        000788828 sz=     3752    free      "               "
  Chunk        0007896d0 sz=      312    freeable assoc with mark
prv=(nil) nxt=(nil)
  Chunk        000789808 sz=       80    freeable assoc with mark
prv=(nil) nxt=(nil)
EXTENT 1 addr=0x77d3e8
  Chunk        00077d3f8 sz=     2448    perm      "perm           "  alo=32
Total heap size    =     6592
FREE LISTS:
 Bucket 0 size=160
 Bucket 1 size=288
 Bucket 2 size=544
 Bucket 3 size=1056
 Bucket 4 size=2080
  Chunk        000788828 sz=     3752    free      "               "
 Bucket 5 size=4128
 Bucket 6 size=16416
 Bucket 7 size=32800
Total free space   =     3752
UNPINNED RECREATABLE CHUNKS (lru first):
PERMANENT CHUNKS:
  Chunk        00077d3f8 sz=     2448    perm      "perm           "  alo=32
Permanent space    =     2448
******************************************************
 Hla: 255

ORA-21500: internal error code, arguments: [17177], [0x000000000], [],
[], [], [], [], []
Errors in file :
ORA-21500: internal error code, arguments: [17177], [0x000000000], [],
[], [], [], [], []

----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
Cannot seek to string table section header in /proc/11022/exe.
Cannot seek to string table section header in /proc/11022/exe.
9688CDEF             CALL     9660C588             0 ? 0 ? 774EC8 ? 0 ? 1 ? 0 ?
96DA64D8             CALLr    00000000             0 ? 0 ? 655680 ? 0 ?
                                                   4FA13060 ? 0 ?
96DA6CD4             CALL     965ED0E8             Sat Jan 27 19:13:16
2007 : Debug: radius_xlat:  'SELECT id,UserName,Attribute,Value,op
FROM radreply WHERE Username = 'alfbericoz@foo.com' ORDER BY id'
0 ? 0 ? FF ? 0 ? 0 ? 0 ?
96DA6898             CALL     96605AC8             4FA13090 ? 0 ? 655680 ? 0 ?
                                                   FF ? 0 ?
96D75B7F             CALL     965FF0C8             0 ? 0 ? 0 ? 0 ? 788ED0 ? 0 ?
96D9135D             CALL     96D757AA             0 ? 0 ? 0 ? 0 ? 0 ? 0 ?
966CA4FA             CALL     96607898             0 ? 0 ? 0 ? 0 ? 0 ? 0 ?
966DF8CE             CALL     966070F8             77DDA8 ? 0 ? 781BF0 ? 0 ?
                                                   4FA15E50 ? 0 ?
966DF582             CALL     965F7D68             0 ? 0 ? B0D0A8C0 ? 3E ?
                                                   B0D0AE20 ? 3E ?
966DBF1E             CALL     965FEC88             0 ? 0 ? 0 ? 0 ? 0 ? 0 ?
9678A292             CALL     9660F088             0 ? 0 ? 4FA161D0 ? 0 ?
                                                   772E10 ?Sat Jan 27
19:13:16 2007 : Debug: radius_xlat:  'SELECT
id,UserName,Attribute,Value,op FROM radreply WHERE Username =
'john@foo.com' ORDER BY id'
 0 ?
962BB4F6             CALL     962BAD60             4FA160C0 ? 0 ? 6536A4 ? 0 ?
                                                   4FA162E2 ? 0 ?
0077C450             CALLs    00000000

----- Argument/Register Address Dump -----

Argument/Register addr=774ec8.  Dump of memory from 0x000774E88 to 0x000774FC8
00000000                   00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 FFFFFFFF 00000000 007773F0 00000000 0077AB38 00000000 0077AEE0 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 007748D0 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000001 00000000 973DC7A0 0000002A 00001000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00FA865A 000000FF 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
Argument/Register addr=655680.  Dump of memory from 0x000655640 to 0x000655780
00000000 00000000 00000000 00000001 00000000 7273752F 636F6C2F 6C2F6C61 722F6269
00000000 735F6D6C 6F5F6C71 6C636172 2E312D65 2E342E31 00006F73 00000481 00000000
00000000 963BD000 0000002A 00654CC0 00000000 97657028 0000002A 00655B00 00000000
00000000 006551F0 00000000 00655680 00000000 00000000 00000000 00655AC8 00000000
00000000 00000000 00000000 97657078 0000002A 97657128 0000002A 97657118 0000002A
00000000 976570C8 0000002A 976570D8 0000002A 976570E8 0000002A 97657158 0000002A
00000000 97657168 0000002A 97657178 0000002A 976570F8 0000002A 97657108 0000002A
00000000 976570A8 0000002A 976570B8 0000002A 97657088 0000002A 97657098 0000002A
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 97657138 0000002A 00000000 00000000 97657188 0000002A 97657148 0000002A
Argument/Register addr=4fa13060.  Dump of memory from 0x04FA13020 to 0x04FA13160
00000000 4FA13060 00000000 96DA6CD9 0000002A 00774EC8 00000000 9688CBFE 0000002A
00000000 00000000 00000000 000000FF 00000000 00000000 00000000 00655680 00000000
00000000 4FA13140 00000000 96DA689D 0000002A 00000028 00000030 4FA13150 00000000
00000000 4FA13090 00000000 00655680 00000000 000000FF 00000000 97663E70 0000002A
00000000 000000FF 00000000 00000002 00000000 00000000 00000000 B0E2D800 0000003E
00000000 4FA131C0 00000000 B0A0A68A 0000003E 00000001 00000000 00000000 00000000
00000000 00774EC8 00000000 96411030 0000002A 00000000 00000000 B0A0A552 0000003E
00000000 00000000 00000000 00000002 00000000 00004319 00000000 00000000 00000000
00000000 00774EC8 00000000 00000000 00000000 B0E2D800 0000003E 00655680 00000000
00000000 4FA131C0 00000000 96D75B84 0000002A 00000000 0000003E 4FA13198 00000000
Argument/Register addr=788ed0.  Dump of memory from 0x000788E90 to 0x000788FD0
00000000                                     00000000 00000000 00040000 00000000
00000000 00050001 00000029 009B51C0 00000000 009F97A6 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000105 00000029 00000000 00000000
00000000 00000000 00000000 00000000 00010001 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000029
00000000 00000002 00000002 00000002 00000000 00000000 00000000 00788E38 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00788B28 00000000
00000000 00000000 00000029 00000191 20B38F00 00788E18 00000000 00000000 00000000
00000000 00000000 00000000 F8E9DACB 00000604
Argument/Register addr=77dda8.  Dump of memory from 0x00077DD68 to 0x00077DEA8
00000000                   00000000 00000000 00000001 00000000 00000000 00000000
00000000 00000000 00000000 00000661 30B38F00 0077D3B8 00000000 00000000 00000000
00000000 00000000 00000000 F8E9DACB 00000401 007748D0 00000000 007748D0 00000000
00000000 40000400 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 0077E2B8 00000000 00000000 00000000
00000000 00000000 0000007C 00000BFF 00000000 00000000 00000000 00000002 00000060
00000000 00000002 00000000 007896F0 00000000 00000112 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000001 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
Argument/Register addr=781bf0.  Dump of memory from 0x000781BB0 to 0x000781CF0
00000000                                     00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 007846C0 00000000
00000000 00000007 00000000 00000000 00000000 00028000 00000002 00000000 00000000
00000000 00000000 00000000 007834E4 00000000 0000000D 00000000 00000000 00000000
00000000 00783518 00000000 00000000 00000001 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 0078351A 00000000 0077DDA8 00000000 00000000 00000000
00000000 00000000 00000000 00783530 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
        Repeat 1 times
00000000 00000000 00000000 00000000 00000000
Argument/Register addr=4fa15e50.  Dump of memory from 0x04FA15E10 to 0x04FA15F50
00000000                                     00000000 00000000 00000000 00000000
00000000 00000000 00000000 00783530 00000000 00781BF0 00000000 007834C8 00000000
00000000 00781BE8 00000000 9734DE24 0000002A 4FA15F90 00000000 966DBF23 0000002A
00000000 0077DDA8 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00774960 00000000 00000000 00000000 00000000 00000001
00000000 00000000 00000000 0077DDA8 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 0077C450 00000000 0077D128 00000000 00000000 00000000
00000000 4FA161D0 00000000 00772DB0 00000000 00653200 00000000 00772DB0 00000000
00000000 00000001 00000000 4FA15FE0 00000000 00000000 00000000 00000000 00000000
00000000 0077DDA8 00000000 0077EFB8 00000000 0077DE28 00000000 0077E178 00000000
00000000 0077E180 00000000 0077C8E8 00000000
Argument/Register addr=4fa161d0.  Dump of memory from 0x04FA16190 to 0x04FA162D0
00000000                                     00000000 00000000 4FA16840 00000000
00000000 955729B2 0000002A 00000000 00000000 00000000 00000000 00000000 00000001
00000000 00000020 00000000 00000000 0000000A 454C4553 72205443 72676461 7270756F
00000000 796C7065 2C64692E 67646172 70756F72 6C706572 72472E79 4E70756F 2C656D61
00000000 67646172 70756F72 6C706572 74412E79 62697274 2C657475 67646172 70756F72
00000000 6C706572 61562E79 2C65756C 67646172 70756F72 6C706572 706F2E79 52462020
00000000 72204D4F 72676461 7270756F 796C7065 6573752C 6F726772 57207075 45524548
Sat Jan 27 19:13:16 2007 : Debug: radius_xlat:  'SELECT
radgroupreply.id,radgroupreply.GroupName,radgroupreply.Attribute,radgroupreply.Value,radgroupreply.op
 FROM radgroupreply,usergroup WHERE usergroup.Username =
'alfbericoz@foo.com' AND usergroup.GroupName = radgroupreply.GroupName
ORDER BY radgroupreply.id'
Sat Jan 27 19:13:16 2007 : Debug: rlm_sql_postgresql: Status: PGRES_COMMAND_OK
Sat Jan 27 19:13:16 2007 : Debug: rlm_sql_postgresql: affected rows =
Sat Jan 27 19:13:16 2007 : Debug: radius_xlat:  'UPDATE radippool
SET nasipaddress = '', pool_key = 0, callingstationid = '',
expiry_time = 'now'::timestamp(0) - '1 second'::interval   WHERE
nasipaddress = '100.125.1.1' AND pool_key = '2952792152' AND username
= 'manoel@foo.com'   AND callingstationid = '#BRAS-01#E110#2136''
Sat Jan 27 19:13:16 2007 : Debug: rlm_sql_postgresql: Status: PGRES_COMMAND_OK
Sat Jan 27 19:13:16 2007 : Debug: rlm_sql_postgresql: Status: PGRES_COMMAND_OK
Sat Jan 27 19:13:16 2007 : Debug: rlm_sql_postgresql: Status: PGRES_COMMAND_OK
00000000 65737520 6F726772 552E7075 6E726573 20656D61 6D27203D 656C6C75 72616D72
Sat Jan 27 19:13:16 2007 : Debug: rlm_sql_postgresql: affected rows = 0
Sat Jan 27 19:13:16 2007 : Debug: rlm_sql_postgresql: affected rows =
Sat Jan 27 19:13:16 2007 : Debug: rlm_sql_postgresql: affected rows =
00000000 736E6974 72657340 746D6F63 632E6C65 622E6D6F 41202772 7520444E 67726573
Sat Jan 27 19:13:16 2007 : Debug: radius_xlat:  'UPDATE radippool
SET nasipaddress = '', pool_key = 0, callingstationid = '',
expiry_time = 'now'::timestamp(0) - '1 second'::interval   WHERE
nasipaddress = '100.125.1.1' AND pool_key = '2147485784' AND username
= 'manoel@foo.com'   AND callingstationid = '#BRAS-01#E80#2136''
Sat Jan 27 19:13:16 2007 : Debug: radius_xlat:  'UPDATE radippool
SET nasipaddress = '', pool_key = 0, callingstationid = '',
expiry_time = 'now'::timestamp(0) - '1 second'::interval   WHERE
nasipaddress = '100.125.1.1' AND pool_key = '2147485849' AND username
= 'cezar@foo.com'   AND callingstationid = '#BRAS-01#E80#2201''
Sat Jan 27 19:13:16 2007 : Debug: radius_xlat:  'COMMIT'
00000000 70756F72 6F72472E 614E7075 3D20656D 64617220 756F7267 70657270 472E796C
00000000 70756F72 656D614E 44524F20 42205245
Argument/Register addr=772e10.  Dump of memory from 0x000772DD0 to 0x000772F10
00000000                                     00000000 00000000 00000000 00000000
00000000 00748250 00000000 00000000 00000000 00772E10 00000000 00000000 00000000
00000000 00000000 00000000 00000051 00000000 007748D0 00000000 0077C450 00000000
00000000 0077D128 00000000 0077DDA8 00000000 009CB560 00000000 009E1E80 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000071 00000000
00000000 00656480 00000000 00000000 00000000 B0E306B8 0000003E 0076D7A0 00000000
00000000 0076D370 00000000 00000000 00000000 0076D480 00000000 00000008 00000040
00000000 00772ED0 00000000 00000000 00000000 0076D690 00000000 00000007 00000020
00000000 00000000 00000000 00000091 00000000 B0E30828 0000003E B0E30828 0000003E
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000
Argument/Register addr=4fa160c0.  Dump of memory from 0x04FA16080 to 0x04FA161C0
00000000 006528B0 00000000 984025B0 0000002A 0091DFA0 00000000 00772DB0 00000000
00000000 4FA161D0 00000000 961B5744 0000002A 00000000 00000000 4FA160D0 00000000
00000000 00000000 00000000 00A02350 00000000 6C6C756D 616D7265 6E697472 65734073
00000000 6D6F6372 2E6C6574 2E6D6F63 00007262 955728DF 0000002A 4FA167B0 00000000
00000000 4FA16776 00000000 00000002 73000000 00000320 00000000 00000000 00000000
00000000 00000000 00000000 4FA16360 00000000 FFFFFFFF 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000002 4FA16698 00000000 955729B2 0000002A
00000000 00000020 0000000E FFFFFFFF FFFFFFFF 00000000 00000000 4FA16840 00000000
00000000 955729B2 0000002A 00000000 00000000 00000000 00000000 00000000 00000001
Argument/Register addr=6536a4.  Dump of memory from 0x000653664 to 0x0006537A4
00000000          472E7075 70756F72 656D614E 72203D20 72676461 7270756F 796C7065
00000000 6F72472E 614E7075 4F20656D 52454452 20594220 67646172 70756F72 6C706572
00000000 64692E79 00000000 000002D1 00000000 41445055 72204554 63616461 53207463
00000000 41205445 53746363 54706F74 3D656D69 445F4F54 28455441 27532527 7979272C
00000000 6D2D7979 64642D6D 32686820 696D3A34 2773733A 41202C29 53746363 69737365
00000000 69546E6F 283D656D 74786528 74636172 63657328 20646E6F 6D6F7266 75632820
00000000 6E657272 69745F74 7473656D 20706D61 5428202D 41445F4F 27284554 2C275325
00000000 79797927 6D6D2D79 2064642D 34326868 3A696D3A 29277373 20292929 7865202B
00000000 63617274 696D2874 6574756E 6F726620 6328206D 65727275 745F746E 73656D69
00000000 706D6174 28202D20 445F4F54 28455441 27532527 7979272C 6D2D7979 64642D6D
00000000 32686820
Argument/Register addr=4fa162e2.  Dump of memory from 0x04FA162A2 to 0x04FA163E2
00000000 70756F72 6F72472E 614E7075 3D20656D 64617220 756F7267 70657270 472E796C
00000000 70756F72 656D614E 44524F20 42205245 61722059 6F726764 65727075 2E796C70
00000000 00006469 00000000 009F7BBC 00000000 4FA16440 00000000 955728D4 0000002A
00000000 00000006 00000000 984025B0 0000002A 009F7BBC 00000000 B0C5CFB9 0000003E
00000000 FBAD8001 00000000 009F7BBC 00000000 009F7BBC 00000000 009F7BBC 00000000
00000000 009F7BBC 00000000 009F7BC7 00000000 FFFFFFFF FFFFFFFF 009F7BBC 00000000
00000000 FFFFFFFF FFFFFFFF 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 4FA164F0 00000000 009CCBB0 00000000
00000000 4FA10000 00000000 00000000 00000000 000003F2 00000000 B0C48276 0000003E
00000000 00000018 00000030 00000000 00000000 4FA163E0 00000000 00521080 00000000
00000000 FFFFFFFF

----- End of Call Stack Trace -----

Segmentation fault

SEVERE! radiusd 2.0 and 1.1.4 dying! Segmentation fault

Guilherme Franco