rlm_sql: readclients segmentation fault
Hi Alan, with the latest cvs head I've experienced following serious bug: radiusd.conf: ... listen { ipaddr = * port = 0 type = auth } listen { ipaddr = * port = 0 type = acct } ... clients.conf: client 127.0.0.1 { secret = testing123 shortname = localhost } sql.conf: ... readclients = yes ... When starting rlm_sql says it's adding nases from DB: freeradius -X:
BEGIN DEBUG ... rlm_sql (sql): Read entry nasname=a.b.c.d,shortname=wlan-gw33,secret=secret rlm_sql (sql): Adding client a.b.c.d (wlan-gw33) to clients list rlm_sql (sql): Released sql socket id: 4 ... END DEBUG
When testing freeradius with radclient from localhost it works OK. However when sending some request from eg. nas a.b.c.d(wlan-gw33) we ends up with segmentation fault:
BEGIN DEBUG Nothing to do. Sleeping until we see a request.
rad_recv: Access-Request packet from host a.b.c.d port 42926, id=82, length=46 (no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)... Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1024 (LWP 16225)] 0x08052987 in client_listener_find () (gdb) (gdb) (gdb) bt #0 0x08052987 in client_listener_find () #1 0x080590df in main () #2 0x4024714f in __libc_start_main () from /lib/libc.so.6
END DEBUG
I can confirm that at least cvs build from 6.4. 2007 did not have such a bug. Thus obviously there must be something rotten in latest cvs commits... Please advise. Milan Holub holub (at) thenet (dot) ch -------------------------------------- TheNet-Internet Services AG, im Bernertechnopark, Morgenstr. 129 CH-3018, Bern, Switzerland 031 998 4333, Fax 031 998 4330 http://www.thenet.ch http://wlan.thenet.ch --------------------------------------
Milan Holub wrote:
Hi Alan,
with the latest cvs head I've experienced following serious bug:
radiusd.conf: ... listen { ipaddr = * port = 0 type = auth } listen { ipaddr = * port = 0 type = acct } ...
clients.conf: client 127.0.0.1 { secret = testing123 shortname = localhost }
sql.conf: ... readclients = yes ...
When starting rlm_sql says it's adding nases from DB: freeradius -X:
BEGIN DEBUG ... rlm_sql (sql): Read entry nasname=a.b.c.d,shortname=wlan-gw33,secret=secret rlm_sql (sql): Adding client a.b.c.d (wlan-gw33) to clients list rlm_sql (sql): Released sql socket id: 4 ... END DEBUG
When testing freeradius with radclient from localhost it works OK. However when sending some request from eg. nas a.b.c.d(wlan-gw33) we ends up with segmentation fault:
BEGIN DEBUG Nothing to do. Sleeping until we see a request.
rad_recv: Access-Request packet from host a.b.c.d port 42926, id=82, length=46 (no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)... Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1024 (LWP 16225)] 0x08052987 in client_listener_find () (gdb) (gdb) (gdb) bt #0 0x08052987 in client_listener_find () #1 0x080590df in main () #2 0x4024714f in __libc_start_main () from /lib/libc.so.6
END DEBUG
I can confirm that at least cvs build from 6.4. 2007 did not have such a bug. Thus obviously there must be something rotten in latest cvs commits...
Please advise.
Milan Holub holub (at) thenet (dot) ch
-------------------------------------- TheNet-Internet Services AG, im Bernertechnopark, Morgenstr. 129 CH-3018, Bern, Switzerland 031 998 4333, Fax 031 998 4330 http://www.thenet.ch http://wlan.thenet.ch -------------------------------------- - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Thats weird, i'm using cvs head from this morning and all is fine. I'll recompile tomorrow when I have time. I'm loading SQL clients in a seperate instance of the SQL module though. Not that, that should make any difference. Oh and i'm binding to a single ip instead of wildcarding. -- Arran Cudbard-Bell (ac221@sussex.ac.uk) Authentication Authorisation & Accounting Officer Infrastructure Services | ENG1 FF08 EXT:3900
Hi Arran, On Mon, Apr 16, 2007 at 02:50:19PM +0100, Arran Cudbard-Bell wrote:
Thats weird, i'm using cvs head from this morning and all is fine.
==> I'm using latest cvs head(at the time of writing). I have my own few minor patches against cvs head but these really should not have any impact...(and did not have before)
I'll recompile tomorrow when I have time.
==> please do
I'm loading SQL clients in a seperate instance of the SQL module though. Not that, that should make any difference.
==> I think so...
Oh and i'm binding to a single ip instead of wildcarding.
==> I've tried but without any impact:-( Milan Holub holub (at) thenet (dot) ch -------------------------------------- TheNet-Internet Services AG, im Bernertechnopark, Morgenstr. 129 CH-3018, Bern, Switzerland 031 998 4333, Fax 031 998 4330 http://www.thenet.ch http://wlan.thenet.ch --------------------------------------
Milan Holub wrote:
Hi Alan,
with the latest cvs head I've experienced following serious bug: ...
You're using SNMP. You ran into an assertion. Try "cvs update". Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Hi all, On Mon, Apr 16, 2007 at 04:40:18PM +0200, Alan DeKok wrote:
You're using SNMP. You ran into an assertion. Try "cvs update". ==> I did cvs update this morning. I don't think it's an assertion(no clean exit on assertion but segmentation fault). I synchronized the radclient binary + dictionaries on remote NAS with the freeradius built. Unfortunately problem persists.
Oh and I forgot: when I move definition of localhost from clients.conf to nas table I'm getting segmentation fault also when sending test requests from localhost as well. Anyone else experience similar behaviour? Milan Holub holub (at) thenet (dot) ch -------------------------------------- TheNet-Internet Services AG, im Bernertechnopark, Morgenstr. 129 CH-3018, Bern, Switzerland 031 998 4333, Fax 031 998 4330 http://www.thenet.ch http://wlan.thenet.ch --------------------------------------
Milan Holub wrote:
Oh and I forgot: when I move definition of localhost from clients.conf to nas table I'm getting segmentation fault also when sending test requests from localhost as well.
*Please* run the server under valgrind to find the source of these problems. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Hi Alan, On Tue, Apr 17, 2007 at 11:45:28AM +0200, Alan DeKok wrote:
*Please* run the server under valgrind to find the source of these problems.
==> finally I managed to compile valgrind and can give you thus its output... I did fresh cvs checkout and then created a debian package on woody(export LDFLAGS='-lz'; dpkg-buildpackage -b -uc -d). Point 2) I've also compiled&tested on debian testing with the same result. After cvs commits from this morning I'm getting segmentation faults in following cases: 1) snmpwalk (read-query) - when reading the NAS entries `/usr/local/bin/valgrind --tool=memcheck --leak-check=full freeradius -X &> /devel/freeradius/debug/fr_snmp_walk_1.txt` when running `snmpwalk -Cc -v 1 -m /devel/freeradius/cvs/radiusd/mibs/RADIUS-AUTH-SERVER-MIB.txt -c verysecret localhost radiusAuth` I get "Segmentation fault":-( ==> full -X debug output + valgrind: http://pastebin.ca/444684 2) when receiving HUP signal ==> full -X debug output + valgrind: http://pastebin.ca/444717 3) snmpset (write-query) - similar to 2) ==> similar output as in 2) 4) on any incoming radius request (when the corresponding NAS is stored in mysql nas table) ==> full -X debug output + valgrind: http://pastebin.ca/444719 Am I doing something wrong? Nobody else experience similar behaviour? Please advise. PS: I like this http://pastebin.ca - it keeps the mailing lists clean... Milan Holub holub (at) thenet (dot) ch -------------------------------------- TheNet-Internet Services AG, im Bernertechnopark, Morgenstr. 129 CH-3018, Bern, Switzerland 031 998 4333, Fax 031 998 4330 http://www.thenet.ch http://wlan.thenet.ch --------------------------------------
Milan Holub wrote:
1) snmpwalk (read-query) - when reading the NAS entries ... I get "Segmentation fault":-(
==> full -X debug output + valgrind: http://pastebin.ca/444684
It looks like a NULL de-reference. i.e. a NULL isn't checked before it's de-referenced to look into a structure. It would help to have more symbols,
2) when receiving HUP signal ==> full -X debug output + valgrind: http://pastebin.ca/444717
Similar comments about debugging symbols appear here. If you're doing CVS updates regularly from CVS head, you MUST delete all of the previous rlm_* libaries before installing! The internal server structures are changing hourly, and without a re-compile, the modules will be looking in the wrong place.
PS: I like this http://pastebin.ca - it keeps the mailing lists clean...
It does appear extremely useful. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Hi Alan, On Tue, Apr 17, 2007 at 03:54:25PM +0200, Milan Holub wrote:
Hi Alan, On Tue, Apr 17, 2007 at 11:45:28AM +0200, Alan DeKok wrote:
*Please* run the server under valgrind to find the source of these problems.
==> finally I managed to compile valgrind and can give you thus its output...
I did fresh cvs checkout and then created a debian package on woody(export LDFLAGS='-lz'; dpkg-buildpackage -b -uc -d). Point 2) I've also compiled&tested on debian testing with the same result.
After cvs commits from this morning I'm getting segmentation faults in following cases:
1) snmpwalk (read-query) - when reading the NAS entries `/usr/local/bin/valgrind --tool=memcheck --leak-check=full freeradius -X &> /devel/freeradius/debug/fr_snmp_walk_1.txt`
when running `snmpwalk -Cc -v 1 -m /devel/freeradius/cvs/radiusd/mibs/RADIUS-AUTH-SERVER-MIB.txt -c verysecret localhost radiusAuth`
I get "Segmentation fault":-(
==> full -X debug output + valgrind: http://pastebin.ca/444684
2) when receiving HUP signal ==> full -X debug output + valgrind: http://pastebin.ca/444717
3) snmpset (write-query) - similar to 2) ==> similar output as in 2)
4) on any incoming radius request (when the corresponding NAS is stored in mysql nas table) ==> full -X debug output + valgrind: http://pastebin.ca/444719
Am I doing something wrong? Nobody else experience similar behaviour?
Please advise.
PS: I like this http://pastebin.ca - it keeps the mailing lists clean...
All segmentation faults were related to the same snmp issue - accessing num_tree structure. Here is a debug output for case 4) rad_recv: Access-Request packet from host <NAS_IN_NAS_TABLE> port 43052, id=161, length=46 Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1024 (LWP 26896)] 0x080529ab in auth_socket_recv (listener=0x815aac0, pfun=0xbfffea1c, prequest=0xbfffea20) at listen.c:389 389 break; (gdb) (gdb) (gdb) bt #0 0x080529ab in auth_socket_recv (listener=0x815aac0, pfun=0xbfffea1c, prequest=0xbfffea20) at listen.c:389 #1 0x08059073 in main (argc=2, argv=0xbffffbb4) at radiusd.c:643 (gdb) print client $1 = (RADCLIENT *) 0x81ff5e8 (gdb) list 384 */ 385 switch(packet->code) { 386 case PW_AUTHENTICATION_REQUEST: 387 RAD_SNMP_CLIENT_INC(listener, client, requests); 388 fun = rad_authenticate; 389 break; 390 391 case PW_STATUS_SERVER: 392 if (!mainconfig.status_server) { 393 RAD_SNMP_TYPE_INC(listener, total_packets_dropped); (gdb) print client->auth $2 = (rad_snmp_client_entry_t *) 0x0 ==> as you can see the rad_snmp_client_entry type should contain some data but it does not. As I do not care much about detail snmp info about every NAS thus I've done following dummy changes in order to fix the segmentation faults: 1)+2)+3) do not insert clients into snmp structure... Index: ./src/main/client.c =================================================================== RCS file: /source/radiusd/src/main/client.c,v retrieving revision 1.56 diff -u -r1.56 client.c --- ./src/main/client.c 17 Apr 2007 09:22:36 -0000 1.56 +++ ./src/main/client.c 18 Apr 2007 14:11:38 -0000 @@ -207,7 +207,7 @@ return 0; } -#ifdef WITH_SNMP +#ifdef WITH_SNMPX if (!tree_num) { tree_num = rbtree_create(client_num_cmp, NULL, 0); } Index: ./src/main/event.c =================================================================== RCS file: /source/radiusd/src/main/event.c,v retrieving revision 1.20 diff -u -r1.20 event.c --- ./src/main/event.c 18 Apr 2007 10:03:30 -0000 1.20 +++ ./src/main/event.c 18 Apr 2007 14:14:32 -0000 @@ -111,24 +111,24 @@ case PW_AUTHENTICATION_ACK: rad_snmp.auth.total_responses++; rad_snmp.auth.total_access_accepts++; - if (client) client->auth->accepts++; + //if (client) client->auth->accepts++; break; case PW_AUTHENTICATION_REJECT: rad_snmp.auth.total_responses++; rad_snmp.auth.total_access_rejects++; - if (client) client->auth->rejects++; + //if (client) client->auth->rejects++; break; case PW_ACCESS_CHALLENGE: rad_snmp.auth.total_responses++; rad_snmp.auth.total_access_challenges++; - if (client) client->auth->challenges++; + //if (client) client->auth->challenges++; break; case PW_ACCOUNTING_RESPONSE: rad_snmp.acct.total_responses++; - if (client) client->auth->responses++; + //if (client) client->auth->responses++; break; /* @@ -138,7 +138,7 @@ case 0: if (request->packet->code == PW_AUTHENTICATION_REQUEST) { rad_snmp.auth.total_bad_authenticators++; - if (client) client->auth->bad_authenticators++; + //if (client) client->auth->bad_authenticators++; } break; 4) - do not work with client->auth since its empty now Index: ./src/include/radius_snmp.h =================================================================== RCS file: /source/radiusd/src/include/radius_snmp.h,v retrieving revision 1.16 diff -u -r1.16 radius_snmp.h --- ./src/include/radius_snmp.h 10 Apr 2007 14:21:23 -0000 1.16 +++ ./src/include/radius_snmp.h 18 Apr 2007 14:08:43 -0000 @@ -81,7 +81,7 @@ } else { if (_listener->type == RAD_LISTEN_ACCT) \ rad_snmp.acct._x++; } } -#define RAD_SNMP_CLIENT_INC(_listener, _client, _x) if (mainconfig.do_snmp) { \ +#define RAD_SNMP_CLIENT_INC(_listener, _client, _x) if (0) { \ if (_listener->type == RAD_LISTEN_AUTH) { \ _client->auth->_x++; \ } else { if (_listener->type == RAD_LISTEN_ACCT) \ This works for me but I believe Alan will fix the issue cleanly ASAP:) Anyway thanks Alan for your hard job on freeradius. Milan Holub holub (at) thenet (dot) ch -------------------------------------- TheNet-Internet Services AG, im Bernertechnopark, Morgenstr. 129 CH-3018, Bern, Switzerland 031 998 4333, Fax 031 998 4330 http://www.thenet.ch http://wlan.thenet.ch --------------------------------------
Milan Holub wrote: ...
(gdb) print client->auth $2 = (rad_snmp_client_entry_t *) 0x0
Ah. client_add() doesn't create the necessary structure. I've just fixed that.
This works for me but I believe Alan will fix the issue cleanly ASAP:)
<g>
Anyway thanks Alan for your hard job on freeradius.
Just wait. I'm trying to get major capabilities into 2.0, or maybe 2.1. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Hi Alan, On Wed, Apr 18, 2007 at 05:09:11PM +0200, Alan DeKok wrote:
Ah. client_add() doesn't create the necessary structure. I've just fixed that.
==> I can confirm it works (cool!) However here is another bug report:): * cvs head * all NASes in nas table(clients.conf not used) * sending HUP results in segmentation fault when re-building up internal clients structure: rlm_sql (sql): Read entry nasname=<IP_IN_NAS_TABLE>,shortname=wlan-gw29,secret=DISABLED rlm_sql (sql): Adding client <IP_IN_NAS_TABLE> (wlan-gw29) to clients list Program received signal SIGSEGV, Segmentation fault. 0x40033e93 in rbtree_insertnode (tree=0x81f1c48, Data=0x8276748) at rbtree.c:248 248 result = tree->Compare(Data, Current->Data); (gdb) bt #0 0x40033e93 in rbtree_insertnode (tree=0x81f1c48, Data=0x8276748) at rbtree.c:248 #1 0x40033f87 in rbtree_insert (tree=0x81f1c48, Data=0x8276748) at rbtree.c:299 #2 0x0804e64b in client_add (clients=0x815bd28, client=0x8276748) at client.c:232 #3 0x4054ef98 in generate_sql_clients (inst=0x82694f0) at rlm_sql.c:338 #4 0x4054ff6d in rlm_sql_instantiate (conf=0x820a308, instance=0x82694e4) at rlm_sql.c:862 #5 0x080561e5 in find_module_instance (modules=0x8206a80, instname=0x820f728 "sql") at modules.c:307 #6 0x08057cf3 in do_compile_modsingle (parent=0x0, component=1, ci=0x820f708, filename=0x8079f48 "radiusd.conf", grouptype=0, modname=0xbfffe478) at modcall.c:1195 #7 0x08058308 in compile_modsingle (parent=0x0, component=1, ci=0x820f708, filename=0x8079f48 "radiusd.conf", modname=0xbfffe478) at modcall.c:1302 #8 0x0805671e in load_component_section (parent=0x0, cs=0x820f3c0, comp=1, filename=0x8079f48 "radiusd.conf") at modules.c:551 #9 0x08056bc0 in setup_modules (reload=1) at modules.c:927 #10 0x08055a9d in read_mainconfig (reload=1) at mainconfig.c:968 #11 0x08058ea2 in main (argc=2, argv=0xbffffbb4) at radiusd.c:540 #0 0x40033e93 in rbtree_insertnode (tree=0x81f1c48, Data=0x8276748) at rbtree.c:248 248 result = tree->Compare(Data, Current->Data); (gdb) list 243 int result; 244 245 /* 246 * See if two entries are identical. 247 */ 248 result = tree->Compare(Data, Current->Data); 249 if (result == 0) { 250 /* 251 * Don't replace the entry. 252 */ (gdb) print Data $1 = (void *) 0x8276748 (gdb) print Current->Data Cannot access memory at address 0x29 ==> if you need more debug outpu then let me know... Milan Holub holub (at) thenet (dot) ch -------------------------------------- TheNet-Internet Services AG, im Bernertechnopark, Morgenstr. 129 CH-3018, Bern, Switzerland 031 998 4333, Fax 031 998 4330 http://www.thenet.ch http://wlan.thenet.ch --------------------------------------
Milan Holub wrote:
However here is another bug report:): * cvs head * all NASes in nas table(clients.conf not used) * sending HUP results in segmentation fault when re-building up internal clients structure:
Ok... I've added more code to re-set pointers on cleanup, and create them on creation. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Hi Alan, On Thu, Apr 19, 2007 at 10:26:36AM +0200, Alan DeKok wrote:
Ok... I've added more code to re-set pointers on cleanup, and create them on creation.
==> and yes it helped! no segmentation fault anymore Milan Holub holub (at) thenet (dot) ch -------------------------------------- TheNet-Internet Services AG, im Bernertechnopark, Morgenstr. 129 CH-3018, Bern, Switzerland 031 998 4333, Fax 031 998 4330 http://www.thenet.ch http://wlan.thenet.ch --------------------------------------
participants (3)
-
Alan DeKok -
Arran Cudbard-Bell -
Milan Holub