Segmentation fault for SNMP query
Hi all, when I've compiled in snmp support (--with-snmp) on current cvs head I got following segmentation fault(does not matter whether NAS are stored in DB or in clients.conf):
DEBUG OUTPUT START ... Ready to process requests. Nothing to do. Sleeping until we see a request. SMUX read start SMUX read len: 12 SMUX message received type: 67 rest len: 4 SMUX_RRSP SMUX_RRSP value: 0 errstat: 0 --- Walking the entire request list --- Nothing to do. Sleeping until we see a request.
SMUX read start SMUX read len: 40 SMUX message received type: 161 rest len: 36 SMUX_GETNEXT SMUX GET message parse: len 36 SMUX GET reqid: 182 len: 32 SMUX GET errstat 0 len: 29 SMUX GET errindex 0 len: 26 SMUX var parse: len 26 SMUX var parse: type 48 len 22 SMUX var parse: type must be 48 Request OID: 1.3.6.1.2.1.67.1.1.1.1.15.1.2.1 SMUX val_type: 5 ASN_NULL SMUX function call index is 2 SMUX function call index is 3 Segmentation fault
DEBUG OUTPUT END
when debugging(I can attach the gdb output if needed) I've found that it fails somewhere on reading internal nas clients data structure. As I'm not interested in nas clients obtained via snmp(I keep them in mysql DB) following is the little patch which basically just ads "return NULL" when snmp asks for next nas client: Index: src/main/radius_snmp.c =================================================================== RCS file: /source/radiusd/src/main/radius_snmp.c,v retrieving revision 1.31 diff -u -r1.31 radius_snmp.c --- src/main/radius_snmp.c 10 Apr 2007 14:21:24 -0000 1.31 +++ src/main/radius_snmp.c 11 Apr 2007 14:41:08 -0000 @@ -194,6 +194,8 @@ RADCLIENT *c; int i, len; + return NULL; + len = *objid_len - v->namelen; if (!mainconfig.clients) Hope this helps someone. Milan Holub holub (at) thenet (dot) ch -------------------------------------- TheNet-Internet Services AG, im Bernertechnopark, Morgenstr. 129 CH-3018, Bern, Switzerland 031 998 4333, Fax 031 998 4330 http://www.thenet.ch http://wlan.thenet.ch --------------------------------------
Milan Holub wrote:
Hi all,
when I've compiled in snmp support (--with-snmp) on current cvs head I got following segmentation fault(does not matter whether NAS are stored in DB or in clients.conf):
I just committed fixes for SNMP. I haven't tested it, but the code that was obviously wrong isn't there any more. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Hi Alan, On Wed, Apr 11, 2007 at 05:51:16PM +0200, Alan DeKok wrote:
Milan Holub wrote:
Hi all,
when I've compiled in snmp support (--with-snmp) on current cvs head I got following segmentation fault(does not matter whether NAS are stored in DB or in clients.conf):
I just committed fixes for SNMP. I haven't tested it, but the code that was obviously wrong isn't there any more.
==> I've tested your recent commits. Here are the results: - when query the radiusAcc and radiusAuth everything works fine(no segmentation faults); multiple queries give correct result - when trying to force reload using snmp: `snmpset -m /devel/freeradius/cvs/radiusd/mibs/RADIUS-AUTH-SERVER-MIB.txt -c verysecret localhost radiusAuthServConfigReset.0 i 2` then 1st reload is OK but after then when trying to either run the snmp-read query or the snmp-write query radius seems to ignore it. * there is no debug activity when running with -X flag and the result of the snmp-read query is empty and result of snmp-write query is following: `snmpset -m /devel/freeradius/cvs/radiusd/mibs/RADIUS-AUTH-SERVER-MIB.txt -c verysecret localhost radiusAuthServConfigReset.0 i 2` Error in packet. Reason: (noSuchName) There is no such variable name in this MIB. Failed object: radiusMIB.radiusAuthentication.radiusAuthServMIB.radiusAuthServMIBObjects.radiusAuthServ.radiusAuthServConfigReset.0 Radius itself seems to react on radius packets; only snmp is ignored after the snmp-write query. Completely same behaviour is observed when doing reload via HUP signal(using my "memory leakage" patch for reload). Please advise.
Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Milan Holub holub (at) thenet (dot) ch -------------------------------------- TheNet-Internet Services AG, im Bernertechnopark, Morgenstr. 129 CH-3018, Bern, Switzerland 031 998 4333, Fax 031 998 4330 http://www.thenet.ch http://wlan.thenet.ch --------------------------------------
Milan Holub wrote:
- when query the radiusAcc and radiusAuth everything works fine(no segmentation faults); multiple queries give correct result
Thanks.
- when trying to force reload using snmp: `snmpset -m /devel/freeradius/cvs/radiusd/mibs/RADIUS-AUTH-SERVER-MIB.txt -c verysecret localhost radiusAuthServConfigReset.0 i 2` then 1st reload is OK but after then when trying to either run the snmp-read query or the snmp-write query radius seems to ignore it
SNMP stops working after a HUP or reload. It's a known issue. It should be fixed before 2.0. As always, patches are welcome.
Reason: (noSuchName) There is no such variable name in this MIB.
Yes. It loses the connection to snmpd, and snmpd therefore says that he RADIUS MIBs are unknown. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
On Thursday 12 April 2007 04:40:47 Milan Holub wrote:
- when trying to force reload using snmp: `snmpset -m /devel/freeradius/cvs/radiusd/mibs/RADIUS-AUTH-SERVER-MIB.txt -c verysecret localhost radiusAuthServConfigReset.0 i 2` then 1st reload is OK but after then when trying to either run the snmp-read query or the snmp-write query radius seems to ignore it. * there is no debug activity when running with -X flag and the result of the snmp-read query is empty and result of snmp-write query is following: `snmpset -m /devel/freeradius/cvs/radiusd/mibs/RADIUS-AUTH-SERVER-MIB.txt -c verysecret localhost radiusAuthServConfigReset.0 i 2` Error in packet. Reason: (noSuchName) There is no such variable name in this MIB. Failed object: radiusMIB.radiusAuthentication.radiusAuthServMIB.radiusAuthServMIBObjects.r adiusAuthServ.radiusAuthServConfigReset.0
Radius itself seems to react on radius packets; only snmp is ignored after the snmp-write query. Completely same behaviour is observed when doing reload via HUP signal(using my "memory leakage" patch for reload).
Please advise.
Try http://bugs.freeradius.org/show_bug.cgi?id=150 I doubt that patch will still apply cleanly due to the many recent changes. I'll see if I can test the CVS head later today and submit a newer patch. Kevin Bonner
On Thursday 12 April 2007 10:32:18 Kevin Bonner wrote:
On Thursday 12 April 2007 04:40:47 Milan Holub wrote:
Radius itself seems to react on radius packets; only snmp is ignored after the snmp-write query. Completely same behaviour is observed when doing reload via HUP signal(using my "memory leakage" patch for reload).
Please advise.
Try http://bugs.freeradius.org/show_bug.cgi?id=150
I doubt that patch will still apply cleanly due to the many recent changes. I'll see if I can test the CVS head later today and submit a newer patch.
It surprises me that it still applies cleanly (just offset) with the current CVS head. Feel free to test the patch and report results in the bug or on the list. It would be nice to see the bug squashed, but it's become a default patch for my local freeradius build so I haven't been bothered with the issue in a long time. Kevin Bonner
Kevin Bonner wrote:
It surprises me that it still applies cleanly (just offset) with the current CVS head.
The SMUX code hasn't changed much. It should probably be replaced with AgentX code, but that can be done later...
Feel free to test the patch and report results in the bug or on the list. It would be nice to see the bug squashed, but it's become a default patch for my local freeradius build so I haven't been bothered with the issue in a long time.
The SMUX code should also try more than 3 times to connect to the SNMP server, and shouldn't try whenever it receives a packet. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Hi Kevin, On Thu, Apr 12, 2007 at 06:19:14PM -0400, Kevin Bonner wrote:
Try http://bugs.freeradius.org/show_bug.cgi?id=150
I doubt that patch will still apply cleanly due to the many recent changes. I'll see if I can test the CVS head later today and submit a newer patch.
It surprises me that it still applies cleanly (just offset) with the current CVS head. Feel free to test the patch and report results in the bug or on the list. It would be nice to see the bug squashed, but it's become a default patch for my local freeradius build so I haven't been bothered with the issue in a long time.
Kevin Bonner
==> I've applied the patch on cvs head. Here are the test results: 1) situation after start of freeradius(syslog output attached as well) freeradius -X ... Listening on SNMP SMUX with OID .1.3.6.1.4.1.3317.1.3.1 Ready to process requests. Nothing to do. Sleeping until we see a request. SMUX read start SMUX read len: 12 SMUX message received type: 67 rest len: 4 SMUX_RRSP SMUX_RRSP value: 0 errstat: 0 Nothing to do. Sleeping until we see a request. tail -f /var/log/syslog ucd-snmp[5234]: [smux_accept] accepted fd 3 from 127.0.0.1:39371 ucd-snmp[5234]: accepted smux peer: oid enterprises.3317.1.3.1, password verysecret, descr radiusd Then freeradius accepts and processes the snmp queries correctly until the reload(either via HUP or snmp-write). ==> you can see the snmp client connections in syslog: ucd-snmp[5234]: Connection from 127.0.0.1 for each request. Unfortunately when the reload is performed the situation is following: * syslog shows re-initilization of snmp upon reload: tail -f /var/log/syslog ucd-snmp[5234]: peer disconnected: enterprises.3317.1.3.1 ucd-snmp[5234]: [smux_accept] accepted fd 3from 127.0.0.1:39374 ucd-snmp[5234]: accepted smux peer: oid enterprises.3317.1.3.1, password verysecret, descr radiusd * but when trying to snmpwalk freeradius does not response to query at all and the snmp client times out: `snmpwalk -Cc -v 1 -m /devel/freeradius/cvs/radiusd/mibs/RADIUS-AUTH-SERVER-MIB.txt -c verysecret localhost radiusAuthServUpTime.0` Timeout: No Response from localhost and the attempt is seen in syslog: ucd-snmp[5234]: Connection from 127.0.0.1 To sum up: the patch applies to 2.0 but it does not work as expected:-( If you have some ideas, then please advise.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Milan Holub holub (at) thenet (dot) ch -------------------------------------- TheNet-Internet Services AG, im Bernertechnopark, Morgenstr. 129 CH-3018, Bern, Switzerland 031 998 4333, Fax 031 998 4330 http://www.thenet.ch http://wlan.thenet.ch --------------------------------------
Kevin Bonner wrote:
Try http://bugs.freeradius.org/show_bug.cgi?id=150
I doubt that patch will still apply cleanly due to the many recent changes. I'll see if I can test the CVS head later today and submit a newer patch.
Please try the latest CVS. I've added a patch based on yours. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Hi Alan, On Mon, Apr 16, 2007 at 01:52:43PM +0200, Alan DeKok wrote:
Kevin Bonner wrote:
Try http://bugs.freeradius.org/show_bug.cgi?id=150
I doubt that patch will still apply cleanly due to the many recent changes. I'll see if I can test the CVS head later today and submit a newer patch.
Please try the latest CVS. I've added a patch based on yours.
==> I've tested latest cvs head: - snmp works until 1st reload(HUP or snmp-write) - then it behaves the same as with Kevin's old patch (described in this thread) == snmp not working after reload - debug flags survive reload (good!) - with my config each reload eats additional 620k of memory per thread! Milan Holub holub (at) thenet (dot) ch -------------------------------------- TheNet-Internet Services AG, im Bernertechnopark, Morgenstr. 129 CH-3018, Bern, Switzerland 031 998 4333, Fax 031 998 4330 http://www.thenet.ch http://wlan.thenet.ch --------------------------------------
Milan Holub wrote:
- snmp works until 1st reload(HUP or snmp-write) - then it behaves the same as with Kevin's old patch (described in this thread) == snmp not working after reload
Hmm... OK.
- debug flags survive reload (good!) - with my config each reload eats additional 620k of memory per thread!
That memory will be cleaned up after a few more HUPs. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Hi Alan, On Mon, Apr 16, 2007 at 03:18:24PM +0200, Alan DeKok wrote:
That memory will be cleaned up after a few more HUPs. ==> Are you sure about that?
ps axu | grep rad: freerad 16235 2.2 1.9 9448 4916 pts/0 S 15:31 0:00 freeradius -X ==> initially we have 9448kb of memory used by freeradius process After 10 HUPs: for i in `seq 10`; do echo HUP $i; kill -HUP 16235; sleep 1; done we end up with 15636kb ps axu | grep rad: freerad 16235 0.8 4.2 15636 10920 pts/0 S 15:31 0:01 freeradius -X 15636-9448=6188 Thus after 10 HUPs the process takes 6188kb more space in memory than at the beginning(and it's incrementing lineary each HUP). Please advise. Milan Holub holub (at) thenet (dot) ch -------------------------------------- TheNet-Internet Services AG, im Bernertechnopark, Morgenstr. 129 CH-3018, Bern, Switzerland 031 998 4333, Fax 031 998 4330 http://www.thenet.ch http://wlan.thenet.ch --------------------------------------
Milan Holub wrote:
Hi Alan,
On Mon, Apr 16, 2007 at 03:18:24PM +0200, Alan DeKok wrote:
That memory will be cleaned up after a few more HUPs. ==> Are you sure about that?
Yes.
After 10 HUPs: for i in `seq 10`; do echo HUP $i; kill -HUP 16235; sleep 1; done
Try 32 HUPs. The memory will increase, but won't grow after that. At some point in the future, it can be fixed to do more cleanups after HUP. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Hi Alan, On Mon, Apr 16, 2007 at 04:39:16PM +0200, Alan DeKok wrote:
Try 32 HUPs. The memory will increase, but won't grow after that.
At some point in the future, it can be fixed to do more cleanups after HUP.
==> well, I've done the tests with 32 reHUPs and I'm getting segmentation fault during the promised cleanup: ...when 32nd HUP received:
BEGIN DEBUG Program received signal SIGSEGV, Segmentation fault. 0x4029ca3b in free () from /lib/libc.so.6 (gdb) bt #0 0x4029ca3b in free () from /lib/libc.so.6 #1 0x0804f05e in cf_pair_free () #2 0x0804f0c5 in cf_section_free () #3 0x0804f1f7 in cf_section_free () #4 0x08055b34 in read_mainconfig () #5 0x08058f5e in main () #6 0x4024714f in __libc_start_main () from /lib/libc.so.6 END DEBUG
Milan Holub holub (at) thenet (dot) ch -------------------------------------- TheNet-Internet Services AG, im Bernertechnopark, Morgenstr. 129 CH-3018, Bern, Switzerland 031 998 4333, Fax 031 998 4330 http://www.thenet.ch http://wlan.thenet.ch --------------------------------------
Milan Holub wrote:
==> well, I've done the tests with 32 reHUPs and I'm getting segmentation fault during the promised cleanup: ...when 32nd HUP received:
Ok... after some work with valgrind, the problem should be fixed. The server shouldn't use more memory after a HUP, and it shouldn't crash, either. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Milan Holub wrote:
==> I've tested latest cvs head: - snmp works until 1st reload(HUP or snmp-write) - then it behaves the same as with Kevin's old patch (described in this thread) == snmp not working after reload
Ok, try now. After some fighting with getting SNMPD to work, I can now see the counters incrementing when I query it via snmpwalk. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Hi Alan, On Mon, Apr 16, 2007 at 04:49:37PM +0200, Alan DeKok wrote:
Ok, try now. After some fighting with getting SNMPD to work, I can now see the counters incrementing when I query it via snmpwalk.
==> I can confirm with latest cvs head snmp is working after reload(either via HUP or snmp-write) ==> that's great! Thanks a lot, Alan. Milan Holub holub (at) thenet (dot) ch -------------------------------------- TheNet-Internet Services AG, im Bernertechnopark, Morgenstr. 129 CH-3018, Bern, Switzerland 031 998 4333, Fax 031 998 4330 http://www.thenet.ch http://wlan.thenet.ch --------------------------------------
On Monday 16 April 2007 07:52:43 Alan DeKok wrote:
Kevin Bonner wrote:
Try http://bugs.freeradius.org/show_bug.cgi?id=150
I doubt that patch will still apply cleanly due to the many recent changes. I'll see if I can test the CVS head later today and submit a newer patch.
Please try the latest CVS. I've added a patch based on yours.
Alan DeKok.
Tested with the CVS head as of this morning and everything looks good to me, even the per-client data. I'm hitting a segfault when testing the cases I listed in bug#150, but I don't think it is related to the SNMP portion of the code. Segfault info is below. Kevin Bonner == cut == (gdb) bt #0 0x00fe97a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x002fca0d in ___newselect_nocancel () from /lib/tls/libc.so.6 #2 0x004ecbb6 in main (argc=2, argv=0xbfe06fc4) at radiusd.c:575 (gdb) up #1 0x002fca0d in ___newselect_nocancel () from /lib/tls/libc.so.6 (gdb) up #2 0x004ecbb6 in main (argc=2, argv=0xbfe06fc4) at radiusd.c:575 575 status = select(max_fd + 1, &readfds, NULL, NULL, ptv); (gdb) list 570 #else 571 DEBUG2("Waking up in %d seconds...", 572 (int) tv.tv_sec); 573 #endif 574 } 575 status = select(max_fd + 1, &readfds, NULL, NULL, ptv); 576 if (status == -1) { 577 /* 578 * On interrupts, we clean up the request 579 * list. We then continue with the loop, (gdb) print ptv $1 = (struct timeval *) 0x0 (gdb) print &readfds $2 = (fd_set *) 0xbfe05ea0 (gdb) print max_fd $3 = 10 == cut ==
Kevin Bonner wrote: ...
Tested with the CVS head as of this morning and everything looks good to me, even the per-client data. I'm hitting a segfault when testing the cases I listed in bug#150, but I don't think it is related to the SNMP portion of the code. Segfault info is below.
Wierd. It's a little odd for select() to core dump. All I can say is re-build cleanly, and try again... Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
participants (3)
-
Alan DeKok -
Kevin Bonner -
Milan Holub