In a cvs version of pre2, which is newer than pre2, on one machine, after some time of inactivity, freeradius does not seem to answer at all- sometimes, not reliable reproducable. Freeradius has a lot of database connections to mysql, maybe they are the culprit for that behaviour? stracing freeradius gives: # strace -f -p 29757 Process 29807 attached with 51 threads - interrupt to quit [pid 29757] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29758] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29759] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29760] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29761] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- ... [pid 29796] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29797] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29798] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29799] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29800] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29801] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29802] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29803] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- --- [pid 29778] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29779] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29780] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29781] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29782] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29783] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29784] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29785] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29786] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- ... [pid 29803] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29804] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29805] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29806] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29807] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29758] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29759] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29760] read(9, <unfinished ...> [pid 29761] read(8, <unfinished ...> [pid 29762] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29763] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29764] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29765] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29766] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [..... [pid 29798] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29799] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29800] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29801] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29802] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29803] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29804] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29805] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29806] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29807] futex(0x8002f654, FUTEX_WAIT, 0, NULL with a few dozens lines deleted. strace's output stops here, also when radius request come in. They are not answered. Any idea, what's going wrong here or how to track that down? I already tried nohup gdb -x gdbargs /sbin/radiusd & with gdbargs containing: run -AX bt but was unable to reproduce that situation in this context. Running radiusd under gdb's control seemed to solve that problem, maybe that's not the final solution. Norbert Wegener
It's mysql that goes to sleep. There is some default setting that kills the idle connections after 8 hours. Once all threads are gone ... Read their list to find out what setting it is and how to ?incease it to 24 hours. Ivan Kalik Kalik Informatika ISP Dana 5/11/2007, "Norbert Wegener" <norbert.wegener@siemens.com> piše:
In a cvs version of pre2, which is newer than pre2, on one machine, after some time of inactivity, freeradius does not seem to answer at all- sometimes, not reliable reproducable. Freeradius has a lot of database connections to mysql, maybe they are the culprit for that behaviour? stracing freeradius gives: # strace -f -p 29757 Process 29807 attached with 51 threads - interrupt to quit [pid 29757] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29758] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29759] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29760] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29761] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- ....
[pid 29796] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29797] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29798] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29799] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29800] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29801] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29802] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29803] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- --- [pid 29778] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29779] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29780] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29781] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29782] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29783] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29784] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29785] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29786] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- .... [pid 29803] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29804] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29805] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29806] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29807] --- SIGSTOP (Stopped (signal)) @ 0 (0) --- [pid 29758] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29759] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29760] read(9, <unfinished ...> [pid 29761] read(8, <unfinished ...> [pid 29762] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29763] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29764] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29765] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29766] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [..... [pid 29798] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29799] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29800] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29801] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29802] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29803] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29804] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29805] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29806] futex(0x8002f654, FUTEX_WAIT, 0, NULL <unfinished ...> [pid 29807] futex(0x8002f654, FUTEX_WAIT, 0, NULL
with a few dozens lines deleted. strace's output stops here, also when radius request come in. They are not answered. Any idea, what's going wrong here or how to track that down? I already tried nohup gdb -x gdbargs /sbin/radiusd & with gdbargs containing: run -AX bt
but was unable to reproduce that situation in this context. Running radiusd under gdb's control seemed to solve that problem, maybe that's not the final solution.
Norbert Wegener
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Norbert Wegener wrote:
In a cvs version of pre2, which is newer than pre2, on one machine, after some time of inactivity, freeradius does not seem to answer at all- sometimes, not reliable reproducable.
When did you take the CVS snapshot? Did -pre2 NOT have this behavior? ...
with a few dozens lines deleted.
It looks like the threads are blocked on reading a descriptor, OR on a mutex. Which mutex they're blocked on is anyone's guess.
strace's output stops here, also when radius request come in. They are not answered. Any idea, what's going wrong here or how to track that down? I already tried nohup gdb -x gdbargs /sbin/radiusd & with gdbargs containing: run -AX bt
but was unable to reproduce that situation in this context.
You can use gdb to attach to a running process, and then type "bt" to see where it's blocked.
Running radiusd under gdb's control seemed to solve that problem, maybe that's not the final solution.
Sounds like it's a signal/mutex issue... Alan DeKok.
Alan DeKok wrote:
Norbert Wegener wrote:
In a cvs version of pre2, which is newer than pre2, on one machine, after some time of inactivity, freeradius does not seem to answer at all- sometimes, not reliable reproducable.
When did you take the CVS snapshot? the cvs is from September, 12. Did -pre2 NOT have this behavior?
I did not notice this with pre2.
...
with a few dozens lines deleted.
It looks like the threads are blocked on reading a descriptor, OR on a mutex. Which mutex they're blocked on is anyone's guess.
..
You can use gdb to attach to a running process, and then type "bt" to see where it's blocked.
I'll try to get a bt. Norbert Wegener
Running radiusd under gdb's control seemed to solve that problem, maybe that's not the final solution.
Sounds like it's a signal/mutex issue...
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
got the opportunity earlier than expected... The system is suse 10.2. # gdb radiusd 30251 GNU gdb 6.5 Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i586-suse-linux"...Using host libthread_db library "/lib/libthread_db.so.1". Attaching to program: /sbin/radiusd, process 30251 Loaded symbols for /sbin/radiusd Reading symbols from /lib/libradius-2.0.0-pre2.so...done. Loaded symbols for //lib/libradius-2.0.0-pre2.so Reading symbols from /lib/libnsl.so.1...done. Loaded symbols for //lib/libnsl.so.1 Reading symbols from /lib/libresolv.so.2...done. Loaded symbols for //lib/libresolv.so.2 Reading symbols from /lib/libpthread.so.0...done. [Thread debugging using libthread_db enabled] [New Thread -1211984192 (LWP 30251)] Loaded symbols for //lib/libpthread.so.0 Reading symbols from /lib/libcrypt.so.1...done. Loaded symbols for //lib/libcrypt.so.1 Reading symbols from /usr/lib/libltdl.so.3...done. Loaded symbols for /usr/lib/libltdl.so.3 Reading symbols from /lib/libdl.so.2...done. Loaded symbols for //lib/libdl.so.2 Reading symbols from /usr/lib/libssl.so.0.9.8...done. Loaded symbols for /usr/lib/libssl.so.0.9.8 Reading symbols from /usr/lib/libcrypto.so.0.9.8...done. Loaded symbols for /usr/lib/libcrypto.so.0.9.8 Reading symbols from /lib/libc.so.6...done. Loaded symbols for //lib/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/libz.so.1...done. Loaded symbols for /lib/libz.so.1 Reading symbols from /lib/libnss_files.so.2...done. Loaded symbols for //lib/libnss_files.so.2 Reading symbols from /lib/rlm_exec-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_exec-2.0.0-pre2.so Reading symbols from /lib/rlm_expr-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_expr-2.0.0-pre2.so Reading symbols from /lib/rlm_expiration-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_expiration-2.0.0-pre2.so Reading symbols from /lib/rlm_logintime-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_logintime-2.0.0-pre2.so Reading symbols from /lib/rlm_sql-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_sql-2.0.0-pre2.so Reading symbols from /lib/rlm_sql_mysql-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_sql_mysql-2.0.0-pre2.so Reading symbols from /usr/lib/mysql/libmysqlclient_r.so.15...done. Loaded symbols for /usr/lib/mysql/libmysqlclient_r.so.15 Reading symbols from /lib/libm.so.6...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/rlm_pap-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_pap-2.0.0-pre2.so Reading symbols from /lib/rlm_chap-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_chap-2.0.0-pre2.so Reading symbols from /lib/rlm_mschap-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_mschap-2.0.0-pre2.so Reading symbols from /lib/rlm_unix-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_unix-2.0.0-pre2.so Reading symbols from /lib/rlm_eap-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_eap-2.0.0-pre2.so Reading symbols from /lib/libeap-2.0.0-pre2.so...done. Loaded symbols for //lib/libeap-2.0.0-pre2.so Reading symbols from /lib/rlm_eap_md5-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_eap_md5-2.0.0-pre2.so Reading symbols from /lib/rlm_eap_leap-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_eap_leap-2.0.0-pre2.so Reading symbols from /lib/rlm_eap_gtc-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_eap_gtc-2.0.0-pre2.so Reading symbols from /lib/rlm_eap_tls-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_eap_tls-2.0.0-pre2.so Reading symbols from /lib/rlm_eap_peap-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_eap_peap-2.0.0-pre2.so Reading symbols from /lib/rlm_eap_mschapv2-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_eap_mschapv2-2.0.0-pre2.so Reading symbols from /lib/rlm_preprocess-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_preprocess-2.0.0-pre2.so Reading symbols from /lib/rlm_realm-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_realm-2.0.0-pre2.so Reading symbols from /lib/rlm_always-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_always-2.0.0-pre2.so Reading symbols from /lib/rlm_attr_rewrite-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_attr_rewrite-2.0.0-pre2.so Reading symbols from /lib/rlm_acct_unique-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_acct_unique-2.0.0-pre2.so Reading symbols from /lib/rlm_files-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_files-2.0.0-pre2.so Reading symbols from /lib/rlm_detail-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_detail-2.0.0-pre2.so Reading symbols from /lib/rlm_radutmp-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_radutmp-2.0.0-pre2.so Reading symbols from /lib/rlm_attr_filter-2.0.0-pre2.so...done. Loaded symbols for //lib/rlm_attr_filter-2.0.0-pre2.so Failed to read a valid object file image from memory. 0xb7f87410 in ?? () (gdb) bt #0 0xb7f87410 in ?? () #1 0xbfae9008 in ?? () #2 0x00004000 in ?? () #3 0x801779d0 in ?? () #4 0xb7f32003 in __read_nocancel () from //lib/libpthread.so.0 #5 0xb7b083c8 in vio_read () from /usr/lib/mysql/libmysqlclient_r.so.15 #6 0xb7b0843e in vio_read_buff () from /usr/lib/mysql/libmysqlclient_r.so.15 #7 0xb7b08eac in net_realloc () from /usr/lib/mysql/libmysqlclient_r.so.15 #8 0xb7b0929b in my_net_read () from /usr/lib/mysql/libmysqlclient_r.so.15 #9 0xb7b03799 in cli_safe_read () from /usr/lib/mysql/libmysqlclient_r.so.15 #10 0xb7b04665 in unpack_fields () from /usr/lib/mysql/libmysqlclient_r.so.15 #11 0xb7b02e14 in mysql_real_query () from /usr/lib/mysql/libmysqlclient_r.so.15 #12 0xb7adaa85 in mysql_query () from /usr/lib/mysql/libmysqlclient_r.so.15 #13 0xb7c0b163 in sql_query (sqlsocket=0x80163e10, config=0x80168d28, querystr=0xbfae9254 "SELECT id, username, attribute, value, op", ' ' <repeats 11 times>, "FROM radcheck", ' ' <repeats 11 times>, "WHERE username = 'host/de7018tc.ww901.siemens.net'", ' ' <repeats 11 times>, "ORDER BY id") at sql_mysql.c:172 #14 0xb7c0b1f5 in sql_select_query (sqlsocket=0x80163e10, config=0x80168d28, querystr=0xbfae9254 "SELECT id, username, attribute, value, op", ' ' <repeats 11 times>, "FROM radcheck", ' ' <repeats 11 times>, "WHERE username = 'host/de7018tc.ww901.siemens.net'", ' ' <repeats 11 times>, "ORDER BY id") ---Type <return> to continue, or q <return> to quit--- at sql_mysql.c:259 #15 0xb7c12bf2 in rlm_sql_select_query (sqlsocket=0x80163e10, inst=0x80163508, query=0xbfae9254 "SELECT id, username, attribute, value, op", ' ' <repeats 11 times>, "FROM radcheck", ' ' <repeats 11 times>, "WHERE username = 'host/de7018tc.ww901.siemens.net'", ' ' <repeats 11 times>, "ORDER BY id") at sql.c:516 #16 0xb7c13122 in sql_getvpdata (inst=0x80163508, sqlsocket=0x80163e10, pair=0xbfaea458, query=0xbfae9254 "SELECT id, username, attribute, value, op", ' ' <repeats 11 times>, "FROM radcheck", ' ' <repeats 11 times>, "WHERE username = 'host/de7018tc.ww901.siemens.net'", ' ' <repeats 11 times>, "ORDER BY id") at sql.c:563 #17 0xb7c10ce3 in rlm_sql_authorize (instance=0x80163508, request=0x8034adb8) at rlm_sql.c:926 #18 0x80011bc0 in modcall (component=1, c=0x801668f0, request=0x8034adb8) at modcall.c:244 #19 0x8000edea in indexed_modcall (space=<value optimized out>, comp=1, idx=0, request=0x8034adb8) at modules.c:411 #20 0x8000635a in rad_authenticate (request=0x8034adb8) at auth.c:540 #21 0x8001a486 in radius_handle_request (request=0x8034adb8, fun=0x80006280 <rad_authenticate>) at event.c:2181 #22 0x80014876 in thread_pool_addrequest (request=0x8034adb8, fun=0x80006280 <rad_authenticate>) at threads.c:847 #23 0x800130f4 in main (argc=2, argv=0xbfaec6e4) at radiusd.c:699 Norbert Wegener
Norbert Wegener wrote:
got the opportunity earlier than expected... The system is suse 10.2.
# gdb radiusd 30251 ... (gdb) bt ... #5 0xb7b083c8 in vio_read () from /usr/lib/mysql/libmysqlclient_r.so.15
So it's blocked on a MySQL read... All I can suggest right now is to try reverting rlm_sql.c to revision 1.172, which is the only difference in that module between -pre2 and Sept. 12. Alan DeKok.
There seems to have changed a bit more.rlm_sql.c did not fit. Even substituting the whole rlm_sql directory with the one from pre 2 did not work: rlm_sql.c: In function 'sql_xlat': rlm_sql.c:200: warning: comparison between signed and unsigned rlm_sql.c: In function 'generate_sql_clients': rlm_sql.c:329: warning: pointer targets in assignment differ in signedness rlm_sql.c:337: error: 'struct main_config_t' has no member named 'clients' gmake[6]: *** [rlm_sql.lo] Fehler 1 gmake[6]: Leaving directory `/root/radiusd/src/modules/rlm_sql' gmake[5]: *** [common] Fehler 2 gmake[5]: Leaving directory `/root/radiusd/src/modules' gmake[4]: *** [all] Fehler 2 gmake[4]: Leaving directory `/root/radiusd/src/modules' gmake[3]: *** [common] Fehler 2 gmake[3]: Leaving directory `/root/radiusd/src' gmake[2]: *** [all] Fehler 2 gmake[2]: Leaving directory `/root/radiusd/src' gmake[1]: *** [common] Fehler 2 gmake[1]: Leaving directory `/root/radiusd' make: *** [all] Fehler 2 Any other suggestions? Norbert Wegener Alan DeKok schrieb:
Norbert Wegener wrote:
got the opportunity earlier than expected... The system is suse 10.2.
# gdb radiusd 30251
...
(gdb) bt
...
#5 0xb7b083c8 in vio_read () from /usr/lib/mysql/libmysqlclient_r.so.15
So it's blocked on a MySQL read...
All I can suggest right now is to try reverting rlm_sql.c to revision 1.172, which is the only difference in that module between -pre2 and Sept. 12.
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Norbert Wegener Siemens AG Siemens IT Solutions and Services SBS GO GIO NW PSU2 Kruppstr. 16 D-46128 Essen, Germany Phone : +49 (0) 201 816-3116 Fax. : +49 (0) 201 816-5581284 mailto:norbert.wegener@siemens.com Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Gerhard Cromme Vorstand: Peter Löscher, Vorsitzender; Johannes Feldmayer, Heinrich Hiesinger, Joe Kaeser, Rudi Lamprecht, Eduardo Montes, Jürgen Radomski, Erich R. Reinhardt, Hermann Requardt, Uriel J. Sharef, Klaus Wucherer Sitz der Gesellschaft: Berlin und München; Registergericht: Berlin Charlottenburg, HRB 12300, München, HRB 6684 WEEE-Reg.-Nr. DE 23691322
Norbert Wegener wrote:
There seems to have changed a bit more.rlm_sql.c did not fit. Even substituting the whole rlm_sql directory with the one from pre 2 did not work:
Arg. OK.
Any other suggestions?
Not right now. If you're willing to do a binary search in CVS to see *when* it started breaking... but that's a lot of work. Alan DeKok.
Alan DeKok wrote:
..
Any other suggestions?
Not right now. If you're willing to do a binary search in CVS to see *when* it started breaking... but that's a lot of work.
Without changing versions of mysql or freeradius I think I already found out, that num_sql_socks seems to have a significant influence on the behaviour. With a value of 5, freeradius now is already running for some hours. A value of 20 stops it from working normally within two hours. Maybe I can find out more. Norbert Wegener
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Hi,
With a value of 5, freeradius now is already running for some hours. A value of 20 stops it from working normally within two hours. Maybe I can find out more.
ah! and this is *changed* behaviour since the pre2 release. I'm here to report that 'beta' version taken via CVS yesterday lunchtime froze with no response within the hour. however, a simple revert to the released pre2 tar.gz version and all ticks along without a freeze. for some reason we too have 20 as our number of SQL connections..... worked for us with CVS code from 17th sept (if that helps to narrow it down??) alan
A.L.M.Buxey@lboro.ac.uk wrote:
I'm here to report that 'beta' version taken via CVS yesterday lunchtime froze with no response within the hour. however, a simple revert to the released pre2 tar.gz version and all ticks along without a freeze.
Dang...
for some reason we too have 20 as our number of SQL connections..... worked for us with CVS code from 17th sept (if that helps to narrow it down??)
That's still over a month ago. And a lot of code changed since then. Is it possible for you to set up a test system, and feed it fake traffic? If so, the issue can be tracked down to a specific day or commit. That would make fixing the problem much, much, easier. Alan DeKok.
Norbert Wegener wrote:
Without changing versions of mysql or freeradius I think I already found out, that num_sql_socks seems to have a significant influence on the behaviour. With a value of 5, freeradius now is already running for some hours. A value of 20 stops it from working normally within two hours.
Umm... weird. I have no idea why that would affect anything. Alan DeKok.
Now it works again with num_sql_socks=30. To make it work, I increased the number of connections on the mysql server via set global max_connections =200; (I have more than one sql-module and each one starts its own connections) It seems, freeradius comes into trouble, when the number of connections allowed by mysql is exceeded. I did not notice that in former versions of freeradius. Hope this information helps to find the real cause for freeradius going to sleep. Norbert Wegener Norbert Wegener schrieb:
Alan DeKok wrote:
..
Any other suggestions?
Not right now. If you're willing to do a binary search in CVS to see *when* it started breaking... but that's a lot of work.
Without changing versions of mysql or freeradius I think I already found out, that num_sql_socks seems to have a significant influence on the behaviour. With a value of 5, freeradius now is already running for some hours. A value of 20 stops it from working normally within two hours. Maybe I can find out more.
Norbert Wegener
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Norbert Wegener Siemens AG Siemens IT Solutions and Services SBS GO GIO NW PSU2 Kruppstr. 16 D-46128 Essen, Germany Phone : +49 (0) 201 816-3116 Fax. : +49 (0) 201 816-5581284 mailto:norbert.wegener@siemens.com Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Gerhard Cromme Vorstand: Peter Löscher, Vorsitzender; Johannes Feldmayer, Heinrich Hiesinger, Joe Kaeser, Rudi Lamprecht, Eduardo Montes, Jürgen Radomski, Erich R. Reinhardt, Hermann Requardt, Uriel J. Sharef, Klaus Wucherer Sitz der Gesellschaft: Berlin und München; Registergericht: Berlin Charlottenburg, HRB 12300, München, HRB 6684 WEEE-Reg.-Nr. DE 23691322
Norbert Wegener wrote:
Now it works again with num_sql_socks=30. To make it work, I increased the number of connections on the mysql server via set global max_connections =200; (I have more than one sql-module and each one starts its own connections)
The only other change I see in the SQL module is rlm_sql.c: @@ -555,7 +556,7 @@ /* * Get the list of groups this user is a member of */ - if (sql_get_grouplist(inst, sqlsocket, req, &group_list)) { + if (sql_get_grouplist(inst, sqlsocket, req, &group_list) < 0) { radlog(L_ERR, "rlm_sql (%s): Error getting group membership", inst->config->xlat_name); /* Remove the username we (maybe) added above */ @@ -605,7 +606,7 @@ /* * Get the list of groups this user is a member of */ - if (sql_get_grouplist(inst, sqlsocket, request, &group_list)) { + if (sql_get_grouplist(inst, sqlsocket, request, &group_list) < 0) { radlog(L_ERR, "rlm_sql (%s): Error retrieving group list", inst->config->xlat_name); return -1;
It seems, freeradius comes into trouble, when the number of connections allowed by mysql is exceeded.
That sounds like it's not properly handling errors returned from the SQL libraries.
I did not notice that in former versions of freeradius. Hope this information helps to find the real cause for freeradius going to sleep.
I don't see much in rlm_sql that could cause this problem. I don't see how changes to the server core can affect the SQL module. Alan DeKok.
participants (4)
-
A.L.M.Buxey＠lboro.ac.uk -
Alan DeKok -
Norbert Wegener -
tnt＠kalik.co.yu