possible radutmp problem in 2.1.4 and/or 2.1.5-git(2009-04-17 state)?
Hello! I use freeradius 1.1.x version a few years ago without any problems with 5 cisco NAS and ~15,000 online user in daytime. This work with realtime mysql accounting, but more user, more accounting record --> growing problem with the records rotate and user search without long sql lock. A few days before I changed the version to 2.1.4 with modified configuration, traditional authentication and separate radrelay process for accounting. If I need rotate the radacct table, only stop the radrelay, and this is ideal for me, I do the rotate without sql locking problem or the tech support team can do a search without long time lock :) But I found some problem. On the 2.1.4 version found the "detail file polling go crazy" problem and the fix from 2.1.5-git. This works very well, thanks :) The second problem both the 2.1.4 and 2.1.5, but the 2.1.5git well tested: In this situation: -or the network connection interrupted for a few seconds or a minute between the radius server and NAS or sql server -or the radiusd need stop, some second still and start -or after 5-10 hour (the longest running time 16 hour), -or the radutmp file size around 1.8-2.2 Mbyte the radius stop responding. I was luck, and catch an strace output, while stop: stat("/usr/local/freeradius/var/log/radius/radacct/radrelay/detail", 0x705d786a5ba0) = -1 ENOENT (No file or directory) stat("/usr/local/freeradius/var/log/radius/radacct/radrelay", {st_mode=S_IFDIR|0755, st_size=6, ...}) open("/usr/local/freeradius/var/log/radius/radacct/radrelay/detail", O_WRONLY|O_CREAT|O_APPEND, 0600) lseek(127, 0, SEEK_SET) = 0 flock(127, LOCK_EX|LOCK_NB) = 0 fstat(127, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 fcntl(127, F_GETFL) = 0x8401 (flags O_WRONLY|O_APPEND|O_LARGEFILE) fstat(127, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x6a8898c4d000 lseek(127, 0, SEEK_CUR) = 0 fstat(127, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 lseek(127, 0, SEEK_SET) = 0 write(127, "Fri Apr 17 21:45:34 2009\n\tAcct-S"..., 492) = 492 lseek(127, 0, SEEK_SET) = 0 flock(127, LOCK_UN) = 0 close(127) = 0 munmap(0x6a8898c4d000, 4096) = 0 open("/usr/local/freeradius/var/log/radius/radutmp", O_RDWR|O_CREAT, 0600) = 127 flock(127, LOCK_EX ...and this point freeze the process, rarely only the kill -9 the solution for stop. The restart is not possible, it can do accept one or two user after start, and freeze again. The only one solution is delete the radutmp file, and start again. At this moment (when can do only 1-2 user) do a debug with -X paramter. The latest lines, twice probe to start: The first probe: [radrelay-detail] expand: /usr/local/freeradius/var/log/radius/radacct/radrelay/detail -> /usr/local/freeradius/var/log/radius/radacct/radrelay/detail [radrelay-detail] /usr/local/freeradius/var/log/radius/radacct/radrelay/detail expands to /usr/local/freeradius/var/log/radius/radacct/radrelay/detail [radrelay-detail] Acquired filelock, tried 1 time(s) [radrelay-detail] expand: %t -> Sun Apr 19 16:34:03 2009 [radrelay-detail] Released filelock ++[radrelay-detail] returns ok [radutmp] expand: /dev/shm/radutmp -> /dev/shm/radutmp [radutmp] expand: %{User-Name} -> myuser@myrealm.hu ...and this pont freeze and not responding. The second probe to start: [radrelay-detail] expand: /usr/local/freeradius/var/log/radius/radacct/radrelay/detail -> /usr/local/freeradius/var/log/radius/radacct/radrelay/detail [radrelay-detail] /usr/local/freeradius/var/log/radius/radacct/radrelay/detail expands to /usr/local/freeradius/var/log/radius/radacct/radrelay/detail [radrelay-detail] Acquired filelock, tried 1 time(s) [radrelay-detail] expand: %t -> Sun Apr 19 16:34:59 2009 [radrelay-detail] Released filelock ++[radrelay-detail] returns ok [radutmp] expand: /dev/shm/radutmp -> /dev/shm/radutmp [radutmp] expand: %{User-Name} -> myotheruser@myrealm.hu ...and this pont freeze and not responding. When delete the radutmp, and start the radius, the authentication is very fast, a custom, radclient style monitoring system say about 10-12 msec for a login. If the radutmp file grow, this time going over 30 msec, when the radutmp file in the /dev/shm "ramdisk". If the radutmp file on standard filesystem, had similar effect, but a little slower respond. Could someone help, what should I do? Ran out of ideas... The radutmp file usage better me, than the sql backend, because the radrelay do a little delay. Thanks, RB
Rozsahegyi Bela wrote: ... = 0
open("/usr/local/freeradius/var/log/radius/radutmp", O_RDWR|O_CREAT, 0600) = 127 flock(127, LOCK_EX
...and this point freeze the process, rarely only the kill -9 the solution for stop.
That looks like another process is locking the file.
When delete the radutmp, and start the radius, the authentication is very fast, a custom, radclient style monitoring system say about 10-12 msec for a login. If the radutmp file grow, this time going over 30 msec, when the radutmp file in the /dev/shm "ramdisk". If the radutmp file on standard filesystem, had similar effect, but a little slower respond.
OK... so what kind of file system are you using when it stops working? NFS?
Could someone help, what should I do? Ran out of ideas...
Don't put any of the server files on NFS.
The radutmp file usage better me, than the sql backend, because the radrelay do a little delay.
If you're using radrelay, the delay doesn't matter. Use SQL for user logins, not radutmp. Alan DeKok.
Thanks for a very quick answer!
...and this point freeze the process, rarely only the kill -9 the solution for stop.
That looks like another process is locking the file.
I do not think so, because this is a dedicated server for radius, but on the next stop, i try lookup with lsof or other strace trick.
When delete the radutmp, and start the radius, the authentication is very fast, a custom, radclient style monitoring system say about 10-12 msec for a login. If the radutmp file grow, this time going over 30 msec, when the radutmp file in the /dev/shm "ramdisk". If the radutmp file on standard filesystem, had similar effect, but a little slower respond.
OK... so what kind of file system are you using when it stops working? NFS?
Nowise :) The /dev/shm is tmpfs, the normal filesystem is XFS and noatime,nodiratime mount option for the radius directory: /dev/md3 on /usr/local/freeradius type xfs (rw,noatime,nodiratime) I have an other idea that this may be a kernel related bug, the current kernel version is 2.6.27.13 with grsec patch, but late night I change the kernel, to exclude this chance.
Could someone help, what should I do? Ran out of ideas...
Don't put any of the server files on NFS.
The radutmp file usage better me, than the sql backend, because the radrelay do a little delay.
If you're using radrelay, the delay doesn't matter. Use SQL for user logins, not radutmp.
What would you propose? SQL select from the big accounting table for already logged in users (~1.2 million row), or a smaller table dedicatad this functions with insert on log in and delete on log off? RB
participants (2)
-
Alan DeKok -
Rozsahegyi Bela