decoupled accounting cron check
Hi, For those out there using decoupled accounting, especially in an 'eduroam' environment, might find the following helpful. I receive a lot of random rubbish from the various NAS's deployed internationally send to my FreeRADIUS installation. Such moments of fun are accounting stop packets with a zero session length (CISCO_ACCOUNTING_HACK) resulting in a DoS when received in the decoupled accounting case...plus the other usual hings that trigger corner cases my custom (bad?) SQL statements do not catch when logging this information to our database. The unfortunate outcome means after a bad accounting packet, the mountpoint I use for recording my journal fills up until FreeRADIUS hangs with no warning (meanwhile FreeRADIUS works fine so it is not something trivially monitored by NAGIOS or such). The solution I slapped together is a quick minutely run script by cron that notifies me by email when a problem occurs. My preference is to place my detail journal files on a separate tmpfs mountpoint (as I use low powered ARM boxes, OpenRD's if you are curious, that only have a NAND): ---- tmpfs /var/log/freeradius/radacct/journal tmpfs nosuid,nodev,noexec,size=32M,mode=700,uid=freerad,gid=freerad 0 0 ---- Then the following script is used. ---- #!/bin/sh MOUNT=/var/log/freeradius/radacct/journal TRIGGER=1024 RCPT="jill@example.com bob@example.com" MESSAGE="FreeRADIUS is on the road to implosion...yer might want to look into it. Cheers" SELF=$(basename $0) if [ -e "/var/lock/$SELF" ]; then if [ $(df "$MOUNT" | tail -n1 | awk '{ print $3 }') -lt $(($TRIGGER/2)) ]; then rm "/var/lock/$SELF" else exit 1 fi fi [ $(df "$MOUNT" | tail -n1 | awk '{ print $3 }') -lt $TRIGGER ] && exit 0 DATE=$(date -R) TO=$(echo $RCPT | sed 's/ /, /g') cat <<EOF | /usr/sbin/sendmail -i $RCPT To: $TO Date: $DATE Subject: $MOUNT exceeds ${TRIGGER}kB $MESSAGE EOF [ $? -eq 0 ] && touch "/var/lock/$SELF" exit 1 ---- That's it. The above script will email you only a single time when more that 1024kB of journal is sitting around waiting to be processed and will re-enable notifications once it drops to half the trigger mark (512kB). Cheers -- Alexander Clouter .sigmonster says: T-shirt: Life is *not* a Cabaret, and stop calling me chum!
So does the detail reader read the packet, find that its invalid and then retry the same packet? -Arran On Mar 6, 2011, at 2:37 PM, Alexander Clouter wrote:
Hi,
For those out there using decoupled accounting, especially in an 'eduroam' environment, might find the following helpful.
I receive a lot of random rubbish from the various NAS's deployed internationally send to my FreeRADIUS installation. Such moments of fun are accounting stop packets with a zero session length (CISCO_ACCOUNTING_HACK) resulting in a DoS when received in the decoupled accounting case...plus the other usual hings that trigger corner cases my custom (bad?) SQL statements do not catch when logging this information to our database.
The unfortunate outcome means after a bad accounting packet, the mountpoint I use for recording my journal fills up until FreeRADIUS hangs with no warning (meanwhile FreeRADIUS works fine so it is not something trivially monitored by NAGIOS or such).
The solution I slapped together is a quick minutely run script by cron that notifies me by email when a problem occurs.
My preference is to place my detail journal files on a separate tmpfs mountpoint (as I use low powered ARM boxes, OpenRD's if you are curious, that only have a NAND): ---- tmpfs /var/log/freeradius/radacct/journal tmpfs nosuid,nodev,noexec,size=32M,mode=700,uid=freerad,gid=freerad 0 0 ----
Then the following script is used. ---- #!/bin/sh
MOUNT=/var/log/freeradius/radacct/journal TRIGGER=1024 RCPT="jill@example.com bob@example.com" MESSAGE="FreeRADIUS is on the road to implosion...yer might want to look into it.
Cheers"
SELF=$(basename $0)
if [ -e "/var/lock/$SELF" ]; then if [ $(df "$MOUNT" | tail -n1 | awk '{ print $3 }') -lt $(($TRIGGER/2)) ]; then rm "/var/lock/$SELF" else exit 1 fi fi
[ $(df "$MOUNT" | tail -n1 | awk '{ print $3 }') -lt $TRIGGER ] && exit 0
DATE=$(date -R) TO=$(echo $RCPT | sed 's/ /, /g')
cat <<EOF | /usr/sbin/sendmail -i $RCPT To: $TO Date: $DATE Subject: $MOUNT exceeds ${TRIGGER}kB
$MESSAGE EOF
[ $? -eq 0 ] && touch "/var/lock/$SELF"
exit 1 ----
That's it. The above script will email you only a single time when more that 1024kB of journal is sitting around waiting to be processed and will re-enable notifications once it drops to half the trigger mark (512kB).
Cheers
-- Alexander Clouter .sigmonster says: T-shirt: Life is *not* a Cabaret, and stop calling me chum!
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Arran Cudbard-Bell <a.cudbardb@gmail.com> wrote:
So does the detail reader read the packet, find that its invalid and then retry the same packet?
Yes...after waiting 30 seconds then retrying. For 'valid' packets, it is handy, as I get to fix my SQL, but there will come a point where is safe[1] and I will probably look to rely more on the approach Alan just posted...but maybe it is more interesting not to. Bah, it's only accounting packets, I have ~99%+ of them, who really cares in the 'eduroam' world if I get the final 10^-$BIGNUM :) Cheers [1] I thought I actually had it covered this week, but it imploded twice and I added a few more CASE's and %{%{...}:-0}'s -- Alexander Clouter .sigmonster says: To teach is to learn twice. -- Joseph Joubert
Alexander Clouter wrote:
The unfortunate outcome means after a bad accounting packet, the mountpoint I use for recording my journal fills up until FreeRADIUS hangs with no warning (meanwhile FreeRADIUS works fine so it is not something trivially monitored by NAGIOS or such).
2.1.10 has Packet-Transmit-Counter preacct { ... if (Packet-Transmit-Counter > 20) { ok return } ... } Should work, I think. In some cases, if the home server doesn't care enough to keep an accounting server up and running, they deserve to lose accounting data. An alternate solution would be to put the retried packets into another detail file, where they wouldn't bother anyone else. An admin could periodically check it, and delete the nonsense records. Alan DeKok.
participants (3)
-
Alan DeKok -
Alexander Clouter -
Arran Cudbard-Bell