System frequently stops responding...

[Mohamed Lrhazi]

Hello,

So I wrote a little bash script to try and detect the event, so as to try
and launch some commands to further debug... It seems to be able to catch
it... currently it just run a server-status request, sleep 5, run
server-status again, then run "top" command...  Please see output bellow.

The way the script works is by simply running "tcpdump port 1812" for 10
seconds and try to detect if we received RADIUS requests and we did not
reply to any.

In this occurrence, the server seems to have failed to reply to the status
command twice... top shows that system load was never high.

I guess next would be to run raddedug in such a moment? Any suggestions on
other data I would want to grab, that can help identify why radiusd locks
up, or fails to respond?


Wed Jul 29 10:10:22 EDT 2015
Sent Status-Server Id 221 from 0.0.0.0:38772 to 127.0.0.1:18121 length 50
        Message-Authenticator = 0x00
        FreeRADIUS-Statistics-Type = Authentication
        Response-Packet-Type = Access-Accept
Sent Status-Server Id 221 from 0.0.0.0:38772 to 127.0.0.1:18121 length 50
        Message-Authenticator = 0x00
        FreeRADIUS-Statistics-Type = Authentication
        Response-Packet-Type = Access-Accept
Sent Status-Server Id 221 from 0.0.0.0:38772 to 127.0.0.1:18121 length 50
        Message-Authenticator = 0x00
        FreeRADIUS-Statistics-Type = Authentication
        Response-Packet-Type = Access-Accept
Received Access-Accept Id 221 from 127.0.0.1:18121 to 0.0.0.0:0 length 140
        FreeRADIUS-Total-Access-Requests = 193666
        FreeRADIUS-Total-Access-Accepts = 18043
        FreeRADIUS-Total-Access-Rejects = 3102
        FreeRADIUS-Total-Access-Challenges = 172407
        FreeRADIUS-Total-Auth-Responses = 193552
        FreeRADIUS-Total-Auth-Duplicate-Requests = 84
        FreeRADIUS-Total-Auth-Malformed-Requests = 0
        FreeRADIUS-Total-Auth-Invalid-Requests = 0
        FreeRADIUS-Total-Auth-Dropped-Requests = 88
        FreeRADIUS-Total-Auth-Unknown-Types = 0
Sent Status-Server Id 28 from 0.0.0.0:40187 to 127.0.0.1:18121 length 50
        Message-Authenticator = 0x00
        FreeRADIUS-Statistics-Type = Authentication
        Response-Packet-Type = Access-Accept
Received Access-Accept Id 28 from 127.0.0.1:18121 to 0.0.0.0:0 length 140
        FreeRADIUS-Total-Access-Requests = 194192
        FreeRADIUS-Total-Access-Accepts = 18044
        FreeRADIUS-Total-Access-Rejects = 3107
        FreeRADIUS-Total-Access-Challenges = 172443
        FreeRADIUS-Total-Auth-Responses = 193594
        FreeRADIUS-Total-Auth-Duplicate-Requests = 140
        FreeRADIUS-Total-Auth-Malformed-Requests = 0
        FreeRADIUS-Total-Auth-Invalid-Requests = 0
        FreeRADIUS-Total-Auth-Dropped-Requests = 145
        FreeRADIUS-Total-Auth-Unknown-Types = 0
top - 10:10:39 up 7 days, 10:42,  3 users,  load average: 0.37, 0.12, 0.07
Tasks: 179 total,   1 running, 178 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.5 us,  0.4 sy,  0.0 ni, 99.1 id,  0.0 wa,  0.0 hi,  0.0 si,
 0.0 st
KiB Mem :  1884696 total,  1368184 free,   165676 used,   350836 buff/cache
KiB Swap:  4194300 total,  4117172 free,    77128 used.  1447748 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
25532 root      20   0  130032   1668   1200 R   5.9  0.1   0:00.02 top
    1 root      20   0  206336   5048   2860 S   0.0  0.3   0:41.37 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.03 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:00.70
ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00
kworker/0:0H
    7 root      rt   0       0      0      0 S   0.0  0.0   0:00.65
migration/0
...