Freeradius Proxy
Hello all, I have setup a freeradius proxy server and proxied my radius requests to two different freeradius servers. everything is working fine except for this error, Thu Jan 23 16:01:08 2025 : Proxy: Marking home server 3.147.84.26 port 1813 as zombie (it has not responded in 30.000000 seconds). Thu Jan 23 16:01:08 2025 : ERROR: (17) ERROR: Failing proxied request for user "90E8688069D5202300000005G1", due to lack of any response from home server 3.147.84.26 port 1813 Thu Jan 23 16:01:08 2025 : Proxy: (30) Marking home server 3.147.84.26 port 1813 alive Thu Jan 23 16:01:08 2025 : Proxy: (30) Received response to status check 30 ID 101 (1 in current sequence) Thu Jan 23 16:01:08 2025 : Proxy: Marking home server 3.147.84.26 port 1813 as zombie (it has not responded in 30.000000 seconds). Thu Jan 23 16:01:08 2025 : ERROR: (18) ERROR: Failing proxied request for user "505A65FC148D202300000005G1", due to lack of any response from home server 3.147.84.26 port 1813 Thu Jan 23 16:01:08 2025 : ERROR: (19) ERROR: Failing proxied request for user "90E868812627202300000005G1", due to lack of any response from home server 3.147.84.26 port 1813 Thu Jan 23 16:01:08 2025 : Proxy: Marking home server 18.191.101.110 port 1813 as zombie (it has not responded in 30.000000 seconds). Thu Jan 23 16:01:08 2025 : ERROR: (20) ERROR: Failing proxied request for user "A8934AEA6A9B202300000005G1", due to lack of any response from home server 18.191.101.110 port 1813 Thu Jan 23 16:01:08 2025 : Proxy: (31) Marking home server 3.147.84.26 port 1813 alive Thu Jan 23 16:01:08 2025 : Proxy: (31) Received response to status check 31 ID 41 (1 in current sequence) Thu Jan 23 16:01:08 2025 : Proxy: (32) Marking home server 18.191.101.110 port 1813 alive Thu Jan 23 16:01:08 2025 : Proxy: (32) Received response to status check 32 ID 20 (1 in current sequence) My two servers are keep getting zombie and alive. But all the requests getting back to my two servers. How to resolve this issue? These are the things I have tried so far, 1. add a custom radius reply to my accounting handling perl script $RAD_REPLY{'Acct-Status-Type'} = 'Interim-Update'; ------- did not work $RAD_REPLY{'Reply-Message'} = 'accounting received'; ---------did not work 2. add a reply from sites-enabled/default 3. tried to reduce/remove attributes for proxy.conf for accounting related home_servers I really appreciate any help you guys can provide me. Thanks, Tharka Tharka Karunanayake Engineer - Research and Development +94774129055 https://thryvz.com<https://thryvz.com/> https://monyfi.com [cid:b8f7da65-8c53-4ca7-a5f7-b0a977b204e3]<https://thryvz.com/>
On Jan 23, 2025, at 5:49 AM, Tharka Karunanayake <tharkak@nvision.lk> wrote:
I have setup a freeradius proxy server and proxied my radius requests to two different freeradius servers. everything is working fine except for this error,
Thu Jan 23 16:01:08 2025 : Proxy: Marking home server 3.147.84.26 port 1813 as zombie (it has not responded in 30.000000 seconds).
That's pretty clear.
My two servers are keep getting zombie and alive. But all the requests getting back to my two servers. How to resolve this issue?
No. The replies aren't making it back. The home server is either going down, or is being very very slow for 30+ seconds at a time.
These are the things I have tried so far,
1. add a custom radius reply to my accounting handling perl script $RAD_REPLY{'Acct-Status-Type'} = 'Interim-Update'; ------- did not work $RAD_REPLY{'Reply-Message'} = 'accounting received'; ---------did not work
Adding attributes to a reply won't help, when the reply isn't being sent.
2. add a reply from sites-enabled/default 3. tried to reduce/remove attributes for proxy.conf for accounting related home_servers
You can't fix a slow home server by poking the proxy. Look at the logs on the home server. I guarantee you that the logs will be FULL of messages complaining about unresponsive child / blocked in module / etc. If you're running a Perl script on the home server ... why? The server doesn't need to run Perl. The *only* reason to use a Perl script is to interact with some weird external API, and the only available libraries which support that are in Perl. Alan DeKok.
Dear Alan, Thank you so much for the quick and on point response.
Look at the logs on the home server. I guarantee you that the logs will be FULL of messages complaining about unresponsive child / blocked in module / etc.
I will sure to look into this.
If you're running a Perl script on the home server ... why? The server doesn't need to run Perl. The *only* reason to use a Perl script is to interact with some weird external API, and the only available libraries which support that are in Perl.
got it thanks. This radius server is something that has been developed 15 years ago by someone else and I was assigned to optimize it very recently. Currently it uses one Perl script to do everything and does not use modules. About 10% of the customers cannot get authenticate too. As a newbie I thought freeradius load balancer would resolve the problem. I will try for a solution and thanks again for pointing me in the right direction. Thanks, Tharka Tharka Karunanayake ________________________________ From: Freeradius-Devel <freeradius-devel-bounces+tharkak=nvision.lk@lists.freeradius.org> on behalf of Alan DeKok <aland@deployingradius.com> Sent: Thursday, January 23, 2025 5:22 PM To: FreeRadius developers mailing list <freeradius-devel@lists.freeradius.org> Subject: Re: Freeradius Proxy On Jan 23, 2025, at 5:49 AM, Tharka Karunanayake <tharkak@nvision.lk> wrote:
I have setup a freeradius proxy server and proxied my radius requests to two different freeradius servers. everything is working fine except for this error,
Thu Jan 23 16:01:08 2025 : Proxy: Marking home server 3.147.84.26 port 1813 as zombie (it has not responded in 30.000000 seconds).
That's pretty clear.
My two servers are keep getting zombie and alive. But all the requests getting back to my two servers. How to resolve this issue?
No. The replies aren't making it back. The home server is either going down, or is being very very slow for 30+ seconds at a time.
These are the things I have tried so far,
1. add a custom radius reply to my accounting handling perl script $RAD_REPLY{'Acct-Status-Type'} = 'Interim-Update'; ------- did not work $RAD_REPLY{'Reply-Message'} = 'accounting received'; ---------did not work
Adding attributes to a reply won't help, when the reply isn't being sent.
2. add a reply from sites-enabled/default 3. tried to reduce/remove attributes for proxy.conf for accounting related home_servers
You can't fix a slow home server by poking the proxy. Look at the logs on the home server. I guarantee you that the logs will be FULL of messages complaining about unresponsive child / blocked in module / etc. If you're running a Perl script on the home server ... why? The server doesn't need to run Perl. The *only* reason to use a Perl script is to interact with some weird external API, and the only available libraries which support that are in Perl. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/devel.html
On Jan 23, 2025, at 10:22 PM, Tharka Karunanayake <tharkak@nvision.lk> wrote:
got it thanks. This radius server is something that has been developed 15 years ago by someone else
It needs to be updated to v3 then, too.
and I was assigned to optimize it very recently. Currently it uses one Perl script to do everything and does not use modules. About 10% of the customers cannot get authenticate too. As a newbie I thought freeradius load balancer would resolve the problem. I will try for a solution and thanks again for pointing me in the right direction.
Adding more capacity hides the fact that the system is slow. My guess is that you can rewrite all of the Perl into v3 unlang. At that point, performance should improve substantially. Also, you should investigate *why* the Perl script is so slow. Even with Perl being inefficient, there is no reason for it to take 30+ seconds to process a packer. That delay is almost always due to database issues. Odds are that fi you fix the database issues, the errors and authentication failures will go away. Alan DeKok.
It needs to be updated to v3 then, too. It is v3. They do update the freeradius server but that's it.
Also, you should investigate *why* the Perl script is so slow. Even with Perl being inefficient, there is no reason for it to take 30+ seconds to process a packer. That delay is almost always due to database issues.
Odds are that fi you fix the database issues, the errors and authentication failures will go away. Thanks again for the response, I also believe the issue is on the database because we use mysql to log accounting instead of .txt or any other file type. As a first step I'll log the accounting into a file, then I'll gradually convert the whole perl script into unlang.
I also have some questions regarding freeDiameter, is it okay to ask it in here? Thanks, Tharka Tharka Karunanayake Engineer - Research and Development +94774129055 https://thryvz.com<https://thryvz.com/> https://monyfi.com [cid:a4e1be99-9203-49a8-987e-ed8fc90ef9ab]<https://thryvz.com/> ________________________________ From: Freeradius-Devel <freeradius-devel-bounces+tharkak=nvision.lk@lists.freeradius.org> on behalf of Alan DeKok <aland@deployingradius.com> Sent: Friday, January 24, 2025 8:58 PM To: FreeRadius developers mailing list <freeradius-devel@lists.freeradius.org> Subject: Re: Freeradius Proxy On Jan 23, 2025, at 10:22 PM, Tharka Karunanayake <tharkak@nvision.lk> wrote:
got it thanks. This radius server is something that has been developed 15 years ago by someone else
It needs to be updated to v3 then, too.
and I was assigned to optimize it very recently. Currently it uses one Perl script to do everything and does not use modules. About 10% of the customers cannot get authenticate too. As a newbie I thought freeradius load balancer would resolve the problem. I will try for a solution and thanks again for pointing me in the right direction.
Adding more capacity hides the fact that the system is slow. My guess is that you can rewrite all of the Perl into v3 unlang. At that point, performance should improve substantially. Also, you should investigate *why* the Perl script is so slow. Even with Perl being inefficient, there is no reason for it to take 30+ seconds to process a packer. That delay is almost always due to database issues. Odds are that fi you fix the database issues, the errors and authentication failures will go away. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/devel.html
On Jan 25, 2025, at 4:30 AM, Tharka Karunanayake <tharkak@nvision.lk> wrote:
Thanks again for the response, I also believe the issue is on the database because we use mysql to log accounting instead of .txt or any other file type. As a first step I'll log the accounting into a file, then I'll gradually convert the whole perl script into unlang.
That should help. Slow databases usually mean that the database is too large (hundreds of gigs of unused data), or is lacking indexes, or there is a lot of lock contention. Those need to be checked and fixed, no matter what your solution.
I also have some questions regarding freeDiameter, is it okay to ask it in here?
freediameter has nothing to do with FreeRADIUS, so no. So far as I can tell, free diameter is essentially dead. Alan DeKok.
participants (2)
-
Alan DeKok -
Tharka Karunanayake