Timeout problem behind load balancer
I've got a few working freeradius servers that we've spread the load with some basic round robin dns shizzle. Today I found out that Google's Compute Engine network load balancers support UDP traffic so I thought I'd try them out. Firstly I thought everything had gone smoothly as I could see Access-Accept appearing in the debug logs. However on further inspection, whilst I always get this message (when using a valid account), my client doesn't actually receive the response and therefore fails. I've tried with a simulator (jradius) and a coova client - all the same. There's no errors in the logs - the last few lines are as so: Thu Jul 24 23:51:23 2014 : Debug: rlm_sql (sql_primary): Reserving sql socket id: 3 Thu Jul 24 23:51:23 2014 : Info: row[0] returned NULL Thu Jul 24 23:51:23 2014 : Debug: rlm_sql (sql_primary): Released sql socket id: 3 Thu Jul 24 23:51:23 2014 : Info: expand: %{sql_primary:select location_id from nas where calledstationid='%{Called-Station-Id-Clean}'} -> Thu Jul 24 23:51:23 2014 : Info: ++} # update request = noop Thu Jul 24 23:51:23 2014 : Info: ++? if ("%{Tmp-String-4}") Thu Jul 24 23:51:23 2014 : Info: expand: %{Tmp-String-4} -> Thu Jul 24 23:51:23 2014 : Info: ? Evaluating ("%{Tmp-String-4}") -> FALSE Thu Jul 24 23:51:23 2014 : Info: ++? if ("%{Tmp-String-4}") -> FALSE Thu Jul 24 23:51:23 2014 : Info: [sql_cache] expand: %{User-Name} -> test Thu Jul 24 23:51:23 2014 : Info: [sql_cache] sql_set_user escaped user --> 'test' Thu Jul 24 23:51:23 2014 : Info: [sql_cache] expand: %{User-Password} -> account Thu Jul 24 23:51:23 2014 : Info: [sql_cache] expand: INSERT INTO radpostauth_cache (username, pass, reply, authdate) VALUES ( '%{User-Name}', '%{%{User-Password}:-%{Chap-Password}}', '%{reply:Packet-Type}', '%S') -> INSERT INTO radpostauth_cache (username, pass, reply, authdate) VALUES ( 'test', 'account', 'Access-Accept', '2014-07-24 23:51:23') Thu Jul 24 23:51:23 2014 : Debug: rlm_sql (sql_cache) in sql_postauth: query is INSERT INTO radpostauth_cache (username, pass, reply, authdate) VALUES ( 'test', 'account', 'Access-Accept', '2014-07-24 23:51:23') Thu Jul 24 23:51:23 2014 : Debug: rlm_sql (sql_cache): Reserving sql socket id: 2 Thu Jul 24 23:51:23 2014 : Debug: rlm_sql (sql_cache): Released sql socket id: 2 Thu Jul 24 23:51:23 2014 : Info: ++[sql_cache] = ok Thu Jul 24 23:51:23 2014 : Info: ++[exec] = noop Thu Jul 24 23:51:23 2014 : Info: +} # group post-auth = ok Sending Access-Accept of id 8 to 213.205.230.210 port 49080 Thu Jul 24 23:51:23 2014 : Info: Finished request 7. Thu Jul 24 23:51:23 2014 : Debug: Going to the next request Thu Jul 24 23:51:23 2014 : Debug: Waking up in 1.9 seconds. Thu Jul 24 23:51:25 2014 : Info: Cleaning up request 7 ID 8 with timestamp +177 Thu Jul 24 23:51:25 2014 : Info: Ready to process requests. I've discussed with Google and they're saying the connection isn't closed and the load-balancer therefore puts the instance in 'unhealthy' mode. This doesn't sound right to me. I think the traffic is being blocked. The only reference I can find to a similar issue was a user years ago who had network issues.. Has anyone tried this successfully - either on GCE or other UDP load balancer? Does anyone have any thoughts about how I could solve?
Simon Morley wrote:
I've discussed with Google and they're saying the connection isn't closed and the load-balancer therefore puts the instance in 'unhealthy' mode.
UDP sessions aren't "closed". The google support guys don't seem to be competent.
Has anyone tried this successfully - either on GCE or other UDP load balancer? Does anyone have any thoughts about how I could solve?
F5 load balancers seem to work. But they are useful only when you're at Telco scale. i.e. 1M+ users. For pretty much everyone else, it doesn't matter. Alan DeKok.
Agreed. So it would seem there is an issue getting the traffic back out again as suspected. We don't have a million users so I shan't bother looking at the F5s. Will hassle Google again. I suspect their health check is randomly removing the machines from the load balancers. Thanks On 25 July 2014 13:24, Alan DeKok <aland@deployingradius.com> wrote:
Simon Morley wrote:
I've discussed with Google and they're saying the connection isn't closed and the load-balancer therefore puts the instance in 'unhealthy' mode.
UDP sessions aren't "closed". The google support guys don't seem to be competent.
Has anyone tried this successfully - either on GCE or other UDP load balancer? Does anyone have any thoughts about how I could solve?
F5 load balancers seem to work. But they are useful only when you're at Telco scale. i.e. 1M+ users. For pretty much everyone else, it doesn't matter.
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Fri, Jul 25, 2014 at 7:48 PM, Simon Morley <simon@polkaspots.com> wrote:
Agreed. So it would seem there is an issue getting the traffic back out again as suspected.
We don't have a million users so I shan't bother looking at the F5s. Will hassle Google again.
I suspect their health check is randomly removing the machines from the load balancers.
I'm curious what they use for udp health check. Do they even have any? Is it something that aware of radius packets? Or is it just basic ping check? -- Fajar
Thanks
On 25 July 2014 13:24, Alan DeKok <aland@deployingradius.com> wrote:
Simon Morley wrote:
I've discussed with Google and they're saying the connection isn't closed and the load-balancer therefore puts the instance in 'unhealthy' mode.
UDP sessions aren't "closed". The google support guys don't seem to be competent.
Has anyone tried this successfully - either on GCE or other UDP load balancer? Does anyone have any thoughts about how I could solve?
F5 load balancers seem to work. But they are useful only when you're at Telco scale. i.e. 1M+ users. For pretty much everyone else, it doesn't matter.
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
That's the frustrating part. Despite being able to balance UDP connections, you need to use an http check. There's not even a tcp option. We've resorted to running nginx on the servers and if we decide to production it, maybe we'll build something into monit to kill nginx if radius dies. Hacky. Essentially it's just checking if the server's up in our case. Which is better than the round-robin stuff we have tried previously. Actually, previously we used Amazon's route 53 to route the traffic but this also needed a tcp / http check. They claim they're not blocking outbound traffic other than the obvious (smtp etc.) but I don't believe that currently. If radius works when we hit the server without the load-balancer and then mysteriously stops sending out, it would appear it is being blocked. On 25 July 2014 14:06, Fajar A. Nugraha <list@fajar.net> wrote:
On Fri, Jul 25, 2014 at 7:48 PM, Simon Morley <simon@polkaspots.com> wrote:
Agreed. So it would seem there is an issue getting the traffic back out again as suspected.
We don't have a million users so I shan't bother looking at the F5s. Will hassle Google again.
I suspect their health check is randomly removing the machines from the load balancers.
I'm curious what they use for udp health check. Do they even have any? Is it something that aware of radius packets? Or is it just basic ping check?
-- Fajar
Thanks
On 25 July 2014 13:24, Alan DeKok <aland@deployingradius.com> wrote:
Simon Morley wrote:
I've discussed with Google and they're saying the connection isn't closed and the load-balancer therefore puts the instance in
mode.
UDP sessions aren't "closed". The google support guys don't seem to be competent.
Has anyone tried this successfully - either on GCE or other UDP load balancer? Does anyone have any thoughts about how I could solve?
F5 load balancers seem to work. But they are useful only when you're at Telco scale. i.e. 1M+ users. For pretty much everyone else, it doesn't matter.
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
'unhealthy' - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Have you analysed the traffic eg put a radius system praise so you can see what's happening. If the load balancer is playing with the UDP then the clients won't like it. Regarding the http... freeradius could be listening on that with a small service. Bit what's the point? You want to check the service you are balancing is working not some other service on another port. I've got a feeling they aren't doing any proper udp stuff and it's all designed for Web stuff. Proper load balancers understand the protocols and will balance accordingly (eg make note of RADIUS attributes etc) alan -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
participants (4)
-
Alan Buxey -
Alan DeKok -
Fajar A. Nugraha -
Simon Morley