Using libkqueue, how stable it is with freeradius 3.x?
Hi, I want to try to use freeradius 3.2.x which is compiled with libkquueue. In my tests, it is working fine. But I also see the list of know issues https://github.com/mheily/libkqueue/blob/master/BUGS.md of libkqueue and just want to double-check if I should not use it. But I hope that since freeradius 4.x by default will use it the library integration is fine and the listed issues are not affecting freeradius 4.x or 3.x. Best regards, Vladimir
On Jun 15, 2023, at 6:36 AM, work vlpl <thework.vlpl@gmail.com> wrote:
Hi, I want to try to use freeradius 3.2.x which is compiled with libkquueue. In my tests, it is working fine. But I also see the list of know issues https://github.com/mheily/libkqueue/blob/master/BUGS.md of libkqueue and just want to double-check if I should not use it.
Why are you trying to use 3.2 with libkqueue? There is pretty much zero benefit.
But I hope that since freeradius 4.x by default will use it the library integration is fine and the listed issues are not affecting freeradius 4.x or 3.x.
v4 requires the most recent development version of libkqueue. We've put many fixes into libkqueue. v3 uses select() etc. on Linux, and kqueue() on OSX and *BSD. If you want to use libkqueue on Linux with v3, then either don't (the recommended approach), or use the most recent code from the git repo. The released packages of libkqueue are very old. Alan DeKok.
On 15 Jun 2023 at 14:33:51, Alan DeKok <aland@deployingradius.com> wrote:
Why are you trying to use 3.2 with libkqueue? There is pretty much zero benefit.
I have a situation when I need to use freeradius as a load balancer to look inside radius packets for attributes because balancing only by IP + port not helping since each new packet from some clients arrives from new source ports (can't fix clients). Select() has a limitation in 1024 fd and during the testing I hit such a limit.
If you want to use libkqueue on Linux with v3, then either don't (the recommended approach), or use the most recent code from the git repo. The released packages of libkqueue are very old.
I am compiling freeradius with 2.6.1 libkqueue, is it ok or do I need to use HEAD for the master branch?
On Jun 15, 2023, at 8:53 AM, work vlpl <thework.vlpl@gmail.com> wrote:
I have a situation when I need to use freeradius as a load balancer to look inside radius packets for attributes because balancing only by IP + port not helping since each new packet from some clients arrives from new source ports (can't fix clients).
Select() has a limitation in 1024 fd and during the testing I hit such a limit.
If you're using UDP, then the client source port doesn't matter. FreeRADIUS doesn't open a new socket for every UDP packet. If you're using TCP, then don't. It's insecure. If you're using TLS, and the server only receives one packet per connection, then the client is broken. But you have the source code to FreeRADIUS. You can add some code to close the TLS socket after the reply packet is sent. I really don't understand why it would need more than 1024 sockets. That just makes no sense.
If you want to use libkqueue on Linux with v3, then either don't (the recommended approach), or use the most recent code from the git repo. The released packages of libkqueue are very old.
I am compiling freeradius with 2.6.1 libkqueue, is it ok or do I need to use HEAD for the master branch?
Did I say use the packaged release, or the most recent one version from git? Alan DeKok.
On 15 Jun 2023 at 14:58:14, Alan DeKok <aland@deployingradius.com> wrote:
If you're using UDP, then the client source port doesn't matter.
FreeRADIUS doesn't open a new socket for every UDP packet.
I apologize for raising this topic, as it may be unrelated to my original question. However, if you see alternative methods of achieving load balancing or recommend a different load balancer, I would greatly appreciate it. So these IoT clients create a huge load and if I am using another load balancer that cares only about ip and port I have a next issue. Clients are UDP and traffic is from one ip address. If I will use a hash or IP address as a balancing key, then balancing will not work - all traffic will be sent to 1 server because the hash from 1 ip address will be the same. Hash from ip + port - I can get a good spread between servers. But the clients or AP sends each new radius packet from a new random port this lead to a situation when packets from 1 radius session are routed to different servers, and the radius server can't authorize users with missing packets from the sequence. And I can't fix clients or AP. So because of that, I can't use the udp source port in the hash key for balancing distribution, then I need to use something else. I think radius attributes like Calling and Called Station ID are good enough. And for load balancing, I am using freeradius and only the proxy module. Freeradius working great and fast. But with many home servers and virtual server configurations that fit my need to fully consume server resources and updated value of FR_EV_MAX_FDS constant, I hit select() limitation. I know it is a weird configuration. And the general suggestion probably will be to add another server for a horizontal scale. But still, I want to try kqueue.
Did I say use the packaged release, or the most recent one version from git?
Got it, thank you!
On Jun 15, 2023, at 10:18 AM, work vlpl <thework.vlpl@gmail.com> wrote:
So these IoT clients create a huge load and if I am using another load balancer that cares only about ip and port I have a next issue. Clients are UDP and traffic is from one ip address.
All packets from one IP should be fine.
If I will use a hash or IP address as a balancing key, then balancing will not work - all traffic will be sent to 1 server because the hash from 1 ip address will be the same.
*What* is doing is doing the hash? Is it the load balancer? If so, use another one. If the load balancing is done in FreeRADIUS (and it should be), see proxy.conf. There is a "client-port-balance" configuration which hashes on client IP+port, which works for sane clients. But FreeRADIUS also supports keyed-load-balance. In which case you can control which attributes are used to do load balancing.
Hash from ip + port - I can get a good spread between servers. But the clients or AP sends each new radius packet from a new random port this lead to a situation when packets from 1 radius session are routed to different servers, and the radius server can't authorize users with missing packets from the sequence. And I can't fix clients or AP.
That sounds like the AP is broken, to be honest. Using a different source port for each packet in an ongoing EAP session is completely stupid.
So because of that, I can't use the udp source port in the hash key for balancing distribution, then I need to use something else. I think radius attributes like Calling and Called Station ID are good enough.
So use "keyed-load-balance", and copy Calling-Station-Id to the Load-Balance-Key attribute. FreeRADIUS will ignore the client IP+port, and key on the Calling-Station-Id (i.e. MAC address). That shouldn't change from packet to packet. So the load balancing will send all packets for one session to the same back-end RADIUS server. Even if the AP is completely broken.
And for load balancing, I am using freeradius and only the proxy module.
That's good.
Freeradius working great and fast. But with many home servers and virtual server configurations that fit my need to fully consume server resources and updated value of FR_EV_MAX_FDS constant, I hit select() limitation
You still haven't explained why it's opening 1024 listeners. So you're keeping some part of your configuration secret, I don't know why. Hint: I can't read your mind. I can't magically log into your serves remotely and see what they do. I can only read the messages you post to the list. And if you don't explain something, then I don't know anything about it.
I know it is a weird configuration. And the general suggestion probably will be to add another server for a horizontal scale. But still, I want to try kqueue.
If you try libkueue on Linux, then you're on your own. It might work. If so, great. If not, we won't spend time fixing it. It is not officially supported for v3. I would suggest using keyed-load-balance as I said before. It should solve the load-balancing issues. And I still have absolutely no idea what you think the server needs 1024 listeners. It makes zero sense, and you're refusing to explain. The underlying problem here is that you're doing massive amounts of work for no reason. You're wasting your time by trying libkqueue, instead of using a documented solution. You're wasting our time by asking for help, and then refusing to describe what you're actually doing. Please get better at explaining things. I'm not a mind reader, and I don't want to be. If you don't explain what you're doing, then I can't help you. Alan DeKok.
On 15 Jun 2023 at 16:43:29, Alan DeKok <aland@deployingradius.com> wrote:
You still haven't explained why it's opening 1024 listeners. So you're keeping some part of your configuration secret, I don't know why.
This is the old setup so I don't know all the details or maybe the reasons are not valid anymore. But the idea was next, since it was not possible to have clients with the same ip address but different shared secrets it was decided to create a virtual server with a different port and shared secret for each client. On normal servers, it is ok I do not hit the limit. But I have a pool of normal servers that I want to put behind a load balancer, and it was seems reasonable to use a load balancer or proxy server to handle the load for more than 1 pool or normal server. So let's assume that I have 2 pools of servers with 256 virtual servers (ports or sockets) in each. This means on the load balancer I need to listen to 512 ports which hit the current limit. And I saw this message, it is probably not what I hit but similar so I've also tried to change the constant in an experimental build and hit the select() limit. https://lists.freeradius.org/pipermail/freeradius-users/2018-March/091045.ht...
If you try libkueue on Linux, then you're on your own. It might work. If so, great. If not, we won't spend time fixing it. It is not officially supported for v3.
You're wasting our time by asking for help, and then refusing to describe what you're actually doing.
Yes, I understand that this is not supported approach, but the setup that I have is old, and it will be a huge deal to redo it from the beginning, so I just asking what to expect from the version that uses libkqueue.
On Jun 15, 2023, at 12:44 PM, work vlpl <thework.vlpl@gmail.com> wrote:
This is the old setup so I don't know all the details or maybe the reasons are not valid anymore. But the idea was next, since it was not possible to have clients with the same ip address but different shared secrets it was decided to create a virtual server with a different port and shared secret for each client.
It might be worth fixing the network so that each client has a unique IP address. That is generally the best way to do it. i.e. if no one else runs into the 1024 socket limit, it's because no one else is doing what you're doing. Which means there's likely a better way to get the same result.
On normal servers, it is ok I do not hit the limit. But I have a pool of normal servers that I want to put behind a load balancer, and it was seems reasonable to use a load balancer or proxy server to handle the load for more than 1 pool or normal server.
So let's assume that I have 2 pools of servers with 256 virtual servers (ports or sockets) in each. This means on the load balancer I need to listen to 512 ports which hit the current limit.
I don't know why you need a "pool" of servers. Is this for authentication and accounting? Nothing goes wrong if you configure both of them to use the same port. The documentation describes how to do this.
Yes, I understand that this is not supported approach, but the setup that I have is old, and it will be a huge deal to redo it from the beginning, so I just asking what to expect from the version that uses libkqueue.
If it works, great. If not, it's a bug in libkqueue. It's likely fixed by using the latest version of libkqueue. If it doesn't work, then we're not going to spend time tracking down the issue. v3 + libkqueue is not supported. Alan DeKok.
participants (2)
-
Alan DeKok -
work vlpl