Hi guys, I'd like to run a FR with more than 256 virtual servers/listening ports. When building and starting the latest source of the v3.0.x branch with more than 256 vs's/ports defined I get: Failed adding event handler for socket: Too many readers I traced this error back to the #define on line 37 of event.c: #define FR_EV_MAX_FDS (256) Are there are any serious repercussions of increasing this and running more than 256 vs's/listening ports? Cheers very much Kev/.
Kev Pearce wrote:
I'd like to run a FR with more than 256 virtual servers/listening ports.
Why? That's a bit unusual.
When building and starting the latest source of the v3.0.x branch with more than 256 vs's/ports defined I get: Failed adding event handler for socket: Too many readers I traced this error back to the #define on line 37 of event.c: #define FR_EV_MAX_FDS (256)
Are there are any serious repercussions of increasing this and running more than 256 vs's/listening ports?
Not really. Thought with 256 sockets open, something like kqueue or epoll becomes a *lot* more efficient. Alan DeKok.
Alan DeKok <aland@deployingradius.com> wrote: >> I'd like to run a FR with more than 256 virtual servers/listening >> ports. > Why? That's a bit unusual. Yeah, why not just run more than once instance? I think that would also spread the load across more cores, and reduce memory contention if your machine is has any NUMA-like properties (many machines with more than one CPU socket tends to have penalties for accessing memory not directly attached to that CPU). -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works | network architect [ ] mcr@sandelman.ca http://www.sandelman.ca/ | ruby on rails [
On 9 Nov 2014, at 20:50, Michael Richardson <mcr@sandelman.ca> wrote:
Alan DeKok <aland@deployingradius.com> wrote:
I'd like to run a FR with more than 256 virtual servers/listening ports.
Why? That's a bit unusual.
Yeah, why not just run more than once instance? I think that would also spread the load across more cores, and reduce memory contention if your machine is has any NUMA-like properties (many machines with more than one CPU socket tends to have penalties for accessing memory not directly attached to that CPU).
Possibly, but you would have to force the processor affinity of each radiusd process so that all the worker threads ran on cores in the same zone as the master thread. From what i've observed the Linux kernel isn't smart enough to do this on its own, else the default weighting that control process affinity don't usually result in it happening without manual intervention. The larger performance increase in this case would come from parallelising socket reads and packet validation, which is currently only done by a single 'master' thread. Anyway, the kqueue stuff Alan just pushed should help with the inefficiencies of selecting over large numbers of FDs, just need to switch to OSX or *BSD :) Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
Kev Pearce wrote:
I'd like to run a FR with more than 256 virtual servers/listening ports. Why? That's a bit unusual.
It's for multi-tenancy by port.
That isn't necessary. See the "virtual_server" example in clients.conf. It is intended for this EXACT purpose. Each tenant has their own RADIUS client. Each RADIUS client points to a different virtual server. You don't need different "listen" sections for each client. In general, when your solution is complicated and running into large limits (that no one else runs into), your solution is sub-optimal. Alan DeKok.
I'd like to run a FR with more than 256 virtual servers/listening ports. Why? That's a bit unusual.
It's for multi-tenancy by port.
That isn't necessary. See the "virtual_server" example in clients.conf. It is intended for this EXACT purpose.
My multi-tenancy needs to work like this: Each customer gets their own udp port to authenticate to. They have their own shared secret that applies to all their clients (i.e any client that connects to their udp port). They can have any client IP address authenticate, use their own shared secret and authenticate their own user list (in mysql). What I have built is a virtual server per customer. Each VS has its own listen port and has its own shared secret (using dynamic clients read from mysql, looked up by a udp port field). The shared secret is setup using 128.0.0.0/1 and 0.0.0.0/1 CIDR hosts read by dynamic clients (as per previous posts I made). This then allows any client to auth to that udp port with that customers shared secret. If there is any way to isolate 'lots' of customer so they each have their own port and shared secret without lots of seperate virtual servers I'd definitely be interested. I use the port as part of my sql queries to authorise users and all this works absolutely great. Cheers Kev/.
Kev Pearce wrote:
My multi-tenancy needs to work like this:
Each customer gets their own udp port to authenticate to.
Yes, we got that. You've said that before. You should talk about the PROBLEM, not the IMPLEMENTATION you've created. Your solution may very well be wrong.
They have their own shared secret that applies to all their clients (i.e any client that connects to their udp port).
Yes, they each connect to their own UDP port. Why? You don't seem to be clear on that.
They can have any client IP address authenticate, use their own shared secret and authenticate their own user list (in mysql).
Did you read my previous message? You can create unique virtual servers per client.
What I have built is a virtual server per customer.
Customer? Tenant? You're using vague terminology. Be specific. Who is doing what and why?
Each VS has its own listen port and has its own shared secret (using dynamic clients read from mysql, looked up by a udp port field).
You're describing your implementation. NOT the problem.
The shared secret is setup using 128.0.0.0/1 and 0.0.0.0/1 CIDR hosts read by dynamic clients (as per previous posts I made). This then allows any client to auth to that udp port with that customers shared secret.
Could you explain why you need to say 5-6 tunes "EACH CUSTOMER GETS THEIR OWN UDP PORT"? It's unnecessary and annoying. It means you either think we're idiots and don't understand it, OR you're stuck on your implementation, and REALLY WANT TO TALK ABOUT IT.
If there is any way to isolate 'lots' of customer so they each have their own port and shared secret without lots of seperate virtual servers I'd definitely be interested.
No. I've explained that you DO NOT need to use separate ports. Instead of listening and trying to understand, you've asked again how to use multiple ports.
I use the port as part of my sql queries to authorise users and all this works absolutely great.
Except that you've got to modify the server source because your design is bad. i.e. your design is crap. It's overly complicated, and unnecessary. Can you answer this one simple question: - do multiple customers use the same the RADIUS client IP? - YES - NO Which is it? My guess is NO. In which case your design is massively over-complicated and unnecessary. Are you prepared to think about it? i.e. pay attention, and have a response which *doesn't* say 6 times "I HAVE MULTIPLE UDP PORTS" ? It's annoying. You're asking for help and then ignoring the answers. Read and learn, or stop asking questions. Alan DeKok.
On 13 Nov 2014, at 09:52, Kev Pearce <email.me@kevp.com> wrote:
I'd like to run a FR with more than 256 virtual servers/listening ports. Why? That's a bit unusual.
It's for multi-tenancy by port.
That isn't necessary. See the "virtual_server" example in clients.conf. It is intended for this EXACT purpose.
My multi-tenancy needs to work like this:
Each customer gets their own udp port to authenticate to. They have their own shared secret that applies to all their clients (i.e any client that connects to their udp port). They can have any client IP address authenticate, use their own shared secret and authenticate their own user list (in mysql).
What I have built is a virtual server per customer. Each VS has its own listen port and has its own shared secret (using dynamic clients read from mysql, looked up by a udp port field). The shared secret is setup using 128.0.0.0/1 and 0.0.0.0/1 CIDR hosts read by dynamic clients (as per previous posts I made). This then allows any client to auth to that udp port with that customers shared secret.
If there is any way to isolate 'lots' of customer so they each have their own port and shared secret without lots of seperate virtual servers I'd definitely be interested.
No, but you can do something similar with client IP ranges. If you're set on using ports, realise that FreeRADIUS uses select() to determine when there's data available to read. select does not scale to thousands of file descriptors, in fact it doesn't perform very will with anything over 30. There's also usually a compiled in upper boundary FD_SETSIZE which is 1024 on most systems. As this is undoubtedly a commercial enterprise perhaps you'd like to sponsor the development work needed to remove the current file descriptor limitation. Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
participants (4)
-
Alan DeKok -
Arran Cudbard-Bell -
Kev Pearce -
Michael Richardson