On 13/02/14 10:14, Michael Schwartzkopff wrote:

...

If our people move over the campus with ~3.000 smartphones with actvated wifi, request numbers increase when they enter new wi-fi cells and trouble begins: There is barely an auth ok or incorrect in the log but lots of discarding duplicates messages and cpu load is going up to 120 and a higher number of messages like

Yes, lots of people have run into this in the last year or so. Are you using Samba + ntlm_auth? The underlying reasons are uncertain, and I'm not sure anyone has a complete understanding, but relevant issues are: 1. Samba sites: * Older versions of Samba with scale limits * AD RPC pipe is, by design, synchronous and head-of-line blocking, so 1 slow/failed auth will block everyone else * Slow AD controllers can trigger the above too 4. Very short EAP timeouts on newer client devices 5. Some NASes e.g. Cisco lightweight wireless use a single UDP source port, so at most 256 requests can be in-flight at any given time 6. Newer client devices staying on the wireless in sleep mode 7. Lack of fast roaming 8. Exhaustion of the thread pool in FreeRADIUS when samba/ldap/whatever slows down ...and more. If you can give a bit more detail about how you are doing your authentication, I can suggest some pointers. There's a bunch of discussion on the list around October time. Probably most relevant, if you are using Samba: 1. Upgrade to 2.2.3, which has a configurable, sensible timeout for ntlm_auth 2. Upgrade to Samba 3.6.x and set "winbind max domain connections = 12" 3. Ensure your AD controllers are responding in a timely fashion by starting a long-running tcpdump ring-buffer capture and post-processing it with tshark to extract MS-RPC PDU IDs & response times. I can give more info on this if you need.

On 13/02/14 17:03, Anja Ruckdaeschel wrote:

Tests with exact one ldap server showed no differences. Ldap response times went e.g. from 0.1 s to 0.2 s when the problem appears, but then we do a lot more ldap queries, too. So I think, that´s quite normal.?

0.2 sec for a query is quite slow. Our median time for an AD MSCHAP auth is around 35 milliseconds, and our SQL lookups are closer to 5 milliseconds. If you do the maths, a thread pool of 100 threads can only serve 500 auths/sec if you have 0.2 sec blocking per auth. Scale numbers appropriately for number of lookups, divided by 11/3 (usual ratio of PEAP outer to PEAP inner packets).

Here are our authentication methods: PAP with or without ldap for switches, routers and dial-in servers (cisco, hp, max ascend) and for Juniper VPN controllers (only those devices do accounting) CHAP PAP with or without ldap for switches and routers (cisco, hp) For Wi-Fi: PEAP/MSCHAPv2 and EAP/TTLS-PAP with 350 lancom aps and 1 Colubris Controller with does 2 different SSIDs with ldap for our users at home and those proxied in via the eduroam-project, for which we are service provider and identity provider

Consider splitting these out into separate processes, listening on separate IP/ports. This will help with brief thread pool exhaustion spikes, packet ID space exhaustion from common NAS source IPs and so on.

90% of our radius requests are coming in over Wi-Fi. ldap, sql and so on are only called in inner-tunnel (3 Packets)

You can, and should, reduce that even further. In later server versions, you can do: authorize { eap { ok = return } ... } ...in the inner tunnel as well, which will skip LDAP/SQL lookups for EAP-Identity and EAP-MSCHAP success/fail packets. You only need the LDAP/SQL for EAP-MSCHAP auth. In earlier versions of the server you can do crazy hacks to match the EAP-Message against a regexp and ignore identity / mschap success/fail packets - search the archives. Also investigate rlm_cache. Cache every possible lookup, even if it's just for a few seconds.

I think we have quite a default config, but with a few modules commented out (e.g. unix, radutmp, ... we don´t need) and a few policies in place to reduce unnecessary stuff like querying ldap or sql in default, when its peap.

Short eap timeouts on client devices and sleep mode increased our radius requests for the last two years. There are single users doing up to ~ 1500 login requests per day from one device. But there is nothing we can do about that :-(

Maybe, but you need to be aware of it. Keep an eye on fast roaming, discuss the issues with your wireless vendor, etc.

"Some NASes e.g. Cisco lightweight wireless use a single UDP source port, so at most 256 requests can be in-flight at any given time" could be our problem to because our lancom access points do the same.

Does this mean that when one NAS is sending more than 256 Access-Requests from one port freeradius cannot process one more at that time from this NAS?

It's not a freeradius thing - *no* RADIUS server could. It's a protocol limitation. A given source/dest ip/port 4-tuple can only carry 256 requests in-flight, so no software could handle more. The NAS would have to open more ports. It would be rare - but not impossible, under heavy load - for 256 requests to be outstanding. When that happens, the NAS will either start dropping auth requests, or re-using IDs. In the latter case, that makes things worse, and the backlog can grow until the backend database (SQL, LDAP, AD, whatever) unblocks the thread pool. TBH, I think 0.2 sec is slow for an LDAP lookup; can you run a local replica?

On 02/13/2014 06:22 AM, Arran Cudbard-Bell wrote:

Hmm, hints and huntgroups should probably be removed in 3.1 or 4.0, there's no real advantage to using them, and they're quite poorly implemented. I doubt the code has changed much since v1.x.x.

NAK to removing them in 3.1, you shouldn't remove features in a minor point release. I would suggest postponing such a change to the next major release, 4.0. -- John

Hi,

...

Hmm, hints and huntgroups should probably be removed in 3.1 or 4.0, there's no real advantage to using them, and they're quite poorly implemented. I doubt the code has changed much since v1.x.x.

NAK to removing them in 3.1, you shouldn't remove features in a minor point release. I would suggest postponing such a change to the next major release, 4.0.

3.0 - 3.1 isnt a minor point release - major changes between the 2. 3.0.1 - 3.0.2 is a minor point release (I'll cite the openssl 0.9.x -> 0.9.y patch as previous precendent ;-) ) alan

Dear Alan! Thank you for your reply. Changed max_request_time back to 30. modules/preprocess is unchanged default config and I did not change default or inner-tunnel at that point of authorize section either.... First thing I do with a huntgroup is to call a policy based on if (Huntgroup-Name == "bla") after eap { ok = return } in authorize. #RUAADD: rzur.radiusrejectcheckdefault Every nas has an entry in an include file for clients.conf like: client 172.31.134.10 { secret = *************** shortname = blafasel nastype = other } and an entry per NAS in an include file for huntrgoups like: ap Client-IP-Address == x.x.x.x ap NAS-IP-Address == x.x.x.x the clients.conf and huntgroups file only have one additional line: the include statement for my files. PS: no samba, no AD, no ntlm_auth, no load balancers Kind regards Anja

...

...

Alan DeKok <aland@deployingradius.com> 13.02.2014 15:18 >>> Anja Ruckdaeschel wrote: Having some performance trouble at our campus site with lots of "Discarding Duplicate request" errors.

Our setup is freeradius 2.2.0 with ldap and sql. ; Intel Xeon CPU E5630 2.53GHz 2CPUs with 8 Core (hyperthreading enabled), 24 GB RAM

That should be enough for many, many, EAP users. I've done testing with 500K clients, loaded from the "clients.conf" file. The server uses a lot of RAM, but it works. And performance isn't really different than with using one client.

If our people move over the campus with ~3.000 smartphones with actvated wifi, request numbers increase when they enter new wi-fi cells and trouble begins: There is barely an auth ok or incorrect in the log but lots of discarding duplicates messages and cpu load is going up to 120 and a higher number of messages like

Tue Feb 11 18:22:37 2014 : Error: rlm_eap: No EAP session matching the State variable.

The EAP sessions are timing out.

Tue Feb 11 09:26:50 2014 : Info: WARNING: Module rlm_preprocess became unblocked for request 241193

That's just weird. The preprocess module doesn't do much, so it shouldn't block for any perod of time.

and

Tue Feb 11 09:26:51 2014 : Info: WARNING: Module rlm_eap became unblocked for request 241193

Same here. The EAP module does SSL, but that's fairly quick for new CPUs.

We have a lot of NAS in our environment: there are over 2200 NAS in total in clients.conf, which are 350 heavily used WLAN access points (auth only, no acct) and switches which do a administrative login only every 5 min, and ~ 10 VPN controllers.

That won't cause a problem.

As having so many NAS, preprocess has to do a max of 2200 expansion like

Fri Feb 7 15:41:16 2014 : Debug: [preprocess] expand: %{Client-IP-Address} -> x.x.x.x

in order to determine the allowed client. That are ~26.400 checks for preprocess in one peap/mschap request for example (~12 request-packtes x NAS).

That will cause a problem. The "preprocess" module is intended for small sites. If you're going to do 2200 expansions with it, you're likely using it incorrectly. You should probably put these rules into a database, and do one SQL query. It will be MUCH faster. And why are you doing these expansions for every packet? What is the preprocess module doing? Odds are that you could do the same thing as one SQL query in the post-auth section, or in the inner-tunnel. That would speed up the system a LOT.

Is there some kind of "recommened maximum number of NAS" for one instance of freeradius?

Here is our thread_pool and request config:

max_request_time = 120

That's useless. Most NASes will give up after about 30s. All you're doing is wasting CPU time by setting this to 120.

cleanup_delay = 5 #256 x NAS (By the way: "This should be 256 multiplied by the number of clients. # e.g. With 4 clients, this number should be 1024. " Is client in this contenxt supplicant or NAS?)

It's the NAS. Please post a short description of what you're doing with the preprocess module. Doing 2200 %{Client-IP-Address} expansions is very, very, wrong. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

Anja Ruckdaeschel wrote:

Every nas has an entry in an include file for clients.conf like: client 172.31.134.10 { secret = *************** shortname = blafasel nastype = other }

That's fine.

and an entry per NAS in an include file for huntrgoups like:

ap Client-IP-Address == x.x.x.x ap NAS-IP-Address == x.x.x.x

That's terrible. Don't do that. Ever. Instead, put the client group information into the "client" section: client 172.31.134.10 { secret = *************** shortname = blafasel nastype = other group = ap } Then do policy checking via %{client:group} instead of Huntgroup-Name. It will do the same thing, and will be *enormously* faster. As a general rule, if you're doing tens of checks, it's OK to put them into a flat-text file. If you're doing thousands of checks, you should really put them into a database. Alan DeKok.

On 14 Feb 2014, at 08:00, Alan Buxey <A.L.M.Buxey@LBORO.AC.UK> wrote:

Enormously? ?? Is the huntgroup file dynamically read all the time?

No. It's O(n) as previously stated. Looking something up in a huntgroup means iterating over every member of the linked list (worst case) and evaluating the condition each time to see if it matched.

I thought it was read at server start time and the values then populated into a lookup table....?

There's no hashing or treeing or anything fancy, just a plain linked list. But people get really horny about it like it's *the* ultimate super efficient NAS categorisation mechanism. It's not, it's shit, it needs to go away. Using the client xlat is significantly more efficient. There's also the getclient xlat which allows client lookups using arbitrary IP addresses. Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS Development Team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2

On 20/02/14 13:05, Anja Ruckdaeschel wrote:

Did the change with policies and default and inner-tunnel with "%{client:group}" instead of Huntgroup-Name.

But what is the equivalent Variable to check that in the users file / files module?

You'll have to copy it into a variable before calling files e.g. authorize { ... update request { Tmp-String-0 = "%{client:group}" } files ... } ...then in "users" DEFAULT Tmp-String-0 == "group1" blah, blah You can define your own variable names; see raddb/dictionary and note carefully the comments about attribute numbers.

Thank you for the hints and tipps. A single request is much faster now, since we are not using the huntgroups file any more but client xlat. We are waiting for our students coming back from holidays to check if the fix did it. We are also still talking to our vendor about changing the udp source port when doing more than 256 in-flight requests. Thank you all very much. Anja

...

...

Alan DeKok <aland@deployingradius.com> 14.02.2014 03:17 >>> Anja Ruckdaeschel wrote: Every nas has an entry in an include file for clients.conf like: client 172.31.134.10 { secret = *************** shortname = blafasel nastype = other }

That's fine.

and an entry per NAS in an include file for huntrgoups like:

ap Client-IP-Address == x.x.x.x ap NAS-IP-Address == x.x.x.x

That's terrible. Don't do that. Ever. Instead, put the client group information into the "client" section: client 172.31.134.10 { secret = *************** shortname = blafasel nastype = other group = ap } Then do policy checking via %{client:group} instead of Huntgroup-Name. It will do the same thing, and will be *enormously* faster. As a general rule, if you're doing tens of checks, it's OK to put them into a flat-text file. If you're doing thousands of checks, you should really put them into a database. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

A.L.M.Buxey@lboro.ac.uk wrote:

...

We are also still talking to our vendor about changing the udp source port when doing more than 256 in-flight requests.

Cisco? ;-)

I'm at IETF. I'll point this out to the people in charge. That might help. Alan DeKok.

Hi, no, it´s not Cisco. There are about 350 Lancom APs without controller, so we didn´t notice for years because a it takes a lot for one single acces point reaching the limit of 256. If we are able to persuade them to fix it, I will tell you. Anja

...

...

<A.L.M.Buxey@lboro.ac.uk> 03.03.2014 10:22 >>> Hi,

We are also still talking to our vendor about changing the udp source port when doing more than 256 in-flight requests.

Cisco? ;-) the latest wireless firmware now has a seperate port for auth and accounting (which helps a bit) but the next release migth improve that more.... we've been nagging them for ages now about the single NAS port for ALL the access points (and hence clients behind them). not really a problem with a few dozen APs...but when youre a few thousand APs is the first big thing you notice :( alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

Anja Ruckdaeschel wrote:

Thank you for the hints and tipps. A single request is much faster now, since we are not using the huntgroups file any more but client xlat.

That's good to hear.

We are also still talking to our vendor about changing the udp source port when doing more than 256 in-flight requests.

Most vendors have done that since 1999. There's no excuse for the vendor to be this broken. Alan DeKok.