Thanks for your input. Yes I’ve checked it. The request rate is nearly zero ( 10 Authentications every 2 minutes ) and there is no database connection at all. The issue get’s resolved when we disable `check_client_connections` in the TLS server. We use `check_client_connections` to do some certificate revocations checks with the python module and I’ve also checked that this script is not blocking something and there is no open process which is blocking further processing. And there was an issue (where this thread was opened with) that the connection was accepted but the socket just don’t reads the data from it which is why I assumed that the issue got back with the changes for BLAST RADIUS. Also there is currently an open issue that the server segfaults with `check_client_connections` enabled https://github.com/FreeRADIUS/freeradius-server/issues/5480. I assume that not many out there use this functionality which is why no other reported it. The only difference now is that it’s not a general problem but happens after a while. — Lineconnect
On 18. Dec 2024, at 16:28, Winfield, Alister (Senior Solutions Architect) via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote:
I assume you have checked some obvious things:
1. Request Rate isn’t hugely spiky, but the average rate is completely within the capability of the server to respond. If it is there is not much you can do except ensure the backlog queues are big enough to handle it without the max latency getting > request timeout. 2. Response Rate = Request Rate. Obvious really if response rate is the same you likely haven’t got an immediate issue but perhaps time to start optimising things. In absence of (1) queues are the sign that you are starting to get close to the ‘hocky sticky’ in a throughput vs latency graph. 3. Bad Clients (many ignore the random geometric back-off so if you reject they come back immediately again and again). If that’s the case, it may be useful to cache rejects so you can reduce the ‘cost’ of those requests. Also look to see if the sending device can limit such request hammering at source rather than ‘causing’ issues.
Now…
My educated guess there is a database or similar that’s blocking threads sooner of later that ends up with the input side queuing data.
Note that THROUGHPUT is almost totally dominated by LATENCY of a request.
Assuming all resources have capacity spare (eg CPU, NIC, etc) then throughput is simply (1/latency per request)*threads. This starts to become non-linear quite quickly because you nearly always become contended on something like I/O kernel calls. Graphed this will look like a hockey stick (graph latency against request rate), first linear growth with tiny latency increases until you start hitting the limits. At this point a tiny increase in request/s causes a vast change in latency. With limits on queueing or timeouts this can end up with total meltdown where response rates actually drop and things don’t get better.
From: Freeradius-Users <freeradius-users-bounces+alister.winfield=sky.uk@lists.freeradius.org> on behalf of nabble@felix.world <nabble@felix.world> Date: Wednesday, 18 December 2024 at 11:35 To: FreeRadius users mailing list <freeradius-users@lists.freeradius.org> Subject: [EXTERNAL] Re: No auth requests through TLS tunnel if connection was checked Hi there,
We’ve some productive instances which experience the same issue but only after a certain time. Therefore I’m currently not able to reproduce it reliably but it’s visible that the Recv-Q is getting bigger and bigger.
root@radius-d6674b64-xt6x4:/# netstat -ap Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 9 0 0.0.0.0:2083 0.0.0.0:* LISTEN 1/radiusd tcp 85 0 radius-d6674b64-xt:2083 10-244-25-182.tra:35878 CLOSE_WAIT 1/radiusd tcp 85 0 radius-d6674b64-xt:2083 10-244-25-182.tra:39682 CLOSE_WAIT 1/radiusd tcp 236288 0 radius-d6674b64-xt:2083 10-244-25-182.tra:37974 ESTABLISHED 1/radiusd tcp 442 0 radius-d6674b64-xt:2083 10-244-25-182.tra:40226 CLOSE_WAIT 1/radiusd tcp 298 0 radius-d6674b64-xt:2083 10-244-25-182.tra:39460 CLOSE_WAIT - tcp 326 0 radius-d6674b64-xt:2083 10-244-25-182.tra:33402 CLOSE_WAIT - tcp 6404 0 radius-d6674b64-xt:2083 10-244-25-182.tra:33096 CLOSE_WAIT 1/radiusd tcp 85 0 radius-d6674b64-xt:2083 10-244-25-182.tra:51816 CLOSE_WAIT 1/radiusd tcp 85 0 radius-d6674b64-xt:2083 10-244-25-182.tra:39432 CLOSE_WAIT 1/radiusd tcp 85 0 radius-d6674b64-xt:2083 10-244-25-182.tra:59936 CLOSE_WAIT 1/radiusd tcp 85 0 radius-d6674b64-xt:2083 10-244-25-182.tra:58720 CLOSE_WAIT 1/radiusd tcp 199 0 radius-d6674b64-xt:2083 10-244-25-182.tra:33400 CLOSE_WAIT - tcp 326 0 radius-d6674b64-xt:2083 10-244-25-182.tra:59178 CLOSE_WAIT - tcp 8023 0 radius-d6674b64-xt:2083 10-244-25-182.tra:51882 CLOSE_WAIT 1/radiusd tcp 85 0 radius-d6674b64-xt:2083 10-244-25-182.tra:57698 CLOSE_WAIT 1/radiusd tcp 6765 0 radius-d6674b64-xt:2083 10-244-25-182.tra:37272 CLOSE_WAIT 1/radiusd tcp 258034 0 radius-d6674b64-xt:2083 10-244-25-182.tra:55148 ESTABLISHED 1/radiusd tcp 247 0 radius-d6674b64-xt:2083 10-244-25-182.tra:58998 CLOSE_WAIT - tcp 85 0 radius-d6674b64-xt:2083 10-244-25-182.tra:49494 CLOSE_WAIT 1/radiusd tcp 1 0 radius-d6674b64-x:38118 20.50.2.37:http CLOSE_WAIT 1/radiusd tcp 5736 0 radius-d6674b64-xt:2083 10-244-25-182.tra:39550 CLOSE_WAIT 1/radiusd tcp 326 0 radius-d6674b64-xt:2083 10-244-25-182.tra:57830 CLOSE_WAIT - tcp 19335 0 radius-d6674b64-xt:2083 10-244-25-182.tra:57654 CLOSE_WAIT 1/radiusd tcp 994 0 radius-d6674b64-xt:2083 10-244-25-182.tra:35026 CLOSE_WAIT 1/radiusd tcp 85 0 radius-d6674b64-xt:2083 10-244-25-182.tra:47572 CLOSE_WAIT 1/radiusd tcp 326 0 radius-d6674b64-xt:2083 10-244-25-182.tra:49282 CLOSE_WAIT - tcp 326 0 radius-d6674b64-xt:2083 10-244-25-182.tra:59172 CLOSE_WAIT - tcp 85 0 radius-d6674b64-xt:2083 10-244-25-182.tra:35852 CLOSE_WAIT 1/radiusd udp 0 0 localhost:18120 0.0.0.0:* 1/radiusd Active UNIX domain sockets (servers and established) Proto RefCnt Flags Type State I-Node PID/Program name Path unix 3 [ ] STREAM CONNECTED 498007517 1/radiusd unix 3 [ ] STREAM CONNECTED 498007516 1/radiusd
I know it's cheeky to ask this without a reproducible test, but the last time there was an obvious error, so I wanted to ask if you could look at it? I fully understand if not.
— Lineconnect
On 12. Apr 2024, at 18:05, Alan DeKok <aland@deployingradius.com> wrote:
Thanks. I've pushed the patch, and another one-line fix which stops that error.
On Apr 12, 2024, at 11:36 AM, nabble@felix.world wrote:
That worked! Now the requests are coming through. Thanks!
One thing to mention is that every time, the radsec client connects, there is one new error.
hread 4 got semaphore Thread 4 handling request 0, (1 handled so far) (0) (TLS) Checking connection to see if it is authorized. (0) # Executing group from file /usr/local/etc/raddb/sites-enabled/default (0) Autz-Type New-TLS-Connection { (0) [ok] = ok (0) } # Autz-Type New-TLS-Connection = ok (0) (TLS) Connection is authorized (0) ERROR: Failed signing packet: ERROR: RADIUS packets must be assigned an Id (0) Sent Access-Accept Id 4294967295 from 0.0.0.0:2083 to 192.168.215.1:32993 length 20 (0) Finished request Thread 4 waiting to be assigned a request Waking up in 0.2 seconds. Waking up in 4.6 seconds.
- Lineconnect
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html -------------------------------------------------------------------- This email is from an external source. Please do not open attachments or click links from an unknown or suspicious origin. Phishing attempts can be reported by using the report message button in Outlook or sending them as an attachment to phishing@sky.uk. Thank you -------------------------------------------------------------------- Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. SKY and the SKY marks are trademarks of Sky Limited and Sky International AG and are used under licence.
Sky UK Limited (Registration No. 2906991), Sky-In-Home Service Limited (Registration No. 2067075), Sky Subscribers Services Limited (Registration No. 2340150) and Sky CP Limited (Registration No. 9513259) are direct or indirect subsidiaries of Sky Limited (Registration No. 2247735). All of the companies mentioned in this paragraph are incorporated in England and Wales and share the same registered office at Grant Way, Isleworth, Middlesex TW7 5QD - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html