On 4 Oct 2013, at 10:37, Arran Cudbard-Bell <a.cudbardb@freeradius.org> wrote:

On 4 Oct 2013, at 10:19, Alex Sharaz <alex.sharaz@york.ac.uk> wrote:

...

Hi, Yesterday caught an email about the release of FR 2.2.2 on Monday to "fix a proxy problem". As I've just migrated 2 of my servers from 2.2.0 to 2.2.1 the sudden release of 2.2.2 sounds important. What does 2.2.2 fix?

Issue with workers not marking requests are being done correctly. Workers appear to get hung, leading to issues. I would upgrade to latest 2.x.x HEAD to avoid disruption if the proxying functionality is heavily used.

Eek! that's what I'm seeing on our outward facing eduroam servers that do nothing but proxy stuff. Time to fix it methinks

There were also quite a few issues with the policy language.

-Arran

Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS Development Team

- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

Hmm like these then? Fri Oct 4 11:24:12 2013 : Info: WARNING: Child is hung for request 17630 in com ponent <core> module <thread>. Fri Oct 4 11:24:13 2013 : Info: WARNING: Child is hung for request 17635 in com ponent <core> module <thread>. Fri Oct 4 11:24:14 2013 : Info: WARNING: Child is hung for request 17634 in com ponent <core> module <thread>. Fri Oct 4 11:24:17 2013 : Info: WARNING: Child is hung for request 17636 in com ponent <core> module <thread>. Fri Oct 4 11:24:44 2013 : Info: WARNING: Child is hung for request 17633 in com ponent <core> module <thread>. Fri Oct 4 11:24:52 2013 : Info: WARNING: Child is hung for request 17635 in com ponent <core> module <thread>. Fri Oct 4 11:24:53 2013 : Info: WARNING: Child is hung for request 17634 in com ponent <core> module <thread>. Fri Oct 4 11:24:55 2013 : Info: WARNING: Child is hung for request 17636 in com ponent <core> module <thread>. Reverted back to 2.2.0 as I never saw these errors with it Rgs A On 4 Oct 2013, at 11:53, A.L.M.Buxey@lboro.ac.uk wrote:

Hi,

a couple of logic issues that meant case/switch and if() worked different to 2.x - thats been fixed. ..and an issue if your server does a lot of proxying work - in which worker threads arent dealt with properly - your log file will be full of <core> and <module> messages if you are being hit. this *MIGHT* be fixed in HEAD. we are testing at the moment (looking good). if you arent doing the former and not hit by the latter you dont need to worry.

alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

Hi early report :( 2.2.2 HEAD still showing: Fri Oct 4 13:20:43 2013 : Info: WARNING: Child is hung for request 3767589 in component <core> module <thread>. Fri Oct 4 13:20:45 2013 : Info: WARNING: Child is hung for request 3767589 in component <core> module <thread>. Fri Oct 4 13:20:47 2013 : Info: WARNING: Child is hung for request 3767589 in component <core> module <thread>. Fri Oct 4 13:20:51 2013 : Info: WARNING: Child is hung for request 3767589 in component <core> module <thread>. Fri Oct 4 13:20:52 2013 : Info: WARNING: Child is hung for request 3766906 in component <core> module <thread>. Fri Oct 4 13:20:56 2013 : Info: WARNING: Child is hung for request 3767589 in component <core> module <thread>. Fri Oct 4 13:21:03 2013 : Info: WARNING: Child is hung for request 3767589 in component <core> module <thread>. Fri Oct 4 13:21:15 2013 : Info: WARNING: Child is hung for request 3767589 in component <core> module <thread>. Fri Oct 4 13:21:17 2013 : Info: WARNING: Child is hung for request 3766906 in component <core> module <thread>. Fri Oct 4 13:21:32 2013 : Info: WARNING: Child is hung for request 3767589 in component <core> module <thread>. Fri Oct 4 13:21:57 2013 : Info: WARNING: Child is hung for request 3767589 in component <core> module <thread>. Fri Oct 4 13:27:32 2013 : Info: WARNING: Child is hung for request 3797280 in component <core> module <thread>. Fri Oct 4 13:27:40 2013 : Info: WARNING: Child is hung for request 3797280 in component <core> module <thread>. Fri Oct 4 13:27:51 2013 : Info: WARNING: Child is hung for request 3797280 in component <core> module <thread>. alan

Hi,

More debug output would help. The last patch came from output sent by Stefan. The patch seems to help. But there's an underlying issue which is harder to debug. It looks like a Linux specific IPv6 problem. I don't see any issue with v4.

interesting......the culprit may have been found. put HEAD onto server this afternoon... the logs had plenty of <core> messages but look Sun Oct 6 15:13:55 2013 : Error: WARNING: Unresponsive child for request 1821224, in component <core> module <thread> Sun Oct 6 15:13:56 2013 : Error: WARNING: Unresponsive child for request 1821229, in component <core> module <thread> Sun Oct 6 15:13:56 2013 : Info: WARNING: Child is hung for request 1821224 in component <core> module <thread>. Sun Oct 6 15:13:57 2013 : Info: WARNING: Child is hung for request 1821229 in component <core> module <thread>. Sun Oct 6 15:13:58 2013 : Info: WARNING: Child is hung for request 1821224 in component <core> module <thread>. Sun Oct 6 15:13:58 2013 : Info: WARNING: Child is hung for request 1821229 in component <core> module <thread>. Sun Oct 6 15:14:00 2013 : Info: WARNING: Child is hung for request 1821224 in component <core> module <thread>. Sun Oct 6 15:14:00 2013 : Info: WARNING: Child is hung for request 1821229 in component <core> module <thread>. Sun Oct 6 15:14:03 2013 : Info: WARNING: Child is hung for request 1820598 in component <core> module <thread>. Sun Oct 6 15:14:04 2013 : Info: WARNING: Child is hung for request 1821224 in component <core> module <thread>. Sun Oct 6 15:14:04 2013 : Info: WARNING: Child is hung for request 1821229 in component <core> module <thread>. Sun Oct 6 15:14:09 2013 : Info: WARNING: Child is hung for request 1821224 in component <core> module <thread>. Sun Oct 6 15:14:09 2013 : Info: WARNING: Child is hung for request 1821229 in component <core> module <thread>. Sun Oct 6 15:14:18 2013 : Info: Ready to process requests. no 'bad logs' since that restart logged. clarification/agreement from Stefan or others? alan

Hi,

If everyone's in favor, I'll release 2.2.2 on Monday.

hold request now its monday AM and the load has gone back to higher levels the server is freaking out and freezing witht he last message in the log being Mon Oct 7 07:50:28 2013 : Error: [event.c:2318] Internal sanity check failed (thats it...no other output - the server needs a restart, it doesnt process anything else once it hits this error) alan

Hi,

clarification/agreement from Stefan or others?

tried the newest GIT this morning and the proxy issues were gone. I haven't seen your "Internal sanity check failed" just yet (and am not looking forward to it :-/ ). Stefan

alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

-- Stefan WINTER Ingenieur de Recherche Fondation RESTENA - Réseau Téléinformatique de l'Education Nationale et de la Recherche 6, rue Richard Coudenhove-Kalergi L-1359 Luxembourg Tel: +352 424409 1 Fax: +352 422473 PGP key updated to 4096 Bit RSA - I will encrypt all mails if the recipient's key is known to me http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC0DE6A358A39DC66

On 04/10/13 13:46, Arran Cudbard-Bell wrote:

If I asked particularly nicely, and promised you a beer at the next networkshop we were both in attendance at, would you be willing to try git head?

I'll roll a v2.2.2_rc0 if it sweetens the deal any? It'd just be really good to know that that particular issue was fixed before rolling out 2.2.2 and then finding it was something else and having to roll 2.2.3 a few weeks later.

I for one will be happy to run git head and see what it does. I'm building as we speak. Cheers, Jonathan

Woah! that's getting g to be lots of beer. I'll run it on one of my outward facing servers. Point me at something I can build and run A On 4 Oct 2013, at 14:33, A.L.M.Buxey@lboro.ac.uk wrote:

Hi,

...

If I asked particularly nicely, and promised you a beer at the next networkshop we were both in attendance at, would you be willing to try git head?

I'll take the beer - am running HEAD since last night on one server :-) (as I said to Alan, i'll report at end of day)

alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Some things started acting differently in 2.2.1 compared to previous releases of 2.x 2.2.2 should revert that so things behave the same - so far that seems to be true but we are still seeing stalled module in core messages that we did not see with 2.2.0 alan - -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -----BEGIN PGP SIGNATURE----- Version: APG v1.0.8 iHkEAREIADkFAlJO8E8yHEFsYW4gQnV4ZXkgKEFsYW4gQnV4ZXkpIDxhLmwubS5i dXhleUBsYm9yby5hYy51az4ACgkQobRdvRSkLC4v6wCcDkljo+wc582+s9TDOJEr Zz7YKAoAnjM3sq4jiTOJdOn7sKwLN83aycJh =/vny -----END PGP SIGNATURE-----