Hi, I've been working on the belief that each separate thread in FreeRADIUS has a separate instantiation of each module, e.g. a new thread calls mod_instantiate at startup and mod_detach at finish. Therefore an instantiation of the module is all within one thread. I'm now thinking about this further, and believe I'm wrong...? Reason - mschap code: I'm grabbing winbind context in mod_instantiate (assuming that is in one thread), using the context in mod_authenticate, then freeing it in mod_detach. That runs fine for a while (all testing was good...) then exhibits problems that I was seeing with a non-threaded library. As I know the library is now thread-safe because of the use of context, I'm guessing that the module instantiation isn't separated per thread? If two threads use the same context at the same time (e.g. are both running with the same module instantiation simultaneously) then obviously Bad Things are going to happen. If this is the case then I guess I'll need to add a connection pool to make sure the contexts aren't shared... nothing major, but will explain what I'm seeing after running under load for ~20 minutes. Cheers, Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On Mon, Mar 23, 2015 at 09:40:55PM +0000, Matthew Newton wrote:
I've been working on the belief that each separate thread in FreeRADIUS has a separate instantiation of each module, e.g. a new thread calls mod_instantiate at startup and mod_detach at finish. Therefore an instantiation of the module is all within one thread.
I'm now thinking about this further, and believe I'm wrong...?
Thinking about it more, and checking the code, yes I'm wrong. Just not sure why I even thought that in the first place. Been staring at this for too long I guess. I'll add a connection pool to the mschap winbind code so it doesn't have threading issues... as always annoying the issues only show up under load, not in normal testing. Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On 23 Mar 2015, at 17:51, Matthew Newton <mcn4@LEICESTER.AC.UK> wrote:
On Mon, Mar 23, 2015 at 09:40:55PM +0000, Matthew Newton wrote:
I've been working on the belief that each separate thread in FreeRADIUS has a separate instantiation of each module, e.g. a new thread calls mod_instantiate at startup and mod_detach at finish. Therefore an instantiation of the module is all within one thread.
I'm now thinking about this further, and believe I'm wrong...?
Thinking about it more, and checking the code, yes I'm wrong. Just not sure why I even thought that in the first place. Been staring at this for too long I guess.
I'll add a connection pool to the mschap winbind code so it doesn't have threading issues... as always annoying the issues only show up under load, not in normal testing.
I was wondering about that... I figured you'd done some TLS magic in the samba patches, but I guess not :) Yes, you'll need a connection pool. There's no per module/thread instantiators or destructors. It's also the reason why you never parent memory allocated when processing a request, from the context of the module instance. Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On Mon, Mar 23, 2015 at 06:08:24PM -0400, Arran Cudbard-Bell wrote:
On 23 Mar 2015, at 17:51, Matthew Newton <mcn4@LEICESTER.AC.UK> wrote:
Thinking about it more, and checking the code, yes I'm wrong. Just not sure why I even thought that in the first place. Been staring at this for too long I guess.
I'll add a connection pool to the mschap winbind code so it doesn't have threading issues... as always annoying the issues only show up under load, not in normal testing.
I was wondering about that...
I figured you'd done some TLS magic in the samba patches, but I guess not :)
I did originally, but the code wasn't great; there were cases where the library could be dlclose'd and threads still have data structures around that wouldn't have called their destructors... yet. This is the correct version.
Yes, you'll need a connection pool.
There's no per module/thread instantiators or destructors.
Feeling rather silly - if asked this I would have told anyone that, but for some reason it didn't come to mind. Oh well. Thread pool is trivial and about 70% done now.
It's also the reason why you never parent memory allocated when processing a request, from the context of the module instance.
Yup, that makes sense. Thanks for rebooting my sanity... Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On Mon, Mar 23, 2015 at 06:08:24PM -0400, Arran Cudbard-Bell wrote:
Yes, you'll need a connection pool.
Just send new PR to add this. I've just fired ~25000 mschap auths through the server (backed by a Samba 4.2 AD DC) in a couple of minutes, and all held up fine, so I'm hoping this time it's right! Cheers, Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On Mon, Mar 23, 2015 at 11:25:30PM +0000, Matthew Newton wrote:
I've just fired ~25000 mschap auths through the server (backed by a Samba 4.2 AD DC) in a couple of minutes, and all held up fine, so I'm hoping this time it's right!
Just for the record - have now had live traffic (all of it...) through 3.0.x HEAD using this for the last two hours, and all looking good. Load isn't incredibly high as we're heading towards end of term (peak ~50 auths/sec), but no hiccups so far. I did initially have the server pausing for a minute or so every now and then, but it turns out the FR3 doesn't like it if the FR2 radmin is pointed at it - it locks the server solid until radmin quits... FR3 radmin seems fine. I may do another small PR to update some of the error messages to make them more helpful for people (yeah, I got confused by my own errors!) Cheers, Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On Mar 24, 2015, at 12:12 PM, Matthew Newton <mcn4@leicester.ac.uk> wrote:
Just for the record - have now had live traffic (all of it...) through 3.0.x HEAD using this for the last two hours, and all looking good. Load isn't incredibly high as we're heading towards end of term (peak ~50 auths/sec), but no hiccups so far.
That’s good.
I did initially have the server pausing for a minute or so every now and then, but it turns out the FR3 doesn't like it if the FR2 radmin is pointed at it - it locks the server solid until radmin quits... FR3 radmin seems fine.
Hmm… it shouldn’t lock. And yes, the radmin in 3.0.7 is very different from earlier versions. Much, much, better.
I may do another small PR to update some of the error messages to make them more helpful for people (yeah, I got confused by my own errors!)
OK, thanks. Alan DeKok.
participants (3)
-
Alan DeKok -
Arran Cudbard-Bell -
Matthew Newton