Spent some time poking at it in v3.1.x. It seems to work OK with multiple threads now, and i've reworked the code to use separate interpreters for each module instance for cleanliness. I don't see any of the crashes previously observed when using multiple rlm_python instances. If heavy users of python in v3.0.x could try v3.1.x it would be appreciated. As v3.0.x was marked as THREAD_UNSAFE, you should see substantial performance improvements in v3.1.x. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS Development Team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On 06/04/16 15:31, Arran Cudbard-Bell wrote:
Spent some time poking at it in v3.1.x.
It seems to work OK with multiple threads now, and i've reworked the code to use separate interpreters for each module instance for cleanliness.
This is a bit of a minefield. C extension modules which are using the "simplified" GIL API will deadlock if accessed in anything other than the first interpreter, IIRC. I also think you can run into problems with C extension modules and unloading if you destroy interpreters. The mod_wsgi docs contain a few of the pitfalls: https://code.google.com/archive/p/modwsgi/wikis/ApplicationIssues.wiki It'll probably be fine for pure python, but it's very chancy with any C extension modules (which are the most interesting bits). Cheers, Phil
On 6 Apr 2016, at 10:57, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
On 06/04/16 15:31, Arran Cudbard-Bell wrote:
Spent some time poking at it in v3.1.x.
It seems to work OK with multiple threads now, and i've reworked the code to use separate interpreters for each module instance for cleanliness.
This is a bit of a minefield.
C extension modules which are using the "simplified" GIL API will deadlock if accessed in anything other than the first interpreter, IIRC.
Wow...
I also think you can run into problems with C extension modules and unloading if you destroy interpreters.
The mod_wsgi docs contain a few of the pitfalls:
https://code.google.com/archive/p/modwsgi/wikis/ApplicationIssues.wiki
Ew OK.
It'll probably be fine for pure python, but it's very chancy with any C extension modules (which are the most interesting bits).
Do you have an example of using a C extension module that's likely to crash/deadlock? Thanks, -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS Development Team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On Apr 6, 2016, at 10:57 AM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
C extension modules which are using the "simplified" GIL API will deadlock if accessed in anything other than the first interpreter, IIRC.
Then we document it. And anyone who doesn't read the documentation gets what they deserve Sorry, but life isn't perfect.
It'll probably be fine for pure python, but it's very chancy with any C extension modules (which are the most interesting bits).
Alas, life isn't perfect. Alan DeKok.
On 6 Apr 2016, at 11:30, Alan DeKok <aland@deployingradius.com> wrote:
On Apr 6, 2016, at 10:57 AM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
C extension modules which are using the "simplified" GIL API will deadlock if accessed in anything other than the first interpreter, IIRC.
Then we document it. And anyone who doesn't read the documentation gets what they deserve
mod_python includes an option to use the first interpreter context, so that's probably a knob we should add too. Python embedding really sucks. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS Development Team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
C extension modules which are using the "simplified" GIL API will deadlock if accessed in anything other than the first interpreter, IIRC.
Then we document it. And anyone who doesn't read the documentation gets what they deserve
mod_python includes an option to use the first interpreter context, so that's probably a knob we should add too.
I suspect this is the best option under the circumstances. :-/ Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800.
Arran Cudbard-Bell wrote:
On 6 Apr 2016, at 11:30, Alan DeKok <aland@deployingradius.com> wrote:
On Apr 6, 2016, at 10:57 AM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
C extension modules which are using the "simplified" GIL API will deadlock if accessed in anything other than the first interpreter, IIRC.
Then we document it. And anyone who doesn't read the documentation gets what they deserve
mod_python includes an option to use the first interpreter context, so that's probably a knob we should add too.
Hmm, I vaguely remember that using mod_python is highly deprecated since many years. So I'm not sure whether you view it as a good coding example.
Python embedding really sucks.
Maybe it's better to not embed any script language interpreter at all and rather encourage people to implement decent external demons listening on a Unix domain socket where FreeRADIUS sends the requests to. Ciao, Michael.
On 9 Apr 2016, at 10:09, Michael Ströder <michael@stroeder.com> wrote:
Arran Cudbard-Bell wrote:
On 6 Apr 2016, at 11:30, Alan DeKok <aland@deployingradius.com> wrote:
On Apr 6, 2016, at 10:57 AM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
C extension modules which are using the "simplified" GIL API will deadlock if accessed in anything other than the first interpreter, IIRC.
Then we document it. And anyone who doesn't read the documentation gets what they deserve
mod_python includes an option to use the first interpreter context, so that's probably a knob we should add too.
Hmm, I vaguely remember that using mod_python is highly deprecated since many years. So I'm not sure whether you view it as a good coding example.
Python embedding really sucks.
Maybe it's better to not embed any script language interpreter at all and rather encourage people to implement decent external demons listening on a Unix domain socket where FreeRADIUS sends the requests to.
For Python, yes. I'm surprised there's not a python-on-a-stick project already. For other languages (like Lua), you can run multiple instances of the interpreter and manage your own shared storage if the instances need to communicate (which they usually don't). -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS Development Team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On 06/04/16 16:30, Alan DeKok wrote:
On Apr 6, 2016, at 10:57 AM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
C extension modules which are using the "simplified" GIL API will deadlock if accessed in anything other than the first interpreter, IIRC.
Then we document it. And anyone who doesn't read the documentation gets what they deserve
Sorry, but life isn't perfect.
No need to apologise - I totally agree there's no perfect solution. FWIW I don't personally consider Python a suitable language for embedding, and I like Python a *lot*. As such, I wouldn't use rlm_python.
participants (5)
-
Alan DeKok -
Arran Cudbard-Bell -
Michael Ströder -
Phil Mayers -
Stefan Paetow