Python Module Threading Issue
I've attached radius.log, a simple example Python program, and the mods-enabled configs for two instances: mods-enabled/python1 mods-enabled/python2 that when invoked from the authorization sections of: sites-enabled/default sites-enabled/inner-tunnel result in a freeradius server crash. This only happens for threaded tests, for example: ./radperf -s -p 10 -D stats.out -f test-small.csv 127.0.0.1 auth testing123 The crash does not occur if the test client doesn't thread requests or if I run 'freeradius -X' with console output. Depending upon the execution, I see these fatal errors: Fatal Python error: GC object already tracked Fatal Python error: PyEval_SaveThread: NULL tstate Thoughts? Thanks, Gary
Hi Alan, Running 3.0.16. Will try upgrading and report back. Gary On Mon, Aug 13, 2018 at 11:44 AM Alan DeKok <aland@deployingradius.com> wrote:
On Aug 13, 2018, at 1:36 PM, Gary Gwin <garygwin@gmail.com> wrote:
I've attached radius.log, a simple example Python program, and the mods-enabled configs for two instances:
Which version are you running? 3.0.17 should be fine.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Hi Alan, I see the same issues with 3.0.17: sudo grep Fatal /var/log/freeradius/radius.log Fatal Python error: PyEval_SaveThread: NULL tstate freeradius -v radiusd: FreeRADIUS Version 3.0.17 (git #cb7c6d3), for host x86_64-pc-linux-gnu Cheers, Gary On Mon, Aug 13, 2018 at 3:13 PM Gary Gwin <garygwin@gmail.com> wrote:
Hi Alan,
Running 3.0.16. Will try upgrading and report back.
Gary
On Mon, Aug 13, 2018 at 11:44 AM Alan DeKok <aland@deployingradius.com> wrote:
On Aug 13, 2018, at 1:36 PM, Gary Gwin <garygwin@gmail.com> wrote:
I've attached radius.log, a simple example Python program, and the mods-enabled configs for two instances:
Which version are you running? 3.0.17 should be fine.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On 13 Aug 2018, at 18:26, Gary Gwin <garygwin@gmail.com> wrote:
Hi Alan,
I see the same issues with 3.0.17:
sudo grep Fatal /var/log/freeradius/radius.log Fatal Python error: PyEval_SaveThread: NULL tstate
freeradius -v radiusd: FreeRADIUS Version 3.0.17 (git #cb7c6d3), for host x86_64-pc-linux-gnu
Could you try master branch (it's not config compatible unfortunately), to see if the issue is fixed there. If it's not, could you submit a pull request for an automated test case so it can be reproduced locally. There are multiple test cases in there already for python so it should be easy to follow one of the existing examples. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS Development Team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
Hi Arran, I built 3.0.18 from the master branch and reproduced the same concurrency issue. The issue manifests under a small load with config of multiple modules in multiple sites. I didn't know how to provide the full context here: https://github.com/FreeRADIUS/freeradius-server/tree/master/src/tests/module... I minimized my test case and changed the naming conventions to match. I've attached: 1) modules.conf - which can probably be appended to modules.conf in the repro 2) mod7.py - minimized Python code to use 3) test-small.csv - test accounts to use with radperf To configure in the default raddb config: 1) sites-enabed/default - add 'mod7_default' before chap in authorize 2) sites-enabled/inner-tunnel - add 'mod7_inner_tunnel' before chap in authorize 3) mods-enabled - I'm sure you know what to do better than me Run: 1) ./radperf -s -p 10 -D stats.out -f test-small.csv 127.0.0.1 auth testing123 Run radiusd as a service. The issue does not manifest running "radiusd -X". Please provide more instructions if you want me to do more to integrate into the Python test environment. Thanks, Gary On Tue, Aug 14, 2018 at 7:56 AM Arran Cudbard-Bell < a.cudbardb@freeradius.org> wrote:
On 13 Aug 2018, at 18:26, Gary Gwin <garygwin@gmail.com> wrote:
Hi Alan,
I see the same issues with 3.0.17:
sudo grep Fatal /var/log/freeradius/radius.log Fatal Python error: PyEval_SaveThread: NULL tstate
freeradius -v radiusd: FreeRADIUS Version 3.0.17 (git #cb7c6d3), for host x86_64-pc-linux-gnu
Could you try master branch (it's not config compatible unfortunately), to see if the issue is fixed there.
If it's not, could you submit a pull request for an automated test case so it can be reproduced locally.
There are multiple test cases in there already for python so it should be easy to follow one of the existing examples.
-Arran
Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS Development Team
FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Aug 16, 2018, at 8:07 PM, Gary Gwin <garygwin@gmail.com> wrote:
I built 3.0.18 from the master branch and reproduced the same concurrency issue. The issue manifests under a small load with config of multiple modules in multiple sites. I didn't know how to provide the full context here:
I think it's difficult to make Python fully thread-safe: https://docs.python.org/2/c-api/init.html ... Note Calling system I/O functions is the most common use case for releasing the GIL, but it can also be useful before calling long-running computations which don’t need access to Python objects, such as compression or cryptographic functions operating over memory buffers. For example, the standard zlib and hashlib modules release the GIL when compressing or hashing data. ... That isn't good. The short-term fix is to edit the module. Change the flag at the end from: .type = RLM_TYPE_THREAD_SAFE, to .type = RLM_TYPE_THREAD_UNSAFE, Recompile, re-install, and it will work. It will also be slower. But.. slow and correct is better than fast and wrong. And for most circumstances (<1K pps) the speed difference shouldn't be noticeable. Alan DeKok.
participants (3)
-
Alan DeKok -
Arran Cudbard-Bell -
Gary Gwin