rlm_python and dynload problem
Aurélien Geron
aureliengeron.freeradius at wifirst.fr
Thu Jan 6 11:26:44 CET 2011
Hi and happy new year to everyone,
In april I wrote the message below about python modules not being able to load dynamic libraries on Debian Lenny. Alan DeKok answered that he would try to fix this in version 2.1.9, and I was very pleased to read the following message in version 2.1.9's Changelog:
"Update configure script for rlm_python to avoid dynamic linking problems on some platforms."
I did not have time to test this ever since, but I just did, and unfortunately, I still have the same problem on Debian Lenny, using the latest FreeRADIUS backport for Lenny (it is based on FreeRADIUS 2.1.10, the Debian package is labelled "2.1.10+dfsg-2~bpo50+1"). Is this a problem that I should report to the package's maintainer for Debian (did he forget some compilation option?), or is it something that should be fixed within FreeRADIUS itself?
Here's FreeRADIUS's debug output:
[...]
Module: Linked to module rlm_python
Module: Instantiating module "python_test" from file /etc/freeradius/modules/python
python_init done
python python_test {
mod_instantiate = "test_freeradius"
func_instantiate = "instantiate"
mod_authorize = "test_freeradius"
func_authorize = "authorize"
mod_authenticate = "test_freeradius"
func_authenticate = "authenticate"
mod_accounting = "test_freeradius"
func_accounting = "accounting"
mod_checksimul = "test_freeradius"
func_checksimul = "checksimul"
mod_pre_proxy = "test_freeradius"
func_pre_proxy = "pre_proxy"
mod_post_proxy = "test_freeradius"
func_post_proxy = "post_proxy"
mod_post_auth = "test_freeradius"
func_post_auth = "post_auth"
mod_recv_coa = "test_freeradius"
func_recv_coa = "recv_coa"
mod_send_coa = "test_freeradius"
func_send_coa = "send_coa"
mod_detach = "test_freeradius"
func_detach = "detach"
}
rlm_python:python_load_function: module 'test_freeradius' is not found
rlm_python:EXCEPT:<type 'exceptions.ImportError'>: /usr/lib/python2.5/lib-dynload/math.so: undefined symbol: PyExc_ValueError
rlm_python:python_load_function: failed to import python function 'test_freeradius.instantiate'
/etc/freeradius/modules/python[69]: Instantiation failed for module "python_test"
/etc/freeradius/sites-enabled/default[205]: Failed to load module "python_test".
/etc/freeradius/sites-enabled/default[62]: Errors parsing authorize section.
The python code looks like this:
import radiusd
import math
def get_radius_attribute(attribute, p):
for attr, val in p:
if attr==attribute:
return val
def instantiate(p):
radiusd.radlog(radiusd.L_DBG, '*** python: instantiate ***')
return radiusd.RLM_MODULE_OK
[...] # all other functions are exactly similar
Everything works fine if I just remove the line "import math".
Thank you very much for your help,
Aurélien Geron
Le 27 avr. 2010 à 01:56, Aurélien Geron a écrit :
> Hi,
>
> I came across a bug when rlm_python executes python code that tries to load a dynamic (shared) module. This bug seems to have been discussed 2 or 3 times on this list, but no really satisfying solution appears to have been found so far (as far as I know), so I thought I might raise the subject again and perhaps try to contribute in finding a solution.
>
> In short, I think the solution to this problem is explained here, but I don't know how to implement it in freeRADIUS : http://docs.python.org/release/2.5.2/ext/link-reqs.html
>
> Here is my setup :
> - running a perfectly standard Debian Lenny (2.6.26-2-amd64)
> - installed the latest freeradius package from the lenny-backports (2.1.8+dfsg-1~bpo50+1)
> - the python debian package is the one installed with Debian Lenny (2.5.2-3)
>
> Here is my config:
>
> ----------------------------------------------
> #######
> # /etc/freeradius/modules/python
> #######
> python python_test {
> mod_instantiate = radiusd_test
> func_instantiate = instantiate
> }
>
> #######
> # /etc/freeradius/radiusd.conf
> #######
> ...
> instantiate {
> python_test
> }
> ...
>
> #######
> # /usr/lib/python2.5/site-packages/radiusd_test.py
> #######
> def instantiate(p):
> radiusd.radlog(radiusd.L_DBG, "THIS WORKS")
> import random # this crashes
> radiusd.radlog(radiusd.L_DBG, "THIS IS NEVER REACHED !")
> ----------------------------------------------
>
> Here is the output of "freeradius -X" :
>
> ----------------------------------------------
> FreeRADIUS Version 2.1.8, for host x86_64-pc-linux-gnu, built on Apr 26 2010 at 21:49:28
> Copyright (C) 1999-2009 The FreeRADIUS server project and contributors.
> There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
> PARTICULAR PURPOSE.
> You may redistribute copies of FreeRADIUS under the terms of the
> GNU General Public License v2.
> Starting - reading configuration files ...
> including configuration file /etc/freeradius/radiusd.conf
> ...
> Module: Linked to module rlm_python
> Module: Instantiating python_test
> python_init done
> python python_test {
> mod_instantiate = "radiusd_test"
> func_instantiate = "instantiate"
> }
> THIS WORKS
> rlm_python:EXCEPT:<type 'exceptions.ImportError'>: /usr/lib/python2.5/lib-dynload/math.so: undefined symbol: PyExc_ValueError
> /etc/freeradius/modules/python[24]: Instantiation failed for module "python_test"
> ----------------------------------------------
>
> The module is properly loaded, the "instantiate" function gets called, and the first log message is output. In fact, any 100% pure python code works fine. The bug happens when a python module gets loaded dynamically : importing any module located in /usr/lib/python2.5/lib-dynload/*.so will crash.
>
>> From what I understand, here's what happens:
> 1) rlm_python was built against libpython2.5.a, but for optimization purposes, the linker stripped out all the symbols that were not used by rlm_python itself (including, for example, PyExc_ValueError). This behavior is platform-dependent, which probably explains why everything works fine on centos, for example.
> 2) when the radiusd_test module runs, it tries to import the "random" module, which itself tries to import the "math" module, which is in lib-dynload, and gets loaded dynamically.
> 3) unfortunately, python fails to load that module because it uses the PyExc_ValueError symbol and it does not know where it is defined (it is located in /usr/lib/libpython2.5.so.1, but unfortunately, the "math" module does not know that.
>
>
> As I said, I believe that the solution to this problem is clearly explained here: http://docs.python.org/release/2.5.2/ext/link-reqs.html
> As it's just a few paragraphs, I'll quote it here, for your convenience:
>
> ----------------------------------------------
> While the configure script shipped with the Python sources will correctly build Python to export the symbols needed by dynamically linked extensions, this is not automatically inherited by applications which embed the Python library statically, at least on Unix. This is an issue when the application is linked to the static runtime library (libpython.a) and needs to load dynamic extensions (implemented as.so files).
> The problem is that some entry points are defined by the Python runtime solely for extension modules to use. If the embedding application does not use any of these entry points, some linkers will not include those entries in the symbol table of the finished executable. Some additional options are needed to inform the linker not to remove these symbols.
> Determining the right options to use for any given platform can be quite difficult, but fortunately the Python configuration already has those values. To retrieve them from an installed Python interpreter, start an interactive interpreter and have a short session like this:
>>>> import distutils.sysconfig
>>>> distutils.sysconfig.get_config_var('LINKFORSHARED')
> '-Xlinker -export-dynamic'
> The contents of the string presented will be the options that should be used. If the string is empty, there's no need to add any additional options. The LINKFORSHARED definition corresponds to the variable of the same name in Python's top-level Makefile.
> ----------------------------------------------
>
> I think the short-term solution to this bug is to modify "src/modules/rlm_python/configure.in" and make it use the LINKFORSHARED linker options when linking rlm_python against libpython2.5.a., This could be done by adding a line such as:
> PY_OTHER_LDFLAGS=`sed -n -e 's/^LINKFORSHARED=\(.*\)/\1/p' $PY_MAKEFILE`
> just under this one:
> PY_OTHER_LIBS=`sed -n -e 's/^LIBS=\(.*\)/\1/p' $PY_MAKEFILE`
> and then using $PY_OTHER_LDFLAGS somewhere... the thing is, I have no idea where, because I am new to all these dynamic modules and embedded interpreters . I tried this though:
> python_ldflags="$PY_LIB_LOC $PY_EXTRA_LIBS $PY_OTHER_LDFLAGS -lpython${PY_VERSION} -lm"
> and recompiled the debian package (dpkg-buildpackage), but the problem was not fixed.
> Can someone tell me where to add those options exactly ? I can try recompiling as many times as necessary.
>
> Apparently, at least one person complains about the fact that you have to link against the static python runtime library (libpython2.5.a) even when you plan to use dynamic modules, and he has opened a bug report here:
> http://bugs.python.org/issue4434
>
> Basically, if I understand correctly, his idea is to have the python fellows declare the proper dependencies in every *.so file, so that the libpython2.5.so.1 file gets loaded automatically when the "math" module (or any other dynamic module) gets loaded. Maybe that's the ideal solution, I really don't know. But it seems to me that we should try to fix freeRADIUS so that it works around this bug before python dependencies are fixed (it make take a while or even never happen). So I thing the only short-medium-term solution is to use LINKFORSHARED linker options.
>
> Thanks for reading this huge message. I hope we can beat this bug.
>
> Aurelien Geron
> -
> List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
More information about the Freeradius-Users
mailing list