Like I said in my original email we are aware of the problems caused by the lack of a timeout and are actively working to fix that, so you don't need to be condescending and rude about the lack of a timeout. My question was that even after dynamo recovered FreeRADIUS continued to fail to call python, and I wanted to make sure that even after we fix the timeout issue that there isn't another problem that could be triggered. Are you suggesting the only possible explanation for what we saw was that all of the worker threads were still waiting for a response from dynamo 2 days later, and that there is no point in debugging why FreeRADIUS was unable to execute rlm_python? I suppose that's possible but it seems unlikely that that many sockets were hung open for that long. Is there any way to see what the worker threads were doing? On Mon, Sep 21, 2015 at 2:37 PM, Arran Cudbard-Bell < a.cudbardb@freeradius.org> wrote:
The first line in our python accounting module is to radlog() so I know
if
the request hit our code, and that wasn't even happening
Hmm, so Python SDK has no timeout, and your complaint is that requests were never timing out?
Yes python is smart enough to release the GIL on blocking I/O, no there is not a mechanism to return control to the caller of the Python interpreter on blocking I/O. At least none that i've found wandering through the shitty Python embedding documentation.
so it seems as though FreeRADIUS+rlm_python had failed in some way. Restarting freeradius resolved the problem.
No, your SDK failed, FreeRADIUS+rlm_python functioned as intended.
I'm fairly certain the long request time problem is because the default timeout for the dynamo SDK is none (wait forever). I'm working on setting a boto timeout to prevent the long response time from happening again, but at the moment I'm concerned about how freeradius was failing to even call python after a while.
No, all worker threads were blocked in the python module so there were no threads to call python after a while, they were all already calling python.
Aside from the boto timeout issue how would we go about debugging the freeradius+python problem to make sure it doesn't happen in the future?
Fix the SDK timeout.
-Arran
Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team
FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html