panic_action and GDB/LLDB
The server now performs checks to determine whether it is running under gdb or lldb, and if it is, will not install the signal handlers required for panic_action to functions. We found that signal handlers for fatal signals interfered with debugger operation, and meant we couldn't get back traces or do interactive debugging under various crash conditions. The detection mechanism is basically: #ifndef HAVE_SYS_PTRACE_H * can't determine debug state, so install the signal handlers - end #endif #ifdef HAVE_CAPABILITY_H * check whether we currently have the CAP_SYS_PTRACE capability * if no, we can't determine debug state, so install the signal handlers - end #endif * fork a child with a pipe between parent and child * have the child attempt to ptrace the parent * if it can parent pauses execution * child writes a status code (to the pipe) indicating that * child detaches * parent resumes and reads status code * signal handlers are installed - end * if it can't then the process is being debugged * child writes a status code (to the pipe) indicating that * don't install signal handlers - end If anyone knows of another universally applicable method to determine if a process is being debugged then i'd be interested to implement it, but the hacky ptrace technique is the best I could come up with. If you're debugging and not running as the root user (which always has CAP_SYS_PTRACE), you can set the capability on the radius binary using: setcap cap_sys_ptrace+ep <path to radiusd bindary> Using setcap on gdb/lldb would also allow to use those utilities without being root. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
Why do the signal handlers get in the way? I'd expect the debugger to get the signal before freeradius does. If you want to resume the program without passing along the signal use the signal 0 command. If you want to mask the signals in the debugger so that the program never sees them do something like handle 11 nopass I don't understand the use case for this change and it seems kind of complex.
On 5 Nov 2014, at 09:02, Sam Hartman <hartmans@mit.edu> wrote:
Why do the signal handlers get in the way? I'd expect the debugger to get the signal before freeradius does.
Alan indicated in his experience, that this was sometimes not the case and that when running the server under a debugger he was unable to get a backtrace on fatal signals if the panic_action signal handlers were installed.
If you want to resume the program without passing along the signal use the signal 0 command. If you want to mask the signals in the debugger so that the program never sees them do something like handle 11 nopass
I don't understand the use case for this change
Principle of least surprise when debugging the server. If you start the server with CAP_SYS_PTRACE in it's effective capabilities list, it just works (which is what happens when you run it as root). That's superior to having to manually mask signals in the debugger.
and it seems kind of complex.
I agree, the code to determine if we can attach to the process is stupidly complex for what it's doing, but I don't know of a better way of detecting if a debugger is already attached. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
Hmm, I actually expect that programs will not change their behavior when under a debugger. I've found far more trouble with attempts to do that or with people being clever and basing decisions on whether they were in a debugger than in setting up debugger signal handlers. I've been debugging fatal signals in freeradius a fair bit recently and I have always seen the debugger behave as I expect.
Sam Hartman wrote:
Hmm, I actually expect that programs will not change their behavior when under a debugger.
I agree. The underlying reason for the changes was the idea to catch fatal signals in normal circumstances. i.e. rather than simply exiting, print out a stack trace of where the problem came from. Once that code went in, the normal gdb workflow didn't work. Which necessitated fixes to the debugging code... and down the rabbit hole we went.
I've been debugging fatal signals in freeradius a fair bit recently and I have always seen the debugger behave as I expect.
<sigh> lldb. It hates people. Alan DeKok.
Looks like the process handle command in lldb is basically the same as the handle command in gdb. It seems like either * a command line option to disable signal handlers or * an lldb debugger script to set sane defaults would be better approaches than this code.
On 5 Nov 2014, at 11:55, Sam Hartman <hartmans@mit.edu> wrote:
Looks like the process handle command in lldb is basically the same as the handle command in gdb.
It seems like either
* a command line option to disable signal handlers
I'm ok with an environmental variable, and for it to default to installing the signal handlers if the environmental variable is not set. Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On 05/11/14 16:27, Arran Cudbard-Bell wrote:
I agree, the code to determine if we can attach to the process is stupidly complex for what it's doing, but I don't know of a better way of detecting if a debugger is already attached.
Maybe don't. Maybe have a command-line switch to disable panic_action signal installers and document that this is needed for debugging? I'm wary about the code. How does it interact with SELinux/grsecurity for example? Failure to be able to ptrace a parent process might be expected in some - many? most? - cases.
participants (4)
-
Alan DeKok -
Arran Cudbard-Bell -
Phil Mayers -
Sam Hartman