howto pstack running freeradius process
Hello, I am investigating one issue with freeradius 2.1.6 custom module and would like to get a stack trace of running process. pstack <pid of radiusd> fails with the following error: 23246: /usr/local/freeradius-2.1.6/sbin/radiusd '': opening object file: No such file or directory Could not open object file. The following commands was used to build the freeradius server: ./configure --prefix=/usr/local/freeradius-2.1.6 --enable-developer make make install Please point me to the right direction. Thanks in advance, George Chelidze
On Thu, 2009-07-23 at 16:10 +0200, Alan DeKok wrote:
George Chelidze wrote:
Hello,
I am investigating one issue with freeradius 2.1.6 custom module and would like to get a stack trace of running process.
This is a local OS issue. It has nothing to do with FreeRADIUS.
Hello Alan, I didn't say it's an issue with freeradius. I said it's an issue with a custom module and I am trying to find the reason that's why I asked about stack trace. Best Regards, George
On 23 July 2009, at 22:09, George Chelidze wrote:
On Thu, 2009-07-23 at 16:10 +0200, Alan DeKok wrote:
George Chelidze wrote:
Hello,
I am investigating one issue with freeradius 2.1.6 custom module and would like to get a stack trace of running process.
This is a local OS issue. It has nothing to do with FreeRADIUS.
Hello Alan,
I didn't say it's an issue with freeradius. I said it's an issue with a custom module and I am trying to find the reason that's why I asked about stack trace.
The approach I use to debug a module is to compile it with gdb (helps to also compile freeradius with gdb). Then run it under gdb with -X. You can then set breakpoints or other gdb trace commands and then feed it the input that causes the problem.
On Thu, 2009-07-23 at 22:27 -0700, Doug Hardie wrote:
On 23 July 2009, at 22:09, George Chelidze wrote:
On Thu, 2009-07-23 at 16:10 +0200, Alan DeKok wrote:
George Chelidze wrote:
Hello,
I am investigating one issue with freeradius 2.1.6 custom module and would like to get a stack trace of running process.
This is a local OS issue. It has nothing to do with FreeRADIUS.
Hello Alan,
I didn't say it's an issue with freeradius. I said it's an issue with a custom module and I am trying to find the reason that's why I asked about stack trace.
The approach I use to debug a module is to compile it with gdb (helps to also compile freeradius with gdb). Then run it under gdb with -X. You can then set breakpoints or other gdb trace commands and then feed it the input that causes the problem.
Hello Doug, Thanks for reply. Unfortunately when I start freeradiusd with -X problem is gone, it only exists when I start it in background, so I'd like to attach to the running daemon and get the stack trace if possible. Best Regards, George
On 23 July 2009, at 22:53, George Chelidze wrote:
On Thu, 2009-07-23 at 22:27 -0700, Doug Hardie wrote:
On 23 July 2009, at 22:09, George Chelidze wrote:
On Thu, 2009-07-23 at 16:10 +0200, Alan DeKok wrote:
George Chelidze wrote:
Hello,
I am investigating one issue with freeradius 2.1.6 custom module and would like to get a stack trace of running process.
This is a local OS issue. It has nothing to do with FreeRADIUS.
Hello Alan,
I didn't say it's an issue with freeradius. I said it's an issue with a custom module and I am trying to find the reason that's why I asked about stack trace.
The approach I use to debug a module is to compile it with gdb (helps to also compile freeradius with gdb). Then run it under gdb with -X. You can then set breakpoints or other gdb trace commands and then feed it the input that causes the problem.
Hello Doug,
Thanks for reply. Unfortunately when I start freeradiusd with -X problem is gone, it only exists when I start it in background, so I'd like to attach to the running daemon and get the stack trace if possible.
that sounds like a threading issue. Have you tried running it restricted to only one thread? There is a parameter in the initialization data (RLM_TYPE_THREAD_UNSAFE) that will prevent it from being multi-threaded. If that doesn't fail, you should have a better handle on the cause.
George Chelidze wrote:
I didn't say it's an issue with freeradius.
If it's not a FreeRADIUS issue, then the question doesn't belong on the list.
I said it's an issue with a custom module and I am trying to find the reason that's why I asked about stack trace.
a) it's your custom module b) it's an issue with pstack on your OS I can't find anything FreeRADIUS related in there. You're asking us to support (for free) a module you wrote, and/or an OS that someone else wrote. Why? Alan DeKok.
On Fri, 2009-07-24 at 08:08 +0200, Alan DeKok wrote:
George Chelidze wrote:
I didn't say it's an issue with freeradius.
If it's not a FreeRADIUS issue, then the question doesn't belong on the list.
I have just realized that this question should have been posted to freeradius-devel list. Sorry for mistake.
You're asking us to support (for free) a module you wrote, and/or an OS that someone else wrote.
Why?
What kind of answer you would like to get? I am afraid I missed something while building freeradius the way I did so I asked what I asked. If I knew that I have built freeradius with enough parameters to get the stack trace and I can't get it because I have some other OS related problem I would never asked this question on this list. I still do not know it, so if someone can give me a hint, I'll be thankful. Best Regards, George
On 07/24/2009 04:27 AM, George Chelidze wrote:
On Fri, 2009-07-24 at 08:08 +0200, Alan DeKok wrote:
George Chelidze wrote:
I didn't say it's an issue with freeradius. If it's not a FreeRADIUS issue, then the question doesn't belong on the list.
I have just realized that this question should have been posted to freeradius-devel list. Sorry for mistake.
You're asking us to support (for free) a module you wrote, and/or an OS that someone else wrote.
Why?
What kind of answer you would like to get? I am afraid I missed something while building freeradius the way I did so I asked what I asked. If I knew that I have built freeradius with enough parameters to get the stack trace and I can't get it because I have some other OS related problem I would never asked this question on this list. I still do not know it, so if someone can give me a hint, I'll be thankful.
I have to agree with Alan, this is not a FreeRADIUS issue. It is clearly an OS and software development environment issue. You haven't even stated what OS and architecture it is and your description of the error is vague at best. The man page for ptrace states it has architecture specific limitations. You built a local copy using your own toolchain and installed it in in a non-standard location, the ball is in your court. Here is a hint which is appropriate for Linux. I assume the process is aborting, if so the easiest thing to do is port-mortium analysis on a core dump using gdb with the assumption you built everything with debugging symbols. Normally Linux does not generate core dumps when a process aborts, you have to ask it to generate the core dump. This is done with "ulimit -c NNN" where NNN is the maximum size of the core dump, by default its zero. Run the server, allow it to crash, then run gdb on the generated core file and the server executable. The other thing you can do attach gdb to the running process and wait for things to go boom, you'll have a complete stack trace and can examine state the moment it happens, in fact this is really just an interactive version of the core dump approach, but without the core dump. It has an advantage of being able to pause the process before things go wrong and examine state. Just one gottcha, because FreeRADIUS uses loadable modules you won't be able to set break points in modules before they're loaded unless you tell gdb you need to do this, "set breakpoint pending on" is your friend in this regard. HTH, John -- John Dennis <jdennis@redhat.com> Looking to carve out IT costs? www.redhat.com/carveoutcosts/
On Fri, 2009-07-24 at 09:15 -0400, John Dennis wrote:
On 07/24/2009 04:27 AM, George Chelidze wrote:
On Fri, 2009-07-24 at 08:08 +0200, Alan DeKok wrote:
George Chelidze wrote:
I didn't say it's an issue with freeradius. If it's not a FreeRADIUS issue, then the question doesn't belong on the list.
I have just realized that this question should have been posted to freeradius-devel list. Sorry for mistake.
You're asking us to support (for free) a module you wrote, and/or an OS that someone else wrote.
Why?
What kind of answer you would like to get? I am afraid I missed something while building freeradius the way I did so I asked what I asked. If I knew that I have built freeradius with enough parameters to get the stack trace and I can't get it because I have some other OS related problem I would never asked this question on this list. I still do not know it, so if someone can give me a hint, I'll be thankful.
I have to agree with Alan, this is not a FreeRADIUS issue. It is clearly an OS and software development environment issue. You haven't even stated what OS and architecture it is and your description of the error is vague at best.
No, It's not a FreeRADIUS issue, it's an issue with my custom module. Let me say it again - I posted to the wrong list, sorry.
The man page for ptrace states it has architecture specific limitations. You built a local copy using your own toolchain and installed it in in a non-standard location, the ball is in your court.
My original question was about pstack not ptrace. If you mean pstack and "__pthread_threads_debug" stuff, I checked it before posting to this list.
Here is a hint which is appropriate for Linux. I assume the process is aborting
No, it's not, however your hists are useful. Thank you. Best Regards, George
participants (4)
-
Alan DeKok -
Doug Hardie -
George Chelidze -
John Dennis