core dumps not working on FreeBSD host
Dear list, since upgrade from 3.0.19 to 3.0.20 I am occasionally getting errors like this one: pid 23749 (radiusd), jid 1, uid 133: exited on signal 11 This issue is somehow related to network / connectivity problems and looks like quite a new problem (I have not seen this in 3.0.19 and earlier releases). It can't be easily triggered, so I am searching for the culprit. To find it I have tried to enable core dumps on FreeBSD for radiusd but without success. I have set allow_core_dumps = yes in radiusd.conf file. Also these sysctls are set: kern.sugid_coredump=1 kern.coredump=1 kern.corefile=/tmp/%N.%P.core There is no problem with getting core dumps for other daemons running on this host, also those running with suid/sgid enabled like bind or nsd, so the issue looks more like daemon related problem than OS problem. Any help would be greatly appreciated! -- Marek Zarychta
W dniu 05.02.2020 o 14:25, Marek Zarychta pisze:
Dear list,
since upgrade from 3.0.19 to 3.0.20 I am occasionally getting errors like this one:
pid 23749 (radiusd), jid 1, uid 133: exited on signal 11
This issue is somehow related to network / connectivity problems and looks like quite a new problem (I have not seen this in 3.0.19 and earlier releases). It can't be easily triggered, so I am searching for the culprit. To find it I have tried to enable core dumps on FreeBSD for radiusd but without success. I have set allow_core_dumps = yes in radiusd.conf file. Also these sysctls are set: kern.sugid_coredump=1 kern.coredump=1 kern.corefile=/tmp/%N.%P.core
There is no problem with getting core dumps for other daemons running on this host, also those running with suid/sgid enabled like bind or nsd, so the issue looks more like daemon related problem than OS problem.
Any help would be greatly appreciated!
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Running radiusd -X and going through debug output I found these lines: Changing value of PR_DUMPABLE not supported on this system Core dumps are enabled Which probably according to https://github.com/FreeRADIUS/freeradius-server/blob/v3.0.x/src/lib/debug.c#... mean that core dumps are not really supported while running FreeRADIUS on this OS ? -- Marek Zarychta
On Feb 5, 2020, at 12:17 PM, Marek Zarychta <zarychtam@plan-b.pwste.edu.pl> wrote:
Running radiusd -X and going through debug output I found these lines:
Changing value of PR_DUMPABLE not supported on this system Core dumps are enabled
Which probably according to https://github.com/FreeRADIUS/freeradius-server/blob/v3.0.x/src/lib/debug.c#...
mean that core dumps are not really supported while running FreeRADIUS on this OS ?
I suppose not. We don't currently had a FreeBSD system for testing. I'm sure it's fixable, but it means spending some time trying to understand what we need to do. Alan DeKok.
W dniu 05.02.2020 o 19:20, Alan DeKok pisze:
On Feb 5, 2020, at 12:17 PM, Marek Zarychta <zarychtam@plan-b.pwste.edu.pl> wrote:
Running radiusd -X and going through debug output I found these lines:
Changing value of PR_DUMPABLE not supported on this system Core dumps are enabled
Which probably according to https://github.com/FreeRADIUS/freeradius-server/blob/v3.0.x/src/lib/debug.c#...
mean that core dumps are not really supported while running FreeRADIUS on this OS ? I suppose not. We don't currently had a FreeBSD system for testing. I'm sure it's fixable, but it means spending some time trying to understand what we need to do.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Dear FreeRadius users and other subscribers of the list, especially those running FreeRADIUS on FreeBSD To close the case I owe you a bit of explanation. FreeRADIUS run on FreeBSD permits to create core dumps, respects "allow_core_dumps" setting in radius.conf but requires radiusd to be run with uid and gid 0 which I was not aware of blindly relying on kern.sugid_coredump setting=1. So getting core dumps for FreeBSD was available from the beginning, though it was not documented. The culprit of not getting these dumps was the fact of running radiusd with uid=133(freeradius) gid=133(freeradius), which is a typical setting. To clarify this I want to assure you that running daemons with effective uid/gid > 0 doesn't prevent them from creating such dumps when sysctl kern.sugid_coredump is set to 1. FreeRADIUS seems to be exceptional in this case. Anyway, I have sorted this out a few weeks ago but so far I wasn't able to reproduce the original error which initiated the investigation. I am really grateful for your assistance and support, especially for Jorge's help. -- Marek Zarychta
Hi Mark, Thank you for the feedback. Btw, the feature was incorporated into the v3.0.x branch (it will be available on the next 3.0.21) and also in master (v4.0.0) branch. Anyway, we have some tests in https://github.com/FreeRADIUS/freeradius-server/pull/3279#issuecomment-58499... <https://github.com/FreeRADIUS/freeradius-server/pull/3279#issuecomment-584993824> --- Jorge Pereira jpereira@freeradius.org
On 4 Mar 2020, at 17:11, Marek Zarychta <zarychtam@plan-b.pwste.edu.pl> wrote:
W dniu 05.02.2020 o 19:20, Alan DeKok pisze:
On Feb 5, 2020, at 12:17 PM, Marek Zarychta <zarychtam@plan-b.pwste.edu.pl> wrote:
Running radiusd -X and going through debug output I found these lines:
Changing value of PR_DUMPABLE not supported on this system Core dumps are enabled
Which probably according to https://github.com/FreeRADIUS/freeradius-server/blob/v3.0.x/src/lib/debug.c#...
mean that core dumps are not really supported while running FreeRADIUS on this OS ? I suppose not. We don't currently had a FreeBSD system for testing. I'm sure it's fixable, but it means spending some time trying to understand what we need to do.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Dear FreeRadius users and other subscribers of the list, especially those running FreeRADIUS on FreeBSD
To close the case I owe you a bit of explanation. FreeRADIUS run on FreeBSD permits to create core dumps, respects "allow_core_dumps" setting in radius.conf but requires radiusd to be run with uid and gid 0 which I was not aware of blindly relying on kern.sugid_coredump setting=1. So getting core dumps for FreeBSD was available from the beginning, though it was not documented. The culprit of not getting these dumps was the fact of running radiusd with uid=133(freeradius) gid=133(freeradius), which is a typical setting. To clarify this I want to assure you that running daemons with effective uid/gid > 0 doesn't prevent them from creating such dumps when sysctl kern.sugid_coredump is set to 1. FreeRADIUS seems to be exceptional in this case.
Anyway, I have sorted this out a few weeks ago but so far I wasn't able to reproduce the original error which initiated the investigation.
I am really grateful for your assistance and support, especially for Jorge's help.
-- Marek Zarychta
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Hi Marek, If you could trigger it easily, maybe you could just start the process with GDB and after catching the crash just uses the command “bt full all threads” to show the entire stack trace and send it to us or create a ticket https://github.com/FreeRADIUS/freeradius-server/issues/new/choose <https://github.com/FreeRADIUS/freeradius-server/issues/new/choose> It would be good if you could do that. -- Jorge Pereira jpereira@networkradius.com
On 5 Feb 2020, at 10:25, Marek Zarychta <zarychtam@plan-b.pwste.edu.pl> wrote:
Dear list,
since upgrade from 3.0.19 to 3.0.20 I am occasionally getting errors like this one:
pid 23749 (radiusd), jid 1, uid 133: exited on signal 11
This issue is somehow related to network / connectivity problems and looks like quite a new problem (I have not seen this in 3.0.19 and earlier releases). It can't be easily triggered, so I am searching for the culprit. To find it I have tried to enable core dumps on FreeBSD for radiusd but without success. I have set allow_core_dumps = yes in radiusd.conf file. Also these sysctls are set: kern.sugid_coredump=1 kern.coredump=1 kern.corefile=/tmp/%N.%P.core
There is no problem with getting core dumps for other daemons running on this host, also those running with suid/sgid enabled like bind or nsd, so the issue looks more like daemon related problem than OS problem.
Any help would be greatly appreciated!
-- Marek Zarychta
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
W dniu 05.02.2020 o 19:20, Jorge Pereira pisze:
Hi Marek,
If you could trigger it easily, maybe you could just start the process with GDB and after catching the crash just uses the command “bt full all threads” to show the entire stack trace and send it to us or create a ticket https://github.com/FreeRADIUS/freeradius-server/issues/new/choose <https://github.com/FreeRADIUS/freeradius-server/issues/new/choose>
It would be good if you could do that.
-- Jorge Pereira jpereira@networkradius.com
I have been using freeradius3-3.0.20 since Nov 26th 2019 on both systems, the issue has occurred only 3 or 4 times since then, so it can't be triggered easily. I believe that running the process in the debugger for weeks is not an acceptable scenario. I suspect that some network related problems triggered the issue but it's not easily reproducible. I must admit that so far all previous versions survived network outages easily. I am running FreeRADIUS on FreeBSD for 10+ years, never encountered strictly OS related issues since both FreeRADIUS and FreeBSD are great pieces of software. I am grateful and really appreciate your work guys as well as the ability to get your support here. With kind regards, -- Marek Zarychta
participants (3)
-
Alan DeKok -
Jorge Pereira -
Marek Zarychta