Error: Cannot update core dump limit

[Jakob Hirsch]

Arran Cudbard-Bell wrote on 2016-07-13 15:17:
>> but FR uses setrlimit() with rlim_max=0, and you can only lower
>> rlim_max, but not raise it (unless you have the CAP_SYS_RESOURCE
>> capability).
> OK... So there's probably an issue there.  Could we get the stack trace so it's clear what logic is being executed.

Sure:

> #0  fr_set_dumpable (allow_core_dumps=false) at src/lib/debug.c:526
> #1  0x00007ffff7bc75a2 in rad_suid_down () at src/main/util.c:1560
> #2  0x0000000000420bf0 in switch_users (cs=0x81fbf0) at src/main/mainconfig.c:651
> #3  main_config_init () at src/main/mainconfig.c:933
> #4  0x0000000000410cc7 in main (argc=2, argv=0x7fffffffe748) at src/main/radiusd.c:347

I tried to track this down myself. I think the cause is this calls chain:

switch_users()
  rad_suid_down()
    fr_reset_dumpable()
      fr_set_dumpable(dump_core) <= Note that dump_core is not set at
this point (only initialized to 0 since it's a static bool), so the core
dump limit is set to 0 by fr_set_dumpable().
  ...
  fr_set_dumpable(allow_core_dumps) <= allow_core_dumps is true here,
but since fr_set_dumpable() was already called with 0=false and
therefore setrlimit(RLIMIT_CORE, &no_core), it's not possible to
re-raise the core dump limit.

An easy fix could be to simply use the hard limit that was already
fetched by fr_set_dumpable_init():

diff --git a/src/lib/debug.c b/src/lib/debug.c
index 797b4b6..eb39148 100644
--- a/src/lib/debug.c
+++ b/src/lib/debug.c
@@ -533,7 +533,7 @@ int fr_set_dumpable(bool allow_core_dumps)
                struct rlimit no_core;

                no_core.rlim_cur = 0;
-               no_core.rlim_max = 0;
+               no_core.rlim_max = core_limits.rlim_max;

                if (setrlimit(RLIMIT_CORE, &no_core) < 0) {
                        fr_strerror_printf("Failed disabling core dumps:
%s", fr_syserror(errno));

Or we could add a flag to fr_set_dumpable_init() which sets dump_core to
the correct value.
OTOH, I don't see why we need to setrlimit() twice, but maybe there is
some good reason for the fr_reset_dumpable() call in rad_suid_down()...

> There's only two calls to setrlimit in the server...
> 
>> I don't know what the call setrlimit(RLIMIT_CORE,
>> {rlim_cur=0, rlim_max=0}) is good for...
> It disables core dumping.

I know. I just wondered why the server is doing that...