Segfault at src/lib/misc.c:1193 in 3.0.4 (3.0.11 looks very similar)
Behavior is similar on two production hosts running Centos 7.2. Radius is setup as proxy only, passes authentication successfully for a time, then abruptly segfaults. Attached gdb to corefile and found this: Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/radiusd -d /etc/raddb'. Program terminated with signal 11, Segmentation fault. #0 fr_ipaddr_cmp (a=0x20, b=0x7f3965a30510) at src/lib/misc.c:1193 1193 if (a->af < b->af) return -1; It looks like it's somehow getting the memory address for "a" set to 0x20, which is obviously invalid, and then it segfaults. Where should I look in the configs for where realm acct_pool servers is being set up and what would I expect to see in a configuration that would allow the server to run successfully and then crash abruptly like this? Obviously let me know what other data to provide for debugging this. Thanks, Mike
On Sep 26, 2016, at 3:45 PM, Mike Ely <me@mikeely.org> wrote:
Behavior is similar on two production hosts running Centos 7.2. Radius is setup as proxy only, passes authentication successfully for a time, then abruptly segfaults. Attached gdb to corefile and found this:
Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/radiusd -d /etc/raddb'. Program terminated with signal 11, Segmentation fault. #0 fr_ipaddr_cmp (a=0x20, b=0x7f3965a30510) at src/lib/misc.c:1193 1193 if (a->af < b->af) return -1;
It looks like it's somehow getting the memory address for "a" set to 0x20, which is obviously invalid, and then it segfaults.
The gdb backtrace would be a lot more useful than just one line.
Where should I look in the configs for where realm acct_pool servers is being set up and what would I expect to see in a configuration that would allow the server to run successfully and then crash abruptly like this?
Is there any particular reason to ask a question like that? Do you want help, or are you interested in insulting the developers? The server should not crash. In fact, we've managed to do authentication proxying in 3.0.4 and 3.0.11 for millions of packets without a server crash. The default configuration doesn't crash. Any configuration I've tried doesn't crash like this. So... what configuration changes did you make? Can you describe what you did?
Obviously let me know what other data to provide for debugging this.
This is documented. http://wiki.freeradius.org/project/bug-reports Alan DeKok.
The configs are complex and have had many hands on them over a long period of time. I am attempting to migrate these configs (some of them set by people who are no longer present where I work) over from a working 2.x system that proxies numerous devices. In order to get the migrated configs to work I had to change the user and group stanzas to within a security{} block and also get rid of calls to the no longer used rlm_acct_unique module. The configurations would require a fair amount of redaction to share publicly, which is why I asked the more general question of where to look within them. Insulting anyone was the nowhere in my mind. I'm trying to figure out an unexpected crash, that's all. Here's the backtrace: #0 fr_ipaddr_cmp (a=0x20, b=0x7f3965a30510) at src/lib/misc.c:1193 #1 0x00007f3967740426 in check_for_realm (instance=<optimized out>, request=0x7f396f3cda40, returnrealm=0x7f3965a30580) at src/modules/rlm_realm/rlm_realm.c:276 #2 0x00007f396774052d in mod_preacct (instance=<optimized out>, request=0x7f396f3cda40) at src/modules/rlm_realm/rlm_realm.c:399 #3 0x00007f396d5f3109 in call_modsingle (request=0x7f396f3cda40, sp=0x7f396f3b3720, component=RLM_COMPONENT_PREACCT) at src/main/modcall.c:318 #4 modcall_recurse (request=0x7f396f3cda40, component=RLM_COMPONENT_PREACCT, depth=1, entry=entry@entry=0x7f3965a309c8) at src/main/modcall.c:587 #5 0x00007f396d5f2a70 in modcall_child (request=<optimized out>, component=<optimized out>, depth=<optimized out>, entry=0x7f3965a309b0, c=<optimized out>, result=0x7f3965a30824) at src/main/modcall.c:420 #6 0x00007f396d5f2c67 in modcall_recurse (request=request@entry=0x7f396f3cda40, component=component@entry=RLM_COMPONENT_PREACCT, depth=depth@entry=0, entry=entry@entry=0x7f3965a309b0) at src/main/modcall.c:798 #7 0x00007f396d5f3e90 in modcall (component=component@entry=RLM_COMPONENT_PREACCT, c=c@entry=0x7f396f3b2da0, request=request@entry=0x7f396f3cda40) at src/main/modcall.c:1138 #8 0x00007f396d5f12bf in indexed_modcall (comp=comp@entry=RLM_COMPONENT_PREACCT, idx=idx@entry=0, request=0x7f396f3cda40) at src/main/modules.c:909 #9 0x00007f396d5f1e3f in module_preacct (request=<optimized out>) at src/main/modules.c:1867 #10 0x00007f396d5e1c65 in rad_accounting (request=0x7f396f3cda40) at src/main/acct.c:56 #11 0x00007f396d601677 in request_running (request=0x7f396f3cda40, action=<optimized out>) at src/main/process.c:1498 #12 0x00007f396d5fb617 in request_handler_thread (arg=0x7f396f3b80f0) at src/main/threads.c:678 #13 0x00007f396b82adc5 in start_thread (arg=0x7f3965a31700) at pthread_create.c:308 #14 0x00007f396b0daced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 From the bt full, this was interesting: #1 0x00007f3967740426 in check_for_realm (instance=<optimized out>, request=0x7f396f3cda40, returnrealm=0x7f3965a30580) at src/modules/rlm_realm/rlm_realm.c:276 i = 1 my_ipaddr = {af = 2, ipaddr = {ip4addr = {s_addr = 167775328}, ip6addr = {__in6_u = { __u6_addr8 = "`\f\000\064\000\376:o9\177\000\000-402", __u6_addr16 = {3168, 13312, 65024, 28474, 32569, 0, 13357, 12848}, __u6_addr32 = {167775328, 1866137088, 32569, 167775328}}}}, prefix = 54 '6', scope = 1664759088} username = 0x0 vp = <optimized out> realm = 0x7f396dfcf890 namebuf = <optimized out> realmname = <optimized out> ptr = <optimized out> returnrealm = 0x7f3965a30580 request = 0x7f396f3cda40 instance = <optimized out> inst = <optimized out> On 09/26/2016 12:53 PM, Alan DeKok wrote:
On Sep 26, 2016, at 3:45 PM, Mike Ely <me@mikeely.org> wrote:
Behavior is similar on two production hosts running Centos 7.2. Radius is setup as proxy only, passes authentication successfully for a time, then abruptly segfaults. Attached gdb to corefile and found this:
Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/radiusd -d /etc/raddb'. Program terminated with signal 11, Segmentation fault. #0 fr_ipaddr_cmp (a=0x20, b=0x7f3965a30510) at src/lib/misc.c:1193 1193 if (a->af < b->af) return -1;
It looks like it's somehow getting the memory address for "a" set to 0x20, which is obviously invalid, and then it segfaults.
The gdb backtrace would be a lot more useful than just one line.
Where should I look in the configs for where realm acct_pool servers is being set up and what would I expect to see in a configuration that would allow the server to run successfully and then crash abruptly like this?
Is there any particular reason to ask a question like that? Do you want help, or are you interested in insulting the developers?
The server should not crash. In fact, we've managed to do authentication proxying in 3.0.4 and 3.0.11 for millions of packets without a server crash.
The default configuration doesn't crash. Any configuration I've tried doesn't crash like this.
So... what configuration changes did you make? Can you describe what you did?
Obviously let me know what other data to provide for debugging this.
This is documented.
http://wiki.freeradius.org/project/bug-reports
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Sep 26, 2016, at 4:13 PM, Mike Ely <me@mikeely.org> wrote:
The configs are complex and have had many hands on them over a long period of time. I am attempting to migrate these configs (some of them set by people who are no longer present where I work) over from a working 2.x system that proxies numerous devices. In order to get the migrated configs to work I had to change the user and group stanzas to within a security{} block and also get rid of calls to the no longer used rlm_acct_unique module.
The recommendation is to create a new configuration instead of migrating an old one. The configurations are similar, but not identical.
The configurations would require a fair amount of redaction to share publicly, which is why I asked the more general question of where to look within them. Insulting anyone was the nowhere in my mind. I'm trying to figure out an unexpected crash, that's all.
Asking leading questions is unpleasant and unnecessary.
Here's the backtrace: #0 fr_ipaddr_cmp (a=0x20, b=0x7f3965a30510) at src/lib/misc.c:1193 #1 0x00007f3967740426 in check_for_realm (instance=<optimized out>, request=0x7f396f3cda40, returnrealm=0x7f3965a30580) at src/modules/rlm_realm/rlm_realm.c:276 #2 0x00007f396774052d in mod_preacct (instance=<optimized out>, request=0x7f396f3cda40) at src/modules/rlm_realm/rlm_realm.c:399 #3 0x00007f396d5f3109 in call_modsingle (request=0x7f396f3cda40, sp=0x7f396f3b3720, component=RLM_COMPONENT_PREACCT) at src/main/modcall.c:318
Hmm.. you have a fail-over pool with a mix of real home servers, and local ones (i.e. virtual servers). This is generally not recommended, but is allowed. I've pushed a fix. Alan DeKok.
Here's the backtrace: #0 fr_ipaddr_cmp (a=0x20, b=0x7f3965a30510) at src/lib/misc.c:1193 #1 0x00007f3967740426 in check_for_realm (instance=<optimized out>, request=0x7f396f3cda40, returnrealm=0x7f3965a30580) at src/modules/rlm_realm/rlm_realm.c:276 #2 0x00007f396774052d in mod_preacct (instance=<optimized out>, request=0x7f396f3cda40) at src/modules/rlm_realm/rlm_realm.c:399 #3 0x00007f396d5f3109 in call_modsingle (request=0x7f396f3cda40, sp=0x7f396f3b3720, component=RLM_COMPONENT_PREACCT) at src/main/modcall.c:318
Hmm.. you have a fail-over pool with a mix of real home servers, and local ones (i.e. virtual servers). This is generally not recommended, but is allowed.
I've pushed a fix.
Alan DeKok.
Unfortunately after applying the two fixes added to rlm_realm.c on 9/26 the segfault persists. Here's the latest backtrace: Core was generated by `/usr/sbin/radiusd -d /etc/raddb'. Program terminated with signal 11, Segmentation fault. #0 0x00007f48dfea1416 in check_for_realm (instance=<optimized out>, request=0x7f48e7870830, returnrealm=0x7f48dc18d580) at src/modules/rlm_realm/rlm_realm.c:276 276 if (realm->acct_pool->servers[i]->ipaddr.af == AF_UNSPEC) continue; (gdb) bt #0 0x00007f48dfea1416 in check_for_realm (instance=<optimized out>, request=0x7f48e7870830, returnrealm=0x7f48dc18d580) at src/modules/rlm_realm/rlm_realm.c:276 #1 0x00007f48dfea153d in mod_preacct (instance=<optimized out>, request=0x7f48e7870830) at src/modules/rlm_realm/rlm_realm.c:403 #2 0x00007f48e5d54109 in call_modsingle (request=0x7f48e7870830, sp=0x7f48e78561b0, component=RLM_COMPONENT_PREACCT) at src/main/modcall.c:318 #3 modcall_recurse (request=0x7f48e7870830, component=RLM_COMPONENT_PREACCT, depth=1, entry=entry@entry=0x7f48dc18d9c8) at src/main/modcall.c:587 #4 0x00007f48e5d53a70 in modcall_child (request=<optimized out>, component=<optimized out>, depth=<optimized out>, entry=0x7f48dc18d9b0, c=<optimized out>, result=0x7f48dc18d824) at src/main/modcall.c:420 #5 0x00007f48e5d53c67 in modcall_recurse (request=request@entry=0x7f48e7870830, component=component@entry=RLM_COMPONENT_PREACCT, depth=depth@entry=0, entry=entry@entry=0x7f48dc18d9b0) at src/main/modcall.c:798 #6 0x00007f48e5d54e90 in modcall (component=component@entry=RLM_COMPONENT_PREACCT, c=c@entry=0x7f48e7855830, request=request@entry=0x7f48e7870830) at src/main/modcall.c:1138 #7 0x00007f48e5d522bf in indexed_modcall (comp=comp@entry=RLM_COMPONENT_PREACCT, idx=idx@entry=0, request=0x7f48e7870830) at src/main/modules.c:909 #8 0x00007f48e5d52e3f in module_preacct (request=<optimized out>) at src/main/modules.c:1867 #9 0x00007f48e5d42c65 in rad_accounting (request=0x7f48e7870830) at src/main/acct.c:56 #10 0x00007f48e5d62677 in request_running (request=0x7f48e7870830, action=<optimized out>) at src/main/process.c:1498 #11 0x00007f48e5d5c617 in request_handler_thread (arg=0x7f48e786f830) at src/main/threads.c:678 #12 0x00007f48e3f8bdc5 in start_thread (arg=0x7f48dc18e700) at pthread_create.c:308 #13 0x00007f48e383bced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
On Sep 30, 2016, at 2:03 PM, Mike Ely <me@mikeely.org> wrote:
Unfortunately after applying the two fixes added to rlm_realm.c on 9/26 the segfault persists. Here's the latest backtrace:
Which is not as useful, because you've run a locally patched version. So the line numbers don't match up with any released version, or any version in fit.
Core was generated by `/usr/sbin/radiusd -d /etc/raddb'. Program terminated with signal 11, Segmentation fault. #0 0x00007f48dfea1416 in check_for_realm (instance=<optimized out>, request=0x7f48e7870830, returnrealm=0x7f48dc18d580) at src/modules/rlm_realm/rlm_realm.c:276 276 if (realm->acct_pool->servers[i]->ipaddr.af == AF_UNSPEC) continue;
The larger question is what are you doing in your configuration? No one else has run into these issues. That makes me think you're doing something fairly unusual. Alan DeKok.
On 09/30/2016 01:59 PM, Alan DeKok wrote:
On Sep 30, 2016, at 2:03 PM, Mike Ely <me@mikeely.org> wrote:
Unfortunately after applying the two fixes added to rlm_realm.c on 9/26 the segfault persists. Here's the latest backtrace:
Which is not as useful, because you've run a locally patched version. So the line numbers don't match up with any released version, or any version in fit. The line number corresponds to the comparison you added to the second update on 9/26: 275: for (i = 0; i < realm->acct_pool->num_home_servers; i++) { 276: if (realm->acct_pool->servers[i]->ipaddr.af == AF_UNSPEC) continue;
Core was generated by `/usr/sbin/radiusd -d /etc/raddb'. Program terminated with signal 11, Segmentation fault. #0 0x00007f48dfea1416 in check_for_realm (instance=<optimized out>, request=0x7f48e7870830, returnrealm=0x7f48dc18d580) at src/modules/rlm_realm/rlm_realm.c:276 276 if (realm->acct_pool->servers[i]->ipaddr.af == AF_UNSPEC) continue;
The larger question is what are you doing in your configuration?
Just proxying for about half a dozen realms. Can a badly-written .conf file really cause radiusd to segfault?
On Sep 30, 2016, at 5:10 PM, Mike Ely <me@mikeely.org> wrote:
The larger question is what are you doing in your configuration?
Just proxying for about half a dozen realms. Can a badly-written .conf file really cause radiusd to segfault?
It shouldn't. But as I said, if you're doing something unusual, who knows. So... *what* are you doing? My question was "what are you doing". i.e. please explain, and show the config. Answering "I'm doing proxying" is giving me information I already know. *What* is your configuration? What is it doing when it dies? Please give SOME information other than "it dies". Alan DeKok.
On 09/30/2016 02:14 PM, Alan DeKok wrote:
On Sep 30, 2016, at 5:10 PM, Mike Ely <me@mikeely.org> wrote:
The larger question is what are you doing in your configuration?
Just proxying for about half a dozen realms. Can a badly-written .conf file really cause radiusd to segfault?
It shouldn't. But as I said, if you're doing something unusual, who knows.
So... *what* are you doing? My question was "what are you doing". i.e. please explain, and show the config. Answering "I'm doing proxying" is giving me information I already know.
*What* is your configuration? What is it doing when it dies? Please give SOME information other than "it dies".
It's proxying requests when it dies. I've yet to discern what request is killing it, and the rate is very low - somewhere around 20/min. Redacting config. May take a while.
On Sep 30, 2016, at 6:08 PM, Mike Ely <me@mikeely.org> wrote:
It's proxying requests when it dies. I've yet to discern what request is killing it, and the rate is very low - somewhere around 20/min.
Redacting config. May take a while.
All I need is the "proxy.conf" file. With IP addresses and secrets changed. Alan DeKok.
All I need is the "proxy.conf" file. With IP addresses and secrets changed.
Alan DeKok.
That helps a lot. I've stripped comments and obfuscated in a way that's logically consistent with our config, so the munged names, IPs, etc should map to one another in the same way as the real ones do. proxy server { synchronous = no retry_delay = 5 retry_count = 3 dead_time = 120 default_fallback = no post_proxy_authorize = no } home_server a.radius.anaheim { type = auth+acct ipaddr = a.radius.anaheim.example.net port = 1812 secret = nope strip } home_server b.radius.anaheim { type = auth+acct ipaddr = b.radius.anaheim.example.net port = 1812 secret = nope strip } home_server c.radius.anaheim { type = auth+acct ipaddr = c.radius.anaheim.example.net port = 1812 secret = nope strip } home_server d.radius.anaheim { type = auth+acct ipaddr = d.radius.anaheim.example.net port = 1812 secret = nope strip } home_server_pool example_load_balance { type = load-balance home_server = a.radius.anaheim home_server = b.radius.anaheim home_server = c.radius.anaheim home_server = d.radius.anaheim } realm example.net { pool = example_load_balance nostrip } realm mickey1.com { pool = example_load_balance strip } realm "fred-test.com" { type = radius authhost = 10.1.3.226:2015 accthost = 10.1.3.226:2016 secret = nope nostrip } realm "fred.barney.com" { type = radius authhost = bigauth3.special1.fred.com:2015 accthost = bigauth4.special1.fred.com:2016 secret = nope nostrip } realm "fred.barney.com" { type = radius authhost = bigauth3.special2.fred.com:2015 accthost = bigauth4.special2.fred.com:2016 secret = nope nostrip } realm "fred.com" { type = radius authhost = bigauth3.special1.fred.com:2015 accthost = bigauth4.special1.fred.com:2016 secret = nope nostrip } realm "fred.com" { type = radius authhost = bigauth3.special2.fred.com:2015 accthost = bigauth4.special2.fred.com:2016 secret = nope nostrip } realm "pluto.net" { type = radius authhost = bigauth3.special1.fred.com:2015 accthost = bigauth4.special1.fred.com:2016 secret = nope nostrip } realm "pluto.net" { type = radius authhost = bigauth3.special2.fred.com:2015 accthost = bigauth4.special2.fred.com:2016 secret = nope nostrip } realm "pluto.com" { type = radius authhost = bigauth3.special1.fred.com:2015 accthost = bigauth4.special1.fred.com:2016 secret = nope nostrip } realm "minnie.net" { type = radius authhost = bigauth3.special1.fred.com:2015 accthost = bigauth4.special1.fred.com:2016 secret = nope nostrip } realm "minnie.net" { type = radius authhost = bigauth3.special2.fred.com:2015 accthost = bigauth4.special2.fred.com:2016 secret = nope nostrip } realm "minnie.com" { type = radius authhost = bigauth3.special1.fred.com:2015 accthost = bigauth4.special1.fred.com:2016 secret = nope nostrip } realm "minnie.com" { type = radius authhost = bigauth3.special2.fred.com:2015 accthost = bigauth4.special2.fred.com:2016 secret = nope nostrip } realm "daffy.net" { type = radius authhost = 10.4.1.38 accthost = 10.4.1.38 secret = nope nostrip } realm "daffy.net" { type = radius authhost = 10.4.2.21 accthost = 10.4.1.38 secret = nope nostrip } realm "daffy.net" { type = radius authhost = 10.5.1.7 accthost = 10.4.1.38 secret = nope nostrip } realm "clarabelle.com" { type = radius authhost = 10.4.1.38 accthost = 10.4.1.38 secret = nope nostrip } realm "clarabelle.com" { type = radius authhost = 10.4.2.21 accthost = 10.4.1.38 secret = nope nostrip } realm "clarabelle.com" { type = radius authhost = 10.5.1.7 accthost = 10.4.1.38 secret = nope nostrip } realm LOCAL { type = radius authhost = LOCAL accthost = LOCAL } realm popeye { type = radius authhost = LOCAL accthost = LOCAL nostrip } realm bluto { type = radius authhost = LOCAL accthost = LOCAL nostrip }
On Oct 3, 2016, at 1:38 PM, Mike Ely <me@mikeely.org> wrote:
All I need is the "proxy.conf" file. With IP addresses and secrets changed.
Alan DeKok.
That helps a lot. I've stripped comments and obfuscated in a way that's logically consistent with our config, so the munged names, IPs, etc should map to one another in the same way as the real ones do.
When it crashes, can you also do: (gdb) p *realm (gdb) p *realm->acct_pool->servers[i] which should also let me know *which* realm and server had the problem. Alan DeKok.
On 10/03/2016 11:36 AM, Alan DeKok wrote:
When it crashes, can you also do:
(gdb) p *realm (gdb) p *realm->acct_pool->servers[i]
which should also let me know *which* realm and server had the problem.
From the coredump of the last segfault (I've pulled the afflicted box out of production and am running the older 2.x hardware): (gdb) p *realm $1 = {name = 0x7f48e63e5f30 "daffy.net", strip_realm = false, auth_pool = 0x7f48e6471770, acct_pool = 0x7f48e6471c60, coa_pool = 0x0} (gdb) p *realm->acct_pool->servers[i] Cannot access memory at address 0x0
On Oct 3, 2016, at 2:45 PM, Mike Ely <me@mikeely.org> wrote:
On 10/03/2016 11:36 AM, Alan DeKok wrote:
When it crashes, can you also do:
(gdb) p *realm (gdb) p *realm->acct_pool->servers[i]
which should also let me know *which* realm and server had the problem.
From the coredump of the last segfault (I've pulled the afflicted box out of production and am running the older 2.x hardware):
(gdb) p *realm $1 = {name = 0x7f48e63e5f30 "daffy.net", strip_realm = false, auth_pool = 0x7f48e6471770, acct_pool = 0x7f48e6471c60, coa_pool = 0x0} (gdb) p *realm->acct_pool->servers[i] Cannot access memory at address 0x0
Huh? Just to confirm, can you do: (gdb) p *realm->acct_pool (gdb) p *realm->acct_pool->servers That's weird. Either the accounting pool has no servers, or it's somehow managed to walk off of the end of the list of accounting servers. Alan DeKok.
On 10/03/2016 11:56 AM, Alan DeKok wrote:
(gdb) p *realm $1 = {name = 0x7f48e63e5f30 "daffy.net", strip_realm = false, auth_pool = 0x7f48e6471770, acct_pool = 0x7f48e6471c60, coa_pool = 0x0} (gdb) p *realm->acct_pool->servers[i] Cannot access memory at address 0x0
Huh?
Just to confirm, can you do:
(gdb) p *realm->acct_pool (gdb) p *realm->acct_pool->servers
That's weird. Either the accounting pool has no servers, or it's somehow managed to walk off of the end of the list of accounting servers.
(gdb) p *realm->acct_pool $2 = {name = 0x7f48e63e5f30 "daffy.net", type = HOME_POOL_FAIL_OVER, server_type = 2, cs = 0x7f48e63e5dc0, virtual_server = 0x0, fallback = 0x0, in_fallback = 0, time_all_dead = 0, num_home_servers = 3, servers = { 0x7f48e64718d0}} (gdb) p *realm->acct_pool->servers $3 = (home_server_t *) 0x7f48e64718d0
On Oct 3, 2016, at 3:02 PM, Mike Ely <me@mikeely.org> wrote:
(gdb) p *realm->acct_pool $2 = {name = 0x7f48e63e5f30 "daffy.net", type = HOME_POOL_FAIL_OVER, server_type = 2, cs = 0x7f48e63e5dc0, virtual_server = 0x0, fallback = 0x0, in_fallback = 0, time_all_dead = 0, num_home_servers = 3, servers = { 0x7f48e64718d0}} (gdb) p *realm->acct_pool->servers $3 = (home_server_t *) 0x7f48e64718d0
And one last one... (gdb) p i My guess is that this will print "3", or "4". It looks like something completely catastrophic is happening. The pool is *supposed* to have 3 servers. But for some reason, the array entry is NULL, *or* the code is walking off of the end of the array somehow. That shouldn't happen. i.e. it's a catastrophic error that signals something completely bizarre is going on. I've never seen this before. Alan DeKok.
On 10/03/2016 12:20 PM, Alan DeKok wrote:
On Oct 3, 2016, at 3:02 PM, Mike Ely <me@mikeely.org> wrote:
(gdb) p *realm->acct_pool $2 = {name = 0x7f48e63e5f30 "daffy.net", type = HOME_POOL_FAIL_OVER, server_type = 2, cs = 0x7f48e63e5dc0, virtual_server = 0x0, fallback = 0x0, in_fallback = 0, time_all_dead = 0, num_home_servers = 3, servers = { 0x7f48e64718d0}} (gdb) p *realm->acct_pool->servers $3 = (home_server_t *) 0x7f48e64718d0
And one last one...
(gdb) p i
My guess is that this will print "3", or "4".
It looks like something completely catastrophic is happening. The pool is *supposed* to have 3 servers. But for some reason, the array entry is NULL, *or* the code is walking off of the end of the array somehow.
That shouldn't happen. i.e. it's a catastrophic error that signals something completely bizarre is going on. I've never seen this before.
(gdb) p i $4 = 1 Welcome to Monday.
On Oct 3, 2016, at 3:26 PM, Mike Ely <me@mikeely.org> wrote:
(gdb) p i $4 = 1
Welcome to Monday.
Well.. the code already checks that the array entries are non-NULL. So the array is fine when the server starts. If it's NULL later, then something else is going on. Maybe there's an unrelated error which smashes the array entry. But as I said, I've never seen this. The unusual (but possible) reason could be that you have a problem with the system RAM. i.e. it's failing. I've seen that before. Alan DeKok.
On 10/03/2016 12:30 PM, Alan DeKok wrote:
Well.. the code already checks that the array entries are non-NULL. So the array is fine when the server starts.
If it's NULL later, then something else is going on. Maybe there's an unrelated error which smashes the array entry.
But as I said, I've never seen this. The unusual (but possible) reason could be that you have a problem with the system RAM. i.e. it's failing.
I've seen that before.
Maybe an arch problem? We get the same problem on both hosts (a/b configuration) so it's unlikely they both have failed sticks of RAM. They are running with Atom D525 processors. Here's /proc/cpuinfo (2 sockets, total 4 cores): processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 28 model name : Intel(R) Atom(TM) CPU D525 @ 1.80GHz stepping : 10 microcode : 0x107 cpu MHz : 1795.613 cache size : 512 KB physical id : 0 siblings : 4 core id : 0 cpu cores : 2 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl tm2 ssse3 cx16 xtpr pdcm movbe lahf_lm dtherm bogomips : 3591.22 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management:
On Oct 3, 2016, at 4:15 PM, Mike Ely <me@mikeely.org> wrote:
Maybe an arch problem? We get the same problem on both hosts (a/b configuration) so it's unlikely they both have failed sticks of RAM.
They are running with Atom D525 processors. Here's /proc/cpuinfo (2 sockets, total 4 cores):
They're x86 machines. Which means they should work. Maybe it's a compiler bug... Alan DeKok.
On 10/03/2016 01:21 PM, Alan DeKok wrote:
They're x86 machines. Which means they should work.
Maybe it's a compiler bug...
That's scary to contemplate. I first saw this issue on a version compiled by CentOS and now it happens on an updated version compiled on our own hardware (with just those two changes added).
I wanted to ask almost the same. What are compiler options? Possibly this: https://github.com/rust-lang/rust/issues/2078 r. On 10/03/2016 10:21 PM, Alan DeKok wrote:
On Oct 3, 2016, at 4:15 PM, Mike Ely <me@mikeely.org> wrote:
Maybe an arch problem? We get the same problem on both hosts (a/b configuration) so it's unlikely they both have failed sticks of RAM.
They are running with Atom D525 processors. Here's /proc/cpuinfo (2 sockets, total 4 cores):
They're x86 machines. Which means they should work.
Maybe it's a compiler bug...
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On 10/03/2016 01:23 PM, Rasto Rickardt wrote:
I wanted to ask almost the same. What are compiler options?
Possibly this:
https://github.com/rust-lang/rust/issues/2078
r.
On 10/03/2016 10:21 PM, Alan DeKok wrote:
On Oct 3, 2016, at 4:15 PM, Mike Ely <me@mikeely.org> wrote:
Maybe an arch problem? We get the same problem on both hosts (a/b configuration) so it's unlikely they both have failed sticks of RAM.
They are running with Atom D525 processors. Here's /proc/cpuinfo (2 sockets, total 4 cores):
They're x86 machines. Which means they should work.
Maybe it's a compiler bug...
Not (from a glance anyhow) the same problem as the rust-lang people were seeing, but from another thread it's possible it needs built with -mcpu=atom. Here are the options for the segfaulting version: # rpm -q --queryformat="%{NAME}: %{OPTFLAGS}\n" freeradius freeradius: -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic This looks interesting (linked from Rasto's thread): http://permalink.gmane.org/gmane.comp.compilers.llvm.cvs/112078 Thoughts?
---- Mike Ely wrote ----
On 10/03/2016 01:23 PM, Rasto Rickardt wrote:
I wanted to ask almost the same. What are compiler options?
Possibly this:
https://github.com/rust-lang/rust/issues/2078
r.
On 10/03/2016 10:21 PM, Alan DeKok wrote:
On Oct 3, 2016, at 4:15 PM, Mike Ely <me@mikeely.org> wrote:
Maybe an arch problem? We get the same problem on both hosts (a/b configuration) so it's unlikely they both have failed sticks of RAM.
They are running with Atom D525 processors. Here's /proc/cpuinfo (2 sockets, total 4 cores):
They're x86 machines. Which means they should work.
Maybe it's a compiler bug...
Not (from a glance anyhow) the same problem as the rust-lang people were seeing, but from another thread it's possible it needs built with -mcpu=atom. Here are the options for the segfaulting version:
# rpm -q --queryformat="%{NAME}: %{OPTFLAGS}\n" freeradius freeradius: -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic
This looks interesting (linked from Rasto's thread): http://permalink.gmane.org/gmane.comp.compilers.llvm.cvs/112078
Thoughts?
I guess that Atom is missing something, L2/L3 cache or some instructions, so the generic option might not be the right one, as it changes with every gcc release. -mtune native or atom you suggested might help. I might even go to core2 as it was the first architecture with 64bit features. r.
On 10/03/2016 12:20 PM, Alan DeKok wrote:
On Oct 3, 2016, at 3:02 PM, Mike Ely <me@mikeely.org> wrote:
(gdb) p *realm->acct_pool $2 = {name = 0x7f48e63e5f30 "daffy.net", type = HOME_POOL_FAIL_OVER, server_type = 2, cs = 0x7f48e63e5dc0, virtual_server = 0x0, fallback = 0x0, in_fallback = 0, time_all_dead = 0, num_home_servers = 3, servers = { 0x7f48e64718d0}} (gdb) p *realm->acct_pool->servers $3 = (home_server_t *) 0x7f48e64718d0
And one last one...
(gdb) p i
My guess is that this will print "3", or "4".
It looks like something completely catastrophic is happening. The pool is *supposed* to have 3 servers. But for some reason, the array entry is NULL, *or* the code is walking off of the end of the array somehow.
That shouldn't happen. i.e. it's a catastrophic error that signals something completely bizarre is going on. I've never seen this before.
So I'm recompiling these packages on the atom with the "right" CPU optimizations and will report success or failure once I know. In the meantime I figured out the triggering condition: The third and final authhost for "daffy.net" is down. When that server gets tried, the segfault occurs. Which is at least nice in that I can now trigger the problem at will. Like I say, I'll let you know when I have answered whether the issue is fixed by different compile options.
So I'm recompiling these packages on the atom with the "right" CPU optimizations and will report success or failure once I know. In the meantime I figured out the triggering condition:
The third and final authhost for "daffy.net" is down. When that server gets tried, the segfault occurs. Which is at least nice in that I can now trigger the problem at will.
Like I say, I'll let you know when I have answered whether the issue is fixed by different compile options.
When i was on 3.0.4 from Centos7 repos, i encountered bunch of mysterious segfaults, but everything solved when i build 3.0.11. I did not changed the config and they disappeared so it was my fault not to use recent version :). r.
Hi,
a working 2.x system that proxies numerous devices. In order to get the migrated configs to work I had to change the user and group stanzas to within a security{} block and also get rid of calls to the no longer used rlm_acct_unique module.
well, yes....the configuration files and options are different in many places. to be honest, if its just proxying then the configs are quite simple and you just have a lot of entries in proxy.conf and clients.conf (or some other file(s) that you write to for local config reasons. I know of several FR instances where there are > 300 different realms with their associated 2 or 3 remote servers in the proxy.conf (and those servers also in clients.conf). alan
participants (5)
-
A.L.M.Buxey＠lboro.ac.uk -
Alan DeKok -
Mike Ely -
Rasto Rickardt -
riki