Bad talloc magic value
Hi, I’m seeing a lot of crashes after upgrading to the NR packaged versions of FreeRADIUS 3.0.23: Tue Jul 6 03:46:10 2021 : Auth: (145) Login OK: [anonymous@realm] (from client roaming1 port 0 cli 70-6F-6C-69-73-68) Bad talloc magic value - unknown value talloc abort: Bad talloc magic value - unknown value Tue Jul 6 03:46:24 2021 : Info: Debugger not attached Tue Jul 6 03:46:24 2021 : Info: systemd watchdog is disabled 3.0.22 ran nicely, but now I’m concerned about 50 crashes with the above message since 3am this morning... Any ideas, gents? *looks at Arran, Alan and Matthew expectantly* I’ve rolled back manually to 3.0.22 as a precaution. With Kind Regards Stefan Paetow Federated Roaming Technical Specialist t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet In line with government advice, at Jisc we’re now working from home and our offices are currently closed. Read our statement on coronavirus<https://www.jisc.ac.uk/about/corporate/coronavirus-statement>. jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800.
On Jul 6, 2021, at 7:19 AM, Stefan Paetow via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote:
I’m seeing a lot of crashes after upgrading to the NR packaged versions of FreeRADIUS 3.0.23:
Tue Jul 6 03:46:10 2021 : Auth: (145) Login OK: [anonymous@realm] (from client roaming1 port 0 cli 70-6F-6C-69-73-68) Bad talloc magic value - unknown value
talloc abort: Bad talloc magic value - unknown value
Tue Jul 6 03:46:24 2021 : Info: Debugger not attached Tue Jul 6 03:46:24 2021 : Info: systemd watchdog is disabled
3.0.22 ran nicely, but now I’m concerned about 50 crashes with the above message since 3am this morning... Any ideas, gents? *looks at Arran, Alan and Matthew expectantly*
We have a patch into the v3.0.x branch. Matthew is looking to release updated packages for 3.0.23. Alan DeKok.
On 7/6/21 1:58 PM, Alan DeKok wrote:
On Jul 6, 2021, at 7:19 AM, Stefan Paetow via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote:
I’m seeing a lot of crashes after upgrading to the NR packaged versions of FreeRADIUS 3.0.23:
Tue Jul 6 03:46:10 2021 : Auth: (145) Login OK: [anonymous@realm] (from client roaming1 port 0 cli 70-6F-6C-69-73-68) Bad talloc magic value - unknown value
talloc abort: Bad talloc magic value - unknown value
Tue Jul 6 03:46:24 2021 : Info: Debugger not attached Tue Jul 6 03:46:24 2021 : Info: systemd watchdog is disabled
3.0.22 ran nicely, but now I’m concerned about 50 crashes with the above message since 3am this morning... Any ideas, gents? *looks at Arran, Alan and Matthew expectantly*> We have a patch into the v3.0.x branch. Matthew is looking to release updated packages for 3.0.23.
A back-port patch other downstream packagers should also use? If yes, which one? Ciao, Michael.
On 07/07/2021 08:16, Michael Ströder via Freeradius-Users wrote:
On 7/6/21 1:58 PM, Alan DeKok wrote:
On Jul 6, 2021, at 7:19 AM, Stefan Paetow via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote: We have a patch into the v3.0.x branch. Matthew is looking to release updated packages for 3.0.23.
A back-port patch other downstream packagers should also use? If yes, which one?
8e204e3b6a and 4d4af808a7 should do it. -- Matthew
Ahhh, excellent. Thank you very much! :-) Stefan Paetow Federated Roaming Technical Specialist t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet In line with government advice, at Jisc we’re now working from home and our offices are currently closed. Read our statement on coronavirus <https://www.jisc.ac.uk/about/corporate/coronavirus-statement>. jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. On 06/07/2021, 12:59, "Freeradius-Users on behalf of Alan DeKok" <freeradius-users-bounces+stefan.paetow=jisc.ac.uk@lists.freeradius.org on behalf of aland@deployingradius.com> wrote: On Jul 6, 2021, at 7:19 AM, Stefan Paetow via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote: > I’m seeing a lot of crashes after upgrading to the NR packaged versions of FreeRADIUS 3.0.23: > > Tue Jul 6 03:46:10 2021 : Auth: (145) Login OK: [anonymous@realm] (from client roaming1 port 0 cli 70-6F-6C-69-73-68) > Bad talloc magic value - unknown value > > talloc abort: Bad talloc magic value - unknown value > > Tue Jul 6 03:46:24 2021 : Info: Debugger not attached > Tue Jul 6 03:46:24 2021 : Info: systemd watchdog is disabled > > 3.0.22 ran nicely, but now I’m concerned about 50 crashes with the above message since 3am this morning... Any ideas, gents? *looks at Arran, Alan and Matthew expectantly* We have a patch into the v3.0.x branch. Matthew is looking to release updated packages for 3.0.23. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Sorry gents, But I've just updated to 3.0.23-2 and I still see this error: Wed Jul 28 10:12:01 2021 : Auth: (186) Login OK: [login@realm] (from client x port 0 cli 02-00-00-00-00-01) Bad talloc magic value - unknown value talloc abort: Bad talloc magic value - unknown value Wed Jul 28 10:12:11 2021 : Info: Debugger not attached Wed Jul 28 10:12:11 2021 : Info: systemd watchdog is disabled Wed Jul 28 10:12:11 2021 : Info: rlm_sql (sql): Driver rlm_sql_postgresql (module rlm_sql_postgresql) loaded and linked This is on RedHat 7. I am seeing it every 2-3 minutes now. Wed Jul 28 10:05:11 2021 : Info: Debugger not attached Wed Jul 28 10:08:18 2021 : Info: Debugger not attached Wed Jul 28 10:09:40 2021 : Info: Debugger not attached Wed Jul 28 10:11:01 2021 : Info: Debugger not attached Wed Jul 28 10:12:11 2021 : Info: Debugger not attached Wed Jul 28 10:13:19 2021 : Info: Debugger not attached Wed Jul 28 10:16:15 2021 : Info: Debugger not attached Wed Jul 28 10:17:38 2021 : Info: Debugger not attached For any avoidance of doubt, ldap module is not enabled (so nss is not a factor). I've returned to 3.0.22 (again). With Kind Regards Stefan Paetow Federated Roaming Technical Specialist t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet In line with government advice, at Jisc we’re now working from home and our offices are currently closed. Read our statement on coronavirus <https://www.jisc.ac.uk/about/corporate/coronavirus-statement>. jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. On 07/07/2021, 21:36, "Freeradius-Users on behalf of Stefan Paetow via Freeradius-Users" <freeradius-users-bounces+stefan.paetow=jisc.ac.uk@lists.freeradius.org on behalf of freeradius-users@lists.freeradius.org> wrote: Ahhh, excellent. Thank you very much! :-) Stefan Paetow Federated Roaming Technical Specialist t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet In line with government advice, at Jisc we’re now working from home and our offices are currently closed. Read our statement on coronavirus <https://www.jisc.ac.uk/about/corporate/coronavirus-statement>. jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. On 06/07/2021, 12:59, "Freeradius-Users on behalf of Alan DeKok" <freeradius-users-bounces+stefan.paetow=jisc.ac.uk@lists.freeradius.org on behalf of aland@deployingradius.com> wrote: On Jul 6, 2021, at 7:19 AM, Stefan Paetow via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote: > I’m seeing a lot of crashes after upgrading to the NR packaged versions of FreeRADIUS 3.0.23: > > Tue Jul 6 03:46:10 2021 : Auth: (145) Login OK: [anonymous@realm] (from client roaming1 port 0 cli 70-6F-6C-69-73-68) > Bad talloc magic value - unknown value > > talloc abort: Bad talloc magic value - unknown value > > Tue Jul 6 03:46:24 2021 : Info: Debugger not attached > Tue Jul 6 03:46:24 2021 : Info: systemd watchdog is disabled > > 3.0.22 ran nicely, but now I’m concerned about 50 crashes with the above message since 3am this morning... Any ideas, gents? *looks at Arran, Alan and Matthew expectantly* We have a patch into the v3.0.x branch. Matthew is looking to release updated packages for 3.0.23. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Hi, just to add to Stefan and Max, we've also witnessed the same behaviour with centos 7 after updating to both the 3.0.23 and 3.0.23-2 packages but haven't had a chance to do any further investigation as yet. Regards, Glen Glen Fallon Network Officer NUIT Newcastle University -----Original Message----- From: Freeradius-Users <freeradius-users-bounces+freeradius=jgf.org.uk@lists.freeradius.org> On Behalf Of Stefan Paetow via Freeradius-Users Sent: 28 July 2021 11:43 To: FreeRadius users mailing list <freeradius-users@lists.freeradius.org> Cc: Stefan Paetow <Stefan.Paetow@jisc.ac.uk> Subject: Re: Bad talloc magic value Sorry gents, But I've just updated to 3.0.23-2 and I still see this error: Wed Jul 28 10:12:01 2021 : Auth: (186) Login OK: [login@realm] (from client x port 0 cli 02-00-00-00-00-01) Bad talloc magic value - unknown value talloc abort: Bad talloc magic value - unknown value Wed Jul 28 10:12:11 2021 : Info: Debugger not attached Wed Jul 28 10:12:11 2021 : Info: systemd watchdog is disabled Wed Jul 28 10:12:11 2021 : Info: rlm_sql (sql): Driver rlm_sql_postgresql (module rlm_sql_postgresql) loaded and linked This is on RedHat 7. I am seeing it every 2-3 minutes now. Wed Jul 28 10:05:11 2021 : Info: Debugger not attached Wed Jul 28 10:08:18 2021 : Info: Debugger not attached Wed Jul 28 10:09:40 2021 : Info: Debugger not attached Wed Jul 28 10:11:01 2021 : Info: Debugger not attached Wed Jul 28 10:12:11 2021 : Info: Debugger not attached Wed Jul 28 10:13:19 2021 : Info: Debugger not attached Wed Jul 28 10:16:15 2021 : Info: Debugger not attached Wed Jul 28 10:17:38 2021 : Info: Debugger not attached For any avoidance of doubt, ldap module is not enabled (so nss is not a factor). I've returned to 3.0.22 (again). With Kind Regards Stefan Paetow Federated Roaming Technical Specialist t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet In line with government advice, at Jisc we’re now working from home and our offices are currently closed. Read our statement on coronavirus <https://www.jisc.ac.uk/about/corporate/coronavirus-statement>. jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. On 07/07/2021, 21:36, "Freeradius-Users on behalf of Stefan Paetow via Freeradius-Users" <freeradius-users-bounces+stefan.paetow=jisc.ac.uk@lists.freeradius.org on behalf of freeradius-users@lists.freeradius.org> wrote: Ahhh, excellent. Thank you very much! :-) Stefan Paetow Federated Roaming Technical Specialist t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet In line with government advice, at Jisc we’re now working from home and our offices are currently closed. Read our statement on coronavirus <https://www.jisc.ac.uk/about/corporate/coronavirus-statement>. jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. On 06/07/2021, 12:59, "Freeradius-Users on behalf of Alan DeKok" <freeradius-users-bounces+stefan.paetow=jisc.ac.uk@lists.freeradius.org on behalf of aland@deployingradius.com> wrote: On Jul 6, 2021, at 7:19 AM, Stefan Paetow via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote: > I’m seeing a lot of crashes after upgrading to the NR packaged versions of FreeRADIUS 3.0.23: > > Tue Jul 6 03:46:10 2021 : Auth: (145) Login OK: [anonymous@realm] (from client roaming1 port 0 cli 70-6F-6C-69-73-68) > Bad talloc magic value - unknown value > > talloc abort: Bad talloc magic value - unknown value > > Tue Jul 6 03:46:24 2021 : Info: Debugger not attached > Tue Jul 6 03:46:24 2021 : Info: systemd watchdog is disabled > > 3.0.22 ran nicely, but now I’m concerned about 50 crashes with the above message since 3am this morning... Any ideas, gents? *looks at Arran, Alan and Matthew expectantly* We have a patch into the v3.0.x branch. Matthew is looking to release updated packages for 3.0.23. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On 28/07/2021 11:49, freeradius@jgf.org.uk wrote:
Hi, just to add to Stefan and Max, we've also witnessed the same behaviour with centos 7 after updating to both the 3.0.23 and 3.0.23-2 packages but haven't had a chance to do any further investigation as yet.
Gah, "thanks". We replicated the bug and did loads of testing, and it looked good, and I know people are running the latest packages successfully in production, where 3.0.23 failed. Can anyone try v3.0.x HEAD? I can make packages available. If it's crashing as quickly as that then it should be a fast test... Getting debugger output using the "panic_action" option would be really helpful, too. -- Matthew
Hi Matthew Looks like I have some time on Monday afternoon I could try this, if no-one else gets back to you. I'm using CentOS 7 Regards Max -----Original Message----- From: Freeradius-Users <freeradius-users-bounces+max.caines=wlv.ac.uk@lists.freeradius.org> On Behalf Of Matthew Newton Sent: 28 July 2021 11:58 To: freeradius-users@lists.freeradius.org Subject: Re: Bad talloc magic value On 28/07/2021 11:49, freeradius@jgf.org.uk wrote:
Hi, just to add to Stefan and Max, we've also witnessed the same behaviour with centos 7 after updating to both the 3.0.23 and 3.0.23-2 packages but haven't had a chance to do any further investigation as yet.
Gah, "thanks". We replicated the bug and did loads of testing, and it looked good, and I know people are running the latest packages successfully in production, where 3.0.23 failed. Can anyone try v3.0.x HEAD? I can make packages available. If it's crashing as quickly as that then it should be a fast test... Getting debugger output using the "panic_action" option would be really helpful, too. -- Matthew - List info/subscribe/unsubscribe? See https://url6.mailanyone.net/v1/?m=1m8hGk-0002KH-5J&i=57e1b682&c=9YA--3GVwFUI... ------------------------------------ This email has been scanned for spam & viruses. If you believe this email should have been stopped by our filters, click the following link to report it (https://portal.mailanyone.net/index.html#/outer/reportspam?token=dXNlcj1tYXg...).
On 30/07/2021 12:35, Caines, Max wrote:
Looks like I have some time on Monday afternoon I could try this, if no-one else gets back to you. I'm using CentOS 7 3.0.23-3 is out and fixes this. It was an issue in the CentOS packaging.
Apart from rebuilding there have been no other changes. New Debian and Ubuntu packages are released for consistency, but are identical to -2. Thanks to Max for testing. -- Matthew
Hi Matthew, just to note we upgraded one of the servers that had this issue with the previous package this morning and all looks good so far. Thanks for the efforts of everyone with this Regards, Glen Glen Fallon Network Officer NUIT Newcastle University -----Original Message----- From: Freeradius-Users <freeradius-users-bounces+freeradius=jgf.org.uk@lists.freeradius.org> On Behalf Of Matthew Newton Sent: 03 August 2021 19:53 To: freeradius-users@lists.freeradius.org Subject: Re: Bad talloc magic value On 30/07/2021 12:35, Caines, Max wrote:
Looks like I have some time on Monday afternoon I could try this, if no-one else gets back to you. I'm using CentOS 7 3.0.23-3 is out and fixes this. It was an issue in the CentOS packaging.
Apart from rebuilding there have been no other changes. New Debian and Ubuntu packages are released for consistency, but are identical to -2. Thanks to Max for testing. -- Matthew - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On 04/08/2021 11:53, freeradius@jgf.org.uk wrote:
Hi Matthew, just to note we upgraded one of the servers that had this issue with the previous package this morning and all looks good so far. Thanks for the efforts of everyone with this
Sounds good. Thanks for the update! -- Matthew
participants (6)
-
Alan DeKok -
Caines, Max -
freeradius@jgf.org.uk -
Matthew Newton -
Michael Ströder -
Stefan Paetow