FR-1.1.2 dies with error
Hi all. We have some trouble with fr-1.1.2 & Oracle-9.2.0.6 & Solaris 9. The process dies periodically with the error: Thu Aug 3 14:27:43 2006 : Error: Assertion failed in request_list.c, line 1012 FR is built with the following configuration: ./configure \ --with-ltdl-lib=libltdl \ --with-ltdl-include=libltdl \ --enable-ltdl-install=yes \ --with-threads \ --without-openssl \ --enable-developer \ --prefix=/opt/fr It did not dump core, thouigh i enabled it in radiusd.conf and said "ulimit -c unlimited". any suggestions? -- Sincerely Yours, Alexander
Alexander Serkin <als@cell.ru> wrote:
We have some trouble with fr-1.1.2 & Oracle-9.2.0.6 & Solaris 9. The process dies periodically with the error:
Thu Aug 3 14:27:43 2006 : Error: Assertion failed in request_list.c, line 1012
It's probably because your DB is slow. See the logs for messages about "unresponsive child". The code path to the assertion is taken only when the request has been marked "done", but there is still a child thread blocked, and working on it. Either fix the DB, or delete the assertion. But if you delete the assertion, odds are that something else will go wrong elsewhere. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Alan DeKok пишет:
It's probably because your DB is slow. See the logs for messages about "unresponsive child". The code path to the assertion is taken only when the request has been marked "done", but there is still a child thread blocked, and working on it.
Either fix the DB, or delete the assertion. But if you delete the assertion, odds are that something else will go wrong elsewhere.
Do you mean just comment out line 1012 in request_list.c ? I think i'll try this first because speeding up DB is not a trivial task by now. Thanks for the hints, -- Sincerely Yours, Alexander
Alexander Serkin <als@cell.ru> wrote:
Do you mean just comment out line 1012 in request_list.c ?
Yes.
I think i'll try this first because speeding up DB is not a trivial task by now.
But it's the real source of the problem... Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
--- Alan DeKok <aland@deployingradius.com> wrote:
Alexander Serkin <als@cell.ru> wrote:
Do you mean just comment out line 1012 in request_list.c ?
Yes.
I think i'll try this first because speeding up DB is not a trivial task by now.
But it's the real source of the problem...
How large a DB is this? And what type of link is there between FR and the DB? Unless there are, literally, (tens of) thousands of records and/or a *slow* link (think "dial-up") and/or ancient hardware there should be some reasonable ways to speed up the DB response. Archiving of records and indexing are two that come to mind first. More complicated, but effective, would be clustering or optimization, even review of the DB version (deprecated?). Alan is correct, you are "fixing" (hiding) a symptom, and I can say from personal experience it *will* bite you in the butt at some point :) The worst part of it, too, will be that the new issue may not be clearly linkable back to the FR problem you have currently and you may not remember this piece of the puzzle. Patching also breaks your upgrade path... Just some friendly advice. Laker
Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
Laker Netman wrote:
How large a DB is this? And what type of link is there between FR and the DB?
It's about 36 million records since april 2005.
Unless there are, literally, (tens of) thousands of records and/or a *slow* link (think "dial-up") and/or ancient hardware there should be some reasonable ways to speed up the DB response. Archiving of records and indexing are two that come to mind first. More complicated, but effective, would be clustering or optimization, even review of the DB version (deprecated?).
I was partially wrong with the environment description. The authentication DB is very small (less than 10000 records in all the tables). It is local on Sun Netra 1120 (2x440MHz) and Oracle 9.2.0.6. It serves about 2 to 5 radius requests per second. And the accounting DB is located on remote server (HP DL380 3GHz, Red Hat Enterprise with Oracle 10.2.0.1), connected to AAA server via 100BaseT link (loaded by 1-5%). The accounting process takes up to 25 requests per second. I suppose this is what bites the radius process periodically.
Alan is correct, you are "fixing" (hiding) a symptom, and I can say from personal experience it *will* bite you in the butt at some point :) The worst part of it, too, will be that the new issue may not be clearly linkable back to the FR problem you have currently and you may not remember this piece of the puzzle.
You are definitely right. We'll consider archiving. Indexing is already done on all the columns taking part in "where" clauses. Commenting rad_assert is just a temporary solution. Just for me to spend weekend with my friends and some beer. And not to be awaken in the night by damned SMS from dead AAA process :-) Thanks, -- Alexander
Hi all, Alan DeKok wrote:
Alexander Serkin <als@cell.ru> wrote:
Do you mean just comment out line 1012 in request_list.c ?
Yes.
I think i'll try this first because speeding up DB is not a trivial task by now.
But it's the real source of the problem...
I'm still trying to investigate the problem with one of my AAA servers. Is it possible to tell on wich request assertion fails? I mean authentication or accounting. We have two servers and use load-balancing between them configured on NASes. But assertion fails periodically with only one of the servers mostly on line 1012 of request_list.c, but sometimes on line 1039 of the same file. -- Sincerely Yours, Alexander
Is there a way to have netflow data per session, instead of just total "octets in" and total "octets out"? I am trying to find a relatively easy way to charge users per netflow data, for example: local data is 50% discount, mail is 30% discount etc. Thank you, Igor
"Igor Smitran" <igor@poen.net> wrote:
Is there a way to have netflow data per session, instead of just total "octets in" and total "octets out"? I am trying to find a relatively easy way to charge users per netflow data, for example: local data is 50% discount, mail is 30% discount etc.
Consult the NAS documentation. If it doesn't say it can send that information, then that information won't be available to FreeRADIUS. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Is there a way to have netflow data per session, instead of just total "octets in" and total "octets out"? I am trying to find a relatively easy way to charge users per netflow data, for example: local data is 50% discount, mail is 30% discount etc.
Consult the NAS documentation. If it doesn't say it can send that information, then that information won't be available to FreeRADIUS.
Alan DeKok.
I have Mikrotik. It can export netflow data but i am not sure what freeradius can do with that? Is it possible to have all netflow for that client inserted into database somehow? Please provide some URL because i am not sure what to search for. Igor
"Igor Smitran" <igor@poen.net> wrote:
I have Mikrotik. It can export netflow data but i am not sure what freeradius can do with that?
Nothing. You will need a netflow server.
Is it possible to have all netflow for that client inserted into database somehow? Please provide some URL because i am not sure what to search for.
"netflow server" ? Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
"Igor Smitran" <igor@poen.net> wrote:
I have Mikrotik. It can export netflow data but i am not sure what freeradius can do with that?
Nothing. You will need a netflow server.
Is it possible to have all netflow for that client inserted into database somehow? Please provide some URL because i am not sure what to search for.
"netflow server" ?
Alan DeKok.
Yes, i know about cflowd and similar netflow tools. I was thinking that maybe there is some solution that can help me to insert flow data for particular user into database together with total octets in, octets out upon disconnect. Thank you, Igor
Igor Smitran wrote:
"netflow server" ?
Alan DeKok.
Yes, i know about cflowd and similar netflow tools. I was thinking that maybe there is some solution that can help me to insert flow data for particular user into database together with total octets in, octets out upon disconnect.
There is no pre-built solution. You would need to roll your own. It's not hard. See the "Exec-Program" examples in acct_users, for examples how to trigger external code/events on accounting start/stop.
On Tue 15 Aug 2006 15:51, Igor Smitran wrote:
"Igor Smitran" <igor@poen.net> wrote:
I have Mikrotik. It can export netflow data but i am not sure what freeradius can do with that?
Nothing. You will need a netflow server.
Is it possible to have all netflow for that client inserted into database somehow? Please provide some URL because i am not sure what to search for.
"netflow server" ?
Alan DeKok.
Yes, i know about cflowd and similar netflow tools. I was thinking that maybe there is some solution that can help me to insert flow data for particular user into database together with total octets in, octets out upon disconnect.
Aside from tinkering with FreeRADIUS code (and running a large number of production servers) I also tinker with and run pmacct which I highly recommend as a netflow/sflow solution. We have a number of deployments of both on the same Postgresql backend and as long as your DB server is specced correctly you shouldn't have any trouble. My pmacct rpms are at: http://software.opensuse.org/download/server:/monitoring/ And my FreeRADIUS rpms are at: http://software.opensuse.org/download/home:/peternixon/ Integration of the 2 different sets of accounting data is left as an excercise to you :-) Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
Peter Nixon <listuser@peternixon.net> wrote:
Aside from tinkering with FreeRADIUS code (and running a large number of production servers) I also tinker with and run pmacct which I highly recommend as a netflow/sflow solution. We have a number of deployments of both on the same Postgresql backend and as long as your DB server is specced correctly you shouldn't have any trouble.
That looks like a fantastic tool, which should be mentioned in the FAQ, as "how to get protocol-specific accounting information". I we were suckers for punishment, we could write a radius plugin for pmacct, so that the RADIUS server could see that traffic, too. But it's probably better to integrate things at the DB layer, rather than the protocol layer. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Peter Nixon <listuser@peternixon.net> wrote:
Aside from tinkering with FreeRADIUS code (and running a large number of production servers) I also tinker with and run pmacct which I highly recommend as a netflow/sflow solution. We have a number of deployments of both on the same Postgresql backend and as long as your DB server is specced correctly you shouldn't have any trouble.
That looks like a fantastic tool, which should be mentioned in the FAQ, as "how to get protocol-specific accounting information".
I we were suckers for punishment, we could write a radius plugin for pmacct, so that the RADIUS server could see that traffic, too. But it's probably better to integrate things at the DB layer, rather than the protocol layer.
Alan DeKok.
It would be good to have all data imidiately accessible, that way one can use exec-wait and do accounting imidiately upon disconnect? Or am i missing the point? :) i am just trying to share some ideas and do some brain storming. My idea was something like this: 1. user tries to authenticate 2. radius authenticates user and starts accounting 3. radius pulls netflow data for particular IP in some time intervals and inserts those into some database table I am not very familiar with freeradius. I've seted it up to do what i want but don't have time to learn more :( so if i am missing the point please let me know :) Igor
On Tue 15 Aug 2006 19:21, Igor Smitran wrote:
Peter Nixon <listuser@peternixon.net> wrote:
Aside from tinkering with FreeRADIUS code (and running a large number of production servers) I also tinker with and run pmacct which I highly recommend as a netflow/sflow solution. We have a number of deployments of both on the same Postgresql backend and as long as your DB server is specced correctly you shouldn't have any trouble.
That looks like a fantastic tool, which should be mentioned in the FAQ, as "how to get protocol-specific accounting information".
I we were suckers for punishment, we could write a radius plugin for pmacct, so that the RADIUS server could see that traffic, too. But it's probably better to integrate things at the DB layer, rather than the protocol layer.
Alan DeKok.
It would be good to have all data imidiately accessible, that way one can use exec-wait and do accounting imidiately upon disconnect? Or am i missing the point? :)
i am just trying to share some ideas and do some brain storming. My idea was something like this: 1. user tries to authenticate 2. radius authenticates user and starts accounting 3. radius pulls netflow data for particular IP in some time intervals and inserts those into some database table
I am not very familiar with freeradius. I've seted it up to do what i want but don't have time to learn more :( so if i am missing the point please let me know :)
Igor, I think you may be a little confused about how netflow works. The router/switch pushes the netflow data to the collector, typically for ALL traffic through the device (although some devices support aggregation of flows and/or statistical sampling). You therefore don't trigger netflow records for a particular IP, you need to wait for the device to send them to you. You can configure flow timeouts and various things but basically in any realistic deployment you are still likely to receive netflow records for several minutes (at least) after traffic stops. I you make pmacct or other collector put flow data into the same postgresql database (different table of course) a simple database JOIN to the radacct table should give you the billing data you need whenever you want it :-) Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
Yes Peter, you are right. My fault. I only tried netflow tools, i never used those in production envrionment. I just checked and saw that i need to pull data from collector, while collector is receiving data from routers. That said it is not possible to have accurate data at disconnect. Sorry for this, i was hotheaded, but i can't help it, it defines me :) Igor
Alexander Serkin <als@cell.ru> wrote:
I'm still trying to investigate the problem with one of my AAA servers.
It's a problem with the DB, not with the server.
Is it possible to tell on wich request assertion fails?
No.
We have two servers and use load-balancing between them configured on NASes. But assertion fails periodically with only one of the servers mostly on line 1012 of request_list.c, but sometimes on line 1039 of the same file.
As you have been told already, the problem is that something is blocking the server, and preventing it from doing work. That is likely the DB used by the server. Any work you do on FreeRADIUS will do little more than hide the problem. You can't make your DB run faster by poking FreeRADIUS. And if the DB is down, there's *nothing* you can do to the server to make it process requests. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Alan DeKok wrote:
Alexander Serkin <als@cell.ru> wrote:
I'm still trying to investigate the problem with one of my AAA servers.
It's a problem with the DB, not with the server.
I understand this, Alan. My experiments with hiding assertion strings in request_list.c came into failure. And i feel that Oracle is not good production server for radius accounting. Or the DB structure is not optimal for our application. Does anybody in the list use the FR&Oracle for the systems serving about 4500 simultaneous connections? It's really not much, but... Will the accounting table partitioning help? -- Sincerely Yours, Alexander
Alexander Serkin <als@cell.ru> wrote:
And i feel that Oracle is not good production server for radius accounting. Or the DB structure is not optimal for our application.
I know of Oracle installations with 400k users. And the default schema works with installations of millions of users.
Does anybody in the list use the FR&Oracle for the systems serving about 4500 simultaneous connections? It's really not much, but... Will the accounting table partitioning help?
How big are your tables? Do you have gigabytes of historical data in them? Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Alan DeKok wrote:
Alexander Serkin <als@cell.ru> wrote:
And i feel that Oracle is not good production server for radius accounting. Or the DB structure is not optimal for our application.
I know of Oracle installations with 400k users. And the default schema works with installations of millions of users.
do you mean 400k active simultaneous connections? And do they use accounting updates (Interim-Update records)?
Does anybody in the list use the FR&Oracle for the systems serving about 4500 simultaneous connections? It's really not much, but... Will the accounting table partitioning help?
How big are your tables? Do you have gigabytes of historical data in them?
The table is about 37 million records since april 2005. -- Sincerely Yours, Alexander
Alexander Serkin <als@cell.ru> wrote:
I know of Oracle installations with 400k users. And the default schema works with installations of millions of users.
do you mean 400k active simultaneous connections? And do they use accounting updates (Interim-Update records)?
There are no simultaneous connections in RADIUS. I mean 400k users, many of whom are online at the same time, and sending accounting updates.
The table is about 37 million records since april 2005.
Is it indexed? And why do you have a years worth of data available to your RADIUS server? It's completely unnecessary. You may need a years worth of data available for your billing software, and long-term records, but those things don't need one second response time. Change your tables so that you have "current" data (i.e. last month or so) available to FreeRADIUS, and leave the year-old data in another table that the server doesn't use. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Alan DeKok wrote:
Alexander Serkin <als@cell.ru> wrote:
I know of Oracle installations with 400k users. And the default schema works with installations of millions of users. do you mean 400k active simultaneous connections? And do they use accounting updates (Interim-Update records)?
There are no simultaneous connections in RADIUS.
Oh, sorry. I didn't mean simultaneous connections. I meant the amount of online users. We have up to ~4500 online connections for ~170k subscribers.
I mean 400k users, many of whom are online at the same time, and sending accounting updates.
The table is about 37 million records since april 2005.
Is it indexed?
yes it is indexed by all columns which take part in "where" clauses of the select and update queries.
And why do you have a years worth of data available to your RADIUS server? It's completely unnecessary.
You may need a years worth of data available for your billing software, and long-term records, but those things don't need one second response time. Change your tables so that you have "current" data (i.e. last month or so) available to FreeRADIUS, and leave the year-old data in another table that the server doesn't use.
We will. This is subject to discuss with our DBAs. Thanks for your comments, Alan. -- Sincerely Yours, Alexander
Alexander Serkin wrote:
And i feel that Oracle is not good production server for radius accounting. Or the DB structure is not optimal for our application. Does anybody in the list use the FR&Oracle for the systems serving about 4500 simultaneous connections? It's really not much, but... Will the accounting table partitioning help?
Maybe you could use "radsqlrelay" for accounting data, so FreeRADIUS doesn't interact with the database anymore. See the manpages for rlm_sql_log(5) and radsqlrelay(8) for more details. -- Nicolas Baradakis
Nicolas Baradakis wrote:
Alexander Serkin wrote:
And i feel that Oracle is not good production server for radius accounting. Or the DB structure is not optimal for our application. Does anybody in the list use the FR&Oracle for the systems serving about 4500 simultaneous connections? It's really not much, but... Will the accounting table partitioning help?
Maybe you could use "radsqlrelay" for accounting data, so FreeRADIUS doesn't interact with the database anymore. See the manpages for rlm_sql_log(5) and radsqlrelay(8) for more details.
didn't think about this yet. Is it possible to run radsqlrelay similar to radrelay - when it feeds the sql log to db while the log is being written by radiusd? -- Sincerely Yours, Alexander
Alexander Serkin wrote:
Nicolas Baradakis wrote:
Maybe you could use "radsqlrelay" for accounting data, so FreeRADIUS doesn't interact with the database anymore. See the manpages for rlm_sql_log(5) and radsqlrelay(8) for more details.
didn't think about this yet. Is it possible to run radsqlrelay similar to radrelay - when it feeds the sql log to db while the log is being written by radiusd?
Oracle is not supported by radsqlrelay? # radsqlrelay -b CDMA -d oracle -h host -P 1521 -p pass -u user -x ./tst-sql error: SQL driver not supported yet: oracle This is very strange because i remeber that i used it once after some maintenance operations with our oracle DB. May be with freeradius-1.1.1. Something changed? -- Sincerely Yours, Alexander
Alexander Serkin wrote:
Nicolas Baradakis wrote:
Maybe you could use "radsqlrelay" for accounting data, so FreeRADIUS doesn't interact with the database anymore. See the manpages for rlm_sql_log(5) and radsqlrelay(8) for more details.
didn't think about this yet. Is it possible to run radsqlrelay similar to radrelay - when it feeds the sql log to db while the log is being written by radiusd?
You'll find an answer in the manpages mentioned earlier.
Oracle is not supported by radsqlrelay?
# radsqlrelay -b CDMA -d oracle -h host -P 1521 -p pass -u user -x ./tst-sql error: SQL driver not supported yet: oracle
There is no Oracle support yet, because I couldn't test it when I wrote radsqlrelay. As Perl DBI includes an Oracle driver, it should be trivial to add. It's just a 2 lines patch in script/radsqlrelay.
This is very strange because i remeber that i used it once after some maintenance operations with our oracle DB. May be with freeradius-1.1.1. Something changed?
radsqlrelay was added in version 1.1.0 and since then it was never changed. (you can check the CVS log if you want) -- Nicolas Baradakis
Nicolas Baradakis wrote:
radsqlrelay was added in version 1.1.0 and since then it was never changed. (you can check the CVS log if you want)
Yes. i was wrong. In my case i've just fed sqllog to sqlplus utility. Finally radsqlrelay works for me with oracle too after patching 2 strings as you said. At least this gives us a chance to free up the production db temporary for optimization. Thanks for the hint, Nicolas. -- Sincerely Yours, Alexander
Alexander Serkin wrote:
Finally radsqlrelay works for me with oracle too after patching 2 strings as you said.
Please create a patch with "diff -u radsqlrelay.orig radsqlrelay" and post it to the list. I'll add it in version 1.1.3. -- Nicolas Baradakis
Nicolas Baradakis wrote:
Alexander Serkin wrote:
Finally radsqlrelay works for me with oracle too after patching 2 strings as you said.
Please create a patch with "diff -u radsqlrelay.orig radsqlrelay" and post it to the list. I'll add it in version 1.1.3.
Here it is: --- radsqlrelay.orig 2006-08-16 15:40:58.220277000 +0400 +++ radsqlrelay 2006-08-16 17:53:20.151452000 +0400 @@ -156,6 +156,8 @@ $data_source = "DBI:mysql:database=$args{b};host=$args{h}"; } elsif (lc($args{d}) eq 'pg') { $data_source = "DBI:Pg:dbname=$args{b};host=$args{h}"; +} elsif (lc($args{d}) eq 'oracle') { + $data_source = "DBI:Oracle:$args{b}"; } else { print STDERR "error: SQL driver not supported yet: $args{d}\n"; exit 1; whith "-b db.domain.tld" i give the database description stored in $TNS_ADMIN/tnsnames.ora: db.domain.tld = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = db.domain.tld)(PORT = 1521)) ) (CONNECT_DATA = (SERVICE_NAME = <DB SID>) ) ) -- Sincerely Yours, Alexander
Alexander Serkin wrote:
Nicolas Baradakis wrote:
Please create a patch with "diff -u radsqlrelay.orig radsqlrelay" and post it to the list. I'll add it in version 1.1.3.
Here it is:
--- radsqlrelay.orig 2006-08-16 15:40:58.220277000 +0400 +++ radsqlrelay 2006-08-16 17:53:20.151452000 +0400 @@ -156,6 +156,8 @@ $data_source = "DBI:mysql:database=$args{b};host=$args{h}"; } elsif (lc($args{d}) eq 'pg') { $data_source = "DBI:Pg:dbname=$args{b};host=$args{h}"; +} elsif (lc($args{d}) eq 'oracle') { + $data_source = "DBI:Oracle:$args{b}"; } else { print STDERR "error: SQL driver not supported yet: $args{d}\n"; exit 1;
Added, thanks.
whith "-b db.domain.tld" i give the database description stored in $TNS_ADMIN/tnsnames.ora:
db.domain.tld = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = db.domain.tld)(PORT = 1521)) ) (CONNECT_DATA = (SERVICE_NAME = <DB SID>) ) )
I've added this to the radsqlrelay(8) manpage, too. -- Nicolas Baradakis
participants (7)
-
Alan DeKok -
Alexander Serkin -
Igor Smitran -
Laker Netman -
Nicolas Baradakis -
Peter Nixon -
Phil Mayers