Freeradius 2.1-1: failure modes
Guys, I have a question about different failure modes in FreeRadius, basically I want to differentiate between different scenarios of failures and return a proper response (if any) to NAS device. Failure example: Backend database is down Right now FreeRadius returns reject to NAS rlm_sql (sql): Failed to connect DB handle #8 rlm_sql (sql): reconnect failed, database down? rlm_sql_getvpdata: database query error [sql] SQL query error; rejecting user rlm_sql (sql): Released sql socket id: 8 ++[sql] returns fail Sending Access-Reject Say I do not want to return reject in this case and I want the switch to understand that radius is down and retry another radius server (a backup/failover). Because if I return reject the NAS device treats this as radius is UP and running. What is recommended behavior for Radius for: DB is down,Authentication infra is down(say kerberos is down)? Should it return REJECT or NOOP? Thank you very much for your responses. Leopold -- View this message in context: http://www.nabble.com/Freeradius-2.1-1%3A-failure-modes-tp22413666p22413666.... Sent from the FreeRadius - User mailing list archive at Nabble.com.
Hi,
Right now FreeRadius returns reject to NAS rlm_sql (sql): Failed to connect DB handle #8 rlm_sql (sql): reconnect failed, database down? rlm_sql_getvpdata: database query error [sql] SQL query error; rejecting user rlm_sql (sql): Released sql socket id: 8 ++[sql] returns fail Sending Access-Reject Say I do not want to return reject in this case and I want the switch to understand that radius is down and retry another radius server (a backup/failover). Because if I return reject the NAS device treats this as radius is UP and running.
if radiusd cannot use the SQL, then its a fail...and a reject. if you dont want this to happen, and radiusd is running, why would you think the NAS should try another radiusd server to get to another DB? if the radius daemon is up, then it should try the other DB instead! this is a much better way - just add the other DB to the SQL config - there are plenty of resouces (and mailing list archives) that show how you can add another sql instance..and use that if the primary fails alternatively, if the DB is down, have a monitor task that can check this (eg a timed PERL process) and if the DB is down, kill the radius daemon. thats just as bad as what you are suggesting..but is 'clean' from the NAS end (particularly its its got status packet ability) alan
Thanks Alan for reply. Connection to another backup DB from the same radius server will help only if the primary DB has real problem. But if there is a network issue to connect to DB and DB is fine just radius daemon fails to communicate to the database so in this case radius needs not to respond to the NAS instead of sending "reject". Killing radius daemon from a script is not a very clean solution. it would be much easier if auth.c file rad_authenticate function would not set request->reply->code = PW_AUTHENTICATION_REJECT; for RLM_MODULE_FAIL What RFC is saying about returning reject for actual hardware/network issue as this is not a real reject(like user does not exist or password is wrong) and NAS treats this reject as radius server is UP but authentication failed. result = module_authorize(autz_type, request); switch (result) { case RLM_MODULE_NOOP: case RLM_MODULE_NOTFOUND: case RLM_MODULE_OK: case RLM_MODULE_UPDATED: break; case RLM_MODULE_HANDLED: return result; case RLM_MODULE_FAIL: case RLM_MODULE_INVALID: case RLM_MODULE_REJECT: case RLM_MODULE_USERLOCK: default: if ((module_msg = pairfind(request->packet->vps, PW_MODULE_FAILURE_MESSAGE)) != NULL) { char msg[MAX_STRING_LEN + 16]; snprintf(msg, sizeof(msg), "Invalid user (%s)", module_msg->vp_strvalue); rad_authlog(msg,request,0); } else { rad_authlog("Invalid user", request, 0); } request->reply->code = PW_AUTHENTICATION_REJECT; return result; } A.L.M.Buxey wrote:
Hi,
Right now FreeRadius returns reject to NAS rlm_sql (sql): Failed to connect DB handle #8 rlm_sql (sql): reconnect failed, database down? rlm_sql_getvpdata: database query error [sql] SQL query error; rejecting user rlm_sql (sql): Released sql socket id: 8 ++[sql] returns fail Sending Access-Reject Say I do not want to return reject in this case and I want the switch to understand that radius is down and retry another radius server (a backup/failover). Because if I return reject the NAS device treats this as radius is UP and running.
if radiusd cannot use the SQL, then its a fail...and a reject. if you dont want this to happen, and radiusd is running, why would you think the NAS should try another radiusd server to get to another DB? if the radius daemon is up, then it should try the other DB instead! this is a much better way - just add the other DB to the SQL config - there are plenty of resouces (and mailing list archives) that show how you can add another sql instance..and use that if the primary fails
alternatively, if the DB is down, have a monitor task that can check this (eg a timed PERL process) and if the DB is down, kill the radius daemon. thats just as bad as what you are suggesting..but is 'clean' from the NAS end (particularly its its got status packet ability)
alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-- View this message in context: http://www.nabble.com/Freeradius-2.1-1%3A-failure-modes-tp22413666p22418589.... Sent from the FreeRadius - User mailing list archive at Nabble.com.
Say I do not want to return reject in this case and I want the switch to understand that radius is down and retry another radius server (a backup/failover).
So call a script that executes killall radiusd - that should do it. But that's insane.
Because if I return reject the NAS device treats this as radius is UP and running.
But your radius server *is* up and running. sql module is not working. So fail-over to another sql server or users file. http://wiki.freeradius.org/Fail-over Ivan Kalik Kalik Informatika ISP
Thanks Ivan for your reply. Regarding redundant databases:yes it can help because the problem will arise only if all of them sql1,sql2,sql3 are not reachable or dead. We are keeping NAS table in database and do not use flat files I am having another issue with "readclients = yes" option I can't keep this option set to yes in all 3 files: sql1.conf,sql2.conf,sql3.conf I want to keep NASes in the database and use DB replication to all 3 databases so all 3 databases have exact same mirrored data. The problem happens that it cannot load NAS table from sql2 because of duplicates taht it cached from sql1. Here is the error I am getting: rlm_sql (sql2): Failed to add client x.y.z.w (someclient) to clients list. Maybe there's a duplicate? Is there any way around it other than using flat files for NAS? tnt-4 wrote:
Say I do not want to return reject in this case and I want the switch to understand that radius is down and retry another radius server (a backup/failover).
So call a script that executes killall radiusd - that should do it. But that's insane.
Because if I return reject the NAS device treats this as radius is UP and running.
But your radius server *is* up and running. sql module is not working. So fail-over to another sql server or users file.
If database is down or radius server cannot connect to the database then for all my purposes radius server is dead, because sql module is the heart of this solution and we do not use flat files at all.
http://wiki.freeradius.org/Fail-over
Ivan Kalik Kalik Informatika ISP
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-- View this message in context: http://www.nabble.com/Freeradius-2.1-1%3A-failure-modes-tp22413666p22423224.... Sent from the FreeRadius - User mailing list archive at Nabble.com.
I am having another issue with "readclients = yes" option I can't keep this option set to yes in all 3 files: sql1.conf,sql2.conf,sql3.conf
I want to keep NASes in the database and use DB replication to all 3 databases so all 3 databases have exact same mirrored data.
The problem happens that it cannot load NAS table from sql2 because of duplicates taht it cached from sql1.
Here is the error I am getting: rlm_sql (sql2): Failed to add client x.y.z.w (someclient) to clients list. Maybe there's a duplicate?
Why is this "a problem"? Ivan Kalik Kalik Informatika ISP
I want to keep NAS table replicated in redundant SQL servers for failover reasons, is this fair? Now, if NAS is kept only in one SQL server this will become a single point of failure and if this particular DB fails then radius application cannot just seemlessly failover to another redundant DB server because NAS table can exist only on one DB server because of freeradius design/limitation. How do you propose solving SQL NAS table replication challenge? I understand that you can tell me keeping NAS info in local flat file(clients.conf) and do not reply on database. Then I'll need scripting on every radius server periodically connecting to DB and generating NAS (clients.conf) on the fly and HUP the radiusd or do not rely on database and hand edit clients.conf file and rsync it. But it would be nice to keep NAS information in DB but also have sql redundancy. Thanks again for you reply. tnt-4 wrote:
I am having another issue with "readclients = yes" option I can't keep this option set to yes in all 3 files: sql1.conf,sql2.conf,sql3.conf
I want to keep NASes in the database and use DB replication to all 3 databases so all 3 databases have exact same mirrored data.
The problem happens that it cannot load NAS table from sql2 because of duplicates taht it cached from sql1.
Here is the error I am getting: rlm_sql (sql2): Failed to add client x.y.z.w (someclient) to clients list. Maybe there's a duplicate?
Why is this "a problem"?
Ivan Kalik Kalik Informatika ISP
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-- View this message in context: http://www.nabble.com/Freeradius-2.1-1%3A-failure-modes-tp22413666p22425466.... Sent from the FreeRadius - User mailing list archive at Nabble.com.
leopold wrote:
I want to keep NAS table replicated in redundant SQL servers for failover reasons, is this fair?
Yes.
How do you propose solving SQL NAS table replication challenge?
Use 2.1.4 when it comes out. I've fixed this problem, and this behavior will be allowed. Alan DeKok.
This is great Alan! This is exact answer I wished to hear Thank again for your help. Alan DeKok-2 wrote:
leopold wrote:
I want to keep NAS table replicated in redundant SQL servers for failover reasons, is this fair?
Yes.
How do you propose solving SQL NAS table replication challenge?
Use 2.1.4 when it comes out. I've fixed this problem, and this behavior will be allowed.
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-- View this message in context: http://www.nabble.com/Freeradius-2.1-1%3A-failure-modes-tp22413666p22433674.... Sent from the FreeRadius - User mailing list archive at Nabble.com.
Ivan/Alan, Other then killing radiusd are there any other solutions to force radius to drop request if all databases cannot be reached ? With the latest release 2.1.4 my failover works fine, but still if there is a network issue connecting to ALL Databases I do not see any practical reason returning Access-Reject as NAS treats radius as alive and cannot failover to another radius server. For my situation since radiusd keeps everything in DB and if ALL databases cannot be contacted the radiusd should not respond at all. Is there any way to force radiusd to drop request and not to respond with Access-Reject? Thanks for your reply. tnt-4 wrote:
Say I do not want to return reject in this case and I want the switch to understand that radius is down and retry another radius server (a backup/failover).
So call a script that executes killall radiusd - that should do it. But that's insane.
Because if I return reject the NAS device treats this as radius is UP and running.
But your radius server *is* up and running. sql module is not working. So fail-over to another sql server or users file.
http://wiki.freeradius.org/Fail-over
Ivan Kalik Kalik Informatika ISP
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-- View this message in context: http://www.nabble.com/Freeradius-2.1-1%3A-failure-modes-tp22413666p22444908.... Sent from the FreeRadius - User mailing list archive at Nabble.com.
leopold wrote:
For my situation since radiusd keeps everything in DB and if ALL databases cannot be contacted the radiusd should not respond at all. Is there any way to force radiusd to drop request and not to respond with Access-Reject?
Try something like this: authorize { ... redundant { sql1 sql2 } if (fail) { update control { Response-Packet-Type = 256 } reject } ... } That should tell the server "don't respond". Alan DeKok.
No luck. For some reason unlang does not catch SQL fail return code. Only if there is no failure I see it is evaluating return code it prints in debug mode ++? if (fail) ? Evaluating (fail) -> FALSE But when SQL return really fails it does not evaluate this condition and nothing is printed in debug mode. I even tried without any if statements just to see how to force it not to respond,but the server always returns Access-Reject Tried to replace number 256 with Do-Not-Respond string and still no luck. update control { Response-Packet-Type = Do-Not-Respond } What could be the issue? Thanks again. aland wrote:
leopold wrote:
For my situation since radiusd keeps everything in DB and if ALL databases cannot be contacted the radiusd should not respond at all. Is there any way to force radiusd to drop request and not to respond with Access-Reject?
Try something like this:
authorize { ...
redundant { sql1 sql2 }
if (fail) { update control { Response-Packet-Type = 256 } reject }
... }
That should tell the server "don't respond".
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-- View this message in context: http://www.nabble.com/Freeradius-2.1-1%3A-failure-modes-tp22413666p22460041.... Sent from the FreeRadius - User mailing list archive at Nabble.com.
leopold wrote:
No luck. For some reason unlang does not catch SQL fail return code.
OK...
Only if there is no failure I see it is evaluating return code it prints in debug mode ++? if (fail) ? Evaluating (fail) -> FALSE
And you deleted the lines JUST ABOVE THAT which gave you the value of the return code. Why?
But when SQL return really fails it does not evaluate this condition and nothing is printed in debug mode.
No. The two-line output you included above shows that it *IS* evaluating the condition, but that for some reason it doesn't match. Alan DeKok.
radius.conf ------------- redundant redundant_sql { # sql1 sql2 fail <----- I tried to comment this line but it does not help } sites-enabled/default --------------------- authorize { ... redundant_sql if (fail) { update control { # Do-Not-Respond Response-Packet-Type = 256 } reject } elsif (notfound) { reject } } 1) Success scenario debug output rlm_sql (sql2): Released sql socket id: 8 +++[sql2] returns ok ++- group redundant_sql returns ok ++? if (fail) ? Evaluating (fail) -> FALSE ++? if (fail) -> FALSE ++? elsif (notfound) ? Evaluating (notfound) -> FALSE ++? elsif (notfound) -> FALSE ++[expiration] returns noop ++[logintime] returns noop 2) When I force DB down scenario rlm_sql_unixodbc: Connection failed rlm_sql (sql2): Failed to connect DB handle #5 rlm_sql (sql2): reconnect failed, database down? rlm_sql_getvpdata: database query error [sql2] SQL query error; rejecting user rlm_sql (sql2): Released sql socket id: 5 +++[sql2] returns fail +++[fail] returns fail ++- group redundant_sql returns fail Invalid user: [xyz] (from client zzzz port 0) Using Post-Auth-Type Reject +- entering group REJECT {...} [attr_filter.access_reject] expand: %{User-Name} -> xyz attr_filter: Matched entry DEFAULT at line 11 ++[attr_filter.access_reject] returns updated Delaying reject of request 2 for 1 seconds You see here it does not evaluate fail condition Alan DeKok-2 wrote:
leopold wrote:
No luck. For some reason unlang does not catch SQL fail return code.
OK...
Only if there is no failure I see it is evaluating return code it prints in debug mode ++? if (fail) ? Evaluating (fail) -> FALSE
And you deleted the lines JUST ABOVE THAT which gave you the value of the return code.
Why?
But when SQL return really fails it does not evaluate this condition and nothing is printed in debug mode.
No. The two-line output you included above shows that it *IS* evaluating the condition, but that for some reason it doesn't match.
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-- View this message in context: http://www.nabble.com/Freeradius-2.1-1%3A-failure-modes-tp22413666p22461816.... Sent from the FreeRadius - User mailing list archive at Nabble.com.
participants (4)
-
A.L.M.Buxey＠lboro.ac.uk -
Alan DeKok -
leopold -
tnt＠kalik.net