HUP handling: a thought
Let's give up on HUP entirely. Instead, we can implement something like: http://www.linuxappliancedesign.com/projects/rta/ I like the idea of configuration via SQL, but I don't like the idea of a fast server doing SQL queries for configuration. And caching the results of the SQL queries just adds more complexity. The solution, I think, is to invert the problem. Instead of making the server do SQL queries to get its configuration, have the SQL queries get the configuration from the internal data structures of the server. The "rta" project above does this. It's only about 1k LoC, which is nice. On the downside, it's not thread-safe, and only supports UPDATE and SELECT, it requires that the rows be in arrays (for the most part), and it has hard-coded limits on the number of rows, columns, name lengths, etc. My thought was to have a look at doing something similar, but with support for INSERT, DELETE, and removing the hard-coded limitations. e.g. have each table as a balanced red-black tree. Each leaf is a "row". Each row is a data structure, with accessor functions. The server code can just access the data structures directly, without worrying about the whole SQL overhead. The SQL code can use the accessor functions to get the actual data. It may take a bit of magic to handle insertions && deletions in a thread-safe way, but it could work... Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Alan DeKok wrote:
Let's give up on HUP entirely. Instead, we can implement something like:
Thanks for all of the hard work you have been doing, Alan. The decision is not mine, of course, but here is my vote. -1 Proper support for HUP is in my view an elemental requirement of any properly functioning UNIX daemon. If you feel its to complex to do it safely within the running code, you can do it like sendmail daemon does, which simply rexecs itself with its launching arguments. Sendmail doesnt support HUP if it wasnt started with a full path.
On Wed 02 May 2007, Joe Maimon wrote:
Alan DeKok wrote:
Let's give up on HUP entirely. Instead, we can implement something like:
Thanks for all of the hard work you have been doing, Alan.
The decision is not mine, of course, but here is my vote.
-1
Proper support for HUP is in my view an elemental requirement of any properly functioning UNIX daemon.
If you feel its to complex to do it safely within the running code, you can do it like sendmail daemon does, which simply rexecs itself with its launching arguments. Sendmail doesnt support HUP if it wasnt started with a full path.
Yep. I vote for this solution as well in absence of anything more elegant. The system should do _something_ when it receives -HUP (other than crash). It this turns out to be a full restart, then so be it... Regards -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
Peter Nixon wrote:
The system should do _something_ when it receives -HUP (other than crash). It this turns out to be a full restart, then so be it...
So there's a requirement to handle HUP. I don't see why. How about a requirement to dynamically update the running configuration? That could be done via re-exec on HUP, or by the crazy SQL scheme I talked about. I don't understand the fixation on HUP. I don't like SQL, but if I can use an SQL client to edit *every* configuration parameter in a running server, I don't see why HUP would *ever* be necessary. Recent versions of OpenLDAP support an "dn=config", or something like that. It means any configuration parameter can be dynamically changed, meaning you *never* have to HUP the server. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Alan DeKok wrote:
Peter Nixon wrote:
The system should do _something_ when it receives -HUP (other than crash). It this turns out to be a full restart, then so be it...
So there's a requirement to handle HUP. I don't see why.
I don't see why this should be a requirement either. Most operating systems have an abstraction layer on top of the daemon, such as RedHat's 'service' command, or FreeBSD's rc.subr framework. Use that as the common abstraction, not HUP signals (if it's commonality you're looking for). Would your SQL scheme support modifying any configuration item, including module specific ones?
Daniel Larsson wrote:
I don't see why this should be a requirement either. Most operating systems have an abstraction layer on top of the daemon, such as RedHat's 'service' command, or FreeBSD's rc.subr framework. Use that as the common abstraction, not HUP signals (if it's commonality you're looking for).
Would your SQL scheme support modifying any configuration item, including module specific ones?
That would be the end goal, yes. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
On Wed 02 May 2007, Alan DeKok wrote:
Peter Nixon wrote:
The system should do _something_ when it receives -HUP (other than crash). It this turns out to be a full restart, then so be it...
So there's a requirement to handle HUP. I don't see why.
Hmm.. Well, I guess you could just ignore the signal, but doing a restart on HUP would seems pretty trivial to do (please correct me if I'm wrong), and would keep things unsurprising to newcomers..
How about a requirement to dynamically update the running configuration? That could be done via re-exec on HUP, or by the crazy SQL scheme I talked about.
Hmm. I just got around to reading the web page... "The database interface uses a subset of the Postgres protocol and is compatible with the Postgres bindings.. " Cool..
I don't understand the fixation on HUP. I don't like SQL, but if I can use an SQL client to edit *every* configuration parameter in a running server, I don't see why HUP would *ever* be necessary.
Recent versions of OpenLDAP support an "dn=config", or something like that. It means any configuration parameter can be dynamically changed, meaning you *never* have to HUP the server.
Sure. Not necessary. But there are old scripts floating around that do HUP, and unix people do tend to expect -HUP to do something... Personally I dont care that much... I just do "rcfreeradius restart" ;-) Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
Peter Nixon wrote:
Hmm.. Well, I guess you could just ignore the signal, but doing a restart on HUP would seems pretty trivial to do (please correct me if I'm wrong), and would keep things unsurprising to newcomers..
Yes, it's trivial to do. Yes, it has issues. If you're proxying, it's OK to lose state on all accounting packets, as the NAS will retransmit, and the home server will manage duplicates. It's not OK to lose state on all authentication packets, as you may process the same login twice, OR you may discard the users ongoing EAP session. If you're not doing EAP, or if you can manage the duplicate login problem, re-execing on HUP is OK. I would prefer a *clean* solution where It Just Works. Discarding traffic is not nice, especially if a little bit of coding may avoid that problem. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Joe Maimon wrote:
Proper support for HUP is in my view an elemental requirement of any properly functioning UNIX daemon.
What does that mean? "Proper" means... what? Please be specific.
If you feel its to complex to do it safely within the running code, you can do it like sendmail daemon does, which simply rexecs itself with its launching arguments. Sendmail doesnt support HUP if it wasnt started with a full path.
Sendmail has two nice features, like Apache, Bind, and most Unix servers. 1) There is no state. 2) Any state that may exist is on disk. dhcpd has state, but a well written dhcp server puts all of it's state into a DB. All of the information about a lease *must* be put into a DB, otherwise it doesn't make sense. RADIUS isn't like that. There *is* state in a RADIUS server. There are packets in flight from a NAS to a home server. Re-execing on HUP is a complete and total non-starter. It means that every HUP will cause *all* current login sessions to be dropped. Add to that ongoing EAP sessions, and every HUP could cause a 5-10s service outage. If that's acceptable, we'll just release 2.0 with documented service outages. I *really* don't like that, though. To answer my first question. People HUP the server to: a) add/edit/delete a client/realm/home-server b) re-read the "users" file c) not much else. If we make the server support (a) without HUP, that will address a large part of my concerns. Did I mention I don't like re-exec'ing the server? Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Hi! Alan DeKok wrote:
To answer my first question. People HUP the server to:
a) add/edit/delete a client/realm/home-server b) re-read the "users" file c) not much else.
Except: I remember vaguely that HUPing was at least once used to re-read updated CRLs and probably also re-reading the list/directory of trusted client-CA certificates for all the EAP-(T)TLS/PEAP stuff. I an email recently send to freeradius-users you recommended to simply restart the daemon for this since HUPing does not work for this purpose anymore.
If we make the server support (a) without HUP, that will address a large part of my concerns.
-- Beste Gruesse / Kind Regards Reimer Karlsen-Masur DFN-PKI FAQ: https://www.pki.dfn.de/faqpki -- Dipl.-Inform. Reimer Karlsen-Masur (PKI Team), Phone +49 40 808077-615 DFN-CERT Services GmbH, https://www.dfn-cert.de, Phone +49 40 808077-555 Sitz / Register: Hamburg, AG Hamburg, HRB 88805, Ust-IdNr.: DE 232129737
Reimer Karlsen-Masur, DFN-CERT wrote:
I remember vaguely that HUPing was at least once used to re-read updated CRLs and probably also re-reading the list/directory of trusted client-CA certificates for all the EAP-(T)TLS/PEAP stuff.
Yes.
I an email recently send to freeradius-users you recommended to simply restart the daemon for this since HUPing does not work for this purpose anymore.
HUP does something in 1.x. It will cause the server to re-read the configuration files. It *may* also cause the server to die unexpectedly. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
I remember vaguely that HUPing was at least once used to re-read updated CRLs and probably also re-reading the list/directory of trusted client-CA certificates for all the EAP-(T)TLS/PEAP stuff.
Yes.
that was me, so to the list of 'other things' other than re-read clients, realms and the users files I'd like to add re-read of CRLs (and certificates). The reason for me being so boring is that a proper implementation of EAP-(T)TLS requires the server to handle all the CA chain and CRL updates crap. CRLs unfortunately DO expire. Expired CRL == the properly implemented EAP-TLS structure falls apart and everybody gets a reject due to 'expired' certs. Therefore, if you release a CRL because you have a properly implemented CA and TLS infrastructure you must also reload it to the radiusd or everything breaks and you get a VERY poor uptime ~_~. If there's no chance to realod a CRL without an /etc/init.d/radiusd restart then I'll stick to restarting the process during offpeak hours. Not using a CRL at all is not an option when a major amount of users enter and/or leave your organization each year and you use TLS. As a side note, I know this sucks. Maybe I'm one othe few around the world fool enough to go all the way down to CRLs. As a foot note: I suppport Alan's idea. Let's forget about HUP. Experience shows HUP is clearly not suited for something with a system state and personally I don't accept a solution that makes an otherwise perfectly stable daemon to occasionally crater. If HUP is not going to work, let's do it another way. SQL sounds good to me. As long as it reloads a CRL too ^_-
HUP does something in 1.x. It will cause the server to re-read the configuration files. It *may* also cause the server to die unexpectedly.
Too bad my test server dies with excellent consistency when I HUP it :P
inverse wrote:
The reason for me being so boring is that a proper implementation of EAP-(T)TLS requires the server to handle all the CA chain and CRL updates crap. CRLs unfortunately DO expire. Expired CRL == the properly implemented EAP-TLS structure falls apart and everybody gets a reject due to 'expired' certs.
Support for OCSP in the server would minimize the reloads due to changing CRL's.
As a foot note: I suppport Alan's idea. Let's forget about HUP. Experience shows HUP is clearly not suited for something with a system state and personally I don't accept a solution that makes an otherwise perfectly stable daemon to occasionally crater.
The problem isn't the HUP, so much as the fact that *everything* changes on HUP. It's tremendously difficult to keep the server running while almost every data structure is modified. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Frank Cusack wrote:
Is there any publicly available OCSP code? I'm interested in this for my own application and haven't seen any.
I thought OpenSSL supported it. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
On May 4, 2007 6:27:50 PM +0200 Alan DeKok <aland@deployingradius.com> wrote:
Frank Cusack wrote:
Is there any publicly available OCSP code? I'm interested in this for my own application and haven't seen any.
I thought OpenSSL supported it.
Through an external application (a shell script I think). It's not part of libssl. Also, OpenSSL's OCSP tool was incomplete, last I checked. -frank
On May 2, 2007 10:55:57 AM +0200 Alan DeKok <aland@deployingradius.com> wrote:
Let's give up on HUP entirely. Instead, we can implement something like:
http://www.linuxappliancedesign.com/projects/rta/
I like the idea of configuration via SQL,
I hate the idea of requiring SQL to configure the server. It's another dependency, one that is complex. It makes inspection of the current config difficult, and complicates config mgmt. Now then, if the config is dynamic via SQL, how's that different than having it dynamic via a signal? You will have to do all the same setup, teardown and run all current transactions to completion while starting new transactions under a new configuration anyway. -frank
Frank Cusack wrote:
I hate the idea of requiring SQL to configure the server. It's another dependency, one that is complex. It makes inspection of the current config difficult, and complicates config mgmt.
I agree. I have issues with SQL myself. However, the page I pointed to is ~1k LoC, and has no external dependencies. Or, try this: The server starts, reads *no* configuration, and just listens on an SQL port for configuration. Then, a separate program groks the humanly-reading *conf files, and turns them into SQL nonsense for feeding to the server. Once that's done, the admin doesn't need to know SQL. HUP could be handled by having the separate program read the new config, and do "diff"s of the SQL statements. It could then feed the appropriate changes to the server.
Now then, if the config is dynamic via SQL, how's that different than having it dynamic via a signal?
Dynamic config via "edit structure X value Y to value Z" is *nothing* like the current "kick the server in the head, and have it re-read everything on the planet" that's done for HUP. Small, targeted changes are easier to manage.
You will have to do all the same setup, teardown and run all current transactions to completion while starting new transactions under a new configuration anyway.
Yes. The problem is simplified by having almost everything tied to a REQUEST. So anything "live" is either a thread handling a REQUEST, or a REQUEST waiting for something. And REQUESTs point to clients & home servers via one pointer, which is easy to manage. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
On May 3, 2007 8:16:37 AM +0200 Alan DeKok <aland@deployingradius.com> wrote:
Frank Cusack wrote:
I hate the idea of requiring SQL to configure the server. It's another dependency, one that is complex. It makes inspection of the current config difficult, and complicates config mgmt.
I agree. I have issues with SQL myself. However, the page I pointed to is ~1k LoC, and has no external dependencies.
Or, try this: The server starts, reads *no* configuration, and just listens on an SQL port for configuration. Then, a separate program groks the humanly-reading *conf files, and turns them into SQL nonsense for feeding to the server. Once that's done, the admin doesn't need to know SQL.
HUP could be handled by having the separate program read the new config, and do "diff"s of the SQL statements. It could then feed the appropriate changes to the server.
I don't see any special advantage then for SQL. It's not like you're going to do specialized config queries against freeradius. You're not going to write your config in SQL statements. Additionally, computing a SQL "diff" sounds error prone to me.
Now then, if the config is dynamic via SQL, how's that different than having it dynamic via a signal?
Dynamic config via "edit structure X value Y to value Z" is *nothing* like the current "kick the server in the head, and have it re-read everything on the planet" that's done for HUP.
Small, targeted changes are easier to manage.
I'm not suggesting that the current method of handling the config is all that's needed.
You will have to do all the same setup, teardown and run all current transactions to completion while starting new transactions under a new configuration anyway.
Yes. The problem is simplified by having almost everything tied to a REQUEST. So anything "live" is either a thread handling a REQUEST, or a REQUEST waiting for something. And REQUESTs point to clients & home servers via one pointer, which is easy to manage.
Easily done without SQL. -frank
Frank Cusack wrote:
I don't see any special advantage then for SQL. It's not like you're going to do specialized config queries against freeradius. You're not going to write your config in SQL statements.
Exactly. The benefit, however, is to avoid writing yet another api/tool to modify a running configuration.
Additionally, computing a SQL "diff" sounds error prone to me.
I wouldn't recommend it, but it's possible.
Yes. The problem is simplified by having almost everything tied to a REQUEST. So anything "live" is either a thread handling a REQUEST, or a REQUEST waiting for something. And REQUESTs point to clients & home servers via one pointer, which is easy to manage.
Easily done without SQL.
Suggestions? After spending some time on this, there isn't a ready-made solution that I've found. Looking on the net, few servers other than OpenLDAP support dynamic run-time changes to the config. And they all do so via a DB interface. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Suggestions?
After spending some time on this, there isn't a ready-made solution that I've found. Looking on the net, few servers other than OpenLDAP support dynamic run-time changes to the config. And they all do so via a DB interface.
I Think loading configs via SQL is an excellent idea, it would allow central configuration of multiple servers, and far better integration with dialup admin (is anyone still developing dialup admin ?). Would also allow easier access to the configuration items instead of having to go through many hundreds of lines of configuration directives in vi . You could even have web based setups for freeRADIUS, which would save a lot of the annoying questions which keep appearing on this list, over and over again. --- Arran
On May 3, 2007, at 9:28 PM, Alan DeKok wrote:
Frank Cusack wrote:
I don't see any special advantage then for SQL. It's not like you're going to do specialized config queries against freeradius. You're not going to write your config in SQL statements.
Exactly. The benefit, however, is to avoid writing yet another api/tool to modify a running configuration.
Yes, though there may not be anything out there that fits what we need.
After spending some time on this, there isn't a ready-made solution that I've found. Looking on the net, few servers other than OpenLDAP support dynamic run-time changes to the config. And they all do so via a DB interface.
Not all. :) ser/openser for instance takes commands via a pipe. My thoughts were to be able to send commands to FR in a similar fashion. "reload clients" <-- only reloads clients structure "reload full" <-- full reload "reload proxy" <-- only reloads proxy/realms structure The problem with HUP handling as it stands today is it causes a reload of "everything". For a highly available service like RADIUS, where state is important, being able to only selectively reload the clients.conf without stepping on all of the aaa sessions "in flight" would be huge. Ultimately, it'd be awesome to manipulate individual elements of the clients/realms structs, aka: "reload client somenas.foo.com" <--- reads clients, builds struct for somenas, pops current struct entry out, pops new struct entry in. fails if there are errors in parsing clients entry, and doesn't alter the live in memory struct. "reload realm foo.com" <--- same as above. That type of granularity would indeed step on the state for "in flight" sessions of the affected realm/client, but that would be acceptable since we are in fact changing it. All unaffected realms/clients would *not* have their state affected. This is very very useful in large environments where you have several thousand client and realm entries. -Chris -- Chris Parker Director, Systems StarNet - US LEC, now a PAETEC Company (888)212-0099 Fax (847)963-1302 Wholesale Internet and VoIP Services http://www.megapop.net NOTICE: Message is sent IN CONFIDENCE to addressees. It may contain information that is privileged, proprietary or confidential.
Chris Parker wrote:
My thoughts were to be able to send commands to FR in a similar fashion.
"reload clients" <-- only reloads clients structure "reload full" <-- full reload "reload proxy" <-- only reloads proxy/realms structure
The problem with HUP handling as it stands today is it causes a reload of "everything" ... Ultimately, it'd be awesome to manipulate individual elements of the clients/realms structs, aka:
The SQL approach would enable that. From some initial investigation, it's not much more work than doing partial reloading.
That type of granularity would indeed step on the state for "in flight" sessions of the affected realm/client, but that would be acceptable since we are in fact changing it. All unaffected realms/clients would *not* have their state affected. This is very very useful in large environments where you have several thousand client and realm entries.
Yup. That's the goal. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
On May 4, 2007 4:28:06 AM +0200 Alan DeKok <aland@deployingradius.com> wrote:
Frank Cusack wrote:
Yes. The problem is simplified by having almost everything tied to a REQUEST. So anything "live" is either a thread handling a REQUEST, or a REQUEST waiting for something. And REQUESTs point to clients & home servers via one pointer, which is easy to manage.
Easily done without SQL.
Suggestions?
Add a config_t * to a REQUEST. This would be exactly how you would do it with some other library that is handling configuration. (Except with some other library the config_t * probably has both data and methods.) The config_t itself would have a refcount (this can be updated via atomic ops which don't require a mutex). Then on HUP (or some type of reconfig message), a new thread runs which reads the new config and globally marks it current. New REQUESTs get assigned the new config. The config thread waits for the refcount on the old config to go to 0, then reclaims the memory and then exits. Also, re-instantiate all modules. The new instance will pick up the new config. When all previously running threads finally complete, destroy the previous module instances.
After spending some time on this, there isn't a ready-made solution that I've found. Looking on the net, few servers other than OpenLDAP support dynamic run-time changes to the config. And they all do so via a DB interface.
BIND named has reconfig and doesn't use a DB interface. -frank
Frank Cusack wrote:
On May 4, 2007 4:28:06 AM +0200 Alan DeKok <aland@deployingradius.com>
Suggestions?
Add a config_t * to a REQUEST. This would be exactly how you would do it with some other library that is handling configuration. (Except with some other library the config_t * probably has both data and methods.)
The config_t itself would have a refcount (this can be updated via atomic ops which don't require a mutex). Then on HUP (or some type of reconfig message), a new thread runs which reads the new config and globally marks it current. New REQUESTs get assigned the new config. The config thread waits for the refcount on the old config to go to 0, then reclaims the memory and then exits.
Also, re-instantiate all modules. The new instance will pick up the new config. When all previously running threads finally complete, destroy the previous module instances.
I assume the problem then comes in the handling of further packets for preexisting state, such as proxy replies and eap? I suppose in that case, abandoning those wouldnt be so terrible, especialy as compared to a full restart which what most admins/distributions in the real world who havent implemented EVERYTHING via sql will be/already are doing.
On May 4, 2007 1:40:16 PM -0400 Joe Maimon <jmaimon@ttec.com> wrote:
Frank Cusack wrote:
On May 4, 2007 4:28:06 AM +0200 Alan DeKok <aland@deployingradius.com>
Suggestions?
Add a config_t * to a REQUEST. This would be exactly how you would do it with some other library that is handling configuration. (Except with some other library the config_t * probably has both data and methods.)
The config_t itself would have a refcount (this can be updated via atomic ops which don't require a mutex). Then on HUP (or some type of reconfig message), a new thread runs which reads the new config and globally marks it current. New REQUESTs get assigned the new config. The config thread waits for the refcount on the old config to go to 0, then reclaims the memory and then exits.
Also, re-instantiate all modules. The new instance will pick up the new config. When all previously running threads finally complete, destroy the previous module instances.
I assume the problem then comes in the handling of further packets for preexisting state, such as proxy replies and eap?
Oh, right. Handling that would be slightly complex. FR itself (as opposed to modules) would have to understand State. So if a packet came in with State, it could decode the first byte or 2 and decide whether to hand it to an old or new instance of a module. The refcount idea would also be invalid in this case, although maybe it's still useful in general (not sure). But for multi-packet transactions, the refcount could go to 0 yet there would still be pending transactions for an old module instance. I guess a first version could simply ignore that problem.
I suppose in that case, abandoning those wouldnt be so terrible, especialy as compared to a full restart which what most admins/distributions in the real world who havent implemented EVERYTHING via sql will be/already are doing.
Well the problem here is orthogonal to SQL AFAICT. -frank
Frank Cusack wrote:
Add a config_t * to a REQUEST. This would be exactly how you would do it with some other library that is handling configuration. (Except with some other library the config_t * probably has both data and methods.)
That is one thought, which requires everything to be tied to a REQUEST* (not a bad idea, in many respects).
The config_t itself would have a refcount (this can be updated via atomic ops which don't require a mutex). Then on HUP (or some type of reconfig message), a new thread runs which reads the new config and globally marks it current. New REQUESTs get assigned the new config. The config thread waits for the refcount on the old config to go to 0, then reclaims the memory and then exits.
Simpler: each config has a timestamp of when it is active, and when it becomes stale. The REQUESTs already have a timestamp, too. Then, say 30s after the new config becomes active, start scanning the request list for REQUESTs that use the old config. If there are none, clean up the old config. i.e. Garbage collection is often cheaper than refcounts. ISC DHCPD uses refcounts for everything. They're a nightmare to keep track of, and they have *significant* performance overhead.
I assume the problem then comes in the handling of further packets for preexisting state, such as proxy replies and eap?
Oh, right. Handling that would be slightly complex. FR itself (as opposed to modules) would have to understand State.
If the REQUEST* has a config_t, the proxy replies will pick that up. The EAP module can also be updated to add a config_t to the state it keeps.
I suppose in that case, abandoning those wouldnt be so terrible, especialy as compared to a full restart which what most admins/distributions in the real world who havent implemented EVERYTHING via sql will be/already are doing.
Well the problem here is orthogonal to SQL AFAICT.
Yes. There are two problems: 1) dynamically changing data structures as they are being used 2) how to tell the server which data structure to change. (1) has to be solved before (2) becomes useful. (2) can be done by any number of means. The reason I've been leaning towards and SQL-like interface is because I don't want to re-invent yet another CLI. We can always write a simple program that translates a sane (i.e. humanly readable) CLI to SQL. The utility of SQL comes in leveraging the large amount of tools that are ready-made to perform data manipulation. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Alan DeKok wrote:
Yes. There are two problems:
1) dynamically changing data structures as they are being used 2) how to tell the server which data structure to change.
(1) has to be solved before (2) becomes useful. (2) can be done by any number of means. The reason I've been leaning towards and SQL-like interface is because I don't want to re-invent yet another CLI.
We can always write a simple program that translates a sane (i.e. humanly readable) CLI to SQL. The utility of SQL comes in leveraging the large amount of tools that are ready-made to perform data manipulation.
I like when the things are simple: edit the config files with your favorite editor, and run a command to reload the server. (kill -HUP or something else) I think an SQL interface would add a considerable administrative overhead. I can easily explain to the sysadmins in my site to copy template files to deploy new clients. Explaining to type SQL commands seems more difficult. Translating plain text files to SQL has a few issues issues, too. When you update a value on a running server, its runtime config is out of sync with its config files. Therefore I don't know how we could make sure that the "humanly readable" version is relevant or not. -- Nicolas Baradakis
Nicolas Baradakis wrote:
I like when the things are simple: edit the config files with your favorite editor, and run a command to reload the server. (kill -HUP or something else)
It's nice, but can result in ~1s hiccups when everything gets reloaded. There's some code in the server right now to notice that a config file hasn't changed, so it won't be reloaded. There's other code to figure out diff's between old/new config. It then tries to keep the *parsed* configuration (e.g. "users" file to data structures), and not do the parsing again. But it's complicated, and it doesn't really work well.
I think an SQL interface would add a considerable administrative overhead. I can easily explain to the sysadmins in my site to copy template files to deploy new clients. Explaining to type SQL commands seems more difficult.
Tell them to copy a few template files, and run a script to do it.
Translating plain text files to SQL has a few issues issues, too. When you update a value on a running server, its runtime config is out of sync with its config files. Therefore I don't know how we could make sure that the "humanly readable" version is relevant or not.
The "RTA" program I originally pointed to does this by dumping SQL commands to a file. When the file is read in, it re-creates the running configuration. See also Mercurial's "revlog" format for how to do high-performance, safe logging of a changing configuration. Or, to turn the problem on it's head: Don't touch the existing configuration. Instead, write any state to a file (or shared memory). Exit. Then, the new server reads the configuration, and bootstraps itself from the pre-existing state. But if all you're doing is adding one realm... why the heck do you want to reload the other 3000 realms to add just one? That doesn't make sense. It's an O(N^2) solution to a problem that should be O(1). Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Alan DeKok wrote:
Nicolas Baradakis wrote:
I like when the things are simple: edit the config files with your favorite editor, and run a command to reload the server. (kill -HUP or something else)
It's nice, but can result in ~1s hiccups when everything gets reloaded. [...]
Indeed. At some point we need to make a tradeoff between a solution that is simple for the administrator but not very efficient, and a complex configuration mechanism that allow extensive control on a running server. (personally I tend to prefer the former solution)
Translating plain text files to SQL has a few issues issues, too. When you update a value on a running server, its runtime config is out of sync with its config files. Therefore I don't know how we could make sure that the "humanly readable" version is relevant or not.
The "RTA" program I originally pointed to does this by dumping SQL commands to a file. When the file is read in, it re-creates the running configuration. See also Mercurial's "revlog" format for how to do high-performance, safe logging of a changing configuration.
The program will be able to store its configuration and restart with the same parameters. But I think the file that is dumped isn't in a humanly readable format. Therefore if the administrator wants to see what the running server is actually doing, he may have to run a few SQL queries.
But if all you're doing is adding one realm... why the heck do you want to reload the other 3000 realms to add just one? That doesn't make sense. It's an O(N^2) solution to a problem that should be O(1).
I agree. But large sites should have multiple servers, therefore perhaps they could address the issue differently. -- Nicolas Baradakis
On Sun 06 May 2007, Nicolas Baradakis wrote:
Alan DeKok wrote:
Nicolas Baradakis wrote:
I like when the things are simple: edit the config files with your favorite editor, and run a command to reload the server. (kill -HUP or something else)
It's nice, but can result in ~1s hiccups when everything gets reloaded. [...]
Indeed. At some point we need to make a tradeoff between a solution that is simple for the administrator but not very efficient, and a complex configuration mechanism that allow extensive control on a running server. (personally I tend to prefer the former solution)
I also tend to agree with you here. At the end the whole premise for high level languages like java, python, ruby etc is that while its a bit less efficient in terms of CPU time, the most valuable resource is infact the human one, and its worth wasting some CPU time to make the developer or admin's job easier... Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
Peter Nixon wrote:
On Sun 06 May 2007, Nicolas Baradakis wrote:
Alan DeKok wrote:
Nicolas Baradakis wrote:
I like when the things are simple: edit the config files with your favorite editor, and run a command to reload the server. (kill -HUP or something else)
It's nice, but can result in ~1s hiccups when everything gets reloaded. [...]
Indeed. At some point we need to make a tradeoff between a solution that is simple for the administrator but not very efficient, and a complex configuration mechanism that allow extensive control on a running server. (personally I tend to prefer the former solution)
I also tend to agree with you here. At the end the whole premise for high level languages like java, python, ruby etc is that while its a bit less efficient in terms of CPU time, the most valuable resource is infact the human one, and its worth wasting some CPU time to make the developer or admin's job easier...
Cheers
Yes, and putting the configuration for freeradius in a SQL database opens the door for other languages like ruby. You can automate the configuration process a great deal, when you have a high level language that can manipulate the SQL data. So in the end it *will* make the admins job easier, so long as they invest the time to start with. Granted configuring freeradius via the SQL CLI, would be slow and painfull, but you don't have to configure it via the SQL CLI !! Thanks, Arran
Nicolas Baradakis wrote:
Indeed. At some point we need to make a tradeoff between a solution that is simple for the administrator but not very efficient, and a complex configuration mechanism that allow extensive control on a running server. (personally I tend to prefer the former solution)
Step 1: Make it work. Step 2: Make it work better/faster/smarter :)
But if all you're doing is adding one realm... why the heck do you want to reload the other 3000 realms to add just one? That doesn't make sense. It's an O(N^2) solution to a problem that should be O(1).
I agree. But large sites should have multiple servers, therefore perhaps they could address the issue differently.
They still have the problem of tracking state for ongoing sessions. Maybe what's needed instead is a state manager. i.e. a program with no knowledge of clients or home servers that manages requests and responses. It can be told about received packets, and can be queried for duplicates. Then, the main server can safely lose track of almost all state on restart, as it could still query the state manager. It could add significant delays to the packet processing, but it should work for everything other than ongoing EAP-TLS sessions. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Alan DeKok> Maybe what's needed instead is a state manager. Alan DeKok> It can be told about received packets, and can be queried Alan DeKok> for duplicates. Then, the main server can safely lose track of almost Alan DeKok> all state on restart, as it could still query the state manager. That is a good idea, but on most applications where a "state machine" was critical, users solved the problem by using a database. FR has the possibility to change those 1000 users without restarting everything. We can easily grow from 1000 users to 1M users. Imho, the next step is to have the possibility to change radius clients on the fly. Not only VoIP carriers or access providers want to manage/grow to 10k radius clients. If IP addresses are not so easy to compare when (re)building large lists, why not having a numeric ID for each client to seek faster the tree and change the IP address, secret, name?.. This numeric ID can also be used for snmp requests and it will be natural for users running a database backend with a "serial" field. Claudiu FILIP @: claudiu@globtel.ro Http://www.globtel.ro
Claudiu Filip wrote:
That is a good idea, but on most applications where a "state machine" was critical, users solved the problem by using a database.
Which won't handle 100 DB lookups per packet. That's why main memory is used: fast access to necessary configuration information.
FR has the possibility to change those 1000 users without restarting everything.
Because it queries a database for those users. Often only once per request. Maybe 2-3 times, but definitely not 10-20. If all of the configuration information was in a DB, it would do 100-200 DB lookups per request. Even if they were cached, that's still a huge load.
We can easily grow from 1000 users to 1M users. Imho, the next step is to have the possibility to change radius clients on the fly. Not only VoIP carriers or access providers want to manage/grow to 10k radius clients.
I know. That's the goal.
If IP addresses are not so easy to compare when (re)building large lists, why not having a numeric ID for each client to seek faster the tree and change the IP address, secret, name?.. This numeric ID can also be used for snmp requests and it will be natural for users running a database backend with a "serial" field.
2.0-pre already does that for SNMP requests. Maybe the solution is to separate the server into multiple processes. One can handle packets going in and out, but wouldn't connect to a DB. Another would connect to the DB, but would only have the first program as a listed as a client. That separation would solve most of the reloading problems. The only difficulty left would be fast sharing of packets between the programs. But I suspect that for almost any real-world traffic, the extra overhead of moving 20 packets/s between two processes would be negligible. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
On May 6, 2007, at 12:34 PM, Nicolas Baradakis wrote:
But if all you're doing is adding one realm... why the heck do you want to reload the other 3000 realms to add just one? That doesn't make sense. It's an O(N^2) solution to a problem that should be O (1).
I agree. But large sites should have multiple servers, therefore perhaps they could address the issue differently.
Being a "large site", I can speak to this. :) We stagger HUPs to reload realm changes today. It *still* causes a noticeable impact on connection stats that are tracked by our largest customers. We had to roll back from 'on- demand' changes to scheduling periodic HUPs to lessen the impact on our performance stats. The impact is that a larger percentage of connection attempts timeout/ fail due to the hit the server takes to reload the configs, even when they are staggered across multiple servers. That's my underlying desire to be able to change the server details for realm "foo" without taking a hit on the 3000 other realms, which aren't changing. -Chris -- Chris Parker Director, Systems StarNet - US LEC, now a PAETEC Company (888)212-0099 Fax (847)963-1302 Wholesale Internet and VoIP Services http://www.megapop.net NOTICE: Message is sent IN CONFIDENCE to addressees. It may contain information that is privileged, proprietary or confidential.
Frank Cusack wrote:
BIND named has reconfig and doesn't use a DB interface.
It doesn't really work well, either. Dan Bernstein's rants aren't entirely wrong. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Alan DeKok wrote:
Let's give up on HUP entirely. Instead, we can implement something like:
How about using this method to expose internal statistics etc, as an alternative to SNMP? It has the potential of providing a far richer set of information that the current MIB (and hopefully easier to get to compile..) Eddie
Eddie Stassen wrote:
How about using this method to expose internal statistics etc, as an alternative to SNMP? It has the potential of providing a far richer set of information that the current MIB (and hopefully easier to get to compile..)
Sure. Except that the rta site has hard-coded limits on the number of tables, rows, columns, etc. So it can't export statistics for clients or home servers, because there may be more than 20... etc. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
participants (12)
-
Alan DeKok -
Arran Cudbard-Bell -
Chris Parker -
Claudiu Filip -
Daniel Larsson -
Eddie Stassen -
Frank Cusack -
inverse -
Joe Maimon -
Nicolas Baradakis -
Peter Nixon -
Reimer Karlsen-Masur, DFN-CERT