Hi, we want to dump all our requests into files, using the detail module. The files should be rotated (and archived) with the standard logrotate tool, which would need the freeradius daemon to to close and reopen the file (automatically or triggered, e.g. by SIGHUP, like the radius.log). But I found now way to tell the daemon (or the detail module) to do that. As I read from the code, this will only happen when the file is unlinked or nothing was written for 30 seconds. gzip will unlink the file afterwards, but it will complain about the changed file size during compression and there is chance of data loss (as something may be written just before the file is deleted). I know I can tell the detail module to separate the files by hour/day/whatever, but that would require significant changes in the subsequent processing, which I want to avoid (at least for now). Is there something I have missed? Or do I have to live with that (or patch the module)? TIA and regards Jakob
On Jun 17, 2016, at 8:50 AM, Jakob Hirsch <jh@plonk.de> wrote:
we want to dump all our requests into files, using the detail module. The files should be rotated (and archived) with the standard logrotate tool, which would need the freeradius daemon to to close and reopen the file (automatically or triggered, e.g. by SIGHUP, like the radius.log). But I found now way to tell the daemon (or the detail module) to do that. As I read from the code, this will only happen when the file is unlinked or nothing was written for 30 seconds. gzip will unlink the file afterwards, but it will complain about the changed file size during compression and there is chance of data loss (as something may be written just before the file is deleted).
Yes. The changes in v3 which make the detail module more thread-safe and faster do have this side effect.
I know I can tell the detail module to separate the files by hour/day/whatever, but that would require significant changes in the subsequent processing, which I want to avoid (at least for now).
Write files hourly. Then, have a cron job which concatenates them together in a format required by the later processing. Alan DeKok.
Alan DeKok wrote on 2016-06-17 14:57:
But I found now way to tell the daemon (or the detail module) to do that. As I read from the code, this will only happen when the file is unlinked or nothing was written for 30 seconds. gzip will unlink the ... Yes. The changes in v3 which make the detail module more thread-safe and faster do have this side effect.
Oh, ok. I saw that there is quite some code for this. We used our own module (originally developed for FR 1.x, later ported to 2.2.x) which simply did a open(data->dump_file, O_WRONLY|O_APPEND|O_CREAT,...) and close() for every packet, and never had threading or performance issues (ok, there are usually not more then 200 requests/s per server). Our module worked quite well, but I wanted to get rid of the custom modules (yes, there are more) as much as possible.
I know I can tell the detail module to separate the files by hour/day/whatever, but that would require significant changes in the subsequent processing, which I want to avoid (at least for now). Write files hourly. Then, have a cron job which concatenates them together in a format required by the later processing.
Ok, I should have mentioned that we also have some scripts that do some online log processing, which would also need modifications...
On Jun 17, 2016, at 9:42 AM, Jakob Hirsch <jh@plonk.de> wrote:
Oh, ok. I saw that there is quite some code for this. We used our own module (originally developed for FR 1.x, later ported to 2.2.x) which simply did a open(data->dump_file, O_WRONLY|O_APPEND|O_CREAT,...) and close() for every packet, and never had threading or performance issues
If you have low packet rates, that works.... If the data you write to the file descriptor is smaller than 4K, and written atomically. And if you don't need to do locking on the file. The detail module doesn't do atomic writes, and it often needs locking for use with the detail file reader, and it handles 10K packets/s. So... the current behavior is better. It's probably possible to do minor code modifications to the detail file to address your needs. Add a configuration item which tells the module to open / close the file each time. Then add in hooks to avoid the calls to exfile(). Probably ~30 LoC. Alan DeKok.
Hi,
we want to dump all our requests into files, using the detail module. The files should be rotated (and archived) with the standard logrotate
logrotate can handle this - you just need to ensure that you are using the options as provided by the latest source code/config files - this will ensure that the files are dealt with correctly and the server restarted as you have already noted. alan
A.L.M.Buxey@lboro.ac.uk wrote on 2016-06-17 15:04:
we want to dump all our requests into files, using the detail module. The files should be rotated (and archived) with the standard logrotate logrotate can handle this - you just need to ensure that you are using the options as provided by the latest source code/config files - this will ensure that the files are dealt with correctly and the server restarted as you have already noted.
which "latest"? 3.0.11 only has "nocreate" in the detail section in scripts/logrotate/freeradius, no server restart. Oh, and "copytruncate" (for radius.log) is definitely something we don't want to use, as it is prone to data loss. Looking more closely at the code, I wonder why HUP won't close the detail file(s). - rlm_detail is marked HUP_SAFE - on HUP, the instance memory is freed automatically, right? (mod_detach only frees the hash table) - exfile_init registers _exfile_free as destructor, so _exfile_free should be called when the instance is freed - _exfile_free closes all files Don't know what I'm missing here... PS: feel free to move this to freeradius-devel, not really a -users issue anymore, I guess :)
On Jun 17, 2016, at 10:26 AM, Jakob Hirsch <jh@plonk.de> wrote:
Looking more closely at the code, I wonder why HUP won't close the detail file(s).
HUP doesn't cause the modules to be reloaded. Only modules with the configuration changed will be reloaded. You can HUP an individual module. $ radmin -e "hup detail" Tho to be honest, I haven't tried that since the exfile code was added. It might work... Alan DeKok.
Alan DeKok wrote on 2016-06-17 17:48:
Looking more closely at the code, I wonder why HUP won't close the detail file(s). HUP doesn't cause the modules to be reloaded. Only modules with the configuration changed will be reloaded.
I thought I can trick FR to do that by touching detail's config file before sending HUP, but that didn't do anything.
You can HUP an individual module. $ radmin -e "hup detail" Tho to be honest, I haven't tried that since the exfile code was added. It might work...
Oh, that would be quite an acceptable workaround for me. I wanted to try that, but for some strange reason I cannot use the control socket feature. The servers seems to expect an IP address in the listener section:
Server was built with: accounting : yes authentication : yes ... control-socket : yes ... Server core libs: freeradius-server : 3.0.11 talloc : 2.0.* ssl : 1.0.1e release pcre : 8.30 2012-02-04 ... freeradius: #### Opening IP addresses and Ports #### listen { type = "control" /etc/freeradius/sites/control-socket[23]: No address specified in listen section
Config is just the (stripped) default from raddb/sites-available/control-socket: listen { type = control socket = ${run_dir}/${name}.sock } I also tried putting it into my "server default" or in a separate server, with the same result. I can add "ipaddr = *" (and some port setting) to make the error go away, but I will get an udp listener then, so that doesn't work either. Any hint on how to get this running? Regards Jakob
On Jun 20, 2016, at 7:39 AM, Jakob Hirsch <jh@plonk.de> wrote:
Alan DeKok wrote on 2016-06-17 17:48:
Looking more closely at the code, I wonder why HUP won't close the detail file(s). HUP doesn't cause the modules to be reloaded. Only modules with the configuration changed will be reloaded.
I thought I can trick FR to do that by touching detail's config file before sending HUP, but that didn't do anything.
That HUPs the detail file, but doesn't cause the exfile cache to be reloaded.
I wanted to try that, but for some strange reason I cannot use the control socket feature. The servers seems to expect an IP address in the listener section:
It works for me. Does the default configuration work? If so, then your edits broke something. Alan DeKok.
Alan DeKok wrote on 2016-06-20 15:02:
Looking more closely at the code, I wonder why HUP won't close the detail file(s). HUP doesn't cause the modules to be reloaded. Only modules with the configuration changed will be reloaded. I thought I can trick FR to do that by touching detail's config file before sending HUP, but that didn't do anything. That HUPs the detail file,
Hm, what exactly do you mean by that? I thought a HUP to the server would reload the config and all modules with a changed config file would be detached and re-instantiated.
but doesn't cause the exfile cache to be reloaded.
Btw, "hup detail" indeed does what we need, i.e. close the detail file and create a new one on the next request. Many thanks for that! Regards Jakob
Hi,
/etc/freeradius/sites/control-socket[23]: No address specified in listen section
not sites-enabled path? what have you been doing to your config?
Config is just the (stripped) default from raddb/sites-available/control-socket:
listen { type = control socket = ${run_dir}/${name}.sock }
works here: listen { type = control socket = ${run_dir}/${name}.sock uid = radiusd gid = radiusd mode = rw } ensure that the server has permissions to write to that ${run_dir} path? alan
A.L.M.Buxey@lboro.ac.uk wrote on 2016-06-20 15:18:
/etc/freeradius/sites/control-socket[23]: No address specified in listen section not sites-enabled path? what have you been doing to your config?
only replaced $INCLUDE sites-enabled/ in radiusd.conf with $INCLUDE sites/ config is deployed automatically, so there's no need to have sites-available and create symlinks in sites-enabled. Should really make no difference.
works here:
listen { type = control socket = ${run_dir}/${name}.sock uid = radiusd gid = radiusd mode = rw }
I already tried this (any many variants), with no success. This is plain 3.0.11, right, with no patches applied? Very strange. It behaves as I have type = auth or acct, but I don't see how this could happen...
ensure that the server has permissions to write to that ${run_dir} path?
I even tried /tmp/sock etc. Using strace, I don't see a syscall trying to use run_dir...
Jakob Hirsch wrote on 2016-06-20 16:48:
listen { type = control socket = ${run_dir}/${name}.sock uid = radiusd gid = radiusd mode = rw }
I already tried this (any many variants), with no success.
Ok, after recompilation (for debugging), it suddenly works. Don't know how or why. Sorry for the noise...
Hi,
I already tried this (any many variants), with no success.
This is plain 3.0.11, right, with no patches applied? Very strange. It behaves as I have type = auth or acct, but I don't see how this could happen...
plain 3.0.11, also plain 2.2.x that we mirgated from.
I even tried /tmp/sock etc. Using strace, I don't see a syscall trying to use run_dir...
${run_dir} is defined, yes? alan
Taking this to -devel, not really a -users topic any more. On 20.06.2016 17:17, A.L.M.Buxey@lboro.ac.uk wrote:
I already tried this (any many variants), with no success. This is plain 3.0.11, right, with no patches applied? Very strange. It behaves as I have type = auth or acct, but I don't see how this could happen... plain 3.0.11, also plain 2.2.x that we mirgated from.
So I fiddled around a bit and found the cause of this: I used configure --without-vmps in the previous build. Not sure how this is related, probably some side effect from the #ifdefs in src/main/listen.c. We use our own build (custom modules) and since we definitely won't need vmps and I prefer slim builds, I had it disabled. Furthermore, the build breaks using --without-dhcp (when trying to compile src/modules/proto_dhcp/dhcpclient.c) or --without-tcp (in src/main/client.c). Of course there's no harm leaving these options enabled, so I'll let them, but I guess that's not intended behaviour :)
participants (3)
-
A.L.M.Buxey＠lboro.ac.uk -
Alan DeKok -
Jakob Hirsch