rotating detail files

[Jakob Hirsch]

Alan DeKok wrote on 2016-06-17 14:57:
>> But I found now way to tell the daemon (or the detail module) to do
>> that. As I read from the code, this will only happen when the file is
>> unlinked or nothing was written for 30 seconds. gzip will unlink the
...
>   Yes.  The changes in v3 which make the detail module more thread-safe and faster do have this side effect.

Oh, ok. I saw that there is quite some code for this. We used our own
module (originally developed for FR 1.x, later ported to 2.2.x) which
simply did a open(data->dump_file, O_WRONLY|O_APPEND|O_CREAT,...) and
close() for every packet, and never had threading or performance issues
(ok, there are usually not more then 200 requests/s per server).
Our module worked quite well, but I wanted to get rid of the custom
modules (yes, there are more) as much as possible.

>> I know I can tell the detail module to separate the files by
>> hour/day/whatever, but that would require significant changes in the
>> subsequent processing, which I want to avoid (at least for now).
>   Write files hourly.  Then, have a cron job which concatenates them together in a format required by the later processing.

Ok, I should have mentioned that we also have some scripts that do some
online log processing, which would also need modifications...