Alan DeKok wrote on 2016-06-17 14:57:
But I found now way to tell the daemon (or the detail module) to do that. As I read from the code, this will only happen when the file is unlinked or nothing was written for 30 seconds. gzip will unlink the ... Yes. The changes in v3 which make the detail module more thread-safe and faster do have this side effect.
Oh, ok. I saw that there is quite some code for this. We used our own module (originally developed for FR 1.x, later ported to 2.2.x) which simply did a open(data->dump_file, O_WRONLY|O_APPEND|O_CREAT,...) and close() for every packet, and never had threading or performance issues (ok, there are usually not more then 200 requests/s per server). Our module worked quite well, but I wanted to get rid of the custom modules (yes, there are more) as much as possible.
I know I can tell the detail module to separate the files by hour/day/whatever, but that would require significant changes in the subsequent processing, which I want to avoid (at least for now). Write files hourly. Then, have a cron job which concatenates them together in a format required by the later processing.
Ok, I should have mentioned that we also have some scripts that do some online log processing, which would also need modifications...