Expansion of \t and \n in message with linelong module and correct_escapes = true with v3.0.6
I have a 3.0.3 FR instance to which some additional logging has been added using the linelog module. The log messages that are written are multi-line and tab indented with \n and \t used in the format option in the linelog modules configuration file. I have also tried migrating another, older (v2.x) FR instance to v3.0.6. I started with a vanilla 3.0.6 configuration and successfully slowly modified that to give the same behaviour as the v2.x version. However when I finally attempted to add the additional logging that had been added to the 3.0.3 instance the log statements appearing in the log file are not multi-line or tab indented, neither the newline \n nor the tab \t have been expanded, \n and \t appear in the log file where I'd expected newlines and tabs to appear. I note in the changelog that for v3.0.5 correct_escapes was introduced and in a vanilla 3.0.6 configuration this is set to true Taking my 3.0.3 configuration as is and running it against a 3.0.6 server it behaves just as it does with a 3.0.3 server - expanding \t and \n . However if I modify the configuration to set correct_escapes to true then literal \t and \n appear in the log file rather than tabs and newlines. With 3.0.6, with correct_escapes=true, is it possible to use backslash escape sequences, or some equivalent to format linelog log messages? It maybe that the expansion of \n and \t in 3.0.3 was just undocumented behaviour that we stumbled upon and utilised rather than expected behaviour, the fact that the module is called line (singular) log may indicated we should not expect it to write multi-line log statements. I realise that I could just set correct_escapes=false in my 3.0.6 instances, I have briefly tried this however much of the default configuration now appears to assume the correct escapes and I'd have to unwind all those changes. I suspect if I did this I'd just be storing up trouble for myself some point in the future. Thanks,, Paul
On Dec 23, 2014, at 10:38 AM, paul.moser@bt.com wrote:
I have a 3.0.3 FR instance to which some additional logging has been added using the linelog module. The log messages that are written are multi-line and tab indented with \n and \t used in the format option in the linelog modules configuration file.
That’s not really the intended use of the “linelog” module. It logs lines of text.. not multiple lines.
I have also tried migrating another, older (v2.x) FR instance to v3.0.6. I started with a vanilla 3.0.6 configuration and successfully slowly modified that to give the same behaviour as the v2.x version. However when I finally attempted to add the additional logging that had been added to the 3.0.3 instance the log statements appearing in the log file are not multi-line or tab indented, neither the newline \n nor the tab \t have been expanded, \n and \t appear in the log file where I'd expected newlines and tabs to appear.
I note in the changelog that for v3.0.5 correct_escapes was introduced and in a vanilla 3.0.6 configuration this is set to true
It’s correct, and it creates many other simplifications of the source tree.
Taking my 3.0.3 configuration as is and running it against a 3.0.6 server it behaves just as it does with a 3.0.3 server - expanding \t and \n . However if I modify the configuration to set correct_escapes to true then literal \t and \n appear in the log file rather than tabs and newlines.
Yes. That’s really the correct behaviour. The reason is that attributes can have embedded \n in them, either as binary or a as a string “\n”. Using those attributes in the “linelog” module would then cause spurious new lines, and break your assumptions about what the log files mean.
With 3.0.6, with correct_escapes=true, is it possible to use backslash escape sequences, or some equivalent to format linelog log messages?
Not with 3.0.6. We can take a look at correcting it for the next release. We’ve made other fixes in 3.0 which made it easier to separate the “literal” strings passed in from the config files, and the “end user” strings taken from a RADIUS packet. We can have the linelog module convert \n -> CR for literal strings you enter, but not do that expansion for strings taken from RADIUS packets.
It maybe that the expansion of \n and \t in 3.0.3 was just undocumented behaviour that we stumbled upon and utilised rather than expected behaviour, the fact that the module is called line (singular) log may indicated we should not expect it to write multi-line log statements.
Both.
I realise that I could just set correct_escapes=false in my 3.0.6 instances, I have briefly tried this however much of the default configuration now appears to assume the correct escapes and I'd have to unwind all those changes. I suspect if I did this I'd just be storing up trouble for myself some point in the future.
I’d recommend keeping the new behaviour. It’s *much* more intuitive. Alan DeKok.
On Tue, Dec 23, 2014 at 11:05:23AM -0500, Alan DeKok wrote:
That’s not really the intended use of the “linelog” module. It logs lines of text.. not multiple lines.
Of course, "intended" and "what's useful" are two different things. We're also (ab-)using linelog to write out detail log format. These are then picked up by the detail reader and fed (as accounting packets) across to a central RADIUS server for common logging. It means we can set exactly which attributes to forward. Because of the way the logging and detail reader works it is also reliable if the central server goes away for some reason. I guess the alternative would be to have an "include" option for the detail module, countering the "suppress" option. But that's only for our use case.
Not with 3.0.6. We can take a look at correcting it for the next release. We’ve made other fixes in 3.0 which made it easier to separate the “literal” strings passed in from the config files, and the “end user” strings taken from a RADIUS packet. We can have the linelog module convert \n -> CR for literal strings you enter, but not do that expansion for strings taken from RADIUS packets.
That would seem the more sensible option - expand \n, \t etc in the string first, then expand %{..}. But I get that string expansion is pretty tricky in the ordering it needs to be done to get a sane output for all cases. Cheers Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
buffered_sql with extra attr_filter rule? alan -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
On Tue, Dec 23, 2014 at 11:48:11PM +0000, Alan Buxey wrote:
buffered_sql with extra attr_filter rule?
Yeah, there would be many ways to achieve the same thing. This is easy and cheap, and doesn't involve running any sort of SQL server. If I were setting it up now I'd probably look at something like a list in redis as a queue, or just feed data from the live RADIUS servers direct into elasticsearch and forget the central logging server. Happy Christmas :) Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
participants (4)
-
Alan Buxey -
Alan DeKok -
Matthew Newton -
paul.moser＠bt.com