detail logs with filebeat, elasticsearch and Kibana

[Lineconnect]

Hi there,

i found a nice way to get the detailed logs of FreeRADIUS into elasticsearch and want to share it with you: 

After reading some of the old threads/presentations(e.g. http://lists.freeradius.org/pipermail/freeradius-users/2018-April/091415.htm or https://www.slideshare.net/MatthewNewton28/freeradius-eduroam-logging-and-elasticsearchl)  i pretty much copied this approach, 
to get the detail log into a json format because there is currenlty no module for the default FreeRADIUS logs. It works in general very good but has the disadvantage to get empyt fields 
in elasticsearch because not all the fields filled at each time, when the linelog module was called(may just my problem) and you properly will not see any vendor specific RADIUS attribute as long you're not updating your linelog module. 

The default FreeRADIUS detail log does only contain key-value pairs which are filled(obviously). To let FreeRADIUS write the detail log with the linemodule you can do something like this(found in the eduroam documentation(https://wiki.freeradius.org/guide/eduroam#configuration_the-outer-virtual-server_mods-available-linelog): 

````
linelog log_reply {
        filename = ${logfile}
        permissions = 0644
        reference = "messages.%{%{reply:Packet-Type}:-format}"
        messages {
               Access-Accept = "%{pairs:request:}"
               Access-Reject =  "%{pairs:request:}"
        }
}
````


Now the log which is written by linelog looks like this: 
````
User-Name = "anonymous", NAS-IP-Address = 127.0.0.1, Calling-Station-Id = "02-00-00-00-00-01", Framed-MTU = 1400, NAS-Port-Type = Wireless-802.11, Service-Type = Framed-User, Connect-Info = "CONNECT 11Mbps 802.11b", EAP-Message = 0x0200000e01616e6f6e796d6f7573, Message-Authenticator = 0x77ad754adf2d2dee13cfc20cb5f02c74, EAP-Type = Identity
````


To get all of the attributes parsed in elasticsearch as field and value you can use a ingest pipeline and the key-value module
- https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest.html
- https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html

The api request to elastic could be look like this: 
````
PUT _ingest/pipeline/radius-log-parsing
{
  "description" : "Parse radius log",
  "processors" : [
    {
      "kv": {
        "field": "message",
        "field_split": ", ",
        "value_split": " = "
      }
    }
  ]
}
````


And if you're using filebeat as logshipper, you can reference that pipeline in your input configuration, so it could look like this: 
````
- type: log 
  paths: 
    - /var/log/freeradius/radius-detail.log
  ignore_older: 5m
  pipeline: "radius-log-parsing"
  tags: ['detail']
````


Hope this helps someone how's trying to get dynamic field in elastic for the FreeRADIUS detail log. 


Best regards, 
Lineconnect