Just looking at graphite and wondering if anyone thinks a native rlm_graphite module to update counters using accounting data would be useful. One big advantage graphite has advantages over other systems in that it can be used to graph very large numbers of metrics. It could be used to produce traffic graphs for every user on a network for example. The module would implement the pickle protocol, and use a connection pool for TCP Connection caching. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
Writing to graphite (and influxdb) would be great. It is common to use statsd (or collectd) as an aggregation front-end to carbon, using a less efficient but easier to write plain-text protocol. Would that be already possible with freeradius, e.g. sending linelog to a udp socket? On 2015-02-18 03:27, Arran Cudbard-Bell wrote:
Just looking at graphite and wondering if anyone thinks a native rlm_graphite module to update counters using accounting data would be useful.
One big advantage graphite has advantages over other systems in that it can be used to graph very large numbers of metrics. It could be used to produce traffic graphs for every user on a network for example.
The module would implement the pickle protocol, and use a connection pool for TCP Connection caching.
On 18 Feb 2015, at 00:44, Zenon Mousmoulas <zmousm@noc.grnet.gr> wrote:
Writing to graphite (and influxdb) would be great. It is common to use statsd (or collectd) as an aggregation front-end to carbon, using a less efficient but easier to write plain-text protocol. Would that be already possible with freeradius, e.g. sending linelog to a udp socket?
Unsure. It may work already if you use something like netcat to create a named domain socket and then point linelog at that. There's nothing in the code that checks if the thing pointed to by filename is a file :) Collectd does have some useful/interesting transport capabilities. How is it used as an aggregator in this case? Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On 2015-02-18 15:45, Arran Cudbard-Bell wrote:
On 18 Feb 2015, at 00:44, Zenon Mousmoulas <zmousm@noc.grnet.gr> wrote:
Writing to graphite (and influxdb) would be great. It is common to use statsd (or collectd) as an aggregation front-end to carbon, using a less efficient but easier to write plain-text protocol. Would that be already possible with freeradius, e.g. sending linelog to a udp socket?
Unsure. It may work already if you use something like netcat to create a named domain socket and then point linelog at that. There's nothing in the code that checks if the thing pointed to by filename is a file :)
Collectd does have some useful/interesting transport capabilities. How is it used as an aggregator in this case?
There is a plugin for doing configurable aggregation of metrics. It's a different concept from statsd though. In any case the point was writing to statsd/collectd (different, but both plain-text protocols) rather than (or in addition to) writing to graphite. Z.
On 18/02/15 13:45, Arran Cudbard-Bell wrote:
Unsure. It may work already if you use something like netcat to create a named domain socket and then point linelog at that. There's nothing in the code that checks if the thing pointed to by filename is a file :)
Maybe not in the code directly, but it uses open() you can't open() a unix socket. This is why you'll see examples on the internet of people doing: echo foo | nc -U /tmp/unixsock ...or: echo foo | socat - UNIX-CONNECT:/tmp/unixsock
On 18 Feb 2015, at 11:25, Phil Mayers <p.mayers@IMPERIAL.AC.UK> wrote:
On 18/02/15 13:45, Arran Cudbard-Bell wrote:
Unsure. It may work already if you use something like netcat to create a named domain socket and then point linelog at that. There's nothing in the code that checks if the thing pointed to by filename is a file :)
Maybe not in the code directly, but it uses open() you can't open() a unix socket.
Ah yes, you need to connect() to it. Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On 18 Feb 2015, at 02:27, Arran Cudbard-Bell <a.cudbardb@freeradius.org> wrote:
The module would implement the pickle protocol, and use a connection pool for TCP Connection caching.
Pushing to Graphite wonder if some sort of message queue (AMQP?) might be more useful in general - graphite (and many other things, like Gralyog2 and log stash) should be able to pull messages from the queue. Would provide a layer of buffering between FreeRADIUS and any database. Thanks, Adam Bishop Systems Development Specialist gpg: 0x6609D460 t: +44 (0)1235 822 245 xmpp: adamb@jabber.dev.ja.net Janet, the UK's research and education network. Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
On Wed, Feb 18, 2015 at 08:11:26AM +0000, Adam Bishop wrote:
On 18 Feb 2015, at 02:27, Arran Cudbard-Bell <a.cudbardb@freeradius.org> wrote:
The module would implement the pickle protocol, and use a connection pool for TCP Connection caching.
Pushing to Graphite wonder if some sort of message queue (AMQP?) might be more useful in general - graphite (and many other things, like Gralyog2 and log stash) should be able to pull messages from the queue.
The other evening I wrote a logstash config to read and process detail files, which seems to work well. I'm planning to tidy it up and bundle it with a kibana dashboard and elasticsearch mapping for inclusion. The logstash config will then allow use of the graphite output of course, rather than feeding into elasticsearch. That said, an rlm_graphite might be useful for those who want to purely make graphs, without the cool log processing side that elasticsearch gives you as well. Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On 18 Feb 2015, at 05:45, Matthew Newton <mcn4@LEICESTER.AC.UK> wrote:
On Wed, Feb 18, 2015 at 08:11:26AM +0000, Adam Bishop wrote:
On 18 Feb 2015, at 02:27, Arran Cudbard-Bell <a.cudbardb@freeradius.org> wrote:
The module would implement the pickle protocol, and use a connection pool for TCP Connection caching.
Pushing to Graphite wonder if some sort of message queue (AMQP?) might be more useful in general - graphite (and many other things, like Gralyog2 and log stash) should be able to pull messages from the queue.
I'd rather do a 0MQ module TBH. From what i've seen at customer sites, high-speed distributed pub/sub models with no central brokers/routers/bridges, better fit requirements for message passing. It may be better to attempt to separate the modules that do marshalling/unmarshalling from the ones that handle transport.
The other evening I wrote a logstash config to read and process detail files, which seems to work well. I'm planning to tidy it up and bundle it with a kibana dashboard and elasticsearch mapping for inclusion.
Similar functionality to splunk, though i'm not sure splunk can output graphs, and it's not free of course.
The logstash config will then allow use of the graphite output of course, rather than feeding into elasticsearch.
That said, an rlm_graphite might be useful for those who want to purely make graphs, without the cool log processing side that elasticsearch gives you as well.
It would also be considerably faster. Don't suppose you want to contribute that config? :) -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On 18 Feb 2015, at 19:32, Arran Cudbard-Bell <a.cudbardb@freeradius.org> wrote:
I'd rather do a 0MQ module TBH. From what i've seen at customer sites, high-speed distributed pub/sub models with no central brokers/routers/bridges, better fit requirements for message passing.
Any message queue would be good, I've got no strong preference on which. They'd all work for buffering log files, aggregating securely from multiple sites, letting customers subscribe to their own log feeds, subscribing to particular "channels" of log output... :) Thanks, Adam Bishop gpg: 0x6609D460 jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
On Wed, Feb 18, 2015 at 02:32:51PM -0500, Arran Cudbard-Bell wrote:
On 18 Feb 2015, at 05:45, Matthew Newton <mcn4@LEICESTER.AC.UK> wrote:
The other evening I wrote a logstash config to read and process detail files, which seems to work well. I'm planning to tidy it up and bundle it with a kibana dashboard and elasticsearch mapping for inclusion.
Similar functionality to splunk, though i'm not sure splunk can output graphs, and it's not free of course.
Splunk can do graphs. It's also IME hideously expensive for any sort of production traffic.
The logstash config will then allow use of the graphite output of course, rather than feeding into elasticsearch.
That said, an rlm_graphite might be useful for those who want to purely make graphs, without the cool log processing side that elasticsearch gives you as well.
It would also be considerably faster. Don't suppose you want to contribute that config? :)
Yes, I'm doing it for contribution. Just needs tidying up first; shouldn't take long. Cheers Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
participants (5)
-
Adam Bishop -
Arran Cudbard-Bell -
Matthew Newton -
Phil Mayers -
Zenon Mousmoulas