rlm_kafka accounting module for Freeradius 3.x
Hello Community, The repo URL: https://github.com/thebinary/rlm_kafka I had created a FreeRADIUS 3.x module a year ago, for pushing Accounting Records to Kafka (uses librdkafka). I created a separate repo outside of FreeRADIUS project, which I used to compile separately by cloning inside the FreeRADIUS source 'src/modules' path. I would like to submit the module if considered useful by the community.
On May 19, 2024, at 10:49, The Binary <binary4bytes@gmail.com> wrote:
Hello Community,
The repo URL: https://github.com/thebinary/rlm_kafka
I had created a FreeRADIUS 3.x module a year ago, for pushing Accounting Records to Kafka (uses librdkafka). I created a separate repo outside of FreeRADIUS project, which I used to compile separately by cloning inside the FreeRADIUS source 'src/modules' path.
I would like to submit the module if considered useful by the community.
I believe there's already work underway to pull it in :) This was the one you were basing the v3 module on right Terry? -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS Development Team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On Sun, 19 May 2024 at 17:54, Arran Cudbard-Bell via Freeradius-Devel < freeradius-devel@lists.freeradius.org> wrote:
On May 19, 2024, at 10:49, The Binary <binary4bytes@gmail.com> wrote: I had created a FreeRADIUS 3.x module a year ago, for pushing Accounting Records to Kafka (uses librdkafka). I created a separate repo outside of FreeRADIUS project, which I used to compile separately by cloning inside the FreeRADIUS source 'src/modules' path.
I would like to submit the module if considered useful by the community.
I believe there's already work underway to pull it in :)
This was the one you were basing the v3 module on right Terry?
Indeed, I was looking at this over the last week to ensure that there is a path for inclusion of the module into FreeRADIUS. We are grateful for the FRv3 submission. We also need to ensure that we have a plan for FRv4... There's some things we need to get into shape for FRv3.2.x, mainly: 1. Durability of the events: The Kafka Producer in librdkafka is an asynchronous implementation that creates an in-application queue serviced by a dedicated IO thread. There exists the possibility for queued events to be lost (delivery failures, application crashes) after the module has returned OK (and FreeRADIUS acked the accounting event). Currently the module does not poll for delivery status reports (DSRs) for messages, resulting in silent data loss. We should at least log which messages were lost, even if FreeRADIUS is no longer handling the request. In v3 there is little that we can do to link FreeRADIUS's request handling to the Kafka queue. When a Kafka DSR is received the original request will have long gone. We may be able to integrate with the librdkafka-managed queue much tighter in FRv4 which is async by design (to be investigated) — slightly complicated because Kafka uses a polling interface to fetch notifications rather than an event-driven interface. So for FRv3 in order to match the usual durability properties for accounting messages, we likely want a default "synchronous mode" that does not tap into the full power of librdkafka (plugging of requests to allow batching, larger compression windows, etc.) but in which the module reports OK ( => Accounting-Response) only upon receiving a successful DSR (durably received by a set of in-sync brokers), otherwise FAIL. The user can then opt to enable async delivery, allowing >10x throughput, provided that they are prepared to relax the durability properties for accounting events. (In practise the user may take a hybrid approach, e.g. Accounting Start/Stops via a slow, durable synchronous route and Accounting I-Us via the fast asynchronous route: Selected by control attributes / separate module instances.) 2. Example module option sets (for sync vs async ; small vs large queues, i.e. risk appetite vs throughput), with careful descriptions of the durability model vs performance. 3. Integration with the FR build system: Autoconf configuration (library detection), etc. 4. Further testing and analysis to ensure that the implementation of the background IO thread does not cause issues for FreeRADIUS as a whole. 5. CI tests: End to end testing within GitHub actions. I have made progress with points 1 to 3. We (NetworkRADIUS) are currently performing a review prior to deciding to complete the work required to accept the module into the Open Source project. As soon as we have something further to share I will reach out.
On May 19, 2024, at 12:49 PM, The Binary <binary4bytes@gmail.com> wrote:
The repo URL: https://github.com/thebinary/rlm_kafka
I had created a FreeRADIUS 3.x module a year ago, for pushing Accounting Records to Kafka (uses librdkafka). I created a separate repo outside of FreeRADIUS project, which I used to compile separately by cloning inside the FreeRADIUS source 'src/modules' path.
I would like to submit the module if considered useful by the community.
We're always happy to accept patches, thanks. As Terry noted, there are a few minor change we need to make in order to have more confidence in the module. i.e. there can be a big difference between "it works for me", and "it works for a million people running different OS's, environments, etc". We hope to have updates for the module soon. Alan DeKok.
participants (4)
-
Alan DeKok -
Arran Cudbard-Bell -
Terry Burton -
The Binary