Changes to VALUE_PAIR structure
I've pushed some changes to the VALUE_PAIR structure: - rearrange struct members to remove padding This saves ~8 bytes of memory on 64-bit machines - allocate only the needed part of the "data" field This prevents allocating 256 entry array for "integers" The result is that the VP structure is now 52 bytes for most common attributes (integer, ipaddr), instead of 312. For many VPs, that should save a *lot* of memory. I'm also investigating moving the "name", "attribute", and "vendor" fields out of the VP, and replacing them with a pointer to a DICT_ATTR. That should save only a small amount of memory, and may not be worth it. For a busy system managing ~10K requests, each having 10 attributes (request + control + reply), this means a savings of 25M of memory. That should in turn raise performance somewhat. I've taken a sweep through the code to remove everything which assumed you could print to vp_strvalue for an integer. It looks good, passes the tests, and shows no valgrind errors. So it should be OK. Let me know if you have any issues. I don't see much point in trying to reduce the memory footprint any further. What would help is allowing arbitrary length strings, but that requires changing a *lot* of code. Alan DeKok.
On 23 Nov 2011, at 14:40, Alan DeKok wrote:
I've pushed some changes to the VALUE_PAIR structure:
- rearrange struct members to remove padding This saves ~8 bytes of memory on 64-bit machines
- allocate only the needed part of the "data" field This prevents allocating 256 entry array for "integers"
The result is that the VP structure is now 52 bytes for most common attributes (integer, ipaddr), instead of 312. For many VPs, that should save a *lot* of memory.
I'm also investigating moving the "name", "attribute", and "vendor" fields out of the VP, and replacing them with a pointer to a DICT_ATTR. That should save only a small amount of memory, and may not be worth it.
I'd imagine that would require a lot of code changes too, there's no API for getting the name of the attribute, you just access the struct member directly. Though that's more of a grep and replace type thing that anything that requires thought.
Let me know if you have any issues.
I don't see much point in trying to reduce the memory footprint any further. What would help is allowing arbitrary length strings, but that requires changing a *lot* of code.
Mmm arbitrary length strings. -Arran Arran Cudbard-Bell a.cudbardb@freeradius.org Betelwiki, Betelwiki, Betelwiki.... http://wiki.freeradius.org/ !
On 11/23/2011 01:40 PM, Alan DeKok wrote:
I don't see much point in trying to reduce the memory footprint any further. What would help is allowing arbitrary length strings, but that requires changing a *lot* of code.
One thing I wondered about a while back was using a better allocator to help with things like this. In particular, "talloc" (GPL, small, used by samba) is very nice; you can do some clever things like: ptr = talloc(ctx, 10); sub = talloc(ptr, 10); ptr == talloc_parent(sub); /* useful in some cases */ talloc_free(ptr); /* also frees sub */ You can "break" the link if you need to move blocks around etc. Not sure if it would help with variable-length strings per se, but it might be preferable to using a dumb allocator if implementing it.
On 11/23/2011 10:41 AM, Phil Mayers wrote:
On 11/23/2011 01:40 PM, Alan DeKok wrote:
I don't see much point in trying to reduce the memory footprint any further. What would help is allowing arbitrary length strings, but that requires changing a *lot* of code.
One thing I wondered about a while back was using a better allocator to help with things like this. In particular, "talloc" (GPL, small, used by samba) is very nice; you can do some clever things like:
ptr = talloc(ctx, 10); sub = talloc(ptr, 10); ptr == talloc_parent(sub); /* useful in some cases */ talloc_free(ptr); /* also frees sub */
You can "break" the link if you need to move blocks around etc.
Not sure if it would help with variable-length strings per se, but it might be preferable to using a dumb allocator if implementing it.
FWIW, we've been using talloc in our SSSD (System Security Services Daemon [1]) implementation. talloc is packaged independently in Fedora. I know SSSD is being picked by Ubuntu so I expect you'll see talloc packages showing up in other distros too independent of Samba. I've worked with a number of packages which use "arenas" or "pools" for allocation like talloc, they have some very attractive aspects. [1] SSSD is a system daemon. Its primary function is to provide access to identity and authentication remote resource through a common framework that can provide caching and offline support to the system. Gives you all the wonders of remote centralized authentication (single sign-on) but a good user experience for laptops disconnected from the network, plus a ton of other features. -- John Dennis <jdennis@redhat.com> Looking to carve out IT costs? www.redhat.com/carveoutcosts/
John Dennis wrote:
FWIW, we've been using talloc in our SSSD (System Security Services Daemon [1]) implementation. talloc is packaged independently in Fedora. I know SSSD is being picked by Ubuntu so I expect you'll see talloc packages showing up in other distros too independent of Samba.
Yes, it's now available in a lot of places.
I've worked with a number of packages which use "arenas" or "pools" for allocation like talloc, they have some very attractive aspects.
Yes. Given that it's nearly 2011, some of the initial assumptions around FreeRADIUS are no longer true. i.e. it's probably OK to have a normative dependency on an external library. Alan DeKok.
Phil Mayers wrote:
One thing I wondered about a while back was using a better allocator to help with things like this. In particular, "talloc" (GPL, small, used by samba) is very nice; you can do some clever things like:
Yeah. Talloc would be nice. It's a lot more useful since they (a) split it out of Samba to be a stand-alone project, and (b) improved the performance to be something reasonable.
Not sure if it would help with variable-length strings per se, but it might be preferable to using a dumb allocator if implementing it.
It also means that memory allocations are more localized, which will likely help performance. Alan DeKok.
participants (4)
-
Alan DeKok -
Arran Cudbard-Bell -
John Dennis -
Phil Mayers