[EXTERNAL] RE: seeking info on max.rate supported by dhcp.relay in v3.0.0.19

[Winfield, Alister]

Things to look at:

You may find that you need to look at the network interface configuration ... Last time I was coding tests against DHCP the "dumb" NIC regularly dropped packets because the ring buffer filled faster than the kernel was fetching them. This is also strongly influenced by how the interrupts are being optimised for example. (For really bursty traffic this can happen at relatively low packet rates). Obviously don't run the test client on the same server.

I'm also aware that you might be processing stuff too late when the ingest rate is beyond the processing speed per packet. If this happens then you almost always end up processing packets that will not be accepted by the client (it having timed out the request). This goes against tuning those queues too big. In practice if it's possible to drop old packets without processing further it can be surprisingly effective under extreme load. The assumption is that not processing allows more clients through the full DORA process thus the load is reduced. This improves load exponentially because less clients offline => less load => greater chance of processing the full set of packets.

Best thing to do is what is always the way with slowness. (teaching to suck eggs I'm sure).

Speed test with a slow ramp up of packets per second until you see some replies dropped or responses arriving after the normal client timeout. This is the fastest your server can cope without tuning. Now start the tuning game, change "one thing" then retest, repeat until you are sure there is no more to optimise or you reach your desired performance.

Oh and remember to check all dependencies. Eg. You might be using a database and the problems really the database under load.


A.


On 01/04/2019, 08:29, "Freeradius-Users on behalf of Katuri, Vikram" <freeradius-users-bounces+alister.winfield=sky.uk at lists.freeradius.org on behalf of Vikram.Katuri at viasat.com> wrote:

    Hi Alan,

    Thanks for the quick response and sorry for the delay in writing back.
    I've tested with the increased buffer sizes(in dhcp.relay server and also linux kernel) as below(25 MB) and still the result is same, couldn't go beyond 2000-2500 4 way transactions (CPU of all the cores on my relay server is fully occupied, which I've missed mentioning in the initial mail).


    # sysctl net.core.wmem_max
    net.core.wmem_max = 26214400
    # sysctl net.core.wmem_default
    net.core.wmem_default = 26214400
    # sysctl net.core.rmem_max
    net.core.rmem_max = 26214400
    # sysctl net.core.rmem_default
    net.core.rmem_default = 26214400



    server dhcp.eth1 {
            listen {
                    ipaddr = *
                    port = 67
                    type = dhcp
                    interface = eth1
                    recv_buff = 26214400


    Please advise on further steps/enhancements.
    Thanks in advance,
    Vikram.

    -----Original Message-----
    From: Freeradius-Users [mailto:freeradius-users-bounces+vikram.katuri=viasat.com at lists.freeradius.org] On Behalf Of Alan DeKok
    Sent: Thursday, March 28, 2019 4:56 PM
    To: FreeRadius users mailing list <freeradius-users at lists.freeradius.org>
    Cc: Chinnapaiyan, Nagamani <Nagamani.Chinnapaiyan at viasat.com>; Munhall, Damon <Damon.Munhall at viasat.com>
    Subject: Re: seeking info on max.rate supported by dhcp.relay in v3.0.0.19

    On Mar 28, 2019, at 7:19 AM, Katuri, Vikram <Vikram.Katuri at viasat.com> wrote:
    > May I please know the max.supported rate for dhcp.relay(with load balancing) in version 3.0.19.

      There is no pre-set limit on the rate of DHCP relay.  It depends on the OS, CPU, etc.

    > I see its getting capped at 5000 incoming discovers and only
    >              2500 of them are able to complete the 4 way DORA(via dhcp.relay).
    >              3700 with discover and request alone handled by the dhcp.relay and request and ack directly sent by the dhcp server to the client bypassing the relay(by not filling in the gateway address, server sends to the ciaddr).
    > Setup details:
    >              5 clients running perfdhcp each trying dhcp transactions at a rate of 1000/sec.
    >              Dhcp.relay on v3.0.19 running on c4.2xlarge instance (8 CPUs) on AWS cloud, 3 v4 dhcp servers behind it running on t2.medium instances on AWS cloud.

      That seems a bit low.  I've tested v3 at 20K packets/s for RADIUS proxying.

      One thing you may try changing is the "recv_buff" setting in the listener.  See sites-available/default for an example.  The same setting should work for DHCP, too.

      The default kernel receive buffer size is set very low.  If you increase it, UDP performance can go up significantly.

      Alan DeKok.


    -
    List info/subscribe/unsubscribe? See https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www.freeradius.org_list_users.html%26d%3DDwIGaQ%26c%3Djcv3orpCsv7C4ly8-ubDoUfrxF5xIGWmptxGWP5vi5w%26r%3DErCz_gtvq5KYouGyGCP2IPAR_J5Yv1HfOlLSlxMEpHA%26m%3DLWo4HS3j5Wmc3KEupfElllAKCu_Rs_NzJZk6bPXAHXY%26s%3D4AtWAQQkoDHdzAV-8zYRp1ciw-cVEmyXl_nLObXEWpU%26e&data=02%7C01%7Calister.winfield%40sky.uk%7Cce1d12fbd06e418295c608d6b66b550f%7C68b865d5cf184b2b82a4a4eddb9c5237%7C0%7C1%7C636896969459816168&sdata=B1rbiTIjFgjdQEtXx4LysxImySxrAR94iUgS1zw4oqo%3D&reserved=0=

    -
    List info/subscribe/unsubscribe? See https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.freeradius.org%2Flist%2Fusers.html&data=02%7C01%7Calister.winfield%40sky.uk%7Cce1d12fbd06e418295c608d6b66b550f%7C68b865d5cf184b2b82a4a4eddb9c5237%7C0%7C1%7C636896969459816168&sdata=3q76J2nQkw%2B1i0Qwff0DeGkefYMLVHSZFUBQ0xe%2B6Ts%3D&reserved=0
    --------------------------------------------------------------------
    This email is from an external source. Please do not open attachments or click links from an unknown or suspicious origin. Phishing attempts can be reported by sending them to phishing at sky.uk as attachments. Thank you
    --------------------------------------------------------------------



Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. SKY and the SKY marks are trademarks of Sky Limited and Sky International AG and are used under licence.

Sky UK Limited (Registration No. 2906991), Sky-In-Home Service Limited (Registration No. 2067075), Sky Subscribers Services Limited (Registration No. 2340150) and Sky CP Limited (Registration No. 9513259) are direct or indirect subsidiaries of Sky Limited (Registration No. 2247735). All of the companies mentioned in this paragraph are incorporated in England and Wales and share the same registered office at Grant Way, Isleworth, Middlesex TW7 5QD