Auth All but only for those in my clients.conf

Hello, been reading the list for months, first time posting. I am trying to setup a FreeRadius server which will basically authenticate all request which it receives from any IP in the clients.conf which has the correct secret. It seems that setting: DEFAULT Auth-Type := Accept in the users file accomplishes the AuthAll part but it is doing an auth all for any IP not just the ones in my clients.conf, also it doesn't seem to matter what the secret is set to on the client end, it still sends and accept. If I turn off the DEFAULT Auth-Type := Accept, my clients.conf including the secret is honored. Is there any way to accomplish what I am trying to do? Thanks in advance.