Different users file per virtual server

Is there a way to have seperate "users" files per virtual server? The location of the users file appears to be specified in modules/files. I can only seem to load modules from the global radiusd.conf, so each server uses the same users file. My goal is to use a different users file based on the IP of the client. -CJ