Aliased IPs

I have a freeradius server which has multiple IPs aliased on the same interface. This works if I specify each IP explicitly in its own listen { } section but if I try to listen on * all responses are sent from the same IP regardless of which IP the request was received on.