Slow Aceess-Reject if pasword id INVALID

Hi, For some reason I'm getting slow response of access-reject when user uses wrong passwords. I'm using MySQL and first I thought that the delay is due to db, but if user password is ok im getting access-accept in 1.9ms. Where if user pass is incorrect im getting access-rject only after 1 minute. And if I run radius in debug mode im getting system message that password is invalid like in 2ms and only after 4000ms in getting the access-reject on the screen Any ideas how to fix that? Thanks!