Mixing pam and ldap

using freeradius 3 I have pam working well. I use pam to authenticate against winbind and google-authenticator. I now want to get the user groups from ldap as pam does not pass them along. I have ldap setup only in authorize. I see the user lookup performed and found but no group lookup is done. When is the group lookup performed? Can I mix ldap authorize with pam authenticate like this or is this not going to work? Is the ldap group lookup dependant on the user ldap authentication being successful?