Troubleshooting apparent failed RADIUS challenges

[Turner, Ryan H]

Hoping this does not invoke a beating from Alan...

We are a LARGE EAP-TLS shop (one of the first), and authenticate 10s of thousands of clients every day.  I am noticing (which doesn't mean it is necessarily our fault), that some percentage of our users that cannot connect appear to be breaking down in the Challenge phase.  Initially it looks like they aren't even trying to authenticate on wireless (we don't log challenges), but a capture will tell me that  a challenge is made, and then there are no responses...

A few questions...  I honestly find the Radius Challenge section difficult to understand.  In our EAP-TLS environment, exactly what is happening during this challenge phase?  Doing searching on this has so far returned nebulous answers.  Exactly what is being checked of verified with an EAP-TLS client?

Secondly, how do you even begin to troubleshoot why certain clients would not progress beyond the challenge?  We have an android user that is onboarded with a certificate that will last a year.  Periodically, it will just stop working.  I will notice an incomplete challenge.  If he reonboards with a new certificate, everything works again for some period (obviously looks like a client issue).  We also notice that some users fail their challenge when traveling abroad with eduroam.

We are running one of the most patched 2.X versions of FreeRadius (we are actively building for a deployment of 3 at this moment).
Ryan Turner