Re: fr_packet_cmp again
Josip Almasi wrote:
Alan DeKok wrote:
Grab the "master" branch. See http://git.freeradius.org for instructions. The internal state machine has been *completely* re-written so that it's sane. If the bug still exists there, I'll be shocked.
Will try.
Died 7 mins after with Error: ASSERT FAILED threads.c[423]: request->magic == REQUEST_MAGIC :/ Started with -s, segfault in libfreeradius-radius-3.0.0.so. Dang its still there!:)))) -------------------------- Program received signal SIGSEGV, Segmentation fault. fr_packet_cmp (a=0xda9130, b=0x6e6f6b7369050d5b) at packet.c:141 141 if (a->sockfd < b->sockfd) return -1; #0 fr_packet_cmp (a=0xda9130, b=0x6e6f6b7369050d5b) at packet.c:141 #1 0x00007ffff7bc0adb in list_find (ht=0x841840, data=0x7fffffffe158) at hash.c:191 #2 fr_hash_table_find (ht=0x841840, data=0x7fffffffe158) at hash.c:454 #3 0x00007ffff7bc0b29 in fr_hash_table_finddata (ht=<value optimized out>, data=<value optimized out>) at hash.c:484 #4 0x00007ffff7bd067a in fr_packet_list_find (pl=<value optimized out>, request=0xda9130) at packet.c:577 #5 0x0000000000428203 in request_receive (listener=0x858090, packet=0xda9130, client=0x7a3050, fun=0x407f60 <rad_accounting>) at process.c:1181 #6 0x0000000000415bb2 in acct_socket_recv (listener=0x858090) at listen.c:1382 #7 0x00000000004295ca in event_socket_handler (xel=<value optimized out>, fd=<value optimized out>, ctx=0x858090) at process.c:3125 #8 0x00007ffff7bd141b in fr_event_loop (el=0x841e00) at event.c:413 #9 0x000000000041d334 in main (argc=<value optimized out>, argv=<value optimized out>) at radiusd.c:412 (gdb) print a=0xda9130 $1 = (const RADIUS_PACKET *) 0xda9130 (gdb) print $1->sockfd $2 = 9 (gdb) print b=0x6e6f6b7369050d5b $3 = (const RADIUS_PACKET *) 0x6e6f6b7369050d5b (gdb) print $3->sockfd Cannot access memory at address 0x6e6f6b7369050d5b -------------------------- Still running another instance of 1.2.11 under valgrind memtest, some 3M packets later it's still fine. Could it be some sort of timing issue, i.e. I have faster box than you so you cant reproduce it? FTR I have 16 cores, 4 x Xeon 2.8 GHz. BTW I like what I saw in 3, response time under heavy load is way better. Regards...
Josip Almasi wrote:
Died 7 mins after with Error: ASSERT FAILED threads.c[423]: request->magic == REQUEST_MAGIC :/
That's terrible. But at least it helps track down where the problem is. It's starting to look like the problem is in the hash table code... there isn't much that's the same between 2.1 and 3.0.
Could it be some sort of timing issue, i.e. I have faster box than you so you cant reproduce it?
Yes.
BTW I like what I saw in 3, response time under heavy load is way better.
That's good to hear. I'd like to see graphs, if at all possible... I'm not sure *why* it's better, but the reason could very well be that the code and algorithms are substantially cleaner. Alan DeKok.
Alan DeKok wrote:
Josip Almasi wrote:
Died 7 mins after with Error: ASSERT FAILED threads.c[423]: request->magic == REQUEST_MAGIC :/
That's terrible. But at least it helps track down where the problem is.
Care to elaborate? But this terrible thing is up to RHEL 6, maybe kernel issue. I mentioned it earlier as non-critical. After a few hundered thousands threads, no new threads can start. Be it freeradius or java threads. I kill radiusd and restart and it still cant start new threads - clone returns resource temporary unavailable. Workaround is using fixed number of threads, as you explained sometimes. To make it even wierder, if we set radiusd uid and gid to 0, it works:) I have no clue how to set this up, sure it's not up to ulimit.
Could it be some sort of timing issue, i.e. I have faster box than you so you cant reproduce it?
Yes.
Want to try it on our boxes? We could set you up an openvpn account. Or, whatever, it's lab anyway, we'll nat it to internet and give you ssh.
BTW I like what I saw in 3, response time under heavy load is way better.
That's good to hear. I'd like to see graphs, if at all possible...
Sorry no graphs. But we narrowed it down to rlm_detail. For auth requests, performance degrades less, to max 20 ms response time. Say, with about 2K auth req/s, it's still about 2ms, quite nice. But for acct it can get to more than a second. When we turn off detail, it's the same as for acct requests. But when we detail to /dev/null, response grows 40 ms. With detailing to files, we got additional 80-120 ms. It was at about 5K acct req/s. All this, IIRC. We may do it again if you're interested. OTOH, FR 3 keeps response at 1-2ms at all times. Regards...
Josip Almasi wrote:
That's terrible. But at least it helps track down where the problem is.
Care to elaborate?
If it's still the same issue in 2.1.x and 3.0, then it's not the internal state machine. I thought it was, but it looks like it's the hash tables.
But this terrible thing is up to RHEL 6, maybe kernel issue. I mentioned it earlier as non-critical. After a few hundered thousands threads, no new threads can start. Be it freeradius or java threads.
Ouch. The only work-around is to set the number of threads high, and to set the number of "spare threads" high. That way few threads will be created or destroyed.
Workaround is using fixed number of threads, as you explained sometimes. To make it even wierder, if we set radiusd uid and gid to 0, it works:) I have no clue how to set this up, sure it's not up to ulimit.
It does sound like ulimit...
Want to try it on our boxes? We could set you up an openvpn account. Or, whatever, it's lab anyway, we'll nat it to internet and give you ssh.
Maybe next week. I'm busy this week.
Sorry no graphs. But we narrowed it down to rlm_detail. For auth requests, performance degrades less, to max 20 ms response time. Say, with about 2K auth req/s, it's still about 2ms, quite nice. But for acct it can get to more than a second.
Wow... that's really horrible. I have no idea why that's happening.
When we turn off detail, it's the same as for acct requests. But when we detail to /dev/null, response grows 40 ms. With detailing to files, we got additional 80-120 ms. It was at about 5K acct req/s. All this, IIRC. We may do it again if you're interested.
That's astonishing. I don't know why that's happening.
OTOH, FR 3 keeps response at 1-2ms at all times.
Much, much, better. Alan DeKok.
Hi,
Workaround is using fixed number of threads, as you explained sometimes. To make it even wierder, if we set radiusd uid and gid to 0, it works:) I have no clue how to set this up, sure it's not up to ulimit.
ulimit -u unlimited ?
OTOH, FR 3 keeps response at 1-2ms at all times.
nice. having looked at the old state machine diagram it looks like the clean layout of the new state machine is definately streamlining behaviour alan
Alan Buxey wrote:
OTOH, FR 3 keeps response at 1-2ms at all times.
nice. having looked at the old state machine diagram it looks like the clean layout of the new state machine is definately streamlining behaviour
How? I'm not arguing, but I don't understand... Alan DeKok.
Alan Buxey wrote:
Hi,
Workaround is using fixed number of threads, as you explained sometimes. To make it even wierder, if we set radiusd uid and gid to 0, it works:) I have no clue how to set this up, sure it's not up to ulimit.
ulimit -u unlimited
?
Nah it's not ulimit thing, it's first thing we did:) And even if it was, killing and restarting process would clean up threads, giving new process fresh start. Only way to get rid of 'resource busy' was server restart. Regards...
Alan DeKok wrote:
It's starting to look like the problem is in the hash table code...
So it was hastables after all. As for performance decay I mentioned earlier, forget it, it was netfilter and such:) Once we got it working and optimized, max response time is 2ms:) Thanks again. Regards...
On 29.04.2011 19:59, Josip Almasi wrote:
It's starting to look like the problem is in the hash table code... So it was hastables after all.
So with the replacement of the hash-table with rb-tree the crash does not happen any more? That would be _really_ nice, since we are hit by this every hours to days (no pattern recognizable). I guess there could be a problem with the handling of hash collisions. How about performance of rbtree? I'm not a data structure expert, but AFAIR btrees are not ideal if you have to often add new nodes (rbtrees seem to be much better at that than ordinary btrees, though). Not that it matters much, the real bootleneck is probably our mysql backend. @Alan: The rbtree code is only in the master branch, right? Any chance to get this in 2.1.x?
Jakob Hirsch wrote:
I guess there could be a problem with the handling of hash collisions.
The code was tested for that.
How about performance of rbtree? I'm not a data structure expert, but AFAIR btrees are not ideal if you have to often add new nodes (rbtrees seem to be much better at that than ordinary btrees, though). Not that it matters much, the real bootleneck is probably our mysql backend.
It shouldn't matter much. Once you use a DB or EAP, the performance of the RADIUS server is irrelevant.
@Alan: The rbtree code is only in the master branch, right? Any chance to get this in 2.1.x?
There was a patch for master. I'll put something into both branches for 2.1.11. Alan DeKok.
Josip Almasi wrote:
So it was hastables after all.
Wow... that's weird. But at least it's a solution.
As for performance decay I mentioned earlier, forget it, it was netfilter and such:)
OK. That makes a bit more sense.
Once we got it working and optimized, max response time is 2ms:)
Good. I'll be sure to make the change for 2.1.11. Alan DeKok.
participants (4)
-
Alan Buxey -
Alan DeKok -
Jakob Hirsch -
Josip Almasi