event.c assertion in pre2....
hi, one of our FR server processes died this morning. quick radius.log check showed: Mon Sep 17 11:24:09 2007 : Error: Assertion failed in event.c, line 842 now, this was running a slightly old 2.0pre2 CVS checkout and i've now updated the codebase to todays available 2.0pre2 code and the server is running - but I just wanted to check...i recall something about this being mentioned on-list a week or 2 back..and i saw from cvs update that event.c has been edited... is the belief that this assertion problem has been sorted/fixed or should I be looking for something more here? alan
A.L.M.Buxey@lboro.ac.uk wrote:
hi,
one of our FR server processes died this morning. quick radius.log check showed:
Mon Sep 17 11:24:09 2007 : Error: Assertion failed in event.c, line 842
Uh... which assertion was that?
now, this was running a slightly old 2.0pre2 CVS checkout and i've now updated the codebase to todays available 2.0pre2 code and the server is running - but I just wanted to check...i recall something about this being mentioned on-list a week or 2 back..and i saw from cvs update that event.c has been edited... is the belief that this assertion problem has been sorted/fixed or should I be looking for something more here?
If we know what assertion it is, that could help know what to do. Alan DeKok.
Hi,
Uh... which assertion was that?
without full debugging I coudlnt say much more than what was in the basic logfile - we dont run production servers with ful debugging enabled by default due to overheads, lack of historical logs etc Mon Sep 17 11:24:09 2007 : Info: rlm_eap_tls: Length Included Mon Sep 17 11:24:09 2007 : Auth: Login incorrect: [host/xxxxxx-Lap1] (from client wism port 29 cli 01-18-F5-0B-C2-13) Mon Sep 17 11:24:09 2007 : Auth: Login incorrect: [anonymous] (from client wism port 29 cli 01-18-F5-0B-C2-13) Mon Sep 17 11:24:09 2007 : Error: Assertion failed in event.c, line 842 alan
A.L.M.Buxey@lboro.ac.uk wrote:
Uh... which assertion was that?
without full debugging I coudlnt say much more than what was in the basic logfile - we dont run production servers with ful debugging enabled by default due to overheads, lack of historical logs etc
If you still have the sources you used to build that version, knowing the contents of line 842 would help. Alan DeKok.
Hi,
If you still have the sources you used to build that version, knowing the contents of line 842 would help.
ah yes. that i can do: main/event.c case REQUEST_PROXIED: rad_assert(request->next_callback != NULL); request->when = request->next_when; callback = request->next_callback; request->next_callback = NULL; break; line 842 is the rad_assert line I note that the latest code i'm running hasnt changed this section... alan
A.L.M.Buxey@lboro.ac.uk wrote:
case REQUEST_PROXIED: rad_assert(request->next_callback != NULL);
Dang.
I note that the latest code i'm running hasnt changed this section...
OK. There's some assumption that's wrong in the system. i.e. wrong logic, or a race condition, or something. The assertion (while annoying) is *much* more helpful than a simple core dump. I'll take a look... I'm traveling this week, so I don't know how much time I have to look at this. Alan DeKok.
A.L.M.Buxey@lboro.ac.uk wrote:
If you still have the sources you used to build that version, knowing the contents of line 842 would help.
I've updated the assertion macro to print it's contents, too.
ah yes. that i can do:
main/event.c
case REQUEST_PROXIED: rad_assert(request->next_callback != NULL); ... I note that the latest code i'm running hasnt changed this section...
I've just taken a sweep through the code, clarifying it, and adding a few more assertions. Hopefully this should work, but I don't really understand why it hit that assertion before... Alan DeKok.
Alan DeKok wrote:
A.L.M.Buxey@lboro.ac.uk wrote:
If you still have the sources you used to build that version, knowing the contents of line 842 would help.
I've updated the assertion macro to print it's contents, too.
ah yes. that i can do:
main/event.c
case REQUEST_PROXIED: rad_assert(request->next_callback != NULL);
...
This is strange, were running 2.0 cvs head on our production servers which are now under varied and reasonable load, and we haven't noticed them going down with this. They are however leaking memory at an alarming rate. After 24hrs up, prvt memory usage was ~100mb. We are reloading the servers every night to combat this, so we don't urgently need a solution, but it would be nice to get this sorted, especially before releasing 2.0 final. Alan you are free to use the Test MAC uf you want to run any code. Just we can't run valgrind on the macs, as no ones done a port of it.
I note that the latest code i'm running hasnt changed this section...
I've just taken a sweep through the code, clarifying it, and adding a few more assertions.
Hopefully this should work, but I don't really understand why it hit that assertion before...
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/devel.html
-- Arran Cudbard-Bell (A.Cudbard-Bell@sussex.ac.uk) Authentication, Authorisation and Accounting Officer Infrastructure Services | ENG1 E1-1-08 University Of Sussex, Brighton EXT:01273 873900 | INT: 3900
Arran Cudbard-Bell wrote:
This is strange, were running 2.0 cvs head on our production servers which are now under varied and reasonable load, and we haven't noticed them going down with this.
That's good.
They are however leaking memory at an alarming rate. After 24hrs up, prvt memory usage was ~100mb.
That's bad. When did this start happening? Or did CVS head always leak memory? Did 1.1.x leak memory? Alan DeKok.
Alan DeKok wrote:
Arran Cudbard-Bell wrote:
This is strange, were running 2.0 cvs head on our production servers which are now under varied and reasonable load, and we haven't noticed them going down with this.
That's good.
Makes me smile :) We've now done 20k authentications on one server. With a mixture of PAP for login to switches, the major eap flavours (EAP-TTLS,EAP-PEAP,EAP-TLS), and whatever the JRS test script uses. We have Linux Boxes authenticating with wpa_supplicant, we have Macs using Internet Connect, We have 2000,XP,Vista Boxes (Though Vista is Evil and very stubborn) using the built in supplicant software. Authentication is done using LDAP. Authorisation is done using SQL and unlang only.
They are however leaking memory at an alarming rate. After 24hrs up, prvt memory usage was ~100mb.
That's bad.
When did this start happening? Or did CVS head always leak memory? Did 1.1.x leak memory?
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/devel.html
1.1.* as far as I can recall, didn't leak memory. I don't think early CVS head versions leaked memory either. -- Arran Cudbard-Bell (A.Cudbard-Bell@sussex.ac.uk) Authentication, Authorisation and Accounting Officer Infrastructure Services | ENG1 E1-1-08 University Of Sussex, Brighton EXT:01273 873900 | INT: 3900
Arran Cudbard-Bell wrote: ...
Authorisation is done using SQL and unlang only.
Nice.
1.1.* as far as I can recall, didn't leak memory. I don't think early CVS head versions leaked memory either.
If you could offer any assistance as to when it started leaking, that would help. Otherwise, I've run it through valgrind for a number of tests, and don't see anything. Alan DeKok.
participants (3)
-
A.L.M.Buxey＠lboro.ac.uk -
Alan DeKok -
Arran Cudbard-Bell