hi. 3.0.7 also seems to be leaking client proxy ssl state. First, where does SSL_free get called for the ssl created in tls_new_client_session? It seems that a talloc destructor is used for tls_new_session, but I don't see the logic that causes the session state to be freed for client sessions? Is there a missing set_destructor call? Secondly, I seem to be getting tls_new_client_session objects leaking some of the time. Causing errors like killing the process on the other end of the connection makes it more likely. I'm judging this because I do have a few sessions allocated on tls.c line 219 still around even after I've shut down the home server that the tls_session was for. Why is the TLS client session hanging off the configuration talloc context not a socket context of some kind?
On Mar 3, 2015, at 5:07 PM, Sam Hartman <hartmans@mit.edu> wrote:
First, where does SSL_free get called for the ssl created in tls_new_client_session? It seems that a talloc destructor is used for tls_new_session, but I don't see the logic that causes the session state to be freed for client sessions?
The underlying issue seems to be the function proxy_new_listener(). It called tls_new_client_session(), but didn’t set the parent properly. I’ll push a fix.
Secondly, I seem to be getting tls_new_client_session objects leaking some of the time. Causing errors like killing the process on the other end of the connection makes it more likely. I'm judging this because I do have a few sessions allocated on tls.c line 219 still around even after I've shut down the home server that the tls_session was for.
Yeah, it’s a parenting issue.
Why is the TLS client session hanging off the configuration talloc context not a socket context of some kind?
Bad code. I’ve pushed a fix. Alan DeKok.
Yeah, it¹s a parenting issue.
Why is the TLS client session hanging off the configuration talloc context not a socket context of some kind?
Bad code. I¹ve pushed a fix.
I'll do another build later and then re-check if this resolves the issue. Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc¹s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
Yeah, it’s a parenting issue. [...]
Bad code. I’ve pushed a fix.
Hi Alan, Just by the way... The old head of 3.0.x yesterday saw the RSS of the FR process grow by 2276K, today's (with your fix in), I see a growth of 356K. Does that sound credible? The system is being hit by one TLS-based Moonshot auth every 5 minutes. I'll leave it running for the next 12 hours or so (like I did the old head). I'll report back tomorrow. With Regards Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet Lumen House, Library Avenue, Harwell Oxford, Didcot, OX11 0SG jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
On Mar 4, 2015, at 1:13 PM, Stefan Paetow <Stefan.Paetow@jisc.ac.uk> wrote:
Just by the way... The old head of 3.0.x yesterday saw the RSS of the FR process grow by 2276K, today's (with your fix in), I see a growth of 356K.
That’s better.
Does that sound credible?
Yes. TBH, I have no idea how large the OpenSSL structures are, other than “OMG huge”.
The system is being hit by one TLS-based Moonshot auth every 5 minutes. I'll leave it running for the next 12 hours or so (like I did the old head). I'll report back tomorrow.
Thanks. Alan DeKok.
"Alan" == Alan DeKok <aland@deployingradius.com> writes:
Alan> On Mar 4, 2015, at 1:13 PM, Stefan Paetow <Stefan.Paetow@jisc.ac.uk> wrote: Alan> Yes. TBH, I have no idea how large the OpenSSL structures Alan> are, other than “OMG huge”. Leaking the ssl structs with 2 client sessions was around 300k so if it doesn't grow much beyond that, 356k could easily be plausible for no leaks we care about. --Sam
Leaking the ssl structs with 2 client sessions was around 300k so if it doesn't grow much beyond that, 356k could easily be plausible for no leaks we care about.
Alan, Sam, Unfortunately I had to restart the server at 8pm last night after realising that there were in fact no proper auth/proxy sessions that occurred (instead, the requests would abort). However, after restarting and checking it worked, looking at the memory usage this morning at 9:36, the FR RSS grew by 2980KB over 13.5 hours. Based on past data, this is a large improvement (where in 3.0.4 it would grow by that amount in around 2.5 hours). The server will continue to run some more for a 24h checkpoint. I hope that's useful. Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet Lumen House, Library Avenue, Harwell Oxford, Didcot, OX11 0SG jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
Leaking the ssl structs with 2 client sessions was around 300k so if it doesn't grow much beyond that, 356k could easily be plausible for no leaks we care about.
Alan, Sam, Unfortunately I had to restart the server at 8pm last night after realising that there were in fact no proper auth/proxy sessions that occurred (instead, the requests would abort). However, after restarting and checking it worked, looking at the memory usage this morning at 9:36, the FR RSS grew by 2980KB over 13.5 hours. Based on past data, this is a large improvement (where in 3.0.4 it would grow by that amount in around 2.5 hours). The server will continue to run some more for a 24h checkpoint. I hope that's useful. Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet Lumen House, Library Avenue, Harwell Oxford, Didcot, OX11 0SG jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
On Mar 5, 2015, at 6:12 AM, Stefan Paetow <Stefan.Paetow@jisc.ac.uk> wrote:
Unfortunately I had to restart the server at 8pm last night after realising that there were in fact no proper auth/proxy sessions that occurred (instead, the requests would abort). However, after restarting and checking it worked, looking at the memory usage this morning at 9:36, the FR RSS grew by 2980KB over 13.5 hours. Based on past data, this is a large improvement (where in 3.0.4 it would grow by that amount in around 2.5 hours). The server will continue to run some more for a 24h checkpoint.
Better, but still not very good. Leaking 2G of RAM is bad. Maybe some more valgrind runs would help. Alan DeKok.
Stefan, if you want to generate your own valgrind the command I'm using is valgrind --log-file=/tmp/valgrind.out --num-callers=50 --leak-check=full --show-reachable=yes freeradius -fxx -l stdout You'll need to make sure your freeradius and trust router have debugging symbols. On Debian, if you're building FR packages, in your build environm ent: export DEB_BUILD_OPTIONS='noopt nostrip' before the build. I think the default non-package build should be fine. For trust-router, simply install moonshot-trust-router-dbg As you know it'll be a couple of weeks before I get back to this.
You'll need to make sure your freeradius and trust router have debugging symbols. On Debian, if you're building FR packages, in your build environm ent:
I shall spin a new set with that in. :-)
As you know it'll be a couple of weeks before I get back to this.
Indeed. Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet Lumen House, Library Avenue, Harwell Oxford, Didcot, OX11 0SG jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
Better, but still not very good. Leaking 2G of RAM is bad. Maybe some more valgrind runs would help.
2MB... ;-) I have a single valgrind run for you... https://www.dropbox.com/sh/3assw6ltxqm84n7/AADESmqfd_gK9LI6StRIjxr3a?dl=0 Let me know what else you need, Alan. Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet Lumen House, Library Avenue, Harwell Oxford, Didcot, OX11 0SG jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
On Mar 5, 2015, at 11:45 AM, Stefan Paetow <Stefan.Paetow@jisc.ac.uk> wrote:
Better, but still not very good. Leaking 2G of RAM is bad. Maybe some more valgrind runs would help.
2MB... ;-)
Ah, that’s fine, then.
I have a single valgrind run for you... https://www.dropbox.com/sh/3assw6ltxqm84n7/AADESmqfd_gK9LI6StRIjxr3a?dl=0
Let me know what else you need, Alan.
There’s a lot of SSL and trust router stuff there. I’ll have to take a deeper look. Alan DeKok.
I have a single valgrind run for you... https://www.dropbox.com/sh/3assw6ltxqm84n7/AADESmqfd_gK9LI6StRIjxr3a?dl=0 There’s a lot of SSL and trust router stuff there. I’ll have to take a deeper look.
I've uploaded another two... both with --trace-children=yes, and one with -v also. Part of the summary in the latter was interesting in that it referred to 210K in 295 blocks indirectly lost. Anyway... I'm happy to do more runs with new code. Let me know. :-) Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet Lumen House, Library Avenue, Harwell Oxford, Didcot, OX11 0SG jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
On Mar 5, 2015, at 12:32 PM, Stefan Paetow <Stefan.Paetow@jisc.ac.uk> wrote:
I've uploaded another two... both with --trace-children=yes, and one with -v also. Part of the summary in the latter was interesting in that it referred to 210K in 295 blocks indirectly lost.
What I find a little odd is that there’s so much memory lost. The server tries hard to clean up all memory on exit. It looks like there are definite losses in the trust router code. There are “open” connection calls, but not “close” connection. I’m not really worried about that. But it’s odd that nothing’s being cleaned up on exit. That makes it much harder to track down real errors. Alan DeKok.
"Alan" == Alan DeKok <aland@deployingradius.com> writes:
Alan> On Mar 5, 2015, at 12:32 PM, Stefan Paetow <Stefan.Paetow@jisc.ac.uk> wrote: >> I've uploaded another two... both with --trace-children=yes, and >> one with -v also. Part of the summary in the latter was >> interesting in that it referred to 210K in 295 blocks indirectly >> lost. Alan> What I find a little odd is that there’s so much memory Alan> lost. The server tries hard to clean up all memory on exit. Alan> It looks like there are definite losses in the trust router Alan> code. There are “open” connection calls, but not “close” Alan> connection. I’m not really worried about that. Noticed that and am working on those. Alan> But it’s odd that nothing’s being cleaned up on exit. That Alan> makes it much harder to track down real errors. What I found is that if I let it run for 50,000 requests or so, looking at the highest usage possibly lost and definitely losts blocks tended to show the current problem.
Alan> What I find a little odd is that there¹s so much memory Alan> lost. The server tries hard to clean up all memory on exit.
:-/
Alan> But it¹s odd that nothing¹s being cleaned up on exit. That Alan> makes it much harder to track down real errors.
What I found is that if I let it run for 50,000 requests or so, looking at the highest usage possibly lost and definitely losts blocks tended to show the current problem.
Ok, I'll change the config to make the sockets pretty short-lived and then launch into a 50,000-request loop? Will that be useful? Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc¹s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
On Mar 5, 2015, at 6:58 PM, Stefan Paetow <Stefan.Paetow@jisc.ac.uk> wrote:
Ok, I'll change the config to make the sockets pretty short-lived and then launch into a 50,000-request loop? Will that be useful?
Yes, thanks. I’ll double-check that the sockets are cleaned up on exit. Alan DeKok.
On Mar 5, 2015, at 6:58 PM, Stefan Paetow <Stefan.Paetow@jisc.ac.uk> wrote:
Ok, I'll change the config to make the sockets pretty short-lived and then launch into a 50,000-request loop? Will that be useful?
Yes, thanks. I’ll double-check that the sockets are cleaned up on exit.
One thing that I have noticed is that in my current 50K-request loop (one request after the other, non-stop, no idle time between them), the memory usage creeps up slowly (between 09:26 and 12:09 the RSS grew by 2408K). I'll change the loop to include a 3 second delay between requests to watch connections being closed and then re-opened once this finishes and then check the RSS sizes again... :-) Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet Lumen House, Library Avenue, Harwell Oxford, Didcot, OX11 0SG jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
I’ll double-check that the sockets are cleaned up on exit.
Hi Alan, I've uploaded a new log, valgrind-25k-tidc-nonstop.out, to the same location as before. This run used 25K tidc requests without a break. Between 9:26 and 17:34, RSS increased by around 3MB. That's roughly what I guess you'd expect if the sockets are reused all the time. I've now restarted the loop with a 15 second delay between results to let the server print the below before the next request kicks in again: (159) Finished request Waking up in 0.2 seconds. Closing TLS socket from client port 48045 (0) >>> TLS 1.0 Alert [length 0002], warning close_notify Client has closed connection Waking up in 0.2 seconds. Waking up in 1.7 seconds. ... shutting down socket auth from client (127.0.0.1, 48045) -> (*, 2083, virtual-server=abfab-idp) (159) <done>: Cleaning up request packet ID 0 with timestamp +342 Waking up in 2.9 seconds. ... cleaning up socket auth from client (127.0.0.1, 48045) -> (*, 2083, virtual-server=abfab-idp) Ready to process requests Between 17:34 and 17:40, using the --trace-origins=yes setting in valgrind, RSS grew from 265220 to 290064... Dunno if this is valgrind specifically that's blowing up memory usage so spectacularly, but the trend I'm seeing every time I do the ps check is similar to what we've seen before. I'll check again later (I'm heading home now), and keep you in the loop. Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet Lumen House, Library Avenue, Harwell Oxford, Didcot, OX11 0SG jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
On Mar 6, 2015, at 12:50 PM, Stefan Paetow <Stefan.Paetow@jisc.ac.uk> wrote:
Between 17:34 and 17:40, using the --trace-origins=yes setting in valgrind, RSS grew from 265220 to 290064... Dunno if this is valgrind specifically that's blowing up memory usage so spectacularly, but the trend I'm seeing every time I do the ps check is similar to what we've seen before.
I'll check again later (I'm heading home now), and keep you in the loop.
It’s still hard to see what’s going on, because nothing is cleaned up. How are you stopping the server? CTRL-C? Can you set up a control socket, and use “radmin -e terminate” to stop the server? That should cause it to exit cleanly. Then, *only* leaked memory should show up on valgrind. If nothing shows up there, then the memory is being tracked, but not cleaned up on time. Alan DeKok.
It¹s still hard to see what¹s going on, because nothing is cleaned up. How are you stopping the server? CTRL-C? Can you set up a control socket, and use ³radmin -e terminate² to stop the server? That should cause it to exit cleanly. Then, *only* leaked memory should show up on valgrind.
I confess... Yes, Ctrl+C. I'll get on the control socket thing and try that instead. I'll restart the tests, sorry about the time wastage. :-/ Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc¹s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
It¹s still hard to see what¹s going on, because nothing is cleaned up. How are you stopping the server? CTRL-C? Can you set up a control socket, and use ³radmin -e terminate² to stop the server? That should cause it to exit cleanly. Then, *only* leaked memory should show up on valgrind.
There's now a valgrind-with-ctrl-sock.out. I'll run the once-a-minute test overnight (11 hours) and pull the valgrind log over tomorrow morning. Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc¹s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
Can you set up a control socket, and use ³radmin -e terminate² to stop the server? That should cause it to exit cleanly. Then, *only* leaked memory should show up on valgrind. I'll run the once-a-minute test overnight (11 hours) and pull the valgrind log over tomorrow morning.
Hi Alan, You should now have a valgrind-once-a-min-plus-socket-proxy-shutdown.out log. One thing I noticed that did not occur in other runs is that FR shuts down the socket proxy in this, and then cleans it up. Between the last run and me terminating FR with the control socket, the RSS grew by just under 100MB. Is it possible that it's the socket proxy that's losing memory? Just a thought. Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
On Mar 7, 2015, at 10:54 AM, Stefan Paetow <Stefan.Paetow@jisc.ac.uk> wrote:
You should now have a valgrind-once-a-min-plus-socket-proxy-shutdown.out log.
OK. I’ve fixed the complaints about uninitialized memory in fr_ipaddr_cmp. The trust router code should be using a library function to do ipaddr conversions, instead of doing there work itself.
One thing I noticed that did not occur in other runs is that FR shuts down the socket proxy in this, and then cleans it up. Between the last run and me terminating FR with the control socket, the RSS grew by just under 100MB. Is it possible that it's the socket proxy that's losing memory?
It’s likely. I pushed some fixes yesterday which SHOULD result in it cleaning up the proxy sockets on exit. But it looks like there’s still work to do. The good news is that the valgrind traces are useful. :) Alan DeKok.
I pushed some fixes yesterday which SHOULD result in it cleaning up the proxy sockets on exit. But it looks like there¹s still work to do.
Ok, I'll rebuild with the latest head and start another run with the same settings.
The good news is that the valgrind traces are useful. :)
And I've learnt something about *nix debugging, thank you :-) Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc¹s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
I pushed some fixes yesterday which SHOULD result in it cleaning up the proxy sockets on exit. But it looks like there¹s still work to do. The good news is that the valgrind traces are useful. :)
Good morning Alan, I have a new valgrind log for you, valgrind-308-0703152328-GMT.out.
From first glance it looks a lot better under valgrind. I'll restart the server in standard mode for another 12 hours and monitor. :-)
Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc¹s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
On Mar 8, 2015, at 8:19 AM, Stefan Paetow <Stefan.Paetow@jisc.ac.uk> wrote:
I have a new valgrind log for you, valgrind-308-0703152328-GMT.out.
From first glance it looks a lot better under valgrind. I'll restart the server in standard mode for another 12 hours and monitor. :-)
OK. I’m more concerned with ongoing process growth. Cleanup on exit is nice and good, but not extremely important. If the process grows slowly over time, that’s a more important issue. Alan DeKok.
OK. I¹m more concerned with ongoing process growth. Cleanup on exit is nice and good, but not extremely important. If the process grows slowly over time, that¹s a more important issue.
Well, Alan, Good news so far... I just checked and after 4 hours, there's no process growth. It's holding at 27600K RSS after starting at 27560K at the first request. I think you've nailed it. :-) Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc¹s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
OK. I¹m more concerned with ongoing process growth. Cleanup on exit is nice and good, but not extremely important. If the process grows slowly over time, that¹s a more important issue.
Ok Alan, After 10 hours there's still no growth in the RSS, so I'd say you've put a stake into that vampire ;-) Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc¹s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
participants (3)
-
Alan DeKok -
Sam Hartman -
Stefan Paetow