rlm_perl - unresponsive child threads

[Alejandro Perez-Mendez]

The queries normally run extremely fast, under ~0.05 sec, but the rPi hardware that it's running on sometimes experiences random hiccups that cause it to slow to a halt. It's hard to predict or prevent, and I don't have the option to upgrade the hardware in this particular case, so I'm stuck dealing with it & trying to find work-arounds.


  Hmm... that sounds bad.  Even a raspberry Pi is more powerful than a server from 20 years ago.  And people were running ISPs on that hardware.

  There's just no reason for a system to *block* for 30s.  That's terrible!



I would focus on debugging this. As Alan says, there is no reason for blocking for 30s, especially when it works fine for the rest of the time.

I can think of a few reasons for that to happen:
1. The RPi is broken (easy and cheap to fix, just replace it). Hint: check output of dmesg, journalctl, etc.
2. There is a huge IO bottleneck somewhere. For instance, sometimes disk caches flush and block the system until it's done. Or another process might be using your storage in intervals. You can definitely tweak your system to make more frequent flushes so you end up with a worse "normal" query  time, but avoid the eventual ultra-slow query that makes your system fail. Hint: use "iotop" to monitor IO usage.
3. There is swapping happening. If you are short of memory, your system might use SWAP memory and that implies moving stuff from RAM to disk. Hint: use "free" to check memory availability.
4. There is a huge CPU bottleneck. If you are running other processes on the RPi, they might be using too much of your CPU cycles from time to time, making your system slow. Hint: Use "top" to monitor CPU usage.

Hope this helps!

--
Alejandro Perez-Mendez
Technical Specialist (AAA), Trust & Identity
Skype alejandro_perez_mendez
jisc.ac.uk


Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under company number. 05747339, VAT number GB 197 0632 86. Jisc’s registered office is: 4 Portwall Lane, Bristol, BS1 6NB. T 0203 697 5800.


Jisc Services Limited is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under company number 02881024, VAT number GB 197 0632 86. The registered office is: 4 Portwall Lane, Bristol, BS1 6NB. T 0203 697 5800.


Jisc Commercial Limited is a wholly owned Jisc subsidiary and a company limited by shares which is registered in England under company number 09316933, VAT number GB 197 0632 86. The registered office is: 4 Portwall Lane, Bristol, BS1 6NB. T 0203 697 5800.


For more details on how Jisc handles your data see our privacy notice here: https://www.jisc.ac.uk/website/privacy-notice