On Apr 17, 2015, at 3:23 PM, Rygl Aleš <ales@rygl.net> wrote:
The goal is to have server which - responds as fast as possible. - is resilient to occasional peaks in the accounting traffic with magnitude of 10 or more - drops packets as little as possible - utilizes parallel processing on the DB side as much as possible
That's usually the goal. :)
I was not able to reach this with InnoDB engine with just sigle queue. It was pretty fast with MEMORY engine but it suffers from locking on table level (and uses just a single connection to DB). I was affraid that talking to DB directly (without fallback to a file buffer) would had problems in case of peaks. So the idea behind is to rather write to a file which IHMO cheaper that perform a DB transaction where I have to wait for a lock.
Hmm... sounds like DB issues, to be honest. Your other message mentioned 3-5s delay writing to the DB. That should *never* happen. A normal DB should be able to handle 200-1000 writes per second.
I can confirm that it is definitely faster. I have followed Arran's recomendations and reduced the number of queues to 12 and it looks good. I will play with this more.
That's good. You should be able to do: accounting { ... redundant { sql detail } ... } And carefully tune the "pool" parameters to the SQL module. If all of the connections are in use, the server will fail-over to using the "detail" module. This process is very quick, so there's no performance issues.
I will try again. Actually I have two more virtual servers in the config with a single queue processing just couple of packets per second and this error was there too. I will do some debugs here.
Maybe try the v3.0.x branch. We'll turn it into 3.0.8 real soon now... Alan DeKok.