So to confirm, your new module is basically files module, but it does NOT cache anything in directory, but re-read the files on disk for every request, is that correct?
Roughly correct. No caching, and reads on disk for every request. We rely on the block-buffer cache to make it go blisteringly quick. The "rough" part is that it isn't a replacement for the "files" module as such. Instead, it is a module you list in the config before "files". Our module then sets a value for use *by* the "files" module. The "files" module is still used exactly as is. I think Alan summarised it best as being similar to this: update request { My-Magic-Attr = "%{sql: SELECT .. from ..}" } Then in the "users" file the value of "My-Magic-Attr" can be used to select particular DEFAULT entries to return attributes. What is different about our module is that "update request" above would look something like this: update request { My-Magic-Attr = "%{read-line-from-file: /blah/%{Username}}" } But we wanted to avoid SQL and move to a file-based system as we had reached the end of our tether on SQL optimisation, budget, debugging, etc.
Using unlang, we then create a failsafe scenario, so if that a concurrent request comes that exceed the number of max sql thread, it would automatically accepted (i.e. basically Auth-Type=accept), but with a low timeout (e.g. 1 hour). That way the user can connect, but it'd reconnect and reauthenticate later when the system is (hopefuly) not-so-busy.
This is very good idea. Note that our problem was more about stability than raw performance. We still don't really know *why* we had the stability issues and are now relaxing with a beer because it's all gone away now :) Lazy? Yes. Happy? Very :)
Interesting. I wonder if we can hack a detail reader to behave similar, e.g.: - send "start transaction" - read lines from detail file - every 10 seconds or before deleting the detail file, send a "commit"
I suspect this would give all the benefits we gained by writing the events to a file and batch loading. Simpler too.